Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > microsoft.public.scripting.vbscript > #12104 > unrolled thread

Textstream Write

Back to article view | Back to microsoft.public.scripting.vbscript

Contents

  Textstream Write "Mayayana" <mayayana@invalid.nospam> - 2019-04-29 08:55 -0400
    Re: Textstream Write JJ <jj4public@vfemail.net> - 2019-04-30 17:45 +0700
      Re: Textstream Write "R.Wieser" <address@not.available> - 2019-04-30 13:21 +0200
    Re: Textstream Write "Mayayana" <mayayana@invalid.nospam> - 2019-04-30 08:51 -0400
      Re: Textstream Write "R.Wieser" <address@not.available> - 2019-04-30 18:22 +0200
        Re: Textstream Write "Mayayana" <mayayana@invalid.nospam> - 2019-04-30 13:01 -0400
          Re: Textstream Write "R.Wieser" <address@not.available> - 2019-04-30 21:37 +0200
            Re: Textstream Write "Mayayana" <mayayana@invalid.nospam> - 2019-04-30 17:06 -0400
              Re: Textstream Write "R.Wieser" <address@not.available> - 2019-05-01 10:01 +0200
                Re: Textstream Write "Mayayana" <mayayana@invalid.nospam> - 2019-05-01 08:37 -0400
                  Re: Textstream Write "R.Wieser" <address@not.available> - 2019-05-01 16:00 +0200
                    Re: Textstream Write "R.Wieser" <address@not.available> - 2019-05-01 16:12 +0200
                      Re: Textstream Write "Mayayana" <mayayana@invalid.nospam> - 2019-05-01 10:47 -0400
                        Re: Textstream Write "R.Wieser" <address@not.available> - 2019-05-01 19:09 +0200
    Re: Textstream Write "Mayayana" <mayayana@invalid.nospam> - 2019-05-04 15:26 -0400
      Re: Textstream Write JJ <jj4public@vfemail.net> - 2019-05-05 20:15 +0700
        Re: Textstream Write "Mayayana" <mayayana@invalid.nospam> - 2019-05-05 10:19 -0400

#12104 — Textstream Write

  Very weird problem. I've been working on an HTA that
allows me to touch-up edit webpages, in WYSIWYG manner.
It starts by loading what's in the BODY into a DIV, then
storing the header and end. When I save changes it takes
the innerHTML from the DIV, adds back the header and end,
and writes to disk with normal Textstream operation.

   The whole thing works fine but I was testing it on various
files and came across one (an article from theregister.co.uk
that has no particular odd qualities) that won't work.

  I load the page fine but saving changes results in a blank file.
Empty. If I show the 3 parts in msgbox they all look right. If
I right each part to disk, the BODY section is empty but both
ends write OK. When I write the whole file it writes the blank
file and then errors with "invalid procedure call or argument".
That seems to be at Write, though I'm not certain because error
line numbers can be off. The code is in an external VBS.

  sMid = MainDiv.innerHTML
  sContent = sBeg & sMid & sEnd

Then it calls the Write routine in a class. All very vanilla:

Public Sub WriteFile(sPath, sContent)
 'on error resume next
    If FSOcc.FileExists(sPath) = True Then FSOcc.DeleteFile sPath, True
  Set TScc = FSOcc.CreateTextFile(sPath, True)
    TScc.Write sContent
    TScc.Close
  Set TScc = Nothing
End Sub

  The only thing I can even imagine causing trouble would
be a null, but I don't see how a null could get into the text.
Even then, the file should write up to the null. And this happens
even if I've made no edits to the page content. (Though IE
does make edits.) I've also tried removing script before loading.
Nothing works. It always comes out a zero length file.

Here's the webpage, for what it's worth:
https://www.theregister.co.uk/2019/04/26/windows_10_storage/

I just picked it randomly, downloading only the HTML.

[toc] | [next] | [standalone]

#12106

On Mon, 29 Apr 2019 08:55:57 -0400, Mayayana wrote:
>   Very weird problem. I've been working on an HTA that
> allows me to touch-up edit webpages, in WYSIWYG manner.
> It starts by loading what's in the BODY into a DIV, then
> storing the header and end. When I save changes it takes
> the innerHTML from the DIV, adds back the header and end,
> and writes to disk with normal Textstream operation.
> 
>    The whole thing works fine but I was testing it on various
> files and came across one (an article from theregister.co.uk
> that has no particular odd qualities) that won't work.
> 
>   I load the page fine but saving changes results in a blank file.
> Empty. If I show the 3 parts in msgbox they all look right. If
> I right each part to disk, the BODY section is empty but both
> ends write OK. When I write the whole file it writes the blank
> file and then errors with "invalid procedure call or argument".
> That seems to be at Write, though I'm not certain because error
> line numbers can be off. The code is in an external VBS.
> 
>   sMid = MainDiv.innerHTML
>   sContent = sBeg & sMid & sEnd
> 
> Then it calls the Write routine in a class. All very vanilla:
> 
> Public Sub WriteFile(sPath, sContent)
>  'on error resume next
>     If FSOcc.FileExists(sPath) = True Then FSOcc.DeleteFile sPath, True
>   Set TScc = FSOcc.CreateTextFile(sPath, True)
>     TScc.Write sContent
>     TScc.Close
>   Set TScc = Nothing
> End Sub
> 
>   The only thing I can even imagine causing trouble would
> be a null, but I don't see how a null could get into the text.
> Even then, the file should write up to the null. And this happens
> even if I've made no edits to the page content. (Though IE
> does make edits.) I've also tried removing script before loading.
> Nothing works. It always comes out a zero length file.
> 
> Here's the webpage, for what it's worth:
> https://www.theregister.co.uk/2019/04/26/windows_10_storage/
> 
> I just picked it randomly, downloading only the HTML.

Interresting problem.

The initial cause is because of the "▼" (code point 0x25BC; black
pointing-down triangle) character which follows the "SHARE" link below the
line which states the author name and article date. My first thought is
that, it doesn't have any ANSI translation, so I tried to encode it using
UTF8. However, the stream still fail on writing the UTF8 encoded string.

I've found out that if the stream is openeded using ASCII mode, the stream
fails on writing some Unicode C1 control characters.

When I tested it using below code...

[code]
set fs = createobject("scripting.filesystemobject")
set f = fs.createtextfile("out.txt", true, false)
s = ""
on error resume next
for i = 0 to 255
  f.write chrw(i)
next
f.close
[/code]

It only writes 229 bytes of data. It fails on writing these character codes:

  0x80, 0x82-0x8C, 0x8E, 0x91-0x9C, 0x9E-0x9F

I thought... maybe that's expected because it at least conflict with the
stream mode. But it got weirder when I found out that if I use Chr() instead
of ChrW() for the above code, all 256 bytes are written. This doesn't make
sense considering that both `TypeName(ChrW(&H80))` and `TypeName(Chr(&H80))`
are both `String` (i.e. VarType 8). So, it seems like there's a hidden flag
within a string (variant) value which indicate that the string is Unicode or
ANSI.

I could think of anything why it behaves like this. The only way I could
think of to solve it is to scan for all characters whose code point is above
0xFF, convert them into UTF8 encoded string with the help of JScript, and
reencode each UTF8 encoded character using `Chr()`. e.g.

[code]
<div id=htm></div>
<script>
function utf8uri(s) {
  return encodeURI(s);
}
</script>
<script language=vbscript>
function encodeutf8(s)
  dim r, c, l, u, j
  l = len(s)
  for i = 1 to l
    c = mid(s, i, 1)
    if ascw(c) > 255 then
      u = split(mid(utf8uri(mid(s, i, 1)), 2), "%")
      c = ""
      for j = 0 to ubound(u)
        c = c & chr(eval("&H" & u(j)))
      next
    end if
    r = r & c
  next
  encodeutf8 = r
end function

set fs = createobject("scripting.filesystemobject")
set f = fs.opentextfile("input.html", 1, false, 0)
htm.innerHTML = f.readall
f.close
set f = fs.createtextfile("output.html", true, false)
f.write encodeutf8(htm.innerHTML)
f.close
</script>
[/code]

[toc] | [prev] | [next] | [standalone]

#12107

JJ, Mayayana,

> Interresting problem.
>
> The initial cause is because of the "Ąż" (code point 0x25BC; black
> pointing-down triangle) character which follows the "SHARE" link

With this explanation I realized I bumped into the very same when I used the 
"Microsoft.XMLHTTP" (and other, similar ones) to download a webpage and save 
the contents.

Although I could display (wscript.echo, MsgBox) the contents with no 
problems, there was no way I could write it to an UTF8 file.   The only 
solution was to write it in Windows Wide-character format.   :-(

In other words: Although the object read a multi-byte character webpage with 
no problems, it could not be saved as the same.  :-(

The "solution" I came up with was to replace all non-ASCII characters with 
HTML encodings representing their values:

function WSToHtml(sSource)
  dim sTemp,i,sChar

  sTemp=sSource
  i=1
  while i<len(sTemp)
    sChar=mid(sTemp,i,1)
    if ascw(sChar)<0 or ascw(sChar)>127 then
      sTemp=replace(sTemp,sChar,"&#x"& hex(ascw(sChar)) &";")
    end if
    i=i+1
  wend

  WSToHtml=sTemp
end function

(mind the "ascw(sChar)<0" in there)

Hope that helps.

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12108

"Mayayana" <mayayana@invalid.nospam> wrote
|  Very weird problem.

  Thanks to you both. I've never seen anything like this and
I don't think I ever would have guessed it. You've saved me
a lot of hair pulling. 25 BC, for instance, should just write
as % and 1/4 sign in English ANSI. Notepad will do that. Though
I see in the page code that it's not actually UTF-8. Rather
it's inserted as &#9660; The bytes in the file are also that. All
ASCII range. Apparently IE is sending the converted version
and Textstream is balking at it. Which implies TS recognizes
UTF-8 but won't handle it! I suspect this may be yet another
child safety feature added by the scrrun authors.

  I realize now that some time ago I wrote a conversion script
that loads HTML into IE, sets the document.charset to UTF-8,
then uses ADODB.Stream to convert it to windows-1252. Maybe
I'll play around with that and see what I get.

[toc] | [prev] | [next] | [standalone]

#12109

Mayayana,

> 25 BC, for instance, should just write as % and 1/4
> sign in English ANSI

Not quite.  If you write it that way into a file you will never be able to 
display the character the value origionally represented.     A character in 
multi-byte encoding /always/ starts with a byte with the highest bit set. 
More set bits follow, depending on the length of the full value.

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12110

"R.Wieser" <address@not.available>

| > 25 BC, for instance, should just write as % and 1/4
| > sign in English ANSI
|
| Not quite.  If you write it that way into a file you will never be able to
| display the character the value origionally represented.

   I understand. What I meant was that all 256 values
correspond to ANSI characters. They should show as
ANSI characters. Not as a downward arrow, but as %
and 1/4. But in this case it's not UTF-8 multi-byte that's
the problem. Rather &#9660; was used and IE is not
sending that through as UTF-8.

  When I try to use Asc or AscW or AscB to see what IE
thinks is there, they all fail.

  So I think the issue here is not a problem with UTF-8
in Textstream but rather I need to filter out &#[over 255];
before giving it to IE in the first place. 

[toc] | [prev] | [next] | [standalone]

#12111

Mayayana,

> I understand. What I meant was that all 256 values
> correspond to ANSI characters.

:-) But thats the problem: You have /way/ more than 256 possible values in 
that string.   You're using VBScript, in it internally stores its text in 
widechars (two bytes a piece), meaning you've got not (just) 256, but 65536 
values on your hands.

>  When I try to use Asc or AscW or AscB to see what
> IE thinks is there, they all fail.

Thats odd.    Could you show the code you're using to display the Asc values 
of your webpage string ?    And could you retry with a fully ASCII string 
(just to see if its the code or the string thats causing the problems) ?

And you could also try to pull your string thru that function I posted - 
just to see if it has got the same problem.

>  So I think the issue here is not a problem with
> UTF-8 in Textstream but rather I need to filter
> out &#[over 255]; before giving it to IE in the first place.

Don't forget the under zero ones.   You're working with signed ints. (Its 
what bit me in the behind the first time :-) ).

And I'm not quite sure I follow.  Wasn't the problem with saving to a 
textfile ?

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12113

"R.Wieser" <address@not.available> wrote

| :-) But thats the problem: You have /way/ more than 256 possible values in
| that string.   You're using VBScript, in it internally stores its text in
| widechars (two bytes a piece), meaning you've got not (just) 256, but 
65536
| values on your hands.
|
    That's not relevant. It's only internal. A normal string
is single byte 1-255, wehther it's ANSI or UTF-8. The UTF-8
is just interpreted differently.

| >  When I try to use Asc or AscW or AscB to see what
| > IE thinks is there, they all fail.
|
| Thats odd.    Could you show the code you're using to display the Asc 
values
| of your webpage string ?    And could you retry with a fully ASCII string
| (just to see if its the code or the string thats causing the problems) ?
|
I didn't save the code, but basically it was
just a loop:

for i = 1 to len(s)
s2 = s2 & CStr(Asc(Mid(s, i, 1)))
next

  It works on a normal string but none of them work
if I have &#9660;, give it to IE, then ask for it back
again. IE has apparently gone unicode-16 with it.

| Don't forget the under zero ones.   You're working with signed ints. (Its
| what bit me in the behind the first time :-) ).
|
   Negative numbers for &#? I've never seen it.
https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references

| And I'm not quite sure I follow.  Wasn't the problem with saving to a
| textfile ?
|
   Yes, but saving it from IE. As you probably know,
when you ask IE for DOM data you get the IE
version. For instance, if you ask for the innerHTML
of a SPAN where you used B and /B you might get
back <STRONG>  </STRONG>. IE apparently converts
the HTML to its own object hierarchy when it's loaded
into the DOM.

  So the problem here is that a webpage with something
like &#9660; seems to be getting the string converted,
part or all, to unicode. But I can't see what it's doing
because all attempts to look at the bytes fail.

   I load a webpage, get the BODY content, put that
into a DIV in the HTA webpage into an HTA, edit as
desired, then ask IE for the innerHTML of the DIV in
the HTA where I put the BODY content. I then add back
the header and end and write it all to disk. That one 9660
seems to infect the whole thing. If I change it to &#160;
for a non-breaking space, it loads and saves just fine. 

[toc] | [prev] | [next] | [standalone]

#12115

Mayayana,

>    That's not relevant. It's only internal

But its that "only internal" what VBS/IE is working with, and why the ANSI 
only output method throws up its hands when you ask it to write such content 
to file. ...

> for i = 1 to len(s)
> s2 = s2 & CStr(Asc(Mid(s, i, 1)))
> next

Suggestion: Check what the output of that CStr is in every loop-step there. 
Provide it the problem char plus a few before and after and see what the 
"s2" strings becomes.

You might well see that the "s2" builds normally upto the problem char, and 
than disappears altogether (as far as I remember adding a null to a string 
doesn't work that well).   Also try to see what you get when you replace 
that CStr( with "|" & hex(

> IE has apparently gone unicode-16 with it.

Thats what I said, VBS/IE internally uses wide-character. :-)

>   Negative numbers for &#? I've never seen it.

No, negative numbers when you AscW a character.

> As you probably know, when you ask IE for
> DOM data you get the IE version.

No, I didn't.  I've not used IE for quite a while now, and have never 
bothered to look at its DOM.

But in that case, why don't you try to WriteFile that converted string 
/before/ you give it to IE ?  You know, trying to eliminate possible problem 
points.    If you're right the resulting file should be OK.  But if that 
file than again (still) is empty than the problem lies elsewhere ...

> That one 9660 seems to infect the whole thing.

Or any of the gazillion other multi-byte characters.    Like forward and 
backward single and double quotes, dashes of different lengths, a triplet of 
dots, etc.

I often save, using FireFox, webpages (documentation) to file.  Quite a few 
of them show, when loaded into an ANSI-only editor, funny groups of symbols 
sprinkled throut the text.   Yes, those multi-byte characters.    In other 
words, delivering webpages as multi-byte (as opposed to ANSI with html 
entities or encodings for the "special" characters) is not uncommon at all.

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12116

"R.Wieser" <address@not.available> wrote

| Suggestion: Check what the output of that CStr is in every loop-step 
there.
| Provide it the problem char plus a few before and after and see what the
| "s2" strings becomes.
|
| You might well see that the "s2" builds normally upto the problem char, 
and
| than disappears altogether

  I suspect you're right. But I've got a solution now. I
just remove any HTML entities over 255.

| But in that case, why don't you try to WriteFile that converted string
| /before/ you give it to IE ?

   I can't do that because the whole thing is based on
giving it to IE. I load the body of a webpage file into a
DIV in my HTA. The rest of the HTA is a WYSIWYG editor.
When editing is done I take the innerHTML of the DIV
and recreate the input file with edits. For instance, I
might add a line return or remove a word, using the mouse
and keyboard directly in the IE browser window. The only
way to be able to do that is through IE's DOM. Of course I
could edit the HTML directly, but this is much more functional.

  This all started when I found pdftohtml.exe in the
"poppler" package and tried it for converting PDF to
HTML. It does a very good job, compared to other
options, but it makes a mess of the result, filling it
with &nbsp; unnecessarily, running headers together
with text, corrupting links, and making occasional
mistakes typical of PDF conversion, like converting
"ol" to "d". I coupled pdftohtml with my own script
to clean up the result, but that can't fix every detail.
So I wanted a tool that would allow me to do touch-ups
on the finished product.

| > That one 9660 seems to infect the whole thing.
|
| Or any of the gazillion other multi-byte characters.    Like forward and
| backward single and double quotes, dashes of different lengths, a triplet 
of
| dots, etc.
|
  No. You seem to be missing what I've been saying. IE
is turning &#9660; into something else. Unicode? I'm not
sure. Multi-byte UTF-8 is not a problem, because the unit
is still a byte. That's what I started out saying. If it were
actually coming through as bytes 25 BC that would be no
problem. They'd be written to disk and render as % and
1/4 sign. The file has other multi-byte characters and
has no trouble with them. UTF-8 is functional as ANSI.
In fact, part of my converter script involves a filter to
convert common UTF-8, like curly quotes, to ASCII, since
almost all UTF-8 webpages in English are unnecessary.

  It might be that I could convert to unicode, then convert
that back to ANSI. But none of that is necessary. I'm just
removing any &#[255+]; altogether. No big loss. But it's
a weird bug. I probably never would have found it by myself.

  You can try what I'm talking about. Give any HTML file to
IE with an HTML entity over 255. Like &#9660; You'll get
the corruption when it comes back. What IE returns is not
25 BC. Nor is it the entity byte. (38 for &. 35 for #. Etc.)
It's something else.
   But you can use any number of bytes over 127 and
there will be no problem. You can send it dozens of &#149;
and it won't choke. The problem is not high bytes but
rather what IE is doing, converting one HTML entity to unicode,
or perhaps even something more creative. The Microsofties
could have designed it for adaptable complexity internally.

  Interestingly, though, if you send in "&#149;" IE gives you
back byte 149, which won't render at all if it's being read as
UTF-8, but will show as a bullet in ANSI English. In other words,
IE is actually converting the HTML entity rather than just
rendering it.

[toc] | [prev] | [next] | [standalone]

#12118

Mayayana,

> But I've got a solution now. I  just remove any HTML
> entities over 255.

And possibly change the meaning of what is on the page ?  Just imagine all 
kinds of double-quotes disappearing (signifying where people talk about what 
others said). :-|

>   I can't do that because the whole thing is based
> on giving it to IE.

I did not mean permanently.  Just as a test, so you can tell if the problem 
already exists /before/ giving it to IE - and as such rule out IE as being 
the culprit (as you presume).

>  No. You seem to be missing what I've been saying.
> IE is turning ▼ into something else.

What "▼" Please ?

When I save that page you talked about in your first message and look at the 
spot JJ indicated I see a 0xE2, 0x96, 0xBC sequence (which definitily looks 
like a multi-byte encoded character), and no HTML entity like that (and no 
value like 9660t either!) anywhere on the page.

In other words, I have no idea what you are talking about there - and I have 
the idea you are rather confused about what happens where or how.   Pardon 
me the bruthish honesty.


But yes, That above three-byte sequence gets converted into a single 
widechar byte (value ranging from 0 to 65535) and stored /for internal use/ 
by IE.

Part of the problem seems to be that you seem to think that what gets stored 
in the DOM is exactly the same as what got loaded as HTML.    Why should it 
?    Just to convert the above three-byte sequence or that "▼" HTML 
entity into a character every time it needs to redraw the page ?   That 
would only slow down the whole thing, for no reason whatsoever.

Also, HTML elements are parsed and stored as DOM elements.   That also means 
thatif your HTML element contained superfluous spaces (between its 
attributes) they will be gone when you "innerHTML" the DOM.


If you want the editor to save the HTML page exactly as it came from the 
other side (forgetting your changes to it for the moment) than you /cannot/ 
use the DOM to edit it, and you will need to load the HTML page ito a buffer 
of your own, edit /that/, and only than provide it to IE (to display) - 
possibly, if you want to keep the full WYSIWYG effect, on every character 
you type/remove.

... I almost wrote "can we get back to the problem please" (finding it and 
than determining how to solve it), but you already mentioned you would just 
throw any non-ANSI characters away.     A bad move though (as I explained in 
the above), but its not upto me.

Though to bad that you didn't take a peek at that VBS function I posted, as 
that would replace the problematic chars with HTML-entities (just before 
writing it to file), thereby keeping the integrity of your webpage intact. 
But again, thats not upto me.

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12119

> In other words, I have no idea what you are talking about there - and I 
> have the idea you are rather confused about what happens where or how. 
> Pardon me the bruthish honesty.

Ah.   I just realized:  You are not loading the page itself, but are letting 
IE do that for you.  Only after that you take a peek at the DOM.   In that 
case IE already has done its conversions (like from multi-byte to 
wide-character), so you can't save it /before/ giving it to IE (as its IE 
doing the loading).

But that just means that you have to work with whatever IE gives you when 
you do your "innerHTML" extraction of your DIV (which still is a perfect 
representation of what you saw in it!).    Not much choice there, right ? 
:-)

But do yourself a favour and /convert/ those non-ANSI characters (instead of 
just dropping them).   If you don't you /will/ get into problems with pages 
that will have gotten a different meaning than the origional. (just imagine 
removing all end-of-line colons from a few lines of text ....)

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12120

"R.Wieser" <address@not.available> wrote

| Ah.   I just realized:  You are not loading the page itself, but are 
letting
| IE do that for you.  Only after that you take a peek at the DOM.   In that
| case IE already has done its conversions (like from multi-byte to
| wide-character), so you can't save it /before/ giving it to IE (as its IE
| doing the loading).
|

Yes.

| But that just means that you have to work with whatever IE gives you when
| you do your "innerHTML" extraction of your DIV (which still is a perfect
| representation of what you saw in it!).    Not much choice there, right ?
| :-)
|

Right.

| But do yourself a favour and /convert/ those non-ANSI characters (instead 
of
| just dropping them).   If you don't you /will/ get into problems with 
pages
| that will have gotten a different meaning than the origional. (just 
imagine
| removing all end-of-line colons from a few lines of text ....)
|
  All bytes are ANSI characters. You don't seem to be distinguishing
between UTF-8 bytes and 2-byte unicode. There's no problem loading
and saving with things like curly quotes, though they can show corrupted.
The problem is only with HTML entities over 255. If you look at the
original page code you'll see after SHARE is &#9660;. IE is converting
that to the unicode character (or something more quirky). If "&#9660;"
is removed the page works fine. I'm not worried about losing upside
down triangle characters. :)

  But I am actually converting much of the UTF-8, as I mentioned above.
I'm converting curly quotes to ", etc. 

[toc] | [prev] | [next] | [standalone]

#12121

Mayayana,

>  All bytes are ANSI characters.

Sigh ....    /which/ "bytes" please ?   The ones used by IE, or the ones in 
your file ?

For the latter ?  Well, you rammed them down a method which does not accept 
anything else, so I believe you.  As for the former ?   You're wrong.  As I 
already said, IE's DOM uses wide-chars, 16 bits a piece.

Also, you /still/ have no clue what multi-byte characters are (or why IE 
converts them to wide-character ones), and seemingly could not care less. 
If you would have read my explanation, my previous message or even just 
googeled than you would have known that they "map" to the same range as your 
ANSI characters, but should be looked at differently.   So no, not all bytes 
are ANSI characters.

> You don't seem to be distinguishing
> between UTF-8 bytes and 2-byte unicode.

No, you're fully right.  / Ofcourse/ I have zero clue what either of that 
is.   And that code I posted ?  Thats doing absolutily nothing.  Just ignore 
it.   As you obviously already did.    Also ignore my suggestions of doing a 
few simple tests on a short strings containing such a problematic character, 
cause you know much better what is going on than anyone else, right ?

Currently I ask myself why I even thought it would be a good idea to try to 
help you.  :-((

Do not bother to respond.    This thread is will be set to 'ignore'.   Go 
find someone else.

Regards,
Rudy Wieser

[toc] | [prev] | [next] | [standalone]

#12122

I got this PDF to HTML converter polished up, in case anyone
is curious:

https://www.jsware.net/jsware/scrfiles.php5#p2h

   It uses the Poppler pdftohtml.exe tool to do an intial
conversion, then cleans up the result. There's also an editor
tool, to do minor edits of the HTML directly in a browser
window. The two together are working nicely to convert
text from PDF to a webpage with a left-side index in which
one can easily change font, paragraph width, etc.

  Thanks to JJ for solving the strange puzzle of the unicode
booby trap. 

[toc] | [prev] | [next] | [standalone]

#12123

On Sat, 4 May 2019 15:26:47 -0400, Mayayana wrote:
> 
>   Thanks to JJ for solving the strange puzzle of the unicode
> booby trap.

The problem is still unexplained, tho. :(

And that problem also means that with Unicode stream mode, it's still
impossible to write UCS2 encoded file containing those unwrittenable
characters.

[toc] | [prev] | [next] | [standalone]

#12124

"JJ" <jj4public@vfemail.net> wrote

| The problem is still unexplained, tho. :(
|
| And that problem also means that with Unicode stream mode, it's still
| impossible to write UCS2 encoded file containing those unwrittenable
| characters.

  It is an intriguing problem. Since IE converts the
actual text to its own object model, I'm guessing
the actual format of the data it sends is a hybrid
and not just plain text. Slightly analogous to RTF.
If you copy RTF or HTML to the clipbaord and
paste into Notepad, you'll get only the text part.
Maybe IE is designed with the assumption that
when you ask for DIV1.innerHTML you intend to
use it within the context of the DOM.... Just a
guess. Since it's not a legitimate string I don't
know how to inspect it. It does return 8 (string)
when tested with VarType. In a test I got 56 for Len
and 112 for LenB. That seems to indicate unicode.
Maybe it could be handled in VB. I've written VBS to
convert unicode to ANSI, but it's quirky. (It's very
handy, though. I discovered that Windows has a lot
of intelligence built into the conversion. For instance,
if I convert Sanskrit S with what looks like an accent
I get S in ANSI. Nice. For an English speaker all those
technical marks are just noise, anyway. So I can
use my script to convert academic Sanskrit to popular
English book version of Sanskrit easily.)

  In any case, IE's quirk is not a problem for my purposes,
now that I know about it. I don't want unicode files. I
also don't have a problem with dropping out special
characters like an upside down triangle. I'm already
converting common UTF-8 characters like curly quotes,
non-breaking spaces, funky dashes, and o with umlaut
to ANSI equivalents. It's too much trouble to be switching
between encodings and it's completely unnecessary in
English.

  That's the first I've heard of UCS-2. I had to look it up.
It appears to be an outdated term. According to Wikipedia,
Windows unicode was derived from UCS-2 but is not UCS-2.
That gets confusing. It used to be there was ANSI and
unicode in the Windows world. Now with the popularity
of UTF-8 there are lots of unicodes. But it seems to be
safe to refer to Windows unicode as unicode-16. 

[toc] | [prev] | [standalone]

Back to top | Article view | microsoft.public.scripting.vbscript

csiph-web