Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #35771

Re: Packed strings in Forth ... any thoughts, advice, or links to prior art?

Show all headers | View raw

on 13/01/2015 02:11:19,  wrote:
> On Monday, January 12, 2015 at 3:03:17 AM UTC-7, Alex McDonald wrote:
>> on 11/01/2015 22:38:47, Hugh wrote:
>>
>> >
>> > One good thing, is that you can compare packed strings for equality
>> > without unpacking them --- you have to unpack to compare for greater
>> > or less than, though.
>> >
>>
>> Not if you want to support Unicode strings properly, although 99 times
>> out of 100 and in most en=us locales that will be true. Unicode strings
>> may be equal and yet have different binary representations; unicode
>> strings are not bit buffers. The letter n in Spanish can be U+00F1 o
> r
>> U+006E U+0303. These are the code points, not the representation. In
>> UTF-8, they are C3 B1 and 6E CC 83 respectively.
> 
> Alex is talking nonsense, as usual.
> 
> It is well known that UTF-8 chars can theoretically be represented in
> sever al ways, but they are supposed to be represented in the shortest
> form --- i f they are being represented in a longer form than
> necessary, then that is invalid UTF-8.

You are confusing UTF-8 for Unicode code points. It's a common mistake. 

> 
> The only exception to the above rule is a variation of UTF-8 called
> "Modifi ed UTF-8" in which 0x00 is represented as 0xC0,0x80 --- this
> is done so UTF -8 can have the zero char in a string, but still use
> 0x00 as a delimiter (a s done in C strings) --- this is a somewhat
> hokey modification, and I don't think it is used very much.

How is this about Unicode? You're talking about UTF-8.

> 
> Even in Modified UTF-8 though, you would never have both 0x00 and
> 0xC0,0x80 being used together, because in Modified UTF-8 0x00 is now
> invalid. So a s tring compare for equality can still be done on packed
> data, just as I desc ribed.
> 

Unicode strings can't be compared for equality without a process called
normalization, regardless of how they are transformed. This has nothing
to do with UTF-8, modified UTF-8, UTF-16, UTF-32 or any other UTF
encoding (or "transformation"; that's what the T in UTF means).

Unicode point -> UTF-8
-------------    --------
U+006E U+0303    6E CC 83
U+00F1           C3 B1

Compared as Unicode strings, they are equal, since they are both the
string "ñ". The UTF-8 encodings are the shortest forms available for the
two code points. However, the UTF-8 bit buffers are not equal. You cannot
treat any transformation of Unicode as bit buffers when doing
comparisons.

Shorter version: comparing Unicode strings requires normalization, even
for equality. Without it, there are potentially significant problems.
Hence the OP will have to unpack and normalize first.

Back to comp.lang.forth | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-11 00:42 -0800
  Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? stephen@mpeforth.com (Stephen Pelc) - 2015-01-11 12:03 +0000
  Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-01-11 12:02 +0000
    Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-11 11:29 -0800
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Bernd Paysan <bernd.paysan@gmx.de> - 2015-01-11 22:24 +0100
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-01-12 10:10 +0000
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-01-12 10:12 +0000
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? albert@spenarnc.xs4all.nl (Albert van der Horst) - 2015-01-12 11:17 +0000
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-12 19:13 -0800
  Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-11 14:38 -0800
    Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-11 17:09 -0800
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? "Elizabeth D. Rather" <erather@forth.com> - 2015-01-11 17:18 -1000
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-11 19:48 -0800
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-12 17:59 -0800
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-12 19:04 -0800
          Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-12 19:52 -0800
            Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-12 20:28 -0800
              Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-13 19:42 -0800
            Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Alexander Skobelev <al.skobelev@gmail.com> - 2015-01-13 08:38 +0300
              Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Alexander Skobelev <al.skobelev@gmail.com> - 2015-01-13 08:43 +0300
              Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-13 19:27 -0800
    Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? "Alex McDonald" <blog@rivadpm.com> - 2015-01-12 10:03 +0000
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-12 18:11 -0800
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? "Alex McDonald" <blog@rivadpm.com> - 2015-01-13 09:56 +0000
          Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Bernd Paysan <bernd.paysan@gmx.de> - 2015-01-14 03:07 +0100
            Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? albert@spenarnc.xs4all.nl (Albert van der Horst) - 2015-01-14 11:27 +0000
              Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Bernd Paysan <bernd.paysan@gmx.de> - 2015-01-15 18:09 +0100
    Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Ilya Tarasov <ilya74.tarasov@gmail.com> - 2015-01-15 06:42 -0800

csiph-web