Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #35703 > unrolled thread

Packed strings in Forth ... any thoughts, advice, or links to prior art?

Back to article view | Back to comp.lang.forth

Contents

  Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-11 00:42 -0800
    Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? stephen@mpeforth.com (Stephen Pelc) - 2015-01-11 12:03 +0000
    Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-01-11 12:02 +0000
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-11 11:29 -0800
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Bernd Paysan <bernd.paysan@gmx.de> - 2015-01-11 22:24 +0100
          Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-01-12 10:10 +0000
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-01-12 10:12 +0000
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? albert@spenarnc.xs4all.nl (Albert van der Horst) - 2015-01-12 11:17 +0000
          Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-12 19:13 -0800
    Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-11 14:38 -0800
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-11 17:09 -0800
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? "Elizabeth D. Rather" <erather@forth.com> - 2015-01-11 17:18 -1000
          Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-11 19:48 -0800
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-12 17:59 -0800
          Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-12 19:04 -0800
            Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-12 19:52 -0800
              Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Michael Barry <barrym95838@yahoo.com> - 2015-01-12 20:28 -0800
                Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-13 19:42 -0800
              Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Alexander Skobelev <al.skobelev@gmail.com> - 2015-01-13 08:38 +0300
                Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Alexander Skobelev <al.skobelev@gmail.com> - 2015-01-13 08:43 +0300
                Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-13 19:27 -0800
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? "Alex McDonald" <blog@rivadpm.com> - 2015-01-12 10:03 +0000
        Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? hughaguilar96@gmail.com - 2015-01-12 18:11 -0800
          Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? "Alex McDonald" <blog@rivadpm.com> - 2015-01-13 09:56 +0000
            Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Bernd Paysan <bernd.paysan@gmx.de> - 2015-01-14 03:07 +0100
              Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? albert@spenarnc.xs4all.nl (Albert van der Horst) - 2015-01-14 11:27 +0000
                Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Bernd Paysan <bernd.paysan@gmx.de> - 2015-01-15 18:09 +0100
      Re: Packed strings in Forth ... any thoughts, advice, or links to prior art? Ilya Tarasov <ilya74.tarasov@gmail.com> - 2015-01-15 06:42 -0800

Page 2 of 2 — ← Prev page 1 [2]

#35802

On Monday, January 12, 2015 at 10:38:17 PM UTC-7, Alexander Skobelev wrote:
> hughaguilar96@gmail.com writes:
> [...]
> > RACE, the processor formerly known as MiniForth, belongs to Testra ---
> > MFX belongs to them too --- I can't provide any of this to you.  Ask
> > them: http://www.testra.com/
> [...]
> 
> When I (from pure curiosity) tried to go to this site it asked for login
> and password. It's very strange. Aren't they publicly available anymore
> or they just filter some IPs?

No they don't --- I just looked at it, and there is no requirement to log in.

Anyway, I haven't worked for Testra in 20 years, so I don't really care.

[toc] | [prev] | [next] | [standalone]

#35742

on 11/01/2015 22:38:47, Hugh wrote:

> 
> One good thing, is that you can compare packed strings for equality
> without unpacking them --- you have to unpack to compare for greater
> or less than, though.
> 

Not if you want to support Unicode strings properly, although 99 times
out of 100 and in most en=us locales that will be true. Unicode strings
may be equal and yet have different binary representations; unicode
strings are not bit buffers. The letter ñ in Spanish can be U+00F1 or
U+006E U+0303. These are the code points, not the representation. In
UTF-8, they are C3 B1 and 6E CC 83 respectively. 

[toc] | [prev] | [next] | [standalone]

#35758

On Monday, January 12, 2015 at 3:03:17 AM UTC-7, Alex McDonald wrote:
> on 11/01/2015 22:38:47, Hugh wrote:
> 
> > 
> > One good thing, is that you can compare packed strings for equality
> > without unpacking them --- you have to unpack to compare for greater
> > or less than, though.
> > 
> 
> Not if you want to support Unicode strings properly, although 99 times
> out of 100 and in most en=us locales that will be true. Unicode strings
> may be equal and yet have different binary representations; unicode
> strings are not bit buffers. The letter ń in Spanish can be U+00F1 or
> U+006E U+0303. These are the code points, not the representation. In
> UTF-8, they are C3 B1 and 6E CC 83 respectively.

Alex is talking nonsense, as usual. 

It is well known that UTF-8 chars can theoretically be represented in several ways, but they are supposed to be represented in the shortest form --- if they are being represented in a longer form than necessary, then that is invalid UTF-8.

The only exception to the above rule is a variation of UTF-8 called "Modified UTF-8" in which 0x00 is represented as 0xC0,0x80 --- this is done so UTF-8 can have the zero char in a string, but still use 0x00 as a delimiter (as done in C strings) --- this is a somewhat hokey modification, and I don't think it is used very much.

Even in Modified UTF-8 though, you would never have both 0x00 and 0xC0,0x80 being used together, because in Modified UTF-8 0x00 is now invalid. So a string compare for equality can still be done on packed data, just as I described.

[toc] | [prev] | [next] | [standalone]

#35771

on 13/01/2015 02:11:19,  wrote:
> On Monday, January 12, 2015 at 3:03:17 AM UTC-7, Alex McDonald wrote:
>> on 11/01/2015 22:38:47, Hugh wrote:
>>
>> >
>> > One good thing, is that you can compare packed strings for equality
>> > without unpacking them --- you have to unpack to compare for greater
>> > or less than, though.
>> >
>>
>> Not if you want to support Unicode strings properly, although 99 times
>> out of 100 and in most en=us locales that will be true. Unicode strings
>> may be equal and yet have different binary representations; unicode
>> strings are not bit buffers. The letter n in Spanish can be U+00F1 o
> r
>> U+006E U+0303. These are the code points, not the representation. In
>> UTF-8, they are C3 B1 and 6E CC 83 respectively.
> 
> Alex is talking nonsense, as usual.
> 
> It is well known that UTF-8 chars can theoretically be represented in
> sever al ways, but they are supposed to be represented in the shortest
> form --- i f they are being represented in a longer form than
> necessary, then that is invalid UTF-8.

You are confusing UTF-8 for Unicode code points. It's a common mistake. 

> 
> The only exception to the above rule is a variation of UTF-8 called
> "Modifi ed UTF-8" in which 0x00 is represented as 0xC0,0x80 --- this
> is done so UTF -8 can have the zero char in a string, but still use
> 0x00 as a delimiter (a s done in C strings) --- this is a somewhat
> hokey modification, and I don't think it is used very much.

How is this about Unicode? You're talking about UTF-8.

> 
> Even in Modified UTF-8 though, you would never have both 0x00 and
> 0xC0,0x80 being used together, because in Modified UTF-8 0x00 is now
> invalid. So a s tring compare for equality can still be done on packed
> data, just as I desc ribed.
> 

Unicode strings can't be compared for equality without a process called
normalization, regardless of how they are transformed. This has nothing
to do with UTF-8, modified UTF-8, UTF-16, UTF-32 or any other UTF
encoding (or "transformation"; that's what the T in UTF means).

Unicode point -> UTF-8
-------------    --------
U+006E U+0303    6E CC 83
U+00F1           C3 B1

Compared as Unicode strings, they are equal, since they are both the
string "ñ". The UTF-8 encodings are the shortest forms available for the
two code points. However, the UTF-8 bit buffers are not equal. You cannot
treat any transformation of Unicode as bit buffers when doing
comparisons.

Shorter version: comparing Unicode strings requires normalization, even
for equality. Without it, there are potentially significant problems.
Hence the OP will have to unpack and normalize first.

[toc] | [prev] | [next] | [standalone]

#35797

Alex McDonald wrote:
> Shorter version: comparing Unicode strings requires normalization, even
> for equality. Without it, there are potentially significant problems.
> Hence the OP will have to unpack and normalize first.

Or normalize and then pack, and store only normalized strings...

However, the only actual problems people see with Unicode and normalization 
is that if someone actually does it (e.g. Apple in its file system).  This 
changes the file names in an unexpected way, and taking Unicode files to a 
Mac and then back to Linux results in garbled file names.  One of the 
problems is that there are actually 4 different normalization forms...

-- 
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://bernd-paysan.de/
net2o ID: kQusJzA;7*?t=uy@X}1GWr!+0qqp_Cn176t4(dQ*

[toc] | [prev] | [next] | [standalone]

#35807

In article <m94j05$5dn$1@dont-email.me>,
Bernd Paysan  <bernd.paysan@gmx.de> wrote:
>Alex McDonald wrote:
>> Shorter version: comparing Unicode strings requires normalization, even
>> for equality. Without it, there are potentially significant problems.
>> Hence the OP will have to unpack and normalize first.
>
>Or normalize and then pack, and store only normalized strings...

I personally think that messing with the strings is wrong.

The right thing is to replace equality with equivalence.
So if you want case-insensitivity in ciforth, at the lowest level
where you compare a string to a dictionary name, a different
comparison is thrown in. There is nothing canonical about the
a->A *->* conversion.
>
>However, the only actual problems people see with Unicode and normalization
>is that if someone actually does it (e.g. Apple in its file system).  This
>changes the file names in an unexpected way, and taking Unicode files to a
>Mac and then back to Linux results in garbled file names.  One of the
>problems is that there are actually 4 different normalization forms...

As regards file names. Files are strings of bytes, which is pretty
straightforward. So should filenames. The representation on the
screen could change, but not the one on disk.

>--
>Bernd Paysan

Groetjes Albert
>
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

[toc] | [prev] | [next] | [standalone]

#35828

Albert van der Horst wrote:
>>Or normalize and then pack, and store only normalized strings...
> 
> I personally think that messing with the strings is wrong.

Me, too.  Just store it as it came from the input.

> The right thing is to replace equality with equivalence.
> So if you want case-insensitivity in ciforth, at the lowest level
> where you compare a string to a dictionary name, a different
> comparison is thrown in. There is nothing canonical about the
> a->A *->* conversion.

Actially, we need case insensitivity in Forth only because traditional Forth 
had all words in upper case, and we no longer like shouting at the computer.  
However, that problem is limited to the ASCII range of characters, and 
there, case-insensitive comparison is rather simple.

> As regards file names. Files are strings of bytes, which is pretty
> straightforward. So should filenames. The representation on the
> screen could change, but not the one on disk.

That's how Linux does it, but Mac OS X doesn't.

-- 
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://bernd-paysan.de/
net2o ID: kQusJzA;7*?t=uy@X}1GWr!+0qqp_Cn176t4(dQ*

[toc] | [prev] | [next] | [standalone]

#35824

понедельник, 12 января 2015 г., 1:38:48 UTC+3 пользователь hughag...@gmail.com написал:
> On Sunday, January 11, 2015 at 1:42:36 AM UTC-7, Michael Barry wrote:
> > I have a hypothetical machine that doesn't address bytes, only 32-
> >  bit words.  Chars, cells, addresses, single-precision integers and
> >  single-precision floats are all 32-bits wide.  
> 
> As I recall, you said that you are designing a hypothetical processor that looks like a cross between the 6502 and 6809, but in 32-bit. If you are planning on eventually putting this in an FPGA, then that is not a good design. Forth code is highly factored, so it is imperative that you can call functions quickly, and also keep memory usage of the threaded-code to a minimum. In the MiniForth, the high bit of every xt indicated if it was a colon word or a primitive, so NEXT could do the right thing. Every call required only one cell of memory; this was as compared to the 6502 or 6809 where one byte is needed for the JSR opcode and the next cell (2 bytes) is the address (so the MiniForth used 2/3 the memory). NEXT got parallelized almost every time, so NEXT generally took zero clock cycles to execute --- EXIT took one clock cycle though, so a colon word wasn't entirely free. If you are designing a processor and intend Forth to be the primary language, you should focus on design issues related to Forth such as I've described above. Also, unless you know a hardware-design language and can design the processor yourself, speculating on possible designs is a waste of time --- believe me, any design you come up with will fall apart in the first week of putting it in hardware --- hardware design is not something that you plan out in detail ahead of time, but it is something that has to be figured out as you go (it is done "bottom up" rather than "top down").
> 
> I would really recommend that you just find an existing processor and go with it --- there is no market for Forth, so you still won't have your feet planted firmly in reality, but at least you will be in a low orbit around reality.
> 
> > I don't have sufficient knowledge of the advanced features of Forth
> >  to know whether or not it is feasible to have all the words that
> >  create, move, compare, compile and manipulate strings be able to
> >  determine the encoding and act appropriately, since frequent
> >  packing and unpacking would be wasteful.  
> 
> I can't think of any way to do it, except to unpack as needed. The MiniForth only addressed 16-bit words, and didn't address bytes, so this was similar to your contrived processor. I wrote primitives to pack and unpack, and they were pretty fast --- as a rule, primitives were a couple of orders of magnitude faster than equivalent colon words --- unfortunately, writing primitives in MiniForth assembly-language was a couple of orders of magnitude more difficult than writing the equivalent colon words.
> 
> One good thing, is that you can compare packed strings for equality without unpacking them --- you have to unpack to compare for greater or less than, though.

5+ years with 32-bit FPGA-based Forth CPU. No problems, great flexibility due to FPGA nature, smallest possible memory footprint, official Microblaze processor core no longer used by anyone around me. What I do wrong?

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.forth

csiph-web