Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #39639 > unrolled thread

RfD: 1 chars = 1

Back to article view | Back to comp.lang.forth

Contents

  RfD: 1 chars = 1 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-06-08 17:34 +0000
    Re: RfD: 1 chars = 1 Paul Rubin <no.email@nospam.invalid> - 2015-06-08 10:59 -0700
      Re: RfD: 1 chars = 1 "Alex McDonald" <blog@rivadpm.com> - 2015-06-08 19:57 +0100
        Re: RfD: 1 chars = 1 Michael Barry <barrym95838@yahoo.com> - 2015-06-08 16:35 -0700
          Re: RfD: 1 chars = 1 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-06-09 06:18 +0000
        Re: RfD: 1 chars = 1 Bernd Paysan <bernd.paysan@gmx.de> - 2015-06-09 23:35 +0200
          Re: RfD: 1 chars = 1 "Alex McDonald" <blog@rivadpm.com> - 2015-06-09 23:49 +0100
            Re: RfD: 1 chars = 1 Bernd Paysan <bernd.paysan@gmx.de> - 2015-06-10 01:41 +0200
              Re: RfD: 1 chars = 1 "Alex McDonald" <blog@rivadpm.com> - 2015-06-10 02:03 +0100
                Re: RfD: 1 chars = 1 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-06-10 12:22 +0000
                  Re: RfD: 1 chars = 1 "Alex McDonald" <blog@rivadpm.com> - 2015-06-11 14:50 +0100
                Re: RfD: 1 chars = 1 Bernd Paysan <bernd.paysan@gmx.de> - 2015-06-10 14:46 +0200
      Re: RfD: 1 chars = 1 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-06-09 07:09 +0000
        Re: RfD: 1 chars = 1 Stefan Mauerhofer <smauerhofer@androsoft.ch> - 2015-06-09 01:52 -0700
        Re: RfD: 1 chars = 1 Bernd Paysan <bernd.paysan@gmx.de> - 2015-06-09 23:29 +0200
        Re: RfD: 1 chars = 1 David Thompson <dave.thompson2@verizon.net> - 2015-06-21 12:46 -0400
    Re: RfD: 1 chars = 1 Matthias Koch <matthias.koch@hot.uni-hannover.de> - 2015-06-09 12:06 +0200
      Re: RfD: 1 chars = 1 Mark Wills <markwills1970@gmail.com> - 2015-06-09 03:40 -0700
        Re: RfD: 1 chars = 1 Matthias Koch <matthias.koch@hot.uni-hannover.de> - 2015-06-09 12:45 +0200
          Re: RfD: 1 chars = 1 rickman <gnuarm@gmail.com> - 2015-06-09 07:17 -0400
            Re: RfD: 1 chars = 1 Matthias Koch <matthias.koch@hot.uni-hannover.de> - 2015-06-09 14:30 +0200
        Re: RfD: 1 chars = 1 "Elizabeth D. Rather" <erather@forth.com> - 2015-06-09 12:50 -0800
          Re: RfD: 1 chars = 1 "HAA" <someone@microsoft.com> - 2015-06-21 17:14 +1000
            Re: RfD: 1 chars = 1 "Elizabeth D. Rather" <erather@forth.com> - 2015-06-20 21:46 -1000
              Re: RfD: 1 chars = 1 "HAA" <someone@microsoft.com> - 2015-06-23 15:05 +1000
                Re: RfD: 1 chars = 1 "Elizabeth D. Rather" <erather@forth.com> - 2015-06-22 19:42 -1000
                  Re: RfD: 1 chars = 1 "HAA" <someone@microsoft.com> - 2015-06-27 14:08 +1000

Page 1 of 2  [1] 2  Next page →

#39639 — RfD: 1 chars = 1

Problem

While Forth-94 and Forth-2012 provide the option to systems that
1 CHARS > 1, to my knowledge no mainstream Forth system has
made use of this option.  Consequently, many Forth programmers write
programs that (according to the standard documentation requirements)
should declare an environmental dependency on 1 CHARS = 1.

Even those programmers that want to avoid this dependency cannot
easily test that their program does not have it, because so few Forth
systems are available that do not satisfy this dependency.  And in any
case, spending an effort to avoid an environmental dependency that all
interesting systems satisfy seems wasteful; alternatively, declaring
the environmental dependency does not beneit anyone, either.



Solution

Characters take one address unit (au)in memory.

CHARS can be implemented as noop
CHAR+ is equivalent to 1+

These words are not removed.  So programmers who like to program for
systems (whether existing or not) where 1 CHARS>1 still can do so.

As a consequence, on byte-addressed machines, characters normally take
one byte; on word-addressed machines, characters and cells take one
machine word.


Remarks

What about word-addressed machines?

Implementing standard Forth on word-addressed machines already
necessitates that char=au: a character in standard Forth must be at
least on address unit wide, and be at most as large as a cell; on
word-addressed machines, cell=au, so char=au follows from that.

There have been occasionaly ideas about supporting a packed character
representation (with more than one character per au) for
word-addressed machines, but that is beyond the scope of this
proposal.

What about nibble-addressed/bit-addressed machines?

No implementations of standard Forth on such machines are known to me;
probably the benefits of standard systems do not outweigh the costs on
such restricted machines.  Such machines are becoming rarer over time,
so if nobody has created such a system yet, it is unlikely to happen
in the future.  

Still, if anybody wants to create such a system, they would have two
options: Declare an environmental restriction of having a non-standard
character size, or implement an address unit size of (e.g.) 8 bits.
Which option is preferable depends on the use of the system.

What about other unusual hardware setups?

I heard of an embedded system with 16-bit address units and 32-bit
chars.  If we standardize on 1 chars = 1, the simplest way forward for
such a system would be to declare an environmental restriction of
having a non-standard char size; alternatively, one could try to
change the char size to 16 bits, or the address unit size to 32 bits.
I don't know enough about the system to make any recommendation among
these options.

What about Unicode?

An important motivation for the introduction of CHARS and CHAR+ in
Forth-94 was probably the seeming move to 16-bit characters with
Unicode in the early 1990s, which also led to 16-bit characters in
Windows NT (released 1993) and in Java (released 1995).

However, 16 bits turned out to be too little for storing Unicode code
points (Unicode 2.0 1996), so UTF-16 (Unicode based on 16-bit basic
units) is is a variable-width encoding; at around the same time UTF-8
was introduced for encoding Unicode with 8-bit basic units, which
requires less conversion effort for software based on 8-bit chars, and
has the same disadvantage of being variable-width as UTF-16 (which
turns out to play little role in most software).

As a result, with a few exceptions (see below) Forth systems have
stayed with 8-bit characters, and they are not alone in this: UTF-8 is
the dominant file representation of Unicode, and most Unix software
handles Unicode also internally as UTF-8 (can anyone fill in here for
Windows).

Jax4th and JavaForth

Jax4th was written by Jack Woehr (Jax) as a proof of concept
implementation of Forth-94, and implements 16-bit characters on a
byte-addressed platform.  It is not maintained (current platform:
Windows NT 3.1).

JavaForth was written in 2014 by Peter Knaggs with primitives in Java.
It uses 32-bit cells, 16-bit chars and 8-bit address units.  Because
Java does not allow access to raw memory, JavaForth has to map Forth
memory to a Java array of some Java type anyway, and is pretty free to
choose address unit, character and cell size; it seems to me that it
would be simplest if Java Forth actually implemented
cell=char=au=32bits.

My impression is that Peter Knaggs chose the address unit and
character size to allow testing whether programs have an environmental
dependency on char=au (but that is still difficult for most programs
because JavaForth implements only the core words).  While one might
feel that the effort that went into getting this to work would be
wasted if this proposal is accepted, this effort has been expended
already, and is no good reason not to standardize the common practice
that char=au.  JavaForth could adopt the proposal by having 8-bit
chars or 16-bit aus, or in any other way; or it could keep the current
choices (and declare an environmental restriction on char>au) to serve
as testing platform for programs that should be able to cope with that
environmental restriction.



Typical use

s" Some string" dup       buffer: s s swap move

instead of Forth-2012:

s" Some string" dup chars buffer: s s swap Cmove



Proposal

3.1.2 Replace "be at least one address unit wide" with "be exactly one
address unit wide".

Keep the rest as-is.  While some wordings could be simplified (in
particular wrt character-aligned addresses, and the definitions of
CHARS and CHAR+), keeping them as-is gives guidance to programmers who
want to write char>au-capable code, and to system implementors who
implement a Forth system with the environmental restriction char>au.


Reference implementation

Nearly all standard Forth systems, e.g., SwiftForth, VFX, iForth,
Gforth, ...


Test cases

T{ 0 char+ -> 1 }T
T{ 1 chars -> 1 }T


Experience

Almost universally implemented and widely used.


Comments

None yet.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2015: http://www.mpeforth.com/euroforth2015/euroforth2015.htm

[toc] | [next] | [standalone]

#39640

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
> As a result, with a few exceptions (see below) Forth systems have
> stayed with 8-bit characters, and they are not alone in this: UTF-8 is
> the dominant file representation of Unicode, and most Unix software
> handles Unicode also internally as UTF-8 (can anyone fill in here for
> Windows).

UTF-8 is a byte encoding of Unicode, but obviously some characters UTF-8
representation use multiple bytes.  The most correct approach is
probably not in the Forth spirit.  It's to have the char type be large
enough to hold any codepoint (i.e. char = 21 bits or larger) and have a
string type that can use a compact representation like UTF-8 instead of
just being a vector of chars.

Windows (and Java, C#, and Objective C) use UTF-16 according to this:
  
  http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/

[toc] | [prev] | [next] | [standalone]

#39641

on 08/06/2015 18:59:36, Paul Rubin wrote:
> anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>> As a result, with a few exceptions (see below) Forth systems have
>> stayed with 8-bit characters, and they are not alone in this: UTF-8 is
>> the dominant file representation of Unicode, and most Unix software
>> handles Unicode also internally as UTF-8 (can anyone fill in here for
>> Windows).
> 
> UTF-8 is a byte encoding of Unicode, but obviously some characters
> UTF-8 representation use multiple bytes. The most correct approach is
> probably not in the Forth spirit. It's to have the char type be large
> enough to hold any codepoint (i.e. char = 21 bits or larger) and have
> a string type that can use a compact representation like UTF-8 instead
> of just being a vector of chars.

See the xchars section of the Forth201x spec. 

> 
> Windows (and Java, C#, and Objective C) use UTF-16 according to this:
> 
> http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/
> 

Windows NT uses UCS-2 which requires 2 bytes. Prior to that it was ASCII
and codepages, a solution so awful we still live with its unreadable
consequences to this day. NT is only of interest as a historical example
of doing the right thing, but being too early and backing the wrong
horse. This decision forced MS into later using UTF-16 rather than UTF-8.

Windows 2000 and on uses UTF-16 which requires 2 or 4 bytes depending on
the codepoint. Windows provides conversion APIs to translate from UTF-x
to UCS-x, ASCII and various codepages using the functions:
MultiByteToWideChar, maps a character string to a UTF-16;
WideCharToMultiByte maps a UTF-16 (what MS calls wide character) string
to a new character string. MS recommends UTF-8 or UTF- 16 for all target
conversions to avoid loss of fidelity. There are no Windows calls that
support codepoints; all character data is assumed to be suitably encoded.

Internally, programs can deal with characters and strings as they see
fit. But to talk to the outside world, and where string functions are
passed to the OS and other applications, translation is required. There
are often two function names in Windows libraries, funcA and funcW; A for
ANSI (by which is meant bytes interpreted by codepage and not UTF-8) and
W for UTF-16. For funcA calls, Windows calls MultiByteToWideChar on
behalf of the caller and then calls funcW.

For programmers in high level languages, for example C, the correct API
is selected automatically by the compiler based on #defines and flags.
The C programmer therefore simply calls func and the compiler obliges
with the correct entry point. That is a luxury that Forth compiler
writers endeavour to provide by a different route. So some (like
Win32Forth) will try to load func, and if that fails, will attempt to
load funcA if CHAR=1 (or funcW if CHAR=2 but I know of no Windows Forths
that use widechars).

Using UTF-8 is not impossible, but it does require some special handling.
funcW calls must be used after translating from UTF-8 to UTF-16 (funcA
calls don't handle UTF-8 as input), which minimally requires buffers be
big enough and that conversion API calls are correctly handled and
buffered by the compiler (or are handled by the programmer explicitly).
All this has been sufficent for me not to attempt to suppport it in
Win32Forth when I was maintaining the kernel. I have not tried to support
it in my Forth either.

I support the proposal that CHARS=1. CHARS=2 would not make the problem
go away since there are codepoints that require UTF-16 encoding in 4
bytes. CHARS=4 would be horrible for everyone.

[toc] | [prev] | [next] | [standalone]

#39644

On Monday, June 8, 2015 at 11:57:08 AM UTC-7, Alex McDonald wrote:
> ...
> I support the proposal that CHARS=1. CHARS=2 would not make the problem
> go away since there are codepoints that require UTF-16 encoding in 4
> bytes. CHARS=4 would be horrible for everyone.

If UTF-32 ever manages to grab a strong-hold, my little word-addressed
  hobby processor is golden ... until then, I certainly can get behind
  the idea that CHAR+ is equivalent to 1+ (and CHARS is nop), because
  anything else would "be horrible" is my case.

But is this leaving someone out in the cold with no substantial gain
  in efficiency?  I was under the impression that CHARS and CHAR+ were
  added for a legitimate reason, and that reason offered the port-
  ability and flexibility to deal with varying existing hardware *AND*
  an uncertain future WRT default encoding schemes.  I'm sorry if this
  qualifies as a dumb question ... I'm an inexperienced Forth n00b.

Mike

[toc] | [prev] | [next] | [standalone]

#39645

Michael Barry <barrym95838@yahoo.com> writes:
>But is this leaving someone out in the cold with no substantial gain
>  in efficiency?  I was under the impression that CHARS and CHAR+ were
>  added for a legitimate reason, and that reason offered the port-
>  ability and flexibility to deal with varying existing hardware *AND*
>  an uncertain future WRT default encoding schemes.

Yes, it certainly was a good and legitimate idea in the early 1990s to
add CHARS and CHAR+ (and this proposal is not for taking them away);
at the time it looked like the world was going to switch to 16-bit
characters on byte-addressed machines.  Things have turned out
differently, and 16-bit (or 32-bit) characters on byte-addressed
machines have not caught on in the Forth world.  This proposal is
about standardizing the common practice that has emerged since then.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2015: http://www.mpeforth.com/euroforth2015/euroforth2015.htm

[toc] | [prev] | [next] | [standalone]

#39671

Alex McDonald wrote:

> Windows NT uses UCS-2 which requires 2 bytes. Prior to that it was ASCII
> and codepages, a solution so awful we still live with its unreadable
> consequences to this day. NT is only of interest as a historical example
> of doing the right thing, but being too early and backing the wrong
> horse. This decision forced MS into later using UTF-16 rather than UTF-8.

IMHO that's wrong.  CP_UTF8 exists, all they need to do is to make it a 
fully supported code page (instead of supporting it only for conversion 
functions), which works for all the *A versions of their API calls.  
Actually, I don't even understand why they have first-class and second-class 
code pages for the *A versions, as internally, all the *A stuff goes through 
a wrapper which converts the string to UTF-16, anyways.  Apart from Windows 
9x, which is dead and burried.

They weren't forced into that decision.  It is sheer stupidity.

-- 
Bernd Paysan
"If you want it done right, you have to do it yourself"
net2o ID: kQusJzA;7*?t=uy@X}1GWr!+0qqp_Cn176t4(dQ*
http://bernd-paysan.de/

[toc] | [prev] | [next] | [standalone]

#39673

on 09/06/2015 22:35:19, Bernd Paysan wrote:
> Alex McDonald wrote:
> 
>> Windows NT uses UCS-2 which requires 2 bytes. Prior to that it was ASCII
>> and codepages, a solution so awful we still live with its unreadable
>> consequences to this day. NT is only of interest as a historical example
>> of doing the right thing, but being too early and backing the wrong
>> horse. This decision forced MS into later using UTF-16 rather than UTF-8.
> 
> IMHO that's wrong. CP_UTF8 exists, all they need to do is to make it a
> fully supported code page (instead of supporting it only for
> conversion functions), which works for all the *A versions of their
> API calls. Actually, I don't even understand why they have first-class
> and second-class code pages for the *A versions, as internally, all
> the *A stuff goes through a wrapper which converts the string to
> UTF-16, anyways. Apart from Windows 9x, which is dead and burried.
> 
> They weren't forced into that decision.  It is sheer stupidity.
> 

UTF-8 postdated Windows NT.

[toc] | [prev] | [next] | [standalone]

#39674

Alex McDonald wrote:

> on 09/06/2015 22:35:19, Bernd Paysan wrote:
>> Alex McDonald wrote:
>> 
>>> Windows NT uses UCS-2 which requires 2 bytes. Prior to that it was ASCII
>>> and codepages, a solution so awful we still live with its unreadable
>>> consequences to this day. NT is only of interest as a historical example
>>> of doing the right thing, but being too early and backing the wrong
>>> horse. This decision forced MS into later using UTF-16 rather than
>>> UTF-8.
>> 
>> IMHO that's wrong. CP_UTF8 exists, all they need to do is to make it a
>> fully supported code page (instead of supporting it only for
>> conversion functions), which works for all the *A versions of their
>> API calls. Actually, I don't even understand why they have first-class
>> and second-class code pages for the *A versions, as internally, all
>> the *A stuff goes through a wrapper which converts the string to
>> UTF-16, anyways. Apart from Windows 9x, which is dead and burried.
>> 
>> They weren't forced into that decision.  It is sheer stupidity.
>> 
> UTF-8 postdated Windows NT.

Did you read my text?  I repeat it for you:  The *A versions in Windows NT 
go through a string conversion wrapper, and all they need to do is add 
CP_UTF8 as first-class codepage.

It doesn't matter if UTF-8 postdated Windows NT or not; which BTW isn't 
true, either:

"Thompson's design was outlined on September 2, 1992, on a placemat in a New 
Jersey diner with Rob Pike. In the following days, Pike and Thompson 
implemented it and updated Plan 9 to use it throughout, and then 
communicated their success back to X/Open.[7]

UTF-8 was first officially presented at the USENIX conference in San Diego, 
from January 25 to 29, 1993."

Windows NT was released 26. Juli 1993.

-- 
Bernd Paysan
"If you want it done right, you have to do it yourself"
net2o ID: kQusJzA;7*?t=uy@X}1GWr!+0qqp_Cn176t4(dQ*
http://bernd-paysan.de/

[toc] | [prev] | [next] | [standalone]

#39676

on 10/06/2015 00:41:54, Bernd Paysan wrote:
> Alex McDonald wrote:
> 
>> on 09/06/2015 22:35:19, Bernd Paysan wrote:
>>> Alex McDonald wrote:
>>>
>>>> Windows NT uses UCS-2 which requires 2 bytes. Prior to that it was ASCII
>>>> and codepages, a solution so awful we still live with its unreadable
>>>> consequences to this day. NT is only of interest as a historical example
>>>> of doing the right thing, but being too early and backing the wrong
>>>> horse. This decision forced MS into later using UTF-16 rather than
>>>> UTF-8.
>>>
>>> IMHO that's wrong. CP_UTF8 exists, all they need to do is to make it a
>>> fully supported code page (instead of supporting it only for
>>> conversion functions), which works for all the *A versions of their
>>> API calls. Actually, I don't even understand why they have first-class
>>> and second-class code pages for the *A versions, as internally, all
>>> the *A stuff goes through a wrapper which converts the string to
>>> UTF-16, anyways. Apart from Windows 9x, which is dead and burried.
>>>
>>> They weren't forced into that decision.  It is sheer stupidity.
>>>
>> UTF-8 postdated Windows NT.
> 
> Did you read my text? I repeat it for you: The *A versions in Windows
> NT go through a string conversion wrapper, and all they need to do is
> add CP_UTF8 as first-class codepage.

Yes, I read your text. What you're suggesting isn't possible. The
codepage is set for the entire OS and *A calls do not provide specific
codepages. It is possible to overide and use codepage 65001 (UTF-8) for
the command line processor (the "DOS console") but that isn't possible
for windowed apps, and it only affects input and output to the console.

"A Windows operating system always has one currently active Windows code
page. All ANSI versions of API functions use the currently active code
page." https://msdn.microsoft.com/en-
us/library/windows/desktop/dd317752%28v=vs.85%29.aspx

> 
> It doesn't matter if UTF-8 postdated Windows NT or not; which BTW
> isn't true, either:
> 
> "Thompson's design was outlined on September 2, 1992, on a placemat in
> a New Jersey diner with Rob Pike. In the following days, Pike and
> Thompson implemented it and updated Plan 9 to use it throughout, and
> then communicated their success back to X/Open.[7]
> 
> UTF-8 was first officially presented at the USENIX conference in San
> Diego, from January 25 to 29, 1993."
> 
> Windows NT was released 26. Juli 1993.

http://tools.ietf.org/html/rfc2044 didn't get ratified by the IETF until
October 1996. It mentions ISO/IEC 10646-1:1993: "UTF-8 is described in
Annex R, adopted but not yet published." UTF-8 also made an appearance in
Unicode 1.1, but the documents date from late 1992, and were published in
1993. They aren't available online on the Unicode website.

It would not have been possible or advisable to change the design of
Windows NT between January and July 1993 based on a USENIX paper, or a
Unicode standard published in 1993; both would have been well beyond the
end of the design phase of Windows NT.

> 

[toc] | [prev] | [next] | [standalone]

#39678

"Alex McDonald" <blog@rivadpm.com> writes:
>on 10/06/2015 00:41:54, Bernd Paysan wrote:
>> Did you read my text? I repeat it for you: The *A versions in Windows
>> NT go through a string conversion wrapper, and all they need to do is
>> add CP_UTF8 as first-class codepage.
>
>Yes, I read your text. What you're suggesting isn't possible. The
>codepage is set for the entire OS and *A calls do not provide specific
>codepages. It is possible to overide and use codepage 65001 (UTF-8) for
>the command line processor (the "DOS console") but that isn't possible
>for windowed apps, and it only affects input and output to the console.

Just because it is not yet implemented in Windows does not mean it is
impossible.  What prevents Microsoft from supporting code page 65001
everywhere, including windowed apps?  Any technical reason? Or is it
just stubbornness?

>It would not have been possible or advisable to change the design of
>Windows NT between January and July 1993 based on a USENIX paper, or a
>Unicode standard published in 1993; both would have been well beyond the
>end of the design phase of Windows NT.

That is likely true, but that does not mean that UTF-8 postdates
Windows NT.  Nor does it mean that it is impossible to add UTF-8 to
Windows as a first-class code page.  GB18030 (from 2005) definitely
postdates Windows NT, and also Windows XP and Windows Server 2003, yet
MicroSoft added support for GB18030 to Windows XP and Windows Server
2003 <https://msdn.microsoft.com/en-us/library/aa578128.aspx> (and of
course also to later versions).  The funny thing is that GB18030 is a
byte-based Unicode Transformation Format, like UTF-8.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2015: http://www.mpeforth.com/euroforth2015/euroforth2015.htm

[toc] | [prev] | [next] | [standalone]

#39698

on 10/06/2015 13:22:56,  wrote:
> "Alex McDonald" <blog@rivadpm.com> writes:
>>on 10/06/2015 00:41:54, Bernd Paysan wrote:
>>> Did you read my text? I repeat it for you: The *A versions in Windows
>>> NT go through a string conversion wrapper, and all they need to do is
>>> add CP_UTF8 as first-class codepage.
>>
>>Yes, I read your text. What you're suggesting isn't possible. The
>>codepage is set for the entire OS and *A calls do not provide specific
>>codepages. It is possible to overide and use codepage 65001 (UTF-8) for
>>the command line processor (the "DOS console") but that isn't possible
>>for windowed apps, and it only affects input and output to the console.
> 
> Just because it is not yet implemented in Windows does not mean it is
> impossible.  What prevents Microsoft from supporting code page 65001
> everywhere, including windowed apps?  Any technical reason? Or is it
> just stubbornness?

The question is above my pay grade. I don't know. 
 
> 
>>It would not have been possible or advisable to change the design of
>>Windows NT between January and July 1993 based on a USENIX paper, or a
>>Unicode standard published in 1993; both would have been well beyond the
>>end of the design phase of Windows NT.
> 
> That is likely true, but that does not mean that UTF-8 postdates
> Windows NT.  Nor does it mean that it is impossible to add UTF-8 to
> Windows as a first-class code page.  GB18030 (from 2005) definitely
> postdates Windows NT, and also Windows XP and Windows Server 2003, yet
> MicroSoft added support for GB18030 to Windows XP and Windows Server
> 2003 <https://msdn.microsoft.com/en-us/library/aa578128.aspx> (and of
> course also to later versions). The funny thing is that GB18030 is a
> byte-based Unicode Transformation Format, like UTF-8.

There is a first class UTF-8 codepage; utf-8 65001.


https://msdn.microsoft.com/en-us/library/windows/desktop/dd317756%28v=vs.85%29.aspx

> 
> - anton

[toc] | [prev] | [next] | [standalone]

#39679

Alex McDonald wrote:
> Yes, I read your text. What you're suggesting isn't possible. The
> codepage is set for the entire OS and *A calls do not provide specific
> codepages.

Yes, it's severely broken, SetLocaleInfoA (the one which allows you to set 
the locale) is user-global, and affects all running applications.  I also 
understand that Microsoft's codebase is so unmaintainable that you couldn't 
change that to become a per-application setting.

If Microsoft was not just a bunch of incompetent code-monkeys, they could 
change the already available conversion function for *A strings to be per-
application, and add a locale ID that uses CP_UTF8, and then, we would have 
first-class UTF-8 support.  As a "quick hack" (something Microsoft really 
likes, because design is not their strength), they could even implement only 
that UTF-8 option per-application, so when you use a new UseCodepageUTF8A() 
call, the *A conversion function would do UTF-8/UTF-16 conversion instead of 
going through the current code page.

> It is possible to overide and use codepage 65001 (UTF-8) for
> the command line processor (the "DOS console") but that isn't possible
> for windowed apps, and it only affects input and output to the console.

Yes, that's because Microsoft didn't implement this.  I know this.  But the 
framework generally allows this, and it would IMHO be a relatively small 
change in the system to actually do it.  All it would require is some 
intelligence, which unfortunately is unavailable and unobtainable at 
Microsoft.

> "A Windows operating system always has one currently active Windows code
> page. All ANSI versions of API functions use the currently active code
> page." https://msdn.microsoft.com/en-> us/library/windows/desktop/dd317752%28v=vs.85%29.aspx

There's actually a system active code page and an "active code page", which 
is per-user.  The per-user codepage is set by SetLocaleInfoA, and doesn't 
affect the other users.  So at least they have already a way to deal with 
different code pages active at the same time (at least in Windows Terminal 
Server, which supports real multi-user).

>> It doesn't matter if UTF-8 postdated Windows NT or not; which BTW
>> isn't true, either:
>> 
>> "Thompson's design was outlined on September 2, 1992, on a placemat in
>> a New Jersey diner with Rob Pike. In the following days, Pike and
>> Thompson implemented it and updated Plan 9 to use it throughout, and
>> then communicated their success back to X/Open.[7]
>> 
>> UTF-8 was first officially presented at the USENIX conference in San
>> Diego, from January 25 to 29, 1993."
>> 
>> Windows NT was released 26. Juli 1993.
> 
> http://tools.ietf.org/html/rfc2044 didn't get ratified by the IETF until
> October 1996. It mentions ISO/IEC 10646-1:1993: "UTF-8 is described in
> Annex R, adopted but not yet published." UTF-8 also made an appearance in
> Unicode 1.1, but the documents date from late 1992, and were published in
> 1993. They aren't available online on the Unicode website.

Microsoft often bases their stuff on early drafts.  CP1252 was based on an 
early ANSI draft, which became ISO-8859-1 years later.  The discrepancy 
between the early draft and the ISO standard is very obvious.  As Microsoft 
was spearheading Unicode, I'd suppose that they did contribute and 
participate in all related events.  The need for a ASCII-compatible byte-
encoding for Unicode became apparent in early 1992.

> It would not have been possible or advisable to change the design of
> Windows NT between January and July 1993 based on a USENIX paper, or a
> Unicode standard published in 1993; both would have been well beyond the
> end of the design phase of Windows NT.

The design of Windows NT supports variable-width ASCII-compatible code 
pages, so no need to change the design.  Just implement that particular code 
page as first-class locale, which is distributed with every localized 
Windows version (like CP1252, which also is available everywhere).

Microsoft has also shown other incapabilites with i18n, e.g. it took them 
until Vista to deviler a universal distribution which can be set to whatever 
language you liked to, instead of buying the localized version.  And that 
multi-language Vista was only half-baked.

-- 
Bernd Paysan
"If you want it done right, you have to do it yourself"
net2o ID: kQusJzA;7*?t=uy@X}1GWr!+0qqp_Cn176t4(dQ*
http://bernd-paysan.de/

[toc] | [prev] | [next] | [standalone]

#39647

Paul Rubin <no.email@nospam.invalid> writes:
>anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>> As a result, with a few exceptions (see below) Forth systems have
>> stayed with 8-bit characters, and they are not alone in this: UTF-8 is
>> the dominant file representation of Unicode, and most Unix software
>> handles Unicode also internally as UTF-8 (can anyone fill in here for
>> Windows).
>
>UTF-8 is a byte encoding of Unicode, but obviously some characters UTF-8
>representation use multiple bytes.  The most correct approach is
>probably not in the Forth spirit.  It's to have the char type be large
>enough to hold any codepoint (i.e. char = 21 bits or larger) and have a
>string type that can use a compact representation like UTF-8 instead of
>just being a vector of chars.

Look at the xchars wordset in Forth-2012.  The thing that holds code
points is an xchar on the stack.  The normal way to store it in memory
is as a string.  And actually it is pretty rare that you need to get
xchars on the stack.

>Windows (and Java, C#, and Objective C) use UTF-16 according to this:
>  
>  http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/

For C# it's clear given where it comes from (and also since it started
out as a Java clone).

Objective C is more interesting.  Looking at the Wikipedia page, it
says that `Objective-C is a thin layer on top of C, and is often
called a "strict superset" of C, meaning that it should be possible to
compile any C program with an Objective-C compiler'.  And it gives
example code like

printf("\n");

or

const char* propertyName = property_getName(*thisProperty);

which I assume use the same char size as C on the same platform.
Maybe <http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/> means the
character/string representation used by NeXTSTEP (first released 1989)
and its successors (MacOS X and iOS).

BTW, while Java uses a 16-bit char type, it has UTF-8 in various
places in .class files, and the Cacao Java compiler developed in our
groupd keeps this as UTF-8 (there is no reason to convert it to
UTF-16).

<http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/> also says that "Ruby,
Go as well as Rust have decided on using UTF-8" and concludes "I'm
definitely expecting more languages to take the UTF-8 route in the
future and just provide more tools to deal with Unicode as part of the
standard library."

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2015: http://www.mpeforth.com/euroforth2015/euroforth2015.htm

[toc] | [prev] | [next] | [standalone]

#39651

Personally I prefer UTF-8. Since UTF-16 can be 2 or 4 bytes per character there is no way to tell the number of UTF-16 characters from the string length. Also there is a potential endian problem with UTF-16. 

[toc] | [prev] | [next] | [standalone]

#39670

Anton Ertl wrote:

> Objective C is more interesting.  Looking at the Wikipedia page, it
> says that `Objective-C is a thin layer on top of C, and is often
> called a "strict superset" of C, meaning that it should be possible to
> compile any C program with an Objective-C compiler'.

The information about Objective C is wrong.  NSString can take values in 
different encodings, UTF-8 and UTF-16 are just two possible ones, and they 
are equally well supported.

-- 
Bernd Paysan
"If you want it done right, you have to do it yourself"
net2o ID: kQusJzA;7*?t=uy@X}1GWr!+0qqp_Cn176t4(dQ*
http://bernd-paysan.de/

[toc] | [prev] | [next] | [standalone]

#39937

On Tue, 09 Jun 2015 07:09:13 GMT, anton@mips.complang.tuwien.ac.at
(Anton Ertl) wrote:

<snip other points>
> BTW, while Java uses a 16-bit char type, it has UTF-8 in various
> places in .class files, and the Cacao Java compiler developed in our
> groupd keeps this as UTF-8 (there is no reason to convert it to
> UTF-16).
> 
To be exact, Java .class files, along with (other) files that use
{Data,Object}{Output,Input}Stream which includes useful things like
RMI and persistence, use 'modified UTF-8':

- the string content is preceded by a 2-byte aka 'short' length field

- surrogate points are encoded individually, rather than UTF-8
encoding the true Uni-char represented by a surrogate pair; this means
surrogate points not part of a valid pair 'work'

- U+0000 is encoded C0 A0 instead of the canonical 00. This allows ALL
string contents to be processed as null-terminated byte strings.

Of course many of the strings in .class files are identifiers for
classes and methods and such, and JLS identifiers can't use U+0000 or
invalid surrogates; I don't know (and didn't check) if there are any
supplementary (>=U+10000) chars useful as 'letter, currency,
connecting, digit, combining, nonspacing, or ignorable'.

[toc] | [prev] | [next] | [standalone]

#39652

> What about nibble-addressed/bit-addressed machines?
> 
> No implementations of standard Forth on such machines are known to me;
> probably the benefits of standard systems do not outweigh the costs on
> such restricted machines.  Such machines are becoming rarer over time,
> so if nobody has created such a system yet, it is unlikely to happen
> in the future.  

May I ask if there currently still is a bit-addressed microcontroller out in the wild and in production to play with ? I dived into this topic and only found the TMS34010 vintage graphics processor.

Matthias

[toc] | [prev] | [next] | [standalone]

#39654

On Tuesday, 9 June 2015 11:06:04 UTC+1, Matthias Koch  wrote:
> > What about nibble-addressed/bit-addressed machines?
> > 
> > No implementations of standard Forth on such machines are known to me;
> > probably the benefits of standard systems do not outweigh the costs on
> > such restricted machines.  Such machines are becoming rarer over time,
> > so if nobody has created such a system yet, it is unlikely to happen
> > in the future.  
> 
> May I ask if there currently still is a bit-addressed microcontroller out in the wild and in production to play with ? I dived into this topic and only found the TMS34010 vintage graphics processor.
> 
> Matthias

I'm not aware of any. Like you, the last I heard/used was the TMS34010
and TMS34020 back in ~93/94.

[toc] | [prev] | [next] | [standalone]

#39656

> I'm not aware of any. Like you, the last I heard/used was the TMS34010
> and TMS34020 back in ~93/94.

Thank you for the reply ! So many interesting ideas in instruction set design :-)

[toc] | [prev] | [next] | [standalone]

#39661

On 6/9/2015 6:45 AM, Matthias Koch wrote:
>> I'm not aware of any. Like you, the last I heard/used was the TMS34010
>> and TMS34020 back in ~93/94.
>
> Thank you for the reply ! So many interesting ideas in instruction set
> design :-)

I believe the CM3 and/or CM4 have bit manipulation instructions, but I 
believe that is not what you mean.  It's been a while since I read about 
them, but I seem to recall that they may map bits to individual addresses.

-- 

Rick

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.forth

csiph-web