Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #39641

Re: RfD: 1 chars = 1

Show all headers | View raw

on 08/06/2015 18:59:36, Paul Rubin wrote:
> anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>> As a result, with a few exceptions (see below) Forth systems have
>> stayed with 8-bit characters, and they are not alone in this: UTF-8 is
>> the dominant file representation of Unicode, and most Unix software
>> handles Unicode also internally as UTF-8 (can anyone fill in here for
>> Windows).
> 
> UTF-8 is a byte encoding of Unicode, but obviously some characters
> UTF-8 representation use multiple bytes. The most correct approach is
> probably not in the Forth spirit. It's to have the char type be large
> enough to hold any codepoint (i.e. char = 21 bits or larger) and have
> a string type that can use a compact representation like UTF-8 instead
> of just being a vector of chars.

See the xchars section of the Forth201x spec. 

> 
> Windows (and Java, C#, and Objective C) use UTF-16 according to this:
> 
> http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/
> 

Windows NT uses UCS-2 which requires 2 bytes. Prior to that it was ASCII
and codepages, a solution so awful we still live with its unreadable
consequences to this day. NT is only of interest as a historical example
of doing the right thing, but being too early and backing the wrong
horse. This decision forced MS into later using UTF-16 rather than UTF-8.

Windows 2000 and on uses UTF-16 which requires 2 or 4 bytes depending on
the codepoint. Windows provides conversion APIs to translate from UTF-x
to UCS-x, ASCII and various codepages using the functions:
MultiByteToWideChar, maps a character string to a UTF-16;
WideCharToMultiByte maps a UTF-16 (what MS calls wide character) string
to a new character string. MS recommends UTF-8 or UTF- 16 for all target
conversions to avoid loss of fidelity. There are no Windows calls that
support codepoints; all character data is assumed to be suitably encoded.

Internally, programs can deal with characters and strings as they see
fit. But to talk to the outside world, and where string functions are
passed to the OS and other applications, translation is required. There
are often two function names in Windows libraries, funcA and funcW; A for
ANSI (by which is meant bytes interpreted by codepage and not UTF-8) and
W for UTF-16. For funcA calls, Windows calls MultiByteToWideChar on
behalf of the caller and then calls funcW.

For programmers in high level languages, for example C, the correct API
is selected automatically by the compiler based on #defines and flags.
The C programmer therefore simply calls func and the compiler obliges
with the correct entry point. That is a luxury that Forth compiler
writers endeavour to provide by a different route. So some (like
Win32Forth) will try to load func, and if that fails, will attempt to
load funcA if CHAR=1 (or funcW if CHAR=2 but I know of no Windows Forths
that use widechars).

Using UTF-8 is not impossible, but it does require some special handling.
funcW calls must be used after translating from UTF-8 to UTF-16 (funcA
calls don't handle UTF-8 as input), which minimally requires buffers be
big enough and that conversion API calls are correctly handled and
buffered by the compiler (or are handled by the programmer explicitly).
All this has been sufficent for me not to attempt to suppport it in
Win32Forth when I was maintaining the kernel. I have not tried to support
it in my Forth either.

I support the proposal that CHARS=1. CHARS=2 would not make the problem
go away since there are codepoints that require UTF-16 encoding in 4
bytes. CHARS=4 would be horrible for everyone.

Back to comp.lang.forth | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

RfD: 1 chars = 1 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-06-08 17:34 +0000
  Re: RfD: 1 chars = 1 Paul Rubin <no.email@nospam.invalid> - 2015-06-08 10:59 -0700
    Re: RfD: 1 chars = 1 "Alex McDonald" <blog@rivadpm.com> - 2015-06-08 19:57 +0100
      Re: RfD: 1 chars = 1 Michael Barry <barrym95838@yahoo.com> - 2015-06-08 16:35 -0700
        Re: RfD: 1 chars = 1 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-06-09 06:18 +0000
      Re: RfD: 1 chars = 1 Bernd Paysan <bernd.paysan@gmx.de> - 2015-06-09 23:35 +0200
        Re: RfD: 1 chars = 1 "Alex McDonald" <blog@rivadpm.com> - 2015-06-09 23:49 +0100
          Re: RfD: 1 chars = 1 Bernd Paysan <bernd.paysan@gmx.de> - 2015-06-10 01:41 +0200
            Re: RfD: 1 chars = 1 "Alex McDonald" <blog@rivadpm.com> - 2015-06-10 02:03 +0100
              Re: RfD: 1 chars = 1 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-06-10 12:22 +0000
                Re: RfD: 1 chars = 1 "Alex McDonald" <blog@rivadpm.com> - 2015-06-11 14:50 +0100
              Re: RfD: 1 chars = 1 Bernd Paysan <bernd.paysan@gmx.de> - 2015-06-10 14:46 +0200
    Re: RfD: 1 chars = 1 anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2015-06-09 07:09 +0000
      Re: RfD: 1 chars = 1 Stefan Mauerhofer <smauerhofer@androsoft.ch> - 2015-06-09 01:52 -0700
      Re: RfD: 1 chars = 1 Bernd Paysan <bernd.paysan@gmx.de> - 2015-06-09 23:29 +0200
      Re: RfD: 1 chars = 1 David Thompson <dave.thompson2@verizon.net> - 2015-06-21 12:46 -0400
  Re: RfD: 1 chars = 1 Matthias Koch <matthias.koch@hot.uni-hannover.de> - 2015-06-09 12:06 +0200
    Re: RfD: 1 chars = 1 Mark Wills <markwills1970@gmail.com> - 2015-06-09 03:40 -0700
      Re: RfD: 1 chars = 1 Matthias Koch <matthias.koch@hot.uni-hannover.de> - 2015-06-09 12:45 +0200
        Re: RfD: 1 chars = 1 rickman <gnuarm@gmail.com> - 2015-06-09 07:17 -0400
          Re: RfD: 1 chars = 1 Matthias Koch <matthias.koch@hot.uni-hannover.de> - 2015-06-09 14:30 +0200
      Re: RfD: 1 chars = 1 "Elizabeth D. Rather" <erather@forth.com> - 2015-06-09 12:50 -0800
        Re: RfD: 1 chars = 1 "HAA" <someone@microsoft.com> - 2015-06-21 17:14 +1000
          Re: RfD: 1 chars = 1 "Elizabeth D. Rather" <erather@forth.com> - 2015-06-20 21:46 -1000
            Re: RfD: 1 chars = 1 "HAA" <someone@microsoft.com> - 2015-06-23 15:05 +1000
              Re: RfD: 1 chars = 1 "Elizabeth D. Rather" <erather@forth.com> - 2015-06-22 19:42 -1000
                Re: RfD: 1 chars = 1 "HAA" <someone@microsoft.com> - 2015-06-27 14:08 +1000

csiph-web