Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c++ > #67674 > unrolled thread

"C++: Size Matters in Platform Compatibility"

Back to article view | Back to comp.lang.c++

Contents

  "C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-16 14:04 -0500
    Re: "C++: Size Matters in Platform Compatibility" Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-08-16 20:45 +0000
    Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-17 15:08 +0200
      Re: "C++: Size Matters in Platform Compatibility" Geoff <geoff@invalid.invalid> - 2019-08-17 12:43 -0700
        Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-18 12:32 +0200
          Re: "C++: Size Matters in Platform Compatibility" Geoff <geoff@invalid.invalid> - 2019-08-18 13:10 -0700
            Re: "C++: Size Matters in Platform Compatibility" Bart <bc@freeuk.com> - 2019-08-18 21:15 +0100
            Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-18 14:49 -0700
              Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-18 14:51 -0700
            Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-19 08:43 +0200
              Re: "C++: Size Matters in Platform Compatibility" "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com> - 2019-08-19 09:15 +0200
              Re: "C++: Size Matters in Platform Compatibility" Öö Tiib <ootiib@hot.ee> - 2019-08-19 01:07 -0700
        Re: "C++: Size Matters in Platform Compatibility" Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-21 17:14 +0200
          Re: "C++: Size Matters in Platform Compatibility" Eli the Bearded <*@eli.users.panix.com> - 2019-08-21 18:30 +0000
            Re: "C++: Size Matters in Platform Compatibility" scott@slp53.sl.home (Scott Lurndal) - 2019-08-21 19:05 +0000
            Re: "C++: Size Matters in Platform Compatibility" Bonita Montero <Bonita.Montero@gmail.com> - 2019-08-21 21:25 +0200
            Re: "C++: Size Matters in Platform Compatibility" Keith Thompson <kst-u@mib.org> - 2019-08-21 12:28 -0700
            Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-22 22:58 -0400
              Re: "C++: Size Matters in Platform Compatibility" Eli the Bearded <*@eli.users.panix.com> - 2019-08-23 19:15 +0000
      Re: "C++: Size Matters in Platform Compatibility" Anton Shepelev <anton.txt@gmail.com> - 2019-08-18 15:59 +0300
        Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-18 16:32 +0200
      Re: "C++: Size Matters in Platform Compatibility" Juha Nieminen <nospam@thanks.invalid> - 2019-08-19 07:29 +0000
        Re: "C++: Size Matters in Platform Compatibility" Paavo Helde <myfirstname@osa.pri.ee> - 2019-08-19 12:20 +0300
        Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-19 11:22 +0200
          Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-23 14:14 -0400
            Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-24 20:56 +0200
              Re: "C++: Size Matters in Platform Compatibility" Robert Wessel <robertwessel2@yahoo.com> - 2019-08-24 23:43 -0500
                Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-25 12:11 +0200
                  Re: "C++: Size Matters in Platform Compatibility" Richard Damon <Richard@Damon-Family.org> - 2019-08-25 07:33 -0700
                    Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-25 20:27 +0200
                  Re: "C++: Size Matters in Platform Compatibility" Robert Wessel <robertwessel2@yahoo.com> - 2019-08-25 10:22 -0500
      Re: "C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-19 15:05 -0500
        Re: "C++: Size Matters in Platform Compatibility" scott@slp53.sl.home (Scott Lurndal) - 2019-08-19 20:17 +0000
          Re: "C++: Size Matters in Platform Compatibility" David Brown <david.brown@hesbynett.no> - 2019-08-20 08:21 +0200
            Re: "C++: Size Matters in Platform Compatibility" Lynn McGuire <lynnmcguire5@gmail.com> - 2019-08-20 14:38 -0500
        Re: "C++: Size Matters in Platform Compatibility" James Kuyper <jameskuyper@alumni.caltech.edu> - 2019-08-21 09:15 -0400
    Re: "C++: Size Matters in Platform Compatibility" Anton Shepelev <anton.txt@gmail.com> - 2019-08-18 15:20 +0300
      Re: "C++: Size Matters in Platform Compatibility" Ben Bacarisse <ben.usenet@bsb.me.uk> - 2019-08-18 14:50 +0100

Page 2 of 2 — ← Prev page 1 [2]

#67757

On 18/08/2019 14:59, Anton Shepelev wrote:
> David Brown:
> 
>> However, it doesn't make sense to have pointer types at
>> all in a stored file or data interchange format.
> 
> He is talking about the use of pointer types as integer
> indices, as (he says) WinAPI does with DWORD_PTR.
> 

It makes sense to use integer indices in a file format - use a suitable 
integer size for the job (using fixed size types, like uint32_t or 
int64_t).  It doesn't make sense to use pointers, because they depend on 
the memory address your data happens to lie at, which will not be the 
same when the file or data is read again.  So using some sort of 
"integer pointer type" (like a pre-C99 uintptr_t alternative) makes 
little sense.

[toc] | [prev] | [next] | [standalone]

#67777

In comp.lang.c++ David Brown <david.brown@hesbynett.no> wrote:
> For character data where plain ASCII won't do, utf-8 is the only sane 
> choice IMHO.  Other formats - utf-16, utf-32, wchar_t, etc., might be 
> used internally for dealing with historical API's.

The problem with UTF-8 is that it's quite complicated to parse and intepret,
and is wasteful in terms of space if the majority of the text is non-ascii
(quite typically almost all non-ascii characters take at least 3 bytes to
store as UTF-8.) It's the most efficient (spacewise) if the majority of the
text is ascii, but one still has to be prepared to recover any unicode
characters from it.

UTF-16 is simpler to parse (still not completely straightforward, but
much simpler), and often more space-efficient if the majority of the
text is non-ascii (such as Japanese).

[toc] | [prev] | [next] | [standalone]

#67781

On 19.08.2019 10:29, Juha Nieminen wrote:
> In comp.lang.c++ David Brown <david.brown@hesbynett.no> wrote:
>> For character data where plain ASCII won't do, utf-8 is the only sane
>> choice IMHO.  Other formats - utf-16, utf-32, wchar_t, etc., might be
>> used internally for dealing with historical API's.
>
> The problem with UTF-8 is that it's quite complicated to parse and intepret,

So it's fortunate that one rarely needs to parse and interpret it. Most 
text manipulations like text search, followed by cut, replace, insert 
can be done on UTF-8 just fine without any need to parse it. The most 
demanding task would be to extract a random size-limited piece from a 
large text which requires recognition of Unicode character borders, but 
even this is pretty straightforward as well.

Of course, at some point the text needs to be parsed and interpreted for 
display, but for example the Scintilla text editor control happily 
accepts UTF-8 as its input and displays it just fine, no effort on my 
part needed. In other places one might need to pre-convert UTF-8 to 
platform-specific UTF-16 or UTF-32, but this is also done by library 
routines, nothing special.

> and is wasteful in terms of space if the majority of the text is non-ascii
> (quite typically almost all non-ascii characters take at least 3 bytes to
> store as UTF-8.) It's the most efficient (spacewise) if the majority of the
> text is ascii, but one still has to be prepared to recover any unicode
> characters from it.
>
> UTF-16 is simpler to parse (still not completely straightforward, but
> much simpler), and often more space-efficient if the majority of the
> text is non-ascii (such as Japanese).

This is in theory. In practice I have heard most Japanese texts contain 
a hefty amount of ASCII numbers, punctuation, XML and HTML tags, so the 
space benefits are nowhere so clear.

[toc] | [prev] | [next] | [standalone]

#67782

On 19/08/2019 09:29, Juha Nieminen wrote:
> In comp.lang.c++ David Brown <david.brown@hesbynett.no> wrote:
>> For character data where plain ASCII won't do, utf-8 is the only sane 
>> choice IMHO.  Other formats - utf-16, utf-32, wchar_t, etc., might be 
>> used internally for dealing with historical API's.
> 
> The problem with UTF-8 is that it's quite complicated to parse and intepret,

It is not much worse than UTF-16.  In both cases, you have to deal with
multi-unit code points.  With UTF-16 you have the added complication of
endianness, which is not an issue with UTF-8.

> and is wasteful in terms of space if the majority of the text is non-ascii
> (quite typically almost all non-ascii characters take at least 3 bytes to
> store as UTF-8.)

No, a fair amount of non-ASCII takes just two bytes.  That applies to
most text written in Latin, Cyrillic, Greek, Hebrew and Arabic alphabets
- including accents, combined letters, and other variants.  Basically
most European scripts are handled by two bytes of UTF-8, matching the
size of UTF-16.

It is only if you have a lot of CJK characters that UTF-16 is more
efficient than UTF-8.

And the reality of document formats is that they usually contain a large
amount of ASCII characters for tags, codes, formatting, markings, etc.
For many formats, these outweigh text characters by orders of magnitude.
 CJK documents are more commonly encoded as UTF-8 rather than UTF-16,
partly because it makes them /smaller/.

Documents are then often packaged together in a lump (along with
pictures or other attachments) and then compressed.  With a large enough
sample, compressed Unicode documents will take similar sizes regardless
of the encoding.

UTF-8 is therefore the most space efficient in practice, and easier to
handle than UTF-16.

> It's the most efficient (spacewise) if the majority of the
> text is ascii, but one still has to be prepared to recover any unicode
> characters from it.
> 
> UTF-16 is simpler to parse (still not completely straightforward, but
> much simpler), and often more space-efficient if the majority of the
> text is non-ascii (such as Japanese).
> 

UTF-16 is a little simpler to parse - if you ignore the issue of
endianness.  But I don't think the details make much difference - with
UTF-8 and UTF-16 you need a decoder function that handles multi-unit codes.

[toc] | [prev] | [next] | [standalone]

#67915

UTF-16 takes
2 bytes for values 0-0xFFFF (Unicode Base Plane)
4 bytes for values 0x10000 - 0x10FFFF (All of current Unicode)

it CAN'T handle higher code points without some other extension which
would get ugly. That is why Unicode has currently defined that 0x10FFFF
will be the highest code point. I suspect at some point there will be
pressure to change that.

UTF-8 takes
1 byte for values 0 - 0x7F (basically ASCII)
2 bytes for values 0x80 - 0xFFF (most western languages)
3 bytes for values 0x1000 - 0xFFFF (Unicode base plane, much of this is
the CJK group)
4 bytes for values 0x10000 - 0x10FFFF, but can actually get to 0x1FFFF
if they extend unicode to higher code points)

in the extension (and earlier definition)
5 bytes for values 0x200000 - 0x3FFFFFF
6 bytes for values 0x400000 - 0x7FFFFFFF

UTF-16 is more compact only for code that has more characters in the
range 0x1000 - 0xFFFF than in the range 0-0x7F. This MIGHT hold for
Chinese/Japanese/Korean documents that have minimal markup in them. It
largely also requires that CJK document has a lot of 'basic' CJK
characters, as one big reason for extending the Unicode standard past 16
bits was that there were a LOT more characters needed then expected to
handle them.

It turns out that UTF-16 was a mistake. It came about because the first
attempts at Unicode thought it could be a 16 bit characters set, and
thus would be UCS-2. It only lives on really because Microsoft decided
to use UCS-2 in windows instead of UTF-8, which admittedly might not
have been a bad decision at the time. When UCS-2 had to become UTF-16 to
handle the full character set, it gets the worse of both sides, it still
is a multi-code character set (but maybe a lot of code can get away with
ignoring it and just not work right with 'esoteric' characters), is less
compact in the most common cases, and has a byte order issue for byte
streams (which most data streams are).

[toc] | [prev] | [next] | [standalone]

#67927

On 23/08/2019 20:14, Richard Damon wrote:
> 
> UTF-16 takes
> 2 bytes for values 0-0xFFFF (Unicode Base Plane)
> 4 bytes for values 0x10000 - 0x10FFFF (All of current Unicode)
> 
> it CAN'T handle higher code points without some other extension which
> would get ugly. That is why Unicode has currently defined that 0x10FFFF
> will be the highest code point. I suspect at some point there will be
> pressure to change that.

It is true that UTF-16 is limited to 1,114,111 characters.  (I didn't 
know that until recently.)  However, Unicode says: 
<https://www.unicode.org/faq/utf_bom.html#utf16-6>

"""
Q: Will UTF-16 ever be extended to more than a million characters?

A: No. Both Unicode and ISO 10646 have policies in place that formally 
limit future code assignment to the integer range that can be expressed 
with current UTF-16 (0 to 1,114,111). Even if other encoding forms (i.e. 
other UTFs) can represent larger intergers, these policies mean that all 
encoding forms will always represent the same set of characters. Over a 
million possible codes is far more than enough for the goal of Unicode 
of encoding characters, not glyphs. Unicode is not designed to encode 
arbitrary data. If you wanted, for example, to give each “instance of a 
character on paper throughout history” its own code, you might need 
trillions or quadrillions of such codes; noble as this effort might be, 
you would not use Unicode for such an encoding. [AF]
"""


> 
> UTF-8 takes
> 1 byte for values 0 - 0x7F (basically ASCII)
> 2 bytes for values 0x80 - 0xFFF (most western languages)
> 3 bytes for values 0x1000 - 0xFFFF (Unicode base plane, much of this is
> the CJK group)
> 4 bytes for values 0x10000 - 0x10FFFF, but can actually get to 0x1FFFF
> if they extend unicode to higher code points)
> 
> in the extension (and earlier definition)
> 5 bytes for values 0x200000 - 0x3FFFFFF
> 6 bytes for values 0x400000 - 0x7FFFFFFF
> 
> UTF-16 is more compact only for code that has more characters in the
> range 0x1000 - 0xFFFF than in the range 0-0x7F. This MIGHT hold for
> Chinese/Japanese/Korean documents that have minimal markup in them. It
> largely also requires that CJK document has a lot of 'basic' CJK
> characters, as one big reason for extending the Unicode standard past 16
> bits was that there were a LOT more characters needed then expected to
> handle them.

Yes.  The reality is that there are few documents that fall into this 
category, thus it is easier to stick to UTF-8 for all documents.

> 
> It turns out that UTF-16 was a mistake. It came about because the first
> attempts at Unicode thought it could be a 16 bit characters set, and
> thus would be UCS-2. It only lives on really because Microsoft decided
> to use UCS-2 in windows instead of UTF-8, which admittedly might not
> have been a bad decision at the time. When UCS-2 had to become UTF-16 to
> handle the full character set, it gets the worse of both sides, it still
> is a multi-code character set (but maybe a lot of code can get away with
> ignoring it and just not work right with 'esoteric' characters), is less
> compact in the most common cases, and has a byte order issue for byte
> streams (which most data streams are).
> 

Agreed.

[toc] | [prev] | [next] | [standalone]

#67928

On Sat, 24 Aug 2019 20:56:52 +0200, David Brown
<david.brown@hesbynett.no> wrote:

>On 23/08/2019 20:14, Richard Damon wrote:
>> 
>> UTF-16 takes
>> 2 bytes for values 0-0xFFFF (Unicode Base Plane)
>> 4 bytes for values 0x10000 - 0x10FFFF (All of current Unicode)
>> 
>> it CAN'T handle higher code points without some other extension which
>> would get ugly. That is why Unicode has currently defined that 0x10FFFF
>> will be the highest code point. I suspect at some point there will be
>> pressure to change that.
>
>It is true that UTF-16 is limited to 1,114,111 characters.  (I didn't 
>know that until recently.)  However, Unicode says: 
><https://www.unicode.org/faq/utf_bom.html#utf16-6>
>
>"""
>Q: Will UTF-16 ever be extended to more than a million characters?
>
>A: No. Both Unicode and ISO 10646 have policies in place that formally 
>limit future code assignment to the integer range that can be expressed 
>with current UTF-16 (0 to 1,114,111). Even if other encoding forms (i.e. 
>other UTFs) can represent larger intergers, these policies mean that all 
>encoding forms will always represent the same set of characters. Over a 
>million possible codes is far more than enough for the goal of Unicode 
>of encoding characters, not glyphs. Unicode is not designed to encode 
>arbitrary data. If you wanted, for example, to give each “instance of a 
>character on paper throughout history” its own code, you might need 
>trillions or quadrillions of such codes; noble as this effort might be, 
>you would not use Unicode for such an encoding. [AF]
>"""


OTOH, the original Unicode spec also specifically said it was a 16-bit
code.  So these things sometime change.

[toc] | [prev] | [next] | [standalone]

#67933

On 25/08/2019 06:43, Robert Wessel wrote:
> On Sat, 24 Aug 2019 20:56:52 +0200, David Brown
> <david.brown@hesbynett.no> wrote:
> 
>> On 23/08/2019 20:14, Richard Damon wrote:
>>>
>>> UTF-16 takes
>>> 2 bytes for values 0-0xFFFF (Unicode Base Plane)
>>> 4 bytes for values 0x10000 - 0x10FFFF (All of current Unicode)
>>>
>>> it CAN'T handle higher code points without some other extension which
>>> would get ugly. That is why Unicode has currently defined that 0x10FFFF
>>> will be the highest code point. I suspect at some point there will be
>>> pressure to change that.
>>
>> It is true that UTF-16 is limited to 1,114,111 characters.  (I didn't 
>> know that until recently.)  However, Unicode says: 
>> <https://www.unicode.org/faq/utf_bom.html#utf16-6>
>>
>> """
>> Q: Will UTF-16 ever be extended to more than a million characters?
>>
>> A: No. Both Unicode and ISO 10646 have policies in place that formally 
>> limit future code assignment to the integer range that can be expressed 
>> with current UTF-16 (0 to 1,114,111). Even if other encoding forms (i.e. 
>> other UTFs) can represent larger intergers, these policies mean that all 
>> encoding forms will always represent the same set of characters. Over a 
>> million possible codes is far more than enough for the goal of Unicode 
>> of encoding characters, not glyphs. Unicode is not designed to encode 
>> arbitrary data. If you wanted, for example, to give each “instance of a 
>> character on paper throughout history” its own code, you might need 
>> trillions or quadrillions of such codes; noble as this effort might be, 
>> you would not use Unicode for such an encoding. [AF]
>> """
> 
> 
> OTOH, the original Unicode spec also specifically said it was a 16-bit
> code.  So these things sometime change.
> 

But surely now that Unicode has support for Ogham Runes, it /can/ be
carved in stone :-)

You are right, of course - guarantees about future changes are hard to
make.  I guess that if Unicode ever starts to encroach on the UTF-16
limits, then UTF-16 will be dropped.

[toc] | [prev] | [next] | [standalone]

#67944

On 8/25/19 3:11 AM, David Brown wrote:
> On 25/08/2019 06:43, Robert Wessel wrote:
>> On Sat, 24 Aug 2019 20:56:52 +0200, David Brown
>> <david.brown@hesbynett.no> wrote:
>>
>>> On 23/08/2019 20:14, Richard Damon wrote:
>>>>
>>>> UTF-16 takes
>>>> 2 bytes for values 0-0xFFFF (Unicode Base Plane)
>>>> 4 bytes for values 0x10000 - 0x10FFFF (All of current Unicode)
>>>>
>>>> it CAN'T handle higher code points without some other extension which
>>>> would get ugly. That is why Unicode has currently defined that 0x10FFFF
>>>> will be the highest code point. I suspect at some point there will be
>>>> pressure to change that.
>>>
>>> It is true that UTF-16 is limited to 1,114,111 characters.  (I didn't 
>>> know that until recently.)  However, Unicode says: 
>>> <https://www.unicode.org/faq/utf_bom.html#utf16-6>
>>>
>>> """
>>> Q: Will UTF-16 ever be extended to more than a million characters?
>>>
>>> A: No. Both Unicode and ISO 10646 have policies in place that formally 
>>> limit future code assignment to the integer range that can be expressed 
>>> with current UTF-16 (0 to 1,114,111). Even if other encoding forms (i.e. 
>>> other UTFs) can represent larger intergers, these policies mean that all 
>>> encoding forms will always represent the same set of characters. Over a 
>>> million possible codes is far more than enough for the goal of Unicode 
>>> of encoding characters, not glyphs. Unicode is not designed to encode 
>>> arbitrary data. If you wanted, for example, to give each “instance of a 
>>> character on paper throughout history” its own code, you might need 
>>> trillions or quadrillions of such codes; noble as this effort might be, 
>>> you would not use Unicode for such an encoding. [AF]
>>> """
>>
>>
>> OTOH, the original Unicode spec also specifically said it was a 16-bit
>> code.  So these things sometime change.
>>
> 
> But surely now that Unicode has support for Ogham Runes, it /can/ be
> carved in stone :-)
> 
> You are right, of course - guarantees about future changes are hard to
> make.  I guess that if Unicode ever starts to encroach on the UTF-16
> limits, then UTF-16 will be dropped.
> 

Yes, I know that statement, but I also know that practically, at some
point the Unicode consortium will have to make some tough decisions on
code points, and at some point will renege on the policy that UTF-16 as
it currently exists will be able to access every code point in all
future versions of Unicode. One thing to remember, and was pointed out,
UTF-16 only exists BECAUSE a previous promise was broken, and UCS-2
wasn't good enough anymore for all of Unicode.

One thing that could cause this is if the CJK set gets fixed and removed
from being second class code points. It may not be well known, but in
the process of developing Unicode, in an attempt to make it work, and to
have enough code points, the characters in the three major pictograph
languages where 'unified' and characters that meant roughly the same
thing were combined into a single code point. It really wouldn't be that
different than saying that we don't need a separate code point for the
character Pi, as that is just the way to write P in Greek. (It isn't
just a typography issue, as it isn't just how you write a given
character, but they are 'similar' words from distinct languages.) The
unification was accepted as it seemed to only possible option to try to
make Unicode a 16 bit character set, and it did allow them to make the
most common characters in the 16 bit basic plane.

[toc] | [prev] | [next] | [standalone]

#67947

On 25/08/2019 16:33, Richard Damon wrote:
> On 8/25/19 3:11 AM, David Brown wrote:
>> On 25/08/2019 06:43, Robert Wessel wrote:
>>> On Sat, 24 Aug 2019 20:56:52 +0200, David Brown
>>> <david.brown@hesbynett.no> wrote:
>>>
>>>> On 23/08/2019 20:14, Richard Damon wrote:
>>>>>
>>>>> UTF-16 takes
>>>>> 2 bytes for values 0-0xFFFF (Unicode Base Plane)
>>>>> 4 bytes for values 0x10000 - 0x10FFFF (All of current Unicode)
>>>>>
>>>>> it CAN'T handle higher code points without some other extension which
>>>>> would get ugly. That is why Unicode has currently defined that 0x10FFFF
>>>>> will be the highest code point. I suspect at some point there will be
>>>>> pressure to change that.
>>>>
>>>> It is true that UTF-16 is limited to 1,114,111 characters.  (I didn't
>>>> know that until recently.)  However, Unicode says:
>>>> <https://www.unicode.org/faq/utf_bom.html#utf16-6>
>>>>
>>>> """
>>>> Q: Will UTF-16 ever be extended to more than a million characters?
>>>>
>>>> A: No. Both Unicode and ISO 10646 have policies in place that formally
>>>> limit future code assignment to the integer range that can be expressed
>>>> with current UTF-16 (0 to 1,114,111). Even if other encoding forms (i.e.
>>>> other UTFs) can represent larger intergers, these policies mean that all
>>>> encoding forms will always represent the same set of characters. Over a
>>>> million possible codes is far more than enough for the goal of Unicode
>>>> of encoding characters, not glyphs. Unicode is not designed to encode
>>>> arbitrary data. If you wanted, for example, to give each “instance of a
>>>> character on paper throughout history” its own code, you might need
>>>> trillions or quadrillions of such codes; noble as this effort might be,
>>>> you would not use Unicode for such an encoding. [AF]
>>>> """
>>>
>>>
>>> OTOH, the original Unicode spec also specifically said it was a 16-bit
>>> code.  So these things sometime change.
>>>
>>
>> But surely now that Unicode has support for Ogham Runes, it /can/ be
>> carved in stone :-)
>>
>> You are right, of course - guarantees about future changes are hard to
>> make.  I guess that if Unicode ever starts to encroach on the UTF-16
>> limits, then UTF-16 will be dropped.
>>
> 
> Yes, I know that statement, but I also know that practically, at some
> point the Unicode consortium will have to make some tough decisions on
> code points, and at some point will renege on the policy that UTF-16 as
> it currently exists will be able to access every code point in all
> future versions of Unicode. One thing to remember, and was pointed out,
> UTF-16 only exists BECAUSE a previous promise was broken, and UCS-2
> wasn't good enough anymore for all of Unicode.
> 
> One thing that could cause this is if the CJK set gets fixed and removed
> from being second class code points. It may not be well known, but in
> the process of developing Unicode, in an attempt to make it work, and to
> have enough code points, the characters in the three major pictograph
> languages where 'unified' and characters that meant roughly the same
> thing were combined into a single code point. It really wouldn't be that
> different than saying that we don't need a separate code point for the
> character Pi, as that is just the way to write P in Greek. (It isn't
> just a typography issue, as it isn't just how you write a given
> character, but they are 'similar' words from distinct languages.) The
> unification was accepted as it seemed to only possible option to try to
> make Unicode a 16 bit character set, and it did allow them to make the
> most common characters in the 16 bit basic plane.
> 

That is interesting information - thanks for posting it.

Personally, I won't shed a tear if UTF-16 disappears.  But the number of 
characters has to increase by a factor of 8 or so before it becomes 
necessary, which could allow for splitting up C, J and K.

[toc] | [prev] | [next] | [standalone]

#67946

On Sun, 25 Aug 2019 12:11:46 +0200, David Brown
<david.brown@hesbynett.no> wrote:

>On 25/08/2019 06:43, Robert Wessel wrote:
>> On Sat, 24 Aug 2019 20:56:52 +0200, David Brown
>> <david.brown@hesbynett.no> wrote:
>> 
>>> On 23/08/2019 20:14, Richard Damon wrote:
>>>>
>>>> UTF-16 takes
>>>> 2 bytes for values 0-0xFFFF (Unicode Base Plane)
>>>> 4 bytes for values 0x10000 - 0x10FFFF (All of current Unicode)
>>>>
>>>> it CAN'T handle higher code points without some other extension which
>>>> would get ugly. That is why Unicode has currently defined that 0x10FFFF
>>>> will be the highest code point. I suspect at some point there will be
>>>> pressure to change that.
>>>
>>> It is true that UTF-16 is limited to 1,114,111 characters.  (I didn't 
>>> know that until recently.)  However, Unicode says: 
>>> <https://www.unicode.org/faq/utf_bom.html#utf16-6>
>>>
>>> """
>>> Q: Will UTF-16 ever be extended to more than a million characters?
>>>
>>> A: No. Both Unicode and ISO 10646 have policies in place that formally 
>>> limit future code assignment to the integer range that can be expressed 
>>> with current UTF-16 (0 to 1,114,111). Even if other encoding forms (i.e. 
>>> other UTFs) can represent larger intergers, these policies mean that all 
>>> encoding forms will always represent the same set of characters. Over a 
>>> million possible codes is far more than enough for the goal of Unicode 
>>> of encoding characters, not glyphs. Unicode is not designed to encode 
>>> arbitrary data. If you wanted, for example, to give each “instance of a 
>>> character on paper throughout history” its own code, you might need 
>>> trillions or quadrillions of such codes; noble as this effort might be, 
>>> you would not use Unicode for such an encoding. [AF]
>>> """
>> 
>> 
>> OTOH, the original Unicode spec also specifically said it was a 16-bit
>> code.  So these things sometime change.
>> 
>
>But surely now that Unicode has support for Ogham Runes, it /can/ be
>carved in stone :-)
>
>You are right, of course - guarantees about future changes are hard to
>make.  I guess that if Unicode ever starts to encroach on the UTF-16
>limits, then UTF-16 will be dropped.


The problem is that dropping UTF-16 is a nasty hit to Windows, Java
and Javascript.  Those are not minor users in the world.

I suspect what would happen at that point is that an additional block
of surrogate characters will be assigned, so that we can have both the
surrogate pairs we now have for UTF-16, plus some new surrogate
"triples".  You could assign a block of 64 code points from the BMP,
for example (if nothing else, grab some more of the private use area
like they did the last time), and get six extra bits for the "triple"
(or assign two 64-code-point blocks and be able to do "quads" as
well).  So the UTF-16 users would get hit with about the same thing
that happened in the transition from UCS-2.

With hindsight*, something like that should have been done the first
time.  Maybe have three or four blocks of 512 surrogate code points
(instead of 2x2048), and allow two or three (or four) surrogate
characters in a sequences (not really unlike what was done with
UTF-8).


*Although calling that a lack of foresight may be too kind.  How many
times have we been bitten by inadequate address spaces now?  65536,
errr..., 1,114,112 character ought to be enough for anyone!

[toc] | [prev] | [next] | [standalone]

#67784

On 8/17/2019 8:08 AM, David Brown wrote:
> On 16/08/2019 21:04, Lynn McGuire wrote:
>> "C++: Size Matters in Platform Compatibility"
>>
>> <https://www.codeproject.com/Tips/5164768/Cplusplus-Size-Matters-in-Platform-Compatibility> 
>>
> 
> Please put < > marks around links.

Why ?

Lynn

[toc] | [prev] | [next] | [standalone]

#67785

Lynn McGuire <lynnmcguire5@gmail.com> writes:
>On 8/17/2019 8:08 AM, David Brown wrote:
>> On 16/08/2019 21:04, Lynn McGuire wrote:
>>> "C++: Size Matters in Platform Compatibility"
>>>
>>> <https://www.codeproject.com/Tips/5164768/Cplusplus-Size-Matters-in-Platform-Compatibility> 
>>>
>> 
>> Please put < > marks around links.
>
>Why ?
>

Because it will prevent many Usenet clients from breaking the URL into
multiple lines and it avoids confusion as to whether adjacent punctuation symbols
are part of the URL or not.

[toc] | [prev] | [next] | [standalone]

#67798

On 19/08/2019 22:17, Scott Lurndal wrote:
> Lynn McGuire <lynnmcguire5@gmail.com> writes:
>> On 8/17/2019 8:08 AM, David Brown wrote:
>>> On 16/08/2019 21:04, Lynn McGuire wrote:
>>>> "C++: Size Matters in Platform Compatibility"
>>>>
>>>> <https://www.codeproject.com/Tips/5164768/Cplusplus-Size-Matters-in-Platform-Compatibility>
>>>>
>>>
>>> Please put < > marks around links.
>>
>> Why ?
>>
> 
> Because it will prevent many Usenet clients from breaking the URL into
> multiple lines and it avoids confusion as to whether adjacent punctuation symbols
> are part of the URL or not.
> 

It also means that the URL will continue to work with many clients, even 
if it /is/ broken into multiple lines.

Lynn, I added the marks to your URL when I made the reply.  By now, it 
will have wrapped to two lines - you can test it and see if it still 
works as a link for your client.

[toc] | [prev] | [next] | [standalone]

#67820

On 8/20/2019 1:21 AM, David Brown wrote:
> On 19/08/2019 22:17, Scott Lurndal wrote:
>> Lynn McGuire <lynnmcguire5@gmail.com> writes:
>>> On 8/17/2019 8:08 AM, David Brown wrote:
>>>> On 16/08/2019 21:04, Lynn McGuire wrote:
>>>>> "C++: Size Matters in Platform Compatibility"
>>>>>
>>>>> <https://www.codeproject.com/Tips/5164768/Cplusplus-Size-Matters-in-Platform-Compatibility> 
>>>>>
>>>>>
>>>>
>>>> Please put < > marks around links.
>>>
>>> Why ?
>>>
>>
>> Because it will prevent many Usenet clients from breaking the URL into
>> multiple lines and it avoids confusion as to whether adjacent 
>> punctuation symbols
>> are part of the URL or not.
>>
> 
> It also means that the URL will continue to work with many clients, even 
> if it /is/ broken into multiple lines.
> 
> Lynn, I added the marks to your URL when I made the reply.  By now, it 
> will have wrapped to two lines - you can test it and see if it still 
> works as a link for your client.

It does.  I use Thunderbird on Windows 7 x64 as both my email and news 
client.

Thanks,
Lynn

[toc] | [prev] | [next] | [standalone]

#67844

On 8/19/19 4:05 PM, Lynn McGuire wrote:
> On 8/17/2019 8:08 AM, David Brown wrote:
>> On 16/08/2019 21:04, Lynn McGuire wrote:
>>> "C++: Size Matters in Platform Compatibility"
>>>
>>> <https://www.codeproject.com/Tips/5164768/Cplusplus-Size-Matters-in-Platform-Compatibility> 
>>>
>>
>> Please put < > marks around links.
> 
> Why ?

Doing so is specified by one of the relevant international standards (I
don't remember which one - I last read that specification about 25 years
ago). Standards are often ignored, but applications whose designers did
pay attention to that clause of that standard may handle links enclosed
in <> better than links that are not so delimited.

[toc] | [prev] | [next] | [standalone]

#67750

Lynn McGuire:

> "C++: Size Matters in Platform Compatibility"
> https://www.codeproject.com/Tips/5164768/Cplusplus-Size-Matters-in-Platform-Compatibility
> Interesting.  Especially on storing Unicode as UTF-8.

My reaction to this article is in fir's manner: highly
skeptical.  The author says obvious things that are much
better discussed in network data transfer and socket
programming in particular.  He adds nothing new and original
except wrong conclusions, e.g.:

   time_t is not guaranteed to be interoperable between
   platforms, so it is best to store time as text and
   convert to time_t accordingly.

whereas the natural solution is any fixed-width binary
datetime format.  Loth to do calendar arithmetics, I should
pass the fields separately, e.g.:

                  Date: 24           Time: 24
          +-----------------+--------------------+
          |BC? Year Mon Day | Hours Min Sec 1/128|
          | 1   14   4   5  |   5    6   6    7  |
          +-----------------+--------------------+

The date and time are separable three-byte blocks.  Each
block is a packed structure with the fields interpreted as
big-endian, according to the network byte order.

-- 
()  ascii ribbon campaign -- against html e-mail
/\  http://preview.tinyurl.com/qcy6mjc [archived]

[toc] | [prev] | [next] | [standalone]

#67755

Anton Shepelev <anton.txt@gmail.com> writes:

> ...  Loth to do calendar arithmetics, I should
> pass the fields separately, e.g.:
>
>                   Date: 24           Time: 24
>           +-----------------+--------------------+
>           |BC? Year Mon Day | Hours Min Sec 1/128|
>           | 1   14   4   5  |   5    6   6    7  |
>           +-----------------+--------------------+
>
> The date and time are separable three-byte blocks.  Each
> block is a packed structure with the fields interpreted as
> big-endian, according to the network byte order.

For the price of two bytes more you can make the ordering explicit and
and avoid any shifting and masking.  Make everything an octet:

   century, year in century, month, day, hour, minute, second, 1/128ths

(Dates BC require the first byte to be signed: 6BC is year 94 in century
-1.)

You could, of course, use units of 512 years instead at the expense of
having a slightly less human-debuggable format.  You gain just having to
do a shift rather than a multiply by 100.

-- 
Ben.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.c++

csiph-web