Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #25652 > unrolled thread

Re: Why No Supplemental Characters In Character Literals?

Back to article view | Back to comp.lang.java.programmer

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Why No Supplemental Characters In Character Literals? Lawrence D'Oliveiro <ldo@geek-central.gen.new_zealand> - 2011-02-04 19:59 +1300
    Re: Why No Supplemental Characters In Character Literals? "Mike Schilling" <mscottschilling@hotmail.com> - 2011-02-04 17:02 -0800
      Re: Why No Supplemental Characters In Character Literals? Ken Wesson <kwesson@gmail.com> - 2011-02-05 04:21 +0100
    Re: Why No Supplemental Characters In Character Literals? Joshua Cranmer <Pidgeot18@verizon.invalid> - 2011-02-04 19:05 -0500
      Re: Why No Supplemental Characters In Character Literals? Arne Vajhøj <arne@vajhoej.dk> - 2011-02-04 19:56 -0500
      Re: Why No Supplemental Characters In Character Literals? "Mike Schilling" <mscottschilling@hotmail.com> - 2011-02-04 16:37 -0800
    Re: Why No Supplemental Characters In Character Literals? "Mike Schilling" <mscottschilling@hotmail.com> - 2011-02-04 00:22 -0800
      Re: Why No Supplemental Characters In Character Literals? Roedy Green <see_website@mindprod.com.invalid> - 2011-02-04 15:03 -0800
      Re: Why No Supplemental Characters In Character Literals? Arne Vajhøj <arne@vajhoej.dk> - 2011-02-04 18:04 -0500
      Re: Why No Supplemental Characters In Character Literals? Lew <noone@lewscanon.com> - 2011-02-04 07:49 -0500
      Re: Why No Supplemental Characters In Character Literals? Lawrence D'Oliveiro <ldo@geek-central.gen.new_zealand> - 2011-02-05 11:26 +1300
    Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals? Joshua Cranmer <Pidgeot18@verizon.invalid> - 2011-02-04 19:13 -0500
      Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals? Arne Vajhøj <arne@vajhoej.dk> - 2011-02-04 20:08 -0500
    Re: Why No Supplemental Characters In Character Literals? Daniele Futtorovic <da.futt.news@laposte.net.invalid> - 2011-02-04 18:37 +0100
      Re: Why No Supplemental Characters In Character Literals? markspace <nospam@nowhere.com> - 2011-02-04 11:27 -0800
    Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals? Joshua Cranmer <Pidgeot18@verizon.invalid> - 2011-02-04 17:28 -0500
    Re: Why No Supplemental Characters In Character Literals? "Mike Schilling" <mscottschilling@hotmail.com> - 2011-02-04 09:10 -0800
      Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals? Roedy Green <see_website@mindprod.com.invalid> - 2011-02-04 15:22 -0800
        Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals? Arne Vajhøj <arne@vajhoej.dk> - 2011-02-04 18:41 -0500
      Re: Why No Supplemental Characters In Character Literals? Arne Vajhøj <arne@vajhoej.dk> - 2011-02-04 18:12 -0500
      Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals? Tom Anderson <twic@urchin.earth.li> - 2011-02-04 21:30 +0000
        Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals? Ken Wesson <kwesson@gmail.com> - 2011-02-05 04:25 +0100
      Re: Why No Supplemental Characters In Character Literals? Arne Vajhøj <arne@vajhoej.dk> - 2011-02-04 12:33 -0500
      Re: Why No Supplemental Characters In Character Literals? Joshua Cranmer <Pidgeot18@verizon.invalid> - 2011-02-04 13:44 -0500
        Re: Why No Supplemental Characters In Character Literals? Roedy Green <see_website@mindprod.com.invalid> - 2011-02-04 15:08 -0800
    Re: Why No Supplemental Characters In Character Literals? Lew <lew@lewscanon.com> - 2011-02-04 12:43 -0800
    Re: Why No Supplemental Characters In Character Literals? Arne Vajhøj <arne@vajhoej.dk> - 2011-02-04 10:49 -0500
    Re: Why No Supplemental Characters In Character Literals? Joshua Cranmer <Pidgeot18@verizon.invalid> - 2011-02-04 08:04 -0500

Page 1 of 2  [1] 2  Next page →

#25652 — Re: Why No Supplemental Characters In Character Literals?

In message <iig6j2$dul$2@news.albasani.net>, Lew wrote:

> On 02/04/2011 12:59 AM, Lawrence D'Oliveiro wrote:
>
>> Why was it decreed in the language spec that characters beyond U+FFFF are
>> not allowed in character literals, when they are allowed everywhere else
>> (in string literals, in the program text, in character and string values
>> etc)?
> 
> Because a 'char' type holds only 16 bits.

No it doesn’t. Otherwise you wouldn’t be allowed supplementary characters in 
character and string values. Which you are.

[toc] | [next] | [standalone]

#25657

"Arne Vajhøj" <arne@vajhoej.dk> wrote in message 
news:4d4ca055$0$23765$14726298@news.sunsite.dk...
> On 04-02-2011 19:37, Mike Schilling wrote:
>> "Joshua Cranmer" <Pidgeot18@verizon.invalid> wrote in message
>> news:iii493$nn8$1@news.eternal-september.org...
>>> On 02/04/2011 05:26 PM, Lawrence D'Oliveiro wrote:
>>>> In message<iigcva$90q$1@news.eternal-september.org>, Mike Schilling
>>>> wrote:
>>>>
>>>>> Yes, it does (contain 16 bits.)
>>>>
>>>> Yeah, I didn’t realize it was spelled out that way in the original
>>>> language
>>>> spec. What a short-sighted decision.
>>>
>>> It would have been stupider to have not specified a guaranteed size
>>> for char. Take C (+ POSIX), where the definitions of sizes are very
>>> loosely defined, and you very quickly get non-portable code. Yes, you
>>> can in theory change the size of, say, time_t independently of other
>>> types, but it doesn't do you much good if half the C code assumes
>>> sizeof(time_t) == sizeof(int). Pinning down the sizes of the types was
>>> a _very good_ move on Java's part.
>>>
>>>> Why was there a need to define the size of a character at all? Even
>>>> in the
>>>> early days of the unification of Unicode and ISO-10646, there was
>>>> already
>>>> provision for UCS-4. Did they really think that could safely be 
>>>> ignored?
>>>
>>> Knowing the results of other properly Unicode-aware code in the first
>>> days of Unicode, I believe that Unicode quite heavily gave an
>>> impression of "Unicode == 16 bit". Java is not the only major platform
>>> to be bitten by now-Unicode-is-32-bits... the Windows platform has
>>> 16-bit characters embedded into it.
>>
>> .NET, which in several cases took advantage of following Java to correct
>> some of its mistakes (e.g. signed bytes), didn’t fix this one.
>
> Which is a bit surprising since high code points were introduced
> when .NET came around.
>
> But they probably had a compatibility issue with p/Invoke and
> Win32 API, COM interop, C++ mixed mode etc. that all had
> to work with existing Win32 model of 16 bit wchars.

Or, relentless micro-optimizers that they are, Microsoft wasn't willing to 
bite off the size/performance issues. 

[toc] | [prev] | [next] | [standalone]

#25778

On Fri, 04 Feb 2011 17:02:37 -0800, Mike Schilling wrote:

> "Arne Vajhøj" <arne@vajhoej.dk> wrote in message
> news:4d4ca055$0$23765$14726298@news.sunsite.dk...
>> On 04-02-2011 19:37, Mike Schilling wrote:
>>> "Joshua Cranmer" <Pidgeot18@verizon.invalid> wrote in message
>>> news:iii493$nn8$1@news.eternal-september.org...
>>>> On 02/04/2011 05:26 PM, Lawrence D'Oliveiro wrote:
>>>>> In message<iigcva$90q$1@news.eternal-september.org>, Mike Schilling
>>>>> wrote:
>>>>>
>>>>>> Yes, it does (contain 16 bits.)
>>>>>
>>>>> Yeah, I didn’t realize it was spelled out that way in the original
>>>>> language
>>>>> spec. What a short-sighted decision.
>>>>
>>>> It would have been stupider to have not specified a guaranteed size
>>>> for char. Take C (+ POSIX), where the definitions of sizes are very
>>>> loosely defined, and you very quickly get non-portable code. Yes, you
>>>> can in theory change the size of, say, time_t independently of other
>>>> types, but it doesn't do you much good if half the C code assumes
>>>> sizeof(time_t) == sizeof(int). Pinning down the sizes of the types
>>>> was a _very good_ move on Java's part.
>>>>
>>>>> Why was there a need to define the size of a character at all? Even
>>>>> in the
>>>>> early days of the unification of Unicode and ISO-10646, there was
>>>>> already
>>>>> provision for UCS-4. Did they really think that could safely be
>>>>> ignored?
>>>>
>>>> Knowing the results of other properly Unicode-aware code in the first
>>>> days of Unicode, I believe that Unicode quite heavily gave an
>>>> impression of "Unicode == 16 bit". Java is not the only major
>>>> platform to be bitten by now-Unicode-is-32-bits... the Windows
>>>> platform has 16-bit characters embedded into it.
>>>
>>> .NET, which in several cases took advantage of following Java to
>>> correct some of its mistakes (e.g. signed bytes), didn’t fix this one.
>>
>> Which is a bit surprising since high code points were introduced when
>> .NET came around.
>>
>> But they probably had a compatibility issue with p/Invoke and Win32
>> API, COM interop, C++ mixed mode etc. that all had to work with
>> existing Win32 model of 16 bit wchars.
> 
> Or, relentless micro-optimizers that they are, Microsoft wasn't willing
> to bite off the size/performance issues.

Relentless micro-optimizers of what, their cashflow?

[toc] | [prev] | [next] | [standalone]

#25668

On 02/04/2011 05:26 PM, Lawrence D'Oliveiro wrote:
> In message<iigcva$90q$1@news.eternal-september.org>, Mike Schilling wrote:
>
>> Yes, it does (contain 16 bits.)
>
> Yeah, I didn’t realize it was spelled out that way in the original language
> spec. What a short-sighted decision.

It would have been stupider to have not specified a guaranteed size for 
char. Take C (+ POSIX), where the definitions of sizes are very loosely 
defined, and you very quickly get non-portable code. Yes, you can in 
theory change the size of, say, time_t independently of other types, but 
it doesn't do you much good if half the C code assumes sizeof(time_t) == 
sizeof(int). Pinning down the sizes of the types was a _very good_ move 
on Java's part.

> Why was there a need to define the size of a character at all? Even in the
> early days of the unification of Unicode and ISO-10646, there was already
> provision for UCS-4. Did they really think that could safely be ignored?

Knowing the results of other properly Unicode-aware code in the first 
days of Unicode, I believe that Unicode quite heavily gave an impression 
of "Unicode == 16 bit". Java is not the only major platform to be bitten 
by now-Unicode-is-32-bits... the Windows platform has 16-bit characters 
embedded into it.

-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#25813

On 04-02-2011 19:37, Mike Schilling wrote:
> "Joshua Cranmer" <Pidgeot18@verizon.invalid> wrote in message
> news:iii493$nn8$1@news.eternal-september.org...
>> On 02/04/2011 05:26 PM, Lawrence D'Oliveiro wrote:
>>> In message<iigcva$90q$1@news.eternal-september.org>, Mike Schilling
>>> wrote:
>>>
>>>> Yes, it does (contain 16 bits.)
>>>
>>> Yeah, I didn’t realize it was spelled out that way in the original
>>> language
>>> spec. What a short-sighted decision.
>>
>> It would have been stupider to have not specified a guaranteed size
>> for char. Take C (+ POSIX), where the definitions of sizes are very
>> loosely defined, and you very quickly get non-portable code. Yes, you
>> can in theory change the size of, say, time_t independently of other
>> types, but it doesn't do you much good if half the C code assumes
>> sizeof(time_t) == sizeof(int). Pinning down the sizes of the types was
>> a _very good_ move on Java's part.
>>
>>> Why was there a need to define the size of a character at all? Even
>>> in the
>>> early days of the unification of Unicode and ISO-10646, there was
>>> already
>>> provision for UCS-4. Did they really think that could safely be ignored?
>>
>> Knowing the results of other properly Unicode-aware code in the first
>> days of Unicode, I believe that Unicode quite heavily gave an
>> impression of "Unicode == 16 bit". Java is not the only major platform
>> to be bitten by now-Unicode-is-32-bits... the Windows platform has
>> 16-bit characters embedded into it.
>
> .NET, which in several cases took advantage of following Java to correct
> some of its mistakes (e.g. signed bytes), didn’t fix this one.

Which is a bit surprising since high code points were introduced
when .NET came around.

But they probably had a compatibility issue with p/Invoke and
Win32 API, COM interop, C++ mixed mode etc. that all had
to work with existing Win32 model of 16 bit wchars.

Arne

[toc] | [prev] | [next] | [standalone]

#25887

"Joshua Cranmer" <Pidgeot18@verizon.invalid> wrote in message 
news:iii493$nn8$1@news.eternal-september.org...
> On 02/04/2011 05:26 PM, Lawrence D'Oliveiro wrote:
>> In message<iigcva$90q$1@news.eternal-september.org>, Mike Schilling 
>> wrote:
>>
>>> Yes, it does (contain 16 bits.)
>>
>> Yeah, I didn’t realize it was spelled out that way in the original 
>> language
>> spec. What a short-sighted decision.
>
> It would have been stupider to have not specified a guaranteed size for 
> char. Take C (+ POSIX), where the definitions of sizes are very loosely 
> defined, and you very quickly get non-portable code. Yes, you can in 
> theory change the size of, say, time_t independently of other types, but 
> it doesn't do you much good if half the C code assumes sizeof(time_t) == 
> sizeof(int). Pinning down the sizes of the types was a _very good_ move on 
> Java's part.
>
>> Why was there a need to define the size of a character at all? Even in 
>> the
>> early days of the unification of Unicode and ISO-10646, there was already
>> provision for UCS-4. Did they really think that could safely be ignored?
>
> Knowing the results of other properly Unicode-aware code in the first days 
> of Unicode, I believe that Unicode quite heavily gave an impression of 
> "Unicode == 16 bit". Java is not the only major platform to be bitten by 
> now-Unicode-is-32-bits... the Windows platform has 16-bit characters 
> embedded into it.


.NET, which in several cases took advantage of following Java to correct 
some of its mistakes (e.g. signed bytes), didn’t fix this one. 

[toc] | [prev] | [next] | [standalone]

#25732

"Lawrence D'Oliveiro" <ldo@geek-central.gen.new_zealand> wrote in message 
news:iig84e$uqu$1@lust.ihug.co.nz...
> In message <iig6j2$dul$2@news.albasani.net>, Lew wrote:
>
>> On 02/04/2011 12:59 AM, Lawrence D'Oliveiro wrote:
>>
>>> Why was it decreed in the language spec that characters beyond U+FFFF 
>>> are
>>> not allowed in character literals, when they are allowed everywhere else
>>> (in string literals, in the program text, in character and string values
>>> etc)?
>>
>> Because a 'char' type holds only 16 bits.
>
> No it doesn’t. Otherwise you wouldn’t be allowed supplementary characters 
> in
> character and string values. Which you are.

Yes, it does (contain 16 bits.) It was defined to do so before there were 
supplemental characters, and there was no way to extend it without breaking 
compatibility with some older programs.

You can't put a supplementary character in a char.  You can put them in 
strings, but only encoded as UTF-16, i.e. into two 16-bit chars. 

[toc] | [prev] | [next] | [standalone]

#25837

On Sat, 05 Feb 2011 11:26:53 +1300, Lawrence D'Oliveiro
<ldo@geek-central.gen.new_zealand> wrote, quoted or indirectly quoted
someone who said :

>Why was there a need to define the size of a character at all? 

Because C did worked that way and lead to non-wora code.
-- 
Roedy Green Canadian Mind Products
http://mindprod.com
To err is human, but to really foul things up requires a computer.
~ Farmer's Almanac
It is breathtaking how a misplaced comma in a computer program can
shred megabytes of data in seconds.

[toc] | [prev] | [next] | [standalone]

#25842

On 04-02-2011 17:26, Lawrence D'Oliveiro wrote:
> In message<iigcva$90q$1@news.eternal-september.org>, Mike Schilling wrote:
>> Yes, it does (contain 16 bits.)
>
> Yeah, I didn’t realize it was spelled out that way in the original language
> spec.

It is. And give that you in another thread talk about problems in JLS,
then I think you should have read it.

It should also be in most Java beginners books.

It is also in the Java tutorial:
 
http://download.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html

 >          What a short-sighted decision.

Back then Unicode was 16 bit.

The increase in bit was done in 1996 after the release of Java 1.0.

>> It was defined to do so before there were supplemental characters ...
>
> Why was there a need to define the size of a character at all?

Well defines data types is a very good thing.

>                                                                Even in the
> early days of the unification of Unicode and ISO-10646, there was already
> provision for UCS-4.

Java decided to do Unicode. And at that time 16 bit was sufficient
for that.

>                Did they really think that could safely be ignored?

Apparently yes.

Given that the 16 bit had just replaced 8 bit, then I think it
is understandable.

Arne

[toc] | [prev] | [next] | [standalone]

#26031

Lawrence D'Oliveiro wrote:
>>>> Why was it decreed in the language spec that characters beyond U+FFFF are
>>>> not allowed in character literals, when they are allowed everywhere else
>>>> (in string literals, in the program text, in character and string values
>>>> etc)?

It takes TWO 'char' values to represent a supplemental character.  'char' != 
"character".

READ the documentation.

Lew wrote:
>>> Because a 'char' type holds only 16 bits.

Lawrence D'Oliveiro wrote:
>> No it doesn’t. Otherwise you wouldn’t be allowed supplementary characters in
>> character and string values. Which you are.

I have an idea for you to try - check the documentation.
<http://java.sun.com/docs/books/jls/third_edition/html/typesValues.html#4.2.1>

and you see in §4.2: "... char, whose values are 16-bit unsigned integers ..."

Mike Schilling wrote:
> Yes, it does (contain 16 bits.) It was defined to do so before there were
> supplemental characters, and there was no way to extend it without breaking
> compatibility with some older programs.
>
> You can't put a supplementary character in a char. You can put them in
> strings, but only encoded as UTF-16, i.e. into two 16-bit chars.

As the tutorials and JLS tell you, should you deign to read the documentation. 
  (It's not a bad idea to do so.)

-- 
Lew
Ceci n'est pas une fenêtre.
.___________.
|###] | [###|
|##/  | *\##|
|#/ * |   \#|
|#----|----#|
||    |  * ||
|o *  |    o|
|_____|_____|
|===========|

[toc] | [prev] | [next] | [standalone]

#26181

In message <iigcva$90q$1@news.eternal-september.org>, Mike Schilling wrote:

> Yes, it does (contain 16 bits.)

Yeah, I didn’t realize it was spelled out that way in the original language 
spec. What a short-sighted decision.

> It was defined to do so before there were supplemental characters ...

Why was there a need to define the size of a character at all? Even in the 
early days of the unification of Unicode and ISO-10646, there was already 
provision for UCS-4. Did they really think that could safely be ignored?

[toc] | [prev] | [next] | [standalone]

#25754 — Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals?

On 02/04/2011 06:41 PM, Arne Vajhøj wrote:
> indexOf works fine with compression, but substring and charAt becomes
> rather expensive.

I have seen it argued that random-access-ish stuff like substring and 
charAt aren't really all that random access, in that they tend to be 
"small" constants away from the beginning, end, or last indexOf computation.

See 
<http://weblogs.mozillazine.org/roc/archives/2008/01/string_theory.html>.

-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#25927 — Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals?

On 04-02-2011 19:13, Joshua Cranmer wrote:
> On 02/04/2011 06:41 PM, Arne Vajhøj wrote:
>> indexOf works fine with compression, but substring and charAt becomes
>> rather expensive.
>
> I have seen it argued that random-access-ish stuff like substring and
> charAt aren't really all that random access, in that they tend to be
> "small" constants away from the beginning, end, or last indexOf
> computation.

Could be.

But would it be practical to use it?

Arne

[toc] | [prev] | [next] | [standalone]

#25764

On 04/02/2011 16:49, Arne Vajhøj allegedly wrote:
> It is very clearly specified that a Java char is 16 bit.
>
> You can't have the codepoints above U+FFFF in a char.
>
> You can have them in a string but then they actually takes
> two chars in that string.
>
> It is rather messy.
>
> If you look at the Java docs for String class you will see:
>
> charAt & codePointAt
> length & codePointCount
>
> which is not a nice API.
>
> But since codepoints above U+FFFF was added after the String
> class was defined, then the options on how to handle it were
> pretty limited.

They've added supplementary character support to String, StringBuilder,
StringBuffer.

Pity they haven't touched upon java.lang.CharSequence. Probably out of
concerns about compatibility.

Anyone got an idea how supplementary character support could be
integrated with CharSequence, or more generally, with an interface
describing a sequence of code points? Creating a sub-interface, e.g.
UnicodeSequence with int codePointAt(int), etc. doesn't seem like it'd
do the trick, since a UnicodeSequence /is-not/ a CharSequence (char
charAt(int) doesn't make sense for a UnicodeSequence). Adding a new
interface would mean you don't get the interoperability with all the
parts of the API that uses CharSequences... The only option would seem
to refactor CharSequence and all the classes that use or implement it.
Which means no backwards-compatibility.

Bloody mess this is.

-- 
DF.

[toc] | [prev] | [next] | [standalone]

#25880

On 2/4/2011 9:37 AM, Daniele Futtorovic wrote:

> Pity they haven't touched upon java.lang.CharSequence. Probably out of
> concerns about compatibility.


You know that Character has static methods for pulling code points out 
of a CharSequence, right?

[toc] | [prev] | [next] | [standalone]

#25782 — Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals?

On 02/04/2011 04:30 PM, Tom Anderson wrote:
> A question to the house, then: has anyone ever invented a data structure
> for strings which allows space-efficient storage for strings in
> different scripts, but also allows time-efficient implementation of the
> common string operations?

I think the real answer is that maybe we need to rethink traditional 
string APIs. Particularly, we have the issues of diacratics, since "A 
[combining diacritic `]" is basically 1 character stored in 3,4, or 8 
bytes, depending on storage format.

I would be surprised if there weren't already some studies on the impact 
of using UTF-8 based strings in UTF-16/-32-ish contexts.

-- 
Beware of bugs in the above code; I have only proved it correct, not 
tried it. -- Donald E. Knuth

[toc] | [prev] | [next] | [standalone]

#25836

"Arne Vajhøj" <arne@vajhoej.dk> wrote in message 
news:4d4c2019$0$23753$14726298@news.sunsite.dk...
> On 04-02-2011 01:59, Lawrence D'Oliveiro wrote:
>> In message<iig6j2$dul$2@news.albasani.net>, Lew wrote:
>>> On 02/04/2011 12:59 AM, Lawrence D'Oliveiro wrote:
>>>> Why was it decreed in the language spec that characters beyond U+FFFF 
>>>> are
>>>> not allowed in character literals, when they are allowed everywhere 
>>>> else
>>>> (in string literals, in the program text, in character and string 
>>>> values
>>>> etc)?
>>>
>>> Because a 'char' type holds only 16 bits.
>>
>> No it doesn’t. Otherwise you wouldn’t be allowed supplementary characters 
>> in
>> character and string values. Which you are.
>
> It is very clearly specified that a Java char is 16 bit.
>
> You can't have the codepoints above U+FFFF in a char.
>
> You can have them in a string but then they actually takes
> two chars in that string.
>
> It is rather messy.
>
> If you look at the Java docs for String class you will see:
>
> charAt & codePointAt
> length & codePointCount
>
> which is not a nice API.
>
> But since codepoints above U+FFFF was added after the String
> class was defined, then the options on how to handle it were
> pretty limited.

The sticky issue is, I think, that chars were defined as 16-bit.  If that 
had been left undefined, they could have been extended to 24 bits, which 
would make things nice and regular again. 

[toc] | [prev] | [next] | [standalone]

#25867 — Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals?

On Fri, 4 Feb 2011 21:30:57 +0000, Tom Anderson <twic@urchin.earth.li>
wrote, quoted or indirectly quoted someone who said :

>I am, however, at a loss to suggest a practical alternative!

What might happen is strings are nominally 32-bit.

You could probably come up with a very rapid compression scheme,
similar to UTF-8 but with a bit more compression, that could be
applied to strings at garbage collection time if they have not been
referenced since the last GC sweep.

String are immutable.  This admits some other flavours of
"compression".

If the high three bytes of the string are 0, store the string
UNCOMPRESSED, as a string of bytes.  All the indexOf indexing
arithmetic works identically.  This behaviour is hidden inside the
JVM. The String class knows nothing about it. It is an implementation
detail of 32-bit strings.

If the high two bytes of the string are 0, store the string
uncompressed as a string of unsigned shorts.

if there are any one bits in the high 2 byte, store as a string of
unsigned ints.

Strings are what you gobble up your RAM with.  If we start supporting
32 bit chars, we have to do something to compensate for the doubling
of RAM use.
 

Short lived strings would still be 32-bit.  They would only be
converted to the other forms if they have been sitting around for a
while.  Interned strings would be immediately converted to canonical
form.

-- 
Roedy Green Canadian Mind Products
http://mindprod.com
To err is human, but to really foul things up requires a computer.
~ Farmer's Almanac
It is breathtaking how a misplaced comma in a computer program can
shred megabytes of data in seconds.

[toc] | [prev] | [next] | [standalone]

#26171 — Re: Efficient unicode string implementation was: Re: Why No Supplemental Characters In Character Literals?

On 04-02-2011 18:22, Roedy Green wrote:
> On Fri, 4 Feb 2011 21:30:57 +0000, Tom Anderson<twic@urchin.earth.li>
> wrote, quoted or indirectly quoted someone who said :
>
>> I am, however, at a loss to suggest a practical alternative!
>
> What might happen is strings are nominally 32-bit.
>
> You could probably come up with a very rapid compression scheme,
> similar to UTF-8 but with a bit more compression, that could be
> applied to strings at garbage collection time if they have not been
> referenced since the last GC sweep.
>
> String are immutable.  This admits some other flavours of
> "compression".
>
> If the high three bytes of the string are 0, store the string
> UNCOMPRESSED, as a string of bytes.  All the indexOf indexing
> arithmetic works identically.  This behaviour is hidden inside the
> JVM. The String class knows nothing about it. It is an implementation
> detail of 32-bit strings.
>
> If the high two bytes of the string are 0, store the string
> uncompressed as a string of unsigned shorts.
>
> if there are any one bits in the high 2 byte, store as a string of
> unsigned ints.
>
> Strings are what you gobble up your RAM with.  If we start supporting
> 32 bit chars, we have to do something to compensate for the doubling
> of RAM use.
>
>
> Short lived strings would still be 32-bit.  They would only be
> converted to the other forms if they have been sitting around for a
> while.  Interned strings would be immediately converted to canonical
> form.

indexOf works fine with compression, but substring and charAt becomes
rather expensive.

Arne

[toc] | [prev] | [next] | [standalone]

#25905

On 04-02-2011 18:08, Roedy Green wrote:
> On Fri, 04 Feb 2011 13:44:08 -0500, Joshua Cranmer
> <Pidgeot18@verizon.invalid>  wrote, quoted or indirectly quoted someone
> who said :
>> Well, the real problem is that Unicode swore that 16 bits were enough
>> for everybody,
>
> be fair. I bought a book showing thousands of Unicode glyphs including
> pages and pages of HAN ideographs.  There were plenty of holes for
> future growth. At the time I thought it was overkill.  When I started
> my career, character sets had 64 glyphs, including control chars.
> Later it was considered "extravagant" to use lower case since it took
> so much longer to print. In the very early days, each installation
> designed its own local character set.  I recall sitting in one such
> meeting, and Vern Detwiler (later of MacDonald Detwiler) explaining
> the virtues of new code called ASCII.

Impressive that he wanted to discuss that with a 12 year old.

Arne

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.java.programmer

csiph-web