Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.javascript > #19110 > unrolled thread

Re: Unicode support

Back to article view | Back to comp.lang.javascript

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Unicode support Christoph Becker <cmbecker69@gmx.de> - 2013-04-01 15:24 +0200
    Re: Unicode support Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-01 18:39 +0200
      Re: Unicode support Mike Duffy <Use.web.form@website_in_sig.com> - 2013-04-01 20:36 +0000
      Re: Unicode support Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-04-30 23:03 +0200
        Re: Unicode support Scott Sauyet <scott.sauyet@gmail.com> - 2013-05-02 05:27 -0700
          Re: Unicode support Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2013-05-03 17:59 +0200

#19110 — Re: Unicode support

Thomas 'PointedEars' Lahn wrote:
>
> Christoph Becker wrote:
> 
>> Thomas 'PointedEars' Lahn wrote:
>>> Christoph Becker wrote:
>>>> I suppose most *higher level languages* define the length of a string as
>>>> the number of characters the string holds.  Cf. ECMAScript's length
>>>> property and TCL's [string length].  Even PHP's mb_strlen() returns the
>>>> number of characters.
>>>
>>> AISB, unfortunately the “length” property of ECMAScript String instances
>>> is not a good example in that regard as it does _not_ mean the number of
>>> characters but the number of 16-bit units (UTF-16 code units, usually).
>>> That is only less obvious than with UTF-8 because all Unicode characters
>>> in the BMP can be encoded using only one such unit.  (I am working on a
> 
> “BMP” means “Basic Multilingual Plane” (U+0000 to U+FFFF).

I'm aware of that. :)

>>> workaround; you could call it “an mb_strlen() for ECMAScript
>>> implementations”.)
>>
>> ECMA-262, Edition 5.1, Section 15.5.5.1, states:
>>
>> | *length*
>> |
>> | The number of characters in the String value represented by this
>> | String object.
>>
>> IMO the spec is quite clear here,
> 
> It is.
> 
>> and so the implementations might be in error.
> 
> For a change, it is not.  You should look up the definition of “character” 
> in the Specification.  

I was not aware of that:

| Throughout the rest of this document, the phrase “code unit” and the
| word “character” will be used to refer to a 16-bit unsigned value
| used to represent a single 16-bit unit of text. The phrase “Unicode
| character” will be used to refer to the abstract linguistic or
| typographical unit represented by a single Unicode scalar value
| (which may be longer than 16 bits and thus may be represented by more
| than one code unit).

Thanks for pointing it out.  It's good to know that one has to take care
when e.g. using String.prototype.charAt() or
String.prototype.substring() on strings that may contain Unicode
characters outside the BMP.

> See also <news:2202673.rtQqbKup0V@PointedEars.de> pp. 
> (yes, in comp.lang.python).
> 
>> Anyway, I appreciate your working on a workaround; may come in handy. :)
> 
> At the latest after first contact ;-)
> 
> You will find it in 
> <http://PointEdears.de/websvn/listing.php?repname=JSX&path=%2Ftrunk%2Fstring%2F> 
> when it is ready for testing.

My feedreader will keep an eye on it. ;)

> F'up2 comp.lang.javascript

-- 
Christoph M. Becker

[toc] | [next] | [standalone]

#19115

Christoph Becker wrote:

> Thomas 'PointedEars' Lahn wrote:
>> Christoph Becker wrote:
>>> Anyway, I appreciate your working on a workaround; may come in handy. :)
>> 
>> At the latest after first contact ;-)
>> 
>> You will find it in
>> 
<http://PointEdears.de/websvn/listing.php?repname=JSX&path=%2Ftrunk%2Fstring%2F>
>> when it is ready for testing.
> 
> My feedreader will keep an eye on it. ;)

Thank you.

Due to recent events I would also like to announce that in the version of 
unicode.js that follows this one there is going to be support for the Vulcan 
Unicode characters (both TGV and MGV) and the Sloof-lirpa script in which 
these languages are written.  Stay tuned.

-- 
\\//,
(\PointedEars/)

Twitter: @PointedEars2
Please do not Cc: me. / Bitte keine Kopien per E-Mail.

[toc] | [prev] | [next] | [standalone]

#19131

Thomas 'PointedEars' Lahn <PointedEars@web.de> wrote in
news:1888759.kj0EFmyP97@PointedEars.de: 


> 
> Due to recent events I would also like to announce that in the version
> of unicode.js that follows this one there is going to be support for
> the Vulcan Unicode characters (both TGV and MGV) and the Sloof-lirpa
> script in which these languages are written.  Stay tuned.

There is something fishy about your post.

I must admit that at face value it looks like a good combination:

http://www.urbandictionary.com/define.php?term=sloof

http://en.memory-alpha.org/wiki/Lirpa


-- 
http://pages.videotron.ca/duffym/index.htm

[toc] | [prev] | [next] | [standalone]

#19545

Thomas 'PointedEars' Lahn wrote:

> Christoph Becker wrote:
>> Thomas 'PointedEars' Lahn wrote:
>>> Christoph Becker wrote:
>>>> Anyway, I appreciate your working on a workaround; may come in handy.
>>>> :)
>>> 
>>> At the latest after first contact ;-)
>>> 
>>> You will find it in
>>> 
> 
<http://PointEdears.de/websvn/listing.php?repname=JSX&path=%2Ftrunk%2Fstring%2F>
>>> when it is ready for testing.
>> 
>> My feedreader will keep an eye on it. ;)
> 
> Thank you.
> 
> Due to recent events I would also like to announce that in the version of
> unicode.js that follows this one there is going to be support for the
> Vulcan Unicode characters (both TGV and MGV) and the Sloof-lirpa script in
> which these languages are written.  Stay tuned.

Now that April Fools' Day is long over, I would like to clarify so as not to 
be misunderstood.

JSX:string/unicode.js is real: I have been working on a library that 
supports Unicode characters beyond the Basic Multilingual Plane; that is 
UTF-16 code sequences for code points beyond U+FFFF, up to U+10FFFF.

A future unicode.js that would support the aforementioned Vulcan script 
(writing) is _not_ going to happen (although I might write another script 
[programming] that supports the Vulcan languages later – if Google can do 
Klingon [1], I can do Vulcan ;-)).

Those were the clues (ROT-13 encoded for those who either like puzzles or 
are not interested in the details of the joke):

- Gurer ner ab Ihypna Havpbqr punenpgref.  (Gurer ner abg rira Xyvatba
  rgp. Havpbqr punenpgref, nygubhtu gurer ner sbagf gung znc pbqr cbvagf gb
  gubfr punenpgref.  V pna cebivqr HEVf gb gubfr jub ner vagrerfgrq.)

- Gurer vf ab “Fybbs-yvecn” fpevcg; “Fybbs-yvecn”, jura ernq onpxjneqf, vf
  “ncevy-sbbyF”.

    [V qvq abg rkcrpg, ubjrire, “fybbs” gb or n erny Ratyvfu
     jbeq.  Nyfb, frireny bgure crbcyr hfrq “fybbs yvecn” va
     gurve Ncevy Sbbyf' wbxr; cnegvphyneyl pheerag VFF Pbzznaqre
     Puevf Unqsvryq (PFN-NFP): 
     <uggcf://gjvggre.pbz/Pzqe_Unqsvryq/fgnghf/318806874935939074>
     Ol pbvapvqrapr, /yvecn/ vf gur anzr bs n gjb-oynqrq Ihypna jrncba
     gung jnf hfrq va unaq-gb-unaq pbzong orsber gur Njnxravat, naq vf
     fgvyy hfrq va gur evghny bs /xbba-hg-xny-vs-srr/ – zneevntr-be-
     punyyratr[-gb-gur-qrngu).]

- Gur Ihypna ynathntrf ner jevggra jvgu n (abg shyyl xabja) fpevcg jurer
  tylcuf fgnaq sbe flyynoyrf vafgrnq bs fvatyr punenpgref (zhpu yvxr Nfvna
  ynathntrf).  Genafxevcgvbaf hfr gur Yngva nycunorg.  Frr nyfb Gur
  Ihypna Ynathntr Vafgvghqr (Erpynzngvba Cebwrpg):
  <uggc://jjj.fgbtrrx.pbz/jvxv/Pngrtbel:Ihypna_Ynathntr_Vafgvghgr>  


The unit test of JSX:string/unicode.js can now be viewed at

  <http://PointedEars.de/scripts/test/unicode>

More features and testcases will follow.  Hopefully the library will be 
useful.

I am also considering to overwrite built-in methods of the String 
constructor and the String prototype object, and augment them with new ones 
(I already had several of them, some superfluous as it turned out, but I 
removed them after extensive refactoring of jsx.string.unicode.WideString 
broke them).  Overwriting and augmenting would be optional each, but done by 
default, through the new jsx.options mechanism (see object.js [2] for 
details).

What do you think?  Would you find, say

  /* returns 1 */
  "𝄞".getLength()

in addition to

  /* returns 1 */
  jsx.string.unicode.WideString("𝄞").getLength()

and as opposed to

  /* yields 2 */
  "𝄞".length

useful?  And would you find notations like

  /* returns a WideString instance containing "𝄞" */
  jsx.string.unicode.WideString("\\u{1D11E}")

and

  /* returns a RegExp that matches "𝄞" */
  jsx.regexp.RegExp("\\u{1D11E}")

useful until ECMAScript Edition 6 (”Harmony”) was formally published and 
interoperably implemented?  (See also [3].)

_________
[1] <http://www.google.com/?hl=xx-klingon>
[2] 
<http://PointedEars.de/websvn/filedetails.php?repname=JSX&path=%2Ftrunk%2Fobject.js>
[3] 
<http://wiki.ecmascript.org/doku.php?id=spec:chapter_6_lexical_structure>
-- 
PointedEars

Twitter: @PointedEars2
Please do not Cc: me. / Bitte keine Kopien per E-Mail.

[toc] | [prev] | [next] | [standalone]

#19578

Thomas 'PointedEars' Lahn wrote:
> JSX:string/unicode.js is real: I have been working on a library that
> supports Unicode characters beyond the Basic Multilingual Plane; that is
> UTF-16 code sequences for code points beyond U+FFFF, up to U+10FFFF.
> [ ... ]
>
> The unit test of JSX:string/unicode.js can now be viewed at
>
>   <http://PointedEars.de/scripts/test/unicode>
>
> More features and testcases will follow.  Hopefully the library will be
> useful.
>
> I am also considering to overwrite built-in methods of the String
> constructor and the String prototype object, and augment them with new ones
> (I already had several of them, some superfluous as it turned out, but I
> removed them after extensive refactoring of jsx.string.unicode.WideString
> broke them).  Overwriting and augmenting would be optional each, but done by
> default, through the new jsx.options mechanism (see object.js [2] for
> details).
>
> What do you think? [ ... ]

While I have no current uses for this, I applaud your efforts, and
would be very happy to know that it was available if I did need it.
I've looked over your regex extension; I was impressed and wished it
had been available for some older projects I had done.  This project
could clearly fill some similar gaps.

But, as I said, I have no uses for it in any current projects.

Good luck,

  -- Scott

[toc] | [prev] | [next] | [standalone]

#19608

Scott Sauyet wrote:

> Thomas 'PointedEars' Lahn wrote:
>> JSX:string/unicode.js is real: I have been working on a library that
>> supports Unicode characters beyond the Basic Multilingual Plane; that is
>> UTF-16 code sequences for code points beyond U+FFFF, up to U+10FFFF.
>> [ ... ]
>>
>> The unit test of JSX:string/unicode.js can now be viewed at
>>
>> <http://PointedEars.de/scripts/test/unicode>
>>
>> More features and testcases will follow.  Hopefully the library will be
>> useful.
>>
>> I am also considering to overwrite built-in methods of the String
>> constructor and the String prototype object, and augment them with new
>> ones […].  Overwriting and augmenting would be optional each, but done by
>> default, through the new jsx.options mechanism (see object.js [2] for
>> details).
>>
>> What do you think? [ ... ]
> 
> While I have no current uses for this, I applaud your efforts, and
> would be very happy to know that it was available if I did need it.
> I've looked over your regex extension; I was impressed and wished it
> had been available for some older projects I had done.  This project
> could clearly fill some similar gaps.
> 
> But, as I said, I have no uses for it in any current projects.
> 
> Good luck,

Thank you very much.

Now unicode.js by default (AISB, it is configurable) also replaces 
String.fromCharCode() to support code points beyond U+FFFF (Number arguments 
from 0 to 0x10FFFF), and adds String.prototype.getLength() so that 
"…".getLength() returns the number of Unicode characters in a string, as 
opposed to the built-in "…".length which yields the number of 16-bit code 
units.  More is going to follow.

Thanks to Cezary Tomczyk's test [1] it is also working in IE 10 on Windows 8 
(it was a refactoring bug in object.js).  However, JScript in IE/MSHTML 10 
apparently lacks the ability to define getters and setters for properties of 
native objects as specified in ECMAScript Edition 5.x, so tests using those 
features will always fail.

[1] <https://twitter.com/cezarytomczyk/status/329850349760442368>

-- 
PointedEars

Twitter: @PointedEars2
Please do not Cc: me. / Bitte keine Kopien per E-Mail.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.javascript

csiph-web