Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #81089 > unrolled thread

Re: python 2.7 and unicode (one more time)

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: python 2.7 and unicode (one more time) Michael Torrie <torriem@gmail.com> - 2014-11-20 09:51 -0700
    Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-20 20:56 +0200
      Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 10:49 +1100
        Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-21 08:36 +0200
          Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 18:02 +1100
            Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-21 09:14 +0200
              Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 18:29 +1100
                Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-21 10:16 +0200
                  Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 19:24 +1100
          Re: python 2.7 and unicode (one more time) Rustom Mody <rustompmody@gmail.com> - 2014-11-21 08:03 -0800
            Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-21 18:36 +0200
              Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-22 03:52 +1100
              Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-22 15:38 +1100
                Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 08:40 +0200
                  Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 00:50 +1100
                    Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 01:17 +1100
                    Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 16:18 +0200
                      Re: python 2.7 and unicode (one more time) Roy Smith <roy@panix.com> - 2014-11-22 09:44 -0500
                        Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 16:56 +0200
                      Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-22 07:44 -0800
                        Re: python 2.7 and unicode (one more time) Rustom Mody <rustompmody@gmail.com> - 2014-11-22 07:57 -0800
                        Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-22 19:49 +0200
                          Re: python 2.7 and unicode (one more time) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-11-22 18:17 +0000
                            Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-22 11:32 -0800
                          Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-22 11:17 -0800
                          Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 07:17 +1100
                          Re: python 2.7 and unicode (one more time) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-11-22 22:04 +0000
                          Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 09:31 +1100
                          Re: python 2.7 and unicode (one more time) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-11-22 23:38 +0000
                            Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-23 04:52 -0800
                          Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-22 20:52 -0500
                          Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 13:11 +1100
                          Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-22 22:10 -0500
                          Re: python 2.7 and unicode (one more time) Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-23 11:33 -0500
                          Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-24 03:39 +1100
                          Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-23 13:13 -0500
                          Re: python 2.7 and unicode (one more time) Dave Angel <d@davea.name> - 2014-11-23 15:31 -0500
                          Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-24 09:22 +1100
                          Re: python 2.7 and unicode (one more time) Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-11-23 20:24 -0500
                          Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-24 00:05 -0500
                  Re: python 2.7 and unicode (one more time) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-11-24 11:51 +1300
                    Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-24 09:59 +1100
                    Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-24 08:57 +0200
                      Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-24 18:18 +1100
                        Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-24 09:40 +0200
                          Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-25 09:56 +1100
                            Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-25 10:08 +1100
                            Re: python 2.7 and unicode (one more time) Marko Rauhamaa <marko@pacujo.net> - 2014-11-25 08:23 +0200
                              Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-25 22:56 +1100
                                Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-26 00:07 +1100
                                  Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-25 06:26 -0800
                Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-22 20:35 -0500
                  Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-23 17:17 +1100
                    Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-23 17:37 +1100
                      Re: python 2.7 and unicode (one more time) wxjmfauth@gmail.com - 2014-11-23 07:40 -0800
      Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-21 11:32 +1100
        Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 12:10 +1100
          Re: python 2.7 and unicode (one more time) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-11-22 02:23 +1100
            Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-22 02:40 +1100
            Re: python 2.7 and unicode (one more time) Tim Chase <python.list@tim.thechases.com> - 2014-11-21 10:00 -0600
        Re: python 2.7 and unicode (one more time) random832@fastmail.us - 2014-11-20 20:31 -0500
        Re: python 2.7 and unicode (one more time) Chris Angelico <rosuav@gmail.com> - 2014-11-21 12:39 +1100

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

#81405

Marko Rauhamaa wrote:
> Unicode strings is not wrong but the technical emphasis on Unicode is as
> strange as a "tire car" or "rectangular door" when "car" and "door" are
> what you usually mean.

The reason Unicode gets emphasised so much is that
until relatively recently, it *wasn't* what "string"
usually meant in Python.

When Python 3 has been around for as long as Python
2 was, things may change.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#81406

On Mon, Nov 24, 2014 at 9:51 AM, Gregory Ewing
<greg.ewing@canterbury.ac.nz> wrote:
> Marko Rauhamaa wrote:
>>
>> Unicode strings is not wrong but the technical emphasis on Unicode is as
>> strange as a "tire car" or "rectangular door" when "car" and "door" are
>> what you usually mean.
>
>
> The reason Unicode gets emphasised so much is that
> until relatively recently, it *wasn't* what "string"
> usually meant in Python.
>
> When Python 3 has been around for as long as Python
> 2 was, things may change.

I doubt it; the bytes() type is sufficiently stringy to require the
distinction to still be made. PEP 461 makes it clear that byte strings
are not blobs of opaque data, but are very definitely ASCII-compatible
objects, for the benefit of boundary code.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81437

Gregory Ewing <greg.ewing@canterbury.ac.nz>:
> Marko Rauhamaa wrote:
>> Unicode strings is not wrong but the technical emphasis on Unicode is as
>> strange as a "tire car" or "rectangular door" when "car" and "door" are
>> what you usually mean.
>
> The reason Unicode gets emphasised so much is that until relatively
> recently, it *wasn't* what "string" usually meant in Python.
>
> When Python 3 has been around for as long as Python 2 was, things may
> change.

Yes, people call strings "Unicdoe strings" because Python2 *did have*
unicode strings separate from regular strings:

    Python2            Python3
    --------------------------------------
    string             bytes (byte string)
    unicode string     string


In Python2 days, Unicode was a fancy, exotic datatype for the
connoisseurs. The rest used strings. Python3 supposedly elevates Unicode
to boring normalcy. Now it's bytes that have fallen into (unmerited)
disfavor.

But old habits die hard; you call cars "automobile cars" instead of
"cars" since, after all, "cars" were always pulled by horses...


Marko

PS Maybe interestingly, Guile went through an analogous transition. As
of Guile 2.0,

  a character is anything in the Unicode Character Database.
  [...]
  Strings are fixed-length sequences of characters.
  [...]
  A bytevector is a raw bit string.

  <URL: https://www.gnu.org/software/guile/manual/html_node/index.html>

However, Guile 1.8 still had:

  The Guile implementation of character sets currently deals only with
  8-bit characters.

  <URL: https://www.gnu.org/software/guile/docs/docs-1.8/guile-ref/inde
  x.html>

and there were no bytevectors.

[toc] | [prev] | [next] | [standalone]

#81438

On Mon, Nov 24, 2014 at 5:57 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Yes, people call strings "Unicdoe strings" because Python2 *did have*
> unicode strings separate from regular strings:
>
>     Python2            Python3
>     --------------------------------------
>     string             bytes (byte string)
>     unicode string     string
>
>
> In Python2 days, Unicode was a fancy, exotic datatype for the
> connoisseurs. The rest used strings. Python3 supposedly elevates Unicode
> to boring normalcy. Now it's bytes that have fallen into (unmerited)
> disfavor.

Py3's byte strings are still strings, though. People don't use
bytearray for everything.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81439

Chris Angelico <rosuav@gmail.com>:

> Py3's byte strings are still strings, though.

Hm. I don't think so. In a plain English sense, maybe, but that kind of
usage can lead to confusion.

For example,

   A subscription selects an item of a sequence (string, tuple or list)
   or mapping (dictionary) object:

   subscription ::=  primary "[" expression_list "]"

   [...]

   A string’s items are characters. A character is not a separate data
   type but a string of exactly one character.

   <URL: https://docs.python.org/3/reference/expressions.html#subscripti
   ons>


The text is probably a bit buggy since it skates over bytes and byte
arrays listed as sequences (by <URL:
https://docs.python.org/3/reference/datamodel.html>). However, your
Python3 implementation would fail if it interpreted bytes objects to be
strings in the above paragraph:

   >>> "abc"[1]
   'b'
   >>> b'abc'[1]
   98

The subscription of a *string* evaluates to a *string*. The subscription
of a *bytes* object evaluates to a *number*.


Marko

[toc] | [prev] | [next] | [standalone]

#81473

Marko Rauhamaa wrote:

>> Py3's byte strings are still strings, though.
> 
> Hm. I don't think so. In a plain English sense, maybe, but that kind of
> usage can lead to confusion.

Only if you are determined to confuse yourself.

People are quite capable of interpreting correctly sentences like:

"My friend Susan and I were talking about Jenny, and she said that she had
had a horrible fight with her boyfriend and was breaking up with him."

and despite the ambiguity correctly interpret who "she" and "her" refers to
each time. Compared to that, correctly understanding the mild complexity
of "string" is trivial.

In Python usage, "string" always refers to the `str` type, unless prefixed
with "byte", in which case it refers to the immutable byte-string type
(`str` in Python 2, `bytes` in Python 3.)

"Unicode string" always refers to the immutable Unicode string type
(`unicode` in Python 2, `str` in Python 3).

"Text string" is more ambiguous. Some people consider the prefix to be
redundant, e.g. "text string" always refers to `str`, while others consider
it to be in opposition to "byte string", i.e. to be a synonym for "Unicode
string".

In all cases apart from an explicit "byte string", the word "string" is
always used for the native array-of-characters type delimited by plain
quotation marks, as used for error messages, user prompts, etc., regardless
whether the implementation is an array of 8-bit bytes (as used by Python
2), or the full Unicode character set (as used by Python 3). So in
practice, provided you know which version of Python is being discussed,
there is never any genuine ambiguity when using the word "string" and no
excuse for confusion.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#81474

On Tue, Nov 25, 2014 at 9:56 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> In all cases apart from an explicit "byte string", the word "string" is
> always used for the native array-of-characters type delimited by plain
> quotation marks, as used for error messages, user prompts, etc., regardless
> whether the implementation is an array of 8-bit bytes (as used by Python
> 2), or the full Unicode character set (as used by Python 3). So in
> practice, provided you know which version of Python is being discussed,
> there is never any genuine ambiguity when using the word "string" and no
> excuse for confusion.

And frequently, even if you're talking about Py2/Py3 cross code,
there's still no ambiguity about the word "string": it means a
default-for-the-language string. The locale.setlocale() function
expects a string as its second parameter, for instance. (And
unfortunately, flatly refuses the other sort, whichever way around
that is.)

ChrisA

[toc] | [prev] | [next] | [standalone]

#81484

Steven D'Aprano <steve+comp.lang.python@pearwood.info>:

> Marko Rauhamaa wrote:
>
>>> Py3's byte strings are still strings, though.
>> 
>> Hm. I don't think so. In a plain English sense, maybe, but that kind of
>> usage can lead to confusion.
>
> Only if you are determined to confuse yourself.
>
> {...]
>
> In Python usage, "string" always refers to the `str` type, unless
> prefixed with "byte", in which case it refers to the immutable
> byte-string type (`str` in Python 2, `bytes` in Python 3.)

You are saying what I'm saying.

Byte strings are *not* strings.

Prairie dogs are not dogs. No need to call dogs "domesticated dogs" to
tell them apart from "prairie dogs".


Marko

[toc] | [prev] | [next] | [standalone]

#81501

Marko Rauhamaa wrote:

> Steven D'Aprano <steve+comp.lang.python@pearwood.info>:
> 
>> Marko Rauhamaa wrote:
>>
>>>> Py3's byte strings are still strings, though.
>>> 
>>> Hm. I don't think so. In a plain English sense, maybe, but that kind of
>>> usage can lead to confusion.
>>
>> Only if you are determined to confuse yourself.
>>
>> {...]
>>
>> In Python usage, "string" always refers to the `str` type, unless
>> prefixed with "byte", in which case it refers to the immutable
>> byte-string type (`str` in Python 2, `bytes` in Python 3.)
> 
> You are saying what I'm saying.
> 
> Byte strings are *not* strings.

Of course they are. They are strings of bytes, just as the name suggests.


> Prairie dogs are not dogs. No need to call dogs "domesticated dogs" to
> tell them apart from "prairie dogs".

But wild dogs *are* dogs, and there is a need to distinguish between wild
dogs and domesticated dogs. 

Just as there is a need to distinguish between byte strings, ASCII strings,
Latin-1 strings, Big5 strings, Unicode strings, Tron strings and cheese
strings.

I think this conversation is going nowhere, so it's probably best to end it.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#81508

On Tue, Nov 25, 2014 at 10:56 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> I think this conversation is going nowhere, so it's probably best to end it.

\0

ChrisA

[toc] | [prev] | [next] | [standalone]

#81512

It was an interesting reading.

[toc] | [prev] | [next] | [standalone]

#81308

On Fri, Nov 21, 2014, at 23:38, Steven D'Aprano wrote:
> I really don't understand what bothers you about this. In Python, we have
> Unicode strings and byte strings. In computing in general, strings can
> consist of Unicode characters, ASCII characters, Tron characters, EBCDID
> characters, ISO-8859-7 characters, and literally dozens of others. It
> boogles my mind that you are so opposed to being explicit about what sort
> of string we are dealing with.

I think he means that it should be implementation-defined with an API
that does not allow programs to make assumptions about the encoding,
like C. To allow for implementations that use a different character set.

[toc] | [prev] | [next] | [standalone]

#81333

random832@fastmail.us wrote:

> On Fri, Nov 21, 2014, at 23:38, Steven D'Aprano wrote:
>> I really don't understand what bothers you about this. In Python, we have
>> Unicode strings and byte strings. In computing in general, strings can
>> consist of Unicode characters, ASCII characters, Tron characters, EBCDID
>> characters, ISO-8859-7 characters, and literally dozens of others. It
>> boogles my mind that you are so opposed to being explicit about what sort
>> of string we are dealing with.
> 
> I think he means that it should be implementation-defined with an API
> that does not allow programs to make assumptions about the encoding,
> like C. To allow for implementations that use a different character set.

Python is not C, and doesn't make every second thing undefined behaviour.

If Python treated the character set as an implementation detail, the
programmer would have no way of knowing whether

s = u"ö"

is legal or not, since you cannot know whether or not ö is a supported
character in the running Python. It might work on your system, and fail for
other people. That is worse than the old distinction between "narrow"
and "wide" builds. It would be a lazy and stupid design, and especially
stupid since there really in no good alternative to Unicode today. ASCII is
not even sufficient for American English, the whole Windows code page idea
is a horrible mess, none of the legacy encodings are suitable for more than
a tiny fraction of the world.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#81336

On Sun, Nov 23, 2014 at 5:17 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> If Python treated the character set as an implementation detail, the
> programmer would have no way of knowing whether
>
> s = u"ö"
>
> is legal or not, since you cannot know whether or not ö is a supported
> character in the running Python. It might work on your system, and fail for
> other people. That is worse than the old distinction between "narrow"
> and "wide" builds. It would be a lazy and stupid design, and especially
> stupid since there really in no good alternative to Unicode today. ASCII is
> not even sufficient for American English, the whole Windows code page idea
> is a horrible mess, none of the legacy encodings are suitable for more than
> a tiny fraction of the world.

(Code pages aren't a Windows concept, of course, though I guess that's
the main place where they're found on PCs today.)

The only trouble with enforcing Unicode is Japanese encodings and the
whole Han unification debate. Ultimately, you have to pick a side: are
you siding with those who say there are fewer characters with multiple
forms, or with those who say there are more distinct characters? If
the former, go with Unicode. If the latter, be prepared to do heaps of
work yourself, and probably be stuck with supporting only Japanese,
because encodings like Shift-JIS aren't going to be able to represent
Scandinavian text.

Me, I'm siding with Unicode. The politicking of Han unification
doesn't interest me, so I'm happy to accept a position that says that
they're all the same character, just as the Roman letter A can be used
in English, Italian, German, Swedish, etc, etc, etc (maybe with some
combining characters for diacriticals). That gives me access to all
the world's languages with a single character set and some trustworthy
encodings. I think it's a fine trade-off: philosophy I don't care
about versus correctness in my code.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81375

Le dimanche 23 novembre 2014 07:37:45 UTC+1, Chris Angelico a écrit :
> 
> Me, I'm siding with Unicode. [...]

So, am I. Unfortunately Python just becomes a anti-unicode
product.
A very interesting aspect of this language (which btw.
I'm no more recommending).

jmf

[toc] | [prev] | [next] | [standalone]

#81154

Marko Rauhamaa wrote:

> Michael Torrie <torriem@gmail.com>:
> 
>> Unicode can only be encoded to bytes.
>> Bytes can only be decoded to unicode.
> 
> I don't really like it how Unicode is equated with text, or even
> character strings.

That surely depends on the context. To be technically correct, Unicode is a
character set together with a set of rules for dealing with them (e.g.
rules for uppercasing characters, sorting rules, etc.). When referring to
the standard, "Unicode" is a noun; when referring to text, it is actually
an adjective being used as a noun. That is, "Unicode text" has become
abbreviated as just "Unicode" in much the same way as "human beings" has
become abbreviated as just "humans".

In that sense, "text is Unicode" just means "in the context in which we are
talking, when I say 'text' I mean 'Unicode text' as opposed to (for
example) 'ASCII text' or 'KOI-8 text'." It certainly doesn't mean that
*all* text in other contexts are Unicode, since that is obviously untrue.

(E.g. there are millions of existing files across the world containing text
which use legacy encodings that are not compatible with Unicode.)


> There's barely any difference between the truth value of these
> statements:
> 
>    Python strings are ASCII.
> 
>    Python strings are Latin-1.
> 
>    Python strings are Unicode.
> 
> Each of those statements is true as long as you stay within the
> respective character sets, and cease to be true when your text contains
> characters outside the character sets.

When we say "Python strings are FOO", we are making a statement about
arbitrary Python strings, not a particular set of concrete examples of
strings. If Python strings are FOO, that means that for all possible Python
strings s, "s is FOO" is a true statement.

We cannot say that Python strings are uppercase, because we can easily find
counter-examples such as 'xyz'. Likewise we cannot say Python strings are
ASCII, or Latin-1, because we can easily find counter-examples such as 'Ř'

On the other hand, Python strings *are* Unicode, because by design Python
strings are limited to Unicode. Every Python string is a Unicode string.


> Now, it is true that Python currently limits itself to the 1,114,112
> Unicode code points. And it likely won't adopt more characters unless
> Unicode does it first. However, text is something more lofty and
> abstract than a sequence of Unicode code points.

You are certainly correct that in it's full generality, "text" is much more
than just a string of code points. Unicode strings is a primitive data
type. A powerful and sophisticated text processing application may even
find Python strings too primitive, possibly needing something like ropes of
graphemes rather than strings of code points.

We Western and Northern European speakers -- and I don't know whether Finns
are counted as Northern Europeans or Eastern Europeans -- are lucky in that
our natural languages are well-covered by Unicode. All our graphemes are
also code points, even the "funny ones with accents". As an English
speaker. I have to remind myself that not every grapheme is a single code
point, but Devanagari or Navajo writers will never make that mistake.


> We shouldn't call strings Unicode any more than we call numbers IEEE or
> times ISO.

We certainly shouldn't call numbers IEEE, but we might very well call them
IEEE-754. Actually, since IEEE-754 covers multiple formats, we have to be
more specific:

Python floats are IEEE-754 double-precision binary floats.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#81156

On Fri, Nov 21, 2014 at 11:32 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> (E.g. there are millions of existing files across the world containing text
> which use legacy encodings that are not compatible with Unicode.)

Not compatible with Unicode? There aren't many character sets out
there that include characters not in Unicode - that was the whole
point. Of course, there are plenty of files in unspecified eight-bit
encodings, so you may have a problem with reliable decoding - but if
you know what the encoding is, you ought to be able to represent each
character in Unicode.

Not compatible with any of the UTFs, that's different. Plenty of that
in the world.

> You are certainly correct that in it's full generality, "text" is much more
> than just a string of code points. Unicode strings is a primitive data
> type. A powerful and sophisticated text processing application may even
> find Python strings too primitive, possibly needing something like ropes of
> graphemes rather than strings of code points.

That's probably more an efficiency point, though. It should be
possible to do a perfect two-way translation between your grapheme
rope and a Python string; otherwise, you'll have great difficulty
saving your file to the disk (which will normally involve representing
the text in Unicode, then encoding that to bytes).

To be sure, a Python string is a poor representational form for a text
editor. But that's largely because it's immutable, so every little
edit would involve massive copying. Depending on what you're doing, it
might be worth using a chunked UTF-8 byte stream (allowing for
insertion at any chunk boundary), or an array of lines, or something
grapheme-based... but all of those questions are performance, not
correctness, issues.

> We Western and Northern European speakers -- and I don't know whether Finns
> are counted as Northern Europeans or Eastern Europeans -- are lucky in that
> our natural languages are well-covered by Unicode. All our graphemes are
> also code points, even the "funny ones with accents". As an English
> speaker. I have to remind myself that not every grapheme is a single code
> point, but Devanagari or Navajo writers will never make that mistake.

I've been working with different languages a bit, lately. Broadly
speaking, you have:

1) Languages which use the Roman alphabet, plus a handful of other
characters (eg Finnish, German). These can be represented largely in
ASCII, and used to be handled fairly easily with a single codepage -
an eight-bit ASCII-compatible encoding.

2) Languages which use a different alphabet (eg Cyrillic - Russian,
Bulgarian). You could possibly cram them into an eight-bit encoding
without tipping ASCII out, but I'm not sure. In Unicode, these
languages are all easily supported by the BMP, as they don't use a
huge number of characters each.

3) Languages which use a non-alphabetic system (eg Korean). I think
they're all still covered by the BMP, but there's no way you can fit
them into eight-bit encodings - one single language will use more than
256 symbols.

4) Ancient, esoteric, or symbolic writing systems. Not fundamentally
different from the above categories except that they're less used, and
the BMP has finite space. These will definitely need the SMP.

But all of them are covered by Unicode. (Sadly, they are NOT all
covered by all fonts, so I've been finding that certain pieces of text
come out as strings of little boxes. But I can at least manipulate the
text, even if I can't read it back.) I can, for example, zip lines of
text like this:

English:
Let it go, let it go!
I am one with the wind and sky
Let it go, let it go!
You'll never see me cry!

Icelandic:
Þetta er nóg, þetta er nóg
Uppi í himni eins og vindablær
Þetta er nóg, komið nóg
Og tár mín enginn sér fær

Russian:
Отпусти и забудь,
Этот мир из твоих грёз.
Отпусти и забудь,
И не будет больше слёз.


Output:
Let it go, let it go!
Þetta er nóg, þetta er nóg
Отпусти и забудь,

I am one with the wind and sky
Uppi í himni eins og vindablær
Этот мир из твоих грёз.

Let it go, let it go!
Þetta er nóg, komið nóg
Отпусти и забудь,

You'll never see me cry!
Og tár mín enginn sér fær
И не будет больше слёз.


In fact, it's trivially easy to write something like this, because all
this text is Unicode. ALL of these languages (and plenty more) are
"well-covered by Unicode". There's still the ongoing debate of Han
unification, plus the progressive work of adding characters for
ancient scripts and such, but AFAIK, all writing systems currently in
use are covered.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81211

Chris Angelico wrote:

> On Fri, Nov 21, 2014 at 11:32 AM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> (E.g. there are millions of existing files across the world containing
>> text which use legacy encodings that are not compatible with Unicode.)
> 
> Not compatible with Unicode? There aren't many character sets out
> there that include characters not in Unicode - that was the whole
> point. Of course, there are plenty of files in unspecified eight-bit
> encodings, so you may have a problem with reliable decoding - but if
> you know what the encoding is, you ought to be able to represent each
> character in Unicode.

What I meant was that some encodings -- namely ASCII and Latin-1 -- the
ordinals are exactly equivalent to Unicode, that is:

# Python 3
for i in range(128):
    assert chr(i).encode('ASCII') == bytes([i])

for i in range(256):
    assert chr(i).encode('Latin-1') == bytes([i])


That's not quite as significant as I thought, though. What is significant is
that a pure ASCII file on disk can be read by a program assuming UTF-8:

for i in range(128):
    assert chr(i).encode('UTF-8') == bytes([i])


although the same is not the case for Latin-1 encoded files.


> Not compatible with any of the UTFs, that's different. Plenty of that
> in the world.
> 
>> You are certainly correct that in it's full generality, "text" is much
>> more than just a string of code points. Unicode strings is a primitive
>> data type. A powerful and sophisticated text processing application may
>> even find Python strings too primitive, possibly needing something like
>> ropes of graphemes rather than strings of code points.
> 
> That's probably more an efficiency point, though. It should be
> possible to do a perfect two-way translation between your grapheme
> rope and a Python string; otherwise, you'll have great difficulty
> saving your file to the disk (which will normally involve representing
> the text in Unicode, then encoding that to bytes).

Well, yes. My point, agreeing with Marko, is that any time you want to do
something even vaguely related to human-readable text, "code points" are
not enough. For example, if I give a string containing the following two
code points in this order:

LATIN SMALL LETTER E
COMBINING CIRCUMFLEX ACCENT

then my application should treat that as a single "character" and display it
as:

LATIN SMALL LETTER E WITH CIRCUMFLEX

which looks like this: ê

rather than two distinct "characters" eˆ

Now, that specific example is a no-brainer, because the Unicode
normalization routines will handle the conversion. But not every
combination of accented characters has a canonical combined form. What
about something like this?

'w\N{COMBINING CIRCUMFLEX ACCENT}\N{COMBINING OGONEK}\N{COMBINING CARON}'

If I insert a character into my string, I want to be able to insert before
the w or after the caron, but not in the middle of those three code points.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#81214

On Sat, Nov 22, 2014 at 2:23 AM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Chris Angelico wrote:
>
>> On Fri, Nov 21, 2014 at 11:32 AM, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>>> (E.g. there are millions of existing files across the world containing
>>> text which use legacy encodings that are not compatible with Unicode.)
>>
>> Not compatible with Unicode? There aren't many character sets out
>> there that include characters not in Unicode - that was the whole
>> point. Of course, there are plenty of files in unspecified eight-bit
>> encodings, so you may have a problem with reliable decoding - but if
>> you know what the encoding is, you ought to be able to represent each
>> character in Unicode.
>
> What I meant was that some encodings -- namely ASCII and Latin-1 -- the
> ordinals are exactly equivalent to Unicode, that is:
>
> That's not quite as significant as I thought, though. What is significant is
> that a pure ASCII file on disk can be read by a program assuming UTF-8:
>
> although the same is not the case for Latin-1 encoded files.

Yep. Thing is, Unicode can't magically convert all files on all
disks... but with a good codec library, you can at least convert
things as you find them. (I was reading MacRoman files earlier this
year. THAT is an encoding I didn't expect I'd find in 2014.)

> Well, yes. My point, agreeing with Marko, is that any time you want to do
> something even vaguely related to human-readable text, "code points" are
> not enough. ... What about something like this?
>
> 'w\N{COMBINING CIRCUMFLEX ACCENT}\N{COMBINING OGONEK}\N{COMBINING CARON}'
>
> If I insert a character into my string, I want to be able to insert before
> the w or after the caron, but not in the middle of those three code points.

Yes, which is a concern. Also a concern is the ability to detect other
boundaries, like words. None of these can be easily solved; all of
them can be dealt with by using the Unicode character data, which is
better than you get for most legacy encodings. In terms of Python
strings, it still makes sense to insert characters between those
combining characters; so what you're saying is that a text editor
widget needs to be aware of more than just code points. Which is
trivially obvious in the presence of RTL text, too; cursor positions
through differing-direction text will be an issue.

The problems you're citing aren't Unicode problems. They stem from the
complexities of human languages. Unicode just makes them a bit more
visible to English-only-speaking programmers.

ChrisA

[toc] | [prev] | [next] | [standalone]

#81218

On 2014-11-22 02:23, Steven D'Aprano wrote:
> LATIN SMALL LETTER E
> COMBINING CIRCUMFLEX ACCENT
> 
> then my application should treat that as a single "character" and
> display it as:
> 
> LATIN SMALL LETTER E WITH CIRCUMFLEX
> 
> which looks like this: ê
> 
> rather than two distinct "characters" eˆ
> 
> Now, that specific example is a no-brainer, because the Unicode
> normalization routines will handle the conversion. But not every
> combination of accented characters has a canonical combined form.
> What about something like this?
> 
> 'w\N{COMBINING CIRCUMFLEX ACCENT}\N{COMBINING OGONEK}\N{COMBINING
> CARON}'
> 
> If I insert a character into my string, I want to be able to insert
> before the w or after the caron, but not in the middle of those
> three code points.

Things get even weirder if you have

 '\N{LATIN SMALL LETTER E WITH CIRCUMFLEX}\N{COMBINING
 OGONEK}\N{COMBINING CARON}'

and when you try to do comparisons like

 s1 = '\N{LATIN SMALL LETTER E WITH CIRCUMFLEX}\N{COMBINING OGONEK}'
 s2 = 'e\N{COMBINING CIRCUMFLEX ACCENT}\N{COMBINING OGONEK}'
 s3 = 'e\N{COMBINING OGONEK}\N{COMBINING CIRCUMFLEX ACCENT}'
 print(s1 == s2)
 print(s1 == s3)
 print(s2 == s3)

Then you also have the case where you want to edit text and the user
wants to remove the COMBINING OGONEK from the character, so you *do*
want to do something akin to

 s4 = ''.join(c for c in s3 if c != '\N{COMBINING OGONEK}')

And yet, weird things happen if you try to remove the circumflex:

  for test in (s1, s2, s3):
    print(test == ''.join(
      c for c in test if c != '\N{COMBINING CIRCUMFLEX ACCENT}'
      )

They all make sense if you understand what's going on under the hood,
but from a visual/conceptual perspective, something feels amiss.

-tkc

[toc] | [prev] | [next] | [standalone]

Page 3 of 4 — ← Prev page 1 2 [3] 4  Next page →

Back to top | Article view | comp.lang.python

csiph-web