Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #77672

Re: How to turn a string into a list of integers?

Show key headers only | View raw

On 2014-09-07 02:47, Steven D'Aprano wrote:
> Kurt Mueller wrote:
>
>> Processing any Unicode string will work with small and wide
>> python 2.7 builds and also with python >3.3?
>> ( parts of small build python will not work with values over 0xFFFF )
>> ( strings with surrogate pairs will not work correctly on small build
>> python )
>
>
> If you limit yourself to code points in the Basic Multilingual Plane, U+0000
> to U+FFFF, then Python's Unicode handling works fine no matter what version
> or implementation is used. Since most people use only the BMP, you may not
> notice any problems.
>
> (Of course, there are performance and memory-usage differences from one
> version to the next, but the functionality works correctly.)
>
> If you use characters from the supplementary planes ("astral characters"),
> then:
>
> * wide builds will behave correctly;
> * narrow builds will wrongly treat astral characters as two
>    independent characters, which means functions like len()
>    and string slicing will do the wrong thing;
> * Python 3.3 doesn't use narrow and wide builds any more,
>    and also behaves correctly with astral characters.
>
>
> So there are three strategies for correct Unicode support in Python:
>
> * avoid astral characters (and trust your users will also avoid them);
>
> * use a wide build;
>
> * use Python 3.3 or higher.
>
>
> In case you are wondering what Python 3.3 does differently, when it builds a
> string, it works out the largest code point in the string. If the largest
> code point is no greater than U+00FF, it stores the string in Latin 1 using
> 8 bits per character; if the largest code point is no greater than U+FFFF,
> then it uses UTF-16 (or UCS-2, since with the BMP they are functionally the
> same); if the string contains any astral characters, then it uses UTF-32.
> So regardless of the string, each character uses a single code unit. Only
> the size of the code unit varies.
>
I don't think you should be saying that it stores the string in Latin-1
or UTF-16 because that might suggest that they are encoded. They aren't.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

How to turn a string into a list of integers? cl@isbd.net - 2014-09-03 13:27 +0100
  Re: How to turn a string into a list of integers? Peter Otten <__peter__@web.de> - 2014-09-03 14:52 +0200
    Re: How to turn a string into a list of integers? cl@isbd.net - 2014-09-03 15:48 +0100
      Re: How to turn a string into a list of integers? Joshua Landau <joshua@landau.ws> - 2014-09-04 22:06 +0100
        Re: How to turn a string into a list of integers? cl@isbd.net - 2014-09-05 09:42 +0100
          Re: How to turn a string into a list of integers? Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2014-09-05 19:56 +0200
            Re: How to turn a string into a list of integers? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-06 15:47 +1000
              Re: How to turn a string into a list of integers? Peter Otten <__peter__@web.de> - 2014-09-06 10:22 +0200
                Re: How to turn a string into a list of integers? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-06 21:17 +1000
              Re: How to turn a string into a list of integers? Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2014-09-06 14:15 +0200
                Re: How to turn a string into a list of integers? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-07 04:19 +1000
                Re: How to turn a string into a list of integers? Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2014-09-06 21:28 +0200
                Re: How to turn a string into a list of integers? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-07 11:47 +1000
                Re: How to turn a string into a list of integers? MRAB <python@mrabarnett.plus.com> - 2014-09-07 15:52 +0100
                Re: How to turn a string into a list of integers? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-08 03:02 +1000
                Re: How to turn a string into a list of integers? Rustom Mody <rustompmody@gmail.com> - 2014-09-07 10:53 -0700
                Re: How to turn a string into a list of integers? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-08 04:08 +1000
                Re: How to turn a string into a list of integers? Rustom Mody <rustompmody@gmail.com> - 2014-09-07 11:34 -0700
                Re: How to turn a string into a list of integers? Chris Angelico <rosuav@gmail.com> - 2014-09-08 10:14 +1000
                Re: How to turn a string into a list of integers? Marko Rauhamaa <marko@pacujo.net> - 2014-09-08 08:44 +0300
                Re: How to turn a string into a list of integers? Chris Angelico <rosuav@gmail.com> - 2014-09-08 15:53 +1000
                Re: How to turn a string into a list of integers? Terry Reedy <tjreedy@udel.edu> - 2014-09-08 03:41 -0400
                Re: How to turn a string into a list of integers? Chris Angelico <rosuav@gmail.com> - 2014-09-08 01:04 +1000
                Re: How to turn a string into a list of integers? Roy Smith <roy@panix.com> - 2014-09-07 11:40 -0400
                Re: How to turn a string into a list of integers? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-08 04:00 +1000
                Re: How to turn a string into a list of integers? Chris Angelico <rosuav@gmail.com> - 2014-09-08 10:12 +1000
              Re: How to turn a string into a list of integers? Chris Angelico <rosuav@gmail.com> - 2014-09-06 22:23 +1000
          Re: How to turn a string into a list of integers? Chris “Kwpolska” Warrick <kwpolska@gmail.com> - 2014-09-05 20:25 +0200
          Re: How to turn a string into a list of integers? Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2014-09-05 21:16 +0200
          Re: How to turn a string into a list of integers? Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2014-09-05 22:41 +0200
      Re: How to turn a string into a list of integers? Chris Angelico <rosuav@gmail.com> - 2014-09-05 10:12 +1000
      Re: How to turn a string into a list of integers? Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-04 20:09 -0600
      Re: How to turn a string into a list of integers? Chris Angelico <rosuav@gmail.com> - 2014-09-05 12:15 +1000
        Re: How to turn a string into a list of integers? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-09-06 14:27 +1000
  Re: How to turn a string into a list of integers? obedrios@gmail.com - 2014-09-03 07:30 -0700

csiph-web