Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #53877

Re: Chardet, file, ... and the Flexible String Representation

Show all headers | View raw

On 9/9/13 10:28 AM, wxjmfauth@gmail.com wrote:
> Le vendredi 6 septembre 2013 17:46:14 UTC+2, Piet van Oostrum a écrit :
>> wxjmfauth@gmail.com writes:
>>
>>
>>
>>> The Flexible String Representation has conceptually to
>>> face the same problem. It splits "unicode" in chunks and
>>> it has to solve two problems at the same time, the coding
>>> and the handling of multiple "char sets". The problem?
>>> It fails.
>>> "This poor Flexible String Representation does not succeed
>>> to solve the problem it create itsself."
>>
>>
>> The FSR does not split unicode in chuncks. It does not create problems and therefore it doesn't have to solve this.
>>
>>
>>
>> The FSR simply stores a Unicode string as an array[*] of ints (the Unicode code points of the characters of the string. That's it. Then it uses a memory-efficient way to store this array of ints. But that has nothing to do with character sets. The same principle could be used for any array of ints.
>>
>>
>>
>> So you are seeking problems where there are none. And you would have a lot more peace of mind if you stopped doing this.
>>
>>
>>
>> [*] array in the C sense.
>>
>> -- 
>>
>> Piet van Oostrum <piet@vanoostrum.org>
>>
>> WWW: http://pietvanoostrum.com/
>>
>> PGP key: [8DAE142BE17999C4]
> ----------
>
>
> Due to its nature, a character cann't be handled in the
> same way a one another type. That's the purpose of the UTF.
>
> -----
>
> Chunk latin-1, perfomance
>
> ref:
>>>> timeit.timeit("a = 'hundred'; 'x' in a")
> 0.13144639994075646
>
>>>> timeit.timeit("a = 'hundrez'; 'x' in a")
> 0.13780295544393084
>
> Chunk ucs2, perfomance
>
>>>> timeit.timeit("a = 'hundre€'; 'x' in a")
> 0.23505392241617074
>
> Chunk ucs4, perfomance
>
>>>> timeit.timeit("a = 'hundre\U0001d11e'; 'x' in a")
> 0.26266673650735584
>
> Comment: Such differences never happen with utf.
>
> -----
>
> Chunk latin-1, memory
>
>>>> sys.getsizeof('a')
> 26
>
> Chunk ucs2, memory
>
>>>> sys.getsizeof('€')
> 40
>
> Comment: 14 bytes more than latin-1
>
> Chunk ucs4, memory
>
>>>> sys.getsizeof('\U0001d11e')
> 44
>
> Comment: 18 bytes more than latin-1
>
> Comment: With utf, a char (in string or not) never exceed 4
>
> bytes.
>
> -----
>
> 'a' + '€' in utf, conceptually
>
> Concatenate the *unicode tranformation units*.
> Some kind of a real direct 'a' + '€'.
>
>
> 'a' + '€' in FSR, conceptually
>
> 1) Check the "internal coding" of 'a'
> 2) Check the "internal coding" of '€'
> 3) Compare these codings
>
> 4a) If they match, concatenate the bytes
>
> 4b) If they do not match
> 	5) Reencode the string which has to
> 	6) Concatenate
> 	7) Set the "internal coding" status for
> 	further processing
>
> -----
>
> Complicate and full of side effects, eg :
>
>>>> sys.getsizeof('a')
> 26
>>>> sys.getsizeof('aé')
> 39
>
> Is not a latin-1 "é" supposed to count as a latin-1 "a" ?
>
> ----
>
> I picked up random methods, there may be variations, basically
> this general behaviour is always expected.
>
>
> jmf
>

jmf, thanks for your reply.  You've calmed my fears that there is 
something wrong with the Flexible String Representation.  None of the 
examples you show demonstrate any behavior contrary to the Unicode spec.

--Ned.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Chardet, file, ... and the Flexible String Representation wxjmfauth@gmail.com - 2013-09-06 02:11 -0700
  Re: Chardet, file, ... and the Flexible String Representation Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-09-06 10:57 +0000
  Re: Chardet, file, ... and the Flexible String Representation Antoon Pardon <antoon.pardon@rece.vub.ac.be> - 2013-09-06 13:10 +0200
  Re: Chardet, file, ... and the Flexible String Representation Ned Batchelder <ned@nedbatchelder.com> - 2013-09-06 07:02 -0400
  Re: Chardet, file, ... and the Flexible String Representation Piet van Oostrum <piet@vanoostrum.org> - 2013-09-06 11:46 -0400
    Re: Chardet, file, ... and the Flexible String Representation Chris Angelico <rosuav@gmail.com> - 2013-09-07 02:04 +1000
    Re: Chardet, file, ... and the Flexible String Representation random832@fastmail.us - 2013-09-06 12:59 -0400
    Re: Chardet, file, ... and the Flexible String Representation Chris Angelico <rosuav@gmail.com> - 2013-09-07 03:04 +1000
    Re: Chardet, file, ... and the Flexible String Representation wxjmfauth@gmail.com - 2013-09-09 07:28 -0700
      Re: Chardet, file, ... and the Flexible String Representation Ned Batchelder <ned@nedbatchelder.com> - 2013-09-09 12:38 -0400
      Re: Chardet, file, ... and the Flexible String Representation Michael Torrie <torriem@gmail.com> - 2013-09-09 11:05 -0600
        Re: Chardet, file, ... and the Flexible String Representation Steven D'Aprano <steve@pearwood.info> - 2013-09-10 04:58 +0000
      Re: Chardet, file, ... and the Flexible String Representation Terry Reedy <tjreedy@udel.edu> - 2013-09-09 16:47 -0400
      Re: Chardet, file, ... and the Flexible String Representation random832@fastmail.us - 2013-09-10 11:36 -0400
    Re: Chardet, file, ... and the Flexible String Representation random832@fastmail.us - 2013-09-09 14:34 -0400
    Re: Chardet, file, ... and the Flexible String Representation Ian Kelly <ian.g.kelly@gmail.com> - 2013-09-09 13:03 -0600
    Re: Chardet, file, ... and the Flexible String Representation random832@fastmail.us - 2013-09-09 15:27 -0400
    Re: Chardet, file, ... and the Flexible String Representation Serhiy Storchaka <storchaka@gmail.com> - 2013-09-12 00:11 +0300

csiph-web