Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #84354 > unrolled thread

Case-insensitive sorting of strings (Python newbie)

Back to article view | Back to comp.lang.python

Contents

  Case-insensitive sorting of strings (Python newbie) John Sampson <jrs.idx@ntlworld.com> - 2015-01-23 17:00 +0000
    Re: Case-insensitive sorting of strings (Python newbie) Michael Ströder <michael@stroeder.com> - 2015-01-23 18:10 +0100
    Re: Case-insensitive sorting of strings (Python newbie) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-01-24 04:56 +1100

#84354 — Case-insensitive sorting of strings (Python newbie)

I notice that the string method 'lower' seems to convert some strings 
(input from a text file) to Unicode but not others.
This messes up sorting if it is used on arguments of 'sorted' since 
Unicode strings come before ordinary ones.

Is there a better way of case-insensitive sorting of strings in a list? 
Is it necessary to convert strings read from a plaintext file
to Unicode? If so, how? This is Python 2.7.8.

Regards

John Sampson

[toc] | [next] | [standalone]

#84357

John Sampson wrote:
> I notice that the string method 'lower' seems to convert some strings (input
> from a text file) to Unicode but not others.
> This messes up sorting if it is used on arguments of 'sorted' since Unicode
> strings come before ordinary ones.

I doubt that. Can you provide a short example?

> Is there a better way of case-insensitive sorting of strings in a list? Is it
> necessary to convert strings read from a plaintext file
> to Unicode? If so, how? This is Python 2.7.8.

Well, if you have non-ASCII chars for many Unicode characters str.lower()
won't give reasonable results. So binary strings containing an encoding of
Unicode character entities should be decoded to Unicode strings first.

Ciao, Michael.

[toc] | [prev] | [next] | [standalone]

#84372

John Sampson wrote:

> I notice that the string method 'lower' seems to convert some strings
> (input from a text file) to Unicode but not others.

I don't think so. You're going to have to show an example.

I *think* what you might be running into is an artifact of printing to a
terminal, which may (or may not) interpret some byte sequences as UTF-8
characters, but I can't replicate it. So I'll have to see an example.
Please state what OS you are running on, and what encoding your terminal is
set to. Also, are you opening the file in text mode or binary mode?


> This messes up sorting if it is used on arguments of 'sorted' since
> Unicode strings come before ordinary ones.
> 
> Is there a better way of case-insensitive sorting of strings in a list?
> Is it necessary to convert strings read from a plaintext file
> to Unicode? If so, how? This is Python 2.7.8.

Best practice is to always convert to Unicode, even if you know your text is
pure ASCII. You *may* be able to get away with not doing so if you know you
have ASCII, but that's still the lazy way. And of course you need to know
what encoding has been used.

There is some overhead with decoding to Unicode, so if performance really is
critical, *and* your needs are quite low, you may be able to get away with
just treating the strings as ASCII byte strings:

with open("my file.txt") as f:
    for line in f:
        print line.lower()


will correctly lowercase ASCII strings. It won't lowercase non-ASCII
letters, and there's a good chance that they may display as raw bytes in
some encoding. Otherwise, I think the best way to approach this may be:


import io
with io.open("my file.txt", encoding='utf-8') as f:
    for line in f:
        print line.lower()


Assuming the file actually is encoded with UTF-8, that ought to work
perfectly.

But to really know what is going on we will need more information.


-- 
Steven

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web