Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #53246

Re: split lines from stdin into a list of unicode strings

Show all headers | View raw

Kurt Mueller wrote:

> I have to say that I am a bit disapointed by the chardet library.
> The encoding for the single character 'ü'
> is detected as {'confidence': 0.99, 'encoding': 'EUC-JP'},
> whereas "file" says:
> $ echo "ü" | file -i -
> /dev/stdin: text/plain; charset=utf-8
> $
> 
> "ü" is a character I use very often, as it is in my name: "Müller":-)

You cannot determine an encoding by a single letter. 

Why should "ü" be more likely than "端"? The only thing you can blame chardet 
for is that its confidence rating is a flat out lie...

For "Müller" on the other side you could probably come up with a (simple) 
heuristic that "ü" is more likely to be surrounded by ascii-letters than 
"端".

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: split lines from stdin into a list of unicode strings Dave Angel <davea@davea.name> - 2013-08-28 11:13 +0000
  Re: split lines from stdin into a list of unicode strings kurt.alfred.mueller@gmail.com - 2013-08-28 05:39 -0700
    Re: split lines from stdin into a list of unicode strings Peter Otten <__peter__@web.de> - 2013-08-29 11:12 +0200
    Re: split lines from stdin into a list of unicode strings Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2013-08-29 13:31 +0200
    Re: split lines from stdin into a list of unicode strings Peter Otten <__peter__@web.de> - 2013-08-29 15:15 +0200
    Re: split lines from stdin into a list of unicode strings Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2013-09-05 09:42 +0200
    Re: split lines from stdin into a list of unicode strings Peter Otten <__peter__@web.de> - 2013-09-05 10:33 +0200
    Re: split lines from stdin into a list of unicode strings Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2013-09-05 15:25 +0200

csiph-web