Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder5.xlned.com!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Thu, 29 Aug 2013 13:31:45 +0200
From: Kurt Mueller <kurt.alfred.mueller@gmail.com>
Organization: Rothenburg
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: split lines from stdin into a list of unicode strings
References: <521DB58E.5000102@gmail.com> <mailman.302.1377688438.19984.python-list@python.org> <b67f4179-53c6-4bd4-b9ea-8852c9048be0@googlegroups.com> <kvn399$a3r$1@ger.gmane.org>
In-Reply-To: <kvn399$a3r$1@ger.gmane.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Precedence: list
Reply-To: kurt.alfred.mueller@gmail.com
Newsgroups: comp.lang.python
Message-ID: <mailman.361.1377775933.19984.python-list@python.org>
Lines: 61
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:53239

Am 29.08.2013 11:12, schrieb Peter Otten:
> kurt.alfred.mueller@gmail.com wrote:
>> On Wednesday, August 28, 2013 1:13:36 PM UTC+2, Dave Angel wrote:
>>> On 28/8/2013 04:32, Kurt Mueller wrote:
>>>> For some text manipulation tasks I need a template to split lines
>>>> from stdin into a list of strings the way shlex.split() does it.
>>>> The encoding of the input can vary.

> You can compromise and read ahead a limited number of lines. Here's my =
demo=20
> script (The interesting part is detect_encoding(), I got a bit distract=
ed by=20
> unrelated stuff...). The script does one extra decode/encode cycle -- i=
t=20
> should be easy to avoid that if you run into performance issues.

Thanks Peter!

I see the idea. It limits the buffersize/memory usage for the detection.


I have to say that I am a bit disapointed by the chardet library.
The encoding for the single character '=FC'
is detected as {'confidence': 0.99, 'encoding': 'EUC-JP'},
whereas "file" says:
$ echo "=FC" | file -i -
/dev/stdin: text/plain; charset=3Dutf-8
$

"=FC" is a character I use very often, as it is in my name: "M=FCller":-)=



I try to use the "python-magic" library which has a similar functionality=

as chardet and is used by the "file" unix-command and it is expandable
with a magicfile, see "man file".


My magic_test script:
-------------------------------------------------------------------
#!/usr/bin/env python
# vim: set fileencoding=3Dutf-8 :
from __future__ import print_function
import magic
strg_chck =3D '=FC'
magc_enco =3D magic.open( magic.MAGIC_MIME_ENCODING )
magc_enco.load()
print( strg_chck + ' encoding=3D' + magc_enco.buffer( strg_chck ) )
magc_enco.close()
-------------------------------------------------------------------
$ magic_test
=FC encoding=3Dutf-8

python-magic seems to me a bit more reliable.


Cheers
--=20
Kurt Mueller