Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Thu, 05 Sep 2013 15:25:50 +0200
From: Kurt Mueller <kurt.alfred.mueller@gmail.com>
Organization: Rothenburg
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: split lines from stdin into a list of unicode strings
References: <521DB58E.5000102@gmail.com> <mailman.302.1377688438.19984.python-list@python.org> <b67f4179-53c6-4bd4-b9ea-8852c9048be0@googlegroups.com> <kvn399$a3r$1@ger.gmane.org> <522835EC.1010407@gmail.com> <l09fki$rfa$1@ger.gmane.org>
In-Reply-To: <l09fki$rfa$1@ger.gmane.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Precedence: list
Reply-To: kurt.alfred.mueller@gmail.com
Newsgroups: comp.lang.python
Message-ID: <mailman.95.1378387574.5461.python-list@python.org>
Lines: 82
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:53707

Am 05.09.2013 10:33, schrieb Peter Otten:
> Kurt Mueller wrote:
>> Am 29.08.2013 11:12, schrieb Peter Otten:
>>> kurt.alfred.mueller@gmail.com wrote:
>>>> On Wednesday, August 28, 2013 1:13:36 PM UTC+2, Dave Angel wrote:
>>>>> On 28/8/2013 04:32, Kurt Mueller wrote:
>>>>>> For some text manipulation tasks I need a template to split lines
>>>>>> from stdin into a list of strings the way shlex.split() does it.
>>>>>> The encoding of the input can vary.
>> I took your script as a template.
>> But I used the libmagic library (pyhton-magic) instead of chardet.
>> See http://linux.die.net/man/3/libmagic
>> and https://github.com/ahupp/python-magic
>> ( I made tests with files of different size, up to 1.2 [GB] )
>> I had following issues:
>> - I a real file, the encoding was detected as 'ascii' for
>> detect_lines=1000.
>>   In line 1002 there was an umlaut character. So then the
>>   line.decode(encoding) failed. I think to add the errors parameter,
>>   line.decode(encoding, errors='replace')
> 
> Tough luck ;) You could try and tackle the problem by skipping leading 
> ascii-only lines. Untested:
> 
> def detect_encoding(instream, encoding, detect_lines, skip_ascii=True):
>     if encoding is None:
>         encoding = instream.encoding
>         if encoding is None:
>             if skip_ascii:
>                 try:
>                     for line in instream:
>                         yield line.decode("ascii")
>                 except UnicodeDecodeError:
>                     pass
>                 else:
>                     return
>             head = [line]
>             head.extend(islice(instream, detect_lines-1))
>             encoding =  chardet.detect("".join(head))["encoding"]
>             instream = chain(head, instream)
>     for line in instream:
>         yield line.decode(encoding)

I find this solution as a generator very nice.
With just some small modifications it runs fine for now.
( line is undefined if skip_ascii is False. )

For ascii only files chardet or libmagic will not be bothered.
And the detect_lines comes not in charge, until there are
some non ascii characters.

------------------------------------------------------------------------------
def decode_stream_lines( inpt_strm, enco_type, numb_inpt, skip_asci=True, ):
    if enco_type is None:
        enco_type = inpt_strm.encoding
        if enco_type is None:
            line_head = []
            if skip_asci:
                try:
                    for line in inpt_strm:
                        yield line.decode( 'ascii' )
                except UnicodeDecodeError:
                    line_head = [ line ] # last line was not ascii
                else:
                    return # all lines were ascii
            line_head.extend( islice( inpt_strm, numb_inpt - 1 ) )
            magc_enco = magic.open( magic.MAGIC_MIME_ENCODING )
            magc_enco.load()
            enco_type = magc_enco.buffer( "".join( line_head ) )
            magc_enco.close()
            print( I_AM + '-ERROR: enco_type=' + repr( enco_type ), file=sys.stderr, )
            if  enco_type.rfind( 'binary' ) >= 0: # binary, application/mswordbinary, application/vnd.ms-excelbinary and the like
                return
            inpt_strm = chain( line_head, inpt_strm )
    for line in inpt_strm:
        yield line.decode( enco_type, errors='replace' )
------------------------------------------------------------------------------


Thank you very much!
-- 
Kurt Mueller