Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!feeder2.ecngs.de!ecngs!feeder.ecngs.de!xlned.com!feeder1.xlned.com!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <kurt.alfred.mueller@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.021
X-Spam-Evidence: '*H*': 0.96; '*S*': 0.00; 'encoding': 0.05; 'wednesday,': 0.07; 'lines.': 0.09; 'subject:into': 0.09; 'template.': 0.09; 'url:github': 0.09; 'kurt': 0.12; 'template': 0.14; 'buffer.': 0.16; 'character.': 0.16; 'detected': 0.16; 'none.': 0.16; 'subject:unicode': 0.16; 'url:linux': 0.16; 'vary.': 0.16; 'size,': 0.16; 'wrote:': 0.18; 'library': 0.18; 'bit': 0.19; 'file,': 0.19; 'split': 0.19; '(the': 0.22; '>>>': 0.22; 'input': 0.22; 'tests': 0.22; 'header:User-Agent:1': 0.23; 'script': 0.25; 'equivalent': 0.26; 'header:In-Reply-To:1': 0.27; 'bigger': 0.30; 'errors': 0.30; 'returned': 0.30; 'subject:list': 0.30; 'lines': 0.31; '>>>>': 0.31; 'run': 0.32; 'text': 0.33; 'subject:from': 0.34; 'but': 0.35; 'add': 0.35; 'there': 0.35; 'should': 0.36; 'seconds': 0.37; 'list': 0.37; 'performance': 0.37; 'message-id:@gmail.com': 0.38; 'ahead': 0.38; 'tasks': 0.38; 'to:addr:python-list': 0.38; 'files': 0.38; 'does': 0.39; 'to:addr:python.org': 0.39; 'read': 0.60; 'easy': 0.60; 'dave': 0.60; 'august': 0.61; 'url:3': 0.61; 'took': 0.61; 'email addr:gmail.com': 0.63; 'real': 0.63; 'more': 0.64; 'different': 0.65; 'header:Reply-To:1': 0.67; 'limit': 0.70; 'reply-to:no real name:2**0': 0.71; 'reply-to:addr:gmail.com': 0.80; 'distracted': 0.84; 'issues:': 0.84; 'angel': 0.91; '2013': 0.98
X-Virus-Scanned: amavisd-new at aerodynamics.ch
Date: Thu, 05 Sep 2013 09:42:36 +0200
From: Kurt Mueller <kurt.alfred.mueller@gmail.com>
Organization: Rothenburg
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: split lines from stdin into a list of unicode strings
References: <521DB58E.5000102@gmail.com> <mailman.302.1377688438.19984.python-list@python.org> <b67f4179-53c6-4bd4-b9ea-8852c9048be0@googlegroups.com> <kvn399$a3r$1@ger.gmane.org>
In-Reply-To: <kvn399$a3r$1@ger.gmane.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: kurt.alfred.mueller@gmail.com
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.80.1378366986.5461.python-list@python.org>
Lines: 36
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1378366986 news.xs4all.nl 15905 [2001:888:2000:d::a6]:58268
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:53679

Am 29.08.2013 11:12, schrieb Peter Otten:
> kurt.alfred.mueller@gmail.com wrote:
>> On Wednesday, August 28, 2013 1:13:36 PM UTC+2, Dave Angel wrote:
>>> On 28/8/2013 04:32, Kurt Mueller wrote:
>>>> For some text manipulation tasks I need a template to split lines
>>>> from stdin into a list of strings the way shlex.split() does it.
>>>> The encoding of the input can vary.

> You can compromise and read ahead a limited number of lines. Here's my demo 
> script (The interesting part is detect_encoding(), I got a bit distracted by 
> unrelated stuff...). The script does one extra decode/encode cycle -- it 
> should be easy to avoid that if you run into performance issues.

I took your script as a template.
But I used the libmagic library (pyhton-magic) instead of chardet.
See http://linux.die.net/man/3/libmagic
and https://github.com/ahupp/python-magic
( I made tests with files of different size, up to 1.2 [GB] )

I had following issues:

- I a real file, the encoding was detected as 'ascii' for detect_lines=1000.
  In line 1002 there was an umlaut character. So then the line.decode(encoding) failed.
  I think to add the errors parameter, line.decode(encoding, errors='replace')

- If the buffer was bigger than about some Megabytes, the returned encoding
  from libmagic was always None. The big files had very long lines ( more than 4k per line ).
  So with detect_lines=1000 this limit was exceeded.

- The magic.buffer() ( the equivalent of chardet.detect() ) takes about 2 seconds
  per megabyte buffer.



-- 
Kurt Mueller