Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #53147

Re: split lines from stdin into a list of unicode strings

Show all headers | View raw

On Wednesday, August 28, 2013 1:13:36 PM UTC+2, Dave Angel wrote:
> On 28/8/2013 04:32, Kurt Mueller wrote:
> > For some text manipulation tasks I need a template to split lines
> > from stdin into a list of strings the way shlex.split() does it.
> > The encoding of the input can vary.

> Does that mean it'll vary from one run of the program to the next, or
> it'll vary from one line to the next?  Your code below assumes the
> latter.  That can greatly increase the unreliability of the already
> dubious chardet algorithm.

The encoding only varies from one launch to the other.
The reason I process each line is memory usage.

Option to have a better reliability of chardet:
I could read all of the input, save the input lines for further
processing in a list, feed the lines into
chardet.universaldetector.UniversalDetector.feed()/close()/result()
and then decode and split/shlex the lines in the list.
That way the chardet oracle would be more reliable, but 
roughly twice as much memory will be used.


> > import chardet
> Is this the one ?
>     https://pypi.python.org/pypi/chardet

Yes.

> > $ cat <some-file> | template.py

> Why not have a separate filter that converts from a (guessed) encoding
> into utf-8, and have the later stage(s)  assume utf-8 ?  That way, the
> filter could be fed clues by the user, or replaced entirely, without
> affecting the main code you're working on.

Working on UNIX-like systems (I am happy to work in a MSFZ)
the processing pipe would be then:

cat <some-file> | recode2utf8 | splitlines.py
memory usage 2 * <some-file> ( plus chardet memory usage )


> Alternatively, just add a commandline argument with the encoding, and
> parse it into enco_type.

cat <some-file> | splitlines.py -e latin9
memory usage 1 * <some-file>

or

cat <some-file> | splitlines.py -e $( codingdetect <some-file> )
memory usage 1 * <some-file>

So, because memory usage is not primary,
I think I will go with the option described above.


-- 
Kurt Müller

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Re: split lines from stdin into a list of unicode strings Dave Angel <davea@davea.name> - 2013-08-28 11:13 +0000
  Re: split lines from stdin into a list of unicode strings kurt.alfred.mueller@gmail.com - 2013-08-28 05:39 -0700
    Re: split lines from stdin into a list of unicode strings Peter Otten <__peter__@web.de> - 2013-08-29 11:12 +0200
    Re: split lines from stdin into a list of unicode strings Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2013-08-29 13:31 +0200
    Re: split lines from stdin into a list of unicode strings Peter Otten <__peter__@web.de> - 2013-08-29 15:15 +0200
    Re: split lines from stdin into a list of unicode strings Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2013-09-05 09:42 +0200
    Re: split lines from stdin into a list of unicode strings Peter Otten <__peter__@web.de> - 2013-09-05 10:33 +0200
    Re: split lines from stdin into a list of unicode strings Kurt Mueller <kurt.alfred.mueller@gmail.com> - 2013-09-05 15:25 +0200

csiph-web