Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Wed, 28 Aug 2013 10:32:14 +0200
From: Kurt Mueller <kurt.alfred.mueller@gmail.com>
Organization: Rothenburg
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8
MIME-Version: 1.0
To: python-list@python.org
Subject: split lines from stdin into a list of unicode strings
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: list
Reply-To: kurt.alfred.mueller@gmail.com
Newsgroups: comp.lang.python
Message-ID: <mailman.294.1377678757.19984.python-list@python.org>
Lines: 51
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:53115

This is a follow up to the Subject
"right adjusted strings containing umlauts"

For some text manipulation tasks I need a template to split lines
from stdin into a list of strings the way shlex.split() does it.
The encoding of the input can vary.
For further processing in Python I need the list of strings to be in unicode.

Here is template.py:

##############################################################################################################
#!/usr/bin/env python
# vim: set fileencoding=utf-8 :
# split lines from stdin into a list of unicode strings
# Muk 2013-08-23
# Python 2.7.3

from __future__ import print_function
import sys
import shlex
import chardet

bool_cmnt = True  # shlex: skip comments
bool_posx = True  # shlex: posix mode (strings in quotes)

for inpt_line in sys.stdin:
    print( 'inpt_line=' + repr( inpt_line ) )
    enco_type = chardet.detect( inpt_line )[ 'encoding' ]           # {'encoding': 'EUC-JP', 'confidence': 0.99}
    print( 'enco_type=' + repr( enco_type ) )
    try:
        strg_inpt = shlex.split( inpt_line, bool_cmnt, bool_posx, ) # shlex does not work on unicode
    except Exception, errr:                                         # usually 'No closing quotation'
        print( "error='%s' on inpt_line='%s'" % ( errr, inpt_line.rstrip(), ), file=sys.stderr, )
        continue
    print( 'strg_inpt=' + repr( strg_inpt ) )                       # list of strings
    strg_unic = [ strg.decode( enco_type ) for strg in strg_inpt ]  # decode the strings into unicode
    print( 'strg_unic=' + repr( strg_unic ) )                       # list of unicode strings
##############################################################################################################

$ cat <some-file> | template.py


Comments are welcome.


TIA
-- 
Kurt Mueller