Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Sat, 28 Sep 2013 13:02:37 +0200
From: Andreas Perstinger <andipersti@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Thunderbird/24.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Weird bahaviour from shlex - line no
References: <CAEvez=x1ySjRtR_7asYtU3FoRsC8c1g0g_OZQ_NE-Ok443bFrg@mail.gmail.com>
In-Reply-To: <CAEvez=x1ySjRtR_7asYtU3FoRsC8c1g0g_OZQ_NE-Ok443bFrg@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.408.1380366168.18130.python-list@python.org>
Lines: 50
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:54937

On 28.09.2013 08:26, Daniel Stojanov wrote:
> Can somebody explain this. The line number reported by shlex depends
> on the previous token. I want to be able to tell if I have just popped
> the last token on a line.
>
[SNIP]
>
> second = shlex.shlex("word1 word2,\nword3")

Punctuation characters like the comma are not considered as word 
characters by default and thus are seen as different tokens (consisting 
of only a single character):

 >>> lexer = shlex.shlex("foo, bar, ...")
 >>> token = lexer.get_token()
 >>> while token != lexer.eof:
...   print token
...   token = lexer.get_token()
...
foo
,
bar
,
.
.
.

If you want to treat them as "word" characters you need to add them to 
the string "wordchars" (a public attribute of the "shlex" instance):

 >>> lexer = shlex.shlex("foo.bar, baz")
 >>> lexer.wordchar += '.,'
 >>> print lexer.get_token()
foo.bar,
 >>> print lexer.get_token()
baz

There is also a "debug" attribute (with three different levels: 1, 2, 3; 
default value 0 means no debug output):

 >>> lexer = shlex.shlex("foo, bar, ...")
 >>> lexer.debug = 1
 >>> print lexer.get_token()
shlex: token='foo'
foo
 >>> print lexer.get_token()
shlex: popping token ','
,

Bye, Andreas