Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #88110
| Date | 2015-03-26 18:47 -0500 |
|---|---|
| From | Tim Chase <python.list@tim.thechases.com> |
| Subject | Re: A simple single line, triple-quoted comment is giving syntax error. Why? |
| References | (8 earlier) <2362875.FIK8ImHTJA@PointedEars.de> <CALwzidmOsagqXs5Nhr5w1w2Ukv9oDp2mxikcRmHd9rKG2x3Low@mail.gmail.com> <mailman.212.1427394775.10327.python-list@python.org> <1582467.9xuzSGXesM@PointedEars.de> <55149305$0$13009$c3e8da3$5496439d@news.astraweb.com> |
| Newsgroups | comp.lang.python |
| Message-ID | <mailman.225.1427414131.10327.python-list@python.org> (permalink) |
On 2015-03-27 10:15, Steven D'Aprano wrote:
> If that's all it is, why don't you just run the tokenizer over it
> and see what it says?
>
> py> from cStringIO import StringIO
> py> code = StringIO('spam = "abcd" "efgh"\n')
> py> import tokenize
> py> for item in tokenize.generate_tokens(code.readline):
> ... print item
> ...
> (1, 'spam', (1, 0), (1, 4), 'spam = "abcd" "efgh"\n')
> (51, '=', (1, 5), (1, 6), 'spam = "abcd" "efgh"\n')
> (3, '"abcd"', (1, 7), (1, 13), 'spam = "abcd" "efgh"\n')
> (3, '"efgh"', (1, 14), (1, 20), 'spam = "abcd" "efgh"\n')
> (4, '\n', (1, 20), (1, 21), 'spam = "abcd" "efgh"\n')
> (0, '', (2, 0), (2, 0), '')
>
>
> Looks to me that the two string literals each get their own token,
Nice. I haven't played with the tokenize module before, but
resolving arguments on comp.lang.python is one of the best possible
uses.
It was interesting to try other feeders to generate_tokens(), my favorite being
>>> import tokenize
>>> i = iter(["spam = 'abc' 'def'"])
>>> for item in tokenize.generate_tokens(lambda: next(i)):
... print(item)
...
TokenInfo(type=1 (NAME), string='spam', start=(1, 0), end=(1, 4), line="spam = 'abc' 'def'")
TokenInfo(type=52 (OP), string='=', start=(1, 5), end=(1, 6), line="spam = 'abc' 'def'")
TokenInfo(type=3 (STRING), string="'abc'", start=(1, 7), end=(1, 12), line="spam = 'abc' 'def'")
TokenInfo(type=3 (STRING), string="'def'", start=(1, 13), end=(1, 18), line="spam = 'abc' 'def'")
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')
It's also nice to have the translation from token-type to token-type-name in Py3
-tkc
Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
A simple single line, triple-quoted comment is giving syntax error. Why? Aditya Raj Bhatt <adityarajbhatt@gmail.com> - 2015-03-18 10:46 -0700
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Laurent Pointal <laurent.pointal@laposte.net> - 2015-03-18 19:04 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Laurent Pointal <laurent.pointal@laposte.net> - 2015-03-18 19:06 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-18 20:53 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-18 20:56 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Terry Reedy <tjreedy@udel.edu> - 2015-03-18 17:47 -0400
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-19 09:58 +1100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Chris Angelico <rosuav@gmail.com> - 2015-03-19 10:23 +1100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Aditya Raj Bhatt <adityarajbhatt@gmail.com> - 2015-03-18 11:23 -0700
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Igor Korot <ikorot01@gmail.com> - 2015-03-18 14:45 -0400
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Laurent Pointal <laurent.pointal@laposte.net> - 2015-03-18 20:08 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Tim Roberts <timr@probo.com> - 2015-03-28 16:34 -0700
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-18 21:21 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Tim Chase <python.list@tim.thechases.com> - 2015-03-18 13:15 -0500
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ben Finney <ben+python@benfinney.id.au> - 2015-03-19 11:06 +1100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Denis McMahon <denismfmcmahon@gmail.com> - 2015-03-19 00:53 +0000
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-22 04:14 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Chris Angelico <rosuav@gmail.com> - 2015-03-22 14:36 +1100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-22 04:49 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Chris Angelico <rosuav@gmail.com> - 2015-03-22 15:11 +1100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-26 05:35 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-25 23:09 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-26 17:45 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-26 12:29 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-26 19:54 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-26 17:27 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-28 19:20 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-29 01:56 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-29 12:41 +0200
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-31 01:23 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? sohcahtoa82@gmail.com - 2015-03-31 16:10 -0700
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Chris Angelico <rosuav@gmail.com> - 2015-04-01 10:36 +1100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-31 17:43 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-04-02 23:31 +0200
Re: A simple single line, triple-quoted comment is giving syntax error. Why? sohcahtoa82@gmail.com - 2015-04-02 15:26 -0700
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-04-02 18:21 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-04-03 16:40 +1100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Chris Angelico <rosuav@gmail.com> - 2015-04-03 16:57 +1100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Marko Rauhamaa <marko@pacujo.net> - 2015-04-03 10:13 +0300
r"\"" ??? (was A simple single line, triple-quoted comment) Rustom Mody <rustompmody@gmail.com> - 2015-04-03 05:52 -0700
Re: r"\"" ??? (was A simple single line, triple-quoted comment) Chris Angelico <rosuav@gmail.com> - 2015-04-04 01:40 +1100
Re: r"\"" ??? (was A simple single line, triple-quoted comment) Rustom Mody <rustompmody@gmail.com> - 2015-04-03 08:14 -0700
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-04-03 09:25 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-04-03 00:40 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-03-27 11:43 +1300
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-28 19:06 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-26 12:32 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-26 19:56 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-03-27 10:15 +1100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Tim Chase <python.list@tim.thechases.com> - 2015-03-26 18:47 -0500
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Ian Kelly <ian.g.kelly@gmail.com> - 2015-03-26 19:38 -0600
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Marko Rauhamaa <marko@pacujo.net> - 2015-03-27 07:34 +0200
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Dave Angel <davea@davea.name> - 2015-03-26 01:23 -0400
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-26 18:25 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2015-03-26 18:30 +0100
Re: A simple single line, triple-quoted comment is giving syntax error. Why? Marko Rauhamaa <marko@pacujo.net> - 2015-03-22 12:00 +0200
csiph-web