Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder1.xlned.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Mon, 15 Jul 2013 16:42:07 +0100
From: Burak Arslan <burak.arslan@arskom.com.tr>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130711 Thunderbird/17.0.7
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Python - remote object protocols and security
References: <CAPTjJmqAEUUrUxaFjAh8qGjBbNuhNp9Nz6RKQDbraOm0kCVJDg@mail.gmail.com> <595253102.8424684.1373892072113.JavaMail.root@sequans.com> <CAPTjJmoP0OHZP+GBjjZxVMwVT0eSXG1azK1NkZov_x4=1jq-xQ@mail.gmail.com>
In-Reply-To: <CAPTjJmoP0OHZP+GBjjZxVMwVT0eSXG1azK1NkZov_x4=1jq-xQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.4731.1373902913.3114.python-list@python.org>
Lines: 72
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:50691

On 07/15/13 13:51, Chris Angelico wrote:
> So the only bit you still need is: How do you transmit this across the
> network? Since it's now all just bytes, that's easy enough to do, eg
> with TCP. But that depends on the rest of your system, and is a quite
> separate question - and quite probably one you already have the answer
> to.

For Json, you need to have a way of delimiting messages -- to my
knowledge, Python's json library does not support parsing streams.

You can send the json document in the body of a Http POST, or a ZeroMQ
message, or in a UDP datagram (if you can guarantee it fits inside one)
or in a simple TCP-based encapsulation mechanism that e.g. prepends the
length of the message to the document.

e.g.

'\x00\x00\x00\x07{"a":1}'

As MessagePack already does this, you can send MessagePack documents via
an ordinary TCP socket and easily recover them on the other side of the
pipe.

>>> import msgpack; from StringIO import StringIO
>>> s = StringIO(msgpack.dumps({"a":1}) + msgpack.dumps({"b":2}))
>>> for doc in msgpack.Unpacker(s):
...     print doc
...
{'a': 1}
{'b': 2}

This won't work with Json:

>>> import json; from StringIO import StringIO
>>> s = StringIO(json.dumps({"a":1}) + json.dumps({"b":2}))
>>> for doc in json.load(s): # or whatever ???
...     print doc
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 368, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 9 - line 1 column 17 (char 8 - 16)

Note that this is a limitation of python's Json parser, not Json itself.

There seems to be a json.scanner module that *sounds* like it provides
this functionality,
but I couldn't find any documentation about it.

Alternatively, PyYaml can also parse streams. yaml.{dump,load}_all()
provide pickle-like unsafe (de)serialization support and
yaml.safe_{dump,load}_all provide msgpack-like safe-but-limited stream
parsing support.


also;

On 07/15/13 13:57, Chris Angelico wrote:
> But what I meant was that the [Json] protocol itself is designed with
> security restrictions in mind. It's designed not to fetch additional
> content from the network (as XML can),

Can you explain how parsing XML can fetch data from the network?


Best,
Burak