Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!tudelft.nl!txtfeed1.tudelft.nl!multikabel.net!newsfeed10.multikabel.net!xlned.com!feeder7.xlned.com!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Sat, 17 Dec 2011 19:29:53 -0600
From: Evan Driscoll <edriscoll@wisc.edu>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Parsing stream of JSON objects incrementally
Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig3835606BD468F277AD2EECDA"
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3772.1324171874.27778.python-list@python.org>
Lines: 74
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:17422

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig3835606BD468F277AD2EECDA
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I'm interested in writing two programs, A and B, which communicate using
JSON. At a high level, A wants to transfer an array to B.

However, I would very much like to make it possible for A and B to run
in parallel, so my current plan is to have A output and B read a
*sequence* of JSON objects. In other words, instead of
  [ {"a": 0},  {"b":0}, {"c": 0} ]
it would just send
  {"a": 0}
  {"b": 0}
  {"c": 0}

I know about the raw_decode() object inside the json.JSONParser class,
and that gets me most of the way there there.

However, what I'm *not* sure about is the best way to get the input to
the raw_decode() function, which expects a "string or buffer":

  >>> d =3D json.JSONDecoder()
  >>> d.raw_decode(sys.stdin)
  Traceback (most recent call last):
    ...
    File "json\scanner.py", line 42, in iterscan
      match =3D self.scanner.scanner(string, idx).match
  TypeError: expected string or buffer

Now I'm not very familiar with the buffer and how it could be used (and
whether a file or stdin could be used as one in an incremental fashion),
but the best way I can come up with is the following:

1. Read a line of input
2. Try to decode it
3. If not, read another line, concatenate it to the end, and try again
4. etc.

That seems... inelegant at least.


Some other information:

* I'm looking for a 2.7 solution ideally
* I'd prefer not to use a different JSON library entirely
* As suggested, I *am* willing to wait for a newline to do processing
* However, I don't want to require exactly one object per line (and
  want to allow both multiple objects on one line and newlines within
  an object)

Evan


--------------enig3835606BD468F277AD2EECDA
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJO7UIRAAoJEAOzoR8eZTzgVfEIAIFMEKKZtuY+I14rKXNHS8fS
FxlRRS8To6Cqdx5EyGvMSFH3P2Hl7xF3dJq/ezV/mUs/u3xhqu4WyRhd4YVq3u8B
GauhKrLy+vLoHRQMRXszk8CFKVeUksCH8+zgiZrgeBby0ALwC8qzvXD3RP9fuPNs
hAPTcJpjXaAdsGgbYAoRUjhLZi9Ak547aexMNwg5TTStFI+fdOmHqD8vvBhQiU1k
OQM7lrXK3pM7IivSpuhfYB9GEFjYAjt1dbhcsSrNXj17rCNhxONpiPH0KndE6HcM
hSE9mNw0hfyRBe5VU738+e7pOCcADhHbTmsqZ8Od1mMgR51GPdwq2joVwbUuAFI=
=/X4G
-----END PGP SIGNATURE-----

--------------enig3835606BD468F277AD2EECDA--