Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path: <miki.tebeka@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.002
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'main()': 0.05; 'sys': 0.05; 'json': 0.07; '__name__': 0.09; 'reply- to:addr:comp.lang.python': 0.09; 'sockets': 0.09; 'to:addr:comp.lang.python': 0.09; 'def': 0.13; "'__main__':": 0.16; '(note': 0.16; 'from:addr:miki.tebeka': 0.16; 'from:name:miki tebeka': 0.16; 'main():': 0.16; 'obj,': 0.16; 'valueerror': 0.16; 'cc:addr:python-list': 0.16; 'cc:no real name:2**0': 0.20; 'header:In-Reply-To:1': 0.22; 'cc:2**0': 0.24; 'import': 0.27; 'yield': 0.29; 'cc:addr:python.org': 0.29; 'example': 0.29; 'line:': 0.30; 'sleep': 0.30; 'subject:skip:i 10': 0.30; 'break': 0.32; 'header:User-Agent:1': 0.33; 'object': 0.33; 'probably': 0.34; 'succeeded': 0.34; 'try:': 0.34; 'except': 0.37; 'received:google.com': 0.37; 'received:209.85': 0.38; 'option': 0.39; 'received:209': 0.40; 'might': 0.40; 'below': 0.63; 'header:Reply-To:1': 0.71; 'reply-to:no real name:2**0': 0.72; 'reply-to:addr:googlegroups.com': 0.74; 'subject:stream': 0.84
Newsgroups: comp.lang.python
Date: Mon, 19 Dec 2011 15:18:19 -0800 (PST)
In-Reply-To: <mailman.3772.1324171874.27778.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=207.171.18.139; posting-account=uo-8fwoAAACKsFzFX78JHudx1V7WDXZ0
References: <mailman.3772.1324171874.27778.python-list@python.org>
User-Agent: G2/1.0
X-Google-Web-Client: true
MIME-Version: 1.0
Subject: Re: Parsing stream of JSON objects incrementally
From: Miki Tebeka <miki.tebeka@gmail.com>
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1
Cc: python-list@python.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: comp.lang.python@googlegroups.com
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Message-ID: <mailman.3832.1324336701.27778.python-list@python.org>
Lines: 30
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1324336701 news.xs4all.nl 6873 [2001:888:2000:d::a6]:46938
X-Complaints-To: abuse@xs4all.nl
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:17529

You probably need to accumulate a buffer and try to decode it, when succeeded return the object and read more. See example below (note that for sockets select might be a better option for reading data).

import json
from time import sleep

def json_decoder(fo):
    buff = ''
    decode = json.JSONDecoder().raw_decode
    while True:
        line = fo.readline()
        if not line:
            break
        buff += line
        print('BUFF: {}'.format(buff))
        try:
            obj, i = decode(buff)
            buff = buff[i:].lstrip()
            yield obj
        except ValueError as e:
            print('ERR: {}'.format(e))
            sleep(0.01) # select will probably be a better option :)


def main():
    import sys
    for obj in json_decoder(sys.stdin):
        print(obj)

if __name__ == '__main__':
    main()