Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #78476

JSON-encoding very long iterators

X-Received by 10.66.227.135 with SMTP id sa7mr37201945pac.19.1412039993864; Mon, 29 Sep 2014 18:19:53 -0700 (PDT)
X-Received by 10.50.132.66 with SMTP id os2mr36572igb.3.1412039993660; Mon, 29 Sep 2014 18:19:53 -0700 (PDT)
Path csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.glorb.com!uq10no768118igb.0!news-out.google.com!rp1ni8527igb.0!nntp.google.com!a13no5795194igq.0!postnews.google.com!glegroupsg2000goo.googlegroups.com!not-for-mail
Newsgroups comp.lang.python
Date Mon, 29 Sep 2014 18:19:53 -0700 (PDT)
Complaints-To groups-abuse@google.com
Injection-Info glegroupsg2000goo.googlegroups.com; posting-host=184.187.176.110; posting-account=wipd3QoAAACkqCR1nR6c6tC2496M_JVa
NNTP-Posting-Host 184.187.176.110
User-Agent G2/1.0
MIME-Version 1.0
Message-ID <4232c92a-101c-4a1b-89b2-e80d67ef7bc6@googlegroups.com> (permalink)
Subject JSON-encoding very long iterators
From alfred@54.org
Injection-Date Tue, 30 Sep 2014 01:19:53 +0000
Content-Type text/plain; charset=ISO-8859-1
Xref csiph.com comp.lang.python:78476

Show key headers only | View raw


I would like to add the ability to JSONEncode large iterators. Right now there is no way to do this without modifying the code.

The JSONEncoder.default() doc string suggests to do this:
        For example, to support arbitrary iterators, you could
        implement default like this::
            def default(self, o):
                try:
                    iterable = iter(o)
                except TypeError:
                    pass
                else:
                    return list(iterable)
                # Let the base class default method raise the TypeError
                return JSONEncoder.default(self, o)

but this method requires the whole serialized object to fit in memory and it's a good chance that your iterator is an iterator to save on memory in the first place.

By changing the code to accept iterators it is then possible to stream json as I did here:
http://stackoverflow.com/a/26094558/289240

This would ideal if it were included in the standard library. Is there any reason why it shouldn't be?

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread


Thread

JSON-encoding very long iterators alfred@54.org - 2014-09-29 18:19 -0700
  Re: JSON-encoding very long iterators Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-29 19:58 -0600
    Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 03:13 -0700
      Re: JSON-encoding very long iterators Chris Angelico <rosuav@gmail.com> - 2014-10-01 23:06 +1000
        Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 15:01 -0700
          Re: JSON-encoding very long iterators Chris Angelico <rosuav@gmail.com> - 2014-10-02 08:54 +1000
            Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 23:05 -0700
              Re: JSON-encoding very long iterators Chris Angelico <rosuav@gmail.com> - 2014-10-02 16:33 +1000
                Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 23:49 -0700

csiph-web