Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #78477

Re: JSON-encoding very long iterators

Show all headers | View raw

On Mon, Sep 29, 2014 at 7:19 PM,  <alfred@54.org> wrote:
> I would like to add the ability to JSONEncode large iterators. Right now there is no way to do this without modifying the code.
>
> The JSONEncoder.default() doc string suggests to do this:
>         For example, to support arbitrary iterators, you could
>         implement default like this::
>             def default(self, o):
>                 try:
>                     iterable = iter(o)
>                 except TypeError:
>                     pass
>                 else:
>                     return list(iterable)
>                 # Let the base class default method raise the TypeError
>                 return JSONEncoder.default(self, o)
>
> but this method requires the whole serialized object to fit in memory and it's a good chance that your iterator is an iterator to save on memory in the first place.
>
> By changing the code to accept iterators it is then possible to stream json as I did here:
> http://stackoverflow.com/a/26094558/289240
>
> This would ideal if it were included in the standard library. Is there any reason why it shouldn't be?

This would cause things that aren't lists to be encoded as lists.
Sometimes that may be desirable, but in general if e.g. a file object
sneaks its way into your JSON encode call, it is more likely correct
to raise an error than to silently encode the file as if it were a
list of strings.  So it should not be the default behavior. That said,
it sounds like it could be made easier to enable streaming from
iterators as an option for those cases where it's desired.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

JSON-encoding very long iterators alfred@54.org - 2014-09-29 18:19 -0700
  Re: JSON-encoding very long iterators Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-29 19:58 -0600
    Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 03:13 -0700
      Re: JSON-encoding very long iterators Chris Angelico <rosuav@gmail.com> - 2014-10-01 23:06 +1000
        Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 15:01 -0700
          Re: JSON-encoding very long iterators Chris Angelico <rosuav@gmail.com> - 2014-10-02 08:54 +1000
            Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 23:05 -0700
              Re: JSON-encoding very long iterators Chris Angelico <rosuav@gmail.com> - 2014-10-02 16:33 +1000
                Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 23:49 -0700

csiph-web