Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #78476 > unrolled thread

JSON-encoding very long iterators

Back to article view | Back to comp.lang.python

Contents

  JSON-encoding very long iterators alfred@54.org - 2014-09-29 18:19 -0700
    Re: JSON-encoding very long iterators Ian Kelly <ian.g.kelly@gmail.com> - 2014-09-29 19:58 -0600
      Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 03:13 -0700
        Re: JSON-encoding very long iterators Chris Angelico <rosuav@gmail.com> - 2014-10-01 23:06 +1000
          Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 15:01 -0700
            Re: JSON-encoding very long iterators Chris Angelico <rosuav@gmail.com> - 2014-10-02 08:54 +1000
              Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 23:05 -0700
                Re: JSON-encoding very long iterators Chris Angelico <rosuav@gmail.com> - 2014-10-02 16:33 +1000
                  Re: JSON-encoding very long iterators Alfred Morgan <alfred@54.org> - 2014-10-01 23:49 -0700

#78476 — JSON-encoding very long iterators

I would like to add the ability to JSONEncode large iterators. Right now there is no way to do this without modifying the code.

The JSONEncoder.default() doc string suggests to do this:
        For example, to support arbitrary iterators, you could
        implement default like this::
            def default(self, o):
                try:
                    iterable = iter(o)
                except TypeError:
                    pass
                else:
                    return list(iterable)
                # Let the base class default method raise the TypeError
                return JSONEncoder.default(self, o)

but this method requires the whole serialized object to fit in memory and it's a good chance that your iterator is an iterator to save on memory in the first place.

By changing the code to accept iterators it is then possible to stream json as I did here:
http://stackoverflow.com/a/26094558/289240

This would ideal if it were included in the standard library. Is there any reason why it shouldn't be?

[toc] | [next] | [standalone]

#78477

On Mon, Sep 29, 2014 at 7:19 PM,  <alfred@54.org> wrote:
> I would like to add the ability to JSONEncode large iterators. Right now there is no way to do this without modifying the code.
>
> The JSONEncoder.default() doc string suggests to do this:
>         For example, to support arbitrary iterators, you could
>         implement default like this::
>             def default(self, o):
>                 try:
>                     iterable = iter(o)
>                 except TypeError:
>                     pass
>                 else:
>                     return list(iterable)
>                 # Let the base class default method raise the TypeError
>                 return JSONEncoder.default(self, o)
>
> but this method requires the whole serialized object to fit in memory and it's a good chance that your iterator is an iterator to save on memory in the first place.
>
> By changing the code to accept iterators it is then possible to stream json as I did here:
> http://stackoverflow.com/a/26094558/289240
>
> This would ideal if it were included in the standard library. Is there any reason why it shouldn't be?

This would cause things that aren't lists to be encoded as lists.
Sometimes that may be desirable, but in general if e.g. a file object
sneaks its way into your JSON encode call, it is more likely correct
to raise an error than to silently encode the file as if it were a
list of strings.  So it should not be the default behavior. That said,
it sounds like it could be made easier to enable streaming from
iterators as an option for those cases where it's desired.

[toc] | [prev] | [next] | [standalone]

#78533

On Monday, September 29, 2014 7:10:18 PM UTC-7, Ian wrote:
> This would cause things that aren't lists to be encoded as lists.
> Sometimes that may be desirable, but in general if e.g. a file object
> sneaks its way into your JSON encode call, it is more likely correct
> to raise an error than to silently encode the file as if it were a
> list of strings.  So it should not be the default behavior. That said,
> it sounds like it could be made easier to enable streaming from
> iterators as an option for those cases where it's desired.

I added a stream flag (off by default) and also added file streaming (thanks for the idea).

https://github.com/Zectbumo/cpython/compare/2.7

What do you think now?

[toc] | [prev] | [next] | [standalone]

#78547

On Wed, Oct 1, 2014 at 8:13 PM, Alfred Morgan <alfred@54.org> wrote:
> I added a stream flag (off by default) and also added file streaming (thanks for the idea).
>
> https://github.com/Zectbumo/cpython/compare/2.7
>
> What do you think now?

I think that you're adding features to Python 2.7, which isn't getting
new features. That won't be merged into trunk. Does your patch apply
to 3.x?

ChrisA

[toc] | [prev] | [next] | [standalone]

#78565

On Wednesday, October 1, 2014 6:07:23 AM UTC-7, Chris Angelico wrote:
> On Wed, Oct 1, 2014 at 8:13 PM, Alfred Morgan wrote:
> > What do you think now?
> 
> I think that you're adding features to Python 2.7, which isn't getting
> new features. That won't be merged into trunk. Does your patch apply
> to 3.x?
> 
> ChrisA

Thanks Chris, Yes I made changes to 2.7 because I'm not familiar with Python3 yet. Once I get some feedback I was going to see if someone was interested in porting it. If not I will learn and do it myself.
The patch will not work on the 3.x code but the code is similar enough to easily port over.

[toc] | [prev] | [next] | [standalone]

#78570

On Thu, Oct 2, 2014 at 8:01 AM, Alfred Morgan <alfred@54.org> wrote:
> On Wednesday, October 1, 2014 6:07:23 AM UTC-7, Chris Angelico wrote:
>> On Wed, Oct 1, 2014 at 8:13 PM, Alfred Morgan wrote:
>> > What do you think now?
>>
>> I think that you're adding features to Python 2.7, which isn't getting
>> new features. That won't be merged into trunk. Does your patch apply
>> to 3.x?
>>
>> ChrisA
>
> Thanks Chris, Yes I made changes to 2.7 because I'm not familiar with Python3 yet. Once I get some feedback I was going to see if someone was interested in porting it. If not I will learn and do it myself.
> The patch will not work on the 3.x code but the code is similar enough to easily port over.

Okay. At some point, you'll have to port your patch to the latest
codebase, though, as the core devs won't want to do that work
themselves.

I recommend getting familiar enough with Python 3 to be able to see if
the patch at least makes sense. (I expect it probably will.) Try out a
vanilla 3.4 or master (3.5ish), then apply your patch, and see if the
improvement is indeed visible. Assuming it is (again, I expect it will
be; JSON isn't materially different in Python 3), you can post a 3.x
patch here, and you'll have rather more interest, I think - certainly
people like me won't be applying your 2.7 patch, as that's not
something that's ever going to "actually happen".

But due to the changes of str/unicode to str/bytes in Py3, you will
find that your patch requires more than just simple tweaks. It won't
be as simple as "apply the patch and fix a merge conflict"; you'll
have to rethink your changes and apply them to the different codebase.
Unfortunately, that's just what happens when you start development on
2.7; there are differences in the code, and there's no getting around
them. It'll be much easier for you if you get familiar with Py3 and do
development there. :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#78579

On Wednesday, October 1, 2014 3:55:19 PM UTC-7, Chris Angelico wrote:
> At some point, you'll have to port your patch to the latest codebase

Okay, done.

https://github.com/Zectbumo/cpython/compare/master

Iterators for JSON is now Python 3 ready.

[toc] | [prev] | [next] | [standalone]

#78580

On Thu, Oct 2, 2014 at 4:05 PM, Alfred Morgan <alfred@54.org> wrote:
> On Wednesday, October 1, 2014 3:55:19 PM UTC-7, Chris Angelico wrote:
>> At some point, you'll have to port your patch to the latest codebase
>
> Okay, done.
>
> https://github.com/Zectbumo/cpython/compare/master
>
> Iterators for JSON is now Python 3 ready.

Cool! I didn't think it'd be that easy, actually (all those references
to 'basestring' are actually still there in the current source...
whodathunkit). Now you can raise a tracker issue on bugs.python.org,
or suggest it on python-ideas, and see whether other people get
excited by the enhancement!

ChrisA

[toc] | [prev] | [next] | [standalone]

#78581

Excellent, thank you.

http://bugs.python.org/issue14573

-alfred

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web