Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #99695 > unrolled thread

New JSON encoding method proposal for custom objects

Back to article view | Back to comp.lang.python

Contents

  New JSON encoding method proposal for custom objects cescus92@gmail.com - 2015-11-29 07:05 -0800
    Re: New JSON encoding method proposal for custom objects Peter Otten <__peter__@web.de> - 2015-11-29 16:46 +0100
    Re: New JSON encoding method proposal for custom objects Denis McMahon <denismfmcmahon@gmail.com> - 2015-11-29 20:40 +0000
      Re: New JSON encoding method proposal for custom objects Marco Kaulea <marco.kaulea@gmail.com> - 2015-11-30 09:42 +0100
        Re: New JSON encoding method proposal for custom objects cescus92@gmail.com - 2015-11-30 04:35 -0800
          Re: New JSON encoding method proposal for custom objects Burak Arslan <burak.arslan@arskom.com.tr> - 2015-12-01 13:20 +0200
            Re: New JSON encoding method proposal for custom objects cescus92@gmail.com - 2015-12-01 04:08 -0800
          Re: New JSON encoding method proposal for custom objects Peter Otten <__peter__@web.de> - 2015-12-01 13:17 +0100
            Re: New JSON encoding method proposal for custom objects cescus92@gmail.com - 2015-12-01 04:24 -0800

#99695 — New JSON encoding method proposal for custom objects

Hello everyone!

I'm writing here since I've read on the Pyhton's documentation that this is the most common path that a new proposal should follow. I'd be really glad if this proposal could become a PEP if I see a good response from the community or, in the worst case, understand why this proposal is not good enough.

In this day I stumbled upon a very simple task: I had a list of instances of a custom class and I had to convert i into a JSON.

Actually, from what I've discovered, there are 2 ways to get the JSON out of a custom non serializable instance:
- defining a new method ( e.g. to_json() ) inside the class that converts the item manually (quicker)
- a custom encoder/decoder that goes to extend the ones from the json lib (cleander, I think)

Since I thought it was the most cost-effective and I was in hurry, I took the first way that drove me to a "problem": if I wanted to get the JSON out from my list, it obviously would have given me the "non serializable object" error since the json lib didn't know how to transform it into JSON!

I was astonished that Python was requiring me such ugly ways to accomplish this simple task! JSON WebServices are becoming more and more popular and Python _must_ keep the pace of time! And here it is my proposal to make this task easier in the best Pythonic way :)

I propose that every custom class that wants to allow its instances to have a JSON representation could implement a simple method, __json__().

The way how it works it very straightforward: every time json lib is required to convert an object that doesn't know, it looks for the __json__() method to learn how to treat the object. The duty of this method should be to reduce the complexity of an object instance into an elementary data type (int, str, list, dict, etc..).

Often the process to get the JSON out of an object is a one-way need, in this way it would become even simpler to do it in the most flawless way so the programmer should no more care to implement some custom logic to handle quickly these situations.

Let me give you a very small example of what a __json__ method could be like:

class CustomObj():
  def __init__(self, p1, p2):
    self.p1 = p1
    self.p2 = p2

  def __json__(self):
    return {
      'p1': self.p1,
      'p2': self.p2
    }

Once more, the job of __json__ should simply be the one of make our object json-readable :)

What do you think of it? Do you think it's good? If not, why?

Do you have any guidance to make this idea advance?

Do you advice me to write down a preliminary PEP or to make directly a pull request to the Python's source code?

Thank you for you're time!
I'm waiting for your feedback :)
Have a good Sunday!

Francesco

[toc] | [next] | [standalone]

#99696

cescus92@gmail.com wrote:

> Hello everyone!
> 
> I'm writing here since I've read on the Pyhton's documentation that this
> is the most common path that a new proposal should follow. I'd be really
> glad if this proposal could become a PEP if I see a good response from the
> community or, in the worst case, understand why this proposal is not good
> enough.
> 
> In this day I stumbled upon a very simple task: I had a list of instances
> of a custom class and I had to convert i into a JSON.
> 
> Actually, from what I've discovered, there are 2 ways to get the JSON out
> of a custom non serializable instance: - defining a new method ( e.g.
> to_json() ) inside the class that converts the item manually (quicker) - a
> custom encoder/decoder that goes to extend the ones from the json lib
> (cleander, I think)
> 
> Since I thought it was the most cost-effective and I was in hurry, I took
> the first way that drove me to a "problem": if I wanted to get the JSON
> out from my list, it obviously would have given me the "non serializable
> object" error since the json lib didn't know how to transform it into
> JSON!
> 
> I was astonished that Python was requiring me such ugly ways to accomplish
> this simple task! JSON WebServices are becoming more and more popular and
> Python _must_ keep the pace of time! And here it is my proposal to make
> this task easier in the best Pythonic way :)
> 
> I propose that every custom class that wants to allow its instances to
> have a JSON representation could implement a simple method, __json__().
> 
> The way how it works it very straightforward: every time json lib is
> required to convert an object that doesn't know, it looks for the
> __json__() method to learn how to treat the object. The duty of this
> method should be to reduce the complexity of an object instance into an
> elementary data type (int, str, list, dict, etc..).
> 
> Often the process to get the JSON out of an object is a one-way need, in
> this way it would become even simpler to do it in the most flawless way so
> the programmer should no more care to implement some custom logic to
> handle quickly these situations.
> 
> Let me give you a very small example of what a __json__ method could be
> like:
> 
> class CustomObj():
>   def __init__(self, p1, p2):
>     self.p1 = p1
>     self.p2 = p2
> 
>   def __json__(self):
>     return {
>       'p1': self.p1,
>       'p2': self.p2
>     }
> 
> Once more, the job of __json__ should simply be the one of make our object
> json-readable :)
> 
> What do you think of it? Do you think it's good? If not, why?

I'm not sure if that is needed often enough so that adding yet another 
__dunder__ method is worthwhile, particularly as the current mechanism 
already makes an implementation of your idea a matter of a few lines:

>>> import json
>>> class Encoder(json.JSONEncoder):
...   def default(self, obj):
...     try: encode = obj.__json__
...     except AttributeError: return super().default(obj)
...     return encode()
... 
>>> class Foo:
...     def __json__(self): return dict(bar=1, baz=2)
... 
>>> json.dumps({1:2, 3:Foo()}, cls=Encoder)
'{"1": 2, "3": {"baz": 2, "bar": 1}}'

Personally I would find it more interesting if there were a unified 
mechanism for json and pickle. At the very least you should come up with a 
decoding mechanism.

> Do you have any guidance to make this idea advance?

Go to python-ideas for a lengthy discussion ;)

> Do you advice me to write down a preliminary PEP or to make directly a
> pull request to the Python's source code?
> 
> Thank you for you're time!
> I'm waiting for your feedback :)
> Have a good Sunday!
> 
> Francesco

[toc] | [prev] | [next] | [standalone]

#99699

On Sun, 29 Nov 2015 07:05:30 -0800, cescus92 wrote:

> In this day I stumbled upon a very simple task: I had a list of
> instances of a custom class and I had to convert i into a JSON.

That's all well and good, but firstly you need to both dumps and loads to 
work properly with json, and secondly there's no mechanism in json that 
tells you what class of object you have.

So you may have a __json_dumps__ that will dump your object out to a json 
string representation, but then how does json.loads recognise that it's 
loading your object to call your object's __json_loads__?

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#99733

Hi,

I think his goal is not to load the data back into python, but to use
it to provide some
kind of REST-Style interface. An example would be, if you have a class
instance of
a record of a movie, that it could be simply encoded and send over the
wire to display.
Without necessarily recreating the class on the other end.
This would prevent any security risks from the deserialization.

On Sun, Nov 29, 2015 at 9:40 PM, Denis McMahon <denismfmcmahon@gmail.com> wrote:
> That's all well and good, but firstly you need to both dumps and loads to
> work properly with json, and secondly there's no mechanism in json that
> tells you what class of object you have.

But I am not sure how useful that PEP/Change would be, assuming the
code in Peter's response
works.

- Marco Kaulea

[toc] | [prev] | [next] | [standalone]

#99735

Hello everyone and thank you for your interest!

The Peter's code is very similar to what I think the default JSON encoder should be.

The advantage of the method that I propose is that you should not care anymore about which encoder you're going to use even in case of different class instances. Imagine if you could just do

  json.dumps({[1,2,3], Obj(), [DifferentObj()] })

Probably it doesn't worth a PEP but it's a very small change that can be great for making painless REST interfaces as Marco underlined (it was exactly what I was meaning, thanks)!

Denis, the way back is not considered in my proposal. But using the standard json.loads() could be nice enough! :)

> Go to python-ideas for a lengthy discussion ;)
Where can I find it?

Thank you again, I hope now my idea is clearer!

[toc] | [prev] | [next] | [standalone]

#99791

hey,

On 11/30/15 14:35, cescus92@gmail.com wrote:
>
> Hello everyone and thank you for your interest!
>
> The Peter's code is very similar to what I think the default JSON encoder should be.
>
> The advantage of the method that I propose is that you should not care anymore about which encoder you're going to use even in case of different class instances. Imagine if you could just do
>
>   json.dumps({[1,2,3], Obj(), [DifferentObj()] })
>
>

FWIW, Spyne can to the exact same thing -- i.e. serialize an object
given its definition to whatever format you want. (currently xml, json,
yaml and msgpack are supported). Here's a json example:

>>> from spyne import *
>>> from spyne.util.dictdoc import get_object_as_json, get_json_as_object
>>> get_object_as_json([1,2,3], Array(Integer))
'[1, 2, 3]'
>>>
>>> from datetime import datetime
>>>
>>> class SomeObject(ComplexModel):
...     s = Unicode
...     i = Integer
...     dt = DateTime
...
>>> get_object_as_json(SomeObject(s='str', i=42, dt=datetime.now()),
complex_as=list)
'[42, "str", "2015-12-01T12:57:23.751631"]'
>>>
>>> get_json_as_object('[42, "str", "2015-12-01T12:55:21.777108"]',
SomeObject, complex_as=list)
SomeObject(i=42, s=u'str', dt=datetime.datetime(2015, 12, 1, 12, 55, 21,
777108))
>>>
>>>

More info: http://spyne.io

Best,
Burak

PS: The astute reader will notice that element order in SomeObject could
be totally random.

* In Python 3, we solve that by returning an odict() in the __prepare__
of the ComplexModel metaclass.
* In Python 2, we solve that by somehow getting hold of AST of the class
definition and deducing the order from there. Yes you read that right! I
know, it's horrible! Don't worry, it's turned off by default. We
recommend the workaround in [1] for Python 2. See [2] and [3] to see how
we integrated it.

[1]: http://spyne.io/docs/2.10/manual/03_types.html#complex
[2]: https://github.com/arskom/spyne/pull/313
[3]:
https://github.com/arskom/spyne/blob/2768c7ff0b5f58aa0e47859fcd69e5bb7aa31aba/spyne/util/meta.py

[toc] | [prev] | [next] | [standalone]

#99794

Hello Burak, thank you for the reply.

> FWIW, Spyne can to the exact same thing -- i.e. serialize an object
> given its definition to whatever format you want. (currently xml, json,
> yaml and msgpack are supported).
My aim is to propose a built-in method to accomplish this common and simple task that, IMHO, shouln't require the use of extra code for legibility's sake.

Anyway the whole Spyne framework is very interesting, thank you for having shared that!
Probably it's going to be useful for a project of mine :)

Have a nice day,
Francesco

[toc] | [prev] | [next] | [standalone]

#99795

cescus92@gmail.com wrote:

>> Go to python-ideas for a lengthy discussion ;)

> Where can I find it?

At the same place where you already found python-list, or here:
 
https://mail.python.org/mailman/listinfo/python-ideas

[toc] | [prev] | [next] | [standalone]

#99796

> At the same place where you already found python-list, or here:
>  
> https://mail.python.org/mailman/listinfo/python-ideas

Yeah, I've found it after a simple search.. I thought it was like an hidden place for just some python's guru lol!

Thank you! :)

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web