Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #77505 > unrolled thread

Storing instances using jsonpickle

Back to article view | Back to comp.lang.python

Contents

  Storing instances using jsonpickle Josh English <Joshua.R.English@gmail.com> - 2014-09-03 13:32 -0700
    Re: Storing instances using jsonpickle Ned Batchelder <ned@nedbatchelder.com> - 2014-09-03 16:52 -0400
      Re: Storing instances using jsonpickle Josh English <Joshua.R.English@gmail.com> - 2014-09-03 15:30 -0700
        Re: Storing instances using jsonpickle MRAB <python@mrabarnett.plus.com> - 2014-09-04 00:39 +0100
          Re: Storing instances using jsonpickle Denis McMahon <denismfmcmahon@gmail.com> - 2014-09-04 01:11 +0000
        Re: Storing instances using jsonpickle Ned Batchelder <ned@nedbatchelder.com> - 2014-09-03 22:18 -0400
          Re: Storing instances using jsonpickle Sam Raker <sam.raker@gmail.com> - 2014-09-03 21:52 -0700
          Re: Storing instances using jsonpickle Josh English <Joshua.R.English@gmail.com> - 2014-09-15 12:30 -0700
        Re: Storing instances using jsonpickle Chris Angelico <rosuav@gmail.com> - 2014-09-04 15:17 +1000
          Re: Storing instances using jsonpickle Dan Sommers <dan@tombstonezero.net> - 2014-09-05 04:38 +0000
            Re: Storing instances using jsonpickle Chris Angelico <rosuav@gmail.com> - 2014-09-05 15:08 +1000
        Re: Storing instances using jsonpickle MRAB <python@mrabarnett.plus.com> - 2014-09-04 12:07 +0100
        Re: Storing instances using jsonpickle MRAB <python@mrabarnett.plus.com> - 2014-09-05 18:04 +0100
          Re: Storing instances using jsonpickle Marko Rauhamaa <marko@pacujo.net> - 2014-09-05 20:16 +0300
            Re: Storing instances using jsonpickle Ned Batchelder <ned@nedbatchelder.com> - 2014-09-05 13:30 -0400
              Re: Storing instances using jsonpickle Marko Rauhamaa <marko@pacujo.net> - 2014-09-05 21:04 +0300
                Re: Storing instances using jsonpickle Ned Batchelder <ned@nedbatchelder.com> - 2014-09-05 16:50 -0400
                  Re: Storing instances using jsonpickle Marko Rauhamaa <marko@pacujo.net> - 2014-09-05 23:57 +0300
            Re: Storing instances using jsonpickle Irmen de Jong <irmen.NOSPAM@xs4all.nl> - 2014-09-06 00:18 +0200
        Re: Storing instances using jsonpickle Chris Angelico <rosuav@gmail.com> - 2014-09-06 10:20 +1000
        Re: Storing instances using jsonpickle MRAB <python@mrabarnett.plus.com> - 2014-09-06 17:32 +0100
        Re: Storing instances using jsonpickle Ned Batchelder <ned@nedbatchelder.com> - 2014-09-06 12:56 -0400
        Re: Storing instances using jsonpickle Terry Reedy <tjreedy@udel.edu> - 2014-09-06 14:27 -0400

Page 1 of 2  [1] 2  Next page →

#77505 — Storing instances using jsonpickle

I am using jsonpickle to store instances of an object into separate data files.

If I make any changes to the original class definition of the object, when I recreate my stored instances, they are recreated using the original class definition, so any new attributes, methods, or properties, are lost.

I think this is happening because JSON is internalizing the class definition, ignoring the updates. Is this true? Is it possible to refresh JSON's knowledge of a class?

[toc] | [next] | [standalone]

#77506

On 9/3/14 4:32 PM, Josh English wrote:
> I am using jsonpickle to store instances of an object into separate data files.
>
> If I make any changes to the original class definition of the object, when I recreate my stored instances, they are recreated using the original class definition, so any new attributes, methods, or properties, are lost.
>
> I think this is happening because JSON is internalizing the class definition, ignoring the updates. Is this true? Is it possible to refresh JSON's knowledge of a class?
>

Pickle (and it looks like jsonpickle) does not invoke the class' 
__init__ method when it reconstitutes objects.  Your new __init__ is not 
being run, so new attributes it defines are not being created.

This is one of the reasons that people avoid pickle: being completely 
implicit is very handy, but also fragile.

-- 
Ned Batchelder, http://nedbatchelder.com

[toc] | [prev] | [next] | [standalone]

#77511

On Wednesday, September 3, 2014 1:53:23 PM UTC-7, Ned Batchelder wrote:

> Pickle (and it looks like jsonpickle) does not invoke the class' 
> __init__ method when it reconstitutes objects.  Your new __init__ is not 
> being run, so new attributes it defines are not being created.
>
> This is one of the reasons that people avoid pickle: being completely 
> implicit is very handy, but also fragile.
> 

I seem to remember having this exact same frustration when I used pickle and shelve 15 years ago. I had hoped to have another way around this. I spent over a decade trying to make an XML-based database work, in part because of this limitation.

Some days I get so frustrated I think the only data structure I should ever use is a dictionary.

I suppose to make this sort of thing work, I should look at creating custom json encoders and decoders.

Thanks,

Josh

[toc] | [prev] | [next] | [standalone]

#77513

On 2014-09-03 23:30, Josh English wrote:
> On Wednesday, September 3, 2014 1:53:23 PM UTC-7, Ned Batchelder
> wrote:
>
>> Pickle (and it looks like jsonpickle) does not invoke the class'
>> __init__ method when it reconstitutes objects.  Your new __init__
>> is not being run, so new attributes it defines are not being
>> created.
>>
>> This is one of the reasons that people avoid pickle: being
>> completely implicit is very handy, but also fragile.
>>
>
> I seem to remember having this exact same frustration when I used
> pickle and shelve 15 years ago. I had hoped to have another way
> around this. I spent over a decade trying to make an XML-based
> database work, in part because of this limitation.
>
> Some days I get so frustrated I think the only data structure I
> should ever use is a dictionary.
>
> I suppose to make this sort of thing work, I should look at creating
> custom json encoders and decoders.
>
I occasionally think about a superset of JSON, called, say, "pyson" ...
ah, name already taken! :-(

In summary, it would be something like this:

JSON supports int, float, string, None (as 'null'), list and dict (the 
key must be string).

It would add tuples, delimited by (...), which are not used otherwise 
(no expressions):

     () => ()
     (0, ) => (0)
     (0, 1) => (0, 1)

The key of a dict could also be int, float, or tuple.

It could support other classes, and could handle named arguments.

For example, sets:

To encode {0, 1, 2):

     Look in encoder dict for encoder function with class's name ('set') 
and call it, passing object.

     Encoder returns positional and keyword arguments: [0, 1, 2] and {}.

     Output name, followed by encoded arguments in parentheses.

     Encoder for set:

     def encode_set(obj):
         return list(obj), {}

To decode 'set(0, 1, 2)':

     Parse name: 'set'.

     Parse contents of parentheses: [0, 1, 2] and {}.

     Look in decoder dict for decoder function with given name ('set') 
and call it, passing arguments.

     Result would be {0, 1, 2}.

     Decoder for set:

     def decode_set(*args):
         return set(args)

pyson.dumps({0, 1, 2}, decoders={'set': decode_set}) would return 
'set(0, 1, 2)'.

pyson.loads('set(0, 1, 2)', encoders={'set': encode_set}) would return 
{0, 1, 2}.

[toc] | [prev] | [next] | [standalone]

#77514

On Thu, 04 Sep 2014 00:39:07 +0100, MRAB wrote:

> It would add tuples, delimited by (...), which are not used otherwise
> (no expressions):

I guess <> and () are both unused as delims by json at present.

I like the idea of other key types than string.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#77522

On 9/3/14 6:30 PM, Josh English wrote:
> On Wednesday, September 3, 2014 1:53:23 PM UTC-7, Ned Batchelder wrote:
>
>> Pickle (and it looks like jsonpickle) does not invoke the class'
>> __init__ method when it reconstitutes objects.  Your new __init__ is not
>> being run, so new attributes it defines are not being created.
>>
>> This is one of the reasons that people avoid pickle: being completely
>> implicit is very handy, but also fragile.
>>
>
> I seem to remember having this exact same frustration when I used pickle and shelve 15 years ago. I had hoped to have another way around this. I spent over a decade trying to make an XML-based database work, in part because of this limitation.
>
> Some days I get so frustrated I think the only data structure I should ever use is a dictionary.
>
> I suppose to make this sort of thing work, I should look at creating custom json encoders and decoders.
>

Typically, you need to decide explicitly on a serialized representation 
for your data.  Even if it's JSON, you need to decide what that JSON 
looks like.  Then you need to write code that converts the JSON-able 
data structure (dict of lists, whatever) into your object.  Often a 
version number is a good idea so that you have some chance of using old 
data as your code changes.

-- 
Ned Batchelder, http://nedbatchelder.com

[toc] | [prev] | [next] | [standalone]

#77533

1) There are, if you want to mess around with them, ways to make pickle "smarter" about class stuff: https://docs.python.org/2/library/pickle.html#pickling-and-unpickling-normal-class-instances . I've never worked with any of this stuff (and people don't seem to like pickle all that much), and I honestly think it might just be easier to serialize class _data_ and then either create a factory that reads in e.g. a json file and outputs class instances, or tweak __init__/__new__ to take the data as an input.
2) re: "pyson," I'm pretty sure there are serialization formats that let you do this kind of thing. I feel like I was _just_ reading about one somewhere, but scrolling back through my browser history for the last 10 days or so turns up nothing (it definitely included some kind of extension functionality, though...). You could possibly approximate something like that with e.g. JSON and some crafty en-/decoding, maybe like '"mySet: {"set": [0,1,2]}, "myTuple": {"tuple": [0,1,2]}, "myObj": {"ClassName": {"foo": "bar", "baz": "quux"}}" + a parser?

On Wednesday, September 3, 2014 10:19:07 PM UTC-4, Ned Batchelder wrote:
> On 9/3/14 6:30 PM, Josh English wrote:
> 
> > On Wednesday, September 3, 2014 1:53:23 PM UTC-7, Ned Batchelder wrote:
> 
> >
> 
> >> Pickle (and it looks like jsonpickle) does not invoke the class'
> 
> >> __init__ method when it reconstitutes objects.  Your new __init__ is not
> 
> >> being run, so new attributes it defines are not being created.
> 
> >>
> 
> >> This is one of the reasons that people avoid pickle: being completely
> 
> >> implicit is very handy, but also fragile.
> 
> >>
> 
> >
> 
> > I seem to remember having this exact same frustration when I used pickle and shelve 15 years ago. I had hoped to have another way around this. I spent over a decade trying to make an XML-based database work, in part because of this limitation.
> 
> >
> 
> > Some days I get so frustrated I think the only data structure I should ever use is a dictionary.
> 
> >
> 
> > I suppose to make this sort of thing work, I should look at creating custom json encoders and decoders.
> 
> >
> 
> 
> 
> Typically, you need to decide explicitly on a serialized representation 
> 
> for your data.  Even if it's JSON, you need to decide what that JSON 
> 
> looks like.  Then you need to write code that converts the JSON-able 
> 
> data structure (dict of lists, whatever) into your object.  Often a 
> 
> version number is a good idea so that you have some chance of using old 
> 
> data as your code changes.
> 
> 
> 
> -- 
> 
> Ned Batchelder, http://nedbatchelder.com

[toc] | [prev] | [next] | [standalone]

#77900

On Wednesday, September 3, 2014 7:19:07 PM UTC-7, Ned Batchelder wrote:

> Typically, you need to decide explicitly on a serialized representation 
> for your data.  Even if it's JSON, you need to decide what that JSON 
> looks like.  Then you need to write code that converts the JSON-able 
> data structure (dict of lists, whatever) into your object.  Often a 
> version number is a good idea so that you have some chance of using old 
> data as your code changes.
> 

Right now my cheap workaround is to define a function that saves the instances __dict__ using json, and to recreate the object, I create a new object using the __init__ method and cycle through the rest of the json keys and apply them to the new object using setattr.

It's a quick and dirty hack, but it seems to be working. I do have to make sure that everything that lands in the instance's __dict__ is serializable, but that's not so tough.

I need to add a version number, though. Good idea, that.


Josh

[toc] | [prev] | [next] | [standalone]

#77535

On Thu, Sep 4, 2014 at 9:39 AM, MRAB <python@mrabarnett.plus.com> wrote:
> I occasionally think about a superset of JSON, called, say, "pyson" ...
> ah, name already taken! :-(

While I'm somewhat sympathetic to the concept, there are some parts of
your description that I disagree with. Am I misreading something? Are
there typos in the description and I'm making something out of
nothing?

> It would add tuples, delimited by (...), which are not used otherwise (no
> expressions):
>
>     () => ()
>     (0, ) => (0)

This seems odd. Part of JSON's convenience is that it's a subset of
JavaScript syntax, so you can just plop a block of JSON into a REPL
and it'll decode correctly. With PyON (or whatever you call it), it'd
be nice to have the same correspondence; for a start, I would strongly
encourage the "trailing comma is permitted" rule (so [1,2,3,] is
equivalent to [1,2,3]), and then I'd have the default encoding for a
single-element tuple include that trailing comma. If (0) is a
one-element tuple, you end up with a subtle difference between a PyON
decode and the Python interpreter, which is likely to cause problems.
It might even be worth actually mandating (not just encouraging) that
one-element tuples have the trailing comma, just to prevent that.

> The key of a dict could also be int, float, or tuple.

Yes! Yes! DEFINITELY do this!! Ahem. Calm down a little, it's not that
outlandish an idea...

> It could support other classes, and could handle named arguments.
>
> For example, sets:
>
> To encode {0, 1, 2):

Do you mean {0, 1, 2} here? I'm hoping you aren't advocating a syntax
that mismatches bracket types. That's only going to cause confusion.

>     Look in encoder dict for encoder function with class's name ('set') and
> call it, passing object.
>
>     Encoder returns positional and keyword arguments: [0, 1, 2] and {}.
>
>     Output name, followed by encoded arguments in parentheses.
>
>     Encoder for set:
>
>     def encode_set(obj):
>         return list(obj), {}
>
> To decode 'set(0, 1, 2)':
>
>     Parse name: 'set'.
>
>     Parse contents of parentheses: [0, 1, 2] and {}.
>
>     Look in decoder dict for decoder function with given name ('set') and
> call it, passing arguments.
>
>     Result would be {0, 1, 2}.
>
>     Decoder for set:
>
>     def decode_set(*args):
>         return set(args)
>
> pyson.dumps({0, 1, 2}, decoders={'set': decode_set}) would return 'set(0, 1,
> 2)'.
>
> pyson.loads('set(0, 1, 2)', encoders={'set': encode_set}) would return {0,
> 1, 2}.

This seems very much overengineered. Keep it much more simple; adding
set notation is well and good, but keyword arguments aren't necessary
there, and I'm not seeing a tremendous use-case for them.

It's a pity Python has the collision of sets and dicts both using
braces. Pike went for two-character delimiters, which might be better
suited here; round brackets aren't used in JSON anywhere, so you can
afford to steal them:

{'this':'is', 'a':'dict'}
({'this','is','a','set'})

Empty sets would be an issue, though, as they'll be different in
Python and this format. But everything else would work fine. You have
a two-character delimiter in PyON, and superfluous parentheses around
set notation in Python.

(Sadly, this doesn't make it Pike-compatible, as Pike uses (<x,y,z>)
for sets. But it wouldn't have been anyway.)

ChrisA

[toc] | [prev] | [next] | [standalone]

#77575

On Thu, 04 Sep 2014 15:17:17 +1000, Chris Angelico wrote:

> On Thu, Sep 4, 2014 at 9:39 AM, MRAB <python@mrabarnett.plus.com> wrote:

>> The key of a dict could also be int, float, or tuple.
> 
> Yes! Yes! DEFINITELY do this!! Ahem. Calm down a little, it's not that
> outlandish an idea...

Using floats is a bad idea.  Consider this python code:

    dictionary = dict()
    original = get_some_floating_point_value()
    dictionary[original] = 'foo'
    string_version = str(original)  # this is where things head south
    duplicate = float(string_version)
    value = dictionary.get(duplicate)

Okay, so what is value?  Is it 'foo'?  Is it None?

(Yes, I can fix this.  If I *know* that original is a float, then I
could use original.hex() instead of str(original).)

HTH,
Dan

[toc] | [prev] | [next] | [standalone]

#77576

On Fri, Sep 5, 2014 at 2:38 PM, Dan Sommers <dan@tombstonezero.net> wrote:
> On Thu, 04 Sep 2014 15:17:17 +1000, Chris Angelico wrote:
>
>> On Thu, Sep 4, 2014 at 9:39 AM, MRAB <python@mrabarnett.plus.com> wrote:
>
>>> The key of a dict could also be int, float, or tuple.
>>
>> Yes! Yes! DEFINITELY do this!! Ahem. Calm down a little, it's not that
>> outlandish an idea...
>
> Using floats is a bad idea.  Consider this python code:
>
>     dictionary = dict()
>     original = get_some_floating_point_value()
>     dictionary[original] = 'foo'
>     string_version = str(original)  # this is where things head south
>     duplicate = float(string_version)
>     value = dictionary.get(duplicate)
>
> Okay, so what is value?  Is it 'foo'?  Is it None?
>
> (Yes, I can fix this.  If I *know* that original is a float, then I
> could use original.hex() instead of str(original).)

There are issues with direct lookups, yes, but you can safely and
easily iterate over that dictionary, and that's going to have plenty
of use.

ChrisA

[toc] | [prev] | [next] | [standalone]

#77541

On 2014-09-04 06:17, Chris Angelico wrote:> On Thu, Sep 4, 2014 at 9:39
AM, MRAB <python@mrabarnett.plus.com> wrote:
 >> I occasionally think about a superset of JSON, called, say, "pyson"
 >> ... ah, name already taken! :-(
 >
 > While I'm somewhat sympathetic to the concept, there are some parts
 > of your description that I disagree with. Am I misreading something?
 > Are there typos in the description and I'm making something out of
 > nothing?
 >
 >> It would add tuples, delimited by (...), which are not used
 >> otherwise (no expressions):
 >>
 >>     () => ()
 >>     (0, ) => (0)
 >
I'm thinking now that it might be better to have 'tuple()' and
'tuple(0)'.

 > This seems odd. Part of JSON's convenience is that it's a subset of
 > JavaScript syntax, so you can just plop a block of JSON into a REPL
 > and it'll decode correctly. With PyON (or whatever you call it), it'd
 > be nice to have the same correspondence; for a start, I would
 > strongly> encourage the "trailing comma is permitted" rule (so
 > [1,2,3,] is equivalent to [1,2,3]), and then I'd have the default
 > encoding for a single-element tuple include that trailing comma. If
 > (0) is a one-element tuple, you end up with a subtle difference
 > between a PyON decode and the Python interpreter, which is likely to
 > cause problems. It might even be worth actually mandating (not just
 > encouraging) that one-element tuples have the trailing comma, just to
 > prevent that.
 >
In that case, if you wanted to encode a (0, ), it would have to be
'tuple([0])', whereas 1+2j would have to be 'complex(i, 2)'. The
encoder would need to return [[0]] for the first case and [1, 2] for
the second.

 >> The key of a dict could also be int, float, or tuple.
 >
 > Yes! Yes! DEFINITELY do this!! Ahem. Calm down a little, it's not
 > that outlandish an idea...
 >
 >> It could support other classes, and could handle named arguments.
 >>
 >> For example, sets:
 >>
 >> To encode {0, 1, 2):
 >
 > Do you mean {0, 1, 2} here? I'm hoping you aren't advocating a syntax
 > that mismatches bracket types. That's only going to cause confusion.
 >
Yes, that's a typo.

 >>     Look in encoder dict for encoder function with class's name
 >> ('set') and call it, passing object.
 >>
 >>     Encoder returns positional and keyword arguments: [0, 1, 2] and
 >>     {}.
 >>
 >>     Output name, followed by encoded arguments in parentheses.
 >>
 >>     Encoder for set:
 >>
 >>     def encode_set(obj):
 >>         return list(obj), {}
 >>
 >> To decode 'set(0, 1, 2)':
 >>
 >>     Parse name: 'set'.
 >>
 >>     Parse contents of parentheses: [0, 1, 2] and {}.
 >>
 >>     Look in decoder dict for decoder function with given name
 >> ('set') and call it, passing arguments.
 >>
 >>     Result would be {0, 1, 2}.
 >>
 >>     Decoder for set:
 >>
 >>     def decode_set(*args):
 >>         return set(args)
 >>
 >> pyson.dumps({0, 1, 2}, decoders={'set': decode_set}) would return
 >> 'set(0, 1, 2)'.
 >>
 >> pyson.loads('set(0, 1, 2)', encoders={'set': encode_set}) would
 >> return {0, 1, 2}.
 >
 > This seems very much overengineered. Keep it much more simple; adding
 > set notation is well and good, but keyword arguments aren't necessary
 > there, and I'm not seeing a tremendous use-case for them.
 >
 > It's a pity Python has the collision of sets and dicts both using
 > braces. Pike went for two-character delimiters, which might be better
 > suited here; round brackets aren't used in JSON anywhere, so you can
 > afford to steal them:
 >
 > {'this':'is', 'a':'dict'}
 > ({'this','is','a','set'})
 >
 > Empty sets would be an issue, though, as they'll be different in
 > Python and this format. But everything else would work fine. You have
 > a two-character delimiter in PyON, and superfluous parentheses around
 > set notation in Python.
 >
 > (Sadly, this doesn't make it Pike-compatible, as Pike uses (<x,y,z>)
 > for sets. But it wouldn't have been anyway.)
 >
To encode {0, 1, 2}:

     Look in encoder dict for encoder function with class's name ('set')
     and call it, passing object.

     Encoder returns name and list of arguments: 'set' and [[0, 1, 2]].

     Output name, followed by encoded arguments in parentheses:
     'set([0, 1, 2])'.

     Encoder for set:

     def encode_set(obj):
         return 'set', [list(obj)]

To decode 'set([0, 1, 2])':

     Parse name: 'set'.

     Parse contents of parentheses as list: [[0, 1, 2]].

     Look in decoder dict for decoder function with given name ('set')
     and call it, passing arguments.

     Result would be {0, 1, 2}.

     Decoder for set:

     def decode_set(args):
         # Error-checking omitted.
         return set(args[0])

pyson.dumps({0, 1, 2}, decoders={'set': decode_set}) would return
'set([0, 1, 2])'.

pyson.loads('set([0, 1, 2])', encoders={'set': encode_set}) would
return {0, 1, 2}.

[toc] | [prev] | [next] | [standalone]

#77592

On 2014-09-04 06:17, Chris Angelico wrote:
> On Thu, Sep 4, 2014 at 9:39 AM, MRAB <python@mrabarnett.plus.com> wrote:
>> I occasionally think about a superset of JSON, called, say, "pyson" ...
>> ah, name already taken! :-(
>
> While I'm somewhat sympathetic to the concept, there are some parts of
> your description that I disagree with. Am I misreading something? Are
> there typos in the description and I'm making something out of
> nothing?
>
>> It would add tuples, delimited by (...), which are not used otherwise (no
>> expressions):
>>
>>     () => ()
>>     (0, ) => (0)
>
> This seems odd. Part of JSON's convenience is that it's a subset of
> JavaScript syntax, so you can just plop a block of JSON into a REPL
> and it'll decode correctly. With PyON (or whatever you call it), it'd
> be nice to have the same correspondence; for a start, I would strongly
> encourage the "trailing comma is permitted" rule (so [1,2,3,] is
> equivalent to [1,2,3]), and then I'd have the default encoding for a
> single-element tuple include that trailing comma. If (0) is a
> one-element tuple, you end up with a subtle difference between a PyON
> decode and the Python interpreter, which is likely to cause problems.
> It might even be worth actually mandating (not just encouraging) that
> one-element tuples have the trailing comma, just to prevent that.
>
[snip]

JSON has 'true' and 'false'.

Python has 'True' and 'False'.

Therefore, if you want it to be able to drop it into Python's REPL, it
won't be compatible with JSON anyway! (Well, not unless you define
'true' and 'false' first.)

[toc] | [prev] | [next] | [standalone]

#77598

MRAB <python@mrabarnett.plus.com>:

> Therefore, if you want it to be able to drop it into Python's REPL, it
> won't be compatible with JSON anyway! (Well, not unless you define
> 'true' and 'false' first.)

Sigh. I was so hopeful JSON would be great. Unfortunately, it flopped by
requiring the parser to heuristically support 5 encoding systems.

Thus, ast.literal_eval() is superior to anything JSON has to offer.


Marko

[toc] | [prev] | [next] | [standalone]

#77600

On 9/5/14 1:16 PM, Marko Rauhamaa wrote:
> MRAB <python@mrabarnett.plus.com>:
>
>> Therefore, if you want it to be able to drop it into Python's REPL, it
>> won't be compatible with JSON anyway! (Well, not unless you define
>> 'true' and 'false' first.)
>
> Sigh. I was so hopeful JSON would be great. Unfortunately, it flopped by
> requiring the parser to heuristically support 5 encoding systems.

I don't understand how JSON has flopped?  The parser may be a bit more 
complex (but not much, it isn't hard to examine the first few bytes), 
but you're using off-the-shelf parsers anyway, so why are you concerned 
by this?

>
> Thus, ast.literal_eval() is superior to anything JSON has to offer.
>
>
> Marko
>


-- 
Ned Batchelder, http://nedbatchelder.com

[toc] | [prev] | [next] | [standalone]

#77604

Ned Batchelder <ned@nedbatchelder.com>:

> I don't understand how JSON has flopped? The parser may be a bit more
> complex (but not much, it isn't hard to examine the first few bytes),
> but you're using off-the-shelf parsers anyway, so why are you
> concerned by this?

There are occasions where you need to take shortcuts. Say you need to
glean information from JSON with bash, grep, awk or straight C. If JSON
was fixed to UTF-8, that would be quite feasible. Being as it is, you
are bound to 3rd-party libraries.

That alone invites ad-hoc encodings. For example, I have run into
"asterisked" JSON, libraries that limit themselves to UTF-8.

Compare that with HTTP, SMTP, or even XML(!). They fix the encoding to
the bit. No need for completely unnecessary options.


Marko

[toc] | [prev] | [next] | [standalone]

#77616

On 9/5/14 2:04 PM, Marko Rauhamaa wrote:
> Ned Batchelder <ned@nedbatchelder.com>:
>
>> I don't understand how JSON has flopped? The parser may be a bit more
>> complex (but not much, it isn't hard to examine the first few bytes),
>> but you're using off-the-shelf parsers anyway, so why are you
>> concerned by this?
>
> There are occasions where you need to take shortcuts. Say you need to
> glean information from JSON with bash, grep, awk or straight C. If JSON
> was fixed to UTF-8, that would be quite feasible. Being as it is, you
> are bound to 3rd-party libraries.
>
> That alone invites ad-hoc encodings. For example, I have run into
> "asterisked" JSON, libraries that limit themselves to UTF-8.
>
> Compare that with HTTP, SMTP, or even XML(!). They fix the encoding to
> the bit. No need for completely unnecessary options.

I see what you mean about JSON, but you are mistaken about HTTP and XML. 
Neither of them dictates the encoding of the data, and both of them 
offer ways to declare the encoding.  This means XML parsers must be 
prepared for many different encodings.

>
>
> Marko
>


-- 
Ned Batchelder, http://nedbatchelder.com

[toc] | [prev] | [next] | [standalone]

#77617

Ned Batchelder <ned@nedbatchelder.com>:

> I see what you mean about JSON, but you are mistaken about HTTP and
> XML. Neither of them dictates the encoding of the data, and both of
> them offer ways to declare the encoding. This means XML parsers must
> be prepared for many different encodings.

You can rest assured that the line

   GET / HTTP/1.1<CR><LF>

is strictly 8-bit ASCII.

I was intentionally exaggerating about XML, but I have yet to encounter
non-UTF-8 XML in the wild.


Marko

[toc] | [prev] | [next] | [standalone]

#77622

On 5-9-2014 19:16, Marko Rauhamaa wrote:

> Thus, ast.literal_eval() is superior to anything JSON has to offer.

Incidentally,

I've made a serialization library based on Python's literal expressions.
It uses ast.literal_eval() to deserialize, and a bit of custom code to serialize Python
objects:   https://pypi.python.org/pypi/serpent


Irmen

[toc] | [prev] | [next] | [standalone]

#77627

On Sat, Sep 6, 2014 at 3:04 AM, MRAB <python@mrabarnett.plus.com> wrote:
> JSON has 'true' and 'false'.
>
> Python has 'True' and 'False'.
>
> Therefore, if you want it to be able to drop it into Python's REPL, it
> won't be compatible with JSON anyway! (Well, not unless you define
> 'true' and 'false' first.)

This is a new spec, so I guess the question is whether it's primarily
"JSON with some more features" or "subset of Python syntax in the same
way that JSON is a subset of JS". If it's the former, then yes, it'd
use "true" and "false", and you'd have to define them; but if the
latter, the spec would simply use "True" and "False". But being able
to guarantee that JSON decodes correctly with this parser (ie make it
a guaranteed superset of JSON) would be of value.

ChrisA

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web