Path: csiph.com!usenet.pasdenom.info!nntpfeed.proxad.net!news.muarf.org!news.roellig-ltd.de!open-news-network.org!border2.nntp.ams1.giganews.com!border1.nntp.ams1.giganews.com!nntp.giganews.com!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <55778208$0$2899$e4fe514c@news2.news.xs4all.nl>
References: <ml7a2p$hp$1@ger.gmane.org> <CAMw+j7K7JmRPo3PnzTE2rZEstdZRF085+VA=v1ieMMHk2Gp6mQ@mail.gmail.com> <CABicbJ+e02OKYi48kbjFFutyLUGgkEQdiPZqiB657AO+QKu11Q@mail.gmail.com> <mailman.325.1433891190.13271.python-list@python.org> <55778208$0$2899$e4fe514c@news2.news.xs4all.nl>
From: Devin Jeanpierre <jeanpierreda@gmail.com>
Date: Tue, 9 Jun 2015 17:47:53 -0700
Subject: Re: enhancement request: make py3 read/write py2 pickle format
To: Irmen de Jong <irmen.NOSPAM@xs4all.nl>
Cc: "comp.lang.python" <python-list@python.org>
Content-Type: text/plain; charset=UTF-8
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.328.1433897321.13271.python-list@python.org>
Lines: 50
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:92381

Passing around data that can be put into ast.literal_eval is
synonymous with passing around data taht can be put into eval. It
sounds like a trap.

Other points against JSON / etc.: the lack of schema makes it easier
to stuff anything in there (not as easily as pickle, mind), and by
returning a plain dict, it becomes easier to require a field than to
allow a field to be missing, which is bad for robustness and bad for
data format migrations. (Protobuf (v3) has schemas and gives every
field a default value.)

For human readable serialized data, text format protocol buffers are
seriously underrated. (Relatedly: underdocumented, too.)

/me lifts head out of kool-aid and gasps for air

-- Devin

On Tue, Jun 9, 2015 at 5:17 PM, Irmen de Jong <irmen.NOSPAM@xs4all.nl> wrote:
> On 10-6-2015 1:06, Chris Angelico wrote:
>> On Wed, Jun 10, 2015 at 6:07 AM, Devin Jeanpierre
>> <jeanpierreda@gmail.com> wrote:
>>> There's a lot of subtle issues with pickle compatibility. e.g.
>>> old-style vs new-style classes. It's kinda hard and it's better to
>>> give up. I definitely agree it's better to use something else instead.
>>> For example, we switched to using protocol buffers, which have much
>>> better compatibility properties and are a bit more testable to boot
>>> (since text format protobufs are always output in a canonical (sorted)
>>> form.)
>>
>> Or use JSON, if your data fits within that structure. It's easy to
>> read and write, it's human-readable, and it's safe (no chance of
>> arbitrary code execution). Forcing yourself to use a format that can
>> basically be processed by ast.literal_eval() is a good discipline -
>> means you don't accidentally save/load too much.
>>
>> ChrisA
>>
>
> I made a specialized serializer for this, which is more expressive than JSON. It outputs
> python literal expressions that can be directly parsed by ast.literal_eval(). You can
> find it on pypi (https://pypi.python.org/pypi/serpent).  It's the default serializer of
> Pyro, and it includes a Java and .NET version as well as an added bonus.
>
>
> Irmen
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list