Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90221 > unrolled thread

Seralization

Back to article view | Back to comp.lang.python

Contents

  Seralization Cecil Westerhof <Cecil@decebal.nl> - 2015-05-09 10:37 +0200
    Re: Seralization Chris Angelico <rosuav@gmail.com> - 2015-05-09 19:16 +1000
      Re: Seralization Cecil Westerhof <Cecil@decebal.nl> - 2015-05-09 13:38 +0200
    Re: Seralization Cecil Westerhof <Cecil@decebal.nl> - 2015-05-09 15:04 +0200

#90221 — Seralization

To make serialization a bit easier I made a few functions to get, save
and convert between the different types. As I see it pickle and json
are probably used the most. I also have a get and save for marshal.
But no conversion to marshal, because in principle you should not use
it, so a conversion to it is not useful. I did define conversion from.

Are there other seralizations that is handy to take care of?

I understood that because the differences between platforms it is
better to do the file operatins in binary mode. Is that true?


The code:
    def get_json(json_file):
        with open(json_file, 'rb') as in_f:
            return json.load(in_f)

    def get_marshal(marshal_file):
        with open(marshal_file, 'rb') as in_f:
            return marshal.load(in_f)

    def get_pickle(pickle_file):
        with open(pickle_file, 'rb') as in_f:
            return pickle.load(in_f)

    def save_json(data, json_file):
        with open(json_file, 'wb') as out_f:
            json.dump(data, out_f)

    def save_marshal(data, marshal_file):
        with open(marshal_file, 'wb') as out_f:
            marshal.dump(data, out_f)

    def save_pickle(data, pickle_file):
        with open(pickle_file, 'wb') as out_f:
            pickle.dump(data, out_f)

    def marshal_to_pickle(marshal_file, pickle_file):
        data_in = get_marshal(marshal_file)
        save_pickle(data_in, pickle_file)
        data_out = get_pickle(pickle_file)
        if data_in != data_out:
            raise SerializationError('Serialization from {0} to {1} not succesfull'.
                                     format(marshal_file, pickle_file))

    def marshal_to_json(marshal_file, json_file):
        data_in = get_marshal(marshal_file)
        save_json(data_in, json_file)
        data_out = get_json(json_file)
        if data_in != data_out:
            raise SerializationError('Serialization from {0} to {1} not succesfull'.
                                     format(marshal_file, json_file))

    def pickle_to_json(pickle_file, json_file):
        data_in = get_pickle(pickle_file)
        save_json(data_in, json_file)
        data_out = get_json(json_file)
        if data_in != data_out:
            raise SerializationError('Serialization from {0} to {1} not succesfull'.
                                     format(pickle_file, json_file))

    def json_to_pickle(json_file, pickle_file):
        data_in = get_json(json_file)
        save_pickle(data_in, pickle_file)
        data_out = get_pickle(pickle_file)
        if data_in == data_out:
            raise SerializationError('Serialization from {0} to {1} not succesfull'.
                                     format(json_file, pickle_file))

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [next] | [standalone]

#90223

On Sat, May 9, 2015 at 6:37 PM, Cecil Westerhof <Cecil@decebal.nl> wrote:
> The code:
>     def get_json(json_file):
>         with open(json_file, 'rb') as in_f:
>             return json.load(in_f)
>
>     def get_marshal(marshal_file):
>         with open(marshal_file, 'rb') as in_f:
>             return marshal.load(in_f)
>
>     def get_pickle(pickle_file):
>         with open(pickle_file, 'rb') as in_f:
>             return pickle.load(in_f)

def get_any(format, filename):
    with open(filename, 'rb') as in_f:
        return format.load(in_f)

>     def save_json(data, json_file):
>         with open(json_file, 'wb') as out_f:
>             json.dump(data, out_f)
>
>     def save_marshal(data, marshal_file):
>         with open(marshal_file, 'wb') as out_f:
>             marshal.dump(data, out_f)
>
>     def save_pickle(data, pickle_file):
>         with open(pickle_file, 'wb') as out_f:
>             pickle.dump(data, out_f)

def save_any(format, data, filename):
    with open(filename,'wb') as out_f:
        format.dump(data, out_f)

>     def marshal_to_pickle(marshal_file, pickle_file):
>         data_in = get_marshal(marshal_file)
>         save_pickle(data_in, pickle_file)
>         data_out = get_pickle(pickle_file)
>         if data_in != data_out:
>             raise SerializationError('Serialization from {0} to {1} not succesfull'.
>                                      format(marshal_file, pickle_file))
>
>     def marshal_to_json(marshal_file, json_file):
>         data_in = get_marshal(marshal_file)
>         save_json(data_in, json_file)
>         data_out = get_json(json_file)
>         if data_in != data_out:
>             raise SerializationError('Serialization from {0} to {1} not succesfull'.
>                                      format(marshal_file, json_file))
>
>     def pickle_to_json(pickle_file, json_file):
>         data_in = get_pickle(pickle_file)
>         save_json(data_in, json_file)
>         data_out = get_json(json_file)
>         if data_in != data_out:
>             raise SerializationError('Serialization from {0} to {1} not succesfull'.
>                                      format(pickle_file, json_file))

def any_to_any(fmt1, fmt2, fn1, fn2):
    data_in = get_any(fmt1, fn1)
    save_any(fmt2, data_in, fn2)
    data_out = get_any(fmt2, fn2)
    if data_in != data_out:
        raise SerializationError('Serialization from {0} to {1} not successful'.
                                     format(fn1, fn2))

formats = [json, pickle, marshal]
for fmt1 in formats:
    for fmt2 in formats:
        globals()["%s_to_%s" % (fmt1.__name__, fmt2.__name__)] = \
            functools.partial(any_to_any, fmt1, fmt2)

>     def json_to_pickle(json_file, pickle_file):
>         data_in = get_json(json_file)
>         save_pickle(data_in, pickle_file)
>         data_out = get_pickle(pickle_file)
>         if data_in == data_out:
>             raise SerializationError('Serialization from {0} to {1} not succesfull'.
>                                      format(json_file, pickle_file))

def json_to_pickle(json_file, pickle_file):
    try:
        any_to_any(json, pickle, json_file, pickle_file)
    except SerializationError:
        pass
    else:
        raise SerializationError('Serialization from {0} to {1} not successful'.
            format(json_file, pickle_file))

There. Much simpler. And maybe you can see that the last one actually
shouldn't exist :)

ChrisA

[toc] | [prev] | [next] | [standalone]

#90233

Op Saturday 9 May 2015 11:16 CEST schreef Chris Angelico:

> On Sat, May 9, 2015 at 6:37 PM, Cecil Westerhof <Cecil@decebal.nl> wrote:
>> The code:
>> def get_json(json_file):
>> with open(json_file, 'rb') as in_f:
>> return json.load(in_f)
>>
>> def get_marshal(marshal_file):
>> with open(marshal_file, 'rb') as in_f:
>> return marshal.load(in_f)
>>
>> def get_pickle(pickle_file):
>> with open(pickle_file, 'rb') as in_f:
>> return pickle.load(in_f)
>
> def get_any(format, filename):
> with open(filename, 'rb') as in_f:
> return format.load(in_f)

I was thinking about something like that and then let the other three
call this one. I think that:
    data = get_json(json_file)
is nicer to do as:
    data = get_any(json, json_file)


> def any_to_any(fmt1, fmt2, fn1, fn2): data_in = get_any(fmt1, fn1)
> save_any(fmt2, data_in, fn2) data_out = get_any(fmt2, fn2) if
> data_in != data_out: raise SerializationError('Serialization from
> {0} to {1} not successful'. format(fn1, fn2))
>
> formats = [json, pickle, marshal]
> for fmt1 in formats:
> for fmt2 in formats:
> globals()["%s_to_%s" % (fmt1.__name__, fmt2.__name__)] = \
> functools.partial(any_to_any, fmt1, fmt2)
>
>> def json_to_pickle(json_file, pickle_file): data_in =
>> get_json(json_file) save_pickle(data_in, pickle_file) data_out =
>> get_pickle(pickle_file) if data_in == data_out: raise
>> SerializationError('Serialization from {0} to {1} not succesfull'.
>> format(json_file, pickle_file))
>
> def json_to_pickle(json_file, pickle_file): try: any_to_any(json,
> pickle, json_file, pickle_file) except SerializationError: pass
> else: raise SerializationError('Serialization from {0} to {1} not
> successful'. format(json_file, pickle_file))

Was thinking about that also. Only should there be no conversion to
marshal I think.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [next] | [standalone]

#90232

Op Saturday 9 May 2015 10:37 CEST schreef Cecil Westerhof:

> To make serialization a bit easier I made a few functions to get,
> save and convert between the different types. As I see it pickle and
> json are probably used the most. I also have a get and save for
> marshal. But no conversion to marshal, because in principle you
> should not use it, so a conversion to it is not useful. I did define
> conversion from.

I used the example of Chris to make it a lot cleaner. I like DRY, but
did not know it was that easy in Python.

The code is now:
    def convert_serialization(format_in, format_out, filename_in, filename_out):
        data_in = get_serialization(format_in, filename_in)
        save_serialization(format_out, data_in, filename_out)
        data_out = get_serialization(format_out, filename_out)
        if data_in != data_out:
            raise SerializationError('Serialization from {0} to {1} not successful'.
                                         format(filename_in, filename_out))

    def get_serialization(format, filename):
        with open(filename, 'rb') as in_f:
            return format.load(in_f)

    def save_serialization(format, data, filename):
        with open(filename, 'wb') as out_f:
            format.dump(data, out_f)

    _all_serialization_formats           = [json, marshal, pickle]
    _convert_to_serialization_formats    = [json, pickle]

    for format_in in _all_serialization_formats:
        globals()[
            'save_%s' % (format_in.__name__)
        ] = functools.partial(save_serialization, format_in)
        globals()[
            'get_%s' % (format_in.__name__)
        ] = functools.partial(get_serialization, format_in)
        for format_out in _convert_to_serialization_formats:
            if format_in == format_out:
                continue
            globals()[
                '%s_to_%s' % (format_in.__name__, format_out.__name__)
            ] = functools.partial(convert_serialization, format_in, format_out)

    del format_in, format_out

I keep the all_serialization_formats and
_convert_to_serialization_formats so you can peek which ones are
supported at the moment.


> Are there other seralizations that is handy to take care of?

That is now a cinch to implement: you only have to append the two
lists. ;-)

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web