Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.python > #78171 > unrolled thread

Pyrolite, lightweight pickle and pyro client library, seeking a bit of testing help

Started byIrmen de Jong <irmen.NOSPAM@xs4all.nl>
First post2014-09-22 19:47 +0200
Last post2014-09-24 00:16 +0200
Articles 5 — 2 participants

Back to article view | Back to comp.lang.python


Contents

  Pyrolite, lightweight pickle and pyro client library, seeking a bit of testing help Irmen de Jong <irmen.NOSPAM@xs4all.nl> - 2014-09-22 19:47 +0200
    Re: Pyrolite, lightweight pickle and pyro client library, seeking a bit of testing help Chris Angelico <rosuav@gmail.com> - 2014-09-23 03:53 +1000
      Re: Pyrolite, lightweight pickle and pyro client library, seeking a bit of testing help Irmen de Jong <irmen.NOSPAM@xs4all.nl> - 2014-09-22 20:23 +0200
        Re: Pyrolite, lightweight pickle and pyro client library, seeking a bit of testing help Chris Angelico <rosuav@gmail.com> - 2014-09-23 04:28 +1000
          Re: Pyrolite, lightweight pickle and pyro client library, seeking a bit of testing help Irmen de Jong <irmen.NOSPAM@xs4all.nl> - 2014-09-24 00:16 +0200

#78171 — Pyrolite, lightweight pickle and pyro client library, seeking a bit of testing help

FromIrmen de Jong <irmen.NOSPAM@xs4all.nl>
Date2014-09-22 19:47 +0200
SubjectPyrolite, lightweight pickle and pyro client library, seeking a bit of testing help
Message-ID<542060bf$0$2929$e4fe514c@news.xs4all.nl>
Hi,

I've developed Pyrolite (https://github.com/irmen/Pyrolite), a lightweight client
library for Java and .NET to gain access to Python servers running Pyro. As such it also
contains a complete pickle and unpickle implementation in these languages.

Quite recently I got a pull request to fix a big/little-endianness decoding issue
regarding machine floats and doubles in pickled arrays.

It was a mistake on my part but it got me thinking: the unit tests of my library should
ideally be run on both little- and big-endian machines, because some of the code depends
on this.

I've only got access to little endian systems myself. So basically I'm looking for some
tips here, for instance can Travis-ci.org perhaps help out? Or can someone here with
access to a big endian system help me out with running the Java and .NET (Microsoft or
Mono) unit test suite of my project on that system? (just a one time request, to verify
the current version).


Irmen de Jong

PS the reason why I'm posting a Java or .NET related issue here is because the core
protocol my library handles is a Python invented one ;)

[toc] | [next] | [standalone]


#78172

FromChris Angelico <rosuav@gmail.com>
Date2014-09-23 03:53 +1000
Message-ID<mailman.14225.1411408441.18130.python-list@python.org>
In reply to#78171
On Tue, Sep 23, 2014 at 3:47 AM, Irmen de Jong <irmen.NOSPAM@xs4all.nl> wrote:
> I've developed Pyrolite (https://github.com/irmen/Pyrolite), a lightweight client
> library for Java and .NET to gain access to Python servers running Pyro. As such it also
> contains a complete pickle and unpickle implementation in these languages.

Does this imply that you send pickle data across the internet and
unpickle it? Isn't that rather dangerous?

ChrisA

[toc] | [prev] | [next] | [standalone]


#78174

FromIrmen de Jong <irmen.NOSPAM@xs4all.nl>
Date2014-09-22 20:23 +0200
Message-ID<5420690b$0$2833$e4fe514c@news.xs4all.nl>
In reply to#78172
On 22-9-2014 19:53, Chris Angelico wrote:
> On Tue, Sep 23, 2014 at 3:47 AM, Irmen de Jong <irmen.NOSPAM@xs4all.nl> wrote:
>> I've developed Pyrolite (https://github.com/irmen/Pyrolite), a lightweight client
>> library for Java and .NET to gain access to Python servers running Pyro. As such it also
>> contains a complete pickle and unpickle implementation in these languages.
> 
> Does this imply that you send pickle data across the internet and
> unpickle it? Isn't that rather dangerous?
> 
> ChrisA


Yes it is, good to point this out.

This is why Pyro has been using a different (and safe) serializer by default for a while
now. You have to plow through the usual security warnings in the docs and make a
conscious effort in your code to enable the pickle serializer if you really want/need it.

Pyrolite also contains a Java and .NET version of that safe serializer so you should not
be using pickle at all when dealing with Pyro, but its implementation is there. And the
pickle code can be used independently.


Irmen

[toc] | [prev] | [next] | [standalone]


#78177

FromChris Angelico <rosuav@gmail.com>
Date2014-09-23 04:28 +1000
Message-ID<mailman.14229.1411410527.18130.python-list@python.org>
In reply to#78174
On Tue, Sep 23, 2014 at 4:23 AM, Irmen de Jong <irmen.NOSPAM@xs4all.nl> wrote:
> This is why Pyro has been using a different (and safe) serializer by default for a while
> now. You have to plow through the usual security warnings in the docs and make a
> conscious effort in your code to enable the pickle serializer if you really want/need it.

Is the safe serializer affected by byte order? If not, you could just
mark this off as a known bug, and say "if anyone has a big-endian
system to test this on, please help out". It would be another
incentive to use the safe serializer rather than pickle. And it'd save
you a lot of trouble. :)

I would have offered one of my systems for the test, except ...
they're all little-endian. I'm all PC-based hardware here (mainly
Linux running on Intel CPUs, though there are some variations).

ChrisA

[toc] | [prev] | [next] | [standalone]


#78255

FromIrmen de Jong <irmen.NOSPAM@xs4all.nl>
Date2014-09-24 00:16 +0200
Message-ID<5421f152$0$2932$e4fe514c@news2.news.xs4all.nl>
In reply to#78177
On 22-9-2014 20:28, Chris Angelico wrote:
> On Tue, Sep 23, 2014 at 4:23 AM, Irmen de Jong <irmen.NOSPAM@xs4all.nl> wrote:
>> This is why Pyro has been using a different (and safe) serializer by default for a while
>> now. You have to plow through the usual security warnings in the docs and make a
>> conscious effort in your code to enable the pickle serializer if you really want/need it.
> 
> Is the safe serializer affected by byte order? If not, you could just
> mark this off as a known bug, and say "if anyone has a big-endian
> system to test this on, please help out". It would be another
> incentive to use the safe serializer rather than pickle. And it'd save
> you a lot of trouble. :)

The safe serializer is this one: https://github.com/irmen/serpent and it is a text based
protocol, so no, it is not affected by byte order. It uses Python literal expressions as
understood by ast.literal_eval().

Note that the endianness bug in my pickle implementation was fixed by the pull request
so it is no longer a known bug. But I still would like to be able to run the test suite
on both endianness platforms (or let someone else try it for me :-) )


> 
> I would have offered one of my systems for the test, except ...
> they're all little-endian. I'm all PC-based hardware here (mainly
> Linux running on Intel CPUs, though there are some variations).

Heh, yes; I used to have a PowerPC G4 mac mini (a big endian platform), but that machine
got replaced by an Intel based one (little endian, as all the other computers I have
access to). Ah well.

Irmen


[toc] | [prev] | [standalone]


Back to top | Article view | comp.lang.python


csiph-web