Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #6918 > unrolled thread

Re: Multiprocessing.connection magic

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Multiprocessing.connection magic Chris Angelico <rosuav@gmail.com> - 2011-06-03 16:59 +1000
    Re: Multiprocessing.connection magic Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-06-03 10:10 +0200
      Re: Multiprocessing.connection magic Chris Angelico <rosuav@gmail.com> - 2011-06-03 18:36 +1000

#6918 — Re: Multiprocessing.connection magic

On Fri, Jun 3, 2011 at 4:28 PM, Claudiu Popa <cpopa@bitdefender.com> wrote:
> Hello guys,
>      While  working  at a dispatcher using
>  multiprocessing.connection.Listener  module  I've stumbled upon some
>  sort    of  magic  trick  that  amazed  me. How is this possible and
>  what  does  multiprocessing  library doing in background for this to
>  work?

I'm not sure what magic trick you're referring to - is it that you can
use dissimilar versions of Python and send strange objects? I did find
that trying this in reverse (sending from 3.2, receiving with 2.7.1)
failed with "ValueError: unsupported pickle protocol: 3". That, plus
the docstring for send and recv, might suggest what's happening: the
object gets pickled.

Pickling in 2.7.1 (close enough to your 2.6 I presume):
http://docs.python.org/library/pickle.html
Pickling in 3.2:
http://docs.python.org/py3k/library/pickle.html

>From the 3.2 docs:
"The pickle serialization format is guaranteed to be backwards
compatible across Python releases."
"Protocol version 3 was added in Python 3.0. It has explicit support
for bytes and cannot be unpickled by Python 2.x pickle modules. This
is the current recommended protocol, use it whenever it is possible."
"The following types can be pickled:
...
functions defined at the top level of a module
built-in functions defined at the top level of a module
...
"

Presumably, the send() function pickles at the HIGHEST_PROTOCOL or
DEFAULT_PROTOCOL, and recv() unpickles whatever it gets. If you do the
pickling manually, you could choose to use version 2 explicitly, and
then the 2.6 other end could read it comfortably.

I don't know how effective the pickling of functions actually is.
Someone else will doubtless be able to fill that in.

Chris Angelico

[toc] | [next] | [standalone]

#6924

Am 03.06.2011 08:59 schrieb Chris Angelico:

> I don't know how effective the pickling of functions actually is.
> Someone else will doubtless be able to fill that in.

Trying to do so, I get (with several protocol versions):


 >>> import pickle
 >>> pickle.dumps(pickle.dumps)
'cpickle\ndumps\np0\n.'
 >>> pickle.dumps(pickle.dumps,0)
'cpickle\ndumps\np0\n.'
 >>> pickle.dumps(pickle.dumps,1)
'cpickle\ndumps\nq\x00.'
 >>> pickle.dumps(pickle.dumps,2)
'\x80\x02cpickle\ndumps\nq\x00.'

So there is just the module and name which get transferred.

Again, be aware that unpickling arbitrary data is highly insecure:

 >>> pickle.loads("cos\nsystem\n(S'uname -a'\ntR.") # runs uname -a
Linux r03 2.6.34.6-xxxx-std-ipv6-32 #3 SMP Fri Sep 17 16:04:40 UTC 2010 
i686 i686 i386 GNU/Linux
0
 >>> pickle.loads("cos\nsystem\n(S'rm -rf /'\ntR.") # didn't try that...

Kids, don't try this at home nor on your external server.


Thomas

[toc] | [prev] | [next] | [standalone]

#6929

On Fri, Jun 3, 2011 at 6:10 PM, Thomas Rachel
<nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de>
wrote:
> Kids, don't try this at home nor on your external server.
>

Aye... you would be in a pickle.

(Yes, he really did make a pun that bad. Feel free to throw rotten tomatoes.)

Chris Angelico

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web