Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #6917 > unrolled thread

Multiprocessing.connection magic

Back to article view | Back to comp.lang.python

Contents

  Multiprocessing.connection magic Claudiu Popa <cpopa@bitdefender.com> - 2011-06-03 09:28 +0300
    Re: Multiprocessing.connection magic Chris Torek <nospam@torek.net> - 2011-06-03 07:03 +0000
      Re: Multiprocessing.connection magic Chris Angelico <rosuav@gmail.com> - 2011-06-03 17:18 +1000
    Re: Multiprocessing.connection magic Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2011-06-03 10:03 +0200
      Re: Multiprocessing.connection magic Chris Angelico <rosuav@gmail.com> - 2011-06-03 18:26 +1000
        Re: Multiprocessing.connection magic Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-06-03 08:50 +0000
          Re: Multiprocessing.connection magic Chris Angelico <rosuav@gmail.com> - 2011-06-03 20:34 +1000

#6917 — Multiprocessing.connection magic

Hello guys,
      While  working  at a dispatcher using
  multiprocessing.connection.Listener  module  I've stumbled upon some
  sort    of  magic  trick  that  amazed  me. How is this possible and
  what  does  multiprocessing  library doing in background for this to
  work?

  Client, Python 2.6


  >>> from multiprocessing.connection import Client
  >>> client = Client(("localhost", 8080))
  >>> import shutil
  >>> client.send(shutil.copy)

  Server, 3.2
  >>> from multiprocessing.connection import Listener
  >>> listener = Listener(("localhost", 8080))
  >>> con = listener.accept()
  >>> data = con.recv()
  >>> data
  <function copy at 0x024611E0>
  >>> help(data)
  Help on function copy in module shutil:

  copy(src, dst)
      Copy data and mode bits ("cp src dst").
    
          The destination may be a directory.

  >>>
  

[toc] | [next] | [standalone]

#6919

In article <mailman.2417.1307082948.9059.python-list@python.org>
Claudiu Popa  <cpopa@bitdefender.com> wrote:
>Hello guys,
>      While  working  at a dispatcher using
>  multiprocessing.connection.Listener  module  I've stumbled upon some
>  sort    of  magic  trick  that  amazed  me. How is this possible and
>  what  does  multiprocessing  library doing in background for this to
>  work?

Most of Python's sharing routines (including multiprocessing
"send", in this case) use the pickle routines to package data
for transport between processes.

Thus, you can "see the magic" pretty simply:

>  Client, Python 2.6
>
>  >>> from multiprocessing.connection import Client
>  >>> client = Client(("localhost", 8080))
>  >>> import shutil
>  >>> client.send(shutil.copy)

Here I just use pickle.dumps() to return (and print, since we are
in the interpreter) the string representation that client.send()
will send:

   >>> import pickle
   >>> import shutil
   >>> pickle.dumps(shutil.copy)
   'cshutil\ncopy\np0\n.'
   >>>

>  Server, 3.2
>  >>> from multiprocessing.connection import Listener
>  >>> listener = Listener(("localhost", 8080))
>  >>> con = listener.accept()
>  >>> data = con.recv()
>  >>> data
>  <function copy at 0x024611E0>
>  >>> help(data)
>  Help on function copy in module shutil:
[snip]

On this end, the (different) version of python simply unpickles the
byte stream.  Starting a new python session (to get rid of any
previous imports):

    $ python
    ...
    >>> import pickle
    >>> pickle.loads('cshutil\ncopy\np0\n.')
    <function copy at 0x86ef0>
    >>> help(_)
    Help on function copy in module shutil:
    ...

The real magic is in the unpickler, which has figured out how to
access shutil.copy without importing shutil into the global namespace:

    >>> shutil
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'shutil' is not defined
    >>>

but we can expose that magic as well, by feeding pickle.loads()
a "bad" string:

    >>> pickle.loads('cNotAModule\nfunc\np0\n.')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1374, in loads
        return Unpickler(file).load()
      File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 858, in load
        dispatch[key](self)
      File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1090, in load_global
        klass = self.find_class(module, name)
      File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1124, in find_class
        __import__(module)
    ImportError: No module named NotAModule
    >>>

Note the rather total lack of security here -- in the receiver, by
doing con.recv(), you are trusting the sender not to send you a
"dangerous" or invalid pickle-data-stream.  This is why the documentation
includes the following:

    Warning: The Connection.recv() method automatically unpickles
    the data it receives, which can be a security risk unless you
    can trust the process which sent the message.

    Therefore, unless the connection object was produced using Pipe()
    you should only use the recv() and send() methods after performing
    some sort of authentication. See Authentication keys.

(i.e., do that :-) -- see the associated section on authentication)
-- 
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html

[toc] | [prev] | [next] | [standalone]

#6920

On Fri, Jun 3, 2011 at 5:03 PM, Chris Torek <nospam@torek.net> wrote:
> The real magic is in the unpickler, which has figured out how to
> access shutil.copy without importing shutil into the global namespace:

So from this I gather that it doesn't actually pickle the code, just
the name. Seems a little odd, but that would explain why this didn't
really work:

>>> def asdf(x):
	x.append(len(x))
	return len(x)

>>> pickle.dumps(asdf)
b'\x80\x03c__main__\nasdf\nq\x00.'
>>> asdf=pickle.dumps(asdf)
>>> pickle.loads(asdf)
b'\x80\x03c__main__\nasdf\nq\x00.'
>>> asdf
b'\x80\x03c__main__\nasdf\nq\x00.'

I tried to do the classic - encode something, discard the original,
attempt to decode. Didn't work.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#6923

Am 03.06.2011 08:28 schrieb Claudiu Popa:
> Hello guys,
>        While  working  at a dispatcher using
>    multiprocessing.connection.Listener  module  I've stumbled upon some
>    sort    of  magic  trick  that  amazed  me. How is this possible and
>    what  does  multiprocessing  library doing in background for this to
>    work?

As Chris already said, it probably uses pickle. Doing so, you should be 
aware that unpickling strings can execute arbitrary code. So be very 
careful if you use something like that...


Thomas

[toc] | [prev] | [next] | [standalone]

#6926

On Fri, Jun 3, 2011 at 6:03 PM, Thomas Rachel
<nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de>
wrote:
> Am 03.06.2011 08:28 schrieb Claudiu Popa:
>>
>> Hello guys,
>>       While  working  at a dispatcher using
>>   multiprocessing.connection.Listener  module  I've stumbled upon some
>>   sort    of  magic  trick  that  amazed  me. How is this possible and
>>   what  does  multiprocessing  library doing in background for this to
>>   work?
>
> As Chris already said, it probably uses pickle. Doing so, you should be
> aware that unpickling strings can execute arbitrary code. So be very careful
> if you use something like that...

Nice piece of safe ambiguity there - two people said that, both named Chris!

Just how many Chrises are there on this list? I have a pet theory that
there's a greater-than-usual correlation between geeks and the name
"Chris", and the Python list has provided a number of supporting
instances.

Chris Angelico

[toc] | [prev] | [next] | [standalone]

#6930

On Fri, 03 Jun 2011 18:26:47 +1000, Chris Angelico wrote:

> Just how many Chrises are there on this list? I have a pet theory that
> there's a greater-than-usual correlation between geeks and the name
> "Chris", and the Python list has provided a number of supporting
> instances.

My theory is that geeks (at least in Australia) gravitate towards the 
names Matt, or sometimes Ben. So much so that when I'm interviewing a new 
coder, I'll sometimes say "You're name's not Matt? That'll cause a bit of 
confusion. Mind if we call you Matt?"


http://www.youtube.com/watch?v=_f_p0CgPeyA


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#6934

On Fri, Jun 3, 2011 at 6:50 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Fri, 03 Jun 2011 18:26:47 +1000, Chris Angelico wrote:
>
>> Just how many Chrises are there on this list? I have a pet theory that
>> there's a greater-than-usual correlation between geeks and the name
>> "Chris", and the Python list has provided a number of supporting
>> instances.
>
> My theory is that geeks (at least in Australia) gravitate towards the
> names Matt, or sometimes Ben. So much so that when I'm interviewing a new
> coder, I'll sometimes say "You're name's not Matt? That'll cause a bit of
> confusion. Mind if we call you Matt?"

Interesting. I'll have to keep my eyes open for the Matts and Bens. Fascinating.

Chris Angelico

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web