Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #2669

Re: Multiprocessing, shared memory vs. pickled copies

Show all headers | View raw

On Apr 5, 2011, at 12:58 PM, John Ladasky wrote:

> Hi Philip,
> 
> Thanks for the reply.
> 
> On Apr 4, 4:34 pm, Philip Semanchuk <phi...@semanchuk.com> wrote:
>> So if you're going to use multiprocessing, you're going to use pickle, and you
>> need pickleable objects.
> 
> OK, that's good to know.

But as Dan Stromberg pointed out, there are some pickle-free ways to communicate between processes using multiprocessing.

> This leads straight into my second question.  I THINK, without knowing
> for sure, that most user classes would pickle correctly by simply
> iterating through __dict__.  So, why isn't this the default behavior
> for Python?  Was the assumption that programmers would only want to
> pickle built-in classes?  

One can pickle user-defined classes:

>>> class Foo(object):
...     pass
... 
>>> import pickle
>>> foo_instance = Foo()
>>> pickle.dumps(foo_instance)
'ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'


And as Robert Kern pointed out, numpy arrays are also pickle-able.

>>> import numpy
>>> pickle.dumps(numpy.zeros(3))
"cnumpy.core.multiarray\n_reconstruct\np0\n(cnumpy\nndarray\np1\n(I0\ntp2\nS'b'\np3\ntp4\nRp5\n(I1\n(I3\ntp6\ncnumpy\ndtype\np7\n(S'f8'\np8\nI0\nI1\ntp9\nRp10\n(I3\nS'<'\np11\nNNNI-1\nI-1\nI0\ntp12\nbI00\nS'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\np13\ntp14\nb."

As a side note, you should always use "new style" classes, particularly since you're exploring the details of Python class construction. "New" is a bit a of misnomer now, as "new" style classes were introduced in Python 2.2. They have been the status quo in Python 2.x for a while now and are the only choice in Python 3.x.

Subclassing object gives you a new style class:
   class Foo(object):

Not subclassing object (as you did in your example) gives you an old style class:
   class Foo:



Cheers
Philip

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-04 13:20 -0700
  Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-04 19:34 -0400
    Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-05 09:58 -0700
      Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-05 13:43 -0400
        Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-06 23:40 -0700
          Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-07 00:41 -0700
            Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-07 09:23 -0400
          Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-07 12:44 -0500
            Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-07 11:39 -0700
              Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-07 15:01 -0500
  Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-04 19:05 -0500
    Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 16:39 -0700
  Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-04 21:16 -0400
  Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-05 10:47 -0500
  Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 17:03 -0700
    Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 17:38 -0700
      Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 18:10 -0700
        Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-09 00:36 -0700
          Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-09 10:15 -0700
            Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-09 13:18 -0700
              Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-10 08:01 -0700
        Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-10 15:35 -0700

csiph-web