Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #2806

Re: Multiprocessing, shared memory vs. pickled copies

Show key headers only | View raw

On 4/7/11 1:39 PM, John Ladasky wrote:
> On Apr 7, 10:44 am, Robert Kern<robert.k...@gmail.com>  wrote:
>> On 4/7/11 1:40 AM, John Ladasky wrote:
>>
>>> On Apr 5, 10:43 am, Philip Semanchuk<phi...@semanchuk.com>    wrote:
>>>> And as Robert Kern pointed out, numpy arrays are also pickle-able.
>>
>>> OK, but SUBCLASSES of numpy.ndarray are not, in my hands, pickling as
>>> I would expect.  I already have lots of code that is based on such
>>> subclasses, and they do everything I want EXCEPT pickle correctly.  I
>>> may have to direct this line of questions to the numpy discussion
>>> group after all.
>>
>> Define the __reduce_ex__() method, not __getstate__(), __setstate__().
>>
>> http://docs.python.org/release/2.6.6/library/pickle.html#pickling-and...
>>
>> ndarrays are extension types, so they use that mechanism.
>
> Thanks, Robert, as you can see, I got on that track shortly after I
> posted my code example.  This is apparently NOT a numpy issue, it's an
> issue for pickling all C extension types.

Yes, but seriously, you should ask on the numpy mailing list. You will probably 
run into more numpy-specific issues. At least, we'd have been able to tell you 
things like "ndarray is an extension type, so look at that part of the 
documentation" quicker.

> Is there a way to examine a Python object, and determine whether it's
> a C extension type or not?

For sure? No, not really. Not at the Python level, at least. You may be able to 
do something at the C level, I don't know.

> Or do you have to deduce that from the
> documentation and/or the source code?
 >
> I started hunting through the numpy source code last night.  It's
> complicated.  I haven't found the ndarray object definition yet.
> Perhaps because I was looking through .py files, when I actually
> should have been looking through .c files?

Yes. The implementation for __reduce__ is in numpy/core/src/multiarray/methods.c 
as array_reduce(). You may want to look in numpy/ma/core.py for the definition 
of MaskedArray. It shows how you would define __reduce__, __getstate__ and 
__setstate__ for a subclass of ndarray.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-04 13:20 -0700
  Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-04 19:34 -0400
    Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-05 09:58 -0700
      Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-05 13:43 -0400
        Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-06 23:40 -0700
          Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-07 00:41 -0700
            Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-07 09:23 -0400
          Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-07 12:44 -0500
            Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-07 11:39 -0700
              Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-07 15:01 -0500
  Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-04 19:05 -0500
    Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 16:39 -0700
  Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-04 21:16 -0400
  Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-05 10:47 -0500
  Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 17:03 -0700
    Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 17:38 -0700
      Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 18:10 -0700
        Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-09 00:36 -0700
          Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-09 10:15 -0700
            Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-09 13:18 -0700
              Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-10 08:01 -0700
        Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-10 15:35 -0700

csiph-web