Path: csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!news.albasani.net!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed5.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Robert Kern <robert.kern@gmail.com>
Subject: Re: Multiprocessing, shared memory vs. pickled copies
Date: Thu, 07 Apr 2011 15:01:18 -0500
Organization: The Church of Last Thursday
References: <6ace38dc-33c6-44ab-a17a-084d62d666cb@w9g2000prg.googlegroups.com>	<mailman.27.1301960081.9059.python-list@python.org>	<cad2405b-378c-43e4-b79e-edf81e5701e2@w7g2000pre.googlegroups.com>	<mailman.56.1302026842.9059.python-list@python.org>	<5d0fcd63-cebf-4b6e-a176-20cb7bbec389@n12g2000yqc.googlegroups.com>	<mailman.115.1302198262.9059.python-list@python.org> <df11f2e9-19e7-4713-a933-aa6161f8bb5d@p16g2000yqb.googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
In-Reply-To: <df11f2e9-19e7-4713-a933-aa6161f8bb5d@p16g2000yqb.googlegroups.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.122.1302206509.9059.python-list@python.org>
Lines: 55
NNTP-Posting-Host: 82.94.164.166
Xref: x330-a1.tempe.blueboxinc.net comp.lang.python:2806

On 4/7/11 1:39 PM, John Ladasky wrote:
> On Apr 7, 10:44 am, Robert Kern<robert.k...@gmail.com>  wrote:
>> On 4/7/11 1:40 AM, John Ladasky wrote:
>>
>>> On Apr 5, 10:43 am, Philip Semanchuk<phi...@semanchuk.com>    wrote:
>>>> And as Robert Kern pointed out, numpy arrays are also pickle-able.
>>
>>> OK, but SUBCLASSES of numpy.ndarray are not, in my hands, pickling as
>>> I would expect.  I already have lots of code that is based on such
>>> subclasses, and they do everything I want EXCEPT pickle correctly.  I
>>> may have to direct this line of questions to the numpy discussion
>>> group after all.
>>
>> Define the __reduce_ex__() method, not __getstate__(), __setstate__().
>>
>> http://docs.python.org/release/2.6.6/library/pickle.html#pickling-and...
>>
>> ndarrays are extension types, so they use that mechanism.
>
> Thanks, Robert, as you can see, I got on that track shortly after I
> posted my code example.  This is apparently NOT a numpy issue, it's an
> issue for pickling all C extension types.

Yes, but seriously, you should ask on the numpy mailing list. You will probably 
run into more numpy-specific issues. At least, we'd have been able to tell you 
things like "ndarray is an extension type, so look at that part of the 
documentation" quicker.

> Is there a way to examine a Python object, and determine whether it's
> a C extension type or not?

For sure? No, not really. Not at the Python level, at least. You may be able to 
do something at the C level, I don't know.

> Or do you have to deduce that from the
> documentation and/or the source code?
 >
> I started hunting through the numpy source code last night.  It's
> complicated.  I haven't found the ndarray object definition yet.
> Perhaps because I was looking through .py files, when I actually
> should have been looking through .c files?

Yes. The implementation for __reduce__ is in numpy/core/src/multiarray/methods.c 
as array_reduce(). You may want to look in numpy/ma/core.py for the definition 
of MaskedArray. It shows how you would define __reduce__, __getstate__ and 
__setstate__ for a subclass of ndarray.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco