Re: Multiprocessing, shared memory vs. pickled copies

Path	csiph.com!x330-a1.tempe.blueboxinc.net!usenet.pasdenom.info!news.albasani.net!feeder.news-service.com!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<philip@semanchuk.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.011
X-Spam-Evidence	'H': 0.98; 'S': 0.00; 'python,': 0.01; 'subclass': 0.09; 'subclasses': 0.09; 'through.': 0.09; 'to:name:python list': 0.09; 'pm,': 0.11; 'somewhere': 0.11; 'am,': 0.14; 'wrote:': 0.14; 'numpy': 0.16; 'received:mindspring.com': 0.16; 'subject:memory': 0.16; 'subject:vs.': 0.16; 'surprising': 0.16; 'header:In-Reply-To:1': 0.22; 'do,': 0.22; 'that?': 0.23; 'objects': 0.24; "what's": 0.24; "i'm": 0.26; "i'll": 0.26; 'depends': 0.29; 'extent': 0.29; 'list': 0.30; 'implement': 0.30; 'easier.': 0.31; 'grasp': 0.31; 'does': 0.31; 'to:addr:python-list': 0.32; 'asking': 0.32; "i've": 0.33; 'got': 0.34; 'that,': 0.35; 'point': 0.35; 'surprised': 0.35; 'usually': 0.36; 'think': 0.36; 'charset:us-ascii': 0.36; 'else': 0.37; 'should': 0.37; 'apr': 0.38; 'url:mail-archive': 0.38; 'but': 0.38; 'url:org': 0.38; 'third': 0.38; 'to:addr:python.org': 0.39; 'header:Mime-Version:1': 0.39; 'received:24': 0.39; 'attempt': 0.40; 'philip': 0.60; 'best': 0.60; 'header:Message-Id:1': 0.62; 'upon': 0.63; 'subject:. ': 0.66; 'luck': 0.71; 'john,': 0.84; 'received:69.73': 0.84; 'tomorrow.': 0.84
Content-Type	text/plain; charset=us-ascii
Mime-Version	1.0 (Apple Message framework v1082)
Subject	Re: Multiprocessing, shared memory vs. pickled copies
From	Philip Semanchuk <philip@semanchuk.com>
In-Reply-To	<03b4730d-8106-4bc1-b79f-55bd62825f4a@w7g2000yqe.googlegroups.com>
Date	Thu, 7 Apr 2011 09:23:27 -0400
Content-Transfer-Encoding	quoted-printable
References	<6ace38dc-33c6-44ab-a17a-084d62d666cb@w9g2000prg.googlegroups.com> <mailman.27.1301960081.9059.python-list@python.org> <cad2405b-378c-43e4-b79e-edf81e5701e2@w7g2000pre.googlegroups.com> <mailman.56.1302026842.9059.python-list@python.org> <5d0fcd63-cebf-4b6e-a176-20cb7bbec389@n12g2000yqc.googlegroups.com> <03b4730d-8106-4bc1-b79f-55bd62825f4a@w7g2000yqe.googlegroups.com>
To	Python list <python-list@python.org>
X-Mailer	Apple Mail (2.1082)
X-AntiAbuse	This header was added to track abuse, please include it with any abuse report
X-AntiAbuse	Primary Hostname - deimos.nocdirect.com
X-AntiAbuse	Original Domain - python.org
X-AntiAbuse	Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse	Sender Address Domain - semanchuk.com
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.12
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.107.1302182620.9059.python-list@python.org> (permalink)
Lines	34
NNTP-Posting-Host	82.94.164.166
X-Trace	1302182620 news.xs4all.nl 81479 [::ffff:82.94.164.166]:44411
X-Complaints-To	abuse@xs4all.nl
Xref	x330-a1.tempe.blueboxinc.net comp.lang.python:2772

Show key headers only | View raw

On Apr 7, 2011, at 3:41 AM, John Ladasky wrote:

> Following up to my own post...
> 
> On Apr 6, 11:40 pm, John Ladasky <lada...@my-deja.com> wrote:
> 
>> What's up with that?
> 
> Apparently, "what's up" is that I will need to implement a third
> method in my ndarray subclass -- namely, __reduce__.
> 
> http://www.mail-archive.com/numpy-discussion@scipy.org/msg02446.html
> 
> I'm burned out for tonight, I'll attempt to grasp what __reduce__ does
> tomorrow.
> 
> Again, I'm going to point out that, given the extent that
> multiprocessing depends upon pickling, pickling should be made
> easier.  This is Python, for goodness' sake!  I'm still surprised at
> the hoops I've got to jump through.

Hi John,
My own experience has been that when I reach a surprising level of hoop jumping, it usually means there's an easier path somewhere else that I'm neglecting. 

But if pickling subclasses of numpy.ndarray objects is what you really feel you need to do, then yes, I think asking on the numpy list is the best idea. 

Good luck
Philip

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-04 13:20 -0700
  Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-04 19:34 -0400
    Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-05 09:58 -0700
      Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-05 13:43 -0400
        Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-06 23:40 -0700
          Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-07 00:41 -0700
            Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-07 09:23 -0400
          Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-07 12:44 -0500
            Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-07 11:39 -0700
              Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-07 15:01 -0500
  Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-04 19:05 -0500
    Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 16:39 -0700
  Re: Multiprocessing, shared memory vs. pickled copies Philip Semanchuk <philip@semanchuk.com> - 2011-04-04 21:16 -0400
  Re: Multiprocessing, shared memory vs. pickled copies Robert Kern <robert.kern@gmail.com> - 2011-04-05 10:47 -0500
  Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 17:03 -0700
    Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 17:38 -0700
      Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-07 18:10 -0700
        Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-09 00:36 -0700
          Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-09 10:15 -0700
            Re: Multiprocessing, shared memory vs. pickled copies John Ladasky <ladasky@my-deja.com> - 2011-04-09 13:18 -0700
              Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-10 08:01 -0700
        Re: Multiprocessing, shared memory vs. pickled copies sturlamolden <sturlamolden@yahoo.no> - 2011-04-10 15:35 -0700

csiph-web