Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!eweka.nl!lightspeed.eweka.nl!194.109.133.87.MISMATCH!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: joshua.landau.ws@gmail.com
In-Reply-To: <021dfe24-af83-4307-856e-441cf35cb93a@googlegroups.com>
References: <96c575da-7601-4023-aa91-e80664f90333@googlegroups.com> <mailman.268.1375810736.1251.python-list@python.org> <4cff0d5e-33ab-42cd-b6d4-2b4fe235a274@googlegroups.com> <mailman.301.1375858161.1251.python-list@python.org> <021dfe24-af83-4307-856e-441cf35cb93a@googlegroups.com>
From: Joshua Landau <joshua@landau.ws>
Date: Wed, 7 Aug 2013 10:47:06 +0100
Subject: Re: Using Pool map with a method of a class and a list
To: Luca Cerone <luca.cerone@gmail.com>
Content-Type: text/plain; charset=UTF-8
Cc: python-list <python-list@python.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.311.1375868873.1251.python-list@python.org>
Lines: 53
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:52127

On 7 August 2013 09:33, Luca Cerone <luca.cerone@gmail.com> wrote:
> To correct my example:
>
> from multiprocessing import Pool
>
> class A(object):
>     def __init__(self,x):
>         self.value = x
>     def fun(self,x):
>         return self.value**x
>
> l = range(100)
> p = Pool(4)
> op = p.map(A(3).fun, l)
>
> doesn't work neither in Python 2.7, nor 3.2 (by the way I can't use Python 3 for my application).

Are you using Windows? Over here on 3.3 on Linux it does. Not on 2.7 though.

>> You will find that
>> http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-> > when-using-pythons-multiprocessing-pool-ma
>> explains the problem in more detail than I understand. I suggest
>> reading it and relaying further questions back to us. Or use Python 3
>
> :) Thanks, but of course I googled and found this link before posting. I don't understand much of the details as well, that's why I posted here.
>
> Anyway, thanks for the attempt :)

Reading there, the simplest method seems to be, in effect:

from multiprocessing import Pool
from functools import partial

class A(object):
    def __init__(self,x):
        self.value = x
    def fun(self,x):
        return self.value**x

def _getattr_proxy_partialable(instance, name, arg):
    return getattr(instance, name)(arg)

def getattr_proxy(instance, name):
    """
    A version of getattr that returns a proxy function that can
    be pickled. Only function calls will work on the proxy.
    """
    return partial(_getattr_proxy_partialable, instance, name)

l = range(100)
p = Pool(4)
op = p.map(getattr_proxy(A(3), "fun"), l)
print(op)