Path: csiph.com!usenet.pasdenom.info!news.franciliens.net!news.muarf.org!nntpfeed.proxad.net!proxad.net!feeder1-2.proxad.net!news.tele.dk!news.tele.dk!small.news.tele.dk!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Peter Otten <__peter__@web.de>
Subject: Re: Using Pool map with a method of a class and a list
Date: Wed, 07 Aug 2013 16:46:09 +0200
Organization: None
References: <96c575da-7601-4023-aa91-e80664f90333@googlegroups.com> <mailman.268.1375810736.1251.python-list@python.org> <4cff0d5e-33ab-42cd-b6d4-2b4fe235a274@googlegroups.com> <mailman.301.1375858161.1251.python-list@python.org> <021dfe24-af83-4307-856e-441cf35cb93a@googlegroups.com> <mailman.311.1375868873.1251.python-list@python.org> <13807c2e-7f9f-45dd-b36e-4cdc7cde6709@googlegroups.com> <CAN1F8qX5PCVsS-g03BTGYvm3opGNCg1giKC-O9cWADytqJogqw@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7Bit
User-Agent: KNode/4.7.3
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.315.1375886779.1251.python-list@python.org>
Lines: 101
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:52134

Joshua Landau wrote:

> On 7 August 2013 11:10, Luca Cerone <luca.cerone@gmail.com> wrote:
>> I can't try it now, I'll let you know later if it works!
>> (Though just by reading I can't really understand what the code does).
> 
> Well,
> 
>>> from multiprocessing import Pool
>>> from functools import partial
>>>
>>> class A(object):
>>>     def __init__(self,x):
>>>         self.value = x
>>>     def fun(self,x):
>>>         return self.value**x
> 
> This is all the same, as with
> 
>>> l = range(100)
>>> p = Pool(4)
> 
> You then wanted to do:
> 
>> op = p.map(A(3).fun, l)
> 
> but bound methods can't be pickled, it seems.
> 
> However, A(3) *can* be pickled. So what we want is a function:
> 
>     def proxy(arg):
>         A(3).fun(arg)
> 
> so we can write:
> 
>> op = p.map(proxy, l)
> 
> To generalise you might be tempted to write:
> 
>     def generic_proxy(instance, name):
>         def proxy(arg):
>             # Equiv. of instance.name(arg)
>             getattr(instance, name)(arg)
> 
> but the inner function won't work as functions-in-functions can't be
> pickled either.
> 
> So we use:
> 
>>> def _getattr_proxy_partialable(instance, name, arg):
>>>     return getattr(instance, name)(arg)
> 
> Which takes all instance, name and arg. Of course we only want our
> function to take arg, so we partial it:
> 
>>> def getattr_proxy(instance, name):
>>>     """
>>>     A version of getattr that returns a proxy function that can
>>>     be pickled. Only function calls will work on the proxy.
>>>     """
>>>     return partial(_getattr_proxy_partialable, instance, name)
> 
> partial objects are picklable, btw.
> 
>>> op = p.map(getattr_proxy(A(3), "fun"), l)
>>> print(op)
> 
> :)


There is also the copy_reg module. Adapting

<http://mail.python.org/pipermail/python-list/2008-July/469164.html>

you get:

import copy_reg
import multiprocessing
import new

def make_instancemethod(inst, methodname):
    return getattr(inst, methodname)

def pickle_instancemethod(method):
    return make_instancemethod, (method.im_self, method.im_func.__name__)

copy_reg.pickle(
    new.instancemethod, pickle_instancemethod, make_instancemethod)

class A(object):
   def __init__(self, a):
       self.a = a
   def fun(self, b):
       return self.a**b

if __name__ == "__main__":
    items = range(10)
    pool = multiprocessing.Pool(4)
    print pool.map(A(3).fun, items)