Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #52134

Re: Using Pool map with a method of a class and a list

Show key headers only | View raw

Joshua Landau wrote:

> On 7 August 2013 11:10, Luca Cerone <luca.cerone@gmail.com> wrote:
>> I can't try it now, I'll let you know later if it works!
>> (Though just by reading I can't really understand what the code does).
> 
> Well,
> 
>>> from multiprocessing import Pool
>>> from functools import partial
>>>
>>> class A(object):
>>>     def __init__(self,x):
>>>         self.value = x
>>>     def fun(self,x):
>>>         return self.value**x
> 
> This is all the same, as with
> 
>>> l = range(100)
>>> p = Pool(4)
> 
> You then wanted to do:
> 
>> op = p.map(A(3).fun, l)
> 
> but bound methods can't be pickled, it seems.
> 
> However, A(3) *can* be pickled. So what we want is a function:
> 
>     def proxy(arg):
>         A(3).fun(arg)
> 
> so we can write:
> 
>> op = p.map(proxy, l)
> 
> To generalise you might be tempted to write:
> 
>     def generic_proxy(instance, name):
>         def proxy(arg):
>             # Equiv. of instance.name(arg)
>             getattr(instance, name)(arg)
> 
> but the inner function won't work as functions-in-functions can't be
> pickled either.
> 
> So we use:
> 
>>> def _getattr_proxy_partialable(instance, name, arg):
>>>     return getattr(instance, name)(arg)
> 
> Which takes all instance, name and arg. Of course we only want our
> function to take arg, so we partial it:
> 
>>> def getattr_proxy(instance, name):
>>>     """
>>>     A version of getattr that returns a proxy function that can
>>>     be pickled. Only function calls will work on the proxy.
>>>     """
>>>     return partial(_getattr_proxy_partialable, instance, name)
> 
> partial objects are picklable, btw.
> 
>>> op = p.map(getattr_proxy(A(3), "fun"), l)
>>> print(op)
> 
> :)


There is also the copy_reg module. Adapting

<http://mail.python.org/pipermail/python-list/2008-July/469164.html>

you get:

import copy_reg
import multiprocessing
import new

def make_instancemethod(inst, methodname):
    return getattr(inst, methodname)

def pickle_instancemethod(method):
    return make_instancemethod, (method.im_self, method.im_func.__name__)

copy_reg.pickle(
    new.instancemethod, pickle_instancemethod, make_instancemethod)

class A(object):
   def __init__(self, a):
       self.a = a
   def fun(self, b):
       return self.a**b

if __name__ == "__main__":
    items = range(10)
    pool = multiprocessing.Pool(4)
    print pool.map(A(3).fun, items)

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-06 10:12 -0700
  Re: Using Pool map with a method of a class and a list Chris Angelico <rosuav@gmail.com> - 2013-08-06 18:38 +0100
    Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-06 12:42 -0700
      Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 07:48 +0100
        Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 01:33 -0700
          Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 10:47 +0100
            Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 03:10 -0700
              Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 12:53 +0100
                Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 15:26 -0700
                Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 23:49 +0100
              Re: Using Pool map with a method of a class and a list Peter Otten <__peter__@web.de> - 2013-08-07 16:46 +0200
              Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 16:52 +0100
              Re: Using Pool map with a method of a class and a list Peter Otten <__peter__@web.de> - 2013-08-07 18:15 +0200
                Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 16:31 -0700
  Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-06 12:37 -0700

csiph-web