Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #52127

Re: Using Pool map with a method of a class and a list

Show key headers only | View raw

On 7 August 2013 09:33, Luca Cerone <luca.cerone@gmail.com> wrote:
> To correct my example:
>
> from multiprocessing import Pool
>
> class A(object):
>     def __init__(self,x):
>         self.value = x
>     def fun(self,x):
>         return self.value**x
>
> l = range(100)
> p = Pool(4)
> op = p.map(A(3).fun, l)
>
> doesn't work neither in Python 2.7, nor 3.2 (by the way I can't use Python 3 for my application).

Are you using Windows? Over here on 3.3 on Linux it does. Not on 2.7 though.

>> You will find that
>> http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-> > when-using-pythons-multiprocessing-pool-ma
>> explains the problem in more detail than I understand. I suggest
>> reading it and relaying further questions back to us. Or use Python 3
>
> :) Thanks, but of course I googled and found this link before posting. I don't understand much of the details as well, that's why I posted here.
>
> Anyway, thanks for the attempt :)

Reading there, the simplest method seems to be, in effect:

from multiprocessing import Pool
from functools import partial

class A(object):
    def __init__(self,x):
        self.value = x
    def fun(self,x):
        return self.value**x

def _getattr_proxy_partialable(instance, name, arg):
    return getattr(instance, name)(arg)

def getattr_proxy(instance, name):
    """
    A version of getattr that returns a proxy function that can
    be pickled. Only function calls will work on the proxy.
    """
    return partial(_getattr_proxy_partialable, instance, name)

l = range(100)
p = Pool(4)
op = p.map(getattr_proxy(A(3), "fun"), l)
print(op)

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-06 10:12 -0700
  Re: Using Pool map with a method of a class and a list Chris Angelico <rosuav@gmail.com> - 2013-08-06 18:38 +0100
    Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-06 12:42 -0700
      Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 07:48 +0100
        Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 01:33 -0700
          Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 10:47 +0100
            Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 03:10 -0700
              Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 12:53 +0100
                Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 15:26 -0700
                Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 23:49 +0100
              Re: Using Pool map with a method of a class and a list Peter Otten <__peter__@web.de> - 2013-08-07 16:46 +0200
              Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 16:52 +0100
              Re: Using Pool map with a method of a class and a list Peter Otten <__peter__@web.de> - 2013-08-07 18:15 +0200
                Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 16:31 -0700
  Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-06 12:37 -0700

csiph-web