Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #52131

Re: Using Pool map with a method of a class and a list

Show key headers only | View raw

On 7 August 2013 11:10, Luca Cerone <luca.cerone@gmail.com> wrote:
> I can't try it now, I'll let you know later if it works!
> (Though just by reading I can't really understand what the code does).

Well,

>> from multiprocessing import Pool
>> from functools import partial
>>
>> class A(object):
>>     def __init__(self,x):
>>         self.value = x
>>     def fun(self,x):
>>         return self.value**x

This is all the same, as with

>> l = range(100)
>> p = Pool(4)

You then wanted to do:

> op = p.map(A(3).fun, l)

but bound methods can't be pickled, it seems.

However, A(3) *can* be pickled. So what we want is a function:

    def proxy(arg):
        A(3).fun(arg)

so we can write:

> op = p.map(proxy, l)

To generalise you might be tempted to write:

    def generic_proxy(instance, name):
        def proxy(arg):
            # Equiv. of instance.name(arg)
            getattr(instance, name)(arg)

but the inner function won't work as functions-in-functions can't be
pickled either.

So we use:

>> def _getattr_proxy_partialable(instance, name, arg):
>>     return getattr(instance, name)(arg)

Which takes all instance, name and arg. Of course we only want our
function to take arg, so we partial it:

>> def getattr_proxy(instance, name):
>>     """
>>     A version of getattr that returns a proxy function that can
>>     be pickled. Only function calls will work on the proxy.
>>     """
>>     return partial(_getattr_proxy_partialable, instance, name)

partial objects are picklable, btw.

>> op = p.map(getattr_proxy(A(3), "fun"), l)
>> print(op)

:)

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-06 10:12 -0700
  Re: Using Pool map with a method of a class and a list Chris Angelico <rosuav@gmail.com> - 2013-08-06 18:38 +0100
    Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-06 12:42 -0700
      Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 07:48 +0100
        Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 01:33 -0700
          Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 10:47 +0100
            Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 03:10 -0700
              Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 12:53 +0100
                Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 15:26 -0700
                Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 23:49 +0100
              Re: Using Pool map with a method of a class and a list Peter Otten <__peter__@web.de> - 2013-08-07 16:46 +0200
              Re: Using Pool map with a method of a class and a list Joshua Landau <joshua@landau.ws> - 2013-08-07 16:52 +0100
              Re: Using Pool map with a method of a class and a list Peter Otten <__peter__@web.de> - 2013-08-07 18:15 +0200
                Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-07 16:31 -0700
  Re: Using Pool map with a method of a class and a list Luca Cerone <luca.cerone@gmail.com> - 2013-08-06 12:37 -0700

csiph-web