Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!xlned.com!feeder5.xlned.com!feeder1.cambriumusenet.nl!feed.tweaknews.nl!194.109.133.87.MISMATCH!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
Sender: joshua.landau.ws@gmail.com
In-Reply-To: <13807c2e-7f9f-45dd-b36e-4cdc7cde6709@googlegroups.com>
References: <96c575da-7601-4023-aa91-e80664f90333@googlegroups.com> <mailman.268.1375810736.1251.python-list@python.org> <4cff0d5e-33ab-42cd-b6d4-2b4fe235a274@googlegroups.com> <mailman.301.1375858161.1251.python-list@python.org> <021dfe24-af83-4307-856e-441cf35cb93a@googlegroups.com> <mailman.311.1375868873.1251.python-list@python.org> <13807c2e-7f9f-45dd-b36e-4cdc7cde6709@googlegroups.com>
From: Joshua Landau <joshua@landau.ws>
Date: Wed, 7 Aug 2013 12:53:40 +0100
Subject: Re: Using Pool map with a method of a class and a list
To: Luca Cerone <luca.cerone@gmail.com>
Content-Type: text/plain; charset=UTF-8
Cc: python-list <python-list@python.org>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.314.1375876463.1251.python-list@python.org>
Lines: 66
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:52131

On 7 August 2013 11:10, Luca Cerone <luca.cerone@gmail.com> wrote:
> I can't try it now, I'll let you know later if it works!
> (Though just by reading I can't really understand what the code does).

Well,

>> from multiprocessing import Pool
>> from functools import partial
>>
>> class A(object):
>>     def __init__(self,x):
>>         self.value = x
>>     def fun(self,x):
>>         return self.value**x

This is all the same, as with

>> l = range(100)
>> p = Pool(4)

You then wanted to do:

> op = p.map(A(3).fun, l)

but bound methods can't be pickled, it seems.

However, A(3) *can* be pickled. So what we want is a function:

    def proxy(arg):
        A(3).fun(arg)

so we can write:

> op = p.map(proxy, l)

To generalise you might be tempted to write:

    def generic_proxy(instance, name):
        def proxy(arg):
            # Equiv. of instance.name(arg)
            getattr(instance, name)(arg)

but the inner function won't work as functions-in-functions can't be
pickled either.

So we use:

>> def _getattr_proxy_partialable(instance, name, arg):
>>     return getattr(instance, name)(arg)

Which takes all instance, name and arg. Of course we only want our
function to take arg, so we partial it:

>> def getattr_proxy(instance, name):
>>     """
>>     A version of getattr that returns a proxy function that can
>>     be pickled. Only function calls will work on the proxy.
>>     """
>>     return partial(_getattr_proxy_partialable, instance, name)

partial objects are picklable, btw.

>> op = p.map(getattr_proxy(A(3), "fun"), l)
>> print(op)

:)