Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #2820

Re: multiprocessing

Show all headers | View raw

On Apr 7, 2011, at 8:57 PM, Kerensa McElroy wrote:

> 
> Hi,
> 
> thanks for your response.
> 
> I checked out multiprocessing.value, however from what I can make out, it works with object of only a very limited type. Is there a way to do this for more complex objects? (In reality, my object is a large multi-dimensional numpy array).

Elsa, 
Are you following the current thread in this list which is talking about sharing numpy arrays via multiprocessing?

http://mail.python.org/pipermail/python-list/2011-April/1269173.html





> Date: Wed, 6 Apr 2011 22:20:06 -0700
> Subject: Re: multiprocessing
> From: drsalists@gmail.com
> To: kerensaelise@hotmail.com
> CC: python-list@python.org
> 
> 
> On Wed, Apr 6, 2011 at 9:06 PM, elsa <kerensaelise@hotmail.com> wrote:
> 
> Hi guys,
> 
> 
> 
> I want to try out some pooling of processors, but I'm not sure if it
> 
> is possible to do what I want to do. Basically, I want to have a
> 
> global object, that is updated during the execution of a function, and
> 
> I want to be able to run this function several times on parallel
> 
> processors. The order in which the function runs doesn't matter, and
> 
> the value of the object doesn't matter to the function, but I do want
> 
> the processors to take turns 'nicely' when updating the object, so
> 
> there are no collisions. Here is an extremely simplified and trivial
> 
> example of what I have in mind:
> 
> 
> 
> from multiprocessing import Pool
> 
> import random
> 
> 
> 
> p=Pool(4)
> 
> myDict={}
> 
> 
> 
> def update(value):
> 
>    global myDict
> 
>    index=random.random()
> 
>    myDict[index]+=value
> 
> 
> 
> total=1000
> 
> 
> 
> p.map(update,range(total))
> 
> 
> 
> 
> 
> After, I would also like to be able to use several processors to
> 
> access the global object (but not modify it). Again, order doesn't
> 
> matter:
> 
> 
> 
> p1=Pool(4)
> 
> 
> 
> def getValues(index):
> 
>    global myDict
> 
>    print myDict[index]
> 
> 
> 
> p1.map(getValues,keys.myDict)
> 
> 
> 
> Is there a way to do this 
> This should give you a synchronized wrapper around an object in shared memory:
> 
> http://docs.python.org/library/multiprocessing.html#multiprocessing.Value
> 
> 
> 		 	   		  -- 
> http://mail.python.org/mailman/listinfo/python-list

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar

Thread

multiprocessing elsa <kerensaelise@hotmail.com> - 2011-04-06 21:06 -0700
  Re: multiprocessing Philip Semanchuk <philip@semanchuk.com> - 2011-04-07 21:05 -0400

csiph-web