Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #71960

Re: Parallel python in the cloud

Show all headers | View raw

On 2014-05-24 07:46, Charles Gagnon wrote:
> We were happily using PiCloud for several long calculations and we very happy with with it. With their realtime cores, we could take really large calculations set and run through fairly quickly.
>
> Now that PiCloud is going away, we ran a few tests on Mutlyvac but so far, we are struggling to accomplish the same thing we had on PiCloud.
>
> I have several "pieces" of my puzzle but can't seem to be able to put it together. I've seen and tried StarCluster and also various parallel python options but all options seem challenging to put together.
>
> The goal is to mimic PiCloud, ie. loop through a function:
>
> def some_NP_func(x, y):
>     ...
>     return z
>
> some_cloud.call(some_NP_func, a1, a2)
>
> Which computes the function on the cloud. We use this often in for loops with arrays of arguments. The other scenario is:
>
> some_cloud.map(some_NP_intense_func, [...], [...])
>
> Which iterates through and returns results. We need to run a lot of this in batch from a scheduler so I always try to avoid interactive environment (how does iPython parallel work in batch?).

IPython parallel works just fine "in batch". As far as your client code (i.e. 
what you wrote above) is concerned, it's just another library. E.g.

https://github.com/ipython/ipython/blob/master/examples/Parallel%20Computing/nwmerge.py
https://github.com/ipython/ipython/blob/master/examples/Parallel%20Computing/itermapresult.py

etc.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Parallel python in the cloud Charles Gagnon <charlesg@unixrealm.com> - 2014-05-23 23:46 -0700
  Re: Parallel python in the cloud Robert Kern <robert.kern@gmail.com> - 2014-05-24 10:57 +0100

csiph-web