Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #98064 > unrolled thread

PyPy subprocess

Back to article view | Back to comp.lang.python

Contents

  PyPy subprocess LJ <luisjosenovoa@gmail.com> - 2015-11-01 13:27 -0800
    PyPy subprocess LJ <luisjosenovoa@gmail.com> - 2015-11-01 13:29 -0800
    Re: PyPy subprocess Chris Angelico <rosuav@gmail.com> - 2015-11-02 09:11 +1100

#98064 — PyPy subprocess

Hi All.

I hope you're having a good weekend. Im working on a large scale optimization problem which invokes the Gurobi solver from python. Before invoking the solver I use pure python to solve five subproblems in parallel using the multiprocessing module. These subproblems are the bottleneck of my procedure. Im wondering if there is a way in which I can use PyPy to solve the just subproblems in parallel, and return to CPython for the overall routines. The reason behind this, is that Gurobipy (python interface for the Gurobi optimization solver) is not compatible with PyPy, and Im seeing savings of 50% in time when using PyPy to somve the subproblems.

Any help would be highly appreciated.

Lj.

[toc] | [next] | [standalone]

#98065

By the way, Im using python 2.7.

Thanks

[toc] | [prev] | [next] | [standalone]

#98068

On Mon, Nov 2, 2015 at 8:27 AM, LJ <luisjosenovoa@gmail.com> wrote:
> Im wondering if there is a way in which I can use PyPy to solve the just subproblems in parallel, and return to CPython for the overall routines.
>

You could. What you'd have would be a setup where the subprocess is
utterly independent of the parent; the fact that they both happen to
be written in Python is almost immaterial. Figure out some way of
telling the subprocess what it needs to do (eg command-line parameters
coming through into sys.argv), and getting a response back (eg writing
something to stdout before terminating), and then you can use the
subprocess module to spin off the pypy processes, then (maybe after
doing some other work) wait for each of them to finish.

The subproblem solver could be written in any language at all, here;
there's complete decoupling. If you need to pass a lot of info down
into them, or up and out again, you'll need to serialize it all into
JSON and use stdin/stdout, or something like that. Takes some care,
but it works beautifully... I once had a "Python in Python" setup like
that to make my Python-based web site trigger something in a newer
version of Python than was shipped with my OS (think of a 2.7 invoking
a 3.5 script - I don't remember the actual versions but it was
something like that).

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web