Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #6095

Re: Multiprocessing: don't push the pedal to the metal?

Show all headers | View raw

On Mon, 2011-05-23 at 12:51 -0700, John Ladasky wrote:
> On May 23, 2:50 am, Adam Tauno Williams <awill...@whitemice.org>
> wrote:
> > I develop an app that uses multiprocessing heavily.  Remember that all
> > these processes are processes - so you can use all the OS facilities
> > regarding processes on them.  This includes setting nice values,
> > schedular options, CPU pinning, etc...
> That's interesting.  Does code exist in the Python library which
> allows the adjustment of CPU pinning and nice levels?  I just had
> another look at the multiprocessing docs, and also at os.subprocess.
> I didn't see anything that pertains to these issues.

"in the Python library" - no.  All these types of behaviors are platform
specific.

For example you can set the "nice" (priority) of a UNIX/LINUX process
using the nice method from the os module.  Our workflow engine does this
on all worker processes it starts - it sends the workers to the lowest
priority.


from os                    import nice as os_priority
...
        try:
            os_priority(20)
        except Exception, e:
            ...

I'm not aware of a tidy way to call sched_setaffinity from Python; but
my own testing indicates that the LINUX kernel is very good at figuring
this out on its own so long as it isn't swamped.  Queuing, rather than
starting, additional workflows if load average exceeds X.Y and setting
the process priority of workers to very-low seems to work very well.

There is <http://pypi.python.org/pypi/affinity> for setting affinity,
but I haven't used it.

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Multiprocessing: don't push the pedal to the metal? John Ladasky <ladasky@my-deja.com> - 2011-05-21 20:58 -0700
  Re: Multiprocessing: don't push the pedal to the metal? John Ladasky <ladasky@my-deja.com> - 2011-05-22 14:06 -0700
    Re: Multiprocessing: don't push the pedal to the metal? Chris Angelico <rosuav@gmail.com> - 2011-05-23 10:32 +1000
    Re: Multiprocessing: don't push the pedal to the metal? Adam Tauno Williams <awilliam@whitemice.org> - 2011-05-23 05:50 -0400
      Re: Multiprocessing: don't push the pedal to the metal? John Ladasky <ladasky@my-deja.com> - 2011-05-23 12:51 -0700
        Re: Multiprocessing: don't push the pedal to the metal? Adam Tauno Williams <awilliam@whitemice.org> - 2011-05-23 16:46 -0400

csiph-web