Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #6006

Re: Multiprocessing: don't push the pedal to the metal?

Show all headers | View raw

Following up to my own post...

Flickr informs me that quite a few of you have been looking at my
graphs of performance vs. the number of sub-processes employed in a
parallelizable task:

On May 21, 8:58 pm, John Ladasky <lada...@my-deja.com> wrote:
> http://www.flickr.com/photos/15579975@N00/5744093219
[...]
> I'll quickly ask my question first, to avoid a TL;DR problem: when you
> have a multi-core CPU with N cores, is it common to see the
> performance peak at N-1, or even N-2 processes?  And so, should you
> avoid using quite as many processes as there are cores?  I was
> expecting diminishing returns for each additional core -- but not
> outright declines.

But no one has offered any insight yet?  Well, I slept on it, and I
had a thought.  Please feel free to shoot it down.

If I spawn N worker sub-processes, my application in fact has N+1
processes in all, because there's also the master process itself.  If
the master process has anything significant to do (and mine does, and
I would surmise that many multi-core applications would be that way),
then the master process may sometimes find itself competing for time
on a CPU core with a worker sub-process.  This could impact
performance even when the demands from the operating system and/or
other applications are modest.

I'd still appreciate hearing from anyone else who has more experience
with multiprocessing.  If there are general rules about how to do this
best, I haven't seen them posted anywhere.  This may not be a Python-
specific issue, of course.

Tag, you're it!

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Multiprocessing: don't push the pedal to the metal? John Ladasky <ladasky@my-deja.com> - 2011-05-21 20:58 -0700
  Re: Multiprocessing: don't push the pedal to the metal? John Ladasky <ladasky@my-deja.com> - 2011-05-22 14:06 -0700
    Re: Multiprocessing: don't push the pedal to the metal? Chris Angelico <rosuav@gmail.com> - 2011-05-23 10:32 +1000
    Re: Multiprocessing: don't push the pedal to the metal? Adam Tauno Williams <awilliam@whitemice.org> - 2011-05-23 05:50 -0400
      Re: Multiprocessing: don't push the pedal to the metal? John Ladasky <ladasky@my-deja.com> - 2011-05-23 12:51 -0700
        Re: Multiprocessing: don't push the pedal to the metal? Adam Tauno Williams <awilliam@whitemice.org> - 2011-05-23 16:46 -0400

csiph-web