Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90176

Re: multiprocessing, queue

Show key headers only | View raw

On Sat, May 9, 2015 at 12:31 AM, Michael Welle <mwe012008@gmx.net> wrote:
>> As a general rule, queues need to have both ends operating
>> simultaneously, otherwise you're likely to have them blocking. In
>> theory, your code should all work with ridiculously low queue sizes;
>> the only cost will be concurrency (since you'd forever be waiting for
>> the queue, so your tasks will all be taking turns). I tested this by
>> changing the Queue() calls to Queue(1), and the code took about twice
>> as long to complete. :)
> ;) I know, as you might guess it's not a real world example. It's just
> to explore the multiprocessing module.

Sure, but even in a real-world example, it shouldn't ever be necessary
to create huge queues. Larger queues allow for inconsistent
performance of producer and/or consumer (eg if your consumer takes 1s
to do each of 500 jobs, then 500s to do one job, it's capable of
coping with a producer that puts one job on the queue every 2s, but
only if the queue can handle ~250 jobs), but otherwise, the only
effect of shrinking the queues is to force the processes into
lock-step. Bigger queues mean that an over-performing producer will
run you out of memory rather than get blocked. At very least, it
should be a safe way to debug your logic - cut the queues down to just
2-3 elements each, and keep on printing out what's in the queue.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

multiprocessing, queue Michael Welle <mwe012008@gmx.net> - 2015-05-08 12:08 +0200
  Re: multiprocessing, queue Chris Angelico <rosuav@gmail.com> - 2015-05-08 23:34 +1000
    Re: multiprocessing, queue Michael Welle <mwe012008@gmx.net> - 2015-05-08 16:31 +0200
      Re: multiprocessing, queue Chris Angelico <rosuav@gmail.com> - 2015-05-09 01:05 +1000

csiph-web