Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #70978

multiprocessing in a while loop?

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

hi, I am struggling to understand how to leverage python's multiprocessing module in a while loop.  the examples I have found seem to assume it is known ahead of time how many items need to be processed.

specifically, I am reading from an external queue.  I currently process items one at a time until the queue is empty.  I wrote a wrapper function to handle connecting to the queue, pulling the next message, and deleting it when I am done.  ideally, I'd like to modify this wrapper function to take an additional argument (processes) to specify the number of messages to process simultaneously.

I've attached a script that captures what I am doing now.  unfortunately, the external queue object is not publicly accessible and I'm not quite sure how to set up a local object that would support testing.  any suggestions would be most welcome.


thanks,
Johan

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

multiprocessing in a while loop? Johan Llewellyn <johan.llewellyn@yahoo.com> - 2014-05-06 11:45 -0700
  Re: multiprocessing in a while loop? Jesse Adam <jaahush@gmail.com> - 2014-06-27 16:27 -0700

csiph-web