Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #70978 > unrolled thread

multiprocessing in a while loop?

Back to article view | Back to comp.lang.python

Contents

  multiprocessing in a while loop? Johan Llewellyn <johan.llewellyn@yahoo.com> - 2014-05-06 11:45 -0700
    Re: multiprocessing in a while loop? Jesse Adam <jaahush@gmail.com> - 2014-06-27 16:27 -0700

#70978 — multiprocessing in a while loop?

[Multipart message — attachments visible in raw view] — view raw

hi, I am struggling to understand how to leverage python's multiprocessing module in a while loop.  the examples I have found seem to assume it is known ahead of time how many items need to be processed.

specifically, I am reading from an external queue.  I currently process items one at a time until the queue is empty.  I wrote a wrapper function to handle connecting to the queue, pulling the next message, and deleting it when I am done.  ideally, I'd like to modify this wrapper function to take an additional argument (processes) to specify the number of messages to process simultaneously.

I've attached a script that captures what I am doing now.  unfortunately, the external queue object is not publicly accessible and I'm not quite sure how to set up a local object that would support testing.  any suggestions would be most welcome.


thanks,
Johan

[toc] | [next] | [standalone]

#73679

Could you post 
a) what the output looks like now (sans the logging part)
b) what output do you expect


In any event, this routine does not look right to me:

def consume_queue(queue_name):
  conn = boto.connect_sqs()
  q = conn.get_queue(queue_name)
  m = q.read()
  while m is not None:
    yield m
    q.delete_message(m)
    logger.debug('message deleted')
    m = q.read()

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web