Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #105346

multiprocessing, pool, queue length

Show all headers | View raw

Hello,

I use a multiprocessing pool. My producer calls pool.map_async()
to fill the pool's job queue. It can do that quite fast, while the
consumer processes need much more time to empty the job queue. Since the
producer can create a lot of jobs, I thought about asking the pool for
the amount of jobs it has in its queue and then only produce more jobs
if the current value is below a threshold. It seems like the pool
doesn't want to tell me the level of the queue, does it? What is a
better strategy to solve this problem? Implementing a pool around
multiprocessing's Process and Queue?

Regards
hmw

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

multiprocessing, pool, queue length Michael Welle <mwe012008@gmx.net> - 2016-03-21 11:25 +0100
  Re: multiprocessing, pool, queue length Ian Kelly <ian.g.kelly@gmail.com> - 2016-03-21 13:12 -0600
    Re: multiprocessing, pool, queue length Michael Welle <mwe012008@gmx.net> - 2016-03-21 20:46 +0100
      Re: multiprocessing, pool, queue length Ian Kelly <ian.g.kelly@gmail.com> - 2016-03-21 15:24 -0600
        Re: multiprocessing, pool, queue length Michael Welle <mwe012008@gmx.net> - 2016-03-22 07:19 +0100

csiph-web