Re: multiprocessing, pool, queue length

Path	csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From	Ian Kelly <ian.g.kelly@gmail.com>
Newsgroups	comp.lang.python
Subject	Re: multiprocessing, pool, queue length
Date	Mon, 21 Mar 2016 13:12:14 -0600
Lines	23
Message-ID	<mailman.456.1458587577.12893.python-list@python.org> (permalink)
References	<0qu4scx89u.ln2@news.c0t0d0s0.de>
Mime-Version	1.0
Content-Type	text/plain; charset=UTF-8
X-Trace	news.uni-berlin.de dT6HRluzII/rxAlB8AFpbQNZQb83kioLB/5ZbzMkQmiA==
Return-Path	<ian.g.kelly@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.006
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'received:209.85.223': 0.03; 'value,': 0.03; '21,': 0.07; 'problem?': 0.07; 'subject:skip:m 10': 0.09; 'value.': 0.15; '2016': 0.16; '4:25': 0.16; 'decrements': 0.16; 'finishes': 0.16; 'jobs.': 0.16; 'producer': 0.16; 'queue?': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'sentinel': 0.16; 'wrote:': 0.16; 'implementing': 0.18; 'produces': 0.22; 'am,': 0.23; 'seems': 0.23; 'header:In-Reply-To:1': 0.24; 'mon,': 0.24; "doesn't": 0.26; 'message-id:@mail.gmail.com': 0.27; 'queue': 0.29; 'task': 0.30; 'michael': 0.33; 'done,': 0.33; 'add': 0.34; 'received:google.com': 0.35; 'quite': 0.35; 'level': 0.35; 'asking': 0.35; 'received:209.85': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'thought': 0.37; 'received:209': 0.38; 'shared': 0.38; 'end': 0.39; 'does': 0.39; 'enough': 0.39; 'to:addr:python.org': 0.40; 'hello,': 0.40; 'more': 0.63; 'mar': 0.65; 'consumer': 0.67; 'jobs': 0.67; 'strategy': 0.69; 'low': 0.83; 'fast,': 0.84; 'to:name:python': 0.84; 'welle': 0.84
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=n4r2yYapmF0A5uozjZ5Wc72VRm5M5/i5ZY4xCdQcrFo=; b=rU+kRHs7tQ5J2QUz15jaDBk79Q+cvRcJNU8D27qafYDuhBJylO6gNUrRaAXc3oDVMO klMQySOeIAOq3XlSz04f8CzSPuCiRWAK+eLsnNnlwAoKI+HrhWmaB9OmPuqy/254PJD1 4P6Fq1Xh4ilskFbcUvFb7GaX5cj4I9x9QE9cubFvDTr+wQ7gezKtD6feW9t4q0QJJpjY 4GsFRo7UcLXgwzeSMvRzQIvnw6yoDHbS6M+z9qsGKcMM8pY+wlkDIqfich3F2OM3tASw nltruJdYdEC3HDB5ktf3wOr4vQgfZfJnBAJXLRqLgqj4X3M8Ek7OkTs3hrXkt6oLCb7O MitQ==
X-Google-DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=n4r2yYapmF0A5uozjZ5Wc72VRm5M5/i5ZY4xCdQcrFo=; b=iyU5E+bJHBU1NtRuJ/dCFlONw27S1f6O48ElQrqn9u+TDT5l+hkGtdYFbMgK8niVLQ w7KcXWA6/FVQ2/zk4gIVHP80AtII5FCddXKlwx3iksI+zc4jXg/JMhSESxAnSspBl4HD 5TdmibR/eI00pYf+YEncB7lnkOzG28gk3cBBL4/Uj7sAXa3gaZkPSY401lFQ6MCMmKqF G+4usMjTSBx9ssHdENEyYIXGx0jjj4S1IwGlWwveviHAw+6rV9fC3D5IJj+4qMiJ56Q3 ncbez3wDWorryyRD2F+wlXg/VJRdYPu4z9yMPf0tkSGgvJQlm/tgQ49ayLqTXle4Nwgl rtxQ==
X-Gm-Message-State	AD7BkJK+5H2DrCmhU5/O9wg7P7M/mSQvB2a5mROzMgdpmKmLIcuhLIqs2g5JaaqVe9cGjfhfeplJyC5cYcgeKg==
X-Received	by 10.107.11.10 with SMTP id v10mr29621262ioi.188.1458587574505; Mon, 21 Mar 2016 12:12:54 -0700 (PDT)
In-Reply-To	<0qu4scx89u.ln2@news.c0t0d0s0.de>
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.21
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref	csiph.com comp.lang.python:105393

Show key headers only | View raw

On Mon, Mar 21, 2016 at 4:25 AM, Michael Welle <mwe012008@gmx.net> wrote:
> Hello,
>
> I use a multiprocessing pool. My producer calls pool.map_async()
> to fill the pool's job queue. It can do that quite fast, while the
> consumer processes need much more time to empty the job queue. Since the
> producer can create a lot of jobs, I thought about asking the pool for
> the amount of jobs it has in its queue and then only produce more jobs
> if the current value is below a threshold. It seems like the pool
> doesn't want to tell me the level of the queue, does it? What is a
> better strategy to solve this problem? Implementing a pool around
> multiprocessing's Process and Queue?

A simple solution would be to have a shared multiprocessing.Value that
tracks how many items are in the pool. Whenever the producer produces
items it increments the Value, and whenever a consumer finishes a job
it decrements the Value.

An alternative solution that doesn't require adding a small amount of
work to every job would be to have the producer add a sentinel task
that does nothing at or near the end of the batch, and either wait on
the result or check it periodically. When it's done, then the pool is
low enough to add more jobs.

Thread

multiprocessing, pool, queue length Michael Welle <mwe012008@gmx.net> - 2016-03-21 11:25 +0100
  Re: multiprocessing, pool, queue length Ian Kelly <ian.g.kelly@gmail.com> - 2016-03-21 13:12 -0600
    Re: multiprocessing, pool, queue length Michael Welle <mwe012008@gmx.net> - 2016-03-21 20:46 +0100
      Re: multiprocessing, pool, queue length Ian Kelly <ian.g.kelly@gmail.com> - 2016-03-21 15:24 -0600
        Re: multiprocessing, pool, queue length Michael Welle <mwe012008@gmx.net> - 2016-03-22 07:19 +0100

csiph-web