Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail From: Ian Kelly Newsgroups: comp.lang.python Subject: Re: multiprocessing, pool, queue length Date: Mon, 21 Mar 2016 15:24:42 -0600 Lines: 15 Message-ID: References: <0qu4scx89u.ln2@news.c0t0d0s0.de> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Trace: news.uni-berlin.de wqGHNwdRmh38JpDcbD7rzwzyKdCsMZp88BB3Qyy1Rq5Q== Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.009 X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'received:209.85.223': 0.03; 'essentially': 0.04; '21,': 0.07; 'subject:skip:m 10': 0.09; '2016': 0.16; 'couples': 0.16; 'producer': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'sentinel': 0.16; 'still,': 0.16; 'wrote:': 0.16; '(or': 0.23; 'header:In-Reply-To:1': 0.24; 'mon,': 0.24; 'message-id:@mail.gmail.com': 0.27; 'idea': 0.28; 'fast.': 0.29; 'finds': 0.29; 'task': 0.30; 'another': 0.32; 'michael': 0.33; 'received:google.com': 0.35; 'but': 0.36; 'should': 0.36; 'instead': 0.36; 'received:209.85': 0.36; 'to:addr:python-list': 0.36; 'pm,': 0.36; 'subject:: ': 0.37; 'creation': 0.38; 'no,': 0.38; 'received:209': 0.38; 'mean': 0.38; 'means': 0.39; 'to:addr:python.org': 0.40; 'more': 0.63; 'mar': 0.65; 'to:name:python': 0.84; 'welle': 0.84 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=rFGvAwGGr3BKDfL7YZqs5NrOqfNiZCJyb7UHiDXTfgU=; b=bMaQCY1R6uZhJM6RbIJQvWQ+9QNo+6OD+8Cwaaf/zrtpIafrAOcwjjVAVAaBWvD9Os YVAegSVWIUthubLcW7GNkvWKxTYQCufuFx8h135ZU/7TpFdIihQz8Rg2DRbuIOD2WcEN TecGVT1zWad7eZvipNDGi8Z8sWt629/1rZpEeLtyZ34o9ww4R5SRRdIHtBZTlRKghRdd AJeguH003YeFNCAWoRBzSJ6rOnPfaEaV5dxVLgft9auoAbpR6s7aQyiKyrOUwgkSmXmp rlv3EfNvEpKmTSiD5yvb0tsZxrkO7whZdD8f/4KdOHOVkkqAY1fEzew49Ku+SsmttxCw CNsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=rFGvAwGGr3BKDfL7YZqs5NrOqfNiZCJyb7UHiDXTfgU=; b=ZWm2vvGxAHULhtVHJhaErtoGicxMBNaE8hV8fkvCmVrBboCoGeKVZ/YvKBF55rQYns 9NjlNNJk3W50Uk5DvumCO8g+ARXnZHy8yVLQ3v917+9Nk0gnVr9CmjlQKCgzo7O0LKHP sApEVPbSps5LupOD11NxgtVMmXSHwuWLMyYjbe9fMpCt2Xn+gQsI9oyOB+PPV3qLaP2Z HzlsQQonkEUSzM36eUkQBCSReXX5u64ncZw2R3RngQKWQ72GuEU/5b/Ttky4HGH7D9D7 7r4a5Oazmss8r54KxpQycoglMyKtb/ttnmQuNb8NGEpbTk+R1DBVaPBhKJ5texX1iU9n Toww== X-Gm-Message-State: AD7BkJLpgudPm3aMDNfaaY4YWrDoTjURVV3y+BVm+rC6fcaPJpi5nbv1GQFMPcSyRtTNRu6UWhbNI1M3E6N0uw== X-Received: by 10.107.11.148 with SMTP id 20mr31615117iol.111.1458595522179; Mon, 21 Mar 2016 14:25:22 -0700 (PDT) In-Reply-To: X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Xref: csiph.com comp.lang.python:105409 On Mon, Mar 21, 2016 at 1:46 PM, Michael Welle wrote: > Wait on the result means to set a multiprocessing.Event if one of the > consumers finds the sentinel task and wait for it on the producer? Hmm, > that might be better than incrementing a counter. But still, it couples > the consumers and the producer more than I like. No, I mean calling AsyncResult.wait() on the result of the sentinel task (or just calling Pool.apply instead of Pool.apply_async in the first place). > Another idea that I had is to use map() instead of map_async() and then > put the producer in its own process. That should work if job creation is > fast. Essentially the same thing.