Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed3.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.140 X-Spam-Level: * X-Spam-Evidence: '*H*': 0.72; '*S*': 0.00; 'debug': 0.07; 'puts': 0.07; 'logic': 0.09; 'sure,': 0.09; 'cc:addr:python-list': 0.11; 'concurrency': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'subject:skip:m 10': 0.16; 'elements': 0.16; 'sat,': 0.16; 'wrote:': 0.18; 'memory': 0.22; 'otherwise,': 0.22; 'cc:addr:python.org': 0.22; 'example.': 0.24; "shouldn't": 0.24; 'cc:2**0': 0.24; 'header:In-Reply-To:1': 0.27; 'michael': 0.29; 'am,': 0.29; 'bigger': 0.30; 'message-id:@mail.gmail.com': 0.30; 'code': 0.31; '(since': 0.31; 'run': 0.32; 'guess': 0.33; 'but': 0.35; 'received:google.com': 0.35; 'complete.': 0.36; 'module.': 0.36; 'should': 0.36; 'changing': 0.37; 'example,': 0.37; 'operating': 0.37; 'performance': 0.37; 'ends': 0.38; 'tasks': 0.38; 'handle': 0.38; 'rather': 0.38; 'even': 0.60; 'took': 0.61; "you're": 0.61; 'real': 0.63; 'taking': 0.65; 'world': 0.66; 'capable': 0.67; '2-3': 0.68; 'real-world': 0.68; '500': 0.70; 'consumer': 0.70; 'safe': 0.72; 'cut': 0.74; 'low': 0.83; '2015': 0.84; 'each,': 0.84; 'ridiculously': 0.84; 'shrinking': 0.84; 'welle': 0.84; 'to:none': 0.92 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type; bh=GpuBqrky95TLcILQB+IjpEiKUuVPXjS1XrKqP6hKVjM=; b=We/P+wf8qWKwhEwYWcwKnvQP2/LJHY1ZVtbRy0JL2UQFcTCWUfVVqUzrBFCRV2fc5v oXeMSfqaUMSrXVAKuVOcPXnxQgDdHXpEB5uGSAdRNILDHHJOMTNa4TRPZd3K2Of4R1nW uf2tsu++mpc4jua8wL04w4FHrwuFpBmSLyoNVn3xqo8WAftQlP9/TM3Yaa6JtM5ZLfEc XYSpAdea/R7xoXViqQhaZQHHI7/SF74Q1tShNNCZvwfZgjPXyRmTKJnd5uT5F7hCH8zJ F6lYbTCKKDdjx4rS/vzuOiN478kHpyi2qCJuJIpNqJZjPo92hwuzhPTI/+JWbSq8OCO/ q9LA== MIME-Version: 1.0 X-Received: by 10.50.108.115 with SMTP id hj19mr4996630igb.34.1431097524448; Fri, 08 May 2015 08:05:24 -0700 (PDT) In-Reply-To: References: Date: Sat, 9 May 2015 01:05:24 +1000 Subject: Re: multiprocessing, queue From: Chris Angelico Cc: "python-list@python.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.20+ Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 24 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1431097527 news.xs4all.nl 2849 [2001:888:2000:d::a6]:44516 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:90176 On Sat, May 9, 2015 at 12:31 AM, Michael Welle wrote: >> As a general rule, queues need to have both ends operating >> simultaneously, otherwise you're likely to have them blocking. In >> theory, your code should all work with ridiculously low queue sizes; >> the only cost will be concurrency (since you'd forever be waiting for >> the queue, so your tasks will all be taking turns). I tested this by >> changing the Queue() calls to Queue(1), and the code took about twice >> as long to complete. :) > ;) I know, as you might guess it's not a real world example. It's just > to explore the multiprocessing module. Sure, but even in a real-world example, it shouldn't ever be necessary to create huge queues. Larger queues allow for inconsistent performance of producer and/or consumer (eg if your consumer takes 1s to do each of 500 jobs, then 500s to do one job, it's capable of coping with a producer that puts one job on the queue every 2s, but only if the queue can handle ~250 jobs), but otherwise, the only effect of shrinking the queues is to force the processes into lock-step. Bigger queues mean that an over-performing producer will run you out of memory rather than get blocked. At very least, it should be a safe way to debug your logic - cut the queues down to just 2-3 elements each, and keep on printing out what's in the queue. ChrisA