Path: csiph.com!usenet.pasdenom.info!news.redatomik.org!newsfeed.xs4all.nl!newsfeed7.news.xs4all.nl!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.010 X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'reason,': 0.07; 'exit.': 0.09; 'fetch': 0.09; 'item,': 0.09; 'threads,': 0.09; 'throws': 0.09; 'url:github': 0.09; 'thread': 0.10; 'exception': 0.13; 'appropriate': 0.14; 'everyone,': 0.15; 'producing': 0.15; 'server,': 0.15; 'cleanly': 0.16; 'finish.': 0.16; 'think?': 0.16; 'threads': 0.16; 'trying': 0.22; 'sets': 0.23; 'script': 0.25; 'message-id:@mail.gmail.com': 0.27; 'errors.': 0.27; 'locks': 0.29; 'queue': 0.29; "i'm": 0.30; 'subject:/': 0.30; 'implement': 0.32; 'point': 0.33; 'stream': 0.33; 'requirements': 0.35; 'received:google.com': 0.35; 'quite': 0.35; 'item': 0.35; 'sometimes': 0.35; 'but': 0.36; 'should': 0.36; 'to:addr:python- list': 0.36; 'version': 0.38; 'to:addr:python.org': 0.40; 'subject:with': 0.40; 'some': 0.40; 'avoid': 0.61; 'here:': 0.63; 'response.': 0.66; 'consumer': 0.67; 'special': 0.73; 'doubts': 0.84; 'housekeeping': 0.84; 'ugly,': 0.84; 'interrupt': 0.91; 'items,': 0.91 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=UEYH3neJPzyiuQ7ZnPX7PZMft+Ki9tQjq+5JQR6GmTo=; b=UtrL9NznoK17pcHS+I8oPCQlXu7lZ6ZgQFOOFM37gb6NcjgV9TQJSd4ZkTFyDy39pS KgkNJeNW5+169opU9sYUoq8KP8ZUa4p6ZCqKGB7GKdHnP+tdztqbewiyA7tSUkSbX+Cz cKKA3kztqUBatlgnyOKAO1vzrLHjL+Tx8z7dfF7Tkbnp/fHpP6YBzZFKJV2b3HRmMJdv PNl/HvSc2QFxH/EFNQcH1pjsefAvccg3NI9x8ohU94kmX299lRv4QbGr97EZsLvtE+rz K/q087++eIY1A4qJX/Y2hmcGju1UHpZoNQgm1sx55MvnMweWzZxT5KhFEK7b24TVFxmt 4LiA== MIME-Version: 1.0 X-Received: by 10.107.40.147 with SMTP id o141mr6125771ioo.83.1438721383001; Tue, 04 Aug 2015 13:49:43 -0700 (PDT) Date: Tue, 4 Aug 2015 17:49:42 -0300 Subject: consumer/producer with asyncio From: David Rios To: python-list@python.org Content-Type: text/plain; charset=UTF-8 X-Mailman-Approved-At: Wed, 05 Aug 2015 11:55:16 +0200 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.20+ Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 29 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1438768517 news.xs4all.nl 2902 [2001:888:2000:d::a6]:38797 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:95008 Hello everyone, I'm trying to implement a producer/consumer using asyncio, but I have some doubts about the best way to do it. I already have a working implementation using threads, and want to port it to asyncio for fun, to learn and also to avoid some problems that I have using threads, namely that sometimes the script locks and never finish. What the script have to do is to connect to a server, stream content from it, and send it for processing as items arrive. The processing part is to connect to other servers based on the item, send the item to process and wait for the response. The threaded version sets a pool of consumer threads that fetch items from a queue, while on the main thread the queue is populated. Special requirements that I have are that the script should be interruptible and should stop cleanly on errors. When I interrupt it or any part throws an exception for any reason, the script should stop producing items, the workers should finish the current job and then stop processing new items, at which point the script should perform appropriate housekeeping and exit. I have a mock implementation using asyncio here: https://gist.github.com/davidrios/011d044b5e7510f085dd But I think it is quite ugly, particularly all the nesting. What do you think?