Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail From: justin walters Newsgroups: comp.lang.python Subject: Re: python - handling HTTP requests asynchronously Date: Fri, 6 May 2016 08:23:01 -0700 Lines: 42 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Trace: news.uni-berlin.de 2YD/tt+t+ff0SMo1LNcOCAtFwdSWiptflpVxH/IunMRA== Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.006 X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'api.': 0.04; 'concurrently': 0.07; 'api': 0.09; 'concurrent': 0.09; 'non- blocking': 0.09; 'subject:skip:a 10': 0.09; 'url:github': 0.09; 'django': 0.13; 'subject:python': 0.14; 'everyone,': 0.15; 'thu,': 0.15; '2016': 0.16; 'entries.': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'subject:handling': 0.16; 'subject:requests': 0.16; 'used:': 0.16; 'wrote:': 0.16; 'app': 0.16; 'memory': 0.17; 'module,': 0.18; 'email addr:gmail.com>': 0.18; 'assign': 0.22; 'tried': 0.24; 'header:In-Reply-To:1': 0.24; 'requests': 0.25; 'command': 0.26; 'rest': 0.26; 'external': 0.27; '(e.g.': 0.27; 'message-id:@mail.gmail.com': 0.27; 'fastest': 0.27; 'for,': 0.29; 'handled': 0.29; 'connection': 0.30; 'url:mailman': 0.30; 'transaction': 0.30; 'e.g.': 0.30; 'task': 0.30; 'entry': 0.31; 'anybody': 0.32; 'run': 0.33; 'url:python': 0.33; 'http': 0.33; "i'll": 0.33; 'similar': 0.33; 'url:listinfo': 0.34; 'this?': 0.34; 'received:google.com': 0.35; 'should': 0.36; 'url:org': 0.36; 'received:209.85': 0.36; 'to:addr:python-list': 0.36; 'pm,': 0.36; 'subject:: ': 0.37; 'skip:& 10': 0.37; 'done.': 0.37; 'pdf': 0.37; 'thought': 0.37; 'received:209': 0.38; 'url:mail': 0.40; 'well.': 0.40; 'to:addr:python.org': 0.40; 'still': 0.40; 'back': 0.62; 'different': 0.63; 'management': 0.64; 'goal': 0.64; 'between': 0.65; 'here': 0.66; 'receive': 0.71; '100': 0.79; '30k': 0.84; 'route': 0.84; "there'll": 0.84; 'generation.': 0.91; 'task,': 0.91 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=BuwOD0skB1XVTdo8xt+kpEA5etvOBXcdKl3lRNNyOgc=; b=UA94YPX0RXfDSAWTD9HjtgmP5z2TcaL0CnqJZYsJPZNGtNsXylzZBGnUTnQpqHIUpC rX3EuqoID6qflX+CJU0RW4cmYJGinHxQTLLh6ybd5YjUwi4Gc0qUWblc7fzA6GpRV8EZ JryxHUMf0JKvTUozepyO4YQqBH15+wqVOxCcAlS1ZdWioHBOY7C1NtB7x4Njw7ZypyNU vUHyOPXsilBFGKI/XyPyqC7YH6PzI3txAY4HaAeCbAiUJ/ak4G+oJKcrntU8PvFCxKD8 atRnb3XP53nJznbsJph8GuhWb8nGbtA9qgtMY4SEnLFejbrKBQDA+TVIcUPnMi6l5Ij3 u2zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=BuwOD0skB1XVTdo8xt+kpEA5etvOBXcdKl3lRNNyOgc=; b=X76fYIkBD4Irtjw9UkvXSgbR9KcjHX7QG+NiUJIul87kQSSGPES0xErSeIBCelWIZP kHG6DMTGCrAKdPwh/K5m7l9nP8Q5HwaMyiSsGuLnQPYpYs80c/39l0odMKp3uJ1PcXQi fgNV/Y5V5145VyYfM/38+bb+mn+9Hm7zUTerr4iPTIT9bPQ+WnxNwCqkLWXx9ssdyxq1 GhILgK3JN57+VtC7SUSqEKesYtbaKcG5iQtpb9XZ599QPPJbuOR86hsaNJ8eA0U7FU32 HnTvBuCUf7BRMduxzue6sH40kwVsH7XpZrhTkz5wWQA3uu+YuU+tjlRle/ZuGUkQFYhx xPXg== X-Gm-Message-State: AOPr4FVBsrARobrV2im5XAV0eOWkOZMBTdSmcAOb4OqfVIZV5ypiM7+N4bdI5CWsMYyAuXxGpfFcAVestLtuRg== X-Received: by 10.25.77.204 with SMTP id a195mr10560042lfb.14.1462548181635; Fri, 06 May 2016 08:23:01 -0700 (PDT) In-Reply-To: X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Mailman-Original-Message-ID: X-Mailman-Original-References: Xref: csiph.com comp.lang.python:108231 On Thu, May 5, 2016 at 11:56 PM, wrote: > Hi everyone, > I need to generate a PDF report for each entry of a django queryset. > There'll be between between 30k and 40k entries. > > The PDF is generated through an external API. Since currently is generated > on demand, this is handled synchronously via an HTTP request/response. That > will be different for this task, since I think I'll use a django management > command to loop through the queryset and perform the PDF generation. > > Which approach should I follow for this task? I thought about 3 possibile > solutions, although are technologies that I never used: > > 1) Celery: assign a task (http request with a different payload) to a > worker, then retrieve it once it's done. > > 2) request-futures: using requests in a non-blocking way. > > 3) multiprocessing module, with e.g. 10 as workers limit. > > > the goal is to use the API concurrently (e.g. send 10 or 100 http requests > simultaneously, depending on how many concurrent requests the API can > handle). > > Anybody here that handled a similar task and can give advices on how to > proceed on this? > -- > https://mail.python.org/mailman/listinfo/python-list > Have you tried channels: https://github.com/andrewgodwin/channels ? If it's an asyncronous request/response cycle you're looking for, it should work well. Essentially, you have worker processes that receive a message over a websocket connection and then send the new message back to the "group". I would recommend using the redis in memory transaction layer if you go this route as it is the fastest and most efficient. The best part about channels is that you can still run a normal django app alongside it. You can have only one app use websockets while the rest use standard http..