Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed3a.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Date: Tue, 12 Aug 2014 19:43:15 +0100
From: MRAB <python@mrabarnett.plus.com>
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: python-list@python.org
Subject: Re: Using asyncio workers in a `concurrent.futures` interface
References: <d6e2ab63-c9bc-4271-abac-17e32afa4c88@googlegroups.com>
In-Reply-To: <d6e2ab63-c9bc-4271-abac-17e32afa4c88@googlegroups.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.12891.1407869004.18130.python-list@python.org>
Lines: 48
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:76131

On 2014-08-12 18:02, cool-RR wrote:
> Hello everybody! I have a question.
>
> I have a Django app running on Heroku. I need to run about 100 worker
> threads there to do uploads/downloads simultaneously. A Heroku Dyno
> has only 512MB of memory, so I'm reluctant to run 100 worker threads.
> (I've had Dynos crash from lack of memory when using 6 threads
> before.)
>
> I heard that the asyncio module is mature and ready for usage, and I
> was happy because I kept hearing about it in the last year, and I saw
> Guido's lecture about it. If I understand correctly it would let me
> run multiple uploads and downloads efficiently in one thread, which
> would conserve more resources than using threads. (Please correct me
> if I'm wrong.)
>
> Now, I am a little clueless about the whole way it's built, using
> coroutines and tricky usage of `yield from`. I figured that since
> this is eventually a library for concurrency, i.e. doing many tasks
> at the same time, there will be an API in a style of "Here are 100
> tasks for you to do concurrently, let me know when they're done."
>
> I looked at the asyncio documentation page and saw that it does
> mention futures and executors, which is my favorite interface for
> doing concurrency. I was happy and I skimmed the docs. But, I
> couldn't find a simple way to use these. I don't want to learn how to
> define coroutines and use `yield from` to switch between them. (I use
> `yield from` regularly and fully understand how it works, I just
> don't write my programs that way.)
>
> What I'm expecting is something like this:
>
>      download_file = lambda url: requests.get(url).content
>      urls = ['http://google.com/file1.jpg', 'http://google.com/file2.jpg', 'http://google.com/file3.jpg'] # etc.
>
>      with AsyncIOExecutor() as asyncio_executor:
>          files = asyncio_executor.map(download_file, urls)
>
> And that's it, no coroutines, no `yield from`. Since, if I understand
> correctly, asyncio requires a mainloop, it would make sense for the
> AsyncIOExecutor to have a thread of its own in which it could run its
> mainloop.
>
> Is this possible? Did someone implement this?
>
Do you really need to upload/download that many at the same time? I'd
put them into a queue and have only a few of uploading/downloading at
any time.