Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #76124

Using asyncio workers in a `concurrent.futures` interface

Show all headers | View raw

Hello everybody! I have a question. 

I have a Django app running on Heroku. I need to run about 100 worker threads there to do uploads/downloads simultaneously. A Heroku Dyno has only 512MB of memory, so I'm reluctant to run 100 worker threads. (I've had Dynos crash from lack of memory when using 6 threads before.) 

I heard that the asyncio module is mature and ready for usage, and I was happy because I kept hearing about it in the last year, and I saw Guido's lecture about it. If I understand correctly it would let me run multiple uploads and downloads efficiently in one thread, which would conserve more resources than using threads. (Please correct me if I'm wrong.) 

Now, I am a little clueless about the whole way it's built, using coroutines and tricky usage of `yield from`. I figured that since this is eventually a library for concurrency, i.e. doing many tasks at the same time, there will be an API in a style of "Here are 100 tasks for you to do concurrently, let me know when they're done."

I looked at the asyncio documentation page and saw that it does mention futures and executors, which is my favorite interface for doing concurrency. I was happy and I skimmed the docs. But, I couldn't find a simple way to use these. I don't want to learn how to define coroutines and use `yield from` to switch between them. (I use `yield from` regularly and fully understand how it works, I just don't write my programs that way.)

What I'm expecting is something like this: 
    
    download_file = lambda url: requests.get(url).content
    urls = ['http://google.com/file1.jpg', 'http://google.com/file2.jpg', 'http://google.com/file3.jpg'] # etc.
    
    with AsyncIOExecutor() as asyncio_executor:
        files = asyncio_executor.map(download_file, urls)

And that's it, no coroutines, no `yield from`. Since, if I understand correctly, asyncio requires a mainloop, it would make sense for the AsyncIOExecutor to have a thread of its own in which it could run its mainloop.

Is this possible? Did someone implement this? 


Thanks for your help,
Ram.

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Using asyncio workers in a `concurrent.futures` interface cool-RR <ram.rachum@gmail.com> - 2014-08-12 10:02 -0700
  Re: Using asyncio workers in a `concurrent.futures` interface Marko Rauhamaa <marko@pacujo.net> - 2014-08-12 21:31 +0300
  Re: Using asyncio workers in a `concurrent.futures` interface MRAB <python@mrabarnett.plus.com> - 2014-08-12 19:43 +0100
  Re: Using asyncio workers in a `concurrent.futures` interface Ian Kelly <ian.g.kelly@gmail.com> - 2014-08-12 15:31 -0600
    Re: Using asyncio workers in a `concurrent.futures` interface Marko Rauhamaa <marko@pacujo.net> - 2014-08-13 08:03 +0300
      Re: Using asyncio workers in a `concurrent.futures` interface Ian Kelly <ian.g.kelly@gmail.com> - 2014-08-13 00:03 -0600

csiph-web