Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #20712 > unrolled thread

asynchronous downloading

Back to article view | Back to comp.lang.python

Contents

  asynchronous downloading Plumo <richardbp@gmail.com> - 2012-02-22 22:58 -0800
    Re: asynchronous downloading Justin Ezequiel <justin.mailinglists@gmail.com> - 2012-02-22 23:25 -0800
    Re: asynchronous downloading Mark Hammond <mhammond@skippinet.com.au> - 2012-02-23 22:20 +1100
    Re: asynchronous downloading Paul Rubin <no.email@nospam.invalid> - 2012-02-23 03:46 -0800
      Re: asynchronous downloading Plumo <richardbp@gmail.com> - 2012-02-23 16:28 -0800
    Re: asynchronous downloading Richard Baron Penman <richardbp@gmail.com> - 2012-02-24 00:50 +1100
    Re: asynchronous downloading Giampaolo Rodolà <g.rodola@gmail.com> - 2012-02-23 18:31 +0100
      Re: asynchronous downloading Plumo <richardbp@gmail.com> - 2012-02-23 17:10 -0800
      Re: asynchronous downloading Plumo <richardbp@gmail.com> - 2012-02-23 17:10 -0800
        Re: asynchronous downloading Fayaz Yusuf Khan <fayaz.yusuf.khan@gmail.com> - 2012-02-23 18:31 -0800
        Re: asynchronous downloading Giampaolo Rodolà <g.rodola@gmail.com> - 2012-02-24 10:03 +0100
        Re: asynchronous downloading Richard Baron Penman <richardbp@gmail.com> - 2012-02-26 08:44 +1100

#20712 — asynchronous downloading

I want to download content asynchronously. This would be straightforward to do threaded or across processes, but difficult asynchronously so people seem to rely on external libraries (twisted / gevent / eventlet). 

(I would use gevent under different circumstances, but currently need to stick to standard libraries.)

I looked around and found there is little interest in developing a proper HTTP client on asyncore. The best I found stopped development a decade ago: http://sourceforge.net/projects/asynchttp/ 

What do you recommend? 
And why is there poor support for asynchronous execution?

Richard

[toc] | [next] | [standalone]

#20713

have you seen http://www.doughellmann.com/PyMOTW/asyncore/

[toc] | [prev] | [next] | [standalone]

#20722

On 23/02/2012 5:58 PM, Plumo wrote:
> I want to download content asynchronously. This would be
> straightforward to do threaded or across processes, but difficult
> asynchronously so people seem to rely on external libraries (twisted
> / gevent / eventlet).

Exactly - the fact it's difficult is why those tools compete.

> (I would use gevent under different circumstances, but currently need
> to stick to standard libraries.)

As above - use threads or processes - they are fine for relatively 
modest tasks.  If your needs go beyond modest, I'd reevaluate your need 
to stick with just the stdlib - even demanding *sync* http apps often 
wind up using modules outside the stdlib.  Look into virtualenv etc if 
permission to install packages is the issue.

Batteries included free, but turbo-chargers are an extra ;)

Mark

[toc] | [prev] | [next] | [standalone]

#20723

Plumo <richardbp@gmail.com> writes:
> What do you recommend? 

Threads.

> And why is there poor support for asynchronous execution?

The freenode #python crowd seems to hate threads and prefer twisted,
which seems to have the features you want and probably handles very
large #'s of connections better than POSIX threads do.  But I find the
whole event-driven model to be an annoying abstraction inversion and
threads to be simpler, so I've stayed with threads.  I keep hearing
boogieman stories about the evil hazards of race conditions etc. but
none of that stuff has ever happened to me (yet) as far as I can tell.
The main thing is to avoid sharing mutable data between threads to the
extent that you can.  Keep the threads isolated from each other except
for communication through Queues and not too much can go wrong.  The
last program I wrote had around 20 threads and one or two condition
variables and I don't think any significant bugs resulted from that.

FWIW, the Erlang language is built around the above concept, it uses
super-lightweight userland threads so it can handle millions of them
concurrently, and it's used successfully for ultra-high-reliability
phone switches and similar applications that are not allowed to fail, so
it must be doing something right.

There are a few schemes like Camaelia (sp?) implementing concurrency
with Python generators or coroutines, but I think they're not widely
used, and Python coroutines are kind of crippled because they don't
carry any stack below their entry point.

[toc] | [prev] | [next] | [standalone]

#20768

My current implementation works fine below a few hundred threads. But each thread takes up a lot of memory so does not scale well. 

I have been looking at Erlang for that reason, but found it is missing useful libraries in other areas.

[toc] | [prev] | [next] | [standalone]

#20727

>> I want to download content asynchronously. This would be
>> straightforward to do threaded or across processes, but difficult
>> asynchronously so people seem to rely on external libraries (twisted
>> / gevent / eventlet).
>
>
> Exactly - the fact it's difficult is why those tools compete.

It is difficult in Python because the async libraries do not offer
much. Straightforward in some other languages.

Do you know why there is little support for asynchronous execution in
the standard libraries?
For large scale downloading I found thread pools do not scale well.

Richard

[toc] | [prev] | [next] | [standalone]

#20737

Il 23 febbraio 2012 07:58, Plumo <richardbp@gmail.com> ha scritto:
> I want to download content asynchronously. This would be straightforward to do threaded or across processes, but difficult asynchronously so people seem to rely on external libraries (twisted / gevent / eventlet).
>
> (I would use gevent under different circumstances, but currently need to stick to standard libraries.)
>
> I looked around and found there is little interest in developing a proper HTTP client on asyncore. The best I found stopped development a decade ago: http://sourceforge.net/projects/asynchttp/
>
> What do you recommend?
> And why is there poor support for asynchronous execution?
>
> Richard
> --
> http://mail.python.org/mailman/listinfo/python-list

If you want to stick with asyncore try to take a look at this:
https://gist.github.com/1519999

> And why is there poor support for asynchronous execution?

I'd say that's true for stdlib only (asyncore/asynchat).
There are plenty of choices amongst third party modules though.
To say one, I particularly like tornado which is simple and powerful:
http://www.tornadoweb.org/documentation/httpclient.html

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/

[toc] | [prev] | [next] | [standalone]

#20772

that example is excellent - best use of asynchat I have seen so far.

I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat. 

[toc] | [prev] | [next] | [standalone]

#20773

that example is excellent - best use of asynchat I have seen so far.

I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat. 

[toc] | [prev] | [next] | [standalone]

#20776

[Multipart message — attachments visible in raw view] — view raw

On Thursday 23 Feb 2012 5:10:25 PM Plumo wrote:
> I read through the python-dev archives and found the fundamental problem is
> no one maintains asnycore / asynchat. 
By all means, scratch your own itch. :)
-- 
Fayaz Yusuf Khan
Cloud developer and architect
Dexetra SS, Bangalore, India
fayaz.yusuf.khan_AT_gmail_DOT_com
fayaz_AT_dexetra_DOT_com
+91-9746-830-823

[toc] | [prev] | [next] | [standalone]

#20793

Il 24 febbraio 2012 02:10, Plumo <richardbp@gmail.com> ha scritto:
> that example is excellent - best use of asynchat I have seen so far.
>
> I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.

Well, actually I do/did.
Point with asyncore/asynchat is that it's original design is so flawed
and simplicistic it doesn't allow actual customization without
breaking compatibility.
See for example:
http://bugs.python.org/issue6692


--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/

[toc] | [prev] | [next] | [standalone]

#20868

>> I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.
>
> Well, actually I do/did.

ah OK. I had read this comment from a few years back:
"IIRC, there was a threat to remove asyncore because there were no
maintainers, no one was fixing bugs, no one was improving it, and no
one was really using it"


> Point with asyncore/asynchat is that it's original design is so flawed
> and simplicistic it doesn't allow actual customization without
> breaking compatibility.

Python3 uses the same API - was there not enough interest to improve it?

Richard

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web