Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #68234 > unrolled thread

Re: which async framework?

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: which async framework? Antoine Pitrou <solipsis@pitrou.net> - 2014-03-11 18:43 +0000
    Re: which async framework? Marko Rauhamaa <marko@pacujo.net> - 2014-03-12 00:28 +0200
      Re: which async framework? Chris Angelico <rosuav@gmail.com> - 2014-03-12 09:38 +1100
        Re: which async framework? Marko Rauhamaa <marko@pacujo.net> - 2014-03-12 01:18 +0200
          Re: which async framework? Chris Angelico <rosuav@gmail.com> - 2014-03-12 10:38 +1100
            Re: which async framework? Marko Rauhamaa <marko@pacujo.net> - 2014-03-12 02:10 +0200
          Re: which async framework? Ian Kelly <ian.g.kelly@gmail.com> - 2014-03-12 03:58 -0600

#68234 — Re: which async framework?

Sturla Molden <sturla.molden <at> gmail.com> writes:
> 
> Antoine Pitrou <solipsis <at> pitrou.net> wrote:
> 
> > Yes, why use a library when you can rewrite it all yourself?
> 
> This assumes something equivalent to the library will have to be written.
> But if it can be replaced with something very minimalistic it is just
> bloat. I would also like to respond that the select module and pywin32 are
> libraries.

Lower-level ones, though. Just like C malloc() is lower-level than
Python's memory management.

This is the usual assumption that high-level libraries are made of useless
cruft piled up by careless programmers. But there are actual reasons 
why these frameworks have a significant amount of code, and people who 
decide to ignore those reasons are simply bound to reimplement 
non-trivial parts of those frameworks in less careful and less tested
ways (and they have to maintain it themselves afterwards).

What irks me with your response is that you phrased it as though writing
a good event loop was an almost trivial thing to do, which it is not
once you start considering multiple use cases and constraints. It is
quite irresponsible to suggest people don't need the power of network
programming frameworks. I would personally distrust a programmer who
chooses to reimplement their own event loop, except if they happen to
have a very brilliant track record.

> Another thing is that co-routines and "yield from" statements just
> makes it hard to follow the logic of the program. 

That's an optional part of Tornado and asyncio, though. It is very
reasonable to use Tornado or asyncio and still code in purely
callback-driven style (even though Guido himself prefers the coroutine
style).

Regards

Antoine.

[toc] | [next] | [standalone]

#68244

Antoine Pitrou <solipsis@pitrou.net>:

> This is the usual assumption that high-level libraries are made of
> useless cruft piled up by careless programmers. But there are actual
> reasons why these frameworks have a significant amount of code, and
> people who decide to ignore those reasons are simply bound to
> reimplement non-trivial parts of those frameworks in less careful and
> less tested ways (and they have to maintain it themselves afterwards).

Maybe, maybe not. You can't assume it one way or another.

> What irks me with your response is that you phrased it as though writing
> a good event loop was an almost trivial thing to do,

It's not rocket science. There are pitfalls, to be sure, but the
resulting, solid event loop is not many lines of code. Have had to
implement it numerous times.

That said, asyncio seems to have the necessary facilities in place.
Definitely worth a closer look.

> I would personally distrust a programmer who chooses to reimplement
> their own event loop, except if they happen to have a very brilliant
> track record.

As with all programming, you have to do it right.

If you can't write your own event loop, you probably can't be trusted
with any multithreaded code, which has much more baffling corner cases.


Marko

[toc] | [prev] | [next] | [standalone]

#68246

On Wed, Mar 12, 2014 at 9:28 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> If you can't write your own event loop, you probably can't be trusted
> with any multithreaded code, which has much more baffling corner cases.

I'm not sure about that. Threads are generally easier to handle,
because each one just does something on its own, sequentially. (The
problem with threads is the resource usage - you need to allocate X
amount of stack space and other execution state, when all you really
need is the socket (or whatever) and some basic state about that.)
Regardless of how you structure your code, you have to ensure that one
handler doesn't tread on another's toes, and with multithreading,
that's _all_ you have to worry about. A proper event loop, handling
events from all sorts of different sources, is far more complicated.

What corner cases are there with threads that you don't have with anything else?

ChrisA

[toc] | [prev] | [next] | [standalone]

#68250

Chris Angelico <rosuav@gmail.com>:

> What corner cases are there with threads that you don't have with
> anything else?

There are numerous. Here's one example: deadlocks due to two threads
taking locks in a different order. The problem crops up naturally with
two intercommunicating classes. It can sometimes be very difficult to
spot or avoid.


Marko

[toc] | [prev] | [next] | [standalone]

#68251

On Wed, Mar 12, 2014 at 10:18 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> What corner cases are there with threads that you don't have with
>> anything else?
>
> There are numerous. Here's one example: deadlocks due to two threads
> taking locks in a different order. The problem crops up naturally with
> two intercommunicating classes. It can sometimes be very difficult to
> spot or avoid.

Yep. Now how is that not a problem when you use some other model, like
an event loop? The standard methods of avoiding deadlocks (like
acquiring locks in strict order) work exactly the same for all models,
and are just as necessary.

ChrisA

[toc] | [prev] | [next] | [standalone]

#68252

Chris Angelico <rosuav@gmail.com>:

> Yep. Now how is that not a problem when you use some other model, like
> an event loop? The standard methods of avoiding deadlocks (like
> acquiring locks in strict order) work exactly the same for all models,
> and are just as necessary.

I was simply saying that if you can work with multiple threads, an event
loop shouldn't be a big deal.


Marko

[toc] | [prev] | [next] | [standalone]

#68267

On Tue, Mar 11, 2014 at 5:38 PM, Chris Angelico <rosuav@gmail.com> wrote:
> On Wed, Mar 12, 2014 at 10:18 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> Chris Angelico <rosuav@gmail.com>:
>>
>>> What corner cases are there with threads that you don't have with
>>> anything else?
>>
>> There are numerous. Here's one example: deadlocks due to two threads
>> taking locks in a different order. The problem crops up naturally with
>> two intercommunicating classes. It can sometimes be very difficult to
>> spot or avoid.
>
> Yep. Now how is that not a problem when you use some other model, like
> an event loop? The standard methods of avoiding deadlocks (like
> acquiring locks in strict order) work exactly the same for all models,
> and are just as necessary.

If you don't have threads then the only locks you need to acquire are
for external resources.  You might still deadlock, but at least your
process won't be deadlocking with itself.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web