Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #69792 > unrolled thread

Re: threading

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: threading Ben Finney <ben+python@benfinney.id.au> - 2014-04-07 13:05 +1000
    Re: threading Roy Smith <roy@panix.com> - 2014-04-06 23:48 -0400
      Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-07 13:56 +1000
        Re: threading Roy Smith <roy@panix.com> - 2014-04-07 08:26 -0400
          Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-07 22:34 +1000
            Re: threading Roy Smith <roy@panix.com> - 2014-04-07 09:22 -0400
              Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-07 14:41 +0100
              Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 16:49 +0300
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 00:27 +1000
                  Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 17:51 +0300
                    Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 01:12 +1000
              Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 00:24 +1000
        Re: threading Rick Johnson <rantingrickjohnson@gmail.com> - 2014-04-08 18:09 -0700
          Re: threading "Neil D. Cerutti" <neilc@norwich.edu> - 2014-04-09 09:50 -0400
            Re: threading Rick Johnson <rantingrickjohnson@gmail.com> - 2014-04-09 08:51 -0700
              Re: threading MRAB <python@mrabarnett.plus.com> - 2014-04-09 18:47 +0100
                Re: threading Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-04-10 11:35 +1200
                  Re: threading Roy Smith <roy@panix.com> - 2014-04-09 19:53 -0400
                    Re: threading Andrew Berg <robotsondrugs@gmail.com> - 2014-04-09 19:02 -0500
                    Re: threading Steven D'Aprano <steve@pearwood.info> - 2014-04-10 02:43 +0000
                      Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 13:08 +1000
                    Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-10 09:23 +0100
                    Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 19:11 +1000
              Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 04:00 +1000
              Re: threading Steven D'Aprano <steve@pearwood.info> - 2014-04-10 03:44 +0000
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 13:54 +1000
      Re: threading Ben Finney <ben+python@benfinney.id.au> - 2014-04-07 15:22 +1000
      Re: threading Ethan Furman <ethan@stoneleaf.us> - 2014-04-08 11:09 -0700
      Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 21:41 +0200
        Re: threading Grant Edwards <invalid@invalid.invalid> - 2014-04-08 20:30 +0000
          Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 00:32 +0200
            Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-08 19:17 -0700
    Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 08:10 +0300
      Re: threading Paul Rubin <no.email@nospam.invalid> - 2014-04-06 22:39 -0700
        Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 08:46 +0300
        Re: threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-04-07 19:47 -0400
          Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-08 08:19 +0300
            Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 10:47 +0000
              Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-08 15:10 +0300
                Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 16:37 +0000
                  Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-08 20:17 +0300
              Re: threading Roy Smith <roy@panix.com> - 2014-04-08 09:19 -0400
                Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 15:44 +0000
                  Re: threading Paul Rubin <no.email@nospam.invalid> - 2014-04-08 09:38 -0700
                    Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-09 14:42 +0100
            Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-09 15:23 +0200
              Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-09 16:55 +0300
                Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-09 16:46 +0200
                  Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-09 20:31 +0300
                    Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 03:52 +1000
                      Re: threading Mark H Harris <harrismh777@gmail.com> - 2014-04-10 08:29 -0500
                    Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 19:20 +0000
            Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-09 23:47 +1000
              Re: threading Roy Smith <roy@panix.com> - 2014-04-09 10:44 -0400
            Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-09 16:30 +0200
              Re: threading Roy Smith <roy@panix.com> - 2014-04-09 10:52 -0400
                Re: threading Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-04-10 11:19 +1200
              Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-09 19:48 +0300
            Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 00:44 +1000
            Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 15:29 +0000
            Re: threading Terry Reedy <tjreedy@udel.edu> - 2014-04-09 12:14 -0400
            Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 02:25 +1000
            Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 16:32 +0000
            Re: threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-04-09 19:44 -0400
            Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 11:05 +1000
            Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-10 11:17 +0200
            Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 19:40 +1000
            Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-10 13:10 +0200
              Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 14:43 +0300
                Re: threading Roy Smith <roy@panix.com> - 2014-04-10 08:56 -0400
                Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-10 15:24 +0000
                  Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 19:20 +0300
                Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-11 01:32 +1000
                  Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 19:25 +0300
                    Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-11 03:08 +1000
                      Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 11:14 -0700
                        Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 22:44 +0300
                          Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 13:21 -0700
                            Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 23:44 +0300
                              Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 22:15 -0700
                                Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 23:50 -0700
                                  Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-11 18:36 +0300
                                    Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-12 01:53 +1000
                                    Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-11 16:58 +0100
                                    Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-11 11:54 -0700
                                      Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-11 22:27 +0300
                          Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-11 01:51 +0200
                            Re: threading Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-04-11 05:35 +0000
                              Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-11 09:26 +0000
                              Re: threading Roy Smith <roy@panix.com> - 2014-04-11 08:36 -0400
                                Re: threading Grant Edwards <invalid@invalid.invalid> - 2014-04-11 16:18 +0000
                          Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-11 02:21 +0200
                          Re: threading Terry Reedy <tjreedy@udel.edu> - 2014-04-10 20:23 -0400
            Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 21:19 +1000
        Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 02:06 +0000
          Re: threading alister <alister.nospam.ware@ntlworld.com> - 2014-04-08 11:07 +0000
            Re: threading Roy Smith <roy@panix.com> - 2014-04-08 09:13 -0400
              Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 23:23 +1000
                Re: threading alister <alister.nospam.ware@ntlworld.com> - 2014-04-08 14:15 +0000
                  Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 16:06 +0000
              Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 15:40 +0000
                Re: threading Paul Rubin <no.email@nospam.invalid> - 2014-04-08 09:46 -0700
                  Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-09 02:46 +1000
                  Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 17:17 +0000
            Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 15:19 +0000

Page 2 of 6 — ← Prev page 1 [2] 3 4 5 6  Next page →

#70003

On Thu, Apr 10, 2014 at 12:43 PM, Steven D'Aprano <steve@pearwood.info> wrote:
> You cannot use plane English!

Cleared for takeoff...

ChrisA

[toc] | [prev] | [next] | [standalone]

#70022

On 10/04/2014 00:53, Roy Smith wrote:
>
> Natural language is a wonderfully expressive thing.  I open the window,
> stick my head out, look up at the sky, and say, "Raining".  Forget the
> pronoun, I don't even have a verb.  And yet everybody understands
> exactly what I mean.
>

In the UK you can stay in bed and say "Raining" and the odds are you'll 
be correct :)

-- 
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com

[toc] | [prev] | [next] | [standalone]

#70025

On Thu, Apr 10, 2014 at 6:23 PM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
> On 10/04/2014 00:53, Roy Smith wrote:
>> Natural language is a wonderfully expressive thing.  I open the window,
>> stick my head out, look up at the sky, and say, "Raining".  Forget the
>> pronoun, I don't even have a verb.  And yet everybody understands
>> exactly what I mean.
>>
>
> In the UK you can stay in bed and say "Raining" and the odds are you'll be
> correct :)

Is the staying-in-bed part critical to that? The last few times I've
been to England, it's only rained a few times. Granted, I've always
come during your summer, but even so, the rumours suggest that rain
should still be plenty common. We've happily driven a costume rack
down the A53 (twice - once empty, once loaded, if I recall correctly),
without worrying about rain. There were a few times when the terrain
was treacherous (imagine this: you're at the top of a moderately-steep
(probably 1 in 10-20) of rough concrete or asphalt, depending on which
part you jog down, and it's been greased up by vehicles standing
there, and then rained on; and you need to run down it at full speed,
catch the porta-cabin before it closes for the last time this year,
get the DVDs that were being run off for you, and run back up at full
speed, all before a ceremony begins), but other than that, it's been
pretty dry every time we've been there.

But we don't stay in bed much.

ChrisA

[toc] | [prev] | [next] | [standalone]

#69972

On Thu, Apr 10, 2014 at 3:47 AM, MRAB <python@mrabarnett.plus.com> wrote:
> On 2014-04-09 16:51, Rick Johnson wrote:
>>      And the evil incarnation of the IMPLICIT PRONOUN raises
>>      it's ugly head!!!
>>
> (And it's "its ugly head", BTW.)

Fundamental rule of the internet: If you criticize someone else's
spelling or grammar, you will make a blooper yourself, usually of
paper-bag proportions.

ChrisA

[toc] | [prev] | [next] | [standalone]

#70005

On Wed, 09 Apr 2014 08:51:09 -0700, Rick Johnson wrote:

> On Wednesday, April 9, 2014 8:50:59 AM UTC-5, Neil D. Cerutti wrote:
>> [...]
>> Plus Rufus Xavier Sasparilla disagrees with it.
> 
> If you think you're going to argue in such an implicit manner as to the
> benefits of pronouns, then you should expect that an astute logician
> such as myself will tear you to shreds.

This reminds me of Wile E. Coyote, announcing himself to others:

"Allow me to introduce myself, Wile E. Coyote, Super-Genius."

http://geekwisdom.wordpress.com/2013/03/16/wile-e-coyote-super-genius/


Rick, if you have to call yourself an astute logician for other to know, 
then you probably aren't.


> And whist i admit your snarky comment is founded on a *superficial* fact

"Whilst". Whist is a card game.

[...]
> ############################################################ 
> # You comment is cleaver propaganda, 

Damn those lying cleavers!

"Clever". A cleaver is a type of heavy chopping knife, usually used for 
cutting through bones.


[...]
> Now, before i utterly destroy you, we must first *ALL* take a lesson in
> linguistics.

Oh noes! I'm going to be destroyed by Rick, Super-Genius!

Rick, before you reform the English language, perhaps you ought to 
concentrate on an easier task. A few years ago you offered to fork Python 
and make it a better language. I promised to be a beta tester for you if 
you did so. How is that coming along? Do you have a repo where I can 
check out progress?



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#70006

On Thu, Apr 10, 2014 at 1:44 PM, Steven D'Aprano <steve@pearwood.info> wrote:
>> Now, before i utterly destroy you, we must first *ALL* take a lesson in
>> linguistics.
>
> Oh noes! I'm going to be destroyed by Rick, Super-Genius!

I'll send him my business card as a salesman for Acme. Easiest job in
the world - just sell him stuff with no guarantees, isn't necessarily
of merchantable quality, and might even backfire and explode, and
he'll buy the lot.

ChrisA

[toc] | [prev] | [next] | [standalone]

#69802

Chris Angelico <rosuav@gmail.com> writes:

> On Mon, Apr 7, 2014 at 1:48 PM, Roy Smith <roy@panix.com> wrote:
> > There is (or at least, was) another reason. Creating a new process
> > used to be far more expensive than creating a new thread. In modern
> > Unix kernels, however, the cost difference has become much less, so
> > this is no longer a major issue.
>
> Unix maybe, but what about Windows? Is it efficient to create
> processes under Windows?

Another reason to avoid Microsoft's operating systems as a programming
target, IMO.

-- 
 \        “We cannot solve our problems with the same thinking we used |
  `\                           when we created them.” —Albert Einstein |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#69896

On 04/06/2014 08:56 PM, Chris Angelico wrote:
> On Mon, Apr 7, 2014 at 1:48 PM, Roy Smith <roy@panix.com> wrote:
>> There is (or at least, was) another reason.  Creating a new process used
>> to be far more expensive than creating a new thread.  In modern  Unix
>> kernels, however, the cost difference has become much less, so this is
>> no longer a major issue.
>
> Unix maybe, but what about Windows? Is it efficient to create
> processes under Windows?

Not a concern until performance is not good enough.

--
~Ethan~

[toc] | [prev] | [next] | [standalone]

#69897

On 07/04/14 05:56, Chris Angelico wrote:
> On Mon, Apr 7, 2014 at 1:48 PM, Roy Smith <roy@panix.com> wrote:
>> There is (or at least, was) another reason.  Creating a new process used
>> to be far more expensive than creating a new thread.  In modern  Unix
>> kernels, however, the cost difference has become much less, so this is
>> no longer a major issue.
>
> Unix maybe, but what about Windows? Is it efficient to create
> processes under Windows?


Processes are very heavy-weight on Windows.

Sturla

[toc] | [prev] | [next] | [standalone]

#69903

On 2014-04-08, Sturla Molden <sturla.molden@gmail.com> wrote:
> On 07/04/14 05:56, Chris Angelico wrote:
>> On Mon, Apr 7, 2014 at 1:48 PM, Roy Smith <roy@panix.com> wrote:
>>> There is (or at least, was) another reason.  Creating a new process used
>>> to be far more expensive than creating a new thread.  In modern  Unix
>>> kernels, however, the cost difference has become much less, so this is
>>> no longer a major issue.
>>
>> Unix maybe, but what about Windows? Is it efficient to create
>> processes under Windows?
>
> Processes are very heavy-weight on Windows.

Not surprising given its VMS heritage.  I remember running shell
scripts under VMS on a VAX-11/780 that took hours to do what would
have taken minutes on an LSI-11 running Unix.  The whole Unix "small
tools working together" paradigm is based on the assumption that
process creation and death are fast and cheap.

-- 
Grant Edwards               grant.b.edwards        Yow! I selected E5 ... but
                                  at               I didn't hear "Sam the Sham
                              gmail.com            and the Pharoahs"!

[toc] | [prev] | [next] | [standalone]

#69907

On 08/04/14 22:30, Grant Edwards wrote:

>>> Unix maybe, but what about Windows? Is it efficient to create
>>> processes under Windows?
>>
>> Processes are very heavy-weight on Windows.
>
> Not surprising given its VMS heritage.  I remember running shell
> scripts under VMS on a VAX-11/780 that took hours to do what would
> have taken minutes on an LSI-11 running Unix.  The whole Unix "small
> tools working together" paradigm is based on the assumption that
> process creation and death are fast and cheap.

That is one reason software tend to be monolithic on Windows, including 
build tools.

Running a configure script used to take forever, but thankfully 
computers are getting faster.

Sturla

[toc] | [prev] | [next] | [standalone]

#69917

On Wednesday, April 9, 2014 4:02:10 AM UTC+5:30, Sturla Molden wrote:
> On 08/04/14 22:30, Grant Edwards wrote:

> >>> Unix maybe, but what about Windows? Is it efficient to create
> >>> processes under Windows?
> >> Processes are very heavy-weight on Windows.
> > Not surprising given its VMS heritage.  I remember running shell
> > scripts under VMS on a VAX-11/780 that took hours to do what would
> > have taken minutes on an LSI-11 running Unix.  The whole Unix "small
> > tools working together" paradigm is based on the assumption that
> > process creation and death are fast and cheap.

> That is one reason software tend to be monolithic on Windows, including 
> build tools.

> Running a configure script used to take forever, but thankfully 
> computers are getting faster.

I was looking at Erlang...
And under similar presumptions that I see on this thread (in a different sense!)
viz.: Either the messiness of callback hell or the error-proneness of threads

However this was Erlang whose basic premise is to question this either-or.

And so I was properly told-off by Joe Armstrong
(roughly the equivalent of being told off by Guido out here :-) )

http://erlang.org/pipermail/erlang-questions/2012-October/069560.html

Note Erlang, Go and Cloud-haskell all set out along a similar route:
http://joneisen.tumblr.com/post/38188396218/concurrency-models-go-vs-erlang

[toc] | [prev] | [next] | [standalone]

#69799

Ben Finney <ben+python@benfinney.id.au>:

> The *whole point* of threading (AFAIK) is to share memory and other
> process-distinct resources.

Another way to look at it is that threads were pushed as a magic bullet
to manage the complexities of network programming. They were fashionable
in Windows and Java. The idea was that the programmer could write linear
code that blocks on I/O and not be overwhelmed by the intricacies.

I don't think it worked out all that well.

Since then both Windows and Java have come up with their own I/O
multiplexing facilities. Now we see Python follow suit with asyncio.

Threads have their uses, but they are such tricky beasts that I would
see first if the problem couldn't be handled with multiplexing and
multiprocessing.

The main need for threads today comes from the database libraries,
which, AFAIK, generally don't provide a nonblocking API.


Marko

[toc] | [prev] | [next] | [standalone]

#69801

Marko Rauhamaa <marko@pacujo.net> writes:
> Since then both Windows and Java have come up with their own I/O
> multiplexing facilities. Now we see Python follow suit with asyncio.

That all happened because threads in those systems are rather expensive.
GHC and Erlang have fast lightweight threads/processes and programming
with them is much more civilized than using async schemes.  Even a low
level language like Forth reached something similar.

I keep hearing about all the perils of threading bugs and it just hasn't
happened to me in Python as far as I know.  The main trick is to not
share any mutable data between threads.  Instead have them communicate
by message passing through Queues.  If you've got a lot of tasks in the
system then it helps to have a bit of abstraction to keep the queues
organized and make the other tasks addressible by name, but it's all
pretty straightforward.  You do take an efficiency hit, but if that's a
big concern you sort of have to look past Python.

Lately I'm messing with Go and it's sort of the same idea.

[toc] | [prev] | [next] | [standalone]

#69803

Paul Rubin <no.email@nospam.invalid>:

> I keep hearing about all the perils of threading bugs and it just
> hasn't happened to me in Python as far as I know.

Good for you. I'm saying the first step to thread-safe code is to have a
healthy fear of the perils.

> The main trick is to not share any mutable data between threads.
> Instead have them communicate by message passing through Queues.

That certainly is a good way and is akin to multiprocessing. You still
need to make sure you don't flood the queues or cause deadlocks by
limiting queue sizes.

It is still easy to accidentally pass references to mutable objects
between threads (and most people don't even try to avoid it).
Multiprocessing naturally enforces the principle.


Marko

[toc] | [prev] | [next] | [standalone]

#69821

On Sun, 06 Apr 2014 22:39:05 -0700, Paul Rubin <no.email@nospam.invalid>
declaimed the following:

>I keep hearing about all the perils of threading bugs and it just hasn't
>happened to me in Python as far as I know.  The main trick is to not
>share any mutable data between threads.  Instead have them communicate
>by message passing through Queues.  If you've got a lot of tasks in the
>system then it helps to have a bit of abstraction to keep the queues
>organized and make the other tasks addressible by name, but it's all
>pretty straightforward.  You do take an efficiency hit, but if that's a
>big concern you sort of have to look past Python.
>

	That's been my experience too... Threading works for me... My attempts
at so called asyncio (whatever language) have always led to my having to
worry about losing data if some handler takes too long to return.

	To me, asyncio is closer to a polling interrupt handler, and I still
need a thread to handle the main processing.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#69832

Dennis Lee Bieber <wlfraed@ix.netcom.com>:

> 	That's been my experience too... Threading works for me... My
> attempts at so called asyncio (whatever language) have always led to
> my having to worry about losing data if some handler takes too long to
> return.
>
> 	To me, asyncio is closer to a polling interrupt handler, and I
> still need a thread to handle the main processing.

Yes, asynchronous processing results in complex, event-driven state
machines that can be hard to get right. However, my experience is that
that's the lesser evil.

About a handler taking too long: you need to guard each state with a
timer. Also, you need then to handle the belated handler after the timer
has expired.

The main problems with threads include:

 * Thread-safety is rarely done right. Also, when it's done wrong, it
   can be virtually impossible to fix it without a significant rewrite.
   This is not a theoretical concern: I have had to deal with the
   resulting nightmares in my work.

 * There is no accepted, taught, industry-wide discipline on proper
   thread-safety practices so every developer has to improvise. I have
   come up with a "bullet-proof" way of developing with threads, but
   even that methodology has nasty corner cases.

 * Thread-safety cannot be abstracted out. IOW, divide and conquer
   doesn't work. You can't hide the locking inside a class and forget
   about it. The entire application must be aware low-level thread
   synchronization needs.

 * Threads assume each state has one exit event. At a bare minimum, each
   thread should be prepared to have the blocking event aborted from the
   outside. Unfortunately, most threading frameworks don't allow for
   graceful aborts (that goes for Java and Python, too).

 * If you have made your design around threads and finally decide
   asynchronous processing would have been a better choice, a complete
   rewrite is required if it is even possible. Library writers often
   only provide blocking I/O functions forcing you to insulate the
   libraries in thread pools.


Marko

[toc] | [prev] | [next] | [standalone]

#69851

Marko Rauhamaa <marko@pacujo.net> wrote:

> The main problems with threads include:
> 
>  * Thread-safety is rarely done right. Also, when it's done wrong, it
>    can be virtually impossible to fix it without a significant rewrite.
>    This is not a theoretical concern: I have had to deal with the
>    resulting nightmares in my work.
> 
>  * There is no accepted, taught, industry-wide discipline on proper
>    thread-safety practices so every developer has to improvise. I have
>    come up with a "bullet-proof" way of developing with threads, but
>    even that methodology has nasty corner cases.
> 
>  * Thread-safety cannot be abstracted out. IOW, divide and conquer
>    doesn't work. You can't hide the locking inside a class and forget
>    about it. The entire application must be aware low-level thread
>    synchronization needs.

The problem here is the belief that "thread-safety cannot be abstracted
out". It can. The solution is to share nothing and send messages through
queues. If you start to use mutexes and conditions all over your code, you
might shoot yourself in the foot, eventually.   

Sturla

[toc] | [prev] | [next] | [standalone]

#69855

Sturla Molden <sturla.molden@gmail.com>:

> The problem here is the belief that "thread-safety cannot be
> abstracted out". It can. The solution is to share nothing and send
> messages through queues. If you start to use mutexes and conditions
> all over your code, you might shoot yourself in the foot, eventually.

Queues are fine if you hermetically insulate your objects. IOW, you
group your objects in single-threaded pseudoprocesses that don't make
any direct method calls to each other. If you do that, you might as well
use real processes. That way, you can even naturally enforce your
insulation assumption against accidents.

In "normal" multithreaded code, different object methods are called from
different threads. And, as you say, you *will* shoot yourself in the
foot.

Now, even in your queued thread world, you will suffer from the fact
that the naive thread programmer uses blocking function calls (as they
have been taught). Imagine, for example, a network outage in the middle
of a database call. The thread might not take a peek at its input queue
during the stuck I/O call and the application cannot abort the
operation. The solution is to use an asyncio main loop and nonblocking
I/O in each of your insulated threads...


Marko

[toc] | [prev] | [next] | [standalone]

#69882

Marko Rauhamaa <marko@pacujo.net> wrote:

> Queues are fine if you hermetically insulate your objects. IOW, you
> group your objects in single-threaded pseudoprocesses that don't make
> any direct method calls to each other. If you do that, you might as well
> use real processes. That way, you can even naturally enforce your
> insulation assumption against accidents.

No, 10,000 processes will not do. First, they use more resources than
threads, at least on Windows. Second, if something fails, you have 10,000
worker processes to kill. You also risk flooding your system with zombies.
So thanks, but no thanks. I would consider a small pool of processes to
utilize all cores on the CPU, though. But each process would have hundreds
or thousands of threads.

.NET AppDomains is an attempt to solve this. Python do not support
AppDomains in the interpreter. It would be nice if it did, though, e.g. a
thread with an isolated interpreter instance (similar to tcl). But
currently the Python interpreter is not designed for that, as it has states
which are lobal to the process, not just the interpreter.

If we insulate objects completely, we must also serialize (pickle) them
before sending them off as messages on a queue. That can be a major
bottleneck. At least it is in numerical computing with Python. Avoiding
serialization is also an important consideration. But IPC itself is very
fast, so the important part is packing a object into a byte string and
unpacking, not the overhead involved in sending it. Unix domain sockets and
Windows named pipes are close to a memcpy in performance, with very little
overhead. Pickle on the other hand is "very slow". Serialization is hard ot
get around if we use processes for multi-core CPUs, but in a pure
multithread design it is not an issue.

Sturla

[toc] | [prev] | [next] | [standalone]

Page 2 of 6 — ← Prev page 1 [2] 3 4 5 6  Next page →

Back to top | Article view | comp.lang.python

csiph-web