Re: Future of Pypy?

From	Paul Rubin <no.email@nospam.invalid>
Newsgroups	comp.lang.python
Subject	Re: Future of Pypy?
Date	2015-02-22 18:04 -0800
Organization	A noiseless patient Spider
Message-ID	<87zj85bcyu.fsf@jester.gateway.pace.com> (permalink)
References	(2 earlier) <mailman.19009.1424611670.18130.python-list@python.org> <pqrjeahei7v04lu0nqgsm01jrebj10hj8e@4ax.com> <mailman.19018.1424625741.18130.python-list@python.org> <87fv9xdb22.fsf@jester.gateway.pace.com> <54ea7ff4$0$12983$c3e8da3$5496439d@news.astraweb.com>

Show all headers | View raw

Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:
> I'm sorry, but the instant somebody says "eliminate the GIL", they lose
> credibility with me. Yes yes, I know that in *your* specific case you've
> done your research and (1) multi-threaded code is the best solution for
> your application and (2) alternatives aren't suitable.

I don't see what the big deal is.  I hear tons of horror stories about
threads and I believe them, but the thing is, they almost always revolve
around acquiring and releasing locks in the wrong order, forgetting to
lock things, stuff like that.  So I've generally respected the terror
and avoided programming in that style, staying with a message passing
style that may take an efficiency hit but seems to avoid almost all
those problems.  TM also helps with lock hazards and it's a beautiful
idea--I just haven't had to use it yet.  The Python IRC channel seems to
rage against threads and promote Twisted for concurrency, but Twisted
has always reminded me of Java.  I use threads in Python all the time
and haven't gotten bitten yet.

> Writing multithreaded code is *hard*. It is not a programming model
> which comes naturally to most human beings. Very few programs are
> inherently parallelizable, although many programs have *parts* which
> can be successfully parallelized.

Parallel algorithms are complicated and specialized but tons of problems
amount to "do the same thing with N different pieces of data", so-called
embarassingly parallel.  The model is you have a bunch of worker threads
reading off a queue and processing the items concurrently.  Sometimes
separate processes works equally well, other times it's good to have
some shared data in memory instead of communicating through sockets.  If
the data is mutable then have one thread own it and access it only with
message passing, Erlang style.  If it's immutable after initialization
(protect it with a condition variable til initialization finishes) then
you can have read-only access from anywhere.

> if you're running single-thread code on a single-core machine and
> still complaining about the GIL, you have no clue.

Even the Raspberry Pi has 4 cores now, and fancy smartphones have had
them for years.  Single core cpu's are specialized and/or historical.

> for some programs, the simplest way to speed it up is to vectorize the
> core parts of your code by using numpy.  No threads needed.

Nice for numerical codes, not so hot for anything else.

> Where are the people flocking to use Jython and IronPython?

Shrug, who knows, those implementations were pretty deficient from what
heard.

> For removal of the GIL to really make a difference: ...
> - you must be performing a task which is parallelizable and not inherently
> sequential (no point using multiple threads if each thread spends all its
> time waiting for the previous thread);

That's most things involving concurrency these days.

> - the task must be one that moving to some other multi-processing model
> (such as greenlets, multiprocess, etc.) is infeasible;

I don't understand this--there can be multiple ways to solve a problem.

> - your threading bottleneck must be primarily CPU-bound, not I/O bound
> (CPython's threads are already very effective at parallelising I/O tasks);

If your concurrent program's workload makes it cpu-bound even 1% of the
time, then you gain something by having it use your extra cores at those
moments, instead of having those cores always do nothing.

> - and you must be using libraries and tools which prevent you moving to
> Jython or IronPython or some other alternative.

I don't get this at all.  Why should I not want Python to have the same
capabilities?

Thread

Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 12:45 +0000
  Re: Future of Pypy? jkn <jkn_gg@nicorp.f9.co.uk> - 2015-02-22 04:58 -0800
    Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 15:30 +0000
      [OT] - BASIC is still not a bad choice, was Re: Future of Pypy? Michael Torrie <torriem@gmail.com> - 2015-02-23 17:24 -0700
  Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 14:27 +0100
    Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 15:36 +0000
      Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 18:22 +0100
        Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 11:02 -0800
          Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 20:51 +0100
            Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 12:14 -0800
              Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 23:13 +0100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 18:45 -0800
          Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 12:18 +1100
            Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 18:04 -0800
              Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-23 13:16 +1100
              Re: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-23 03:16 +0000
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 19:45 -0800
                Re: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-23 04:00 +0000
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 22:13 -0800
                Re: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-23 07:32 +0000
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 16:11 -0800
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 11:31 +1100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 17:50 -0800
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 13:03 +1100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 20:40 -0800
                Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 17:57 +1100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-27 13:40 -0800
                Re: Future of Pypy? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-02-27 18:47 -0500
                Are threads bad? - was: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-24 00:35 +0000
                Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 21:27 -0800
                Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 16:57 +1100
                Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 22:23 -0800
                Re: Are threads bad? - was: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-24 10:08 +0200
                Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-24 15:53 -0800
                Re: Are threads bad? - was: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-25 07:25 +0200
                Re: Are threads bad? - was: Future of Pypy? Marcos Almeida Azevedo <marcos.al.azevedo@gmail.com> - 2015-02-25 13:34 +0800
                Re: Are threads bad? - was: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-25 07:46 +0200
                Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-25 16:54 +1100
                Re: Are threads bad? - was: Future of Pypy? Marcos Almeida Azevedo <marcos.al.azevedo@gmail.com> - 2015-02-25 13:58 +0800
                Re: Are threads bad? - was: Future of Pypy? Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-24 23:02 -0700
                Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-25 17:07 +1100
                Re: Are threads bad? - was: Future of Pypy? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-02-25 16:37 +0000
                Re: Are threads bad? - was: Future of Pypy? Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-25 10:00 -0700
                Re: Are threads bad? - was: Future of Pypy? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-02-25 17:16 +0000
                Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-26 04:22 +1100
                Re: Are threads bad? - was: Future of Pypy? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-02-25 19:44 -0500
                Re: Are threads bad? - was: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-25 00:59 +0000
                Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-26 21:55 -0800
              Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-23 14:25 +1100
              Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 18:41 +1100
                Re: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-23 10:16 +0200
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-23 20:19 +1100
                Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 17:56 +1100
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 18:16 +1100
                Re: Future of Pypy? wxjmfauth@gmail.com - 2015-02-23 23:57 -0800
                Re: Future of Pypy? Ethan Furman <ethan@stoneleaf.us> - 2015-02-23 11:39 -0800
                Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 13:15 +1100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 17:47 -0800
                Re: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-24 10:12 +0200
      Re: Future of Pypy? Emile van Sebille <emile@fenx.com> - 2015-02-24 09:57 -0800
  Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 01:05 +1100
    Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 15:44 +0000
      Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 19:20 +0000
        Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 22:45 +0100
          Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-23 14:04 +0000
            Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-23 17:16 +0100
  Re: Future of Pypy? Terry Reedy <tjreedy@udel.edu> - 2015-02-23 01:34 -0500
  Re: Future of Pypy? Dave Cook <davecook@nowhere.net> - 2015-02-23 11:36 +0000
    Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-23 14:13 +0000
      Cython - was: Future of Pypy? Stefan Behnel <stefan_ml@behnel.de> - 2015-02-23 16:43 +0100
      Re: Future of Pypy? Dave Cook <davecook@nowhere.net> - 2015-02-23 23:23 +0000

csiph-web