Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #86283

Re: Future of Pypy?

From Paul Rubin <no.email@nospam.invalid>
Newsgroups comp.lang.python
Subject Re: Future of Pypy?
Date 2015-02-23 17:47 -0800
Organization A noiseless patient Spider
Message-ID <8761asaxmz.fsf@jester.gateway.pace.com> (permalink)
References (4 earlier) <mailman.19018.1424625741.18130.python-list@python.org> <87fv9xdb22.fsf@jester.gateway.pace.com> <54ea7ff4$0$12983$c3e8da3$5496439d@news.astraweb.com> <87zj85bcyu.fsf@jester.gateway.pace.com> <54ead9aa$0$12930$c3e8da3$5496439d@news.astraweb.com>

Show all headers | View raw

Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:
> Deadlocks, livelocks, and the simple fact that debugging threaded code is 
> much, much, much harder than debugging single-thread code.

Deadlocks and livelocks can happen in multi-process code just like with
threads.  Debugging threaded code hasn't been too bad a problem for me
so far.  The basic tactic is use the logging module a lot (it is thread
safe), log the i/o events coming into the system if you observe
misbehaviour, and play the log back through your test harness to
reproduce the problem and debug it by normal means.  You have to do
something like that anyway, if the indeterminacy in the system comes
from the unpredictable ordering of external i/o events.  IMHO you'd get
basicaly similar bugs with other concurrency mechanisms.

Other programs like Postgres are written in dangerous languages like C
and use millions of LOC and vast numbers of fine-grained locks and are
still considered very reliable.  Python threads communicating through
queues has to be a breeze by comparison.  Compared to what Linux kernel
hackers have to deal with, it's not even on the same planet.  This book
is amazing:

https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html

> No. You [generic you] sux, for not investigating whether multiprocessing 
> will do the job. It even has the same public API as threads.

I will give multiprocessing a try next time the matter comes up (it
hasn't always been available) but it doesn't seem as flexible.  E.g. say
I have an employee handling thread (untested pseudocode, forgive
errors):

   def employee_loop(queue):
     while True:
       func, args = queue.get()
       func(*args)

   def adjust_salary(person, update_func):
       salaries[person] = update_func(salaries[person])

   threading.Thread(target=employee_loop,
                    args=[employee_request_queue]).start()

Now in another thread, I want to give you a 50% raise:

    employee_request_queue.put(
      (adjust_salary, ('Steven', 
                         lambda old_salary: old_salary * 1.5)))

Can I do that with multiprocessing without a bunch more boilerplate?

> Even in GIL-less Python, two threads aren't twice as fast as one
> thread. So there comes a point of diminishing returns

Well if two threads are 1.5x as fast as one thread, that's a win.

> Python does have the same capabilities. Jython and IronPython aren't 
> different languages, they are Python.

Oh ok.  I think of Jython as a Java thing and IronPython as a Windows
thing and I don't want to deal with the monstrous JVM or CLR systems.
So I unconsciously didn't think of them as Python.  I do think of PyPy
as Python.

> If you want *CPython* to work without a GIL, well, are you
> volunteering to do the work? It is a massive job, and the core devs
> aren't terribly interested.

It's not a massive job, it's been done and it worked, the problem is
that locking all the CPython refcount updates slowed down the important
single-cpu case so it was never adopted.  So the real massive job would
be moving to a tracing GC and changing every extension module ever
written.  But, that's what PyPy is (among other things) so I'm waiting
to hear from Laura why PyPy has a GIL.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 12:45 +0000
  Re: Future of Pypy? jkn <jkn_gg@nicorp.f9.co.uk> - 2015-02-22 04:58 -0800
    Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 15:30 +0000
      [OT] - BASIC is still not a bad choice, was Re: Future of Pypy? Michael Torrie <torriem@gmail.com> - 2015-02-23 17:24 -0700
  Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 14:27 +0100
    Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 15:36 +0000
      Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 18:22 +0100
        Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 11:02 -0800
          Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 20:51 +0100
            Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 12:14 -0800
              Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 23:13 +0100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 18:45 -0800
          Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 12:18 +1100
            Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 18:04 -0800
              Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-23 13:16 +1100
              Re: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-23 03:16 +0000
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 19:45 -0800
                Re: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-23 04:00 +0000
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 22:13 -0800
                Re: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-23 07:32 +0000
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 16:11 -0800
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 11:31 +1100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 17:50 -0800
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 13:03 +1100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 20:40 -0800
                Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 17:57 +1100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-27 13:40 -0800
                Re: Future of Pypy? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-02-27 18:47 -0500
                Are threads bad? - was: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-24 00:35 +0000
                Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 21:27 -0800
                Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 16:57 +1100
                Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 22:23 -0800
                Re: Are threads bad? - was: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-24 10:08 +0200
                Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-24 15:53 -0800
                Re: Are threads bad? - was: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-25 07:25 +0200
                Re: Are threads bad? - was: Future of Pypy? Marcos Almeida Azevedo <marcos.al.azevedo@gmail.com> - 2015-02-25 13:34 +0800
                Re: Are threads bad? - was: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-25 07:46 +0200
                Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-25 16:54 +1100
                Re: Are threads bad? - was: Future of Pypy? Marcos Almeida Azevedo <marcos.al.azevedo@gmail.com> - 2015-02-25 13:58 +0800
                Re: Are threads bad? - was: Future of Pypy? Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-24 23:02 -0700
                Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-25 17:07 +1100
                Re: Are threads bad? - was: Future of Pypy? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-02-25 16:37 +0000
                Re: Are threads bad? - was: Future of Pypy? Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-25 10:00 -0700
                Re: Are threads bad? - was: Future of Pypy? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-02-25 17:16 +0000
                Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-26 04:22 +1100
                Re: Are threads bad? - was: Future of Pypy? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-02-25 19:44 -0500
                Re: Are threads bad? - was: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-25 00:59 +0000
                Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-26 21:55 -0800
              Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-23 14:25 +1100
              Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 18:41 +1100
                Re: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-23 10:16 +0200
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-23 20:19 +1100
                Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 17:56 +1100
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 18:16 +1100
                Re: Future of Pypy? wxjmfauth@gmail.com - 2015-02-23 23:57 -0800
                Re: Future of Pypy? Ethan Furman <ethan@stoneleaf.us> - 2015-02-23 11:39 -0800
                Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 13:15 +1100
                Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 17:47 -0800
                Re: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-24 10:12 +0200
      Re: Future of Pypy? Emile van Sebille <emile@fenx.com> - 2015-02-24 09:57 -0800
  Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 01:05 +1100
    Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 15:44 +0000
      Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 19:20 +0000
        Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 22:45 +0100
          Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-23 14:04 +0000
            Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-23 17:16 +0100
  Re: Future of Pypy? Terry Reedy <tjreedy@udel.edu> - 2015-02-23 01:34 -0500
  Re: Future of Pypy? Dave Cook <davecook@nowhere.net> - 2015-02-23 11:36 +0000
    Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-23 14:13 +0000
      Cython - was: Future of Pypy? Stefan Behnel <stefan_ml@behnel.de> - 2015-02-23 16:43 +0100
      Re: Future of Pypy? Dave Cook <davecook@nowhere.net> - 2015-02-23 23:23 +0000

csiph-web