Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #86105 > unrolled thread

Future of Pypy?

Back to article view | Back to comp.lang.python

Contents

  Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 12:45 +0000
    Re: Future of Pypy? jkn <jkn_gg@nicorp.f9.co.uk> - 2015-02-22 04:58 -0800
      Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 15:30 +0000
        [OT] - BASIC is still not a bad choice, was Re: Future of Pypy? Michael Torrie <torriem@gmail.com> - 2015-02-23 17:24 -0700
    Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 14:27 +0100
      Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 15:36 +0000
        Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 18:22 +0100
          Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 11:02 -0800
            Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 20:51 +0100
              Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 12:14 -0800
                Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 23:13 +0100
                  Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 18:45 -0800
            Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 12:18 +1100
              Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 18:04 -0800
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-23 13:16 +1100
                Re: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-23 03:16 +0000
                  Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 19:45 -0800
                    Re: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-23 04:00 +0000
                      Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-22 22:13 -0800
                        Re: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-23 07:32 +0000
                          Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 16:11 -0800
                            Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 11:31 +1100
                              Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 17:50 -0800
                                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 13:03 +1100
                                  Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 20:40 -0800
                                    Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 17:57 +1100
                                      Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-27 13:40 -0800
                                        Re: Future of Pypy? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-02-27 18:47 -0500
                            Are threads bad? - was: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-24 00:35 +0000
                              Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 21:27 -0800
                                Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 16:57 +1100
                                  Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 22:23 -0800
                                  Re: Are threads bad? - was: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-24 10:08 +0200
                                    Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-24 15:53 -0800
                                      Re: Are threads bad? - was: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-25 07:25 +0200
                                        Re: Are threads bad? - was: Future of Pypy? Marcos Almeida Azevedo <marcos.al.azevedo@gmail.com> - 2015-02-25 13:34 +0800
                                          Re: Are threads bad? - was: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-25 07:46 +0200
                                            Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-25 16:54 +1100
                                            Re: Are threads bad? - was: Future of Pypy? Marcos Almeida Azevedo <marcos.al.azevedo@gmail.com> - 2015-02-25 13:58 +0800
                                            Re: Are threads bad? - was: Future of Pypy? Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-24 23:02 -0700
                                            Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-25 17:07 +1100
                                            Re: Are threads bad? - was: Future of Pypy? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-02-25 16:37 +0000
                                            Re: Are threads bad? - was: Future of Pypy? Ian Kelly <ian.g.kelly@gmail.com> - 2015-02-25 10:00 -0700
                                            Re: Are threads bad? - was: Future of Pypy? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-02-25 17:16 +0000
                                            Re: Are threads bad? - was: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-26 04:22 +1100
                                            Re: Are threads bad? - was: Future of Pypy? Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2015-02-25 19:44 -0500
                                Re: Are threads bad? - was: Future of Pypy? Ryan Stuart <ryan.stuart.85@gmail.com> - 2015-02-25 00:59 +0000
                                  Re: Are threads bad? - was: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-26 21:55 -0800
                Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-23 14:25 +1100
                Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 18:41 +1100
                  Re: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-23 10:16 +0200
                  Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-23 20:19 +1100
                    Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 17:56 +1100
                      Re: Future of Pypy? Chris Angelico <rosuav@gmail.com> - 2015-02-24 18:16 +1100
                        Re: Future of Pypy? wxjmfauth@gmail.com - 2015-02-23 23:57 -0800
                  Re: Future of Pypy? Ethan Furman <ethan@stoneleaf.us> - 2015-02-23 11:39 -0800
                    Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-24 13:15 +1100
                  Re: Future of Pypy? Paul Rubin <no.email@nospam.invalid> - 2015-02-23 17:47 -0800
                    Re: Future of Pypy? Marko Rauhamaa <marko@pacujo.net> - 2015-02-24 10:12 +0200
        Re: Future of Pypy? Emile van Sebille <emile@fenx.com> - 2015-02-24 09:57 -0800
    Re: Future of Pypy? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-02-23 01:05 +1100
      Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 15:44 +0000
        Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-22 19:20 +0000
          Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-22 22:45 +0100
            Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-23 14:04 +0000
              Re: Future of Pypy? Laura Creighton <lac@openend.se> - 2015-02-23 17:16 +0100
    Re: Future of Pypy? Terry Reedy <tjreedy@udel.edu> - 2015-02-23 01:34 -0500
    Re: Future of Pypy? Dave Cook <davecook@nowhere.net> - 2015-02-23 11:36 +0000
      Re: Future of Pypy? Dave Farrance <DaveFarrance@OMiTTHiSyahooANDTHiS.co.uk> - 2015-02-23 14:13 +0000
        Cython - was: Future of Pypy? Stefan Behnel <stefan_ml@behnel.de> - 2015-02-23 16:43 +0100
        Re: Future of Pypy? Dave Cook <davecook@nowhere.net> - 2015-02-23 23:23 +0000

Page 1 of 4  [1] 2 3 4  Next page →

#86105 — Future of Pypy?

As an engineer, I can quickly knock together behavioural models of
electronic circuits,  complete units, and control systems in Python, then
annoyingly in a few recent cases, have to re-write in C for speed.

I've tried PyPy, the just-in-time compiler for Python, and that is
impressively, hugely fast in comparison, but it's no good making these
models if I can't display the results in a useful way, and at the moment
PyPy just doesn't have the huge range of useful time-saving libraries that
CPython has.  It's still quicker to do a re-write in the more cumbersome C
than try to work with PyPy because C, like CPython, also has many useful
libraries.

A few years back, I recall people saying that PyPy was going to be the
future of Python, but it seems to me that CPython still has the lion's
share of the momentum, is developing faster and has ever more libraries,
while PyPy is struggling to get enough workers to even get Numpy
completed.

Maybe there's not enough people like me that have really felt the need for
the speed.  Or maybe it's simply the accident of the historical
development path that's set-in-stone an interpreter rather than a JIT.
Anybody got a useful perspective on this?

[toc] | [next] | [standalone]

#86108

On Sunday, 22 February 2015 12:45:15 UTC, Dave Farrance  wrote:
> As an engineer, I can quickly knock together behavioural models of
> electronic circuits,  complete units, and control systems in Python, then
> annoyingly in a few recent cases, have to re-write in C for speed.
> 
> I've tried PyPy, the just-in-time compiler for Python, and that is
> impressively, hugely fast in comparison, but it's no good making these
> models if I can't display the results in a useful way, and at the moment
> PyPy just doesn't have the huge range of useful time-saving libraries that
> CPython has.  It's still quicker to do a re-write in the more cumbersome C
> than try to work with PyPy because C, like CPython, also has many useful
> libraries.
> 
> A few years back, I recall people saying that PyPy was going to be the
> future of Python, but it seems to me that CPython still has the lion's
> share of the momentum, is developing faster and has ever more libraries,
> while PyPy is struggling to get enough workers to even get Numpy
> completed.
> 
> Maybe there's not enough people like me that have really felt the need for
> the speed.  Or maybe it's simply the accident of the historical
> development path that's set-in-stone an interpreter rather than a JIT.
> Anybody got a useful perspective on this?

I'm curious what ...behavioural... models you are creating quickly in Python that then need rewriting in C for speed. SPICE? some other CAD? Might be interesting to learn more about what and how you are actually doing.

How about running your front end (simulation) work in PyPy, and the backend display work on CPython, if there are some missing features in PyPy that you need. This may be more or less easy depending on your requirements and any intermediate format you have.

Or you could offer to assist in the PyPy porting? Or express an interest in specific libraries being ported?

    Cheers
    Jon N

[toc] | [prev] | [next] | [standalone]

#86127

jkn <jkn_gg@nicorp.f9.co.uk> wrote:

> I'm curious what ...behavioural... models you are creating quickly in
> Python that then need rewriting in C for speed. SPICE? some other CAD?
> Might be interesting to learn more about what and how you are actually
> doing.

The convert-to-C cases were complex filtering functions.  I do make good
use of spice-based tools, but I often find it useful to make a more
abstract model, usually before completing the design.  This helps with
component selection, finalizing the design, and making sure that I
understood the what the circuit would do.

I started work 1980ish, had an early 6502-based home computer, and my then
place of work had some 6502-based Pet computers, so I gained the ability
to quickly write BASIC programs as an engineering aid.  Later, when BASIC
dropped into obscurity, I switched to C and C++, although I always found
that cumbersome compared to the old BASIC.  Later still, when I found that
my Google queries for code examples started returning more Python than C,
I tried that -- and discovered that Python was like BASIC, only better.

But that's just me.  Other hardware engineers use a variety of modeling
applications.  Or don't need to because they're just that clever?  Or they
give the modeling work to system engineers who will use whatever apps that
system engineers use, and will return a result a few weeks later.
Personally, I've tended to get used to writing code in just one
general-purpose language, and it seems to me that I get a useful result
relatively quickly.

> How about running your front end (simulation) work in PyPy, and the
> backend display work on CPython, if there are some missing features in
> PyPy that you need. This may be more or less easy depending on your
> requirements and any intermediate format you have.

Maybe I should look at that again.  In the case of the filter models,
their usefulness had grown to the point that requiring support by other
people was a possibility, so converting them to C seemed better than
writing something that bridged between two language implementations.

> Or you could offer to assist in the PyPy porting? Or express an interest
> in specific libraries being ported?

I'm a hardware engineer not a software engineer, so I have to plead lack
of ability there.  I do appreciate the work that's done on Python, and
have to be grateful for what is available, since I'm not paying for it.

[toc] | [prev] | [next] | [standalone]

#86279 — [OT] - BASIC is still not a bad choice, was Re: Future of Pypy?

On 02/22/2015 08:30 AM, Dave Farrance wrote:
> I started work 1980ish, had an early 6502-based home computer, and my then
> place of work had some 6502-based Pet computers, so I gained the ability
> to quickly write BASIC programs as an engineering aid.  Later, when BASIC
> dropped into obscurity, I switched to C and C++, although I always found
> that cumbersome compared to the old BASIC.  Later still, when I found that
> my Google queries for code examples started returning more Python than C,
> I tried that -- and discovered that Python was like BASIC, only better.

I know BASIC is a hiss and a byword for many folks, and certainly it's
off topic here.  But modern BASIC, particarly in the form of the
FreeBASIC[1] compiler, is very much alive and well on all modern
platforms (including Windows, Mac, Linux, and Arm devices).  BASIC today
appears to be every bit as structured as any modern language.

It's relatively easy to translate C header files into FreeBASIC include
files, but it is tedious.  I would love to have a FreeBASIC translation
of the libpython header files and then I could write Python extensions
in FreeBASIC.

Back on topic, have you tried using Cython to compile python modules
that need a speedup into a binary module?

[1] http://www.freebasic.net/

[toc] | [prev] | [next] | [standalone]

#86112

In a message of Sun, 22 Feb 2015 12:45:03 +0000, Dave Farrance writes:

>Maybe there's not enough people like me that have really felt the need for
>the speed.  Or maybe it's simply the accident of the historical
>development path that's set-in-stone an interpreter rather than a JIT.
>Anybody got a useful perspective on this?

I don't understand 'an interpreter rather than a JIT'.  PyPy has a
JIT, that sort of is the whole point.

One problem is that hacking on PyPy itself is hard.  Lots of people
find it too hard, and give up.  (Of course, lots of people give up
on hacking CPython too.  I think that hacking on PyPy is harder than
hacking on CPython, but I am quite biased.)  So this is a barrier to
getting more people to work on it.

Provided your code is in pure python, PyPy already works great for you.
as you found out.  Pure Python libraries aren't a problem, either.

The problem arises when you want to add your fancy graphic library to
what you do, because chances are that fancy library is a C extension.
And there is no magic 'sprinke this dust here' on a C extension that
makes it acceptable to PyPy.  It's a hard job.  The PyPy team has
gone after, and rewritten how to do this I think 5 times now.  Maybe
more.  Every time the goal has been to make it easier for the average
programmer to make an interface, and, of course to not slow things down
too much.  C extensions, in general, make PyPy code run a lot slower because
you cannot get the JIT in there to speed things up, so you may be
stuck with unjitted PyPy performance, even on your python code,
which isn't speedy.  You also find lots of bugs in the C extensions,
which don't get noticed until you, for instance, no longer have a ref
counting GC.

Some of the things aren't really bugs, exactly, just that the person
who wrote the thing knew far, far, too much about how CPython works
and has produced something that had no need or desire to be portable
anywhere else.  The closer the person who wrote the extension
was 'to the metal' .. knowing _exactly_ how CPython does things and how
to squeeze that for the tiniest last drop of performance improvement ....
the harder things are for whoever wants to get it to work with PyPy, which
has a competely different architecture and a whole lot of other assumptions.

And the slower that it will run.  So, if it is a small thing, then the
usual suggestion is to rewrite it in pure python and let the JIT handle
it.  Very, very often the result is faster than the pure C code, but
clearly this isn't something you want to do with a huge graphics
library ...

There is hope that we can take another crack at this problem using
things learned from the Transactional Memory stuff, but nobody is promising
anything yet.  Also Armin has had a new neat thought.

https://mail.python.org/pipermail/pypy-dev/2015-February/013085.html

If we can get rid of the warmup time, then PyPy should be more popular
than it is now.  Lots of people run PyPy once, un-warmed up, see no
big improvement, and decide it's not for them.

But the problem of

'here is a huge chunk of C code, designed to work with Language X.
Now make it work with Language Y (PyPy) which isn't written in C'
can only be simplified to a certain extent.  There comes a point
where _this is just bloody hard_.

Laura

[toc] | [prev] | [next] | [standalone]

#86129

Laura Creighton <lac@openend.se> wrote:

>I don't understand 'an interpreter rather than a JIT'.  PyPy has a
>JIT, that sort of is the whole point.

Yes.  I meant that from my end-user, non-software-engineer perspective, it
looked as though CPython was proceeding with leaps and bounds and that
PyPy remained mostly a proof-of-concept after several years.

But thanks for your description of the issues.  So once the core issues
are sorted out, if enough people can be found to work on it, then
hopefully the library conversions will follow apace.

[toc] | [prev] | [next] | [standalone]

#86137

In a message of Sun, 22 Feb 2015 15:36:42 +0000, Dave Farrance writes:
>Laura Creighton <lac@openend.se> wrote:
>
>>I don't understand 'an interpreter rather than a JIT'.  PyPy has a
>>JIT, that sort of is the whole point.
>
>Yes.  I meant that from my end-user, non-software-engineer perspective, it
>looked as though CPython was proceeding with leaps and bounds and that
>PyPy remained mostly a proof-of-concept after several years.

Oh no, that was the state of the world about 8 years ago.  PyPy works
great, and more and more people are using it in production all the
time.

>But thanks for your description of the issues.  So once the core issues
>are sorted out, if enough people can be found to work on it, then
>hopefully the library conversions will follow apace.

Well, most libraries just run, of course. It's the ones that are written
in C that cause most of the problems.

One of the problems, from a pure Have-PyPy-Take-Over-the-World
perspective is that as things were moving along quite merrily,
the CPython core developers decided to invent Python 3.0.  This
meant that everybody on the planet, who wanted their library to
work with Python 3.0 had to convert it to work there.

There was, and still is, an enormous amount of resentment about this.
For a lot of people, the perception was, and still is that the benefits
of Python 3.x over Python 2.x was not worth breaking backwards
compatibility.  And there still are plenty of places whose plan is
to use Python 2.7 indefinitely into the far future.  I've got 15
years worth of commerical python code out there in the world, and
nobody wants it converted enough to pay me to do so.  Their position
is that it runs quite well enough as it is.  I'm sure not
going to convert the stuff for fun.  Practically every Python consultant on
the planet is in the same boat.

Things will get converted when there is a compelling business argument
to do so, but not before, and for a lot of code the answer is never.

Given the nature of this political problem, it is not surprising that
the proponents of Python 3.0 spent a lot of effort talking about the
benefits, and praising the people who converted their stuff, and
making a huge effort in the public relations lines.  The whole thing
could have blown up in their faces, as is quite common when you decide
to make the 'second version' of a language.  It happened to Perl.  So
this creates buzz and warm feelings about Python 3.0.

In contrast, on the PyPy team, there is nobody who doesn't consider
public relations and marketing and 'creating the warm fuzzy feelings in
the users' somewhere between 'unpleasant duty' and 'sheer torture'.
The set of human skills you need to have to be good as this sort of
thing is not a set that we have, either collectively or in individuals.
We're much more into 'letting the code speak for itself', which, of
course, it does not do.

A lot of us, indeed were raised on a philosophy that it is morally wrong
to try to influence people.  You can give them options, and you can
even explain the options that you are giving them, and you can argue
with others in favour of certain options, but when it comes right down to
it, everybody has to make their own decision.

This is all well and virtuous, but the fact of the matter is that a large
number of people aren't competant to make their own decisions, and even
among those that are there exist a large number who very
much don't want to do such a thing.  If you are trying to get such people
to adopt your software, you need to provide a completely different
experience.  They need to feel comfortable, and safe, and among a
large community, and, well, I don't know what else they want.  That is
part of the problem.  I am pretty sure that what they want is something
that I never pay a lot of attention to. I mean  I'm a charter member of
the 'always-sacrifice-comfort-in-order-to-have-fun-and-interesting-times'
club.  And my marketing skills, such as they are, are much above average
for the PyPy gang - though some members are learning a bit, slowly, through
necessity.  But you notice that we have only 1 blog, and things are added
to it very slowly.  There are people all over planet python who blog about
things every week, for fun.  There is no way we can compete with them.

So, until some people with such skills decide to take an interest in
PyPy, our marketing effort is going to limp. And I personally feel
pretty bad about asking some poor soul who has just made his C extension
work with 3.0 go back and do it _again_ in a more PyPy friendly way.

But if Armin gets the Transactional Memory to be usable in a robust
way, (as opposed to now where it is only a bit more than a proof of
concept) then things could rocket off again.  Because one thing we
do know is that people who are completely and utterly ignorant about
whether having multiple cores will improve their code still want to
use a language that lets them use the multiple processors.  If the
TM dream of having that just happen, seemlessly (again, no promises)
is proven to be true, well ....  we think that the hordes will suddenly
be interested in PyPy.

But you never know.  Lots of people were doing
-complain-complain-python-is-too-slow, and wouldn't take the argument
that developer time is worth more than CPU time and the like. We
made them PyPy.  They've stopped complaining about the speed, but they
aren't using PyPy either.  Seems they didn't really need the speed,
after all.  It is tempting to believe that they just liked
complaining.  So maybe the 'cannot use all my cores' complaint is
fueled by other people who complain for the fun of it.  Then there
will be no hordes even when we crack that nut too.

But it is very, very interesting and a whole lot of fun.

Laura

[toc] | [prev] | [next] | [standalone]

#86144

Laura Creighton <lac@openend.se> writes:
> Because one thing we do know is that people who are completely and
> utterly ignorant about whether having multiple cores will improve
> their code still want to use a language that lets them use the
> multiple processors.  If the TM dream of having that just happen,
> seemlessly (again, no promises) is proven to be true, well ....  we
> think that the hordes will suddenly be interested in PyPy.

TM is a useful feature but it's unlikely to be the thing that attracts
"the hordes".  More important is to eliminate the GIL and hopefully have
lightweight (green) threads that can still run on multiple cores, like
in GHC and Erlang.  Having them communicate by mailboxes/queues is
sufficient most of the time, and in Erlang it's the only method allowed
in theory (there are some optimizations taking place behind the scenes).
TM hasn't gotten that much uptake in GHC (one of the earliest HLL
implementations of TM) in part because its performance cost is
significant when there's contention.  I wonder if Clojure programmers
use it more.

[toc] | [prev] | [next] | [standalone]

#86151

In a message of Sun, 22 Feb 2015 11:02:29 -0800, Paul Rubin writes:
>Laura Creighton <lac@openend.se> writes:
>> Because one thing we do know is that people who are completely and
>> utterly ignorant about whether having multiple cores will improve
>> their code still want to use a language that lets them use the
>> multiple processors.  If the TM dream of having that just happen,
>> seemlessly (again, no promises) is proven to be true, well ....  we
>> think that the hordes will suddenly be interested in PyPy.
>
>TM is a useful feature but it's unlikely to be the thing that attracts
>"the hordes".  More important is to eliminate the GIL and hopefully have
>lightweight (green) threads that can still run on multiple cores, like
>in GHC and Erlang.  Having them communicate by mailboxes/queues is
>sufficient most of the time, and in Erlang it's the only method allowed
>in theory (there are some optimizations taking place behind the scenes).
>TM hasn't gotten that much uptake in GHC (one of the earliest HLL
>implementations of TM) in part because its performance cost is
>significant when there's contention.  I wonder if Clojure programmers
>use it more.

The GIL isn't going away from PyPy any time real soon, alas.  Armin has
some pretty cool ideas about what to do about contention, but if
you want to hear them, its better if you go post that to pypy-dev@python.org
so you can get it from the man directly rather that hearing my
paraphrase.  Or ask away on the #pypy channel on freenode ...

But this reminds me that I have to get Lennart Augustsson and Armin
Rigo in the same room some time.  Should be fun.

Laura

[toc] | [prev] | [next] | [standalone]

#86153

Laura Creighton <lac@openend.se> writes:
> The GIL isn't going away from PyPy any time real soon, alas.

I thought the GIL's main purpose was to avoid having to lock all the
CPython refcount updates, so if PyPy has tracing GC, why is there still
a GIL?  And how is TM going to help with parallelism if the GIL is still
there?

> Armin has some pretty cool ideas about what to do about contention,
> but if you want to hear them, its better if you go post that to
> pypy-dev@python.org...  Or ask away on the #pypy channel on freenode

It would be nice if he blogged something about them.

> But this reminds me that I have to get Lennart Augustsson and Armin
> Rigo in the same room some time.  Should be fun.

I thought the STM stuff in GHC was done by the Simon's.  Armin should
certainly have Simon Marlow's book about concurrency and Haskell:

http://chimera.labs.oreilly.com/books/1230000000929/index.html

[toc] | [prev] | [next] | [standalone]

#86163

In a message of Sun, 22 Feb 2015 12:14:45 -0800, Paul Rubin writes:
>Laura Creighton <lac@openend.se> writes:
>> The GIL isn't going away from PyPy any time real soon, alas.
>
>I thought the GIL's main purpose was to avoid having to lock all the
>CPython refcount updates, so if PyPy has tracing GC, why is there still
>a GIL?  And how is TM going to help with parallelism if the GIL is still
>there?

This requires a long answer.   a very long answer.
More later, I must work this evening.

>> Armin has some pretty cool ideas about what to do about contention,
>> but if you want to hear them, its better if you go post that to
>> pypy-dev@python.org...  Or ask away on the #pypy channel on freenode
>
>It would be nice if he blogged something about them.

You are asking for water to roll up-hill.  If you want the joy of
hearing the cool ideas as Armin has them, you need to hang out on
the irc channel.  Of course, if you are interested in such things
this makes hanging out there worthwhile.

>> But this reminds me that I have to get Lennart Augustsson and Armin
>> Rigo in the same room some time.  Should be fun.
>
>I thought the STM stuff in GHC was done by the Simon's.  Armin should
>certainly have Simon Marlow's book about concurrency and Haskell:

Of course, but if you think that Lennart Augustsson is not familiar
with every aspect of every Haskell compiler on the planet .... well
then I know Lennart better than you do.  And given that Lennart is
a friend, well really a good friend of my lover and a something-better-
than-an-acquaintance with me ---- I should make the effort to get these
two under the same roof (mine, by preference) for the fun of the
experience.

So thank you for giving me this idea ...

Laura

[toc] | [prev] | [next] | [standalone]

#86181

Laura Creighton <lac@openend.se> writes:
> And given that Lennart is a friend, well really a good friend of my
> lover and a something-better- than-an-acquaintance with me ---- I
> should make the effort to get these two under the same roof (mine, by
> preference) for the fun of the experience.

Oh cool, right, I forgot that you are in Sweden where he is.  I've never
met him but he's pretty visible online and yes, he sure knows a lot
about Haskell.

[toc] | [prev] | [next] | [standalone]

#86173

Paul Rubin wrote:

> Laura Creighton <lac@openend.se> writes:
>> Because one thing we do know is that people who are completely and
>> utterly ignorant about whether having multiple cores will improve
>> their code still want to use a language that lets them use the
>> multiple processors.  If the TM dream of having that just happen,
>> seemlessly (again, no promises) is proven to be true, well ....  we
>> think that the hordes will suddenly be interested in PyPy.
> 
> TM is a useful feature but it's unlikely to be the thing that attracts
> "the hordes".  More important is to eliminate the GIL 

*rolls eyes*

I'm sorry, but the instant somebody says "eliminate the GIL", they lose
credibility with me. Yes yes, I know that in *your* specific case you've
done your research and (1) multi-threaded code is the best solution for
your application and (2) alternatives aren't suitable.

Writing multithreaded code is *hard*. It is not a programming model which
comes naturally to most human beings. Very few programs are inherently
parallelizable, although many programs have *parts* which can be
successfully parallelized. 

I think that for many people, "the GIL" is just a bogeyman, or is being
blamed for their own shortcomings. To take an extreme case, if you're
running single-thread code on a single-core machine and still complaining
about the GIL, you have no clue.

(That's not *you personally* Paul, it's a generic "you".)

There are numerous alternatives for those who are genuinely running into
GIL-related issues. Jeff Knupp has a good summary:

http://www.jeffknupp.com/blog/2013/06/30/pythons-hardest-problem-revisited/

One alternative that he misses is that for some programs, the simplest way
to speed it up is to vectorize the core parts of your code by using numpy.
No threads needed.

For those who think that the GIL and the GIL alone is the problem, consider
that Jython is nearly as old as CPython, it goes back at least 15 years.
IronPython has been around for a long time too, and is possibly faster than
CPython even in single threaded code. Neither has a GIL. Both are mature
implementations, built on well-known, powerful platforms with oodles of
business credibility (the JVM and .Net). IronPython even has the backing of
Microsoft, it is one of the few non-Microsoft languages with a privileged
position in the .Net ecosystem.

Where are the people flocking to use Jython and IronPython?

In fairness, there are good reasons why some people cannot use Jython or
IronPython, or one of the other alternatives. But that demonstrates that
the problem is more complex than just "the GIL".

For removal of the GIL to really make a difference:

- you must have at least two cores (that, at least, applies to most people
these days);

- you must be performing a task which is parallelizable and not inherently
sequential (no point using multiple threads if each thread spends all its
time waiting for the previous thread);

- the task must be one that moving to some other multi-processing model
(such as greenlets, multiprocess, etc.) is infeasible;

- you must actually use multiple threads, and use them properly (no busy
wait loops);

- your threading bottleneck must be primarily CPU-bound, not I/O bound
(CPython's threads are already very effective at parallelising I/O tasks);

- and you must be using libraries and tools which prevent you moving to
Jython or IronPython or some other alternative.

I can't help but feel that the set of people for whom removal of the GIL
would actually help is much smaller than, and different to, the set of
people who complain about the GIL.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#86176

Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:
> I'm sorry, but the instant somebody says "eliminate the GIL", they lose
> credibility with me. Yes yes, I know that in *your* specific case you've
> done your research and (1) multi-threaded code is the best solution for
> your application and (2) alternatives aren't suitable.

I don't see what the big deal is.  I hear tons of horror stories about
threads and I believe them, but the thing is, they almost always revolve
around acquiring and releasing locks in the wrong order, forgetting to
lock things, stuff like that.  So I've generally respected the terror
and avoided programming in that style, staying with a message passing
style that may take an efficiency hit but seems to avoid almost all
those problems.  TM also helps with lock hazards and it's a beautiful
idea--I just haven't had to use it yet.  The Python IRC channel seems to
rage against threads and promote Twisted for concurrency, but Twisted
has always reminded me of Java.  I use threads in Python all the time
and haven't gotten bitten yet.

> Writing multithreaded code is *hard*. It is not a programming model
> which comes naturally to most human beings. Very few programs are
> inherently parallelizable, although many programs have *parts* which
> can be successfully parallelized.

Parallel algorithms are complicated and specialized but tons of problems
amount to "do the same thing with N different pieces of data", so-called
embarassingly parallel.  The model is you have a bunch of worker threads
reading off a queue and processing the items concurrently.  Sometimes
separate processes works equally well, other times it's good to have
some shared data in memory instead of communicating through sockets.  If
the data is mutable then have one thread own it and access it only with
message passing, Erlang style.  If it's immutable after initialization
(protect it with a condition variable til initialization finishes) then
you can have read-only access from anywhere.

> if you're running single-thread code on a single-core machine and
> still complaining about the GIL, you have no clue.

Even the Raspberry Pi has 4 cores now, and fancy smartphones have had
them for years.  Single core cpu's are specialized and/or historical.

> for some programs, the simplest way to speed it up is to vectorize the
> core parts of your code by using numpy.  No threads needed.

Nice for numerical codes, not so hot for anything else.

> Where are the people flocking to use Jython and IronPython?

Shrug, who knows, those implementations were pretty deficient from what
heard.

> For removal of the GIL to really make a difference: ...
> - you must be performing a task which is parallelizable and not inherently
> sequential (no point using multiple threads if each thread spends all its
> time waiting for the previous thread);

That's most things involving concurrency these days.

> - the task must be one that moving to some other multi-processing model
> (such as greenlets, multiprocess, etc.) is infeasible;

I don't understand this--there can be multiple ways to solve a problem.

> - your threading bottleneck must be primarily CPU-bound, not I/O bound
> (CPython's threads are already very effective at parallelising I/O tasks);

If your concurrent program's workload makes it cpu-bound even 1% of the
time, then you gain something by having it use your extra cores at those
moments, instead of having those cores always do nothing.

> - and you must be using libraries and tools which prevent you moving to
> Jython or IronPython or some other alternative.

I don't get this at all.  Why should I not want Python to have the same
capabilities?

[toc] | [prev] | [next] | [standalone]

#86177

On Mon, Feb 23, 2015 at 1:04 PM, Paul Rubin <no.email@nospam.invalid> wrote:
>> if you're running single-thread code on a single-core machine and
>> still complaining about the GIL, you have no clue.
>
> Even the Raspberry Pi has 4 cores now, and fancy smartphones have had
> them for years.  Single core cpu's are specialized and/or historical.

Or virtual. I have a quad-core + hyperthreading CPU, but most of my
VMs (in fact, all except the one that runs a Python buildbot, I
think!) are restricted to a single core. If you rent a basic cheap
machine from a cloud provider, you'll quite possibly be getting a
single core, too.

ChrisA

[toc] | [prev] | [next] | [standalone]

#86186

[Multipart message — attachments visible in raw view] — view raw

On Mon Feb 23 2015 at 12:05:46 PM Paul Rubin <no.email@nospam.invalid>
wrote:

> I don't see what the big deal is.  I hear tons of horror stories about
> threads and I believe them, but the thing is, they almost always revolve
> around acquiring and releasing locks in the wrong order, forgetting to
> lock things, stuff like that.  So I've generally respected the terror
> and avoided programming in that style, staying with a message passing
> style that may take an efficiency hit but seems to avoid almost all
> those problems.  TM also helps with lock hazards and it's a beautiful
> idea--I just haven't had to use it yet.  The Python IRC channel seems to
> rage against threads and promote Twisted for concurrency, but Twisted
> has always reminded me of Java.  I use threads in Python all the time
> and haven't gotten bitten yet.
>
>
Many people have written at length about why it's bad. The most recent
example I have come across is here ->
https://glyph.twistedmatrix.com/2014/02/unyielding.html

It's not a specific Python problem. I must be in the limited crowd that
believes that the GIL is a *feature* of Python. Then again, maybe it isn't
so limited since the GIL-less python implementations haven't really taken
off.

I have yet to come across a scenario I couldn't solve with either
Processes, NumPy or event loops. Yes, when using processes, the passing of
data can be annoying sometimes. But that is far less annoying then trying
to debug something that shares state across threads.

It's great that you haven't been bitten yet. But, the evidence seems to
suggest the either you *will* be bitten at some point or, you already have
been, and you just don't know it.

Cheers


> > Writing multithreaded code is *hard*. It is not a programming model
> > which comes naturally to most human beings. Very few programs are
> > inherently parallelizable, although many programs have *parts* which
> > can be successfully parallelized.
>
> Parallel algorithms are complicated and specialized but tons of problems
> amount to "do the same thing with N different pieces of data", so-called
> embarassingly parallel.  The model is you have a bunch of worker threads
> reading off a queue and processing the items concurrently.  Sometimes
> separate processes works equally well, other times it's good to have
> some shared data in memory instead of communicating through sockets.  If
> the data is mutable then have one thread own it and access it only with
> message passing, Erlang style.  If it's immutable after initialization
> (protect it with a condition variable til initialization finishes) then
> you can have read-only access from anywhere.
>
> > if you're running single-thread code on a single-core machine and
> > still complaining about the GIL, you have no clue.
>
> Even the Raspberry Pi has 4 cores now, and fancy smartphones have had
> them for years.  Single core cpu's are specialized and/or historical.
>
> > for some programs, the simplest way to speed it up is to vectorize the
> > core parts of your code by using numpy.  No threads needed.
>
> Nice for numerical codes, not so hot for anything else.
>
> > Where are the people flocking to use Jython and IronPython?
>
> Shrug, who knows, those implementations were pretty deficient from what
> heard.
>
> > For removal of the GIL to really make a difference: ...
> > - you must be performing a task which is parallelizable and not
> inherently
> > sequential (no point using multiple threads if each thread spends all its
> > time waiting for the previous thread);
>
> That's most things involving concurrency these days.
>
> > - the task must be one that moving to some other multi-processing model
> > (such as greenlets, multiprocess, etc.) is infeasible;
>
> I don't understand this--there can be multiple ways to solve a problem.
>
> > - your threading bottleneck must be primarily CPU-bound, not I/O bound
> > (CPython's threads are already very effective at parallelising I/O
> tasks);
>
> If your concurrent program's workload makes it cpu-bound even 1% of the
> time, then you gain something by having it use your extra cores at those
> moments, instead of having those cores always do nothing.
>
> > - and you must be using libraries and tools which prevent you moving to
> > Jython or IronPython or some other alternative.
>
> I don't get this at all.  Why should I not want Python to have the same
> capabilities?
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#86190

Ryan Stuart <ryan.stuart.85@gmail.com> writes:
> Many people have written at length about why it's bad. The most recent
> example I have come across is here ->
> https://glyph.twistedmatrix.com/2014/02/unyielding.html

That article is about the hazards of mutable state shared between
threads.  The key to using threads safely is to not do that.  So the
"transfer" example in the article would instead be a message handler in
the thread holding the account data, and it would do the transfer in the
usual sequential way.  You'd start a transfer by sending a message
through a Queue, and get back a reply through another queue.

In Erlang that style is enforced: it has basically no such thing as data
mutation or sharing.  In Python it's straightforward to write in that
style; it's just that the language doesn't stop you from departing from
it, sort of like a language with weak type-checking.  You can survive it
if the code complexity isn't too bad.  In most programs I've dealt with,
the number of distinct handlers is not all that large (there might be
1000 threads in a high-concurrency network server, but they're all doing
about the same thing).  So it hasn't been that hard to "color inside the
lines" with a bit of thoughtfulness and code inspection.

You might like this:

http://jlouisramblings.blogspot.com/2012/08/getting-25-megalines-of-code-to-behave.html

[toc] | [prev] | [next] | [standalone]

#86191

[Multipart message — attachments visible in raw view] — view raw

On Mon Feb 23 2015 at 1:50:40 PM Paul Rubin <no.email@nospam.invalid> wrote:

> That article is about the hazards of mutable state shared between
> threads.  The key to using threads safely is to not do that.  So the
> "transfer" example in the article would instead be a message handler in
> the thread holding the account data, and it would do the transfer in the
> usual sequential way.  You'd start a transfer by sending a message
> through a Queue, and get back a reply through another queue.
>

I think that is a pretty accurate summary. In fact, the article even says
that. So, just to iterate its point, if you are using non-blocking Queues
to communicate to these threads, then you just have a communicating event
loop. Given that Queues work perfectly with with processes as well, what is
the point of using a thread? Using a process/fork is far safer in that
someone can't "accidentally" decide to alter mutable state in the future.


> You might like this:
>
> http://jlouisramblings.blogspot.com/2012/08/getting-
> 25-megalines-of-code-to-behave.html


Thanks for this, I'll take a look.

Cheers


>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#86193

Ryan Stuart <ryan.stuart.85@gmail.com> writes:
> I think that is a pretty accurate summary. In fact, the article even
> says that. So, just to iterate its point, if you are using
> non-blocking Queues to communicate to these threads, then you just
> have a communicating event loop. Given that Queues work perfectly with
> with processes as well, what is the point of using a thread?

What do you mean about Queues working with processes?  I meant
Queue.Queue.  There is multiprocessing.Queue but that's much less
capable, and it uses cumbersome IPC like pipes or sockets instead of a
lighter weight lock.  Threads can also share read-only data and you can
pass arbitrary objects (such as code callables that you want the other
thread to execute--this is quite useful) through Queue.Queue.  I don't
think you can do that with the multiprocessing module.

[toc] | [prev] | [next] | [standalone]

#86199

[Multipart message — attachments visible in raw view] — view raw

On Mon Feb 23 2015 at 4:15:42 PM Paul Rubin <no.email@nospam.invalid> wrote:
>
> What do you mean about Queues working with processes?  I meant
> Queue.Queue.  There is multiprocessing.Queue but that's much less
> capable, and it uses cumbersome IPC like pipes or sockets instead of a
> lighter weight lock.  Threads can also share read-only data and you can
> pass arbitrary objects (such as code callables that you want the other
> thread to execute--this is quite useful) through Queue.Queue.  I don't
> think you can do that with the multiprocessing module.
>

These things might be convenient but they are error prone for the reasons
pointed out. Also, the majority can be achieved via the process approach.
For example, using fork to take a copy of the current process (including
the heap) you want to use will give you access to any callables on the
heap.

The vital point here is that fork takes a *copy* of the process and runs in
a *separate* memory space. This means there can be no accidents here. If it
were to run in the same memory space like a thread then bugs anywhere in
the code you run could cause very nasty problems. This includes not only
bugs in your code, but also bugs in any other library code. Not only are
they nasty, they could be close to invisible.

And when I saw any code you run, I literally mean any. Even if you are
extra careful to not touch any shared state in your code, you can almost be
guaranteed that code higher up the stack, like malloc for example, *will*
be using shared state.

The risk of unintended and difficult to track issues when using threads is
very high because of shared state. Even if you aren't sharing state in your
code directly, code higher up the stack will be sharing state. That is the
whole point of a thread, that's what they were invented for. Using threads
safely might well be impossible much less verifiable. So when there are
other options that are just as viable/functional, result in far less risk
and are often much quicker to implement correctly, why wouldn't you use
them? If it were easy to use threads in a verifiably safe manner, then
there probably wouldn't be a GIL.

Cheers


> --
> https://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

Page 1 of 4  [1] 2 3 4  Next page →

Back to top | Article view | comp.lang.python

csiph-web