Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #26472 > unrolled thread

On-topic: alternate Python implementations

Back to article view | Back to comp.lang.python

Contents

  On-topic: alternate Python implementations Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-08-04 06:15 +0000
    Re: On-topic: alternate Python implementations Chris Angelico <rosuav@gmail.com> - 2012-08-04 16:34 +1000
      Re: On-topic: alternate Python implementations Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-08-04 10:54 +0000
        Re: On-topic: alternate Python implementations Stefan Krah <stefan-usenet@bytereef.org> - 2012-08-04 13:18 +0200
          Re: On-topic: alternate Python implementations Paul Rubin <no.email@nospam.invalid> - 2012-08-04 08:59 -0700
            Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-04 18:55 +0200
              Re: On-topic: alternate Python implementations Paul Rubin <no.email@nospam.invalid> - 2012-08-04 11:18 -0700
                Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-04 21:06 +0200
                  Re: On-topic: alternate Python implementations Paul Rubin <no.email@nospam.invalid> - 2012-08-04 13:43 -0700
                    Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-04 23:24 +0200
                Re: On-topic: alternate Python implementations MRAB <python@mrabarnett.plus.com> - 2012-08-04 20:24 +0100
            Re: On-topic: alternate Python implementations Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-08-05 00:54 +0000
              Re: On-topic: alternate Python implementations Paul Rubin <no.email@nospam.invalid> - 2012-08-04 18:38 -0700
                Re: On-topic: alternate Python implementations Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-08-05 02:19 +0000
                  Re: On-topic: alternate Python implementations John Nagle <nagle@animats.com> - 2012-08-06 22:57 -0700
                Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-05 07:37 +0200
              Re: On-topic: alternate Python implementations Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-04 23:09 -0400
        Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-04 13:32 +0200
    Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-04 08:40 +0200
      Re: On-topic: alternate Python implementations Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-08-04 07:49 +0000
        Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-04 11:10 +0200
          Re: On-topic: alternate Python implementations Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2012-08-04 14:51 +0200
            Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-04 15:53 +0200
              Re: On-topic: alternate Python implementations Ulrich Eckhardt <ulrich.eckhardt@dominolaser.com> - 2012-08-08 10:29 +0200
            Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-04 16:03 +0200
        Re: On-topic: alternate Python implementations Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-08-04 11:05 +0100
        Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-04 12:59 +0200
        Re: On-topic: alternate Python implementations Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-08-04 19:24 +0100
          Re: On-topic: alternate Python implementations Temia Eszteri <lamialily@cleverpun.com> - 2012-08-04 11:34 -0700
          Re: On-topic: alternate Python implementations Ben Finney <ben+python@benfinney.id.au> - 2012-08-06 01:21 +1000
        Re: On-topic: alternate Python implementations Zero Piraeus <schesis@gmail.com> - 2012-08-04 14:42 -0400
        Re: On-topic: alternate Python implementations Zero Piraeus <schesis@gmail.com> - 2012-08-04 14:56 -0400
        Re: On-topic: alternate Python implementations Ethan Furman <ethan@stoneleaf.us> - 2012-08-05 07:27 -0700
    Re: On-topic: alternate Python implementations Tim Roberts <timr@probo.com> - 2012-08-04 13:07 -0700
    Re: On-topic: alternate Python implementations jwp <james.pye@gmail.com> - 2012-08-04 15:05 -0700
    Re: On-topic: alternate Python implementations Jürgen A. Erhard <jae+python@jaerhard.com> - 2012-08-05 01:25 +0200
    Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-05 07:46 +0200
    Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-05 09:51 +0200
    Re: On-topic: alternate Python implementations Jürgen A. Erhard <jae+python@jaerhard.com> - 2012-08-05 14:28 +0200
    Re: On-topic: alternate Python implementations alex23 <wuwei23@gmail.com> - 2012-08-05 20:40 -0700
      Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-06 08:21 +0200
    Re: On-topic: alternate Python implementations Stefan Behnel <stefan_ml@behnel.de> - 2012-08-06 08:46 +0200
    Alternate Python extensions (was alternate Python implementations) rusi <rustompmody@gmail.com> - 2012-08-06 21:23 -0700
      Re: Alternate Python extensions (was alternate Python implementations) Stefan Behnel <stefan_ml@behnel.de> - 2012-08-07 07:09 +0200

Page 1 of 3  [1] 2 3  Next page →

#26472 — On-topic: alternate Python implementations

Most people are aware, if only vaguely, of the big Four Python 
implementations:

CPython, or just Python, the reference implementation written in C.
IronPython, written in .NET.
Jython, written in Java.
PyPy, the optimizing implementation written in Python (actually, it's 
written in a subset of Python, RPython).

But the Python ecosystem is a lot bigger than just those four. Here are 
just a few other implementations that you might be interested in:


Stackless - the "forgetten Python", Stackless is, I believe, the oldest 
implementation behind only CPython itself. It's a fork of CPython with 
the calling stack removed and fast and lightweight microthreads, and is 
used extensively in EVE Online.

http://www.stackless.com/


Nuitka - optimising Python compiler written in C++, supports Python 2.6 
and 2.7, claims to be up to twice as fast as CPython.

http://nuitka.net/pages/overview.html


WPython - another optimizing version of Python with wordcodes instead of 
bytecodes.

http://code.google.com/p/wpython/


CLPython, an implementation of Python written in Common Lisp.

http://common-lisp.net/project/clpython/


CapPython is an experimental restricted version of Python with 
capabilities.

http://plash.beasts.org/wiki/CapPython
http://en.wikipedia.org/wiki/Object-capability_model


Berp - a compiler which works by translating Python to Haskell and 
compiling that.

https://github.com/bjpop/berp/wiki



Give them some love!



-- 
Steven

[toc] | [next] | [standalone]

#26473

On Sat, Aug 4, 2012 at 4:15 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> CLPython, an implementation of Python written in Common Lisp.
>
> Berp - a compiler which works by translating Python to Haskell and
> compiling that.

Okay. WHY? CLPython gives some reason, but how often do you need to
bridge that particular pair of languages? And why compile Python via
Haskell, when C is available as a "high level assembly language"?

The mind boggles...

ChrisA

[toc] | [prev] | [next] | [standalone]

#26480

On Sat, 04 Aug 2012 16:34:17 +1000, Chris Angelico wrote:

> On Sat, Aug 4, 2012 at 4:15 PM, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> CLPython, an implementation of Python written in Common Lisp.
>>
>> Berp - a compiler which works by translating Python to Haskell and
>> compiling that.
> 
> Okay. WHY? CLPython gives some reason, but how often do you need to
> bridge that particular pair of languages? And why compile Python via
> Haskell, when C is available as a "high level assembly language"?

For much the same reason that PyPy uses RPython when C is available. 
Because Haskell is available as a high level non-assembly language.

Berp is based on the Glasgow Haskell Compiler, which is a modern, 
efficient, optimizing compiler capable of producing excellent quality 
machine code on Windows, Mac, Linux and many Unixes. It gives you all the 
advantages of a high-level language with high-level data structures, type 
inference, and a compiler capable of generating optimized, fast, machine 
code.

Who would want to deal with C's idiosyncrasies, low-powered explicit type 
system, difficult syntax, and core-dumps, when you could use something 
better? Apart from C programmers, of course.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#26482

Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> Who would want to deal with C's idiosyncrasies, low-powered explicit type 
> system, difficult syntax, and core-dumps, when you could use something 
> better?

In the free software world, apparently many people like C. C is also
quite popular in the zero-fault software world: Several verification
tools do exist and Leroy et al. are writing a certified compiler for
C to plug the hole between the verified source code and the generated
assembly.


Stefan Krah

[toc] | [prev] | [next] | [standalone]

#26491

Stefan Krah <stefan-usenet@bytereef.org> writes:
> In the free software world, apparently many people like C. C is also
> quite popular in the zero-fault software world: Several verification
> tools do exist and Leroy et al. are writing a certified compiler for
> C to plug the hole between the verified source code and the generated
> assembly.

C is pretty poor as a compiler target: how would you translate Python
generators into C, for example?  How would you handle garbage
collection?

C isn't so great for high-assurance stuff either, compared to (say) Ada.
People do use it in critical apps, but that's just because it is (or
anyway used to be) so ubiquitous.  I'm wondering what you mean about
verification tools, other than analyzers like Coverity that mainly check
for bugs that in a safer language would be caught by the compiler.
Compcert is not all that C-specific it has been adapted to compile a
Haskell-derived language (Habit).

Haskell doesn't sound all that great as a translation target for Python
either, unfortunately, because its execution semantics are so different.
GHC is a very powerful compiler but it was made to compile Haskell code
that people actually write, and may do less good of a job with compiler
output from an imperative language like Python.  Compiling Python to
Scheme and then using a Scheme compiler might be a more natural fit.
But, compiling to Haskell was probably pretty convenient for that
particular project.

Finally, Python itself isn't all that well suited for compilation, given
its high dynamicity.  It will be interesting to see if the language
evolves due to PyPy.

[toc] | [prev] | [next] | [standalone]

#26492

Paul Rubin, 04.08.2012 17:59:
> Stefan Krah writes:
>> In the free software world, apparently many people like C. C is also
>> quite popular in the zero-fault software world: Several verification
>> tools do exist and Leroy et al. are writing a certified compiler for
>> C to plug the hole between the verified source code and the generated
>> assembly.
> 
> C is pretty poor as a compiler target: how would you translate Python
> generators into C, for example?

Depends. If you have CPython available, that'd be a straight forward
extension type. Otherwise, I guess you'd either have a class for them in
C++ or a struct in C. Not exactly complex.

For the yielding, you can use labels and goto. Given that you generate the
code, that's pretty straight forward as well.


> How would you handle garbage collection?

CPython does it automatically for us at least. Lacking that, you'd use one
of the available garbage collection implementations, or provide none at all.


> Haskell doesn't sound all that great as a translation target for Python
> either, unfortunately, because its execution semantics are so different.
> GHC is a very powerful compiler but it was made to compile Haskell code
> that people actually write, and may do less good of a job with compiler
> output from an imperative language like Python.  Compiling Python to
> Scheme and then using a Scheme compiler might be a more natural fit.
> But, compiling to Haskell was probably pretty convenient for that
> particular project.

You'd have some kind of emulation layer that does most of the translation
at runtime. That's why I said that you shouldn't expect too much of a
performance gain from what the platform gives you for the underlying
implementation. It can optimise the emulator, but it won't see enough of
the Python code to make anything efficient out of it. Jython is an example
for that.


> Finally, Python itself isn't all that well suited for compilation, given
> its high dynamicity.

You can get pretty far with static code analysis, optimistic optimisations
and code specialisation. We've decided against whole program analysis in
Cython not only for compiler complexity reasons but also because it would
let the normal compilation time explode for gains that you can much easier
get by manual optimisation. Obviously, optimising JIT compilers can do much
more here (because they actually have to do less), although they won't
always be able to figure out the right thing to do either. That's where
manual optimisation wins again.

Stefan

[toc] | [prev] | [next] | [standalone]

#26494

Stefan Behnel <stefan_ml@behnel.de> writes:
>> C is pretty poor as a compiler target: how would you translate Python
>> generators into C, for example?
> Depends. If you have CPython available, that'd be a straight forward
> extension type.

Calling CPython hardly counts as compiling Python into C.

> For the yielding, you can use labels and goto. Given that you generate
> the code, that's pretty straight forward as well.

You're going to compile the whole Python program into a single C
function so that you can do gotos inside of it?  What happens if the
program imports a generator?

>> How would you handle garbage collection?
> CPython does it automatically for us at least. 

You mean you're going to have all the same INCREF/DECREF stuff on every
operation in compiled data?  Ugh.

> Lacking that, you'd use one of the available garbage collection
> implementations,

What implementations would those be?  There's the Boehm GC which is
useful for some purposes but not really suitable at large scale, from
what I can tell.  Is there something else?

> or provide none at all.

You're going to let the program just leak memory until it crashes??

> you shouldn't expect too much of a performance gain from what the
> platform gives you for the underlying implementation. It can optimise
> the emulator, but it won't see enough of the Python code to make
> anything efficient out of it. Jython is an example for that.

Compare that to the performance gain of LuaJIT and it starts to look
like something is wrong with that approach, or maybe some issue inherent
in Python itself.

> You can get pretty far with static code analysis, optimistic
> optimisations and code specialisation.

It seems very hard to do reasonable optimizations in the presence of
standard Python techniques like dynamically poking class instance
attributes.  I guess some optimizations are still possible, like storing
attributes named as literals in the program in fixed slots, saving some
dictionary lookups even though the slot contents would have to still be
mutable.

[toc] | [prev] | [next] | [standalone]

#26499

Paul Rubin, 04.08.2012 20:18:
> Stefan Behnel writes:
>>> C is pretty poor as a compiler target: how would you translate Python
>>> generators into C, for example?
>> Depends. If you have CPython available, that'd be a straight forward
>> extension type.
> 
> Calling CPython hardly counts as compiling Python into C.

CPython is written in C, though. So anything that CPython does can be done
in C. It's not like the CPython project used a completely unusual way of
writing C code.

Besides, I find your above statement questionable. You will always need
some kind of runtime infrastructure when you "compile Python into C", so
you can just as well use CPython for that instead of reimplementing it
completely from scratch. Both Cython and Nuitka do exactly that, and one of
the major advantages of that approach is that they can freely interact with
arbitrary code (Python or not) that was written for CPython, regardless of
its native dependencies. What good would it be to throw all of that away,
just for the sake of having "pure C code generation"?


>> For the yielding, you can use labels and goto. Given that you generate
>> the code, that's pretty straight forward as well.
> 
> You're going to compile the whole Python program into a single C
> function so that you can do gotos inside of it?  What happens if the
> program imports a generator?

No, you are going to compile only the generator function into a function
that uses gotos, maybe with an additional in-out struct parameter that
holds its state. Then, on entry, you read the label (or its ID) from the
previous state, reset local variables and jump to the label. On exit, you
store the state back end return. Cython does it that way. Totally straight
forward, as I said.


>>> How would you handle garbage collection?
>> CPython does it automatically for us at least.
> 
> You mean you're going to have all the same INCREF/DECREF stuff on every
> operation in compiled data?  Ugh.

If you don't like that, you can experiment with anything from a dedicated
GC to transactional memory.


>> Lacking that, you'd use one of the available garbage collection
>> implementations,
> 
> What implementations would those be?  There's the Boehm GC which is
> useful for some purposes but not really suitable at large scale, from
> what I can tell.  Is there something else?

No idea - I'll look it up when I need one. Last I heard, PyPy had a couple
of GCs to choose from, but I don't know how closely the are tied into its
infrastructure.


>> or provide none at all.
> 
> You're going to let the program just leak memory until it crashes??

Well, it's not like CPython leaks memory until it crashes, now does it? And
it's written in C. So there must be ways to handle this also in C.

Remember that CPython didn't even have a GC before something around 2.0,
IIRC. That worked quite ok in most cases and simply left the tricky cases
to the programmers. It really depends on what your requirements are. Small
embedded systems, time critical code and real-time systems are often much
better off without garbage collection. It's pure convenience, after all.


>> you shouldn't expect too much of a performance gain from what the
>> platform gives you for the underlying implementation. It can optimise
>> the emulator, but it won't see enough of the Python code to make
>> anything efficient out of it. Jython is an example for that.
> 
> Compare that to the performance gain of LuaJIT and it starts to look
> like something is wrong with that approach, or maybe some issue inherent
> in Python itself.

Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT
compiler specifically for Lua code. How is that similar to the Jython
runtime that runs *on top of* the JVM with its generic byte code based JIT
compiler?

Basically, LuaJIT's JIT compiler works at the same level as the one in
PyPy, which is why both can theoretically provide the same level of
performance gains.


>> You can get pretty far with static code analysis, optimistic
>> optimisations and code specialisation.
> 
> It seems very hard to do reasonable optimizations in the presence of
> standard Python techniques like dynamically poking class instance
> attributes.  I guess some optimizations are still possible, like storing
> attributes named as literals in the program in fixed slots, saving some
> dictionary lookups even though the slot contents would have to still be
> mutable.

Sure. Even when targeting the CPython runtime with the generated C code
(like Cython or Nuitka), you can still do a lot. And sure, static code
analysis will never be able to infer everything that a JIT compiler can see.

Stefan

[toc] | [prev] | [next] | [standalone]

#26506

Stefan Behnel <stefan_ml@behnel.de> writes:
>> Calling CPython hardly counts as compiling Python into C.
> CPython is written in C, though. So anything that CPython does can be
> done in C. It's not like the CPython project used a completely unusual
> way of writing C code.

CPython is a relatively simple interpreter, and executing code 
by invoking such an interpreter IMHO doesn't count as "compiling" it in
any meaningful way.  

> You will always need some kind of runtime infrastructure when you
> "compile Python into C", so you can just as well use CPython for that
> instead of reimplementing it completely from scratch. 

Maybe there's parts of Cpython you can re-use, but having the CPython
interpreter be the execution engine for "compiled" Python generators
again fails the seriousness test of what it means to compile code.  If
you mean something other than that, you might explain more clearly.

> Both Cython and Nuitka do exactly that, 

I didn't know about Nuitka; it looks interesting but (at least after a
few minutes looking) I don't have much sense of how it works.

> No, you are going to compile only the generator function into a function
> that uses gotos, maybe with an additional in-out struct parameter that
> holds its state.

Yeah, ok, I guess that can work, given python generators are limited
to returning through just one stack level.  You might want to avoid
copying locals by just putting everything into a struct, that has to
be retained across entries/exits.

> If you don't like that, you can experiment with anything from a dedicated
> GC to transactional memory.

OK, but then CPython is no longer managing the memory.

> Last I heard, PyPy had a couple of GCs to choose from,

PyPy doesn't compile to C, but I guess compiling to C doesn't preclude
precise GC, as long as the generated C code carefully tracks what C
objects can contain GC-able pointers, and follows some constraints about
when the GC can run.  Some other compilers do this so it's not as big a
deal as it sounded like at first.  OK.
>
>>> or provide none at all.
>> You're going to let the program just leak memory until it crashes??
> Well, it's not like CPython leaks memory until it crashes...

I was counting CPython's reference counting as a rudimentary form of GC,
though I guess that's terminology that not everyone agrees on.

> Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT
> compiler specifically for Lua code. How is that similar to the Jython
> runtime that runs *on top of* the JVM with its generic byte code based
> JIT compiler?

I thought LuaJIT compiles the existing Lua VM code, but I haven't
looked at it closely or used it.

>> It seems very hard to do reasonable optimizations in the presence of
>> standard Python techniques
>
> Sure. Even when targeting the CPython runtime with the generated C
> code (like Cython or Nuitka), you can still do a lot. And sure, static
> code analysis will never be able to infer everything that a JIT
> compiler can see.

I think even a JIT can't avoid a lot of pain and slowdown, without
complex whole-program analysis and requiring the application to follow
some special conventions, like never importing at "runtime".

[toc] | [prev] | [next] | [standalone]

#26510

Paul Rubin, 04.08.2012 22:43:
> Stefan Behnel writes:
>>> Calling CPython hardly counts as compiling Python into C.
>> CPython is written in C, though. So anything that CPython does can be
>> done in C. It's not like the CPython project used a completely unusual
>> way of writing C code.
> 
> CPython is a relatively simple interpreter, and executing code 
> by invoking such an interpreter IMHO doesn't count as "compiling" it in
> any meaningful way.

Oh, CPython is substantially more than an interpreter. The eval loop is
only *one* way to use the runtime environment. Remember that it has many
builtin types and functions as well as a huge standard library. Much of the
runtime environment is already written in C or can be compiled down to C.
If you compile Python code into C code that avoids the eval loop and only
uses the CPython runtime environment (which is what Cython does), I think
that qualifies as compiling Python code to C. It's definitely the most
practical and user friendly way to do it.


>> You will always need some kind of runtime infrastructure when you
>> "compile Python into C", so you can just as well use CPython for that
>> instead of reimplementing it completely from scratch. 
> 
> Maybe there's parts of Cpython you can re-use, but having the CPython
> interpreter be the execution engine for "compiled" Python generators
> again fails the seriousness test of what it means to compile code.  If
> you mean something other than that, you might explain more clearly.

See above.


>> Both Cython and Nuitka do exactly that, 
> 
> I didn't know about Nuitka; it looks interesting but (at least after a
> few minutes looking) I don't have much sense of how it works.

It's mostly like Cython but without the type system, i.e. without all the
stuff that makes it useful in real life. Just a bare
Python-to-C++-in-CPython compiler, without much of a way to make it do what
you want.


>> Last I heard, PyPy had a couple of GCs to choose from,
> 
> PyPy doesn't compile to C

RPython (usually) does, though, and my guess is that the memory management
part of the runtime is written in RPython.


> but I guess compiling to C doesn't preclude
> precise GC, as long as the generated C code carefully tracks what C
> objects can contain GC-able pointers, and follows some constraints about
> when the GC can run.  Some other compilers do this so it's not as big a
> deal as it sounded like at first.  OK.

Yep, C really becomes a lot nicer when you generate it.


>> Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT
>> compiler specifically for Lua code. How is that similar to the Jython
>> runtime that runs *on top of* the JVM with its generic byte code based
>> JIT compiler?
> 
> I thought LuaJIT compiles the existing Lua VM code, but I haven't
> looked at it closely or used it.

Ok. It obviously reuses code, but the VM part of it is really different
from standard Lua.

Stefan

[toc] | [prev] | [next] | [standalone]

#26501

On 04/08/2012 20:06, Stefan Behnel wrote:
> Paul Rubin, 04.08.2012 20:18:
>> Stefan Behnel writes:
>>>> C is pretty poor as a compiler target: how would you translate Python
>>>> generators into C, for example?
>>> Depends. If you have CPython available, that'd be a straight forward
>>> extension type.
>>
>> Calling CPython hardly counts as compiling Python into C.
>
> CPython is written in C, though. So anything that CPython does can be done
> in C. It's not like the CPython project used a completely unusual way of
> writing C code.
>
> Besides, I find your above statement questionable. You will always need
> some kind of runtime infrastructure when you "compile Python into C", so
> you can just as well use CPython for that instead of reimplementing it
> completely from scratch. Both Cython and Nuitka do exactly that, and one of
> the major advantages of that approach is that they can freely interact with
> arbitrary code (Python or not) that was written for CPython, regardless of
> its native dependencies. What good would it be to throw all of that away,
> just for the sake of having "pure C code generation"?
>
>
>>> For the yielding, you can use labels and goto. Given that you generate
>>> the code, that's pretty straight forward as well.
>>
>> You're going to compile the whole Python program into a single C
>> function so that you can do gotos inside of it?  What happens if the
>> program imports a generator?
>
> No, you are going to compile only the generator function into a function
> that uses gotos, maybe with an additional in-out struct parameter that
> holds its state. Then, on entry, you read the label (or its ID) from the
> previous state, reset local variables and jump to the label. On exit, you
> store the state back end return. Cython does it that way. Totally straight
> forward, as I said.
>
>
>>>> How would you handle garbage collection?
>>> CPython does it automatically for us at least.
>>
>> You mean you're going to have all the same INCREF/DECREF stuff on every
>> operation in compiled data?  Ugh.
>
> If you don't like that, you can experiment with anything from a dedicated
> GC to transactional memory.
>
>
>>> Lacking that, you'd use one of the available garbage collection
>>> implementations,
>>
>> What implementations would those be?  There's the Boehm GC which is
>> useful for some purposes but not really suitable at large scale, from
>> what I can tell.  Is there something else?
>
> No idea - I'll look it up when I need one. Last I heard, PyPy had a couple
> of GCs to choose from, but I don't know how closely the are tied into its
> infrastructure.
>
>
>>> or provide none at all.
>>
>> You're going to let the program just leak memory until it crashes??
>
> Well, it's not like CPython leaks memory until it crashes, now does it? And
> it's written in C. So there must be ways to handle this also in C.
>
> Remember that CPython didn't even have a GC before something around 2.0,
> IIRC. That worked quite ok in most cases and simply left the tricky cases
> to the programmers. It really depends on what your requirements are. Small
> embedded systems, time critical code and real-time systems are often much
> better off without garbage collection. It's pure convenience, after all.
>
[snip]
CPython relied entirely on reference counting, so memory could leak you 
if inadvertently created a cycle of memory references. That problem was
fixed when a mark-and-sweep mechanism was added (it's called
occasionally to collect any unreachable cycles).

[toc] | [prev] | [next] | [standalone]

#26516

On Sat, 04 Aug 2012 08:59:18 -0700, Paul Rubin wrote:

> C isn't so great for high-assurance stuff either, compared to (say) Ada.
> People do use it in critical apps, but that's just because it is (or
> anyway used to be) so ubiquitous.

And then they are shocked, SHOCKED I say!, when their app has enough 
buffer overflow security vulnerabilities to sink a battleship.

[half a wink]


> Haskell doesn't sound all that great as a translation target for Python
> either, unfortunately, because its execution semantics are so different.

I have no opinion on that either way, except to say that if some 
developer wants to experiment with Python-in-Haskell, good on him or her. 
Trying something new is how progress is made.


[...]
> Finally, Python itself isn't all that well suited for compilation, given
> its high dynamicity.  It will be interesting to see if the language
> evolves due to PyPy.

Python is a dynamic language, but most Python code is relatively static. 
Runtime optimizations that target the common case, but fall back to 
unoptimized code in the rare cases that the optimization doesn't apply, 
offer the opportunity of big speedups for most code at the cost of 
trivial slowdowns when you do something unusual.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#26518

Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:
> Runtime optimizations that target the common case, but fall back to 
> unoptimized code in the rare cases that the optimization doesn't apply, 
> offer the opportunity of big speedups for most code at the cost of 
> trivial slowdowns when you do something unusual.

The problem is you can't always tell if the unusual case is being
exercised without an expensive dynamic check, which in some cases must
be repeated in every iteration of a critical inner loop, even though it
turns out that the program never actually uses the unusual case.

[toc] | [prev] | [next] | [standalone]

#26520

On Sat, 04 Aug 2012 18:38:33 -0700, Paul Rubin wrote:

> Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:
>> Runtime optimizations that target the common case, but fall back to
>> unoptimized code in the rare cases that the optimization doesn't apply,
>> offer the opportunity of big speedups for most code at the cost of
>> trivial slowdowns when you do something unusual.
> 
> The problem is you can't always tell if the unusual case is being
> exercised without an expensive dynamic check, which in some cases must
> be repeated in every iteration of a critical inner loop, even though it
> turns out that the program never actually uses the unusual case.

I never said optimizing Python was easy :)

Obviously if the check is expensive enough, the optimization isn't going 
to be worth doing. But often the check is not so expensive, or is just a 
matter of tedious and careful book-keeping.

I don't wish to dispute that optimizing Python is hard, but it's not a 
Hard Problem like factorizing huge integers, or solving the Palestine/
Israeli conflict. It's hard like cleaning your house after a gang of 
drunken frat boys have partied all weekend.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#26685

On 8/4/2012 7:19 PM, Steven D'Aprano wrote:
> On Sat, 04 Aug 2012 18:38:33 -0700, Paul Rubin wrote:
> 
>> Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:
>>> Runtime optimizations that target the common case, but fall back to
>>> unoptimized code in the rare cases that the optimization doesn't apply,
>>> offer the opportunity of big speedups for most code at the cost of
>>> trivial slowdowns when you do something unusual.
>>
>> The problem is you can't always tell if the unusual case is being
>> exercised without an expensive dynamic check, which in some cases must
>> be repeated in every iteration of a critical inner loop, even though it
>> turns out that the program never actually uses the unusual case.

   There are other approaches. PyPy uses two interpreters and a JIT
compiler to handle the hard cases.  When code does something unexpected
to other code, the backup interpreter is used to get control out of
the trouble spot so that the JIT compiler can then recompile the
code.  (I think; I've read the paper but haven't looked at the
internals.)

   This is hard to implement and hard to get right.

				John Nagle

[toc] | [prev] | [next] | [standalone]

#26525

Paul Rubin, 05.08.2012 03:38:
> Steven D'Aprano writes:
>> Runtime optimizations that target the common case, but fall back to 
>> unoptimized code in the rare cases that the optimization doesn't apply, 
>> offer the opportunity of big speedups for most code at the cost of 
>> trivial slowdowns when you do something unusual.
> 
> The problem is you can't always tell if the unusual case is being
> exercised without an expensive dynamic check, which in some cases must
> be repeated in every iteration of a critical inner loop, even though it
> turns out that the program never actually uses the unusual case.

Cython does a lot of optimistic optimisations. That's where a large part of
that huge C file comes from that Cython generates from even simple Python code.

For example, in CPython, C function calls are so ridiculously faster than
Python function calls that it's worth some effort if it saves you from
packing an argument tuple to call into a Python function. In fact, we've
been thinking about ways to export C signatures from Python function
objects, so that code implemented in C (or a C compatible language) can be
called directly from other code implemented in C. That's very common in the
CPython ecosystem.

There are a lot of simple things that quickly add up into a much better
performance on average.

Stefan

[toc] | [prev] | [next] | [standalone]

#26522

On 05 Aug 2012 00:54:57 GMT, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> declaimed the following in
gmane.comp.python.general:

> On Sat, 04 Aug 2012 08:59:18 -0700, Paul Rubin wrote:
> 
> > C isn't so great for high-assurance stuff either, compared to (say) Ada.
> > People do use it in critical apps, but that's just because it is (or
> > anyway used to be) so ubiquitous.
> 
> And then they are shocked, SHOCKED I say!, when their app has enough 
> buffer overflow security vulnerabilities to sink a battleship.
> 
> [half a wink]
>
	{adding the other half}

	One has to realize that it is quite difficult to sink said
battleship -- a complete electrical system failure will still leave it
floating...

	In contrast, keeping an inherently unstable, fly by wire, fighter
jet in the air is much more difficult...
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#26483

Steven D'Aprano, 04.08.2012 12:54:
> Berp is based on the Glasgow Haskell Compiler, which is a modern, 
> efficient, optimizing compiler capable of producing excellent quality 
> machine code on Windows, Mac, Linux and many Unixes. It gives you all the 
> advantages of a high-level language with high-level data structures, type 
> inference, and a compiler capable of generating optimized, fast, machine 
> code.

Although all those optimisations don't mean that Python code would run fast
on top of it. Just because you translate Python to another language and
platform doesn't mean that there's any benefit from the underlying platform
optimisations. Both PyPy and Cython run Python code faster than CPython,
but not because they eventually translate it into machine code but because
they optimise and specialise it along the way, based on its high-level code
constructs. One big success of the Unladen Swallow project was to show that
bare JIT compilation is mostly worthless for high level languages.


> Who would want to deal with C's idiosyncrasies, low-powered explicit type 
> system, difficult syntax, and core-dumps, when you could use something 
> better?

The core developers of both CPython and Cython aim for exactly that. They
write C so you don't have to. But keep in mind that C is still *the* lingua
franca of software development. A major reason why Python is (slowly)
catching up these days is that the main implementation is written in C and
makes it easy to interface with C code.

Stefan

[toc] | [prev] | [next] | [standalone]

#26474

Steven D'Aprano, 04.08.2012 08:15:
> Most people are aware, if only vaguely, of the big Four Python 
> implementations:
> 
> CPython, or just Python, the reference implementation written in C.
> IronPython, written in .NET.
> Jython, written in Java.
> PyPy, the optimizing implementation written in Python (actually, it's 
> written in a subset of Python, RPython).
> 
> But the Python ecosystem is a lot bigger than just those four. Here are 
> just a few other implementations that you might be interested in:
> 
> 
> Stackless - the "forgetten Python", Stackless is, I believe, the oldest 
> implementation behind only CPython itself. It's a fork of CPython with 
> the calling stack removed and fast and lightweight microthreads, and is 
> used extensively in EVE Online.
> 
> http://www.stackless.com/
> 
> 
> Nuitka - optimising Python compiler written in C++, supports Python 2.6 
> and 2.7, claims to be up to twice as fast as CPython.
> 
> http://nuitka.net/pages/overview.html
> 
> 
> WPython - another optimizing version of Python with wordcodes instead of 
> bytecodes.
> 
> http://code.google.com/p/wpython/
> 
> 
> CLPython, an implementation of Python written in Common Lisp.
> 
> http://common-lisp.net/project/clpython/
> 
> 
> CapPython is an experimental restricted version of Python with 
> capabilities.
> 
> http://plash.beasts.org/wiki/CapPython
> http://en.wikipedia.org/wiki/Object-capability_model
> 
> 
> Berp - a compiler which works by translating Python to Haskell and 
> compiling that.
> 
> https://github.com/bjpop/berp/wiki

And not to forget Cython, which is the only static Python compiler that is
widely used. Compiles and optimises Python to C code that uses the CPython
runtime and allows for easy manual optimisations to get C-like performance
out of it.

http://cython.org/

Stefan

[toc] | [prev] | [next] | [standalone]

#26476

On Sat, 04 Aug 2012 08:40:16 +0200, Stefan Behnel wrote:

> And not to forget Cython, which is the only static Python compiler that
> is widely used. Compiles and optimises Python to C code that uses the
> CPython runtime and allows for easy manual optimisations to get C-like
> performance out of it.
> 
> http://cython.org/

Cython is great, but I question that it is a *Python* implementation. 
That's not a criticism of Cython, but it is different from Python. Take 
this example code from the tutorial:

from libc.math cimport sin

cdef double f(double x):
    return sin(x*x)

If that's Python code, then I'm Ethel the Aardvark.

Cython is very Python-like, but there is no doubt in my mind that it is a 
superset of Python and therefore a different language.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web