Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #90578 > unrolled thread

Building CPython

Back to article view | Back to comp.lang.python

Contents

  Building CPython BartC <bc@freeuk.com> - 2015-05-13 20:36 +0100
    Re: Building CPython Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-13 20:58 +0100
    Re: Building CPython Terry Reedy <tjreedy@udel.edu> - 2015-05-13 18:34 -0400
      Re: Building CPython BartC <bc@freeuk.com> - 2015-05-14 16:51 +0100
        Re: Building CPython Chris Angelico <rosuav@gmail.com> - 2015-05-15 02:09 +1000
          Re: Building CPython BartC <bc@freeuk.com> - 2015-05-14 18:02 +0100
            Re: Building CPython Dave Angel <davea@davea.name> - 2015-05-14 13:10 -0400
            Re: Building CPython Chris Angelico <rosuav@gmail.com> - 2015-05-15 03:11 +1000
              Re: Building CPython BartC <bc@freeuk.com> - 2015-05-14 18:32 +0100
                Re: Building CPython Chris Angelico <rosuav@gmail.com> - 2015-05-15 03:45 +1000
            Re: Building CPython Terry Reedy <tjreedy@udel.edu> - 2015-05-14 14:50 -0400
              Re: Building CPython Christian Gollwitzer <auriocus@gmx.de> - 2015-05-15 12:51 +0200
                Re: Building CPython Terry Reedy <tjreedy@udel.edu> - 2015-05-15 17:19 -0400
        Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-14 19:29 +0300
          Re: Building CPython BartC <bc@freeuk.com> - 2015-05-14 22:55 +0100
            Re: Building CPython MRAB <python@mrabarnett.plus.com> - 2015-05-14 23:19 +0100
            Re: Building CPython BartC <bc@freeuk.com> - 2015-05-15 01:50 +0100
              Re: Building CPython Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-05-15 18:05 +1200
                Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-15 11:59 +0300
                  Re: Building CPython wxjmfauth@gmail.com - 2015-05-15 02:07 -0700
                    Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-15 12:20 +0300
                      Re: Building CPython wxjmfauth@gmail.com - 2015-05-15 02:51 -0700
                        Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-15 13:52 +0300
                          Re: Building CPython Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-15 22:10 +1000
                            Re: Building CPython Chris Angelico <rosuav@gmail.com> - 2015-05-15 22:34 +1000
                            Re: Building CPython wxjmfauth@gmail.com - 2015-05-15 07:11 -0700
                          Re: Building CPython Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-15 13:41 +0100
                      Re: Building CPython Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-15 13:38 +0100
                  Re: Building CPython Chris Angelico <rosuav@gmail.com> - 2015-05-15 19:43 +1000
                    Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-15 13:50 +0300
                      Re: Building CPython Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-15 22:43 +1000
                        Re: Building CPython Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-15 09:00 -0600
                        Re: Building CPython Ian Kelly <ian.g.kelly@gmail.com> - 2015-05-15 09:04 -0600
                        Re: Building CPython Chris Angelico <rosuav@gmail.com> - 2015-05-16 01:06 +1000
                  Re: Building CPython Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-15 20:14 +1000
                    Re: Building CPython Chris Angelico <rosuav@gmail.com> - 2015-05-15 20:25 +1000
                  Re: Building CPython Terry Reedy <tjreedy@udel.edu> - 2015-05-15 17:05 -0400
                  Re: Building CPython BartC <bc@freeuk.com> - 2015-05-15 22:54 +0100
                    Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-16 01:44 +0300
                      Re: Building CPython Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-16 00:27 +0100
                        Re: Building CPython Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-16 11:55 +1000
                          Re: Building CPython Chris Angelico <rosuav@gmail.com> - 2015-05-16 12:15 +1000
                          Re: Building CPython Mark Lawrence <breamoreboy@yahoo.co.uk> - 2015-05-16 03:17 +0100
                      Re: Building CPython BartC <bc@freeuk.com> - 2015-05-16 01:43 +0100
                        Re: Building CPython MRAB <python@mrabarnett.plus.com> - 2015-05-16 02:16 +0100
                        Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-16 11:08 +0300
                          Re: Building CPython Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-16 19:40 +1000
                            Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-16 16:59 +0300
                              Re: Building CPython Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2015-05-17 04:18 +1000
                                Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-16 21:55 +0300
                                  Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-17 01:51 +0300
                                    Re: Building CPython Marko Rauhamaa <marko@pacujo.net> - 2015-05-17 23:49 +0300
                    Re: Building CPython Terry Reedy <tjreedy@udel.edu> - 2015-05-15 19:54 -0400
                Re: Building CPython BartC <bc@freeuk.com> - 2015-05-15 10:32 +0100
                  Re: Building CPython Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2015-05-16 12:55 +1200
                    Re: Building CPython Jonas Wielicki <jonas@wielicki.name> - 2015-05-17 14:25 +0200
                      Re: Building CPython BartC <bc@freeuk.com> - 2015-05-17 14:41 +0100

Page 1 of 3  [1] 2 3  Next page →

#90578 — Building CPython

I'm interested in playing with the CPython sources. I need to be able to 
build under Windows, but don't want to use make files (which rarely work 
properly), nor do a 6GB installation of Visual Studio Express which is 
what seems to be needed (I'm hopeless with complicated IDEs anyway).

Is it possible to do this by using mingw-gcc to compile the .c files of 
the Python sources one by one, or is it one of those complicated 
projects where some of the source is generated as it goes along?

I thought I'd start with the source file containing Py_Main and continue 
from there, but some modules compile and some don't, obscure errors that 
I don't want to investigate unless it's going to be worthwhile (ie. 
eventually ending up with a python.exe that can run simple .py programs).

-- 
Bartc

[toc] | [next] | [standalone]

#90582

On 13/05/2015 20:36, BartC wrote:
> I'm interested in playing with the CPython sources. I need to be able to
> build under Windows, but don't want to use make files (which rarely work
> properly), nor do a 6GB installation of Visual Studio Express which is
> what seems to be needed (I'm hopeless with complicated IDEs anyway).
>
> Is it possible to do this by using mingw-gcc to compile the .c files of
> the Python sources one by one, or is it one of those complicated
> projects where some of the source is generated as it goes along?
>
> I thought I'd start with the source file containing Py_Main and continue
> from there, but some modules compile and some don't, obscure errors that
> I don't want to investigate unless it's going to be worthwhile (ie.
> eventually ending up with a python.exe that can run simple .py programs).
>

Before you spend too much time on the mingw route, check out the 
outstanding issues for it on the bug tracker.  Then you'll maybe realise 
that using the supported VS setup is far easier.  Everything you need is 
in the PCBuild directory, including pcbuild.sln for use with VS and 
build.bat for command line use.  Details here 
https://docs.python.org/devguide/setup.html#windows

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#90587

On 5/13/2015 3:36 PM, BartC wrote:
> I'm interested in playing with the CPython sources. I need to be able to
> build under Windows, but don't want to use make files (which rarely work
> properly), nor do a 6GB installation of Visual Studio Express which is
> what seems to be needed (I'm hopeless with complicated IDEs anyway).

Once installed hg or tortoisehg (I use this) and VSE are installed and 
repository cloned, are half done. At command prompt, with top directory 
of repository as current directory enter
tools\scripts\external.bat
Double-clicking file in Explorer does not work.
Usually only needs to be done once per branch after x.y.0 release as 
dependencies are usually not updated for bugfix releases.

Then in same directory enter
pcbuild\python.sln
or double click in Explorer or open VSE and open this file.
Hit F7, wait until get line like
========== Build: 1 succeeded, 0 failed, 24 up-to-date, 1 skipped,
hit F5, pin python_d to taskbar (optional, but handy), and go.

And read devguide.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#90608

On 13/05/2015 23:34, Terry Reedy wrote:
> On 5/13/2015 3:36 PM, BartC wrote:
>> I'm interested in playing with the CPython sources. I need to be able to
>> build under Windows, but don't want to use make files (which rarely work
>> properly), nor do a 6GB installation of Visual Studio Express which is
>> what seems to be needed (I'm hopeless with complicated IDEs anyway).
>
> Once installed hg or tortoisehg (I use this) and VSE are installed and
> repository cloned, are half done. At command prompt, with top directory
> of repository as current directory enter
> tools\scripts\external.bat
> Double-clicking file in Explorer does not work.
> Usually only needs to be done once per branch after x.y.0 release as
> dependencies are usually not updated for bugfix releases.
>
> Then in same directory enter
> pcbuild\python.sln
> or double click in Explorer or open VSE and open this file.
> Hit F7, wait until get line like
> ========== Build: 1 succeeded, 0 failed, 24 up-to-date, 1 skipped,
> hit F5, pin python_d to taskbar (optional, but handy), and go.

OK, the answer seems to be No then - you can't just trivially compile 
the C modules that comprise the sources with the nearest compiler to 
hand. So much for C's famous portability!

(Actually, I think you already lost me on your first line.)

That's a shame because I wanted to tinker with the main dispatcher loop 
to try and find out what exactly is making it slow. Nothing that seems 
obvious at first sight. (The comments even talk about improving branch 
prediction on certain architectures, even though the performance is a 
couple of magnitudes away from that kind of optimisation being relevant.)

Perhaps I was hoping there were some options turned on by default which, 
if disabled, would suddenly double the speed of simple benchmarks. Now I 
won't be able to find out...

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#90610

On Fri, May 15, 2015 at 1:51 AM, BartC <bc@freeuk.com> wrote:
> OK, the answer seems to be No then - you can't just trivially compile the C
> modules that comprise the sources with the nearest compiler to hand. So much
> for C's famous portability!
>
> (Actually, I think you already lost me on your first line.)
>
> That's a shame because I wanted to tinker with the main dispatcher loop to
> try and find out what exactly is making it slow. Nothing that seems obvious
> at first sight. (The comments even talk about improving branch prediction on
> certain architectures, even though the performance is a couple of magnitudes
> away from that kind of optimisation being relevant.)

C's portability isn't really sufficient for building a huge project,
so what you generally end up with is a huge slab of common code that
doesn't change from platform to platform, plus a (relatively) tiny
section of platform-specific code, such as makefiles/project files,
linker definitions, and so on. When you start hacking on CPython, you
don't generally have to consider which platform you're aiming at, as
long as you're building on one of the ones that's well supported;
trying to port Python to a new compiler on a known OS is actually
about as much work as porting it to a known compiler on a new OS. (I
know this, because I've attempted both - using mingw on Windows, and
gcc on OS/2. It's a big job either way.)

If you want to just quickly play around with CPython's sources, I
would strongly recommend getting yourself a Linux box. Either spin up
some actual hardware with actual Linux, or grab a virtualization
engine like VMWare, VirtualBox, etc, etc, and installing into a VM.
With a Debian-based Linux (Debian, Ubuntu, Mint, etc), you should
simply be able to:

sudo apt-get build-dep python3

to get all the build dependencies for Python 3; that, plus the source
code, should be enough to get you a'building. Similar simplicities are
available for other Linux distros, but I'll let someone else recommend
them.

Even when you have all the appropriate build tools, Windows can be at
times a pain for building software on. The general philosophy of
Windows is that you should normally be taking ready-made binaries; the
general philosophy of Linux is that it's perfectly normal to spin up
your own binaries from the distributed source code. It's not
impossible to build on Windows, by any means, but be prepared for
extra hassles and less support from the OS than you might like.

ChrisA

[toc] | [prev] | [next] | [standalone]

#90614

On 14/05/2015 17:09, Chris Angelico wrote:
> On Fri, May 15, 2015 at 1:51 AM, BartC <bc@freeuk.com> wrote:
>> OK, the answer seems to be No then - you can't just trivially compile the C
>> modules that comprise the sources with the nearest compiler to hand. So much
>> for C's famous portability!
>>
>> (Actually, I think you already lost me on your first line.)

> If you want to just quickly play around with CPython's sources, I
> would strongly recommend getting yourself a Linux box. Either spin up
> some actual hardware with actual Linux, or grab a virtualization
> engine like VMWare, VirtualBox, etc, etc, and installing into a VM.
> With a Debian-based Linux (Debian, Ubuntu, Mint, etc), you should
> simply be able to:
>
> sudo apt-get build-dep python3

Actually I had VirtualBox with Ubuntu, but I don't know my way around 
Linux and preferred doing things under Windows (and with all my own tools).

But it's now building under Ubuntu.

(Well, I'm not sure what it's doing exactly; the instructions said type 
make, then make test, then make install, and it's still doing make test.

I hope there's a quicker way of re-building an executable after a minor 
source file change, otherwise doing any sort of development is going to 
be impractical.)

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#90616

On 05/14/2015 01:02 PM, BartC wrote:
> On 14/05/2015 17:09, Chris Angelico wrote:
>> On Fri, May 15, 2015 at 1:51 AM, BartC <bc@freeuk.com> wrote:
>>> OK, the answer seems to be No then - you can't just trivially compile
>>> the C
>>> modules that comprise the sources with the nearest compiler to hand.
>>> So much
>>> for C's famous portability!
>>>
>>> (Actually, I think you already lost me on your first line.)
>
>> If you want to just quickly play around with CPython's sources, I
>> would strongly recommend getting yourself a Linux box. Either spin up
>> some actual hardware with actual Linux, or grab a virtualization
>> engine like VMWare, VirtualBox, etc, etc, and installing into a VM.
>> With a Debian-based Linux (Debian, Ubuntu, Mint, etc), you should
>> simply be able to:
>>
>> sudo apt-get build-dep python3
>
> Actually I had VirtualBox with Ubuntu, but I don't know my way around
> Linux and preferred doing things under Windows (and with all my own tools).
>
> But it's now building under Ubuntu.
>
> (Well, I'm not sure what it's doing exactly; the instructions said type
> make, then make test, then make install, and it's still doing make test.
>
> I hope there's a quicker way of re-building an executable after a minor
> source file change, otherwise doing any sort of development is going to
> be impractical.)
>

That's what make is good for.  It compares the datestamps of the source 
files against the obj files (etc.) and recompiles only when the source 
is newer.  (It's more complex, but that's the idea)

-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#90617

On Fri, May 15, 2015 at 3:02 AM, BartC <bc@freeuk.com> wrote:
> Actually I had VirtualBox with Ubuntu, but I don't know my way around Linux
> and preferred doing things under Windows (and with all my own tools).
>
> But it's now building under Ubuntu.
>
> (Well, I'm not sure what it's doing exactly; the instructions said type
> make, then make test, then make install, and it's still doing make test.
>
> I hope there's a quicker way of re-building an executable after a minor
> source file change, otherwise doing any sort of development is going to be
> impractical.)

The whole point of 'make' is to rebuild only the parts that need to be
rebuilt (either they've changed, or they depend on something that was
changed). Sometimes practically everything needs to be rebuilt, if you
do some really fundamental change, but generally not.

The three parts to the build process are:

1) make - actually generate an executable. Takes ages the first time,
will be a lot quicker if you haven't changed much.
2) make test - run the entire test suite. Takes just as long every
time, but most of it won't have changed.
3) make install (needs root access, so probably 'sudo make install') -
install this as your primary build of Python.

When you start tinkering, I suggest just running make; rerunning the
test suite isn't necessary till you're all done, and even then it's
only important for making sure that your change hasn't broken anything
anywhere else. Test your actual changes by simply running the
freshly-built Python - most likely that'll be "./python". Working that
way is fairly quick - you can tweak some C code, see the results, and
go back and tweak some more, all without pausing for a sword fight.
But once you go and rerun the full test suite, well... that's when
it's time for some:

https://xkcd.com/303/

ChrisA

[toc] | [prev] | [next] | [standalone]

#90619

On 14/05/2015 18:11, Chris Angelico wrote:
> On Fri, May 15, 2015 at 3:02 AM, BartC <bc@freeuk.com> wrote:

>> I hope there's a quicker way of re-building an executable after a minor
>> source file change, otherwise doing any sort of development is going to be
>> impractical.)
>
> The whole point of 'make' is to rebuild only the parts that need to be
> rebuilt (either they've changed, or they depend on something that was
> changed). Sometimes practically everything needs to be rebuilt, if you
> do some really fundamental change, but generally not.
>
> The three parts to the build process are:
>
> 1) make - actually generate an executable. Takes ages the first time,
> will be a lot quicker if you haven't changed much.
> 2) make test - run the entire test suite. Takes just as long every
> time, but most of it won't have changed.
> 3) make install (needs root access, so probably 'sudo make install') -
> install this as your primary build of Python.
>
> When you start tinkering, I suggest just running make; rerunning the
> test suite isn't necessary till you're all done, and even then it's
> only important for making sure that your change hasn't broken anything
> anywhere else. Test your actual changes by simply running the
> freshly-built Python - most likely that'll be "./python".

OK, thanks. I didn't even know where the executable was put! Now I don't 
need 'make install', while 'make test' I won't bother with any more.

Making a small change and typing 'make' took 5 seconds, which is 
reasonable enough (although I had to use the copy of the source in 
Windows to find where the main.c file I needed was located).

Now Python 3.4.3 says "Bart's Python".


-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#90621

On Fri, May 15, 2015 at 3:32 AM, BartC <bc@freeuk.com> wrote:
> OK, thanks. I didn't even know where the executable was put! Now I don't
> need 'make install', while 'make test' I won't bother with any more.
>
> Making a small change and typing 'make' took 5 seconds, which is reasonable
> enough (although I had to use the copy of the source in Windows to find
> where the main.c file I needed was located).
>
> Now Python 3.4.3 says "Bart's Python".

Haha. I don't usually bother rebranding my builds of things; though
that's partly because normally I'm making very few changes, and all in
the hope that they'll be accepted upstream anyway.

Incidentally, a quick 'make' can range anywhere from a fraction of a
second to quite a long time, depending mainly on the speed of your
hard drive and the performance of your disk cache. On my Linux, a
"null make" (ie when literally nothing has changed - just rerunning
make) takes about half a second, and that's dealing with a module
that's failing to build. When you rebuild lots of times all at once,
you'll pretty much be working in RAM the whole time, assuming you have
enough of it available.

ChrisA

[toc] | [prev] | [next] | [standalone]

#90627

On 5/14/2015 1:11 PM, Chris Angelico wrote:

> 2) make test - run the entire test suite. Takes just as long every
> time, but most of it won't have changed.

The test runner has an option, -jn, to run tests in n processes instead 
of just 1.  On my 6 core pentium, -j5 cuts time to almost exactly 1/5th 
of otherwise.  -j10 seems faster but have not times it.  I suspect that 
'make test' does not use -j option.


-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#90665

Am 14.05.15 um 20:50 schrieb Terry Reedy:
> On 5/14/2015 1:11 PM, Chris Angelico wrote:
>
>> 2) make test - run the entire test suite. Takes just as long every
>> time, but most of it won't have changed.
>
> The test runner has an option, -jn, to run tests in n processes instead
> of just 1.  On my 6 core pentium, -j5 cuts time to almost exactly 1/5th
> of otherwise.  -j10 seems faster but have not times it.  I suspect that
> 'make test' does not use -j option.
>

Just to clarify, -j is an option of GNU make to run the Makefile in 
parallel. Unless the Makefile is buggy, this should result in the same 
output. You can also set an environment variable to enable this 
permanently (until you log out) like

export MAKEFLAGS=-j5

Put this into your .bashrc or .profile, and it'll become permanent.

	Christian

[toc] | [prev] | [next] | [standalone]

#90693

On 5/15/2015 6:51 AM, Christian Gollwitzer wrote:
> Am 14.05.15 um 20:50 schrieb Terry Reedy:
>> On 5/14/2015 1:11 PM, Chris Angelico wrote:
>>
>>> 2) make test - run the entire test suite. Takes just as long every
>>> time, but most of it won't have changed.
>>
>> The test runner has an option, -jn, to run tests in n processes instead
>> of just 1.  On my 6 core pentium, -j5 cuts time to almost exactly 1/5th
>> of otherwise.  -j10 seems faster but have not times it.  I suspect that
>> 'make test' does not use -j option.
>>
>
> Just to clarify, -j is an option of GNU make to run the Makefile in
> parallel. Unless the Makefile is buggy, this should result in the same
> output. You can also set an environment variable to enable this
> permanently (until you log out) like
>
> export MAKEFLAGS=-j5
>
> Put this into your .bashrc or .profile, and it'll become permanent.

This is not applicable when running on Windows.  I was specifically 
referring, for instance, to entering at command prompt

current_dir> python -m test -j10

AFAIK, test.__main__ does not look at environmental variables.

ps. I have been wondering why 'j' rather than, say 'c' or 'p' was used 
for 'run in parallel on multiple cores'.  Your comment suggests that 'j' 
follows precedent.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#90611

BartC <bc@freeuk.com>:

> That's a shame because I wanted to tinker with the main dispatcher
> loop to try and find out what exactly is making it slow. Nothing that
> seems obvious at first sight.

My guess is the main culprit is attribute lookup in two ways:

 * Each object attribute reference involves a dictionary lookup.

 * Each method call involves *two* dictionary lookups plus an object
   creation.

Tell us what you find out.


Marko

[toc] | [prev] | [next] | [standalone]

#90632

On 14/05/2015 17:29, Marko Rauhamaa wrote:
> BartC <bc@freeuk.com>:
>
>> That's a shame because I wanted to tinker with the main dispatcher
>> loop to try and find out what exactly is making it slow. Nothing that
>> seems obvious at first sight.
>
> My guess is the main culprit is attribute lookup in two ways:
>
>   * Each object attribute reference involves a dictionary lookup.
>
>   * Each method call involves *two* dictionary lookups plus an object
>     creation.
>
> Tell us what you find out.

I'm just starting but I can tell you that it isn't because debug mode 
(Py_DEBUG defined) was left on by mistake!

What is interesting however is that on the very simple test I'm doing (a 
while loop incrementing a variable up to 100 million), the timings under 
Windows are:

Python 2.5       9.2 seconds
Python 3.1      13.1
Python 3.4.3    17.0
Python 3.4.3    14.3 (under Ubuntu on same machine, using the version
                      I built today)

That's quite a big range!

PyPy does it in 0.7 seconds. The same program within my own *simple* 
bytecode interpreter (not for Python) has a fastest time of 1.5 seconds 
but makes use of ASM. A version 100% in (gcc) C can manage 2 seconds.

So far I haven't find anything that explains the discrepancy (the 
languages are different, mine is simpler, but the Python code isn't 
doing anything that complicated, only LOAD_FASTs and such, and LOAD_FAST 
is apparently just an array access.

But the nearly 2:1 difference between new and old Python versions is 
also intriguing.

def whiletest():
	i=0
	while i<=100000000:
		i=i+1

whiletest()

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#90633

On 2015-05-14 22:55, BartC wrote:
> On 14/05/2015 17:29, Marko Rauhamaa wrote:
>> BartC <bc@freeuk.com>:
>>
>>> That's a shame because I wanted to tinker with the main dispatcher
>>> loop to try and find out what exactly is making it slow. Nothing that
>>> seems obvious at first sight.
>>
>> My guess is the main culprit is attribute lookup in two ways:
>>
>>   * Each object attribute reference involves a dictionary lookup.
>>
>>   * Each method call involves *two* dictionary lookups plus an object
>>     creation.
>>
>> Tell us what you find out.
>
> I'm just starting but I can tell you that it isn't because debug mode
> (Py_DEBUG defined) was left on by mistake!
>
> What is interesting however is that on the very simple test I'm doing (a
> while loop incrementing a variable up to 100 million), the timings under
> Windows are:
>
> Python 2.5       9.2 seconds
> Python 3.1      13.1
> Python 3.4.3    17.0
> Python 3.4.3    14.3 (under Ubuntu on same machine, using the version
>                        I built today)
>
> That's quite a big range!
>
> PyPy does it in 0.7 seconds. The same program within my own *simple*
> bytecode interpreter (not for Python) has a fastest time of 1.5 seconds
> but makes use of ASM. A version 100% in (gcc) C can manage 2 seconds.
>
> So far I haven't find anything that explains the discrepancy (the
> languages are different, mine is simpler, but the Python code isn't
> doing anything that complicated, only LOAD_FASTs and such, and LOAD_FAST
> is apparently just an array access.
>
> But the nearly 2:1 difference between new and old Python versions is
> also intriguing.
>
> def whiletest():
> 	i=0
> 	while i<=100000000:
> 		i=i+1
>
> whiletest()
>
Python 2.x has int and long; Python 3 has int, which is the old 'long'.
Try Python 2 with longs.

[toc] | [prev] | [next] | [standalone]

#90634

On 14/05/2015 22:55, BartC wrote:

 > def whiletest():
 >      i=0
 >      while i<=100000000:
 >          i=i+1
 >
 > whiletest()
 >

> Python 2.5       9.2 seconds
> Python 3.1      13.1
> Python 3.4.3    17.0
> Python 3.4.3    14.3 (under Ubuntu on same machine, using the version
>                       I built today)
>
> That's quite a big range!
>
> PyPy does it in 0.7 seconds. The same program within my own *simple*
> bytecode interpreter (not for Python) has a fastest time of 1.5 seconds
> but makes use of ASM. A version 100% in (gcc) C can manage 2 seconds.

(Actually my ASM-aided code took 0.5 seconds (some crucial section had 
been commented out). Faster than PyPy and just using simple brute-force 
methods.)

> So far I haven't find anything that explains the discrepancy (the
> languages are different, mine is simpler, but the Python code isn't
> doing anything that complicated, only LOAD_FASTs and such, and LOAD_FAST
> is apparently just an array access.

It turns out the LOAD_FASTs were the simple byte-codes!

I'm just starting to find out just how much of a big complicated mess 
this project really is. I wouldn't be surprised if there aren't many 
people who actually understand it all, and that would explain why no-one 
seems to have had much luck in getting the speed up (if anything, it's 
getting slower).

I still have no idea yet exactly what an object comprises. I know that 
even an innocuous-looking LOAD_CONST actually loads a tuple from a table 
(complete with flag testing and bounds checks, although taking those out 
made no discernible difference).

It appears to be those "<=" and "+" operations in the code above where 
much of the time is spent. When I trace out the execution paths a bit 
more, I'll have a better picture of how many lines of C code are 
involved in each iteration.

-- 
Bartc

[toc] | [prev] | [next] | [standalone]

#90649

BartC wrote:
> It appears to be those "<=" and "+" operations in the code above where 
> much of the time is spent. When I trace out the execution paths a bit 
> more, I'll have a better picture of how many lines of C code are 
> involved in each iteration.

The path from decoding a bytecode to the C code that
implements it can be rather convoluted, but there are
reasons for each of the complications -- mainly to
do with supporting the ability to override operators
with C and/or Python code.

If you removed those abilities, the implemention
would be simpler, and possibly faster. But then the
language wouldn't be Python any more.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#90656

Gregory Ewing <greg.ewing@canterbury.ac.nz>:

> BartC wrote:
>> It appears to be those "<=" and "+" operations in the code above
>> where much of the time is spent. When I trace out the execution paths
>> a bit more, I'll have a better picture of how many lines of C code
>> are involved in each iteration.
>
> The path from decoding a bytecode to the C code that implements it can
> be rather convoluted, but there are reasons for each of the
> complications -- mainly to do with supporting the ability to override
> operators with C and/or Python code.
>
> If you removed those abilities, the implemention would be simpler, and
> possibly faster. But then the language wouldn't be Python any more.

I agree that Python's raison-d'être is its dynamism and expressive
power. It definitely shouldn't be sacrificed for performance.

However, in some respects, Python might be going overboard with its
dynamism; are all those dunder methods really needed? Must "false" be
defined so broadly? Must a method lookup necessarily involve object
creation?


Marko

[toc] | [prev] | [next] | [standalone]

#90657

Le vendredi 15 mai 2015 10:59:41 UTC+2, Marko Rauhamaa a écrit :
> 
> I agree that Python's raison-d'être is its dynamism and expressive
> power. It definitely shouldn't be sacrificed for performance.
> 

Step #1:

Implement unicode correctly.

jmf

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.python

csiph-web