Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #40949

Re: Apparent magic number problem

Show all headers | View raw

On 09/03/2013 3:51 AM, Peter Otten wrote:
> Colin J. Williams wrote:
>> The program runs correctly under each version, but it runs more slowly
>> under 3.2.
>
>> This is probably due to the fact that the .pyc file is created for the
>> Python 2.7 execution.
>
>> When Python 3.2 is run it fails to create a new .pyc file and if the 2.7
>> .pyc is offered directly a magic number problem is reported.
>
> (1) .pyc files are only created if a module is imported
> (2) The 2.7 .pyc file is put alongside the .py file whereas the 3.2 .pyc is
> put into the __pycache__ subfolder. No clash can occur.
>
> A simple example:
>
> $ ls
> mod.py
> $ cat mod.py
> print("hello world")
>
> Run it; no pyc is created:
>
> $ python2.7 mod.py
> hello world
> $ ls
> mod.py
>
> Import it using 2.7:
>
> $ python2.7 -c 'import mod'
> hello world
> $ ls
> mod.py  mod.pyc
>
> Import it using 3.2:
>
> $ python3.2 -c 'import mod'
> hello world
> $ ls
> mod.py  mod.pyc  __pycache__
> $ ls __pycache__/
> mod.cpython-32.pyc
>
> Run the compiled code:
>
> $ python2.7 mod.pyc
> hello world
> $ python3.2 __pycache__/mod.cpython-32.pyc
> hello world
>
> But I'm with Steven, it's unlikely that the module compilation phase is
> responsible for a noticeable slowdown.
>
>
Thanks to Steven and Peter for their responses.

My main problem appears to be with:

Profile with Python 2.7
        11   25.736    2.340   25.736    2.340 
{numpy.linalg.lapack_lite.dgesv}

Profile with Python 3.2
        11  152.111   13.828  152.111   13.828 {built-in method dgesv}

In other words, the Python 3.2 linear equation solve takes longer than 
with Python 2.7.   I'll pursue this with the numpy folk.

There also appears to be a problem with the generation of the .pyc. 
Please see the example below:

rem temp.bat
dir *.pyc
del *.pyc
C:\python32\python.exe profiler.py Intel P4 2.8GHz 2MB Ram 221 GB Free 
Disk cjw> prof3.txt
dir *.pyc

This is executed with: tmp.bat > tmp.lst

profiler.py contains:
#-------------------------------------------------------------------------------
# Name:        profiler.py
# Purpose:
#
# Author:      cjw
#
# Created:     17/02/2013
# Copyright:   (c) cjw 2013
# Licence:     <your licence>
#-------------------------------------------------------------------------------

import  cProfile as p, pstats as s, sys

def main():
     v= sys.version
     statsFile= 'FPSStats' + v[0] + v[2] + '.txt'
     p.run('import testFPSpeed; testFPSpeed.main()',
                          statsFile)
     t= s.Stats(statsFile)
     t.strip_dirs().sort_stats('cumulative').print_stats(40)

     pass

if __name__ == '__main__':
     main()

tmp.lst contains:

C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>rem temp.bat

C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>dir *.pyc
  Volume in drive C has no label.
  Volume Serial Number is D001-FAC4

  Directory of C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi


C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>del *.pyc

C:\Documents and Settings\cjw.P4A\My 
Documents\devpy\raspi>C:\python32\python.exe profiler.py Intel P4 2.8GHz 
2MB Ram 221 GB Free Disk cjw 1>prof3.txt

C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>dir *.pyc
  Volume in drive C has no label.
  Volume Serial Number is D001-FAC4

  Directory of C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi


C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>rem temp.bat

C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>dir *.pyc
  Volume in drive C has no label.
  Volume Serial Number is D001-FAC4

  Directory of C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi


C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>del *.pyc

C:\Documents and Settings\cjw.P4A\My 
Documents\devpy\raspi>C:\python32\python.exe profiler.py Intel P4 2.8GHz 
2MB Ram 221 GB Free Disk cjw 1>prof3.txt

C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>dir *.pyc
  Volume in drive C has no label.
  Volume Serial Number is D001-FAC4

  Directory of C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi




tmp.lst contains:

C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>rem temp.bat

C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>dir *.pyc
  Volume in drive C has no label.
  Volume Serial Number is D001-FAC4

  Directory of C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi


C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>del *.pyc

C:\Documents and Settings\cjw.P4A\My 
Documents\devpy\raspi>C:\python32\python.exe profiler.py Intel P4 2.8GHz 
2MB Ram 221 GB Free Disk cjw 1>prof3.txt

C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi>dir *.pyc
  Volume in drive C has no label.
  Volume Serial Number is D001-FAC4

  Directory of C:\Documents and Settings\cjw.P4A\My Documents\devpy\raspi

 From this we see that no .pys was reported for testFPSpeed.py.

However, looking at the directory itself. the .pyc is in fact created.

Thus,  the .pyc is, if necessary, generated upon the import of a .py and 
so this does not explain the time difference between 2.7 and 3.2.

Thanks to Steven for pointing to the __cache__ directory, I find no 
reference to it in the docs.

Colin W.
PS  I wish we could format the text in these messages.

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Apparent  magic number problem "Colin J. Williams" <cjw@ncf.ca> - 2013-03-08 19:48 -0500
  Re: Apparent  magic number problem Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-09 05:22 +0000
  Re: Apparent  magic number problem Peter Otten <__peter__@web.de> - 2013-03-09 09:51 +0100
    Re: Apparent  magic number problem "Colin J. Williams" <cjw@ncf.ca> - 2013-03-09 10:16 -0500
      Re: Apparent  magic number problem Peter Otten <__peter__@web.de> - 2013-03-10 11:56 +0100

csiph-web