Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #15264 > unrolled thread

Re: leftover pyc files

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: leftover pyc files Andrea Crotti <andrea.crotti.0@gmail.com> - 2011-11-02 15:45 +0000
    Re: leftover pyc files Jonathan Hartley <tartley@tartley.com> - 2011-11-03 00:49 -0700
      Re: leftover pyc files Jonathan Hartley <tartley@tartley.com> - 2011-11-03 00:50 -0700
      Re: leftover pyc files Jonathan Hartley <tartley@tartley.com> - 2011-11-03 00:50 -0700
      Re: leftover pyc files Jonathan Hartley <tartley@tartley.com> - 2011-11-03 01:45 -0700
      Re: leftover pyc files Jonathan Hartley <tartley@tartley.com> - 2011-11-03 01:45 -0700
    Re: leftover pyc files Jonathan Hartley <tartley@tartley.com> - 2011-11-03 00:49 -0700
      Re: leftover pyc files Peter Otten <__peter__@web.de> - 2011-11-03 11:51 +0100
      Re: leftover pyc files Andrea Crotti <andrea.crotti.0@gmail.com> - 2011-11-03 17:54 +0000
        Re: leftover pyc files Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-04 06:48 +0000
        Re: leftover pyc files Jonathan Hartley <tartley@tartley.com> - 2011-11-04 02:27 -0700
          Re: leftover pyc files Andrea Crotti <andrea.crotti.0@gmail.com> - 2011-11-04 10:34 +0000
          Re: leftover pyc files Chris Angelico <rosuav@gmail.com> - 2011-11-04 21:39 +1100
            Re: leftover pyc files 88888 Dihedral <dihedral88888@googlemail.com> - 2011-11-04 06:31 -0700
            Re: leftover pyc files 88888 Dihedral <dihedral88888@googlemail.com> - 2011-11-04 06:31 -0700
          Re: leftover pyc files Andrea Crotti <andrea.crotti.0@gmail.com> - 2011-11-04 11:00 +0000
          Re: leftover pyc files Terry Reedy <tjreedy@udel.edu> - 2011-11-04 16:01 -0400
            Re: leftover pyc files Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2011-11-04 23:15 +0000
        Re: leftover pyc files Jonathan Hartley <tartley@tartley.com> - 2011-11-04 02:27 -0700
          Re: leftover pyc files Jonathan Hartley <tartley@tartley.com> - 2011-11-04 02:29 -0700
      Re: leftover pyc files Chris Angelico <rosuav@gmail.com> - 2011-11-04 08:36 +1100
      Re: leftover pyc files David Robinow <drobinow@gmail.com> - 2011-11-03 20:14 -0400

Page 1 of 2  [1] 2  Next page →

#15264 — Re: leftover pyc files

On 11/02/2011 03:03 PM, Peter Otten wrote:
> Switch to Python3.2 ;)

Yes I saw that and it would be great, unfortunately we're stuck with 
Python 2.5 :O
for some more time.

Anyway now the code that does it is a recursive thing ilke
def _clean_orphaned_pycs(self, directory, simulate=False):
     """Remove all the pyc files without a correspondent module
     """
     files = listdir(directory)
     # going down the tree recursively (like os.walk)
     for x in files:
         if x.endswith('.pyc'):
             py_file = x.split('.pyc')[0] + '.py'

             if (not simulate) and (py_file not in files):
                 deletable_pyc = path.join(directory, x)
                 print 'DELETING pyc %s as it has no py %s' % \
                     (deletable_pyc, py_file)
                 remove(deletable_pyc)

         #TODO: move out these special cases
         if path.isdir(path.join(directory, x)) \
             and x != '.svn' \
             and not x.endswith('.egg-info'):

             self._clean_orphaned_pycs(path.join(directory, x))


Can be done much better probably with os.walk or os.path.walk,
any suggestions?

[toc] | [next] | [standalone]

#15279

A task like this is more suited to bash than Python:

find . -name '*.pyc' -exec rm '{}' ';'

[toc] | [prev] | [next] | [standalone]

#15280

This can be added to git as a post-checkout hook:

In your project's .git/hooks/post-checkout:

#!/usr/bin/env bash
cd ./$(git rev-parse --show-cdup)
find . -name '*.pyc' -exec rm '{}' ';'

[toc] | [prev] | [next] | [standalone]

#15281

This can be added to git as a post-checkout hook:

In your project's .git/hooks/post-checkout:

#!/usr/bin/env bash
cd ./$(git rev-parse --show-cdup)
find . -name '*.pyc' -exec rm '{}' ';'

[toc] | [prev] | [next] | [standalone]

#15283

This can be added to your project's .git/hooks/post-checkout:

#!/usr/bin/env bash
cd ./$(git rev-parse --show-cdup)
find . -name '*.pyc' -exec rm '{}' ';'

[toc] | [prev] | [next] | [standalone]

#15284

This can be added to your project's .git/hooks/post-checkout:

#!/usr/bin/env bash
cd ./$(git rev-parse --show-cdup)
find . -name '*.pyc' -exec rm '{}' ';'

[toc] | [prev] | [next] | [standalone]

#15282

A task like this is more suited to bash than Python:

find . -name '*.pyc' -exec rm '{}' ';'

[toc] | [prev] | [next] | [standalone]

#15289

Jonathan Hartley wrote:

> A task like this is more suited to bash than Python:
> 
> find . -name '*.pyc' -exec rm '{}' ';'

You forgot to exclude the .svn directory from the search and didn't limit 
the deletion to pyc-files whose corresponding py-file doesn't exist.

How would your line look with these modifications?

[toc] | [prev] | [next] | [standalone]

#15297

All these ideas (shell and git hooks) are nice, but unfortunately
- it's svn not git
- it's windows not *nix
- we have to remove only the ones without the corresponding *py...

[toc] | [prev] | [next] | [standalone]

#15324

On Thu, 03 Nov 2011 17:54:52 +0000, Andrea Crotti wrote:

> All these ideas (shell and git hooks) are nice, but unfortunately - it's
> svn not git
> - it's windows not *nix
> - we have to remove only the ones without the corresponding *py...

Does it matter? The other .pyc files will be recreated when they are next 
needed. Just nuke all the .pyc files and be done with it. Or if you can't 
bear having to wait for Python to compile them as needed, you can force 
compilation by running your test suite (you do have a test suite, don't 
you?) or by using the compileall.py script.

python -m compileall some_directory 

should do the trick.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#15328

I like to install a Bash shell of some kind on windows boxes I work on, specifically so I can use shell commands like this, just like on any other operating system. Cywin works just fine for this.

svn also has hooks, but sadly not a checkout hook:
http://svnbook.red-bean.com/en/1.1/ch05s02.html
I guess you could write your own two-line wrapper script which does the checkout and then deletes the pyc files.

I don't understand why deleting only some pyc files is important. Can't you just delete them all and let Python re generate the ones you need? I once saw someone create the longest python file they could to see how long generating pyc files takes, and it is very very quick indeed. A human could not notice the delay even for the largest of projects.

Finally, someone asked about skipping .svn dirs: find has a '-prune' option, you can read about it on the manpage.

[toc] | [prev] | [next] | [standalone]

#15333

On 11/04/2011 09:27 AM, Jonathan Hartley wrote:
> I like to install a Bash shell of some kind on windows boxes I work on, specifically so I can use shell commands like this, just like on any other operating system. Cywin works just fine for this.
>
> svn also has hooks, but sadly not a checkout hook:
> http://svnbook.red-bean.com/en/1.1/ch05s02.html
> I guess you could write your own two-line wrapper script which does the checkout and then deletes the pyc files.
>
> I don't understand why deleting only some pyc files is important. Can't you just delete them all and let Python re generate the ones you need? I once saw someone create the longest python file they could to see how long generating pyc files takes, and it is very very quick indeed. A human could not notice the delay even for the largest of projects.
>
> Finally, someone asked about skipping .svn dirs: find has a '-prune' option, you can read about it on the manpa

Uhm yes it makes sense also to just remove all of them, I don't know why 
it was done like this but
probably for "performance" reasons.

I will try both ways, but I would definitively avoid the shell 
scripting, because it's mainly windows
but it has to be multi-platform..

[toc] | [prev] | [next] | [standalone]

#15334

On Fri, Nov 4, 2011 at 9:34 PM, Andrea Crotti <andrea.crotti.0@gmail.com> wrote:
> Uhm yes it makes sense also to just remove all of them, I don't know why it
> was done like this but
> probably for "performance" reasons.
>
> I will try both ways, but I would definitively avoid the shell scripting,
> because it's mainly windows
> but it has to be multi-platform..

If you're removing them all, you don't need to use a powerful shell.
Much much easier! Just recursively del *.pyc and you're done.

ChrisA

[toc] | [prev] | [next] | [standalone]

#15345

Uhn, thanks for the easy way Just delete all *.pyc recursively. spend another 5-20
minutes to recompile all to get everything sync.. That is trivial!

[toc] | [prev] | [next] | [standalone]

#15347

Uhn, thanks for the easy way Just delete all *.pyc recursively. spend another 5-20
minutes to recompile all to get everything sync.. That is trivial!

[toc] | [prev] | [next] | [standalone]

#15336

On 11/04/2011 10:39 AM, Chris Angelico wrote:
> If you're removing them all, you don't need to use a powerful shell. 
> Much much easier! Just recursively del *.pyc and you're done. ChrisA 

Discussing with the guy that did it I think it's actually a good idea 
instead, because removing them *all* means.
- remove a few thousand files every time where maybe 0 would be removed 
otherwise
- recompile the same thousand of files

It might not be so relevant but for 10 lines of code more I guess it's 
fine..

[toc] | [prev] | [next] | [standalone]

#15354

For those not aware, the compiled file caching and import system was 
changed for 3.2. Given test.py, the compiled file is no longer test.pyc, 
in the same directory, but (for cpython32) 
__pycache__/test.cpython-32.pyc. Given the statement 'import test', the 
__pycache__ directory is only searched (for the name given above) after 
finding test.py. So if 'test.py' is deleted or renamed to 'mytest.py', 
an unchanged 'import test' will *fail* instead of importing the obsolete 
.pyc file. So there is no longer a functional reason to delete such 
obsolete files. However, the OP is stuck with 2.5.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#15357

On Fri, 04 Nov 2011 16:01:14 -0400, Terry Reedy wrote:

> For those not aware, the compiled file caching and import system was
> changed for 3.2. Given test.py, the compiled file is no longer test.pyc,
> in the same directory, but (for cpython32)
> __pycache__/test.cpython-32.pyc. Given the statement 'import test', the
> __pycache__ directory is only searched (for the name given above) after
> finding test.py. So if 'test.py' is deleted or renamed to 'mytest.py',
> an unchanged 'import test' will *fail* instead of importing the obsolete
> .pyc file. So there is no longer a functional reason to delete such
> obsolete files. However, the OP is stuck with 2.5.

Oh, I don't know, removing obsolete and useless crud from your file 
system seems like a good enough functional reason to me :)

However, the behaviour of Python 3.2 is a little more complicated than 
the above, since it must still support .pyc only software. If you have 
a .pyc file in the PYTHONPATH, but *outside* of a __pycache__ directory, 
and it is compiled for the correct version of Python, it will still be 
imported as usual.

I'm not sure what happens if you have two .pyc files, one inside the 
cache and one outside.

-- 
Steven

[toc] | [prev] | [next] | [standalone]

#15329

I like to install a Bash shell of some kind on windows boxes I work on, specifically so I can use shell commands like this, just like on any other operating system. Cywin works just fine for this.

svn also has hooks, but sadly not a checkout hook:
http://svnbook.red-bean.com/en/1.1/ch05s02.html
I guess you could write your own two-line wrapper script which does the checkout and then deletes the pyc files.

I don't understand why deleting only some pyc files is important. Can't you just delete them all and let Python re generate the ones you need? I once saw someone create the longest python file they could to see how long generating pyc files takes, and it is very very quick indeed. A human could not notice the delay even for the largest of projects.

Finally, someone asked about skipping .svn dirs: find has a '-prune' option, you can read about it on the manpage.

[toc] | [prev] | [next] | [standalone]

#15330

Apologies for all my messasges appearing twice. I'm using google groups web ui and have no idea why it's doing that. I'll stop using it.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web