Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #38794 > unrolled thread

Python Makefiles... are they possible?

Back to article view | Back to comp.lang.python

Contents

  Python Makefiles... are they possible? Malcolm White <white.m88@gmail.com> - 2013-02-12 16:44 -0800
    Re: Python Makefiles... are they possible? Joel Goldstick <joel.goldstick@gmail.com> - 2013-02-12 19:55 -0500
    Re: Python Makefiles... are they possible? Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-02-13 01:00 +0000
      Re: Python Makefiles... are they possible? Roy Smith <roy@panix.com> - 2013-02-12 20:06 -0500
        Re: Python Makefiles... are they possible? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-02-13 05:54 +0000
          Re: Python Makefiles... are they possible? Benjamin Schollnick <benjamin@schollnick.net> - 2013-02-13 05:45 -0500
          Re: Python Makefiles... are they possible? Dave Angel <d@davea.name> - 2013-02-13 08:33 -0500
          Re: Python Makefiles... are they possible? Roy Smith <roy@panix.com> - 2013-02-13 08:44 -0500
            Re: Python Makefiles... are they possible? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-02-14 13:46 +1100
              Re: Python Makefiles... are they possible? Roy Smith <roy@panix.com> - 2013-02-13 22:02 -0500
              Re: Python Makefiles... are they possible? Chris Angelico <rosuav@gmail.com> - 2013-02-14 17:43 +1100

#38794 — Python Makefiles... are they possible?

I have written a piece of code that will be part of a larger repository of related programs. Within this repository, it is standard to issue a 'make' command to compile any desired program. Is it possible to create a Makefile to compile a simple Python program? Based on what I have come across so far, this is not (at least not typically) the way things are done with Python.

Thanks for any pointers in the right direction!

[toc] | [next] | [standalone]

#38795

[Multipart message — attachments visible in raw view] — view raw

On Tue, Feb 12, 2013 at 7:44 PM, Malcolm White <white.m88@gmail.com> wrote:

> I have written a piece of code that will be part of a larger repository of
> related programs. Within this repository, it is standard to issue a 'make'
> command to compile any desired program. Is it possible to create a Makefile
> to compile a simple Python program? Based on what I have come across so
> far, this is not (at least not typically) the way things are done with
> Python.
>
> Thanks for any pointers in the right direction!
>

Python is an interpreted language.  The source code is 'compiled' into
bytecode when invoked.  Make is a process that converts source code to
compiled code, so it doesn't apply to python

> --
> http://mail.python.org/mailman/listinfo/python-list
>



-- 
Joel Goldstick
http://joelgoldstick.com

[toc] | [prev] | [next] | [standalone]

#38796

On 13 February 2013 00:44, Malcolm White <white.m88@gmail.com> wrote:
> I have written a piece of code that will be part of a larger repository of related programs. Within this repository, it is standard to issue a 'make' command to compile any desired program. Is it possible to create a Makefile to compile a simple Python program? Based on what I have come across so far, this is not (at least not typically) the way things are done with Python.

You can use a Makefile for anything you want in a Python project.
However Python code is not (typically) compiled so it is not common
practise to compile it with or without a Makefile. When part of a
Python project is compiled because, for example it bundles some C code
to be used within Python, the compilation needs to performed in way
that will be compatible with Python so the process is normally
controlled by Python, through a setup.py file. In this case
compilation is done with something like 'python setup.py build' (Of
course there's nothing to stop you from adding that command to a
Makefile and invoking it with 'make').

I often use Makefiles in Python projects for other purposes, though,
such as running tests with 'make test' or building documentation with
'make doc'.


Oscar

[toc] | [prev] | [next] | [standalone]

#38797

In article <mailman.1731.1360717275.2939.python-list@python.org>,
 Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

> On 13 February 2013 00:44, Malcolm White <white.m88@gmail.com> wrote:
> > I have written a piece of code that will be part of a larger repository of 
> > related programs. Within this repository, it is standard to issue a 'make' 
> > command to compile any desired program. Is it possible to create a Makefile 
> > to compile a simple Python program? Based on what I have come across so 
> > far, this is not (at least not typically) the way things are done with 
> > Python.
> 
> You can use a Makefile for anything you want in a Python project.
> However Python code is not (typically) compiled so it is not common
> practise to compile it with or without a Makefile. When part of a
> Python project is compiled because, for example it bundles some C code
> to be used within Python, the compilation needs to performed in way
> that will be compatible with Python so the process is normally
> controlled by Python, through a setup.py file. In this case
> compilation is done with something like 'python setup.py build' (Of
> course there's nothing to stop you from adding that command to a
> Makefile and invoking it with 'make').
> 
> I often use Makefiles in Python projects for other purposes, though,
> such as running tests with 'make test' or building documentation with
> 'make doc'.

One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".  
It avoids all sorts of nasty and hard to track down bugs (consider what 
happens if you move a .py file from one place in your source tree to 
another and leave the old .pyc behind).

[toc] | [prev] | [next] | [standalone]

#38810

On Tue, 12 Feb 2013 20:06:35 -0500, Roy Smith wrote:

> One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
> It avoids all sorts of nasty and hard to track down bugs (consider what
> happens if you move a .py file from one place in your source tree to
> another and leave the old .pyc behind).


How often do you move files around in the source tree? Meanwhile, *every* 
time you run make, you take a performance hit on every Python module in 
your project, whether it has moved or not.

Seems to me like a fairly heavy-handed response for something quite rare, 
but I suppose that depends on how often you run make.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#38815

[Multipart message — attachments visible in raw view] — view raw

>> One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
>> It avoids all sorts of nasty and hard to track down bugs (consider what
>> happens if you move a .py file from one place in your source tree to
>> another and leave the old .pyc behind).
> 
> How often do you move files around in the source tree? Meanwhile, *every* 
> time you run make, you take a performance hit on every Python module in 
> your project, whether it has moved or not.
> 
> Seems to me like a fairly heavy-handed response for something quite rare, 
> but I suppose that depends on how often you run make.

If the performance hit doesn't really matter.  

Then simply walk the build tree, compare time date stamps, anything that doesn't match up in the make directory, gets deleted.  Anything that has different Date Created / Date Modified time from the build tree match, get's deleted.

This way, we are preserving any files that should be identical.  But there should be some mechanism documented to forcibly clear the build cache.

		- Benjamin

[toc] | [prev] | [next] | [standalone]

#38816

On 02/13/2013 12:54 AM, Steven D'Aprano wrote:
> On Tue, 12 Feb 2013 20:06:35 -0500, Roy Smith wrote:
>
>> One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
>> It avoids all sorts of nasty and hard to track down bugs (consider what
>> happens if you move a .py file from one place in your source tree to
>> another and leave the old .pyc behind).
>
>
> How often do you move files around in the source tree? Meanwhile, *every*
> time you run make, you take a performance hit on every Python module in
> your project, whether it has moved or not.
>
> Seems to me like a fairly heavy-handed response for something quite rare,
> but I suppose that depends on how often you run make.
>
>
>
>

That's why I assumed that his command was triggered by the "make clean" 
command.


-- 
DaveA

[toc] | [prev] | [next] | [standalone]

#38817

In article <511b2a7c$0$11096$c3e8da3@news.astraweb.com>,
 Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> On Tue, 12 Feb 2013 20:06:35 -0500, Roy Smith wrote:
> 
> > One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
> > It avoids all sorts of nasty and hard to track down bugs (consider what
> > happens if you move a .py file from one place in your source tree to
> > another and leave the old .pyc behind).
> 
> 
> How often do you move files around in the source tree?

It has happened enough times to make us look for a solution.  Which 
means "more than once".

> Meanwhile, *every* time you run make, you take a performance hit on 
> every Python module in your project, whether it has moved or not.

The performance hit is minimal.  The hours of tearing out your hair 
trying to figure out why bizarre things are happening is not.

Another solution would be if there was a flag you could give to Python 
to tell it, "Only import a .pyc if the corresponding .py file exists".  
It's already checking to see if the .py is newer, so this wouldn't even 
cost anything.

[toc] | [prev] | [next] | [standalone]

#38839

Roy Smith wrote:

> In article <511b2a7c$0$11096$c3e8da3@news.astraweb.com>,
>  Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:
> 
>> On Tue, 12 Feb 2013 20:06:35 -0500, Roy Smith wrote:
>> 
>> > One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
>> > It avoids all sorts of nasty and hard to track down bugs (consider what
>> > happens if you move a .py file from one place in your source tree to
>> > another and leave the old .pyc behind).
>> 
>> 
>> How often do you move files around in the source tree?
> 
> It has happened enough times to make us look for a solution.  Which
> means "more than once".

Maybe the solution is education, not a technical fix.

I've suspicious of technical fixes for developer problems, because in my
experience that strategy ends in a race to the bottom where you end up with
coding standards and procedures that assume everyone is a code monkey who
can barely spell PC. It doesn't protect the monkeys, because there is no
end to the ways they can screw up, while the competent developers suffer
under repressive, B&D procedures that hinder more than help.

YMMV.

I prefer to keep the .pyc files, and only remove them when necessary, rather
than to remove them whether it's necessary or not. It's not just because
I'm an arrogant SOB who expects my team of developers to know at least more
than me, therefore if I know enough to look for orphaned .pyc files so
should they. It's because I am a big believer that your development system
should be as close as possible to the eventual deployment system as is
practical. Your app will (probably) use .pyc files when it is deployed, so
you should do the same when developing. Otherwise you can get bugs in
deployment that you cannot reproduce in development because the
environments are different.


>> Meanwhile, *every* time you run make, you take a performance hit on
>> every Python module in your project, whether it has moved or not.
> 
> The performance hit is minimal.

I guess that depends on the size of your project and how much you care about
the start up time. But as general advice, no, it may not be minimal.

[...]


> Another solution would be if there was a flag you could give to Python
> to tell it, "Only import a .pyc if the corresponding .py file exists".
> It's already checking to see if the .py is newer, so this wouldn't even
> cost anything.

That's called "Python 3.3" :-)



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#38840

In article <511c501d$0$6512$c3e8da3$5496439d@news.astraweb.com>,
 Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> I prefer to keep the .pyc files, and only remove them when necessary, rather
> than to remove them whether it's necessary or not. It's not just because
> I'm an arrogant SOB who expects my team of developers to know at least more
> than me, therefore if I know enough to look for orphaned .pyc files so
> should they. It's because I am a big believer that your development system
> should be as close as possible to the eventual deployment system as is
> practical. Your app will (probably) use .pyc files when it is deployed, so
> you should do the same when developing.

Heh.  Our deployment system rolls out all the source code from scratch 
on every deploy.

> Meanwhile, *every* time you run make, you take a performance hit on
> every Python module in your project, whether it has moved or not.

Yup.

[toc] | [prev] | [next] | [standalone]

#38848

On Thu, Feb 14, 2013 at 1:46 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> I prefer to keep the .pyc files, and only remove them when necessary, rather
> than to remove them whether it's necessary or not.

Solution to that could be just to have your makefile wipe out
orphanned pyc files, rather than all of them. Still might be a
performance hit (if it has to wade through piles of pyc/py files to
see which ones aren't needed), but otherwise should be safe.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web