Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.unix.programmer > #244

Re: Makefile dependency graphs

Cross-posted to 2 groups.

Show all headers | View raw

> An incremental build process, aka an incrementally correct build
> process, is a build process which automated the dependency analysis,
> determines which steps need to be performed, and executes them. The
> "correct" part refers to how the build process will produce results
> equivalent to that from a full clean rebuild. The "incremental" part
> refers to how the build process uses the automated dependency analysis
> to automatically (as in without user input) to skip certain
> unnecessary steps. The build process obviously requires some user
> configuration to set up, such as specifying makefiles, but with that
> out of the way, it's usually a simple "push this button", such as
> calling "make" from the command line.

Most of my (BSD) makefiles for executables are fairly simple:
_________________________________
PROG = programname
SRCS = a.c b.c c.c main.c
CFLAGS = -g -Wall 

.include <bsd.prog.mk>
_________________________________
followed by running "make depend".


I find most of the problems where you don't get the same result as a
clean build come from dependencies that are not specified.  It should
rebuild just about everything if any of the following change:
	- The compiler
	- The OS kernel
	- The CPU chip (different mask level)
	- RAM chips (previous or current version might have had stuck bits)
	- The network card (I've seen code which gets the MAC address of
	  the build system and embeds it into the executable.  That code
	  got deleted rather quickly after I spotted that.)
	- System libraries
	- System include files
	- Which user is doing the build
	- The user's environment variables when running the build
	  (the user's $PATH might affect which version of gcc is
	  being used, and other environment variables might affect
	  which set of cross-compile include headers are used).
	- The makefile, especially if something in it like CFLAGS
	  is changed.
	- The make command line, especially if make conditionals 
	  cause the value of something like CFLAGS to change.

There is also no way, for example, to know whether the compiler in
use depends on the /bin/mkdir executable or not, and which set of
pieces (preprocesser, compiler, assembler, linker, etc.) it uses.

Compilers often have to use a triple-compile test to ensure that they
work properly:
	Compile source X with any old compiler you have yielding C1.
	Compile source X with C1 yielding C2.
	Compile source X with C2 yielding C3.

If C2 and C3 are not byte-for-byte identical, something is wrong.
Tell me how make is going to detect that this is necessary.
(GCC builds routinely do this test, but that's because it's
written into the build process for the compiler.)


I have run into the interesting situation that certain information
can be passed down from previous generations of a tool.  A certain
Z80 assembler started off simple, and grew with the needs of
developers.  At one point, the ability to use the symbol for an
opcode as the value of the opcode was added.  Later, this feature
was used in the assembler symbol table itself.  Some years later,
and dozens of generations of the assembler, someone pointed out
that the value generated for a particularly obscure opcode was
wrong.  We checked the generated code against the Z80 data sheet.
Yep, it was wrong.  We went into the source code and discovered
that there was no place in the source code to change that value.
It had been passed from build tool to target for a couple of years.

> I claim to be considered "truly" incrementally correct, you need to
> satisfy my stronger requirement.
> 
> First, let us note that makefiles are in a sense another form of
> source code. They're checked into source control, and the executables
> which make the end executable(s) from the (cpp et al) source code also
> read in the makefiles. They're rather indistinguishable from the build
> perspective. Depending on exactly how you structure your makefiles,
> code developers commonly edit these files, and mistakes can result in
> the build process no longer being incrementally correct. That is, when
> the developer hits "make", he can get output which is not equivalent
> to a full clean rebuild.

So for strictly being equivalent to a full clean rebuild, your tool
has to function with empty makefiles.

> Thus, when I talk about incremental correctness of a build process, I
> partition the build system into two separate pieces - the source code,
> and the build "executables". Anything which the developer commonly
> edits, such as the common makefile, should count as source code and
> not "build system executable". Perhaps the per-platform rule makefiles
> could be considered part of the "build system executable". The key is
> that we want to guarantee the incremental correctness of the build
> conditionally on the correctness of the "build system". I want lots of
> tests on the build system "executables" so that developers do not have
> to constantly fear whether their incremental build is correct, or
> whether a bug will be resolved by a full clean rebuild. I can tell you
> at my company that one of my first replies to certain classes of
> obscure errors is "Did you do a full clean rebuild?".
 
> So, with that out of the way, let me talk about why I think make's
> directed acyclic dependency graph of file timestamps is inherently
> broken. The best is by example. I've found 3 distinct problems that
> require some interesting "makefile coding" to get incrementally
> correct.
> 
> 1- Remove a node from a wildcard list. Ex: delete a cpp file. (I
> assume your makefiles use wildcards to compile all the cpp files in a
> directory. If they don't, then I think your build system has usability
> issues.) A naive make build system will not notice that the
> corresponding object file ought to be deleted, and the corresponding
> linked target ought to be rebuilt. Remember, (naive) make only
> rebuilds when there's a prereq with a newer file timestamp than a
> target. A full clean rebuild would produce different results.

I have found wildcards in makefiles to be a severe nuisance (even
though the feature is available), and usually they don't do what I
want.  It is not uncommon that I have source from something else
that I'm importing and adapting/editing, or test programs, or
checked-out previous versions of code, still in that directory, and
"compile all the cpp files in a directory" is often *NOT* going to
work (due to, for example, duplicate copies of main(), duplicated
functions, or pieces of source files that don't compile on their
own.  Some of those apparently source files are *generated* files
(from yacc, bison, a shell script, or the output from another program.)


> 2- Add a node which hides another on a search path. Ex: add a header

If you do this intentionally where the "node" is source code (header
files especially), maintenace programmers are going to want to kill
you.  I've run into places where I've been trying to figure out code,
adding debugging messages which never show up, and finally sprinkle
the code with "#error screw you" statements, which still don't show
up, and discovered I've been editing the wrong file.

I can see your point of you switch from building object files
in the source directory to building object files in individual
object directories.

> file which "hides" another header file on an include path. If you add
> a new header file which "obscures" another, (naive) make won't detect
> this, and a full clean rebuild would produce different results.
> 
> 3- Change compilation options. A naive make solution won't trigger a
> rebuild if you add or remove a cpp preprocessor command line macro
> define. Again, a full clean rebuild will have different results than
> your incremental build.

The same applies to compilation options like adding -g for debug
symbols, then discovering only part of the program has symbols.
For this, keeping "make clean" functional and using it helps a lot.
Yes, I wish there were a better solution here.

> Of course, I do draw the line somewhere. You can't prevent a developer
> who's out to break it. Nor can you prevent acts of god, compiler
> upgrades, and such, but I think that failure to encompass the normal
> "makefiles" as source code in terms of the definition of incremental
> correctness is a mistake.

make normally doesn't have access to the previous version of the
makefile used for the last build.  This would be needed to handle
issues you bring up, like changing compiler flags or compiler command-line
defines.

Back to comp.unix.programmer | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-25 04:54 -0700
  Re: Makefile portability Leclerc <gordan.sikic.remove@this.inet.hr> - 2011-04-26 08:51 +0200
    Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-26 04:25 -0700
    Re: Makefile portability Nobody <nobody@nowhere.com> - 2011-04-26 16:18 +0100
      Re: Makefile portability William Ahern <william@wilbur.25thandClement.com> - 2011-04-26 11:19 -0700
        Re: Makefile portability Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM> - 2011-04-27 18:53 +0100
          Re: Makefile portability "Charlie Gibbs" <cgibbs@kltpzyxm.invalid> - 2011-04-27 14:14 -0800
            Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-30 01:07 -0700
              Re: Makefile portability Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM> - 2011-04-30 13:45 +0100
            Re: Makefile portability Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM> - 2011-04-30 13:14 +0100
            Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-28 00:02 -0700
              Re: Makefile portability Rui Maciel <rui.maciel@gmail.com> - 2011-04-28 20:43 +0100
                Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-28 13:57 -0700
                Re: Makefile portability Rui Maciel <rui.maciel@gmail.com> - 2011-04-29 11:18 +0100
                Re: Makefile portability Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM> - 2011-04-29 20:43 +0100
                Re: Makefile portability Rui Maciel <rui.maciel@gmail.com> - 2011-04-30 11:38 +0100
              Re: Makefile portability and file utility programs Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM> - 2011-04-29 20:27 +0100
                Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-29 15:26 -0700
                Re: Makefile portability Rainer Weikusat <rweikusat@mssgmbh.com> - 2011-04-29 23:45 +0100
                Re: Makefile portability Rui Maciel <rui.maciel@gmail.com> - 2011-04-30 11:26 +0100
      Re: Makefile portability Leclerc <gordan.sikic.remove@this.inet.hr> - 2011-04-27 09:28 +0200
  Re: Makefile portability Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-26 11:28 -0700
    Re: Makefile portability Nobody <nobody@nowhere.com> - 2011-04-26 21:41 +0100
      Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-26 23:17 -0700
        Re: Makefile portability Nobody <nobody@nowhere.com> - 2011-04-27 14:08 +0100
      Re: Makefile portability Leo Havmøller <rtxleh@nospam.nospam> - 2011-04-27 13:15 +0200
        Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-27 05:50 -0700
          Re: Makefile portability Rui Maciel <rui.maciel@gmail.com> - 2011-04-28 20:36 +0100
            Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-28 14:12 -0700
              Re: Makefile portability Rui Maciel <rui.maciel@gmail.com> - 2011-04-29 11:03 +0100
                Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-29 09:30 -0700
      Re: Makefile portability Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-27 15:08 -0700
        Re: Makefile portability Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-27 15:17 -0700
          Re: Makefile dependency graphs gordonb.d36yw@burditt.org (Gordon Burditt) - 2011-04-30 02:02 -0500
            Re: Makefile dependency graphs Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-30 00:36 -0700
          Re: Makefile dependency graphs Michael Press <rubrum@pacbell.net> - 2011-04-30 01:13 -0700
            Re: Makefile dependency graphs William Ahern <william@wilbur.25thandClement.com> - 2011-04-30 11:30 -0700
              Re: Makefile dependency graphs Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-30 15:44 -0700
              Re: Makefile dependency graphs Michael Press <rubrum@pacbell.net> - 2011-05-03 11:48 -0700
          Re: Makefile dependency graphs Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM> - 2011-04-30 12:59 +0100
          Re: Makefile dependency graphs Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM> - 2011-04-29 02:55 +0100
            Re: Makefile dependency graphs William Ahern <william@wilbur.25thandClement.com> - 2011-04-28 19:52 -0700
              Re: Makefile dependency graphs Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-29 02:10 -0700
                Re: Makefile dependency graphs Rainer Weikusat <rweikusat@mssgmbh.com> - 2011-04-29 20:59 +0100
                Re: Makefile dependency graphs Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-29 15:21 -0700
                Re: Makefile dependency graphs Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM> - 2011-04-29 20:08 +0100
                Re: Makefile dependency graphs Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-29 15:08 -0700
                Re: Makefile dependency graphs Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-29 16:11 -0700
              Re: Makefile dependency graphs William Ahern <william@wilbur.25thandClement.com> - 2011-04-29 16:03 -0700
                Re: Makefile dependency graphs Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-29 16:48 -0700
                Re: Makefile dependency graphs Joshua Maurice <joshuamaurice@gmail.com> - 2011-04-29 16:54 -0700
  Re: Makefile portability Rui Maciel <rui.maciel@gmail.com> - 2011-04-28 20:33 +0100
    Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-04-28 14:20 -0700
    Re: Makefile portability Nobody <nobody@nowhere.com> - 2011-04-30 18:03 +0100
      Re: Makefile portability William Ahern <william@wilbur.25thandClement.com> - 2011-04-30 12:01 -0700
      Re: Makefile portability Rui Maciel <rui.maciel@gmail.com> - 2011-05-01 12:33 +0100
        Re: Makefile portability Nobody <nobody@nowhere.com> - 2011-05-02 03:44 +0100
          Re: Makefile portability William Ahern <william@wilbur.25thandClement.com> - 2011-05-01 21:27 -0700
          Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-05-02 00:11 -0700
      Re: Makefile portability Rainer Weikusat <rweikusat@mssgmbh.com> - 2011-05-02 13:41 +0100
        Re: Makefile portability Freedom on the Oceans <alex.buell@munted.org.uk> - 2011-05-02 14:36 +0100
          Re: Makefile portability tm <thomas.mertes@gmx.at> - 2011-05-02 08:09 -0700
        Re: Makefile portability Nobody <nobody@nowhere.com> - 2011-05-02 23:49 +0100
          Re: Makefile portability Rainer Weikusat <rweikusat@mssgmbh.com> - 2011-05-03 10:41 +0100
            Re: Makefile portability Nobody <nobody@nowhere.com> - 2011-05-05 08:30 +0100

csiph-web