Re: software disenchantment

From	David Brown <david.brown@hesbynett.no>
Newsgroups	comp.lang.c
Subject	Re: software disenchantment
Date	2018-10-11 14:10 +0200
Organization	A noiseless patient Spider
Message-ID	<ppneji$ke$1@dont-email.me> (permalink)
References	(15 earlier) <4TuvD.69650$522.16377@fx35.am4> <g27ccqFjb2cU7@mid.individual.net> <32vvD.21117$134.7822@fx03.am4> <ppn5kf$clr$1@dont-email.me> <RVFvD.15651$I34.1016@fx43.am4>

Show all headers | View raw

On 11/10/18 12:50, Bart wrote:
> On 11/10/2018 10:37, David Brown wrote:
>> On 11/10/18 00:28, Bart wrote:
> 
>>> Would 2500 object files need to be accessed in the course of even an
>>> incremental build? Or is it clever enough to patch in only the ones that
>>> have changed?
> 
>> So for any incremental build, usually only a small fraction (typically
>> just one) of the object files get re-built.
>>
>> If you have incremental linking, then much of the previous link results
>> are also kept - if not, then you may have to read all the object files
>> for the link.
> 
> I said 2500 OBJECT files. 

Yes, I know you said that.

> So that already assumes that only some source
> files are re-compiled.

No, it does not imply this.  But that does not matter to my point.

It is not clear what you mean by "would 2500 object files need to be
accessed" - accessed in what way?  I took that to mean "read" - if you
had meant "written" or "generated", I think you would have said that.
The only reason you read object files in a typical build process is for
the linking.

It is common for the link to use all the object files in one link
command.  But if you have a bigger project, you might use incremental
linking - then you don't need to read all the object files for each
link.  (Link-time optimisation is also possible, but more complex.)

> 
>> Some timings for a project with 113 object files are:
>>
>> Serial "clean" build: 51s
>> Parallel "clean" build: 12s
>> Incremental build for one C file changed: 1.8s
>>
>> Does that explain why people use incremental builds?
> 
> Incremental builds and multi-processing. It's what you have to throw at
> a problem when the basic methods are slow and inefficient.
> 

No, it is what you use to make everything go much faster.  It does not
matter how fast your compilations are - with big enough projects,
"compile everything, one file at a time" will be slow in the end.  If
your super compiler takes a mere 50 ms to compile a file, then with 2500
files it takes 2 minutes.  A parallel clean build on an 8-core PC will
take 15 seconds.  An incremental build will take 50 ms.

Serial build-everything-everytime builds are fine for toy projects - but
not for more serious work, /regardless/ of the compiler speed.

> What I'm investigating (as are many other people, to try and solve some
> of the problems highlighted in the article in the OP) is how much it is
> possible to reimplement those basic methods.

Enjoy your hobby experiments.  Just don't imagine they are relevant to
people working on more serious development.

In C++, compiler speed is more of an issue due to the trend towards big
header-only template libraries.  So there is a plan there for "modules"
to avoid reading source header files in compilation, and instead read
pre-chewed binary module files for imports.

> 
> Look at the 'gold' linker, especially created from scratch to be more
> efficient than 'ld'. Someone else who was dissatisfied with the status
> quo and wasn't content just to buy a faster machine with more cores.
> 

Look at the "clang/llvm" compiler, created from scratch and intending to
be faster than other compilers.  And it /was/ faster than gcc to begin
with.  But by the time it reached a similar level of features to gcc, it
has a similar run-time.  It turns out that a good compiler with strong
optimisations, lots of features, powerful static analysis and warnings,
and flexible support for targets simply takes a while to run.

That does not mean there isn't scope for improvement in compile times -
merely that the gains are not huge, even when you start from scratch.
(And unlike for a linker, making a real compiler from scratch is a huge
undertaking.)

> Why wasn't incremental linking enough of a solution for them?

"gold" was not created to be more efficient than "ld" - it was created
to add features that "ld" did not have (such as plugins, and link-time
optimisation support).  The speed increase was a byproduct of removing
support for older formats, targets, etc., that allowed much of it to be
simpler.

And whether you use "ld" or "gold", incremental linking can speed up big
links.

> 
>> (And I don't care if your compiler can chew through a gazillion lines
>> per second - it won't do what I need, so its speed is irrelevant.)
> 
> You wouldn't. But it does what /I/ need. And it shows what /can/ be
> done. 

I am glad that your compiler does what /you/ need - and I am in no way
disparaging the work you have done or the results you have achieved.
But it does not compare to a top-class compiler, and it does not do what
I need or want from a compiler.  It is thus utterly irrelevant to /me/.

It is like you telling me you have built a go-kart that can get you to
your work and the shops, and is vastly more fuel efficient than my big,
lumpy normal car.  But I want to drive on the motorway - it is
irrelevant how efficient your go-kart is at going to the local shops,
because it will not do the job I need.

And no, your go-kart says /nothing/ about "what /can/ be done" - because
it gives no clue as to how fuel efficient a vehicle could be when it
meets the requirements of other drivers.

> (And there is still a lot more it can do, but it's already too
> fast to accurately measure for my size of projects, where are other
> normally insignificant overheads start to dominate.)
> 
>>> (And a build system done differently, could eliminate explicit object
>>> files.)
>>>
>>
>> What a great idea - increase build times by a factor of 10 for my
>> project, and perhaps 100 for one the size of Ian's.
> 
> Well, I've eliminated object files for most purposes. My build times
> have improved. (Actually, I've eliminated them because they simply
> weren't needed. They are replaced with .asm files.)
> 

Again, great for your little programs - useless for bigger ones.

> Perhaps super-fast, super-efficient compiler tools are not going to
> provide sub-one-second 'clean build' turnaround times for the some of
> the giant projects presented here.
> 
> But imagine if the underlying tools were even one magnitude faster? Your
> parallel clean build would be faster than your incremental build, and it
> wouldn't matter which files were modified. It would always be
> consistently that speed.
> 

If the underlying compiler (and linker) was one magnitude faster, it
would make the parallel clean build as fast as the current incremental
build.  So what?  I would still use incremental builds - and that would
then be faster still.

And when the next project has twice as many files, the incremental build
is still the same time - because it generally only compiles one file.
The parallel clean build now takes twice as long, however.

Let me stress that I appreciate the aim of having faster compilation -
that is always a good thing in itself.  It is not a major aim, however,
and I don't put it ahead of features of the compiler.

There is no point in /not/ having a parallel incremental build system -
clean builds offer no advantages, and however fast your compiler is,
incremental builds offer additional speedups that scale well with size.

Thread

software disenchantment Bart <bc@freeuk.com> - 2018-10-06 15:13 +0100
  Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-06 09:54 -0700
  Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-06 10:26 -0700
    Re: software disenchantment fir <profesor.fir@gmail.com> - 2018-10-14 04:26 -0700
  Re: software disenchantment antispam@math.uni.wroc.pl - 2018-10-09 18:20 +0000
    Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-09 11:54 -0700
      Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-09 21:22 +0100
        Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-09 17:47 -0700
          Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 11:28 +0100
            Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 11:38 +0100
              Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-10 05:49 -0700
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-10 15:47 +0200
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 09:58 -0400
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-10 14:39 +0000
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-10 09:42 -0700
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 13:18 -0400
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 20:31 +0100
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 17:53 -0500
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 10:06 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 02:34 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 12:03 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 03:59 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 16:56 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 09:02 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 19:39 +0200
                Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-15 19:53 +0000
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-16 03:54 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 20:25 +0200
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 13:16 +0000
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 07:15 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 17:05 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 09:15 -0700
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 17:33 +0000
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 19:53 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 17:59 +0100
                Re: software disenchantment nobody@example.org (Scott) - 2018-10-10 16:53 +0000
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 12:45 -0500
                Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-10 11:04 -0700
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 14:22 -0500
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 15:53 -0400
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 09:06 +1300
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 13:21 -0700
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-10 20:27 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 22:54 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:01 +1300
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:05 +1300
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 23:16 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:22 +1300
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 23:28 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:33 +1300
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 11:37 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 11:50 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 14:10 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 05:24 -0700
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 15:12 +0200
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 13:13 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 14:44 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 16:38 +0200
                Re: software disenchantment mark.bluemel@gmail.com - 2018-10-11 08:09 -0700
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 17:43 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 17:36 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 18:56 +0100
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-11 11:54 -0700
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 20:09 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 13:10 +0000
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-13 12:57 +0200
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 14:27 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 22:03 +0200
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 15:56 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 11:30 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 23:34 +0100
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-11 16:24 -0700
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 01:46 +0100
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-11 18:58 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-12 08:50 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 11:12 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 12:55 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 12:10 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 13:29 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 13:06 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 14:30 +0200
                Re: software disenchantment James Kuyper <jameskuyper@alumni.caltech.edu> - 2018-10-12 08:53 -0400
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 15:11 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 13:55 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 15:13 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 16:10 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 15:22 +0000
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-12 09:12 -0700
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-13 13:00 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-13 12:52 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-14 08:44 +1300
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-13 21:03 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-14 10:13 +1300
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-13 23:02 +0100
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-13 16:47 -0700
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-14 01:05 +0100
                Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-13 18:12 -0700
                Re: software disenchantment Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-10-14 03:05 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-14 20:14 +0200
                Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-14 11:25 -0700
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 13:17 +0000
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 13:16 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 14:36 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 14:25 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 15:55 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 15:22 +0000
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-12 08:26 -0700
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 15:51 +0000
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-13 13:06 +0200
                Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-15 19:30 +0000
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 12:39 +0200
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-12 08:44 +0200
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 13:01 +0200
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 20:41 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 20:02 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 22:13 +0200
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 13:03 -0500
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-11 11:33 -0700
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 21:04 +0200
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 20:15 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 13:17 +0200
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 13:10 +0000
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 19:22 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 10:58 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 02:37 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 12:06 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 04:00 -0700
                Re: software disenchantment Cecil Bayona <cbayona@cbayona.com> - 2018-10-11 13:03 -0500
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 19:36 +0100
                Re: software disenchantment Cecil Bayona <cbayona@cbayona.com> - 2018-10-11 15:37 -0500
                Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-10 14:01 -0700
        Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 09:02 +1300
      Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-10 19:11 +0000
        Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-10 13:08 -0700
          Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-11 14:52 +0000
            Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 14:58 +0000
            Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 17:19 +0200
              Re: software disenchantment Dan Purgert <dan@djph.net> - 2018-10-12 09:49 +0000
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-13 09:30 +0200
                Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-15 08:53 +0000
                Re: software disenchantment Dan Purgert <dan@djph.net> - 2018-10-15 09:44 +0000
    Re: software disenchantment Jens Stuckelberger <Jens_Stuckelberger@nowhere.net> - 2018-10-10 13:36 +0000
  Re: software disenchantment "Chris M. Thomasson" <invalid_chris_thomasson@invalid.invalid> - 2018-10-10 13:51 -0700
    Re: software disenchantment nobody@example.org (Scott) - 2018-10-11 01:53 +0000
    Re: software disenchantment Dan Purgert <dan@djph.net> - 2018-10-11 10:20 +0000
    Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 12:53 +0100
      Re: software disenchantment "Chris M. Thomasson" <invalid_chris_thomasson@invalid.invalid> - 2018-10-11 23:18 -0700

csiph-web