Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.c > #136342 > unrolled thread
| Started by | Bart <bc@freeuk.com> |
|---|---|
| First post | 2018-10-06 15:13 +0100 |
| Last post | 2018-10-11 23:18 -0700 |
| Articles | 20 on this page of 147 — 22 participants |
Back to article view | Back to comp.lang.c
software disenchantment Bart <bc@freeuk.com> - 2018-10-06 15:13 +0100
Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-06 09:54 -0700
Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-06 10:26 -0700
Re: software disenchantment fir <profesor.fir@gmail.com> - 2018-10-14 04:26 -0700
Re: software disenchantment antispam@math.uni.wroc.pl - 2018-10-09 18:20 +0000
Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-09 11:54 -0700
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-09 21:22 +0100
Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-09 17:47 -0700
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 11:28 +0100
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 11:38 +0100
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-10 05:49 -0700
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-10 15:47 +0200
Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 09:58 -0400
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-10 14:39 +0000
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-10 09:42 -0700
Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 13:18 -0400
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 20:31 +0100
Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 17:53 -0500
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 10:06 +0200
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 02:34 -0700
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 12:03 +0200
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 03:59 -0700
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 16:56 +0200
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 09:02 -0700
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 19:39 +0200
Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-15 19:53 +0000
Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-16 03:54 -0500
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 20:25 +0200
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 13:16 +0000
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 07:15 -0700
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 17:05 +0200
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 09:15 -0700
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 17:33 +0000
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 19:53 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 17:59 +0100
Re: software disenchantment nobody@example.org (Scott) - 2018-10-10 16:53 +0000
Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 12:45 -0500
Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-10 11:04 -0700
Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 14:22 -0500
Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 15:53 -0400
Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 09:06 +1300
Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 13:21 -0700
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-10 20:27 +0000
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 22:54 +0100
Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:01 +1300
Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:05 +1300
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 23:16 +0100
Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:22 +1300
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 23:28 +0100
Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:33 +1300
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 11:37 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 11:50 +0100
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 14:10 +0200
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 05:24 -0700
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 15:12 +0200
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 13:13 +0000
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 14:44 +0100
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 16:38 +0200
Re: software disenchantment mark.bluemel@gmail.com - 2018-10-11 08:09 -0700
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 17:43 +0100
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 17:36 +0000
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 18:56 +0100
Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-11 11:54 -0700
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 20:09 +0100
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 13:10 +0000
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-13 12:57 +0200
Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 14:27 -0500
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 22:03 +0200
Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 15:56 -0500
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 11:30 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 23:34 +0100
Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-11 16:24 -0700
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 01:46 +0100
Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-11 18:58 -0700
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-12 08:50 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 11:12 +0100
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 12:55 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 12:10 +0100
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 13:29 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 13:06 +0100
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 14:30 +0200
Re: software disenchantment James Kuyper <jameskuyper@alumni.caltech.edu> - 2018-10-12 08:53 -0400
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 15:11 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 13:55 +0100
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 15:13 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 16:10 +0100
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 15:22 +0000
Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-12 09:12 -0700
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-13 13:00 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-13 12:52 +0100
Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-14 08:44 +1300
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-13 21:03 +0100
Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-14 10:13 +1300
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-13 23:02 +0100
Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-13 16:47 -0700
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-14 01:05 +0100
Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-13 18:12 -0700
Re: software disenchantment Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-10-14 03:05 +0100
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-14 20:14 +0200
Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-14 11:25 -0700
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 13:17 +0000
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 13:16 +0000
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 14:36 +0100
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 14:25 +0000
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 15:55 +0100
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 15:22 +0000
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-12 08:26 -0700
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 15:51 +0000
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-13 13:06 +0200
Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-15 19:30 +0000
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 12:39 +0200
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-12 08:44 +0200
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 13:01 +0200
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 20:41 +0200
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 20:02 +0100
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 22:13 +0200
Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 13:03 -0500
Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-11 11:33 -0700
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 21:04 +0200
Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 20:15 -0500
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 13:17 +0200
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 13:10 +0000
Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 19:22 -0500
Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 10:58 +0200
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 02:37 -0700
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 12:06 +0200
Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 04:00 -0700
Re: software disenchantment Cecil Bayona <cbayona@cbayona.com> - 2018-10-11 13:03 -0500
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 19:36 +0100
Re: software disenchantment Cecil Bayona <cbayona@cbayona.com> - 2018-10-11 15:37 -0500
Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-10 14:01 -0700
Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 09:02 +1300
Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-10 19:11 +0000
Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-10 13:08 -0700
Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-11 14:52 +0000
Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 14:58 +0000
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 17:19 +0200
Re: software disenchantment Dan Purgert <dan@djph.net> - 2018-10-12 09:49 +0000
Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-13 09:30 +0200
Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-15 08:53 +0000
Re: software disenchantment Dan Purgert <dan@djph.net> - 2018-10-15 09:44 +0000
Re: software disenchantment Jens Stuckelberger <Jens_Stuckelberger@nowhere.net> - 2018-10-10 13:36 +0000
Re: software disenchantment "Chris M. Thomasson" <invalid_chris_thomasson@invalid.invalid> - 2018-10-10 13:51 -0700
Re: software disenchantment nobody@example.org (Scott) - 2018-10-11 01:53 +0000
Re: software disenchantment Dan Purgert <dan@djph.net> - 2018-10-11 10:20 +0000
Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 12:53 +0100
Re: software disenchantment "Chris M. Thomasson" <invalid_chris_thomasson@invalid.invalid> - 2018-10-11 23:18 -0700
Page 3 of 8 — ← Prev page 1 2 [3] 4 5 6 7 8 Next page →
| From | Ian Collins <ian-news@hotmail.com> |
|---|---|
| Date | 2018-10-11 09:06 +1300 |
| Message-ID | <g274dhFjb2cU4@mid.individual.net> |
| In reply to | #136571 |
On 11/10/18 08:53, Rick C. Hodgin wrote: > On 10/10/2018 3:22 PM, BGB wrote: >> On 10/10/2018 1:04 PM, Thiago Adams wrote: >>> The solution is pre-compiled headers. We still have the "source" >>> and the pre-compiled headers can optimize the parser. > > Pre-compiled headers don't always get you there. You can have > pre-parsed headers, but even then if a #define comes into play > which re-defines something referenced somewhere, your pre- > compiled header will be invalid and may require a full repars- > ing anyway. Which is why compilers usually require the precompiled headers header include to be the first item in a compilation unit. -- Ian.
[toc] | [prev] | [next] | [standalone]
| From | "Rick C. Hodgin" <rick.c.hodgin@gmail.com> |
|---|---|
| Date | 2018-10-10 13:21 -0700 |
| Message-ID | <80b8860c-aebf-4041-aee8-6e8fd515fa0f@googlegroups.com> |
| In reply to | #136573 |
On Wednesday, October 10, 2018 at 4:06:17 PM UTC-4, Ian Collins wrote: > On 11/10/18 08:53, Rick C. Hodgin wrote: > > On 10/10/2018 3:22 PM, BGB wrote: > >> On 10/10/2018 1:04 PM, Thiago Adams wrote: > >>> The solution is pre-compiled headers. We still have the "source" > >>> and the pre-compiled headers can optimize the parser. > > > > Pre-compiled headers don't always get you there. You can have > > pre-parsed headers, but even then if a #define comes into play > > which re-defines something referenced somewhere, your pre- > > compiled header will be invalid and may require a full repars- > > ing anyway. > > Which is why compilers usually require the precompiled headers header > include to be the first item in a compilation unit. It still limits their utility because you can only put in things that are wholly fixed into that area, such as OS or remote library files, lets they be re-compiled anyway. Having the ability to parse out every source files, and then to be able to do a hard lookup on individual changes as source code is being edited in real-time ... it's far more desirable because it addresses the entire code base with a processing time spread out over normal developer IDE use. -- Rick C. Hodgin
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2018-10-10 20:27 +0000 |
| Message-ID | <jhtvD.161950$SN2.73697@fx38.iad> |
| In reply to | #136575 |
"Rick C. Hodgin" <rick.c.hodgin@gmail.com> writes: >On Wednesday, October 10, 2018 at 4:06:17 PM UTC-4, Ian Collins wrote: >> On 11/10/18 08:53, Rick C. Hodgin wrote: >> > On 10/10/2018 3:22 PM, BGB wrote: >> >> On 10/10/2018 1:04 PM, Thiago Adams wrote: >> >>> The solution is pre-compiled headers. We still have the "source" >> >>> and the pre-compiled headers can optimize the parser. >> > >> > Pre-compiled headers don't always get you there. You can have >> > pre-parsed headers, but even then if a #define comes into play >> > which re-defines something referenced somewhere, your pre- >> > compiled header will be invalid and may require a full repars- >> > ing anyway. >> >> Which is why compilers usually require the precompiled headers header >> include to be the first item in a compilation unit. >Having the ability to parse out every source files, and then to >be able to do a hard lookup on individual changes as source code >is being edited in real-time ... it's far more desirable because >it addresses the entire code base with a processing time spread >out over normal developer IDE use. But it's a non-existent problem. The project I work on at the moment as per the sloccount utility: Totals grouped by language (dominant language first): ansic: 2160423 (72.79%) cpp: 452648 (15.25%) python: 317663 (10.70%) asm: 34952 (1.18%) perl: 1103 (0.04%) sh: 1093 (0.04%) Total Physical Source Lines of Code (SLOC) = 2,967,882 full build timing: real 2m49.10s user 17m0.46s sys 1m50.75s Less than three minutes[*] for a two million line compile is in the noise. And incremental compiles (the norm for a developer) are basically instananeous (recompile one or two files and relink). I don't see that spending any time at all with respect to optimization of handling header files would have any real world benefit (note that there are over a million lines of header files in the above project). [*] the more observent may notice that it took about 19 minutes of CPU time, but less than three minutes of wall time. Parallelism is your friend.
[toc] | [prev] | [next] | [standalone]
| From | Bart <bc@freeuk.com> |
|---|---|
| Date | 2018-10-10 22:54 +0100 |
| Message-ID | <8yuvD.13782$5A2.519@fx25.am4> |
| In reply to | #136576 |
On 10/10/2018 21:27, Scott Lurndal wrote: > "Rick C. Hodgin" <rick.c.hodgin@gmail.com> writes: >> On Wednesday, October 10, 2018 at 4:06:17 PM UTC-4, Ian Collins wrote: >>> On 11/10/18 08:53, Rick C. Hodgin wrote: >>>> On 10/10/2018 3:22 PM, BGB wrote: >>>>> On 10/10/2018 1:04 PM, Thiago Adams wrote: >>>>>> The solution is pre-compiled headers. We still have the "source" >>>>>> and the pre-compiled headers can optimize the parser. >>>> >>>> Pre-compiled headers don't always get you there. You can have >>>> pre-parsed headers, but even then if a #define comes into play >>>> which re-defines something referenced somewhere, your pre- >>>> compiled header will be invalid and may require a full repars- >>>> ing anyway. >>> >>> Which is why compilers usually require the precompiled headers header >>> include to be the first item in a compilation unit. > >> Having the ability to parse out every source files, and then to >> be able to do a hard lookup on individual changes as source code >> is being edited in real-time ... it's far more desirable because >> it addresses the entire code base with a processing time spread >> out over normal developer IDE use. > > But it's a non-existent problem. > > The project I work on at the moment as per the sloccount utility: > > Totals grouped by language (dominant language first): > ansic: 2160423 (72.79%) > cpp: 452648 (15.25%) > python: 317663 (10.70%) > asm: 34952 (1.18%) > perl: 1103 (0.04%) > sh: 1093 (0.04%) > > Total Physical Source Lines of Code (SLOC) = 2,967,882 > > full build timing: > > real 2m49.10s > user 17m0.46s > sys 1m50.75s > > Less than three minutes[*] for a two million line compile is in the noise. And > incremental compiles (the norm for a developer) are basically > instananeous (recompile one or two files and relink). What counts as 'instant' here? My threshold is about 0.2 seconds, above which any delay becomes increasingly annoying. > I don't see that spending any time at all with respect to optimization > of handling header files would have any real world benefit (note that there > are over a million lines of header files in the above project). > > [*] the more observent may notice that it took about 19 minutes of > CPU time, but less than three minutes of wall time. Parallelism is your > friend. Do you have a figure for the number of C source lines that are actually processed? (Due to repeatedly processing the same headers etc.) What part does file caching play here? Working only with a 2.6Mloc of cached C/C++ sources and the 19 minutes (ignoring the other stuff; does the Python need to be compiled?) that gives a throughput of under 2.5 Kloc per second, per core. I think Tiny C can probably do at least 100 times that. And, unless the code is a single monolithic source file, can also be employed on multiple cores. That would give the possibility of a cold compile, for the 2M lines of C sources, of a few seconds. What I'm saying is that 3 minutes' build time, even 3M lines of code, is still three minutes. Too long to have to wait for anything without finding something else to do. And by the time you get back to it, it will be a lot more than 3 minutes. And compiling from scratch is preferable to incremental compile which is subject to conditions. So new ways of streamlining the process will always be welcome. -- bart
[toc] | [prev] | [next] | [standalone]
| From | Ian Collins <ian-news@hotmail.com> |
|---|---|
| Date | 2018-10-11 11:01 +1300 |
| Message-ID | <g27b6mFjb2cU5@mid.individual.net> |
| In reply to | #136580 |
On 11/10/18 10:54, Bart wrote: > On 10/10/2018 21:27, Scott Lurndal wrote: >> "Rick C. Hodgin" <rick.c.hodgin@gmail.com> writes: >> >>> Having the ability to parse out every source files, and then to >>> be able to do a hard lookup on individual changes as source code >>> is being edited in real-time ... it's far more desirable because >>> it addresses the entire code base with a processing time spread >>> out over normal developer IDE use. >> >> But it's a non-existent problem. >> >> The project I work on at the moment as per the sloccount utility: >> >> Totals grouped by language (dominant language first): >> ansic: 2160423 (72.79%) >> cpp: 452648 (15.25%) >> python: 317663 (10.70%) >> asm: 34952 (1.18%) >> perl: 1103 (0.04%) >> sh: 1093 (0.04%) >> >> Total Physical Source Lines of Code (SLOC) = 2,967,882 >> >> full build timing: >> >> real 2m49.10s >> user 17m0.46s >> sys 1m50.75s >> >> Less than three minutes[*] for a two million line compile is in the noise. And >> incremental compiles (the norm for a developer) are basically >> instananeous (recompile one or two files and relink). > > What counts as 'instant' here? My threshold is about 0.2 seconds, above > which any delay becomes increasingly annoying. You obviously haven't worked with Android or Angular... > What I'm saying is that 3 minutes' build time, even 3M lines of code, is > still three minutes. Too long to have to wait for anything without > finding something else to do. And by the time you get back to it, it > will be a lot more than 3 minutes. You do do something ease, move on to your next update while the build is running. Big multiple developer code bases take time to build, that's the way of the wold. > And compiling from scratch is preferable to incremental compile which is > subject to conditions. So new ways of streamlining the process will > always be welcome. No, compiling from scratch isn't preferable to incremental builds. -- Ian.
[toc] | [prev] | [next] | [standalone]
| From | Ian Collins <ian-news@hotmail.com> |
|---|---|
| Date | 2018-10-11 11:05 +1300 |
| Message-ID | <g27bd6Fjb2cU6@mid.individual.net> |
| In reply to | #136583 |
On 11/10/18 11:01, Ian Collins wrote: > > You do do something ease, move on to your next update while the build is > running. Big multiple developer code bases take time to build, that's > the way of the wold. So little text, so many typos... -- Ian.
[toc] | [prev] | [next] | [standalone]
| From | Bart <bc@freeuk.com> |
|---|---|
| Date | 2018-10-10 23:16 +0100 |
| Message-ID | <4TuvD.69650$522.16377@fx35.am4> |
| In reply to | #136583 |
On 10/10/2018 23:01, Ian Collins wrote: > On 11/10/18 10:54, Bart wrote: >> And compiling from scratch is preferable to incremental compile which is >> subject to conditions. So new ways of streamlining the process will >> always be welcome. > > No, compiling from scratch isn't preferable to incremental builds. Even if both took the same amount of time? (Or both were fast enough to be insignificant.) (My latest tools (not for C), /only/ compile from scratch. It means I can make any change anywhere, and know it will taken into account throughout the project, without incurring any sort of build penalty. Build time for my largest project, of 50K lines, can be 0.25 seconds. On one core.)
[toc] | [prev] | [next] | [standalone]
| From | Ian Collins <ian-news@hotmail.com> |
|---|---|
| Date | 2018-10-11 11:22 +1300 |
| Message-ID | <g27ccqFjb2cU7@mid.individual.net> |
| In reply to | #136588 |
On 11/10/18 11:16, Bart wrote: > On 10/10/2018 23:01, Ian Collins wrote: >> On 11/10/18 10:54, Bart wrote: > >>> And compiling from scratch is preferable to incremental compile which is >>> subject to conditions. So new ways of streamlining the process will >>> always be welcome. >> >> No, compiling from scratch isn't preferable to incremental builds. > > Even if both took the same amount of time? (Or both were fast enough to > be insignificant.) That ain't going to happen for big projects. The C & C++ part of our project (which isn't too big) contains over 2500 source files. -- Ian.
[toc] | [prev] | [next] | [standalone]
| From | Bart <bc@freeuk.com> |
|---|---|
| Date | 2018-10-10 23:28 +0100 |
| Message-ID | <32vvD.21117$134.7822@fx03.am4> |
| In reply to | #136590 |
On 10/10/2018 23:22, Ian Collins wrote: > On 11/10/18 11:16, Bart wrote: >> On 10/10/2018 23:01, Ian Collins wrote: >>> On 11/10/18 10:54, Bart wrote: >> >>>> And compiling from scratch is preferable to incremental compile >>>> which is >>>> subject to conditions. So new ways of streamlining the process will >>>> always be welcome. >>> >>> No, compiling from scratch isn't preferable to incremental builds. >> >> Even if both took the same amount of time? (Or both were fast enough to >> be insignificant.) > > That ain't going to happen for big projects. The C & C++ part of our > project (which isn't too big) contains over 2500 source files. Would 2500 object files need to be accessed in the course of even an incremental build? Or is it clever enough to patch in only the ones that have changed? If all object files do have to be loaded, then that is not so different from having to load source files. (And a build system done differently, could eliminate explicit object files.) -- bart
[toc] | [prev] | [next] | [standalone]
| From | Ian Collins <ian-news@hotmail.com> |
|---|---|
| Date | 2018-10-11 11:33 +1300 |
| Message-ID | <g27d2hFjb2cU8@mid.individual.net> |
| In reply to | #136593 |
On 11/10/18 11:28, Bart wrote: > On 10/10/2018 23:22, Ian Collins wrote: >> On 11/10/18 11:16, Bart wrote: >>> On 10/10/2018 23:01, Ian Collins wrote: >>>> On 11/10/18 10:54, Bart wrote: >>> >>>>> And compiling from scratch is preferable to incremental compile >>>>> which is >>>>> subject to conditions. So new ways of streamlining the process will >>>>> always be welcome. >>>> >>>> No, compiling from scratch isn't preferable to incremental builds. >>> >>> Even if both took the same amount of time? (Or both were fast enough to >>> be insignificant.) >> >> That ain't going to happen for big projects. The C & C++ part of our >> project (which isn't too big) contains over 2500 source files. > > Would 2500 object files need to be accessed in the course of even an > incremental build? Or is it clever enough to patch in only the ones that > have changed? Make and Ninja know what to rebuild by maintaining dependency trees. > If all object files do have to be loaded, then that is not so different > from having to load source files. > > (And a build system done differently, could eliminate explicit object > files.) It could, but why would it? You need something that can be quantised to support distributed building. -- Ian.
[toc] | [prev] | [next] | [standalone]
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2018-10-11 11:37 +0200 |
| Message-ID | <ppn5kf$clr$1@dont-email.me> |
| In reply to | #136593 |
On 11/10/18 00:28, Bart wrote: > On 10/10/2018 23:22, Ian Collins wrote: >> On 11/10/18 11:16, Bart wrote: >>> On 10/10/2018 23:01, Ian Collins wrote: >>>> On 11/10/18 10:54, Bart wrote: >>> >>>>> And compiling from scratch is preferable to incremental compile >>>>> which is >>>>> subject to conditions. So new ways of streamlining the process will >>>>> always be welcome. >>>> >>>> No, compiling from scratch isn't preferable to incremental builds. >>> >>> Even if both took the same amount of time? (Or both were fast enough to >>> be insignificant.) >> >> That ain't going to happen for big projects. The C & C++ part of our >> project (which isn't too big) contains over 2500 source files. > > Would 2500 object files need to be accessed in the course of even an > incremental build? Or is it clever enough to patch in only the ones that > have changed? Do you know what the term "incremental build" means? A typical setup has a set of rules to say that, for example, file "a.o" depends on "a.c", "a.h", "b.h" and "c.h", and "b.o" depends on "b.c", "a.h" and "b.h". So if you change file "a.c" and rebuild, only "a.o" is generated - "b.o" does not need to be touched. But if you change file "a.h", then both "a.o" and "b.o" get regenerated. So for any incremental build, usually only a small fraction (typically just one) of the object files get re-built. If you have incremental linking, then much of the previous link results are also kept - if not, then you may have to read all the object files for the link. > > If all object files do have to be loaded, then that is not so different > from having to load source files. > You load the object files for linking. You load source files for two purposes - one is to automatically generate dependency trees, which is usually quite a fast operation, and the other is to compile them - often the slow part of the process. My biggest projects are a good deal smaller than Ian's in file counts. Some timings for a project with 113 object files are: Serial "clean" build: 51s Parallel "clean" build: 12s Incremental build for one C file changed: 1.8s Does that explain why people use incremental builds? (And I don't care if your compiler can chew through a gazillion lines per second - it won't do what I need, so its speed is irrelevant.) > (And a build system done differently, could eliminate explicit object > files.) > What a great idea - increase build times by a factor of 10 for my project, and perhaps 100 for one the size of Ian's.
[toc] | [prev] | [next] | [standalone]
| From | Bart <bc@freeuk.com> |
|---|---|
| Date | 2018-10-11 11:50 +0100 |
| Message-ID | <RVFvD.15651$I34.1016@fx43.am4> |
| In reply to | #136615 |
On 11/10/2018 10:37, David Brown wrote: > On 11/10/18 00:28, Bart wrote: >> Would 2500 object files need to be accessed in the course of even an >> incremental build? Or is it clever enough to patch in only the ones that >> have changed? > So for any incremental build, usually only a small fraction (typically > just one) of the object files get re-built. > > If you have incremental linking, then much of the previous link results > are also kept - if not, then you may have to read all the object files > for the link. I said 2500 OBJECT files. So that already assumes that only some source files are re-compiled. > Some timings for a project with 113 object files are: > > Serial "clean" build: 51s > Parallel "clean" build: 12s > Incremental build for one C file changed: 1.8s > > Does that explain why people use incremental builds? Incremental builds and multi-processing. It's what you have to throw at a problem when the basic methods are slow and inefficient. What I'm investigating (as are many other people, to try and solve some of the problems highlighted in the article in the OP) is how much it is possible to reimplement those basic methods. Look at the 'gold' linker, especially created from scratch to be more efficient than 'ld'. Someone else who was dissatisfied with the status quo and wasn't content just to buy a faster machine with more cores. Why wasn't incremental linking enough of a solution for them? > (And I don't care if your compiler can chew through a gazillion lines > per second - it won't do what I need, so its speed is irrelevant.) You wouldn't. But it does what /I/ need. And it shows what /can/ be done. (And there is still a lot more it can do, but it's already too fast to accurately measure for my size of projects, where are other normally insignificant overheads start to dominate.) >> (And a build system done differently, could eliminate explicit object >> files.) >> > > What a great idea - increase build times by a factor of 10 for my > project, and perhaps 100 for one the size of Ian's. Well, I've eliminated object files for most purposes. My build times have improved. (Actually, I've eliminated them because they simply weren't needed. They are replaced with .asm files.) Perhaps super-fast, super-efficient compiler tools are not going to provide sub-one-second 'clean build' turnaround times for the some of the giant projects presented here. But imagine if the underlying tools were even one magnitude faster? Your parallel clean build would be faster than your incremental build, and it wouldn't matter which files were modified. It would always be consistently that speed. -- bart
[toc] | [prev] | [next] | [standalone]
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2018-10-11 14:10 +0200 |
| Message-ID | <ppneji$ke$1@dont-email.me> |
| In reply to | #136624 |
On 11/10/18 12:50, Bart wrote: > On 11/10/2018 10:37, David Brown wrote: >> On 11/10/18 00:28, Bart wrote: > >>> Would 2500 object files need to be accessed in the course of even an >>> incremental build? Or is it clever enough to patch in only the ones that >>> have changed? > >> So for any incremental build, usually only a small fraction (typically >> just one) of the object files get re-built. >> >> If you have incremental linking, then much of the previous link results >> are also kept - if not, then you may have to read all the object files >> for the link. > > I said 2500 OBJECT files. Yes, I know you said that. > So that already assumes that only some source > files are re-compiled. No, it does not imply this. But that does not matter to my point. It is not clear what you mean by "would 2500 object files need to be accessed" - accessed in what way? I took that to mean "read" - if you had meant "written" or "generated", I think you would have said that. The only reason you read object files in a typical build process is for the linking. It is common for the link to use all the object files in one link command. But if you have a bigger project, you might use incremental linking - then you don't need to read all the object files for each link. (Link-time optimisation is also possible, but more complex.) > >> Some timings for a project with 113 object files are: >> >> Serial "clean" build: 51s >> Parallel "clean" build: 12s >> Incremental build for one C file changed: 1.8s >> >> Does that explain why people use incremental builds? > > Incremental builds and multi-processing. It's what you have to throw at > a problem when the basic methods are slow and inefficient. > No, it is what you use to make everything go much faster. It does not matter how fast your compilations are - with big enough projects, "compile everything, one file at a time" will be slow in the end. If your super compiler takes a mere 50 ms to compile a file, then with 2500 files it takes 2 minutes. A parallel clean build on an 8-core PC will take 15 seconds. An incremental build will take 50 ms. Serial build-everything-everytime builds are fine for toy projects - but not for more serious work, /regardless/ of the compiler speed. > What I'm investigating (as are many other people, to try and solve some > of the problems highlighted in the article in the OP) is how much it is > possible to reimplement those basic methods. Enjoy your hobby experiments. Just don't imagine they are relevant to people working on more serious development. In C++, compiler speed is more of an issue due to the trend towards big header-only template libraries. So there is a plan there for "modules" to avoid reading source header files in compilation, and instead read pre-chewed binary module files for imports. > > Look at the 'gold' linker, especially created from scratch to be more > efficient than 'ld'. Someone else who was dissatisfied with the status > quo and wasn't content just to buy a faster machine with more cores. > Look at the "clang/llvm" compiler, created from scratch and intending to be faster than other compilers. And it /was/ faster than gcc to begin with. But by the time it reached a similar level of features to gcc, it has a similar run-time. It turns out that a good compiler with strong optimisations, lots of features, powerful static analysis and warnings, and flexible support for targets simply takes a while to run. That does not mean there isn't scope for improvement in compile times - merely that the gains are not huge, even when you start from scratch. (And unlike for a linker, making a real compiler from scratch is a huge undertaking.) > Why wasn't incremental linking enough of a solution for them? "gold" was not created to be more efficient than "ld" - it was created to add features that "ld" did not have (such as plugins, and link-time optimisation support). The speed increase was a byproduct of removing support for older formats, targets, etc., that allowed much of it to be simpler. And whether you use "ld" or "gold", incremental linking can speed up big links. > >> (And I don't care if your compiler can chew through a gazillion lines >> per second - it won't do what I need, so its speed is irrelevant.) > > You wouldn't. But it does what /I/ need. And it shows what /can/ be > done. I am glad that your compiler does what /you/ need - and I am in no way disparaging the work you have done or the results you have achieved. But it does not compare to a top-class compiler, and it does not do what I need or want from a compiler. It is thus utterly irrelevant to /me/. It is like you telling me you have built a go-kart that can get you to your work and the shops, and is vastly more fuel efficient than my big, lumpy normal car. But I want to drive on the motorway - it is irrelevant how efficient your go-kart is at going to the local shops, because it will not do the job I need. And no, your go-kart says /nothing/ about "what /can/ be done" - because it gives no clue as to how fuel efficient a vehicle could be when it meets the requirements of other drivers. > (And there is still a lot more it can do, but it's already too > fast to accurately measure for my size of projects, where are other > normally insignificant overheads start to dominate.) > >>> (And a build system done differently, could eliminate explicit object >>> files.) >>> >> >> What a great idea - increase build times by a factor of 10 for my >> project, and perhaps 100 for one the size of Ian's. > > Well, I've eliminated object files for most purposes. My build times > have improved. (Actually, I've eliminated them because they simply > weren't needed. They are replaced with .asm files.) > Again, great for your little programs - useless for bigger ones. > Perhaps super-fast, super-efficient compiler tools are not going to > provide sub-one-second 'clean build' turnaround times for the some of > the giant projects presented here. > > But imagine if the underlying tools were even one magnitude faster? Your > parallel clean build would be faster than your incremental build, and it > wouldn't matter which files were modified. It would always be > consistently that speed. > If the underlying compiler (and linker) was one magnitude faster, it would make the parallel clean build as fast as the current incremental build. So what? I would still use incremental builds - and that would then be faster still. And when the next project has twice as many files, the incremental build is still the same time - because it generally only compiles one file. The parallel clean build now takes twice as long, however. Let me stress that I appreciate the aim of having faster compilation - that is always a good thing in itself. It is not a major aim, however, and I don't put it ahead of features of the compiler. There is no point in /not/ having a parallel incremental build system - clean builds offer no advantages, and however fast your compiler is, incremental builds offer additional speedups that scale well with size.
[toc] | [prev] | [next] | [standalone]
| From | Malcolm McLean <malcolm.arthur.mclean@gmail.com> |
|---|---|
| Date | 2018-10-11 05:24 -0700 |
| Message-ID | <f15ed3e1-8466-47c8-83e0-28712233df68@googlegroups.com> |
| In reply to | #136630 |
On Thursday, October 11, 2018 at 1:10:38 PM UTC+1, David Brown wrote: > > There is no point in /not/ having a parallel incremental build system - > clean builds offer no advantages, and however fast your compiler is, > incremental builds offer additional speedups that scale well with size. > If you can compile simply with gcc *.c then it's a blessing. Not practical for big programs. But not every program needs to be big, especially if you write it in C. Baby X is a workable GUI, and it's about 40 fairly short .c files for the core - not that much really.
[toc] | [prev] | [next] | [standalone]
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2018-10-11 15:12 +0200 |
| Message-ID | <ppni7m$mdi$1@dont-email.me> |
| In reply to | #136631 |
On 11/10/18 14:24, Malcolm McLean wrote: > On Thursday, October 11, 2018 at 1:10:38 PM UTC+1, David Brown wrote: >> >> There is no point in /not/ having a parallel incremental build system - >> clean builds offer no advantages, and however fast your compiler is, >> incremental builds offer additional speedups that scale well with size. >> > If you can compile simply with gcc *.c then it's a blessing. Not practical > for big programs. But not every program needs to be big, especially if > you write it in C. Baby X is a workable GUI, and it's about 40 fairly > short .c files for the core - not that much really. > Not every program needs to be big, that is true. If you compile with "gcc *.c" as part of your development process, you don't understand how to use development tools. It is fine to say you want the code to work that way when you are finished - if you release source code for others to use, you want it to be flexible. But this is the /development/ process we are talking about now, when compile time speed and incremental builds are relevant. Using a development tool without understanding it, like using "gcc" without any flags, is like buying an electric screwdriver to screw in some nails - and then hammering on the nails with the unopened box the screwdrivers comes in.
[toc] | [prev] | [next] | [standalone]
| From | scott@slp53.sl.home (Scott Lurndal) |
|---|---|
| Date | 2018-10-11 13:13 +0000 |
| Message-ID | <I%HvD.102471$9O2.64504@fx03.iad> |
| In reply to | #136630 |
David Brown <david.brown@hesbynett.no> writes: >On 11/10/18 12:50, Bart wrote: >> I said 2500 OBJECT files. > >Yes, I know you said that. > >> So that already assumes that only some source >> files are re-compiled. > >No, it does not imply this. But that does not matter to my point. > >It is not clear what you mean by "would 2500 object files need to be >accessed" - accessed in what way? I took that to mean "read" - if you >had meant "written" or "generated", I think you would have said that. >The only reason you read object files in a typical build process is for >the linking. > >It is common for the link to use all the object files in one link >command. But if you have a bigger project, you might use incremental >linking - then you don't need to read all the object files for each >link. (Link-time optimisation is also possible, but more complex.) In our project, we bundle the objects for each subdirectory into an archive library, and link against that. So one open per subdir by the linker rather than opening a thousand individual object files; not that it likely makes much difference vis-a-vis link speed, unless, as is the case here, one is working over NFS.
[toc] | [prev] | [next] | [standalone]
| From | Bart <bc@freeuk.com> |
|---|---|
| Date | 2018-10-11 14:44 +0100 |
| Message-ID | <7tIvD.18127$RM7.167@fx24.am4> |
| In reply to | #136630 |
On 11/10/2018 13:10, David Brown wrote:
> On 11/10/18 12:50, Bart wrote:
>> Look at the 'gold' linker, especially created from scratch to be more
>> efficient than 'ld'. Someone else who was dissatisfied with the status
>> quo and wasn't content just to buy a faster machine with more cores.
>>
>
> Look at the "clang/llvm" compiler, created from scratch and intending to
> be faster than other compilers.
Not with llvm, which they will have known from the start would be a
huge, cumbersome dependency.
> But by the time it reached a similar level of features to gcc, it
> has a similar run-time. It turns out that a good compiler with strong
> optimisations, lots of features, powerful static analysis and warnings,
> and flexible support for targets simply takes a while to run.
There are workarounds. For example, you don't always need all that
analysis and all the optimisation, you just want to make a small change,
that might affect the whole program, and try it to see what happens.
To that end, the build system might simply employ Tiny C to do a quick
and cheerful version suitable for building dozens of times in a short
time, and then only use the big guns when all those other 'requirements'
kick in.
>> You wouldn't. But it does what /I/ need. And it shows what /can/ be
>> done.
>
> I am glad that your compiler does what /you/ need - and I am in no way
> disparaging the work you have done or the results you have achieved.
> But it does not compare to a top-class compiler, and it does not do what
> I need or want from a compiler. It is thus utterly irrelevant to /me/.
Is what you want actually that typical?
My needs must be fairly atypical then. I just want to make a change to
some source code, then create an executable so I can test those changes.
How it does that source -> executable step is of little interest, other
than it shouldn't get in the way. I certainly don't want to hang around
while it does it.
Is that really that unusual?
Take a program 'pc' that I want to try some changes on: 34 modules
comprising 45Kloc, another 10Kloc in 16 source files, included as data.
This is not C, but rebuilding from scratch is like this (using the
command line to demonstrate, usually it's done from an IDE):
C:\qq>timeit "mm pc"
MM Compiling pc.m to pc.exe
mm pc 0.24
That's straight to x64, so let's try generating intermediate C, here
done as two steps:
C:\qq>timeit "mm -c64 pc"
MM Compiling pc.m to pc.c
mm -c64 pc 0.14
C:\qq>timeit "bcc -exe pc"
BCC Compiling pc.c to pc.exe
bcc -exe pc 0.22
Total 0.36, but there is intermediate C source as well as intermediate
ASM. But let's try that last step with gcc (warnings have been left out):
C:\qq>timeit "gcc -opc.exe pc.c"
gcc -opc.exe pc.c 3.80
Total now 3.94 seconds.
So successive solutions have taken 0.24 seconds or 0.36 seconds with
mine, and 3.84 seconds with gcc, which is 10-15 times slower.
(And this is not even optimised. With -O2, it takes 11 seconds. -O3, 15
seconds. There is no option here of incremental compiling or linking,
since the C intermediate is a single file.)
So, which solution should I go for: the 0.25 seconds, 0.36 seconds, or 4
seconds? Bear in the 4-second build runs 30% slower than the 0.25-second
one. Very difficult choice...
>
> It is like you telling me you have built a go-kart that can get you to
> your work and the shops, and is vastly more fuel efficient than my big,
> lumpy normal car.
No, I'd want to drive your big car to work and to the shops. So do many
people. But according to the article in the OP, we are all gradually
being forced to drive 40-ton trucks everywhere, and very complicated
ones that no single person properly understands, with no choice.
>> Well, I've eliminated object files for most purposes. My build times
>> have improved. (Actually, I've eliminated them because they simply
>> weren't needed. They are replaced with .asm files.)
>>
>
> Again, great for your little programs - useless for bigger ones.
Why? My assembler can assemble /and/ link at millions of lines per second.
SQLITE3 is a decent-sized library, and even with my poor code
generation, it is only 240K lines of ASM. How big are we talking about?
>> Perhaps super-fast, super-efficient compiler tools are not going to
>> provide sub-one-second 'clean build' turnaround times for the some of
>> the giant projects presented here.
>>
>> But imagine if the underlying tools were even one magnitude faster? Your
>> parallel clean build would be faster than your incremental build, and it
>> wouldn't matter which files were modified. It would always be
>> consistently that speed.
>>
>
> If the underlying compiler (and linker) was one magnitude faster, it
> would make the parallel clean build as fast as the current incremental
> build. So what? I would still use incremental builds - and that would
> then be faster still.
But then you don't need to bother. You can use a simpler, more reliable
build system. Perhaps even Malcolm's 'gcc *.c'.
> And when the next project has twice as many files, the incremental build
> is still the same time - because it generally only compiles one file.
I don't believe this about only ever having to compile one file. Suppose
you make a change to a header that is included everywhere else?
Nor about the next project always being bigger. Suppose it's smaller?
(And anyway there are ways of handling scalability of projects; I
haven't needed do address that because mine are smallish.)
--
bart
[toc] | [prev] | [next] | [standalone]
| From | David Brown <david.brown@hesbynett.no> |
|---|---|
| Date | 2018-10-11 16:38 +0200 |
| Message-ID | <ppnn96$m7e$1@dont-email.me> |
| In reply to | #136637 |
On 11/10/18 15:44, Bart wrote: > On 11/10/2018 13:10, David Brown wrote: >> On 11/10/18 12:50, Bart wrote: > >>> Look at the 'gold' linker, especially created from scratch to be more >>> efficient than 'ld'. Someone else who was dissatisfied with the status >>> quo and wasn't content just to buy a faster machine with more cores. >>> >> >> Look at the "clang/llvm" compiler, created from scratch and intending to >> be faster than other compilers. > > Not with llvm, which they will have known from the start would be a > huge, cumbersome dependency. > Let me repeat myself. One of the intentions with the whole llvm system, and the clang front end, was to make a compiler that was significantly faster than gcc while being as powerful. It failed there - it is not hugely faster. (It has been a very successful and useful project in other respects.) >> But by the time it reached a similar level of features to gcc, it >> has a similar run-time. It turns out that a good compiler with strong >> optimisations, lots of features, powerful static analysis and warnings, >> and flexible support for targets simply takes a while to run. > > There are workarounds. For example, you don't always need all that > analysis and all the optimisation, you just want to make a small change, > that might affect the whole program, and try it to see what happens. > > To that end, the build system might simply employ Tiny C to do a quick > and cheerful version suitable for building dozens of times in a short > time, and then only use the big guns when all those other 'requirements' > kick in. You are joking, right? Tiny C is an interesting little project - it is not something you would use for serious development work. And, in case you hadn't guessed, it is useless to me for the work I do. > >>> You wouldn't. But it does what /I/ need. And it shows what /can/ be >>> done. >> >> I am glad that your compiler does what /you/ need - and I am in no way >> disparaging the work you have done or the results you have achieved. >> But it does not compare to a top-class compiler, and it does not do what >> I need or want from a compiler. It is thus utterly irrelevant to /me/. > > Is what you want actually that typical? > I can't comment on that. Let those that think your compiler would be good enough for them, stand up. > My needs must be fairly atypical then. I just want to make a change to > some source code, then create an executable so I can test those changes. > Your compiler won't compile my source code. It will not work on my targets. It will not support the features I use in the language. It will not support the compiler-specific code (at least some of which will work, through conditional compilation, with several major compilers for the target). It will not do the quality control or static analysis I need. It will not generate the additional output files I need, such as stack usage analysis. It will not support the language versions I need. It will not support the optimisations I need - in my world, too big or too slow is a failure. It will not give the debugging options I need. It will not give the host support I need. It will not give the maintenance and future roadmap I need. It will not give the support I need. I am a /professional/. My code needs to be high quality, and my company needs to be able to stand behind the quality of their development process and the code written. I can't tell my customers "I ran this through a compiler written by a guy off the internet. I'm not sure if the results are right - the guy doesn't really understand C and refuses to look at the standards documents - but the build times were several seconds faster than my old tools". > How it does that source -> executable step is of little interest, other > than it shouldn't get in the way. I certainly don't want to hang around > while it does it. > > Is that really that unusual? > ... > So, which solution should I go for: the 0.25 seconds, 0.36 seconds, or 4 > seconds? Bear in the 4-second build runs 30% slower than the 0.25-second > one. Very difficult choice... Your first selection criteria is for features that give you correct code, encourage readable and maintainable code, and help identify errors as early as possible in the process. Your second selection criteria is for efficient generated code. If everything else is good enough, /then/ you can consider build times as a third level of criteria. If your tools are good enough you, that's great. They are not good enough for me. > >> >> It is like you telling me you have built a go-kart that can get you to >> your work and the shops, and is vastly more fuel efficient than my big, >> lumpy normal car. > > No, I'd want to drive your big car to work and to the shops. So do many > people. But according to the article in the OP, we are all gradually > being forced to drive 40-ton trucks everywhere, and very complicated > ones that no single person properly understands, with no choice. > I don't care about what the OP's article says. This is a discussion between people in this group - the article may have inspired it, but the author is not in this discussion. I am no fan of unnecessary bloat. But I am quite happy to accept that some programs or systems are big, because most of it is useful to someone even if it is not useful to /me/. This is, I think, missing from your perspective on software - you are only concerned about what matters to /you/. That's okay when you are writing your own compiler that no one but you will ever use - it's a different matter when writing tools to be used by lots of people. >>> Well, I've eliminated object files for most purposes. My build times >>> have improved. (Actually, I've eliminated them because they simply >>> weren't needed. They are replaced with .asm files.) >>> >> >> Again, great for your little programs - useless for bigger ones. > > Why? My assembler can assemble /and/ link at millions of lines per second. > As always, no one else cares about your personal tools. It is a strange idea to use an assembly file as the intermediary format, rather than object files. I fully agree that assembling is a quick part of the process - but it is still faster for a linker to use object files than to use assembly files. > SQLITE3 is a decent-sized library, and even with my poor code > generation, it is only 240K lines of ASM. How big are we talking about? > > >>> Perhaps super-fast, super-efficient compiler tools are not going to >>> provide sub-one-second 'clean build' turnaround times for the some of >>> the giant projects presented here. >>> >>> But imagine if the underlying tools were even one magnitude faster? Your >>> parallel clean build would be faster than your incremental build, and it >>> wouldn't matter which files were modified. It would always be >>> consistently that speed. >>> >> >> If the underlying compiler (and linker) was one magnitude faster, it >> would make the parallel clean build as fast as the current incremental >> build. So what? I would still use incremental builds - and that would >> then be faster still. > > But then you don't need to bother. You can use a simpler, more reliable > build system. Perhaps even Malcolm's 'gcc *.c'. > My build system is to type "make". It is quite simple. It is vastly more simple than to include the compiler options I need - plus all the other programs involved in building. (Build processes are often more involved than just compile+link.) The makefile, of course, contains a lot of information - but it is flexible, re-usable, and highly reliable. If you don't want to write makefiles, there are alternative build systems that may be easier to learn - there is no doubt that the makefile language is a little quirky. Using a proper build system is not a bother. I have used it for decades, including for assembly and occasional other languages. > >> And when the next project has twice as many files, the incremental build >> is still the same time - because it generally only compiles one file. > > I don't believe this about only ever having to compile one file. Suppose > you make a change to a header that is included everywhere else? > As I explained earlier, if you change a header you will have to re-compile the files that include it (directly or indirectly). And of course if you change several C files between builds, they all need re-compiled. But if you have a regular re-build habit (and it's just "ctrl-B" in an IDE), most of your builds will compile just one file. > Nor about the next project always being bigger. Suppose it's smaller? > Of course it might be smaller. The average size tends towards bigger, however. > (And anyway there are ways of handling scalability of projects; I > haven't needed do address that because mine are smallish.) > Yes - and if you need to work on bigger projects, you'll find that every scalability solution will involve incremental parallel builds.
[toc] | [prev] | [next] | [standalone]
| From | mark.bluemel@gmail.com |
|---|---|
| Date | 2018-10-11 08:09 -0700 |
| Message-ID | <50c51f36-76c0-47ea-80cd-69800abbf58e@googlegroups.com> |
| In reply to | #136641 |
On Thursday, 11 October 2018 15:38:43 UTC+1, David Brown wrote: > ... I can't tell my customers "I ran this > through a compiler written by a guy off the internet. I'm not sure if > the results are right - the guy doesn't really understand C and refuses > to look at the standards documents - but the build times were several > seconds faster than my old tools". I can't decide whether this should be sig or an xkcd strip.
[toc] | [prev] | [next] | [standalone]
| From | Bart <bc@freeuk.com> |
|---|---|
| Date | 2018-10-11 17:43 +0100 |
| Message-ID | <I4LvD.16148$xE.7647@fx22.am4> |
| In reply to | #136641 |
On 11/10/2018 15:38, David Brown wrote: > On 11/10/18 15:44, Bart wrote: > You are joking, right? > > Tiny C is an interesting little project - it is not something you would > use for serious development work. So what /is/ it used for? It obviously can't be used to compile C99 source programs into executable files, because David Brown said so. > I can't comment on that. Let those that think your compiler would be > good enough for them, stand up. No, my C compiler is a toy, for someone who wants it as a plug-in for their gcc for for their specific OS and for their specific target. My other ones, not so much, or if the C compiler used for new projects on its own terms. > Your compiler won't compile my source code. It will not work on my > targets. It will not support the features I use in the language. It > will not support the compiler-specific code (at least some of which will > work, through conditional compilation, with several major compilers for > the target). It will not do the quality control or static analysis I > need. It will not generate the additional output files I need, such as > stack usage analysis. It will not support the language versions I need. > It will not support the optimisations I need - in my world, too big or > too slow is a failure. It will not give the debugging options I need. > It will not give the host support I need. It will not give the > maintenance and future roadmap I need. It will not give the support I need. It sounds like you don't really want a compiler, but a bunch of other stuff. > I am a /professional/. My code needs to be high quality, and my company > needs to be able to stand behind the quality of their development > process and the code written. I can't tell my customers "I ran this > through a compiler written by a guy off the internet. Because nobody has ever used a compiler downloaded off the internet. > I'm not sure if > the results are right - the guy doesn't really understand C and refuses > to look at the standards documents - but the build times were several > seconds faster than my old tools". gcc got it right first time of course. Nearly 200 new releases over 30 years were just minor tweaks. > If everything else is good enough, /then/ you can consider build times > as a third level of criteria. If I have to wait even one minute for each new build, then for me it would be useless for development. > It is a strange idea to use an assembly file as the intermediary format, > rather than object files. I fully agree that assembling is a quick part > of the process - but it is still faster for a linker to use object files > than to use assembly files. But they have to generated first. If an object is generated (sometimes, even with gcc, from intermediate assembly) only to immediately linked, then the object serves little purpose other than as an intermediate format between programs. If compiler can translate multiple source files into an executable in one invocation, then that data can stay in memory and doesn't need a discrete form on disk. >> But then you don't need to bother. You can use a simpler, more reliable >> build system. Perhaps even Malcolm's 'gcc *.c'. >> > > My build system is to type "make". It is quite simple. Typing 'make' might be simple, but the makefile it uses might be anything but. BTW how does 'make' know which project to build, if there are several possibilities? -- bart
[toc] | [prev] | [next] | [standalone]
Page 3 of 8 — ← Prev page 1 2 [3] 4 5 6 7 8 Next page →
Back to top | Article view | comp.lang.c
csiph-web