Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #136571

Re: software disenchantment

From "Rick C. Hodgin" <rick.c.hodgin@gmail.com>
Newsgroups comp.lang.c
Subject Re: software disenchantment
Date 2018-10-10 15:53 -0400
Organization Liberty Software Foundation
Message-ID <ppllbg$bkr$1@dont-email.me> (permalink)
References (6 earlier) <7FkvD.14919$Nt6.2825@fx19.am4> <968306fe-f05b-45cb-93a4-3081ec0d2fd0@googlegroups.com> <ppldrc$n44$1@news.albasani.net> <5c209401-54a3-410d-85fd-a5bc50ba93ca@googlegroups.com> <ppljgb$8jc$1@news.albasani.net>

Show all headers | View raw

On 10/10/2018 3:22 PM, BGB wrote:
> On 10/10/2018 1:04 PM, Thiago Adams wrote:
>> The solution is pre-compiled headers. We still have the "source"
>> and the pre-compiled headers can optimize the parser.

Pre-compiled headers don't always get you there.  You can have
pre-parsed headers, but even then if a #define comes into play
which re-defines something referenced somewhere, your pre-
compiled header will be invalid and may require a full repars-
ing anyway.

The ability to have random #define values, and conditional
compilation makes it unfeasible outside of a certain type of
system.

> The header source would still exist in such a database, just it wouldn't be 
> used directly by the "#include" mechanism when including a "header" which 
> exists within this database.
> 
> In a previous attempt, I had split up ifdef's into two different groups 
> (depending on the define): those which were still handled during 
> preprocessing, and those which were "delayed" until a later stage in the 
> compiler (effectively, they were translated to a different syntax). All of 
> the relevant branches would then be compiled just with attributes encoding 
> which defines a given declaration or code-block depends on, ... With N/A 
> contents being pruned away in the compiler's backend).
> 
> The problem was that this imposed constraints on how this feature could be 
> used (*) and still didn't adequately address some issues I had initially set 
> out to "solve" (like a single binary working with different C library 
> implementations or OS APIs; at best case the C library would need a wrapper 
> or for the compiler to provide its own C library and similar).
> 
> 
> *: For example, everything within the block had to be "syntactically valid" 
> on its own, couldn't be used for macro expansion, required types to exist 
> regardless of whether or not the block was used, ...
> 
> This latter limitation was a big mess for, say, having sections of code 
> specific to Win32 or the POSIX APIs, since these would depend on 
> headers/types/... which may or may not exist depending on the target. This 
> limitation effectively ruined its usability for one of its main potential 
> use-cases (since everything for every target will need to exist during 
> building, which would mean building with both Windows and POSIX headers at 
> the same time; neither of which exist in forms to be aware of or deal 
> gracefully with the quirks of this mechanism; as we don't want Win32 stuff 
> polluting a POSIX target or the reverse).
> 
> The result would be needing effectively an abstract wrapper library over the 
> OS specific APIs, basically defeating the whole point.
> 
> Not long after, I more or less gave up on this idea.

The solution is to tokenize every word in every #include file used
in the system, and store it all in a B+tree type index for quick
access by name.  As something changes, once you have the name, you
reference the lines which relate to the token and see if the change
impacts compilation.

Then, each thing that is changed spawns its own separate child lookup
to look for more that needs compilation, and it repeats until it
exhausts itself.  And if you get beyond some number of N things that
need to be re-compiled, it may be quicker to stop the examination and
do a full recompile, so that has to have a threshold there, examined
from prior compile times, etc.

This all exists as an identifying algorithm to determine what needs
to be re-compiled based on what's changed in the source code.  Some
things impact entire functions, such as adding new local variables,
which will change the stack alignment, and may impact the stack history
in the running ABI if some recursion is in play, etc.

It's all very logical, but it's all very meticulous, and very memory
intensive.  It would work for projects that are a few 100K lines of
code max, with minimal output files (DLLs and EXE files).  Beyond
that, you'd be better off to do full compiles and a more traditional
software development cycle.

I do not plan to ever have projects more than a few hundred thousand
lines of code each.

> Some more narrowly defined conditions can be addressed via more constrained 
> mechanisms; such as using 'unsigned long' for "size_t" which can 
> automagically vary to be the correct size for using the same IL code for 32 
> and 64-bit builds (with a common C library impl), mostly because things like 
> struct layout and similar aren't fixed until after the 3AC is generated.
> 
> Similarly, differences in the ASM modules (between 32 and 64 bit ISA variants 
> or for dealing with optional ISA features; or entirely separate ISA's if need 
> be) can be handled by using a secondary ".ifdef" mechanism which was handled 
> much later (during ASM parsing; by which stage these specifics are known), vs 
> during the normal preprocessing stage (which happens prior to producing the IL).
> 
> So, in a much more limited form of the original idea still exists for the 
> assembler.

For me, the goals are always to look at things from the developer's
point of view.  What would he/she need tool-wise to make their job
easier.  And even if it's harder for me to get it setup for them, I
am one person (or we would be one team) working on the compiler end,
and there may be thousands of developers who later use this tool.

The extra investment of thought and labor results in a desired end-
result of a feature-rich tool more people can use to be productive.
It will save them time, which when multiplied out and compared to
my extra time, will make mine seem small by comparison.

-- 
Rick C. Hodgin

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

software disenchantment Bart <bc@freeuk.com> - 2018-10-06 15:13 +0100
  Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-06 09:54 -0700
  Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-06 10:26 -0700
    Re: software disenchantment fir <profesor.fir@gmail.com> - 2018-10-14 04:26 -0700
  Re: software disenchantment antispam@math.uni.wroc.pl - 2018-10-09 18:20 +0000
    Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-09 11:54 -0700
      Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-09 21:22 +0100
        Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-09 17:47 -0700
          Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 11:28 +0100
            Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 11:38 +0100
              Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-10 05:49 -0700
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-10 15:47 +0200
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 09:58 -0400
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-10 14:39 +0000
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-10 09:42 -0700
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 13:18 -0400
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 20:31 +0100
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 17:53 -0500
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 10:06 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 02:34 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 12:03 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 03:59 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 16:56 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 09:02 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 19:39 +0200
                Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-15 19:53 +0000
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-16 03:54 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 20:25 +0200
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 13:16 +0000
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 07:15 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 17:05 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 09:15 -0700
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 17:33 +0000
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 19:53 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 17:59 +0100
                Re: software disenchantment nobody@example.org (Scott) - 2018-10-10 16:53 +0000
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 12:45 -0500
                Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-10 11:04 -0700
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 14:22 -0500
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 15:53 -0400
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 09:06 +1300
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-10 13:21 -0700
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-10 20:27 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 22:54 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:01 +1300
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:05 +1300
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 23:16 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:22 +1300
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-10 23:28 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 11:33 +1300
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 11:37 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 11:50 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 14:10 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 05:24 -0700
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 15:12 +0200
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 13:13 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 14:44 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 16:38 +0200
                Re: software disenchantment mark.bluemel@gmail.com - 2018-10-11 08:09 -0700
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 17:43 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 17:36 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 18:56 +0100
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-11 11:54 -0700
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 20:09 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 13:10 +0000
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-13 12:57 +0200
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 14:27 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 22:03 +0200
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 15:56 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 11:30 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 23:34 +0100
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-11 16:24 -0700
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 01:46 +0100
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-11 18:58 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-12 08:50 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 11:12 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 12:55 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 12:10 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 13:29 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 13:06 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 14:30 +0200
                Re: software disenchantment James Kuyper <jameskuyper@alumni.caltech.edu> - 2018-10-12 08:53 -0400
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 15:11 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 13:55 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 15:13 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 16:10 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 15:22 +0000
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-12 09:12 -0700
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-13 13:00 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-13 12:52 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-14 08:44 +1300
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-13 21:03 +0100
                Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-14 10:13 +1300
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-13 23:02 +0100
                Re: software disenchantment Keith Thompson <kst-u@mib.org> - 2018-10-13 16:47 -0700
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-14 01:05 +0100
                Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-13 18:12 -0700
                Re: software disenchantment Ben Bacarisse <ben.usenet@bsb.me.uk> - 2018-10-14 03:05 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-14 20:14 +0200
                Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-14 11:25 -0700
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 13:17 +0000
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 13:16 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 14:36 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 14:25 +0000
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-12 15:55 +0100
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 15:22 +0000
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-12 08:26 -0700
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-12 15:51 +0000
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-13 13:06 +0200
                Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-15 19:30 +0000
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 12:39 +0200
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-12 08:44 +0200
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 13:01 +0200
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 20:41 +0200
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 20:02 +0100
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 22:13 +0200
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 13:03 -0500
                Re: software disenchantment "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2018-10-11 11:33 -0700
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 21:04 +0200
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-11 20:15 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-12 13:17 +0200
                Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 13:10 +0000
                Re: software disenchantment BGB <cr88192@hotmail.com> - 2018-10-10 19:22 -0500
                Re: software disenchantment David Brown <david.brown@hesbynett.no> - 2018-10-11 10:58 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 02:37 -0700
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 12:06 +0200
                Re: software disenchantment Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2018-10-11 04:00 -0700
                Re: software disenchantment Cecil Bayona <cbayona@cbayona.com> - 2018-10-11 13:03 -0500
                Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 19:36 +0100
                Re: software disenchantment Cecil Bayona <cbayona@cbayona.com> - 2018-10-11 15:37 -0500
                Re: software disenchantment Thiago Adams <thiago.adams@gmail.com> - 2018-10-10 14:01 -0700
        Re: software disenchantment Ian Collins <ian-news@hotmail.com> - 2018-10-11 09:02 +1300
      Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-10 19:11 +0000
        Re: software disenchantment David Kleinecke <dkleinecke@gmail.com> - 2018-10-10 13:08 -0700
          Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-11 14:52 +0000
            Re: software disenchantment scott@slp53.sl.home (Scott Lurndal) - 2018-10-11 14:58 +0000
            Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-11 17:19 +0200
              Re: software disenchantment Dan Purgert <dan@djph.net> - 2018-10-12 09:49 +0000
                Re: software disenchantment Wouter Verhelst <w@uter.be> - 2018-10-13 09:30 +0200
                Re: software disenchantment Jorgen Grahn <grahn+nntp@snipabacken.se> - 2018-10-15 08:53 +0000
                Re: software disenchantment Dan Purgert <dan@djph.net> - 2018-10-15 09:44 +0000
    Re: software disenchantment Jens Stuckelberger <Jens_Stuckelberger@nowhere.net> - 2018-10-10 13:36 +0000
  Re: software disenchantment "Chris M. Thomasson" <invalid_chris_thomasson@invalid.invalid> - 2018-10-10 13:51 -0700
    Re: software disenchantment nobody@example.org (Scott) - 2018-10-11 01:53 +0000
    Re: software disenchantment Dan Purgert <dan@djph.net> - 2018-10-11 10:20 +0000
    Re: software disenchantment Bart <bc@freeuk.com> - 2018-10-11 12:53 +0100
      Re: software disenchantment "Chris M. Thomasson" <invalid_chris_thomasson@invalid.invalid> - 2018-10-11 23:18 -0700

csiph-web