Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #106473 > unrolled thread

Linker question

Back to article view | Back to comp.lang.c

Contents

  Linker question silusilusilu@gmail.com - 2017-03-19 04:42 -0700
    Re: Linker question Malcolm McLean <malcolm.mclean5@btinternet.com> - 2017-03-19 05:10 -0700
    Re: Linker question Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-03-19 12:59 +0000
      Re: Linker question silusilusilu@gmail.com - 2017-03-20 04:06 -0700
        Re: Linker question Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-03-20 13:05 +0000
          Re: Linker question silusilusilu@gmail.com - 2017-03-20 06:24 -0700
            Re: Linker question Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-03-20 13:51 +0000
              Re: Linker question Manfred <noname@invalid.add> - 2017-03-20 16:03 +0100
                Re: Linker question Ben Bacarisse <ben.usenet@bsb.me.uk> - 2017-03-20 17:09 +0000
    Re: Linker question gof@somewhere.invalid (Adam Wysocki) - 2017-03-20 09:59 +0000
      Re: Linker question silusilusilu@gmail.com - 2017-03-20 04:13 -0700
        Re: Linker question gof@somewhere.invalid (Adam Wysocki) - 2017-03-21 09:44 +0000

#106473 — Linker question

Hi All,
not a C programming question but something related: i think this is the right place for my question...correct me if i'm wrong :)
I'm working on a project with several C files: all files are statically linked in an unique binary.
My final goal is create a unique repository with only object files: if i modify one .c file i should not recompile other files if it is not strictly necessary.
This is done by the linker. For example i have three files:

file1.c file2.c file3.c file1.h file2.h file3.h

function_infile1 calls function_infile2; function_infile2 calls function_infile3

As output of my compiler i obtain

file1.o file2.o file3.o

If i change something inside function_infile3 (without modify its function prototype), file2.o remains untouched.
If i modify file3.h inserting for example a new define or a new prototype function i should recompile file2 even if it not uses any of the new functions or defines in file3.h.
Is there a way to avoid it?
Thanks 

[toc] | [next] | [standalone]

#106474

On Sunday, March 19, 2017 at 11:42:32 AM UTC, silusi...@gmail.com wrote:
> Hi All,
> not a C programming question but something related: i think this is the right place for my question...correct me if i'm wrong :)
> I'm working on a project with several C files: all files are statically linked in an unique binary.
> My final goal is create a unique repository with only object files: if i modify one .c file i should not recompile other files if it is not strictly necessary.
> This is done by the linker. For example i have three files:
> 
> file1.c file2.c file3.c file1.h file2.h file3.h
> 
> function_infile1 calls function_infile2; function_infile2 calls function_infile3
> 
> As output of my compiler i obtain
> 
> file1.o file2.o file3.o
> 
> If i change something inside function_infile3 (without modify its function prototype), file2.o remains untouched.
> If i modify file3.h inserting for example a new define or a new prototype function i should recompile file2 even if it not uses any of the new functions or defines in file3.h.
> Is there a way to avoid it?
> Thanks
>
Make was designed for exactly this job.

I hesitate to recommend it as Make has many problems, and is now usually automatically
generated. Look at CMake after you have written a few Makefiles.

[toc] | [prev] | [next] | [standalone]

#106484

silusilusilu@gmail.com writes:


> not a C programming question but something related: i think this is
> the right place for my question...correct me if i'm wrong :)
> I'm working on a project with several C files: all files are
> statically linked in an unique binary.
> My final goal is create a unique repository with only object files: if
> i modify one .c file i should not recompile other files if it is not
> strictly necessary.  This is done by the linker. For example i have
> three files:
>
> file1.c file2.c file3.c file1.h file2.h file3.h
>
> function_infile1 calls function_infile2; function_infile2 calls
> function_infile3

You don't say what file include which others.  The pattern of calls
suggests that

  file1.c includes file2.h
  file2.c includes file3.h

It is that right?

> As output of my compiler i obtain
>
> file1.o file2.o file3.o
>
> If i change something inside function_infile3 (without modify its
> function prototype), file2.o remains untouched.
> If i modify file3.h inserting for example a new define or a new
> prototype function i should recompile file2 even if it not uses any of
> the new functions or defines in file3.h.
>
> Is there a way to avoid it?

Not any simply way.  I think the suggestion to use make has missed your
point.  Make will have to be told that file2.o depends on file3.h
because its information is based solely on what files include which
others.  I don't know of an automatic way to avoid compiling file2.c
simply because it does not use any of the specific changes to one of the
included files.  (Of course, *I* may have missed your point but this is
what I think you want.)

Naturally, you can split file3.h into different parts and only include
into file2.c exactly those parts that it definitively needs, but
complicating the structure to avoid a few redundant re-compilations is
probably a bad idea.  Does it really matter that much?

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#106544

Thanks for your reply

> > not a C programming question but something related: i think this is
> > the right place for my question...correct me if i'm wrong :)
> > I'm working on a project with several C files: all files are
> > statically linked in an unique binary.
> > My final goal is create a unique repository with only object files: if
> > i modify one .c file i should not recompile other files if it is not
> > strictly necessary.  This is done by the linker. For example i have
> > three files:
> >
> > file1.c file2.c file3.c file1.h file2.h file3.h
> >
> > function_infile1 calls function_infile2; function_infile2 calls
> > function_infile3
> 
> You don't say what file include which others.  The pattern of calls
> suggests that
> 
>   file1.c includes file2.h
>   file2.c includes file3.h
> 
> It is that right?

Yes
 
> > As output of my compiler i obtain
> >
> > file1.o file2.o file3.o
> >
> > If i change something inside function_infile3 (without modify its
> > function prototype), file2.o remains untouched.
> > If i modify file3.h inserting for example a new define or a new
> > prototype function i should recompile file2 even if it not uses any of
> > the new functions or defines in file3.h.
> >
> > Is there a way to avoid it?
> 
> Not any simply way.  I think the suggestion to use make has missed your
> point.  Make will have to be told that file2.o depends on file3.h
> because its information is based solely on what files include which
> others.  I don't know of an automatic way to avoid compiling file2.c
> simply because it does not use any of the specific changes to one of the
> included files.  (Of course, *I* may have missed your point but this is
> what I think you want.)

you haven't missed the point: i have the problem you have underlined

> Naturally, you can split file3.h into different parts and only include
> into file2.c exactly those parts that it definitively needs, but
> complicating the structure to avoid a few redundant re-compilations is
> probably a bad idea.  Does it really matter that much?

It is a requirement

[toc] | [prev] | [next] | [standalone]

#106553

silusilusilu@gmail.com writes:
<snip>

>> ...  I think the suggestion to use make has missed your
>> point.  Make will have to be told that file2.o depends on file3.h
>> because its information is based solely on what files include which
>> others.  I don't know of an automatic way to avoid compiling file2.c
>> simply because it does not use any of the specific changes to one of the
>> included files.  (Of course, *I* may have missed your point but this is
>> what I think you want.)
>
> you haven't missed the point: i have the problem you have underlined
>
>> Naturally, you can split file3.h into different parts and only include
>> into file2.c exactly those parts that it definitively needs, but
>> complicating the structure to avoid a few redundant re-compilations is
>> probably a bad idea.  Does it really matter that much?
>
> It is a requirement

Then your simplest solution is might be to break the include files up,
but I am puzzled by why this is a requirement.  If you say a bit more
about why this is important, you might get some better suggestions.
Elsewhere, you comment that one of the source files might not exist.
That suggests that your situation is not the normal one envisioned by
build systems.

You can't always get finer compile granularity just by splitting the
include files up because, sometimes, it comes down to what #defines are
in play.  It's possible that some quite complex use of scons could
accommodate this situation but I am not really familiar with it.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#106555

> > It is a requirement
> 
> Then your simplest solution is might be to break the include files up,
> but I am puzzled by why this is a requirement.  If you say a bit more
> about why this is important, you might get some better suggestions.
> Elsewhere, you comment that one of the source files might not exist.
> That suggests that your situation is not the normal one envisioned by
> build systems.

Currently we have a unique repository with all the source code of our project, however my boss decided to put in the repository only object files for "sensitive" source code.
This possibility should not block other team members to compile the entire build and make their changes on the remaining files. 
I disagree with that choice but for me it is a requirement

> You can't always get finer compile granularity just by splitting the
> include files up because, sometimes, it comes down to what #defines are
> in play.  It's possible that some quite complex use of scons could
> accommodate this situation but I am not really familiar with it.

I don't know how scons can be used to avoid a compilation problem like this one.
(but i don't know scons very well)

[toc] | [prev] | [next] | [standalone]

#106559

silusilusilu@gmail.com writes:

>> > It is a requirement
>> 
>> Then your simplest solution is might be to break the include files up,
>> but I am puzzled by why this is a requirement.  If you say a bit more
>> about why this is important, you might get some better suggestions.
>> Elsewhere, you comment that one of the source files might not exist.
>> That suggests that your situation is not the normal one envisioned by
>> build systems.
>
> Currently we have a unique repository with all the source code of our
> project, however my boss decided to put in the repository only object
> files for "sensitive" source code.
>
> This possibility should not block other team members to compile the
> entire build and make their changes on the remaining files.  I
> disagree with that choice but for me it is a requirement

Ah, so what you need is not so much something that avoids unneeded
recompiles but something that can tell the difference between a change
where the old .o file could safely be used and a change where that would
actually build an incorrect program.

You could live with an unnecessary recompile, provided you could tell
when the error can be safely ignored.  And where you do have all the
source, the occasional extra compile is, again, not really the problem.

>> You can't always get finer compile granularity just by splitting the
>> include files up because, sometimes, it comes down to what #defines are
>> in play.  It's possible that some quite complex use of scons could
>> accommodate this situation but I am not really familiar with it.
>
> I don't know how scons can be used to avoid a compilation problem like
> this one.  (but i don't know scons very well)

SCons appears to be entirely programmable so, one way or another, there
would be a way to implement almost any conceivable policy.  It might
take more effort than persuading your boss that not breaking the build
system is a better option, but it can probably be done.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#106570

On 3/20/2017 2:51 PM, Ben Bacarisse wrote:
> silusilusilu@gmail.com writes:
>
>>>> It is a requirement
>>>
>>> Then your simplest solution is might be to break the include files up,
>>> but I am puzzled by why this is a requirement.  If you say a bit more
>>> about why this is important, you might get some better suggestions.
>>> Elsewhere, you comment that one of the source files might not exist.
>>> That suggests that your situation is not the normal one envisioned by
>>> build systems.
>>
>> Currently we have a unique repository with all the source code of our
>> project, however my boss decided to put in the repository only object
>> files for "sensitive" source code.
>>
>> This possibility should not block other team members to compile the
>> entire build and make their changes on the remaining files.  I
>> disagree with that choice but for me it is a requirement
>
> Ah, so what you need is not so much something that avoids unneeded
> recompiles but something that can tell the difference between a change
> where the old .o file could safely be used and a change where that would
> actually build an incorrect program.

Doesn't the requirement sound like handling the "sensitive" modules as a 
closed-source library?
Such modules could in fact even be packed in a static library, IIUC.
This way of handling would require the usual for libraries:
A clear identification of the modules (library) interfaces, i.e. header 
files that declare all and only the functions (APIs) defined in the 
modules, and prototype declarations of callback functions (if any) used 
by the modules.
(plus obviously managing symbol uniqueness, global data objects etc..)
Did *I* miss the point?

>
> You could live with an unnecessary recompile, provided you could tell
> when the error can be safely ignored.  And where you do have all the
> source, the occasional extra compile is, again, not really the problem.
>
>>> You can't always get finer compile granularity just by splitting the
>>> include files up because, sometimes, it comes down to what #defines are
>>> in play.  It's possible that some quite complex use of scons could
>>> accommodate this situation but I am not really familiar with it.
>>
>> I don't know how scons can be used to avoid a compilation problem like
>> this one.  (but i don't know scons very well)
>
> SCons appears to be entirely programmable so, one way or another, there
> would be a way to implement almost any conceivable policy.  It might
> take more effort than persuading your boss that not breaking the build
> system is a better option, but it can probably be done.
>

[toc] | [prev] | [next] | [standalone]

#106592

Manfred <noname@invalid.add> writes:

> On 3/20/2017 2:51 PM, Ben Bacarisse wrote:
>> silusilusilu@gmail.com writes:
>>
>>>>> It is a requirement
>>>>
>>>> Then your simplest solution is might be to break the include files up,
>>>> but I am puzzled by why this is a requirement.  If you say a bit more
>>>> about why this is important, you might get some better suggestions.
>>>> Elsewhere, you comment that one of the source files might not exist.
>>>> That suggests that your situation is not the normal one envisioned by
>>>> build systems.
>>>
>>> Currently we have a unique repository with all the source code of our
>>> project, however my boss decided to put in the repository only object
>>> files for "sensitive" source code.
>>>
>>> This possibility should not block other team members to compile the
>>> entire build and make their changes on the remaining files.  I
>>> disagree with that choice but for me it is a requirement
>>
>> Ah, so what you need is not so much something that avoids unneeded
>> recompiles but something that can tell the difference between a change
>> where the old .o file could safely be used and a change where that would
>> actually build an incorrect program.
>
> Doesn't the requirement sound like handling the "sensitive" modules as
> a closed-source library?
> Such modules could in fact even be packed in a static library, IIUC.
> This way of handling would require the usual for libraries: A clear
> identification of the modules (library) interfaces, i.e. header files
> that declare all and only the functions (APIs) defined in the modules,
> and prototype declarations of callback functions (if any) used by the
> modules.  (plus obviously managing symbol uniqueness, global data
> objects etc..)  Did *I* miss the point?

No, I think you have described one of the suggestions I made but from
"the other side".  When I talked about splitting up the include files so
that only the parts that actually relate to the (what I now know are
hidden) source files that should not be re-compiled, I was, rather
clumsily, suggesting that a fixed API be defined for those files.

Your way of putting it -- thinking of them as a closed-source library
with a fixed API -- is clearer though it may not map naturally onto the
actual situation in hand.  That does not mean it's not worth using it.
If the code can be reorganised to make that separation clear, it may
improve the structure overall.

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#106540

silusilusilu@gmail.com wrote:

> If i change something inside function_infile3 (without modify its 
> function prototype), file2.o remains untouched.
>
> If i modify file3.h inserting for example a new define or a new 
> prototype function i should recompile file2 even if it not uses any of 
> the new functions or defines in file3.h.

What you're looking for is called dependency tracking. Take a look at 
a tool called scons, it's a build system that has a nice, built-in 
dependency tracking mechanism.

It's also possible to generate dependencies with gcc to include in 
a normal makefile.

-- 
http://www.chmurka.net/

[toc] | [prev] | [next] | [standalone]

#106545

Thanks for your reply

> > If i change something inside function_infile3 (without modify its 
> > function prototype), file2.o remains untouched.
> >
> > If i modify file3.h inserting for example a new define or a new 
> > prototype function i should recompile file2 even if it not uses any of 
> > the new functions or defines in file3.h.
> 
> What you're looking for is called dependency tracking. Take a look at 
> a tool called scons, it's a build system that has a nice, built-in 
> dependency tracking mechanism.
> 
> It's also possible to generate dependencies with gcc to include in 
> a normal makefile.

I don't know if make,cmake,scons are the right tool for me.

Premise: i don't know those tools very well so my statements can be wrong.

During a compilation you will always use gcc: the prepocessor will try to include my file3.h in file2.c and it will not be possible if file2.c is not present and there is only file2.o.

Or i'm wrong?

[toc] | [prev] | [next] | [standalone]

#106647

silusilusilu@gmail.com wrote:

> I don't know if make,cmake,scons are the right tool for me.
> 
> Premise: i don't know those tools very well so my statements can be wrong.
> 
> During a compilation you will always use gcc: the prepocessor will try 
> to include my file3.h in file2.c and it will not be possible if file2.c 
> is not present and there is only file2.o.
>
> Or i'm wrong?

You're right, but even before compilation, a building tool (make, cmake, 
scons etc.) needs to know which files to invoke gcc on. There's a special 
flag to gcc (-MM) that makes dependencies.

I just created four files, each containing one line:

- test1.c:	#include "test3.h"
- test2.c:	int main;
- test3.h:	#include "test4.h"
- test4.h;	/* empty */

The goal is to discover that test1.o (object file) depends on test1.c, 
test3.h and test4.h, but test2.o depends only on test2.c.

Let's use gcc -MM:

$ gcc -MM test1.c
test1.o: test1.c test3.h test4.h
$ gcc -MM test2.c
test2.o: test2.c

It can be used to generate a dependency file to include in a Makefile, 
read by GNU make, something like that:

dep:    .depend

.depend:        $(SRCS)
        $(CC) -MM $(CFLAGS) $(SRCS) 1> .depend

ifneq ($(wildcard .depend),)
include .depend
endif

The magic of scons is that it does the dependency tracking automatically, 
without any need for gcc -MM (which is especially important to me, as in 
my work I use a compiler that doesn't support generating dependency file, 
that's why I switched to scons).

I created Sconstruct file (a makefile for scons) with only one line:

Program('test', ['test1.c', 'test2.c'])

I called scons and it compiled the program test.

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o test1.o -c test1.c
gcc -o test2.o -c test2.c
gcc -o test test1.o test2.o
scons: done building targets.

Now where the magic begins. I modified test2.c and called scons once 
again. It should recompile only test2.o and relink test.

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o test2.o -c test2.c
gcc -o test test1.o test2.o
scons: done building targets.

And now I modified test4.h. It should recompile only test1.c and rebuild 
test...

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o test1.o -c test1.c
gcc -o test test1.o test2.o
scons: done building targets.

Works automatically and out of the box!

As I don't want to specify all source files in Sconstruct file, I changed 
the line to:

Program('test', Glob('*.c'))

And it works the same.

-- 
http://www.chmurka.net/

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web