Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > comp.lang.c++ > #89449 > unrolled thread

Link at runtime instead of build-time

Started byFrederick Virchanza Gotham <cauldwell.thomas@gmail.com>
First post2023-03-05 09:59 -0800
Last post2023-03-06 23:58 +0100
Articles 12 — 5 participants

Back to article view | Back to comp.lang.c++


Contents

  Link at runtime instead of build-time Frederick Virchanza Gotham <cauldwell.thomas@gmail.com> - 2023-03-05 09:59 -0800
    Re: Link at runtime instead of build-time Christian Gollwitzer <auriocus@gmx.de> - 2023-03-05 19:12 +0100
      Re: Link at runtime instead of build-time Frederick Virchanza Gotham <cauldwell.thomas@gmail.com> - 2023-03-05 11:11 -0800
        Re: Link at runtime instead of build-time Christian Gollwitzer <auriocus@gmx.de> - 2023-03-05 22:45 +0100
          Re: Link at runtime instead of build-time Frederick Virchanza Gotham <cauldwell.thomas@gmail.com> - 2023-03-05 16:10 -0800
            Re: Link at runtime instead of build-time Christian Gollwitzer <auriocus@gmx.de> - 2023-03-06 09:29 +0100
              Re: Link at runtime instead of build-time Frederick Virchanza Gotham <cauldwell.thomas@gmail.com> - 2023-03-06 02:37 -0800
        Re: Link at runtime instead of build-time "Alf P. Steinbach" <alf.p.steinbach@gmail.com> - 2023-03-06 15:45 +0100
          Re: Link at runtime instead of build-time Muttley@dastardlyhq.com - 2023-03-06 17:14 +0000
            Re: Link at runtime instead of build-time Frederick Virchanza Gotham <cauldwell.thomas@gmail.com> - 2023-03-06 13:41 -0800
              Re: Link at runtime instead of build-time gazelle@shell.xmission.com (Kenny McCormack) - 2023-03-06 22:20 +0000
              Re: Link at runtime instead of build-time Christian Gollwitzer <auriocus@gmx.de> - 2023-03-06 23:58 +0100

#89449 — Link at runtime instead of build-time

FromFrederick Virchanza Gotham <cauldwell.thomas@gmail.com>
Date2023-03-05 09:59 -0800
SubjectLink at runtime instead of build-time
Message-ID<6e54715f-9094-46c2-94ec-82aa9d7eaccdn@googlegroups.com>
Let's say I have one program and one shared library on a *nix computer.

I build my program to link dynamically with the library as follows:

g++ -o prog prog.cpp -l:libmonkey.so

So let's say that this produces an executable file that works fine.

Next I change the build command to the following:

g++ -o prog prog.cpp

And so now of course I get a linker error because of unresolved symbols (I'm missing the functions implemented in the library). To overcome this linker error, I change the build command again:

g++ -o prog prog.cpp -Wl,--unresolved-symbols=ignore-in-object-files

So now we get a viable executable file, but when we run it, it segfaults. It segfaults when it tries to access something from the shared library.

So next I make a one-line change at the beginning of 'main':

int main(void)
{
    dlopen("libmonkey.so", RTLD_NOW|RTLD_GLOBAL);

    // The rest of main goes here
}

So now the program works fine.

I think however, that this strategy only works when the program doesn't use global variables which are exported from the library. That is to say, it works fine for exported functions but not for exported global variables. Am I right here?

I have more testing to do on this, but I need to get this working. I need to remove the library from the 'NEEDED' section in the program's ELF file, and instead I have to use 'dlopen' at runtime to load it in.

How can I get this to work with the global variables also? I've tried declaring the global variable as a weak symbol in the program, and that seems to work, but I don't want to have to go looking for a list of all the global objects exported from the shared library.

Also I realise that things get complicated if there's a global object in the program whose constructor accesses the shared library (in which case I need to make use of __attribute__((constructor)). But I'll deal with thay conundrum later.

[toc] | [next] | [standalone]


#89450

FromChristian Gollwitzer <auriocus@gmx.de>
Date2023-03-05 19:12 +0100
Message-ID<tu2m2g$1ddsn$1@dont-email.me>
In reply to#89449
Am 05.03.23 um 18:59 schrieb Frederick Virchanza Gotham:
> Let's say I have one program and one shared library on a *nix computer.
> 
> I build my program to link dynamically with the library as follows:
> 
> g++ -o prog prog.cpp -l:libmonkey.so
> 
> So let's say that this produces an executable file that works fine.
> 
> Next I change the build command to the following:
> 
> g++ -o prog prog.cpp
> 
> And so now of course I get a linker error because of unresolved symbols (I'm missing the functions implemented in the library). To overcome this linker error, I change the build command again:
> 
> g++ -o prog prog.cpp -Wl,--unresolved-symbols=ignore-in-object-files
> 
> So now we get a viable executable file, but when we run it, it segfaults. It segfaults when it tries to access something from the shared library.
> 
> So next I make a one-line change at the beginning of 'main':
> 
> int main(void)
> {
>      dlopen("libmonkey.so", RTLD_NOW|RTLD_GLOBAL);
> 
>      // The rest of main goes here
> }
> 
> So now the program works fine.
> 
> I think however, that this strategy only works when the program doesn't use global variables which are exported from the library. That is to say, it works fine for exported functions but not for exported global variables. Am I right here?
> 
> I have more testing to do on this, but I need to get this working. I need to remove the library from the 'NEEDED' section in the program's ELF file, and instead I have to use 'dlopen' at runtime to load it in.


As always, I'm questioning this NEED. Why do you want to load the 
required library at runtime? Do you want to adjust the path at 
initialization? If it is only about packaging the library with the 
application, you could use the run-path when linking; e.g. say your 
library is distributed in a lib folder alongside your executable, then

-Wl,-rpath,'$ORIGIN'/lib

in the linker line would do that. Another alternative is to write a 
small shell script which adjusts LD_LIBRARY_PATH before invoking the binary.

I can see the need for dlopen() only in case where the functionality is 
somewhat optional, like in a plugin system.

	Christian

[toc] | [prev] | [next] | [standalone]


#89454

FromFrederick Virchanza Gotham <cauldwell.thomas@gmail.com>
Date2023-03-05 11:11 -0800
Message-ID<bd9d76af-a28d-4c27-9b1b-4246cc25a8e4n@googlegroups.com>
In reply to#89450
> As always, I'm questioning this NEED.


I'm combining a console program and a GUI program into one program (i.e. one executable).

Depending upon the command-line arguments to the program, it might enter into GUI mode, and if it does, it needs to load the GUI libraries.

But if it runs in console mode, I don't want the system to need to have the Gui libraries installed.

[toc] | [prev] | [next] | [standalone]


#89459

FromChristian Gollwitzer <auriocus@gmx.de>
Date2023-03-05 22:45 +0100
Message-ID<tu32id$1en3f$1@dont-email.me>
In reply to#89454
Am 05.03.23 um 20:11 schrieb Frederick Virchanza Gotham:
> 
>> As always, I'm questioning this NEED.
> 
> 
> I'm combining a console program and a GUI program into one program (i.e. one executable).
> 
> Depending upon the command-line arguments to the program, it might enter into GUI mode, and if it does, it needs to load the GUI libraries.
> 
> But if it runs in console mode, I don't want the system to need to have the Gui libraries installed.

This usually works by exporting a known entry function from the library, 
load the library by dlopen() and find the function using dlsym(). Put 
all the functionality into the library, so that you can invoke the GUI 
stuff with a simple call. It doesn't make sense to have global variables 
in the main program have influenced by loadable library.

If you need a lot of the functionality of the main program in the 
library, then instead put this functionality in a library if its own 
(say, libengine.so). Then the main program reduces to

if (std::string(argv[1] == "-gui") {
	dlopen();
	run_gui = dlsym("...");
	run_gui();
} else {
	run_cmdline();
}

Link both this main and the GUI library with libengine.so.

	Christian

[toc] | [prev] | [next] | [standalone]


#89464

FromFrederick Virchanza Gotham <cauldwell.thomas@gmail.com>
Date2023-03-05 16:10 -0800
Message-ID<35a8ec8a-bb4a-480e-a04b-4db8e3f3614an@googlegroups.com>
In reply to#89459
On Sunday, March 5, 2023 at 9:46:04 PM UTC, Christian Gollwitzer wrote:
>
> If you need a lot of the functionality of the main program in the 
> library, then instead put this functionality in a library if its own 
> (say, libengine.so).


My program has to be one statically-linked executable file.

I found the following doing a web search:

https://stackoverflow.com/questions/20658809/dynamic-loading-and-weak-symbol-resolution

If you scroll down on that page, a guy called Praxeolitic writes "Here is an example in which the executable itself has an undefined symbol that is resolved through dynamic loading", and he gives a code example. His code example is /exactly/ what I'm trying to achieve, but it doesn't work on modern Linux. It seems to have worked 9 years ago, but something has changed since then. I've tried using gcc-4.8 on modern Linux but the executable I get doesn't work.

Does anyone know what might have changed in Linux in the past 10 years to stop this working? Or does anyone have an email address for Praxeolitic? Anyone know who Praxeolitic is?

I think I'll download an old copy of Ubuntu from 10 years ago and run it in a VM and try this out.

[toc] | [prev] | [next] | [standalone]


#89470

FromChristian Gollwitzer <auriocus@gmx.de>
Date2023-03-06 09:29 +0100
Message-ID<tu489m$1kqqq$2@dont-email.me>
In reply to#89464
Am 06.03.23 um 01:10 schrieb Frederick Virchanza Gotham:
> On Sunday, March 5, 2023 at 9:46:04 PM UTC, Christian Gollwitzer wrote:
>>
>> If you need a lot of the functionality of the main program in the
>> library, then instead put this functionality in a library if its own
>> (say, libengine.so).
> 
> 
> My program has to be one statically-linked executable file.
> 
> I found the following doing a web search:
> 
> https://stackoverflow.com/questions/20658809/dynamic-loading-and-weak-symbol-resolution
> 
> If you scroll down on that page, a guy called Praxeolitic writes "Here is an example in which the executable itself has an undefined symbol that is resolved through dynamic loading", and he gives a code example. His code example is /exactly/ what I'm trying to achieve, but it doesn't work on modern Linux. It seems to have worked 9 years ago, but something has changed since then. I've tried using gcc-4.8 on modern Linux but the executable I get doesn't work.
> 
> Does anyone know what might have changed in Linux in the past 10 years to stop this working? Or does anyone have an email address for Praxeolitic? Anyone know who Praxeolitic is?
> 
> I think I'll download an old copy of Ubuntu from 10 years ago and run it in a VM and try this out.

Have you seen the comment below this message from 2018, that the dynamic 
linker only respects weak symbols when you set LD_DYNAMIC_WEAK? Quoting 
from the referenced page:

LD_DYNAMIC_WEAK (since glibc 2.1.91)
               By default, when searching shared libraries to resolve a
               symbol reference, the dynamic linker will resolve to the
               first definition it finds.

               Old glibc versions (before 2.2), provided a different
               behavior: if the linker found a symbol that was weak, it
               would remember that symbol and keep searching in the
               remaining shared libraries.  If it subsequently found a
               strong definition of the same symbol, then it would
               instead use that definition.  (If no further symbol was
               found, then the dynamic linker would use the weak symbol
               that it initially found.)

               The old glibc behavior was nonstandard.  (Standard
               practice is that the distinction between weak and strong
               symbols should have effect only at static link time.)  In
               glibc 2.2, the dynamic linker was modified to provide the
               current behavior (which was the behavior that was provided
               by most other implementations at that time).

               Defining the LD_DYNAMIC_WEAK environment variable (with
               any value) provides the old (nonstandard) glibc behavior,
               whereby a weak symbol in one shared library may be
               overridden by a strong symbol subsequently discovered in
               another shared library.  (Note that even when this
               variable is set, a strong symbol in a shared library will
               not override a weak definition of the same symbol in the
               main program.)

               Since glibc 2.3.4, LD_DYNAMIC_WEAK is ignored in secure-
               execution mode.


This sounds very hackish to me. Most portable (and easiest) solution 
will be a shell script. The overall goal reminds me to the Tk toolkit. 
In Tcl, you can do "package require Tk" which loads the Tk toolkit 
dynamically. However, there are at least two files - Tk is its own 
shared object, which depends on libX11 etc., while Tcl alone does not 
depend on libX11. For a single file executable (called starpack) they go 
through some hoops, especially attaching a load to the end of the 
executable (similar to a self-extracting ZIP file), which is addressed 
as a virtual file system. Are you sure it is worth to go through this 
for your program? The simplest solution (besides a shell script) would 
be to split the GUI part into its own library, which is installed as a 
separate package.

	Christian

[toc] | [prev] | [next] | [standalone]


#89475

FromFrederick Virchanza Gotham <cauldwell.thomas@gmail.com>
Date2023-03-06 02:37 -0800
Message-ID<23844782-9237-40ce-a51e-40341db05545n@googlegroups.com>
In reply to#89470
On Monday, March 6, 2023 at 8:29:58 AM UTC, Christian Gollwitzer wrote:
>
> Have you seen the comment below this message from 2018, that the dynamic 
> linker only respects weak symbols when you set LD_DYNAMIC_WEAK? 


LD_DYNAMIC_WEAK isn't what's going on here.

I managed to get it all working just now. When building the executable, you have to give -Wl,--export-dynamic

Now it's working as intended.

[toc] | [prev] | [next] | [standalone]


#89477

From"Alf P. Steinbach" <alf.p.steinbach@gmail.com>
Date2023-03-06 15:45 +0100
Message-ID<tu4u9t$1n67n$1@dont-email.me>
In reply to#89454
On 2023-03-05 8:11 PM, Frederick Virchanza Gotham wrote:
> 
>> As always, I'm questioning this NEED.
> 
> 
> I'm combining a console program and a GUI program into one program
> (i.e. one executable).
> 
> Depending upon the command-line arguments to the program, it might
> enter into GUI mode, and if it does, it needs to load the GUI
> libraries.
> 
> But if it runs in console mode, I don't want the system to need to
> have the Gui libraries installed.

[C:\Users\Alf P. Steinbach\Desktop]
 > where devenv
C:\Program Files\Microsoft Visual 
Studio\2022\Community\Common7\IDE\devenv.com
C:\Program Files\Microsoft Visual 
Studio\2022\Community\Common7\IDE\devenv.exe


Guess which one has console subsystem and which has GUI subsystem?

Well, not that hard to guess, because invoked by command should choose 
the console subsystem one:


[C:\Users\Alf P. Steinbach\Desktop]
 > set pathext
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.LNK
PATHEXT-SYS=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC


In passing, some years ago MS sabotaged the use of `.lnk` files as 
commands, as I recall it had to do with argument passing. The question 
of how to differentiate between sabotage and incompetence is an unsolved 
philosophical question. But I simply label it sabotage when it comes 
from Microsoft b/c I don't think one should discount earlier history.


- Alf

[toc] | [prev] | [next] | [standalone]


#89486

FromMuttley@dastardlyhq.com
Date2023-03-06 17:14 +0000
Message-ID<tu571n$8fl$1@dont-email.me>
In reply to#89477
On Mon, 6 Mar 2023 15:45:17 +0100
"Alf P. Steinbach" <alf.p.steinbach@gmail.com> wrote:
>On 2023-03-05 8:11 PM, Frederick Virchanza Gotham wrote:
>> 
>>> As always, I'm questioning this NEED.
>> 
>> 
>> I'm combining a console program and a GUI program into one program
>> (i.e. one executable).
>> 
>> Depending upon the command-line arguments to the program, it might
>> enter into GUI mode, and if it does, it needs to load the GUI
>> libraries.
>> 
>> But if it runs in console mode, I don't want the system to need to
>> have the Gui libraries installed.
>
>[C:\Users\Alf P. Steinbach\Desktop]
> > where devenv
>C:\Program Files\Microsoft Visual 
>Studio\2022\Community\Common7\IDE\devenv.com
>C:\Program Files\Microsoft Visual 
>Studio\2022\Community\Common7\IDE\devenv.exe
>
>
>Guess which one has console subsystem and which has GUI subsystem?
>
>Well, not that hard to guess, because invoked by command should choose 
>the console subsystem one:

Use a version of *nix. It doesn't have this cretinous differentiation between 
a console or GUI program. They're just programs.

[toc] | [prev] | [next] | [standalone]


#89493

FromFrederick Virchanza Gotham <cauldwell.thomas@gmail.com>
Date2023-03-06 13:41 -0800
Message-ID<fa55ce4e-39e4-45e0-9bc0-736dacce73e5n@googlegroups.com>
In reply to#89486
Actually instead I think I'm going to link the program dynamically as normal, but then after the executable file is built, use 'patchelf' on it to remove some of the 'NEEDED' libraries, so then I can just use 'dlopen' at runtime.

[toc] | [prev] | [next] | [standalone]


#89495

Fromgazelle@shell.xmission.com (Kenny McCormack)
Date2023-03-06 22:20 +0000
Message-ID<tu5ove$o7c4$1@news.xmission.com>
In reply to#89493
In article <fa55ce4e-39e4-45e0-9bc0-736dacce73e5n@googlegroups.com>,
Frederick Virchanza Gotham  <cauldwell.thomas@gmail.com> wrote:
>
>Actually instead I think I'm going to link the program dynamically as
>normal, but then after the executable file is built, use 'patchelf' on it
>to remove some of the 'NEEDED' libraries, so then I can just use 'dlopen'
>at runtime.

You do you...

-- 
Kenny, I'll ask you to stop using quotes of mine as taglines.

    - Rick C Hodgin -

[toc] | [prev] | [next] | [standalone]


#89497

FromChristian Gollwitzer <auriocus@gmx.de>
Date2023-03-06 23:58 +0100
Message-ID<tu5r77$3d8k$1@dont-email.me>
In reply to#89493
Am 06.03.23 um 22:41 schrieb Frederick Virchanza Gotham:
> 
> Actually instead I think I'm going to link the program dynamically as normal, but then after the executable file is built, use 'patchelf' on it to remove some of the 'NEEDED' libraries, so then I can just use 'dlopen' at runtime.

Who would have thought that you come up with another bizarre hack. I'm 
beginning to think that you do these things for fun. I can sympathise 
with this idea, hacking is fun.

	Christian

[toc] | [prev] | [standalone]


Back to top | Article view | comp.lang.c++


csiph-web