Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.debian.devel > #101547 > unrolled thread
| Started by | Zebediah Figura <zfigura@codeweavers.com> |
|---|---|
| First post | 2021-09-05 03:40 +0200 |
| Last post | 2022-04-17 02:30 +0200 |
| Articles | 20 on this page of 53 — 8 participants |
Back to article view | Back to linux.debian.devel
Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-05 03:40 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-05 09:20 +0200
Re: Wine MinGW system libraries Stephen Kitt <skitt@debian.org> - 2021-09-05 18:30 +0200
Re: Wine MinGW system libraries Adrian Bunk <bunk@debian.org> - 2021-09-12 10:00 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-12 15:40 +0200
Re: Wine MinGW system libraries Adrian Bunk <bunk@debian.org> - 2021-09-12 16:10 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-12 16:20 +0200
Re: Wine MinGW system libraries Adrian Bunk <bunk@debian.org> - 2021-09-12 16:40 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-12 17:20 +0200
Re: Wine MinGW system libraries Adrian Bunk <bunk@debian.org> - 2021-09-12 17:30 +0200
Re: Wine MinGW system libraries Helmut Grohne <helmut@subdivi.de> - 2021-09-13 17:50 +0200
Re: Wine MinGW system libraries Stephen Kitt <skitt@debian.org> - 2021-09-13 20:20 +0200
Re: Wine MinGW system libraries Helmut Grohne <helmut@subdivi.de> - 2021-09-13 22:10 +0200
Re: Wine MinGW system libraries Adrian Bunk <bunk@debian.org> - 2021-09-13 22:30 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-13 23:00 +0200
Re: Wine MinGW system libraries Adrian Bunk <bunk@debian.org> - 2021-09-13 23:40 +0200
Re: Wine MinGW system libraries Stephen Kitt <skitt@debian.org> - 2021-09-12 17:40 +0200
Re: Wine MinGW system libraries Adrian Bunk <bunk@debian.org> - 2021-09-12 18:40 +0200
Re: Wine MinGW system libraries Stephen Kitt <skitt@debian.org> - 2021-09-12 19:10 +0200
Re: Wine MinGW system libraries Adrian Bunk <bunk@debian.org> - 2021-09-12 20:20 +0200
Re: Wine MinGW system libraries Stephen Kitt <skitt@debian.org> - 2021-09-13 20:30 +0200
Re: Wine MinGW system libraries Stephen Kitt <skitt@debian.org> - 2021-09-05 18:20 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-05 19:20 +0200
Re: Wine MinGW system libraries Stephen Kitt <skitt@debian.org> - 2021-09-06 09:10 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-06 10:20 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-06 18:40 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-06 21:00 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-07 00:00 +0200
Re: Wine MinGW system libraries Paul Wise <pabs@debian.org> - 2021-09-07 02:50 +0200
Re: Wine MinGW system libraries "Bastien Roucariès" <roucaries.bastien@gmail.com> - 2021-09-07 12:40 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-07 17:50 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-07 19:30 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-07 20:20 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-07 23:20 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-08 01:50 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-08 09:50 +0200
Re: Wine MinGW system libraries Simon McVittie <smcv@debian.org> - 2021-09-08 10:20 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-08 17:50 +0200
Re: Wine MinGW system libraries Paul Wise <pabs@debian.org> - 2021-09-09 03:20 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-09 06:50 +0200
Re: Wine MinGW system libraries Paul Wise <pabs@debian.org> - 2021-09-09 07:20 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-09 07:40 +0200
Re: Wine MinGW system libraries Paul Wise <pabs@debian.org> - 2021-09-09 07:50 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2021-09-09 08:10 +0200
Re: Wine MinGW system libraries Paul Wise <pabs@debian.org> - 2021-09-09 09:40 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-10 11:50 +0200
Re: Wine MinGW system libraries Paul Wise <pabs@debian.org> - 2021-09-11 08:10 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-12 17:20 +0200
Re: Wine MinGW system libraries Paul Wise <pabs@debian.org> - 2021-09-08 01:10 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-08 01:30 +0200
Re: Wine MinGW system libraries Adrian Bunk <bunk@debian.org> - 2021-09-11 17:40 +0200
Re: Wine MinGW system libraries Bastien ROUCARIES <roucaries.bastien@gmail.com> - 2021-09-11 20:10 +0200
Re: Wine MinGW system libraries Zebediah Figura <zfigura@codeweavers.com> - 2022-04-17 02:30 +0200
Page 2 of 3 — ← Prev page 1 [2] 3 Next page →
| From | Stephen Kitt <skitt@debian.org> |
|---|---|
| Date | 2021-09-13 20:30 +0200 |
| Message-ID | <CWYSm-1yj-13@gated-at.bofh.it> |
| In reply to | #101677 |
[Multipart message — attachments visible in raw view] — view raw
On Sun, 12 Sep 2021 20:54:56 +0300, Adrian Bunk <bunk@debian.org> wrote: > On Sun, Sep 12, 2021 at 07:03:54PM +0200, Stephen Kitt wrote: > > On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk <bunk@debian.org> wrote: > > > On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote: > > > >... > > > > While one could imagine adding support to all the appropriate > > > > source packages to build similar “Architecture: all” packages, that > > > > would require convincing all the relevant maintainers, > > > > > > Adding a new release architecture (partial or not) requires convincing > > > the ftp team and the release team. > > > > > > It would also require many software changes. > > > > Yes, it would. However, it requires convincing a well-defined set of > > people *once*. If we handle Windows dependencies using “Architecture: > > all” packages, we would either end up with something like Fedora’s > > MinGW-w64 SIG (with a complete set of independent packages), or we would > > end up having to convince a huge set of package maintainers over and over > > again. > > Based on the package list provided in the initial email in this thread > (and guessing the amount of transitive dependencies) I would have guessed > that there are perhaps ~ 30 packages that would have to be touched once. ... which means ~30 maintainers who need convincing individually. My experience so far is that maintainers tend not to be interested (justifiably) in Windows cross-builds. > And after that things will just continue working, just like with udebs > which are in some ways similar to what is being discussed here. udebs are mostly a packaging concern. Keeping Windows cross-builds working tends to require a bit more effort unfortunately. > > > How would for example the dependencies of wine:amd64 be fulfilled? > > > > > > Just supporting that package dependencies might only be fulfilled by > > > also using packages from a different architecture would require changes > > > in many places, like for example the testing migration scripts that > > > ensure installability after migration. > > > > In the same way as the none-arch packages would be. Yes, it’s a lot of > > work, but it’s useful for more than just Wine, and it can be done once > > for all the interested architectures. > > It is also the kind of change where it might be required that it is > fully supported in one release before it can be used by packages in the > next release, if for example any of dpkg/apt/aptitude/... in the > previous stable gives an error when seeing dependencies to another > architecture. That is indeed a good point, thanks for bringing it up! Regards, Stephen
[toc] | [prev] | [next] | [standalone]
| From | Stephen Kitt <skitt@debian.org> |
|---|---|
| Date | 2021-09-05 18:20 +0200 |
| Message-ID | <CU32a-31p-19@gated-at.bofh.it> |
| In reply to | #101547 |
[Multipart message — attachments visible in raw view] — view raw
Hi Zebediah, On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura <zfigura@codeweavers.com> wrote: > I'm a contributor to the Wine project. To summarize the following mail, > Wine needs special versions of some of its normal dependencies, such as > libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > sending out a mail to major distributions in order to get some feedback > from our packagers on how these should be built and packaged. > > For a long time Wine has built all of its Win32 libraries (DLLs and > EXEs) as ELF binaries. For various reasons related to application > compatibility, we have started building our binaries as PE instead, > using the MinGW cross-compiler. It is our intent to expand this to some > of our dependencies as well. The list of dependencies that we intend to > build using MinGW is not quite fixed yet, but we expect it to include > and be mostly limited to the following: > > * libvkd3d > * libFAudio > * libgnutls > * zlib (currently included via manual source import) > * libmpg123 > * libgsm > * libpng > * libjpeg-turbo > * libtiff > * libfreetype > * liblcms2 > * jxrlib > > and dependencies of the above packages (not including CRT dependencies, > which Wine provides). > > There is currently some internal discussion about how these dependencies > should be built and linked. There are essentially three questions I see > that need to be resolved, and while these resolutions have a significant > impact on the Wine building and development process, they also have an > impact on distributions, and accordingly I'd like to get input from our > packagers to ensure that their considerations are accurately taken into > account. > > (1) Should we build via source import, or link statically, or dynamically? > > Source imports are dispreferred by Debian [1], on the grounds that they > cause duplication of libraries on disk and in memory, and make it harder > to handle security updates. They also make building and bisecting > harder. Static libraries don't seem to be expressly discouraged, but > share most of the same downsides (see also [2]). > > Note however that if they are linked dynamically, we need to make sure > that we load our packages instead of MinGW builds of open-source > libraries with applications ship with. There's some internal discussion > about whether this is possible while using "stock" builds of MinGW > libraries, but, due to the way the Win32 loader works, we will probably > need to compile each library, and its dependencies, with a separate, > wine-specific name, e.g. "libwinefreetype-6.dll" and > "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note > that all we actually need to change is the name; we don't need to patch > the source. Assuming Debian provides the dependencies (which is currently true only for zlib), this could be handled in packaging by providing symlinks, couldn’t it? Not in the Wine prefixes, but elsewhere. (The Wine team also maintains libvkd3d and libFaudio, so we can take care of those at least.) > Accordingly, although static linking and source imports are generally > disprefered, it may quite likely be preferable in our case. We don't get > the benefits of on-disk deduplication, since Wine is essentially the > only piece of software which needs these libraries. > > (2) If we use dynamic libraries, should dependencies be included in the > main wine package, or packaged separately? > > This is mostly a question for packagers, although it also relates to (3). > > I expect that Debian will want to answer "packaged separately" here, on > the grounds that this lets them update (say) Wine's libgnutls > separately, and in sync with ELF libgnutls, if some security fix is > needed. There is a snag, though: we need libraries to be copied into the > prefix (there's some internal effort to allow using something like > symlinks instead, but this hard and not done yet). Normally we perform > this copy every time Wine is updated, but if Wine and its dependencies > aren't updated on the same schedule, we may end up loading an old > version of a dependency in the prefix. Debian packaging doesn’t touch anything in users’ home directories, so this would have to be handled in Wine itself, perhaps in a similar fashion to existing provisions for Gecko and Mono. > (3) If dependencies are packaged separately, should Wine build them as > part of its build tree (e.g. using submodules), or find and link > (statically or dynamically) to existing binaries? > > Linking to existing binaries is generally preferable: it avoids > duplication on disk; it reduces compile times when compiling a single > package from source (especially the first time). However, we aren't > going to benefit from on-disk duplication. And, most importantly, unlike > with ELF dependencies, there is no standardized way to locate MinGW > libraries—especially if it comes to Wine-specific libraries. We would > need a way for Wine's configure script to find these packages—and > ideally find them automatically, or else fall back to a submodule-based > approach. > > If we rely on distributions to provide our dependencies, the best idea I > have here would be something like a x86_64-w64-mingw32-pkg-config > (Fedora actually already ships this, but I think Fedora is the only > one). And if we use shared libraries rather than static, things get > worse: we need to know the exact path of each library and its > dependencies at runtime so that we can copy (or symlink) them into a > user's WINEPREFIX. For ELF this is the job of ld.so. For MinGW there is > no standardized install location across distributions. This also works in Debian: $ sudo apt install libz-mingw-w64-dev mingw-w64-tools $ x86_64-w64-mingw32-pkg-config --libs zlib -L/usr/x86_64-w64-mingw32/lib -lz > For what it's worth, the current proposed solution (which has the > support of the Wine maintainer) involves source imports and submodules. > There's probably room for changing our approach even after things are > committed, but I'd still like to get early feedback from distributions, > and make sure that their interests are accurately represented, before we > commit. Realistically, I think this is the best approach for now. As Debian adds support for PE libraries, we can replace the source imports in the Wine source code; this is done in many other Debian packages for projects which vendor dependencies. Regards, Stephen
[toc] | [prev] | [next] | [standalone]
| From | Zebediah Figura <zfigura@codeweavers.com> |
|---|---|
| Date | 2021-09-05 19:20 +0200 |
| Message-ID | <CU3Yg-3BQ-13@gated-at.bofh.it> |
| In reply to | #101552 |
On 9/5/21 11:19 AM, Stephen Kitt wrote: > Hi Zebediah, > > On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura <zfigura@codeweavers.com> > wrote: >> I'm a contributor to the Wine project. To summarize the following mail, >> Wine needs special versions of some of its normal dependencies, such as >> libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm >> sending out a mail to major distributions in order to get some feedback >> from our packagers on how these should be built and packaged. >> >> For a long time Wine has built all of its Win32 libraries (DLLs and >> EXEs) as ELF binaries. For various reasons related to application >> compatibility, we have started building our binaries as PE instead, >> using the MinGW cross-compiler. It is our intent to expand this to some >> of our dependencies as well. The list of dependencies that we intend to >> build using MinGW is not quite fixed yet, but we expect it to include >> and be mostly limited to the following: >> >> * libvkd3d >> * libFAudio >> * libgnutls >> * zlib (currently included via manual source import) >> * libmpg123 >> * libgsm >> * libpng >> * libjpeg-turbo >> * libtiff >> * libfreetype >> * liblcms2 >> * jxrlib >> >> and dependencies of the above packages (not including CRT dependencies, >> which Wine provides). >> >> There is currently some internal discussion about how these dependencies >> should be built and linked. There are essentially three questions I see >> that need to be resolved, and while these resolutions have a significant >> impact on the Wine building and development process, they also have an >> impact on distributions, and accordingly I'd like to get input from our >> packagers to ensure that their considerations are accurately taken into >> account. >> >> (1) Should we build via source import, or link statically, or dynamically? >> >> Source imports are dispreferred by Debian [1], on the grounds that they >> cause duplication of libraries on disk and in memory, and make it harder >> to handle security updates. They also make building and bisecting >> harder. Static libraries don't seem to be expressly discouraged, but >> share most of the same downsides (see also [2]). >> >> Note however that if they are linked dynamically, we need to make sure >> that we load our packages instead of MinGW builds of open-source >> libraries with applications ship with. There's some internal discussion >> about whether this is possible while using "stock" builds of MinGW >> libraries, but, due to the way the Win32 loader works, we will probably >> need to compile each library, and its dependencies, with a separate, >> wine-specific name, e.g. "libwinefreetype-6.dll" and >> "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note >> that all we actually need to change is the name; we don't need to patch >> the source. > > Assuming Debian provides the dependencies (which is currently true only for > zlib), this could be handled in packaging by providing symlinks, couldn’t it? > Not in the Wine prefixes, but elsewhere. Almost :-/ Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. The problem is that libwinefreetype-1.dll is still going to link to libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. > (The Wine team also maintains libvkd3d and libFaudio, so we can take care of > those at least.) > >> Accordingly, although static linking and source imports are generally >> disprefered, it may quite likely be preferable in our case. We don't get >> the benefits of on-disk deduplication, since Wine is essentially the >> only piece of software which needs these libraries. >> >> (2) If we use dynamic libraries, should dependencies be included in the >> main wine package, or packaged separately? >> >> This is mostly a question for packagers, although it also relates to (3). >> >> I expect that Debian will want to answer "packaged separately" here, on >> the grounds that this lets them update (say) Wine's libgnutls >> separately, and in sync with ELF libgnutls, if some security fix is >> needed. There is a snag, though: we need libraries to be copied into the >> prefix (there's some internal effort to allow using something like >> symlinks instead, but this hard and not done yet). Normally we perform >> this copy every time Wine is updated, but if Wine and its dependencies >> aren't updated on the same schedule, we may end up loading an old >> version of a dependency in the prefix. > > Debian packaging doesn’t touch anything in users’ home directories, so this > would have to be handled in Wine itself, perhaps in a similar fashion to > existing provisions for Gecko and Mono. > >> (3) If dependencies are packaged separately, should Wine build them as >> part of its build tree (e.g. using submodules), or find and link >> (statically or dynamically) to existing binaries? >> >> Linking to existing binaries is generally preferable: it avoids >> duplication on disk; it reduces compile times when compiling a single >> package from source (especially the first time). However, we aren't >> going to benefit from on-disk duplication. And, most importantly, unlike >> with ELF dependencies, there is no standardized way to locate MinGW >> libraries—especially if it comes to Wine-specific libraries. We would >> need a way for Wine's configure script to find these packages—and >> ideally find them automatically, or else fall back to a submodule-based >> approach. >> >> If we rely on distributions to provide our dependencies, the best idea I >> have here would be something like a x86_64-w64-mingw32-pkg-config >> (Fedora actually already ships this, but I think Fedora is the only >> one). And if we use shared libraries rather than static, things get >> worse: we need to know the exact path of each library and its >> dependencies at runtime so that we can copy (or symlink) them into a >> user's WINEPREFIX. For ELF this is the job of ld.so. For MinGW there is >> no standardized install location across distributions. > > This also works in Debian: > > $ sudo apt install libz-mingw-w64-dev mingw-w64-tools > $ x86_64-w64-mingw32-pkg-config --libs zlib > -L/usr/x86_64-w64-mingw32/lib -lz Ah, cool, I looked for that and somehow missed it. Note that's the wrong path for dynamic libraries, though; those go in /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe hardcoding a list won't be too painful, though... >> For what it's worth, the current proposed solution (which has the >> support of the Wine maintainer) involves source imports and submodules. >> There's probably room for changing our approach even after things are >> committed, but I'd still like to get early feedback from distributions, >> and make sure that their interests are accurately represented, before we >> commit. > > Realistically, I think this is the best approach for now. As Debian adds > support for PE libraries, we can replace the source imports in the Wine > source code; this is done in many other Debian packages for projects which > vendor dependencies. > > Regards, > > Stephen >
[toc] | [prev] | [next] | [standalone]
| From | Stephen Kitt <skitt@debian.org> |
|---|---|
| Date | 2021-09-06 09:10 +0200 |
| Message-ID | <CUgVt-3A0-19@gated-at.bofh.it> |
| In reply to | #101555 |
[Multipart message — attachments visible in raw view] — view raw
On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura <zfigura@codeweavers.com> wrote: > On 9/5/21 11:19 AM, Stephen Kitt wrote: > > On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura > > <zfigura@codeweavers.com> wrote: > >> I'm a contributor to the Wine project. To summarize the following mail, > >> Wine needs special versions of some of its normal dependencies, such as > >> libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > >> sending out a mail to major distributions in order to get some feedback > >> from our packagers on how these should be built and packaged. > >> > >> For a long time Wine has built all of its Win32 libraries (DLLs and > >> EXEs) as ELF binaries. For various reasons related to application > >> compatibility, we have started building our binaries as PE instead, > >> using the MinGW cross-compiler. It is our intent to expand this to some > >> of our dependencies as well. The list of dependencies that we intend to > >> build using MinGW is not quite fixed yet, but we expect it to include > >> and be mostly limited to the following: > >> > >> * libvkd3d > >> * libFAudio > >> * libgnutls > >> * zlib (currently included via manual source import) > >> * libmpg123 > >> * libgsm > >> * libpng > >> * libjpeg-turbo > >> * libtiff > >> * libfreetype > >> * liblcms2 > >> * jxrlib > >> > >> and dependencies of the above packages (not including CRT dependencies, > >> which Wine provides). > >> > >> There is currently some internal discussion about how these dependencies > >> should be built and linked. There are essentially three questions I see > >> that need to be resolved, and while these resolutions have a significant > >> impact on the Wine building and development process, they also have an > >> impact on distributions, and accordingly I'd like to get input from our > >> packagers to ensure that their considerations are accurately taken into > >> account. > >> > >> (1) Should we build via source import, or link statically, or > >> dynamically? > >> > >> Source imports are dispreferred by Debian [1], on the grounds that they > >> cause duplication of libraries on disk and in memory, and make it harder > >> to handle security updates. They also make building and bisecting > >> harder. Static libraries don't seem to be expressly discouraged, but > >> share most of the same downsides (see also [2]). > >> > >> Note however that if they are linked dynamically, we need to make sure > >> that we load our packages instead of MinGW builds of open-source > >> libraries with applications ship with. There's some internal discussion > >> about whether this is possible while using "stock" builds of MinGW > >> libraries, but, due to the way the Win32 loader works, we will probably > >> need to compile each library, and its dependencies, with a separate, > >> wine-specific name, e.g. "libwinefreetype-6.dll" and > >> "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note > >> that all we actually need to change is the name; we don't need to patch > >> the source. > > > > Assuming Debian provides the dependencies (which is currently true only > > for zlib), this could be handled in packaging by providing symlinks, > > couldn’t it? Not in the Wine prefixes, but elsewhere. > > Almost :-/ > > Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. > The problem is that libwinefreetype-1.dll is still going to link to > libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. Ah yes, I hadn’t thought it through. So really Wine needs its own parallel ecosystem of DLLs in any case, which effectively means building them along with Wine (from Wine’s perspective, ignoring distribution constraints and preferences). > > This also works in Debian: > > > > $ sudo apt install libz-mingw-w64-dev mingw-w64-tools > > $ x86_64-w64-mingw32-pkg-config --libs zlib > > -L/usr/x86_64-w64-mingw32/lib -lz > > Ah, cool, I looked for that and somehow missed it. > > Note that's the wrong path for dynamic libraries, though; those go in > /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe > hardcoding a list won't be too painful, though... In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why the .pc file points there. But as you point out, that doesn’t help at runtime. Also, having pkg-config support doesn’t really help with a parallel set of DLLs, does it? > >> For what it's worth, the current proposed solution (which has the > >> support of the Wine maintainer) involves source imports and submodules. > >> There's probably room for changing our approach even after things are > >> committed, but I'd still like to get early feedback from distributions, > >> and make sure that their interests are accurately represented, before we > >> commit. > > > > Realistically, I think this is the best approach for now. As Debian adds > > support for PE libraries, we can replace the source imports in the Wine > > source code; this is done in many other Debian packages for projects which > > vendor dependencies. I still think this is true. With requirement for a Wine-specific set of DLLs, improving the situation in Debian would involve supporting source build-dependencies, i.e. being able to say that a given package wants access to the source code of another package. That’s something that’s been brought up previously, and is worked around by providing binary packages containing package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t really upstream’s concern, in that it’s a Debian implementation detail. Going back to your original three questions, I think that the best approach for you as upstream is to focus on providing a complete set of source code (including dependencies) which works, and to make it friendlier to distributions, make the build process capable of handling alternative locations for the dependencies’ source code or even build artifacts. (This has a number of knock-on effects — in particular, you should ensure that the upstream source code for all your dependencies works with Wine, i.e. that Wine doesn’t require Wine-specific patches to any of its dependencies.) Given how varied MinGW-w64 handling is in different distributions, pushing things further risks making it easier for one distribution and harder for the others... Regards, Stephen
[toc] | [prev] | [next] | [standalone]
| From | Bastien ROUCARIES <roucaries.bastien@gmail.com> |
|---|---|
| Date | 2021-09-06 10:20 +0200 |
| Message-ID | <CUi1c-4dD-7@gated-at.bofh.it> |
| In reply to | #101556 |
Le lun. 6 sept. 2021 à 06:57, Stephen Kitt <skitt@debian.org> a écrit : > > On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura <zfigura@codeweavers.com> > wrote: > > On 9/5/21 11:19 AM, Stephen Kitt wrote: > > > On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura > > > <zfigura@codeweavers.com> wrote: > > >> I'm a contributor to the Wine project. To summarize the following mail, > > >> Wine needs special versions of some of its normal dependencies, such as > > >> libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm > > >> sending out a mail to major distributions in order to get some feedback > > >> from our packagers on how these should be built and packaged. > > >> > > >> For a long time Wine has built all of its Win32 libraries (DLLs and > > >> EXEs) as ELF binaries. For various reasons related to application > > >> compatibility, we have started building our binaries as PE instead, > > >> using the MinGW cross-compiler. It is our intent to expand this to some > > >> of our dependencies as well. The list of dependencies that we intend to > > >> build using MinGW is not quite fixed yet, but we expect it to include > > >> and be mostly limited to the following: > > >> > > >> * libvkd3d > > >> * libFAudio > > >> * libgnutls > > >> * zlib (currently included via manual source import) > > >> * libmpg123 > > >> * libgsm > > >> * libpng > > >> * libjpeg-turbo > > >> * libtiff > > >> * libfreetype > > >> * liblcms2 > > >> * jxrlib > > >> > > >> and dependencies of the above packages (not including CRT dependencies, > > >> which Wine provides). > > >> > > >> There is currently some internal discussion about how these dependencies > > >> should be built and linked. There are essentially three questions I see > > >> that need to be resolved, and while these resolutions have a significant > > >> impact on the Wine building and development process, they also have an > > >> impact on distributions, and accordingly I'd like to get input from our > > >> packagers to ensure that their considerations are accurately taken into > > >> account. > > >> > > >> (1) Should we build via source import, or link statically, or > > >> dynamically? > > >> > > >> Source imports are dispreferred by Debian [1], on the grounds that they > > >> cause duplication of libraries on disk and in memory, and make it harder > > >> to handle security updates. They also make building and bisecting > > >> harder. Static libraries don't seem to be expressly discouraged, but > > >> share most of the same downsides (see also [2]). > > >> > > >> Note however that if they are linked dynamically, we need to make sure > > >> that we load our packages instead of MinGW builds of open-source > > >> libraries with applications ship with. There's some internal discussion > > >> about whether this is possible while using "stock" builds of MinGW > > >> libraries, but, due to the way the Win32 loader works, we will probably > > >> need to compile each library, and its dependencies, with a separate, > > >> wine-specific name, e.g. "libwinefreetype-6.dll" and > > >> "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note > > >> that all we actually need to change is the name; we don't need to patch > > >> the source. > > > > > > Assuming Debian provides the dependencies (which is currently true only > > > for zlib), this could be handled in packaging by providing symlinks, > > > couldn’t it? Not in the Wine prefixes, but elsewhere. > > > > Almost :-/ > > > > Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. > > The problem is that libwinefreetype-1.dll is still going to link to > > libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. > > Ah yes, I hadn’t thought it through. So really Wine needs its own parallel > ecosystem of DLLs in any case, which effectively means building them along > with Wine (from Wine’s perspective, ignoring distribution constraints and > preferences). I will really prefer something less laborious. Like using flexdll for your own library. Moreover using a custom loader will decrease the patching burden for debian side, so a win win. Bastien > > > This also works in Debian: > > > > > > $ sudo apt install libz-mingw-w64-dev mingw-w64-tools > > > $ x86_64-w64-mingw32-pkg-config --libs zlib > > > -L/usr/x86_64-w64-mingw32/lib -lz > > > > Ah, cool, I looked for that and somehow missed it. > > > > Note that's the wrong path for dynamic libraries, though; those go in > > /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe > > hardcoding a list won't be too painful, though... > > In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why > the .pc file points there. But as you point out, that doesn’t help at > runtime. > > Also, having pkg-config support doesn’t really help with a parallel set of > DLLs, does it? > > > >> For what it's worth, the current proposed solution (which has the > > >> support of the Wine maintainer) involves source imports and submodules. > > >> There's probably room for changing our approach even after things are > > >> committed, but I'd still like to get early feedback from distributions, > > >> and make sure that their interests are accurately represented, before we > > >> commit. > > > > > > Realistically, I think this is the best approach for now. As Debian adds > > > support for PE libraries, we can replace the source imports in the Wine > > > source code; this is done in many other Debian packages for projects which > > > vendor dependencies. > > I still think this is true. With requirement for a Wine-specific set of DLLs, > improving the situation in Debian would involve supporting source > build-dependencies, i.e. being able to say that a given package wants access > to the source code of another package. That’s something that’s been brought > up previously, and is worked around by providing binary packages containing > package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t > really upstream’s concern, in that it’s a Debian implementation detail. > > Going back to your original three questions, I think that the best approach > for you as upstream is to focus on providing a complete set of source code > (including dependencies) which works, and to make it friendlier to > distributions, make the build process capable of handling alternative > locations for the dependencies’ source code or even build artifacts. (This > has a number of knock-on effects — in particular, you should ensure that the > upstream source code for all your dependencies works with Wine, i.e. that > Wine doesn’t require Wine-specific patches to any of its dependencies.) > > Given how varied MinGW-w64 handling is in different distributions, pushing > things further risks making it easier for one distribution and harder for the > others... > > Regards, > > Stephen
[toc] | [prev] | [next] | [standalone]
| From | Zebediah Figura <zfigura@codeweavers.com> |
|---|---|
| Date | 2021-09-06 18:40 +0200 |
| Message-ID | <CUpP3-OE-9@gated-at.bofh.it> |
| In reply to | #101556 |
On 9/6/21 1:57 AM, Stephen Kitt wrote: > On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura <zfigura@codeweavers.com> > wrote: >> On 9/5/21 11:19 AM, Stephen Kitt wrote: >>> On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura >>> <zfigura@codeweavers.com> wrote: >>>> I'm a contributor to the Wine project. To summarize the following mail, >>>> Wine needs special versions of some of its normal dependencies, such as >>>> libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm >>>> sending out a mail to major distributions in order to get some feedback >>>> from our packagers on how these should be built and packaged. >>>> >>>> For a long time Wine has built all of its Win32 libraries (DLLs and >>>> EXEs) as ELF binaries. For various reasons related to application >>>> compatibility, we have started building our binaries as PE instead, >>>> using the MinGW cross-compiler. It is our intent to expand this to some >>>> of our dependencies as well. The list of dependencies that we intend to >>>> build using MinGW is not quite fixed yet, but we expect it to include >>>> and be mostly limited to the following: >>>> >>>> * libvkd3d >>>> * libFAudio >>>> * libgnutls >>>> * zlib (currently included via manual source import) >>>> * libmpg123 >>>> * libgsm >>>> * libpng >>>> * libjpeg-turbo >>>> * libtiff >>>> * libfreetype >>>> * liblcms2 >>>> * jxrlib >>>> >>>> and dependencies of the above packages (not including CRT dependencies, >>>> which Wine provides). >>>> >>>> There is currently some internal discussion about how these dependencies >>>> should be built and linked. There are essentially three questions I see >>>> that need to be resolved, and while these resolutions have a significant >>>> impact on the Wine building and development process, they also have an >>>> impact on distributions, and accordingly I'd like to get input from our >>>> packagers to ensure that their considerations are accurately taken into >>>> account. >>>> >>>> (1) Should we build via source import, or link statically, or >>>> dynamically? >>>> >>>> Source imports are dispreferred by Debian [1], on the grounds that they >>>> cause duplication of libraries on disk and in memory, and make it harder >>>> to handle security updates. They also make building and bisecting >>>> harder. Static libraries don't seem to be expressly discouraged, but >>>> share most of the same downsides (see also [2]). >>>> >>>> Note however that if they are linked dynamically, we need to make sure >>>> that we load our packages instead of MinGW builds of open-source >>>> libraries with applications ship with. There's some internal discussion >>>> about whether this is possible while using "stock" builds of MinGW >>>> libraries, but, due to the way the Win32 loader works, we will probably >>>> need to compile each library, and its dependencies, with a separate, >>>> wine-specific name, e.g. "libwinefreetype-6.dll" and >>>> "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note >>>> that all we actually need to change is the name; we don't need to patch >>>> the source. >>> >>> Assuming Debian provides the dependencies (which is currently true only >>> for zlib), this could be handled in packaging by providing symlinks, >>> couldn’t it? Not in the Wine prefixes, but elsewhere. >> >> Almost :-/ >> >> Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. >> The problem is that libwinefreetype-1.dll is still going to link to >> libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. > > Ah yes, I hadn’t thought it through. So really Wine needs its own parallel > ecosystem of DLLs in any case, which effectively means building them along > with Wine (from Wine’s perspective, ignoring distribution constraints and > preferences). > >>> This also works in Debian: >>> >>> $ sudo apt install libz-mingw-w64-dev mingw-w64-tools >>> $ x86_64-w64-mingw32-pkg-config --libs zlib >>> -L/usr/x86_64-w64-mingw32/lib -lz >> >> Ah, cool, I looked for that and somehow missed it. >> >> Note that's the wrong path for dynamic libraries, though; those go in >> /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe >> hardcoding a list won't be too painful, though... > > In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why > the .pc file points there. But as you point out, that doesn’t help at > runtime. > > Also, having pkg-config support doesn’t really help with a parallel set of > DLLs, does it? I mean... eh. In theory you could say "here's a library called libwinefreetype, and to find it you do `pkg-config --cflags --libs libwinefreetype`", but that does strike me as more than a little janky. > >>>> For what it's worth, the current proposed solution (which has the >>>> support of the Wine maintainer) involves source imports and submodules. >>>> There's probably room for changing our approach even after things are >>>> committed, but I'd still like to get early feedback from distributions, >>>> and make sure that their interests are accurately represented, before we >>>> commit. >>> >>> Realistically, I think this is the best approach for now. As Debian adds >>> support for PE libraries, we can replace the source imports in the Wine >>> source code; this is done in many other Debian packages for projects which >>> vendor dependencies. > > I still think this is true. With requirement for a Wine-specific set of DLLs, > improving the situation in Debian would involve supporting source > build-dependencies, i.e. being able to say that a given package wants access > to the source code of another package. That’s something that’s been brought > up previously, and is worked around by providing binary packages containing > package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t > really upstream’s concern, in that it’s a Debian implementation detail. > > Going back to your original three questions, I think that the best approach > for you as upstream is to focus on providing a complete set of source code > (including dependencies) which works, and to make it friendlier to > distributions, make the build process capable of handling alternative > locations for the dependencies’ source code or even build artifacts. (This > has a number of knock-on effects — in particular, you should ensure that the > upstream source code for all your dependencies works with Wine, i.e. that > Wine doesn’t require Wine-specific patches to any of its dependencies.) > > Given how varied MinGW-w64 handling is in different distributions, pushing > things further risks making it easier for one distribution and harder for the > others... Thanks for the detailed response! It's probably worth pointing out that: (1) if we use static linking, we should be able to use distribution libraries unmodified. Of course, static linking comes with its own downsides... (2) Like I mentioned or hinted at in my original email, it *may* be possible to use distribution dynamic libraries unmodified. It's not clear that it can be done without breaking compatibility, but if it can, would that change anything? ἔρρωσθε, Zebediah
[toc] | [prev] | [next] | [standalone]
| From | Bastien ROUCARIES <roucaries.bastien@gmail.com> |
|---|---|
| Date | 2021-09-06 21:00 +0200 |
| Message-ID | <CUs0z-27a-7@gated-at.bofh.it> |
| In reply to | #101559 |
[Multipart message — attachments visible in raw view] — view raw
Le lun. 6 sept. 2021 à 18:36, Zebediah Figura <zfigura@codeweavers.com> a écrit : > On 9/6/21 1:57 AM, Stephen Kitt wrote: > > On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura < > zfigura@codeweavers.com> > > wrote: > >> On 9/5/21 11:19 AM, Stephen Kitt wrote: > >>> On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura > >>> <zfigura@codeweavers.com> wrote: > >>>> I'm a contributor to the Wine project. To summarize the following > mail, > >>>> Wine needs special versions of some of its normal dependencies, such > as > >>>> libfreetype and libgnutls, built using the MinGW cross-compiler, and > I'm > >>>> sending out a mail to major distributions in order to get some > feedback > >>>> from our packagers on how these should be built and packaged. > >>>> > >>>> For a long time Wine has built all of its Win32 libraries (DLLs and > >>>> EXEs) as ELF binaries. For various reasons related to application > >>>> compatibility, we have started building our binaries as PE instead, > >>>> using the MinGW cross-compiler. It is our intent to expand this to > some > >>>> of our dependencies as well. The list of dependencies that we intend > to > >>>> build using MinGW is not quite fixed yet, but we expect it to include > >>>> and be mostly limited to the following: > >>>> > >>>> * libvkd3d > >>>> * libFAudio > >>>> * libgnutls > >>>> * zlib (currently included via manual source import) > >>>> * libmpg123 > >>>> * libgsm > >>>> * libpng > >>>> * libjpeg-turbo > >>>> * libtiff > >>>> * libfreetype > >>>> * liblcms2 > >>>> * jxrlib > >>>> > >>>> and dependencies of the above packages (not including CRT > dependencies, > >>>> which Wine provides). > >>>> > >>>> There is currently some internal discussion about how these > dependencies > >>>> should be built and linked. There are essentially three questions I > see > >>>> that need to be resolved, and while these resolutions have a > significant > >>>> impact on the Wine building and development process, they also have an > >>>> impact on distributions, and accordingly I'd like to get input from > our > >>>> packagers to ensure that their considerations are accurately taken > into > >>>> account. > >>>> > >>>> (1) Should we build via source import, or link statically, or > >>>> dynamically? > >>>> > >>>> Source imports are dispreferred by Debian [1], on the grounds that > they > >>>> cause duplication of libraries on disk and in memory, and make it > harder > >>>> to handle security updates. They also make building and bisecting > >>>> harder. Static libraries don't seem to be expressly discouraged, but > >>>> share most of the same downsides (see also [2]). > >>>> > >>>> Note however that if they are linked dynamically, we need to make sure > >>>> that we load our packages instead of MinGW builds of open-source > >>>> libraries with applications ship with. There's some internal > discussion > >>>> about whether this is possible while using "stock" builds of MinGW > >>>> libraries, but, due to the way the Win32 loader works, we will > probably > >>>> need to compile each library, and its dependencies, with a separate, > >>>> wine-specific name, e.g. "libwinefreetype-6.dll" and > >>>> "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. > Note > >>>> that all we actually need to change is the name; we don't need to > patch > >>>> the source. > >>> > >>> Assuming Debian provides the dependencies (which is currently true only > >>> for zlib), this could be handled in packaging by providing symlinks, > >>> couldn’t it? Not in the Wine prefixes, but elsewhere. > >> > >> Almost :-/ > >> > >> Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. > >> The problem is that libwinefreetype-1.dll is still going to link to > >> libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. > > > > Ah yes, I hadn’t thought it through. So really Wine needs its own > parallel > > ecosystem of DLLs in any case, which effectively means building them > along > > with Wine (from Wine’s perspective, ignoring distribution constraints and > > preferences). > > > >>> This also works in Debian: > >>> > >>> $ sudo apt install libz-mingw-w64-dev mingw-w64-tools > >>> $ x86_64-w64-mingw32-pkg-config --libs zlib > >>> -L/usr/x86_64-w64-mingw32/lib -lz > >> > >> Ah, cool, I looked for that and somehow missed it. > >> > >> Note that's the wrong path for dynamic libraries, though; those go in > >> /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe > >> hardcoding a list won't be too painful, though... > > > > In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why > > the .pc file points there. But as you point out, that doesn’t help at > > runtime. > > > > Also, having pkg-config support doesn’t really help with a parallel set > of > > DLLs, does it? > > I mean... eh. In theory you could say "here's a library called > libwinefreetype, and to find it you do `pkg-config --cflags --libs > libwinefreetype`", but that does strike me as more than a little janky. > > > > >>>> For what it's worth, the current proposed solution (which has the > >>>> support of the Wine maintainer) involves source imports and > submodules. > >>>> There's probably room for changing our approach even after things are > >>>> committed, but I'd still like to get early feedback from > distributions, > >>>> and make sure that their interests are accurately represented, before > we > >>>> commit. > >>> > >>> Realistically, I think this is the best approach for now. As Debian > adds > >>> support for PE libraries, we can replace the source imports in the Wine > >>> source code; this is done in many other Debian packages for projects > which > >>> vendor dependencies. > > > > I still think this is true. With requirement for a Wine-specific set of > DLLs, > > improving the situation in Debian would involve supporting source > > build-dependencies, i.e. being able to say that a given package wants > access > > to the source code of another package. That’s something that’s been > brought > > up previously, and is worked around by providing binary packages > containing > > package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t > > really upstream’s concern, in that it’s a Debian implementation detail. > > > > Going back to your original three questions, I think that the best > approach > > for you as upstream is to focus on providing a complete set of source > code > > (including dependencies) which works, and to make it friendlier to > > distributions, make the build process capable of handling alternative > > locations for the dependencies’ source code or even build artifacts. > (This > > has a number of knock-on effects — in particular, you should ensure that > the > > upstream source code for all your dependencies works with Wine, i.e. that > > Wine doesn’t require Wine-specific patches to any of its dependencies.) > > > > Given how varied MinGW-w64 handling is in different distributions, > pushing > > things further risks making it easier for one distribution and harder > for the > > others... > > Thanks for the detailed response! > > It's probably worth pointing out that: > > (1) if we use static linking, we should be able to use distribution > libraries unmodified. Of course, static linking comes with its own > downsides... > > (2) Like I mentioned or hinted at in my original email, it *may* be > possible to use distribution dynamic libraries unmodified. It's not > clear that it can be done without breaking compatibility, but if it can, > would that change anything? > Yes i really prefer (2) it means we could use multiarch easilly.... Please try to document thé blocking point. > > ἔρρωσθε, > Zebediah > >
[toc] | [prev] | [next] | [standalone]
| From | Zebediah Figura <zfigura@codeweavers.com> |
|---|---|
| Date | 2021-09-07 00:00 +0200 |
| Message-ID | <CUuOK-3QG-5@gated-at.bofh.it> |
| In reply to | #101560 |
On 9/6/21 1:34 PM, Bastien ROUCARIES wrote: > Le lun. 6 sept. 2021 à 18:36, Zebediah Figura <zfigura@codeweavers.com> a > écrit : > >> On 9/6/21 1:57 AM, Stephen Kitt wrote: >>> On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura < >> zfigura@codeweavers.com> >>> wrote: >>>> On 9/5/21 11:19 AM, Stephen Kitt wrote: >>>>> On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura >>>>> <zfigura@codeweavers.com> wrote: >>>>>> I'm a contributor to the Wine project. To summarize the following >> mail, >>>>>> Wine needs special versions of some of its normal dependencies, such >> as >>>>>> libfreetype and libgnutls, built using the MinGW cross-compiler, and >> I'm >>>>>> sending out a mail to major distributions in order to get some >> feedback >>>>>> from our packagers on how these should be built and packaged. >>>>>> >>>>>> For a long time Wine has built all of its Win32 libraries (DLLs and >>>>>> EXEs) as ELF binaries. For various reasons related to application >>>>>> compatibility, we have started building our binaries as PE instead, >>>>>> using the MinGW cross-compiler. It is our intent to expand this to >> some >>>>>> of our dependencies as well. The list of dependencies that we intend >> to >>>>>> build using MinGW is not quite fixed yet, but we expect it to include >>>>>> and be mostly limited to the following: >>>>>> >>>>>> * libvkd3d >>>>>> * libFAudio >>>>>> * libgnutls >>>>>> * zlib (currently included via manual source import) >>>>>> * libmpg123 >>>>>> * libgsm >>>>>> * libpng >>>>>> * libjpeg-turbo >>>>>> * libtiff >>>>>> * libfreetype >>>>>> * liblcms2 >>>>>> * jxrlib >>>>>> >>>>>> and dependencies of the above packages (not including CRT >> dependencies, >>>>>> which Wine provides). >>>>>> >>>>>> There is currently some internal discussion about how these >> dependencies >>>>>> should be built and linked. There are essentially three questions I >> see >>>>>> that need to be resolved, and while these resolutions have a >> significant >>>>>> impact on the Wine building and development process, they also have an >>>>>> impact on distributions, and accordingly I'd like to get input from >> our >>>>>> packagers to ensure that their considerations are accurately taken >> into >>>>>> account. >>>>>> >>>>>> (1) Should we build via source import, or link statically, or >>>>>> dynamically? >>>>>> >>>>>> Source imports are dispreferred by Debian [1], on the grounds that >> they >>>>>> cause duplication of libraries on disk and in memory, and make it >> harder >>>>>> to handle security updates. They also make building and bisecting >>>>>> harder. Static libraries don't seem to be expressly discouraged, but >>>>>> share most of the same downsides (see also [2]). >>>>>> >>>>>> Note however that if they are linked dynamically, we need to make sure >>>>>> that we load our packages instead of MinGW builds of open-source >>>>>> libraries with applications ship with. There's some internal >> discussion >>>>>> about whether this is possible while using "stock" builds of MinGW >>>>>> libraries, but, due to the way the Win32 loader works, we will >> probably >>>>>> need to compile each library, and its dependencies, with a separate, >>>>>> wine-specific name, e.g. "libwinefreetype-6.dll" and >>>>>> "libwineharfbuzz.dll". For a detailed explantion see footnote [3]. >> Note >>>>>> that all we actually need to change is the name; we don't need to >> patch >>>>>> the source. >>>>> >>>>> Assuming Debian provides the dependencies (which is currently true only >>>>> for zlib), this could be handled in packaging by providing symlinks, >>>>> couldn’t it? Not in the Wine prefixes, but elsewhere. >>>> >>>> Almost :-/ >>>> >>>> Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. >>>> The problem is that libwinefreetype-1.dll is still going to link to >>>> libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll. >>> >>> Ah yes, I hadn’t thought it through. So really Wine needs its own >> parallel >>> ecosystem of DLLs in any case, which effectively means building them >> along >>> with Wine (from Wine’s perspective, ignoring distribution constraints and >>> preferences). >>> >>>>> This also works in Debian: >>>>> >>>>> $ sudo apt install libz-mingw-w64-dev mingw-w64-tools >>>>> $ x86_64-w64-mingw32-pkg-config --libs zlib >>>>> -L/usr/x86_64-w64-mingw32/lib -lz >>>> >>>> Ah, cool, I looked for that and somehow missed it. >>>> >>>> Note that's the wrong path for dynamic libraries, though; those go in >>>> /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe >>>> hardcoding a list won't be too painful, though... >>> >>> In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why >>> the .pc file points there. But as you point out, that doesn’t help at >>> runtime. >>> >>> Also, having pkg-config support doesn’t really help with a parallel set >> of >>> DLLs, does it? >> >> I mean... eh. In theory you could say "here's a library called >> libwinefreetype, and to find it you do `pkg-config --cflags --libs >> libwinefreetype`", but that does strike me as more than a little janky. >> >>> >>>>>> For what it's worth, the current proposed solution (which has the >>>>>> support of the Wine maintainer) involves source imports and >> submodules. >>>>>> There's probably room for changing our approach even after things are >>>>>> committed, but I'd still like to get early feedback from >> distributions, >>>>>> and make sure that their interests are accurately represented, before >> we >>>>>> commit. >>>>> >>>>> Realistically, I think this is the best approach for now. As Debian >> adds >>>>> support for PE libraries, we can replace the source imports in the Wine >>>>> source code; this is done in many other Debian packages for projects >> which >>>>> vendor dependencies. >>> >>> I still think this is true. With requirement for a Wine-specific set of >> DLLs, >>> improving the situation in Debian would involve supporting source >>> build-dependencies, i.e. being able to say that a given package wants >> access >>> to the source code of another package. That’s something that’s been >> brought >>> up previously, and is worked around by providing binary packages >> containing >>> package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t >>> really upstream’s concern, in that it’s a Debian implementation detail. >>> >>> Going back to your original three questions, I think that the best >> approach >>> for you as upstream is to focus on providing a complete set of source >> code >>> (including dependencies) which works, and to make it friendlier to >>> distributions, make the build process capable of handling alternative >>> locations for the dependencies’ source code or even build artifacts. >> (This >>> has a number of knock-on effects — in particular, you should ensure that >> the >>> upstream source code for all your dependencies works with Wine, i.e. that >>> Wine doesn’t require Wine-specific patches to any of its dependencies.) >>> >>> Given how varied MinGW-w64 handling is in different distributions, >> pushing >>> things further risks making it easier for one distribution and harder >> for the >>> others... >> >> Thanks for the detailed response! >> >> It's probably worth pointing out that: >> >> (1) if we use static linking, we should be able to use distribution >> libraries unmodified. Of course, static linking comes with its own >> downsides... >> >> (2) Like I mentioned or hinted at in my original email, it *may* be >> possible to use distribution dynamic libraries unmodified. It's not >> clear that it can be done without breaking compatibility, but if it can, >> would that change anything? >> > > Yes i really prefer (2) it means we could use multiarch easilly.... > > Please try to document thé blocking point. So this is mostly documented in the footnote to my original mail, but I'll copy it here for reference. The basic problem is that applications can and often do ship with PE builds of cross-platform libraries. These libraries can be ahead of Wine's system libraries, behind them, or even built with custom patches. Accordingly we really don't want to load "our" freetype in place of "their" freetype, or "theirs" in place of "ours". But because of the way the Win32 loader works, you can generally only have one DLL of a given name loaded in a process, and further attempts to dlopen() [as it were] "libfreetype-6.dll" will return the handle to the already loaded library, potentially breaking either Wine or the application. The easiest way to avoid ever doing that is by giving each library a different name (or by not using dynamic linking at all.) Failing that, we need a way to determine whether a library is a system dependency—and thus can *only* be loaded from the system DLL path—or an application dependency—and thus should *never* be loaded from the system DLL path. I think this can be done, but it gets tricky, and it's not clear that there aren't going to be subtle problems from having two different DLLs with the same name loaded in the same process. Especially keeping in mind that Windows programs do absolutely insane things when it comes to depending on OS internals.
[toc] | [prev] | [next] | [standalone]
| From | Paul Wise <pabs@debian.org> |
|---|---|
| Date | 2021-09-07 02:50 +0200 |
| Message-ID | <CUxtf-5tR-1@gated-at.bofh.it> |
| In reply to | #101561 |
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > The basic problem is that applications can and often do ship with PE > builds of cross-platform libraries. These libraries can be ahead of > Wine's system libraries, behind them, or even built with custom patches. > Accordingly we really don't want to load "our" freetype in place of > "their" freetype, or "theirs" in place of "ours". But because of the way > the Win32 loader works, you can generally only have one DLL of a given > name loaded in a process, and further attempts to dlopen() [as it were] > "libfreetype-6.dll" will return the handle to the already loaded > library, potentially breaking either Wine or the application. I don't know the details here, but my immediate thought when reading this is that you need some kind of namespace. I then found that linker namespaces are a thing, perhaps they would provide the solution for you. It sounds like the OpenGL shim loader solution listed on the glibc wiki might work for your use-case. Or perhaps the LD_AUDIT feature would also work. https://www.sourceware.org/glibc/wiki/LinkerNamespaces -- bye, pabs https://wiki.debian.org/PaulWise
[toc] | [prev] | [next] | [standalone]
| From | "Bastien Roucariès" <roucaries.bastien@gmail.com> |
|---|---|
| Date | 2021-09-07 12:40 +0200 |
| Message-ID | <CUGGd-3cU-7@gated-at.bofh.it> |
| In reply to | #101562 |
Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : > On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > > The basic problem is that applications can and often do ship with PE > > builds of cross-platform libraries. These libraries can be ahead of > > Wine's system libraries, behind them, or even built with custom patches. > > Accordingly we really don't want to load "our" freetype in place of > > "their" freetype, or "theirs" in place of "ours". But because of the way > > the Win32 loader works, you can generally only have one DLL of a given > > name loaded in a process, and further attempts to dlopen() [as it were] > > "libfreetype-6.dll" will return the handle to the already loaded > > library, potentially breaking either Wine or the application. > > I don't know the details here, but my immediate thought when reading > this is that you need some kind of namespace. I then found that linker > namespaces are a thing, perhaps they would provide the solution for > you. It sounds like the OpenGL shim loader solution listed on the > glibc wiki might work for your use-case. Or perhaps the LD_AUDIT > feature would also work. > > https://www.sourceware.org/glibc/wiki/LinkerNamespaces I agree with pabs, implementing name space is a good solution and will allow to be distrib friendly. Moreover it will be best if marking a library as wine system one could be done post build. We will prefer to not modify upstream (like libpng) source. It is easier for us to apply small patch to upstream, or better in your case to add a post link linker script or even some compiler flag that will add some notes section to dll in order to mark it as use namespace system, or so on. The problem on windows side is well known as dll hell Renaming of the lib could be done post install. Moreover a crazy idea why not renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll (note the version is manadaroty) Therefore we could use upstream source. Then add a small linker script that will rewrite the depends of libpng-2.0.0.sll to .sll file (aka binary sed s/.dll/.sll/g). then add a small wraper (shim) like said pab that load the .sll library, the best pratice will be a command tool that take the name of the sll and create magically the dll Therefore from distrib point of side we only changes the way we build the package, it could be automated, it is scalable and it does not need to patch package by package. Only to add some linker script magic Bastien
[toc] | [prev] | [next] | [standalone]
| From | Zebediah Figura <zfigura@codeweavers.com> |
|---|---|
| Date | 2021-09-07 17:50 +0200 |
| Message-ID | <CULwd-629-1@gated-at.bofh.it> |
| In reply to | #101563 |
On 9/7/21 5:16 AM, Bastien Roucariès wrote: > Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : >> On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: >>> The basic problem is that applications can and often do ship with PE >>> builds of cross-platform libraries. These libraries can be ahead of >>> Wine's system libraries, behind them, or even built with custom patches. >>> Accordingly we really don't want to load "our" freetype in place of >>> "their" freetype, or "theirs" in place of "ours". But because of the way >>> the Win32 loader works, you can generally only have one DLL of a given >>> name loaded in a process, and further attempts to dlopen() [as it were] >>> "libfreetype-6.dll" will return the handle to the already loaded >>> library, potentially breaking either Wine or the application. >> >> I don't know the details here, but my immediate thought when reading >> this is that you need some kind of namespace. I then found that linker >> namespaces are a thing, perhaps they would provide the solution for >> you. It sounds like the OpenGL shim loader solution listed on the >> glibc wiki might work for your use-case. Or perhaps the LD_AUDIT >> feature would also work. >> >> https://www.sourceware.org/glibc/wiki/LinkerNamespaces > I agree with pabs, implementing name space is a good solution > and will allow to be distrib friendly. > > Moreover it will be best if marking a library as wine system one could be done > post build. We will prefer to not modify upstream (like libpng) source. > > It is easier for us to apply small patch to upstream, or better in your case > to add a post link linker script or even some compiler flag that will add some > notes section to dll in order to mark it as use namespace system, or so on. > > The problem on windows side is well known as dll hell > > Renaming of the lib could be done post install. Moreover a crazy idea why not > renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll (note the > version is manadaroty) Therefore we could use upstream source. Then add a > small linker script that will rewrite the depends of libpng-2.0.0.sll to > .sll file (aka binary sed s/.dll/.sll/g). > > then add a small wraper (shim) like said pab that load the .sll library, the > best pratice will be a command tool that take the name of the sll and create > magically the dll > > Therefore from distrib point of side we only changes the way we build the > package, it could be automated, it is scalable and it does not need to patch > package by package. Only to add some linker script magic The problem isn't particularly about detecting what is and isn't a system library; I think we can come up with a reliable heuristic for that, without even needing linker namespaces or anything. The outstanding problem seems to be more about potentially breaking applications because they see two identically named DLLs loaded in the same process. Applications can and do trawl the internal loader state, although the Win32 loader also exposes some APIs so they don't even have to. It may also be that the aforementioned heuristic is too hacky to be accepted upstream into Wine.
[toc] | [prev] | [next] | [standalone]
| From | Bastien ROUCARIES <roucaries.bastien@gmail.com> |
|---|---|
| Date | 2021-09-07 19:30 +0200 |
| Message-ID | <CUN4Z-73U-5@gated-at.bofh.it> |
| In reply to | #101566 |
[Multipart message — attachments visible in raw view] — view raw
I disagree. Le mar. 7 sept. 2021 à 17:48, Zebediah Figura <zfigura@codeweavers.com> a écrit : > On 9/7/21 5:16 AM, Bastien Roucariès wrote: > > Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : > >> On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > >>> The basic problem is that applications can and often do ship with PE > >>> builds of cross-platform libraries. These libraries can be ahead of > >>> Wine's system libraries, behind them, or even built with custom > patches. > >>> Accordingly we really don't want to load "our" freetype in place of > >>> "their" freetype, or "theirs" in place of "ours". But because of the > way > >>> the Win32 loader works, you can generally only have one DLL of a given > >>> name loaded in a process, and further attempts to dlopen() [as it were] > >>> "libfreetype-6.dll" will return the handle to the already loaded > >>> library, potentially breaking either Wine or the application. > >> > >> I don't know the details here, but my immediate thought when reading > >> this is that you need some kind of namespace. I then found that linker > >> namespaces are a thing, perhaps they would provide the solution for > >> you. It sounds like the OpenGL shim loader solution listed on the > >> glibc wiki might work for your use-case. Or perhaps the LD_AUDIT > >> feature would also work. > >> > >> https://www.sourceware.org/glibc/wiki/LinkerNamespaces > > I agree with pabs, implementing name space is a good solution > > and will allow to be distrib friendly. > > > > Moreover it will be best if marking a library as wine system one could > be done > > post build. We will prefer to not modify upstream (like libpng) source. > > > > It is easier for us to apply small patch to upstream, or better in your > case > > to add a post link linker script or even some compiler flag that will > add some > > notes section to dll in order to mark it as use namespace system, or so > on. > > > > The problem on windows side is well known as dll hell > > > > Renaming of the lib could be done post install. Moreover a crazy idea > why not > > renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll > (note the > > version is manadaroty) Therefore we could use upstream source. Then add > a > > small linker script that will rewrite the depends of libpng-2.0.0.sll > to > > .sll file (aka binary sed s/.dll/.sll/g). > > > > then add a small wraper (shim) like said pab that load the .sll library, > the > > best pratice will be a command tool that take the name of the sll and > create > > magically the dll > > > > Therefore from distrib point of side we only changes the way we build the > > package, it could be automated, it is scalable and it does not need to > patch > > package by package. Only to add some linker script magic > > The problem isn't particularly about detecting what is and isn't a > system library; I think we can come up with a reliable heuristic for > that, without even needing linker namespaces or anything. > > The outstanding problem seems to be more about potentially breaking > applications because they see two identically named DLLs loaded in the > same process. Applications can and do trawl the internal loader state, > although the Win32 loader also exposes some APIs so they don't even have > to. > > It may also be that the aforementioned heuristic is too hacky to be > accepted upstream into Wine. > I do not propose to change the loader, i propose to change the link or post link stage of system library. You could rename the lib and change the dépend to new name after build. So you will have an unique name per system lib without changing the way you build these lib. It is équivalent to use a namespace resolved at late build time.
[toc] | [prev] | [next] | [standalone]
| From | Bastien ROUCARIES <roucaries.bastien@gmail.com> |
|---|---|
| Date | 2021-09-07 20:20 +0200 |
| Message-ID | <CUNRn-7zg-3@gated-at.bofh.it> |
| In reply to | #101568 |
[Multipart message — attachments visible in raw view] — view raw
Le mar. 7 sept. 2021 à 19:05, Bastien ROUCARIES <roucaries.bastien@gmail.com> a écrit : > I disagree. > > Le mar. 7 sept. 2021 à 17:48, Zebediah Figura <zfigura@codeweavers.com> a > écrit : > >> On 9/7/21 5:16 AM, Bastien Roucariès wrote: >> > Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : >> >> On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: >> >>> The basic problem is that applications can and often do ship with PE >> >>> builds of cross-platform libraries. These libraries can be ahead of >> >>> Wine's system libraries, behind them, or even built with custom >> patches. >> >>> Accordingly we really don't want to load "our" freetype in place of >> >>> "their" freetype, or "theirs" in place of "ours". But because of the >> way >> >>> the Win32 loader works, you can generally only have one DLL of a given >> >>> name loaded in a process, and further attempts to dlopen() [as it >> were] >> >>> "libfreetype-6.dll" will return the handle to the already loaded >> >>> library, potentially breaking either Wine or the application. >> >> >> >> I don't know the details here, but my immediate thought when reading >> >> this is that you need some kind of namespace. I then found that linker >> >> namespaces are a thing, perhaps they would provide the solution for >> >> you. It sounds like the OpenGL shim loader solution listed on the >> >> glibc wiki might work for your use-case. Or perhaps the LD_AUDIT >> >> feature would also work. >> >> >> >> https://www.sourceware.org/glibc/wiki/LinkerNamespaces >> > I agree with pabs, implementing name space is a good solution >> > and will allow to be distrib friendly. >> > >> > Moreover it will be best if marking a library as wine system one could >> be done >> > post build. We will prefer to not modify upstream (like libpng) source. >> > >> > It is easier for us to apply small patch to upstream, or better in your >> case >> > to add a post link linker script or even some compiler flag that will >> add some >> > notes section to dll in order to mark it as use namespace system, or so >> on. >> > >> > The problem on windows side is well known as dll hell >> > >> > Renaming of the lib could be done post install. Moreover a crazy idea >> why not >> > renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll >> (note the >> > version is manadaroty) Therefore we could use upstream source. Then >> add a >> > small linker script that will rewrite the depends of libpng-2.0.0.sll >> to >> > .sll file (aka binary sed s/.dll/.sll/g). >> > >> > then add a small wraper (shim) like said pab that load the .sll >> library, the >> > best pratice will be a command tool that take the name of the sll and >> create >> > magically the dll >> > >> > Therefore from distrib point of side we only changes the way we build >> the >> > package, it could be automated, it is scalable and it does not need to >> patch >> > package by package. Only to add some linker script magic >> >> The problem isn't particularly about detecting what is and isn't a >> system library; I think we can come up with a reliable heuristic for >> that, without even needing linker namespaces or anything. >> >> The outstanding problem seems to be more about potentially breaking >> applications because they see two identically named DLLs loaded in the >> same process. Applications can and do trawl the internal loader state, >> although the Win32 loader also exposes some APIs so they don't even have >> to. >> >> It may also be that the aforementioned heuristic is too hacky to be >> accepted upstream into Wine. >> > I do not propose to change the loader, i propose to change the link or > post link stage of system library. You could rename the lib and change the > dépend to new name after build. So you will have an unique name per system > lib without changing the way you build these lib. > > It is équivalent to use a namespace resolved at late build time. > And it is already coded :) https://pypi.org/project/machomachomangler/ > >
[toc] | [prev] | [next] | [standalone]
| From | Zebediah Figura <zfigura@codeweavers.com> |
|---|---|
| Date | 2021-09-07 23:20 +0200 |
| Message-ID | <CUQFA-UT-11@gated-at.bofh.it> |
| In reply to | #101568 |
On 9/7/21 12:05 PM, Bastien ROUCARIES wrote: > I disagree. > > Le mar. 7 sept. 2021 à 17:48, Zebediah Figura <zfigura@codeweavers.com> a > écrit : > >> On 9/7/21 5:16 AM, Bastien Roucariès wrote: >>> Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : >>>> On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: >>>>> The basic problem is that applications can and often do ship with PE >>>>> builds of cross-platform libraries. These libraries can be ahead of >>>>> Wine's system libraries, behind them, or even built with custom >> patches. >>>>> Accordingly we really don't want to load "our" freetype in place of >>>>> "their" freetype, or "theirs" in place of "ours". But because of the >> way >>>>> the Win32 loader works, you can generally only have one DLL of a given >>>>> name loaded in a process, and further attempts to dlopen() [as it were] >>>>> "libfreetype-6.dll" will return the handle to the already loaded >>>>> library, potentially breaking either Wine or the application. >>>> >>>> I don't know the details here, but my immediate thought when reading >>>> this is that you need some kind of namespace. I then found that linker >>>> namespaces are a thing, perhaps they would provide the solution for >>>> you. It sounds like the OpenGL shim loader solution listed on the >>>> glibc wiki might work for your use-case. Or perhaps the LD_AUDIT >>>> feature would also work. >>>> >>>> https://www.sourceware.org/glibc/wiki/LinkerNamespaces >>> I agree with pabs, implementing name space is a good solution >>> and will allow to be distrib friendly. >>> >>> Moreover it will be best if marking a library as wine system one could >> be done >>> post build. We will prefer to not modify upstream (like libpng) source. >>> >>> It is easier for us to apply small patch to upstream, or better in your >> case >>> to add a post link linker script or even some compiler flag that will >> add some >>> notes section to dll in order to mark it as use namespace system, or so >> on. >>> >>> The problem on windows side is well known as dll hell >>> >>> Renaming of the lib could be done post install. Moreover a crazy idea >> why not >>> renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll >> (note the >>> version is manadaroty) Therefore we could use upstream source. Then add >> a >>> small linker script that will rewrite the depends of libpng-2.0.0.sll >> to >>> .sll file (aka binary sed s/.dll/.sll/g). >>> >>> then add a small wraper (shim) like said pab that load the .sll library, >> the >>> best pratice will be a command tool that take the name of the sll and >> create >>> magically the dll >>> >>> Therefore from distrib point of side we only changes the way we build the >>> package, it could be automated, it is scalable and it does not need to >> patch >>> package by package. Only to add some linker script magic >> >> The problem isn't particularly about detecting what is and isn't a >> system library; I think we can come up with a reliable heuristic for >> that, without even needing linker namespaces or anything. >> >> The outstanding problem seems to be more about potentially breaking >> applications because they see two identically named DLLs loaded in the >> same process. Applications can and do trawl the internal loader state, >> although the Win32 loader also exposes some APIs so they don't even have >> to. >> >> It may also be that the aforementioned heuristic is too hacky to be >> accepted upstream into Wine. >> > I do not propose to change the loader, i propose to change the link or post > link stage of system library. You could rename the lib and change the > dépend to new name after build. So you will have an unique name per system > lib without changing the way you build these lib. > > It is équivalent to use a namespace resolved at late build time. > Ah, so what you're proposing is that we do some sort of objcopy-like patching at runtime, e.g. when copying the dependency into the prefix. It probably won't be easy, but it might be more feasible than modifying the loader. I'll look into it...
[toc] | [prev] | [next] | [standalone]
| From | Bastien ROUCARIES <roucaries.bastien@gmail.com> |
|---|---|
| Date | 2021-09-08 01:50 +0200 |
| Message-ID | <CUT0K-2dY-3@gated-at.bofh.it> |
| In reply to | #101571 |
Le mar. 7 sept. 2021 à 21:16, Zebediah Figura <zfigura@codeweavers.com> a écrit : > > On 9/7/21 12:05 PM, Bastien ROUCARIES wrote: > > I disagree. > > > > Le mar. 7 sept. 2021 à 17:48, Zebediah Figura <zfigura@codeweavers.com> a > > écrit : > > > >> On 9/7/21 5:16 AM, Bastien Roucariès wrote: > >>> Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : > >>>> On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > >>>>> The basic problem is that applications can and often do ship with PE > >>>>> builds of cross-platform libraries. These libraries can be ahead of > >>>>> Wine's system libraries, behind them, or even built with custom > >> patches. > >>>>> Accordingly we really don't want to load "our" freetype in place of > >>>>> "their" freetype, or "theirs" in place of "ours". But because of the > >> way > >>>>> the Win32 loader works, you can generally only have one DLL of a given > >>>>> name loaded in a process, and further attempts to dlopen() [as it were] > >>>>> "libfreetype-6.dll" will return the handle to the already loaded > >>>>> library, potentially breaking either Wine or the application. > >>>> > >>>> I don't know the details here, but my immediate thought when reading > >>>> this is that you need some kind of namespace. I then found that linker > >>>> namespaces are a thing, perhaps they would provide the solution for > >>>> you. It sounds like the OpenGL shim loader solution listed on the > >>>> glibc wiki might work for your use-case. Or perhaps the LD_AUDIT > >>>> feature would also work. > >>>> > >>>> https://www.sourceware.org/glibc/wiki/LinkerNamespaces > >>> I agree with pabs, implementing name space is a good solution > >>> and will allow to be distrib friendly. > >>> > >>> Moreover it will be best if marking a library as wine system one could > >> be done > >>> post build. We will prefer to not modify upstream (like libpng) source. > >>> > >>> It is easier for us to apply small patch to upstream, or better in your > >> case > >>> to add a post link linker script or even some compiler flag that will > >> add some > >>> notes section to dll in order to mark it as use namespace system, or so > >> on. > >>> > >>> The problem on windows side is well known as dll hell > >>> > >>> Renaming of the lib could be done post install. Moreover a crazy idea > >> why not > >>> renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll > >> (note the > >>> version is manadaroty) Therefore we could use upstream source. Then add > >> a > >>> small linker script that will rewrite the depends of libpng-2.0.0.sll > >> to > >>> .sll file (aka binary sed s/.dll/.sll/g). > >>> > >>> then add a small wraper (shim) like said pab that load the .sll library, > >> the > >>> best pratice will be a command tool that take the name of the sll and > >> create > >>> magically the dll > >>> > >>> Therefore from distrib point of side we only changes the way we build the > >>> package, it could be automated, it is scalable and it does not need to > >> patch > >>> package by package. Only to add some linker script magic > >> > >> The problem isn't particularly about detecting what is and isn't a > >> system library; I think we can come up with a reliable heuristic for > >> that, without even needing linker namespaces or anything. > >> > >> The outstanding problem seems to be more about potentially breaking > >> applications because they see two identically named DLLs loaded in the > >> same process. Applications can and do trawl the internal loader state, > >> although the Win32 loader also exposes some APIs so they don't even have > >> to. > >> > >> It may also be that the aforementioned heuristic is too hacky to be > >> accepted upstream into Wine. > >> > > I do not propose to change the loader, i propose to change the link or post > > link stage of system library. You could rename the lib and change the > > dépend to new name after build. So you will have an unique name per system > > lib without changing the way you build these lib. > > > > It is équivalent to use a namespace resolved at late build time. > > > > Ah, so what you're proposing is that we do some sort of objcopy-like > patching at runtime, e.g. when copying the dependency into the prefix. > > It probably won't be easy, but it might be more feasible than modifying > the loader. I'll look into it... Yes sort of obcopy patching under linux it is called patchelf... If it work, it could be latter done in memory like paul wise suggest implementing namespace and LT_AUDIT (but it is a long term goal) 1. capture library loading 2. do the patching at loading time (using the same code then patchelf for pe replacing file read/write by mmap operation) And voila bastien
[toc] | [prev] | [next] | [standalone]
| From | Bastien ROUCARIES <roucaries.bastien@gmail.com> |
|---|---|
| Date | 2021-09-08 09:50 +0200 |
| Message-ID | <CV0vf-7wz-1@gated-at.bofh.it> |
| In reply to | #101576 |
Hi, Adding smcv to the thread. Le mar. 7 sept. 2021 à 23:25, Bastien ROUCARIES <roucaries.bastien@gmail.com> a écrit : > > Le mar. 7 sept. 2021 à 21:16, Zebediah Figura > <zfigura@codeweavers.com> a écrit : > > > > On 9/7/21 12:05 PM, Bastien ROUCARIES wrote: > > > I disagree. > > > > > > Le mar. 7 sept. 2021 à 17:48, Zebediah Figura <zfigura@codeweavers.com> a > > > écrit : > > > > > >> On 9/7/21 5:16 AM, Bastien Roucariès wrote: > > >>> Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit : > > >>>> On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote: > > >>>>> The basic problem is that applications can and often do ship with PE > > >>>>> builds of cross-platform libraries. These libraries can be ahead of > > >>>>> Wine's system libraries, behind them, or even built with custom > > >> patches. > > >>>>> Accordingly we really don't want to load "our" freetype in place of > > >>>>> "their" freetype, or "theirs" in place of "ours". But because of the > > >> way > > >>>>> the Win32 loader works, you can generally only have one DLL of a given > > >>>>> name loaded in a process, and further attempts to dlopen() [as it were] > > >>>>> "libfreetype-6.dll" will return the handle to the already loaded > > >>>>> library, potentially breaking either Wine or the application. > > >>>> > > >>>> I don't know the details here, but my immediate thought when reading > > >>>> this is that you need some kind of namespace. I then found that linker > > >>>> namespaces are a thing, perhaps they would provide the solution for > > >>>> you. It sounds like the OpenGL shim loader solution listed on the > > >>>> glibc wiki might work for your use-case. Or perhaps the LD_AUDIT > > >>>> feature would also work. > > >>>> > > >>>> https://www.sourceware.org/glibc/wiki/LinkerNamespaces > > >>> I agree with pabs, implementing name space is a good solution > > >>> and will allow to be distrib friendly. > > >>> > > >>> Moreover it will be best if marking a library as wine system one could > > >> be done > > >>> post build. We will prefer to not modify upstream (like libpng) source. > > >>> > > >>> It is easier for us to apply small patch to upstream, or better in your > > >> case > > >>> to add a post link linker script or even some compiler flag that will > > >> add some > > >>> notes section to dll in order to mark it as use namespace system, or so > > >> on. > > >>> > > >>> The problem on windows side is well known as dll hell > > >>> > > >>> Renaming of the lib could be done post install. Moreover a crazy idea > > >> why not > > >>> renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll > > >> (note the > > >>> version is manadaroty) Therefore we could use upstream source. Then add > > >> a > > >>> small linker script that will rewrite the depends of libpng-2.0.0.sll > > >> to > > >>> .sll file (aka binary sed s/.dll/.sll/g). > > >>> > > >>> then add a small wraper (shim) like said pab that load the .sll library, > > >> the > > >>> best pratice will be a command tool that take the name of the sll and > > >> create > > >>> magically the dll > > >>> > > >>> Therefore from distrib point of side we only changes the way we build the > > >>> package, it could be automated, it is scalable and it does not need to > > >> patch > > >>> package by package. Only to add some linker script magic > > >> > > >> The problem isn't particularly about detecting what is and isn't a > > >> system library; I think we can come up with a reliable heuristic for > > >> that, without even needing linker namespaces or anything. > > >> > > >> The outstanding problem seems to be more about potentially breaking > > >> applications because they see two identically named DLLs loaded in the > > >> same process. Applications can and do trawl the internal loader state, > > >> although the Win32 loader also exposes some APIs so they don't even have > > >> to. > > >> > > >> It may also be that the aforementioned heuristic is too hacky to be > > >> accepted upstream into Wine. > > >> > > > I do not propose to change the loader, i propose to change the link or post > > > link stage of system library. You could rename the lib and change the > > > dépend to new name after build. So you will have an unique name per system > > > lib without changing the way you build these lib. > > > > > > It is équivalent to use a namespace resolved at late build time. > > > > > > > Ah, so what you're proposing is that we do some sort of objcopy-like > > patching at runtime, e.g. when copying the dependency into the prefix. > > > > It probably won't be easy, but it might be more feasible than modifying > > the loader. I'll look into it... > Yes sort of obcopy patching under linux it is called patchelf... > > If it work, it could be latter done in memory like paul wise suggest > implementing namespace and LT_AUDIT (but it is a long term goal) > 1. capture library loading > 2. do the patching at loading time (using the same code then patchelf > for pe replacing file read/write by mmap operation) > > And voila Simon, do you think you could implement a version of libcapasule for PE object ? Bastien > bastien
[toc] | [prev] | [next] | [standalone]
| From | Simon McVittie <smcv@debian.org> |
|---|---|
| Date | 2021-09-08 10:20 +0200 |
| Message-ID | <CV0Yi-7W8-11@gated-at.bofh.it> |
| In reply to | #101580 |
On Wed, 08 Sep 2021 at 07:31:59 +0000, Bastien ROUCARIES wrote:
> Simon, do you think you could implement a version of libcapasule for PE object ?
Given that libcapsule is very glibc- and ELF-specific, doesn't work
properly without new glibc feature work that isn't in Debian yet (I've
lost track of whether/when it got merged upstream, it might be in 2.34)
and wasn't written by me, the answer is likely to be no.
As far as I understand it, the PE loader used for Wine is part of Wine,
so it has total control over the libraries that it loads and how it loads
them. This means that if Wine developers (the experts on this codebase)
have decided a libcapsule-like approach isn't feasible or isn't desirable,
they are probably right.
More generally, if Wine developers wanted Wine's dependency libraries
to be in one version of reality, and Windows DLLs to be in a different
version of reality, then there's a straightforward way to do that:
use ELF .so dependency libraries, like older versions of Wine did, and
don't give them a representation in the PE layer. Presumably they have
put in the work to move from ELF to PE dependency libraries because it
has some compelling advantage, rather than creating work for themselves
for no reason!
smcv
[toc] | [prev] | [next] | [standalone]
| From | Zebediah Figura <zfigura@codeweavers.com> |
|---|---|
| Date | 2021-09-08 17:50 +0200 |
| Message-ID | <CV7ZL-3Qu-3@gated-at.bofh.it> |
| In reply to | #101581 |
On 9/8/21 3:13 AM, Simon McVittie wrote: > On Wed, 08 Sep 2021 at 07:31:59 +0000, Bastien ROUCARIES wrote: >> Simon, do you think you could implement a version of libcapasule for PE object ? > > Given that libcapsule is very glibc- and ELF-specific, doesn't work > properly without new glibc feature work that isn't in Debian yet (I've > lost track of whether/when it got merged upstream, it might be in 2.34) > and wasn't written by me, the answer is likely to be no. > > As far as I understand it, the PE loader used for Wine is part of Wine, > so it has total control over the libraries that it loads and how it loads > them. This means that if Wine developers (the experts on this codebase) > have decided a libcapsule-like approach isn't feasible or isn't desirable, > they are probably right. > > More generally, if Wine developers wanted Wine's dependency libraries > to be in one version of reality, and Windows DLLs to be in a different > version of reality, then there's a straightforward way to do that: > use ELF .so dependency libraries, like older versions of Wine did, and > don't give them a representation in the PE layer. Presumably they have > put in the work to move from ELF to PE dependency libraries because it > has some compelling advantage, rather than creating work for themselves > for no reason! Yeah, basically :-) The basic reason is that Wine is essentially trying to split itself into "PE" and "Unix" parts. The idea is that all of the "user" code is built in PE format, and only calls into unix code by making a sort of fake syscall. The "kernel" code is built in ELF format. There are a few basic reasons for doing this: * Many applications, mostly anti-cheat and anti-tamper engines, depend on our libraries actually being in PE format on disk, and matching that in memory. We used to have "fake" DLLs for this, but they weren't verisimilar enough. * Some of the important hosts for Wine have dropped support for 32-bit libraries or are going to drop it. Mac is the obvious example here, but many commercial Linux distributions also want to drop support. By limiting the surface through which code can transition between PE and Unix code, it becomes feasible to do 32-to-64 translation, where previously this was quite infeasible. * There's some demand for running Win32 debuggers under wine. These have always worked more than a little tenuously, but a couple of them are causing problems when they try to break in and unwind from Unix code. By acting like we're making actual syscalls, things work much better. Of the several dozen modules we have that currently call into Unix libraries, about half of them aren't going to use PE dependencies. Most of these have already been fully split into PE and Unix parts, including things like the socket layer and winegstreamer. This raises the question: why not just call into Unix for the other half as well? There are a few reasons: * Making fake syscalls is, like real syscalls, not cheap. It involves a full context switch into and out of Unix code. * Making callbacks from Unix code is difficult, and not cheap either. For some libraries this doesn't matter, but for others it really does. * Writing the wrappers between Unix and PE code involves some nontrivial work in itself. The smaller the interface is, the easier things are for us.
[toc] | [prev] | [next] | [standalone]
| From | Paul Wise <pabs@debian.org> |
|---|---|
| Date | 2021-09-09 03:20 +0200 |
| Message-ID | <CVgTn-VZ-1@gated-at.bofh.it> |
| In reply to | #101596 |
[Multipart message — attachments visible in raw view] — view raw
On 9/8/21 3:13 AM, Simon McVittie wrote: > As far as I understand it, the PE loader used for Wine is part of Wine, > so it has total control over the libraries that it loads and how it loads > them. This means that if Wine developers (the experts on this codebase) > have decided a libcapsule-like approach isn't feasible or isn't desirable, > they are probably right. Could the PE loader add support for some sort of namespaces and then be told to load Wine dependencies within a Wine namespace? Does the Microsoft Windows loader support anything like this? Recompiling every PE library with special flags or renaming every PE library with a Wine prefix seems a bit tedious if just the PE loader and anything that requests Wine deps can use a namespace/capsule/etc. On Wed, 2021-09-08 at 10:42 -0500, Zebediah Figura wrote: > The basic reason is that Wine is essentially trying to split itself into > "PE" and "Unix" parts. The idea is that all of the "user" code is built > in PE format, and only calls into unix code by making a sort of fake > syscall. The "kernel" code is built in ELF format. > > There are a few basic reasons for doing this: Definitely seems like the right thing to do for Wine. -- bye, pabs https://wiki.debian.org/PaulWise
[toc] | [prev] | [next] | [standalone]
| From | Zebediah Figura <zfigura@codeweavers.com> |
|---|---|
| Date | 2021-09-09 06:50 +0200 |
| Message-ID | <CVkaB-2YX-1@gated-at.bofh.it> |
| In reply to | #101608 |
On 9/8/21 8:12 PM, Paul Wise wrote: > On 9/8/21 3:13 AM, Simon McVittie wrote: >> As far as I understand it, the PE loader used for Wine is part of Wine, >> so it has total control over the libraries that it loads and how it loads >> them. This means that if Wine developers (the experts on this codebase) >> have decided a libcapsule-like approach isn't feasible or isn't desirable, >> they are probably right. > > Could the PE loader add support for some sort of namespaces and then be > told to load Wine dependencies within a Wine namespace? > > Does the Microsoft Windows loader support anything like this? > > Recompiling every PE library with special flags or renaming every PE > library with a Wine prefix seems a bit tedious if just the PE loader > and anything that requests Wine deps can use a namespace/capsule/etc. > The closest thing that Windows has is SxS, which is meant to solve I think some of the same problems (well, it's mainly meant to solve the versioning problem that Microsoft dug themselves into). The problem is that I believe it requires the namespace to be effectively hard-coded into the library in question, in the form of a manifest resource. Unfortunately, while thinking about the answer to this question, I realized another snag, which I think really does make using identically-named dynamic libraries impossible: if system library A loads system library B dynamically, i.e. does the equivalent of a dlopen(), we have no way of knowing which namespace to apply to it. [AFAICT dlmopen() has this problem too, and just isn't meant for that use case.] Even rewriting the import table doesn't help here.
[toc] | [prev] | [next] | [standalone]
Page 2 of 3 — ← Prev page 1 [2] 3 Next page →
Back to top | Article view | linux.debian.devel
csiph-web