Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1197689 > unrolled thread

Bug#1071246: libglib2.0-dev's python3 dependency alternative is posing challenges to cross building

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#1071246: libglib2.0-dev's python3 dependency alternative is posing challenges to cross building Helmut Grohne <helmut@subdivi.de> - 2024-05-17 08:20 +0200
    Bug#1071246: libglib2.0-dev's python3 dependency alternative is posing challenges to cross building Simon McVittie <smcv@debian.org> - 2024-05-17 11:50 +0200
      Bug#1071246: libglib2.0-dev's python3 dependency alternative is posing challenges to cross building Helmut Grohne <helmut@subdivi.de> - 2024-05-21 00:10 +0200
        Bug#1071246: libglib2.0-dev's python3 dependency alternative is posing challenges to cross building Simon McVittie <smcv@debian.org> - 2024-05-21 12:10 +0200

#1197689 — Bug#1071246: libglib2.0-dev's python3 dependency alternative is posing challenges to cross building

Package: libglib2.0-dev
Version: 2.80.2-1
User: debian-cross@lists.debian.org
Usertags: ftcbfs
Control: affects -1 + src:tkgate
X-Debbugs-Cc: debian-cross@lists.debian.org

Hi Simon,

you recently added an alternative python3 to the qemu dependency. This
is posing a challenge to cross building now, but the devil is in the
detail.

On a technical level, python3 | qemu expresses what we want. The crossqa
service sees this and has special handling for python3. It considers
python3.*-minimal to not be installable for the host architecture (as
that pratically fails postinst). We're checking dependencies with dose
and dose is quite clever in coming up with creative solutions. It has no
problems figuring out that qemu is the solution here. Hence, reverse
dependencies (tkgate being an example here) are scheduled for building
and then apt looks at the dependencies. It sees python3 as the first
alternative and is generally happy with installing the host one. Hence,
it doesn't look beyond and doesn't consider qemu. The python3:any
dependency show up later and the host's python3 happens to satisfy it.
Hence apt is happy until python3-minimal:host.postinst fails.

Arguably, this is a problem of the builder. Package depedencies cannot
tell whether we can install a foreign python or not. That's a question
only the builder can answer and David added a fiddle to apt quite a
while ago that is called "BarbarianArchitectures" and documented as a
list of architectures considered too foreign to satisfy M-A:foreign.
Even though python3 isn't actually M-A:foreign, adding this setting
immediately makes the build work and you can try that by adding this to
your sbuild invocation:

    --chroot-setup-commands='echo "APT::BarbarianArchitectures {\"arm64\"};" > /etc/apt/apt.conf.d/b.conf'

I also tried reordering the python3 dependencies in libglib2.0-dev such
that the python3:any comes first, but that did not have any effect.

Do we see another way than changing the way that sbuild (and pbuilder)
performs cross builds?

I note that this bug has a bit of urgency, because affected packages
will be marked as bd-uninstallable and that is taken as a sign to stop
trying to build the relevant architecture and makes crossqa.d.n stop
doing builds until the next mirror push. So this bug breaks QA.

Helmut

[toc] | [next] | [standalone]

#1197701

On Fri, 17 May 2024 at 08:14:41 +0200, Helmut Grohne wrote:
> Package: libglib2.0-dev
> Version: 2.80.2-1
> User: debian-cross@lists.debian.org
> Usertags: ftcbfs
> Control: affects -1 + src:tkgate
> X-Debbugs-Cc: debian-cross@lists.debian.org
> 
> you recently added an alternative python3 to the qemu dependency. This
> is posing a challenge to cross building now, but the devil is in the
> detail.

This is happening because upstream GLib 2.80.x has taken over
the lower-level parts of the GObject-Introspection toolchain
from src:gobject-introspection. As a result, providing its full
functionality requires the ability to run host-architecture binaries
(for at least gi-compile-repository, which will eventually replace
gobject-introspection's g-ir-compiler).

In older versions, src:gobject-introspection would likely have had
the same issue; the only difference is that libglib2.0-dev affects
more packages.

As a reminder of the design here, g-ir-scanner from in the
gobject-introspection source package examines shared libraries and outputs
GIR XML, which is analogous to C headers: usually, but not always,
architecture-independent. g-ir-compiler or gi-compile-repository takes the
GIR XML and compiles it into architecture-dependent binary typelibs.
The GIR XML is used directly by some language bindings (typically at
compile-time for static languages like Rust and C++), and the typelibs are
used for others (typically at runtime for dynamic languages like Perl and
Paython).

I had initially hoped that we could wrap the build-architecture
g-ir-compiler/gi-compile-repository to compile GIR XML into typelibs,
but that was broken and caused a RC bug (#1066900), so we cannot do that.

In principle we could build one copy of gi-compile-repository per (build
architecture, host architecture) pair, like cross-compilers, but that
seems bad from a maintainabiity point of view. Upstream does not support
this use-case (and in general has little interest in cross g-i) so it
would be very easy for it to regress, it would require either GLib or
some other package to know a complete list of all interesting Debian
architectures in the same way that gcc-13-cross does, and running
g-ir-scanner fundamentally requires executing code from the host
architecture *anyway* because that's just how g-i works.

If the dependency was changed to:

    something-native | qemu-user | qemu-user-static,
    python3:any,

where "something-native" is any package that is either M-A: no or
M-A: allowed (but must not be M-A: foreign, and in practice is required
to be installed from a runnable architecture and not a barbarian
architecture) but is *not* Python, would that solve the problem?

Most of the obvious Essential or Build-Essential packages like base-files
are Multi-Arch: foreign, therefore not suitable for this purpose, but for
example a short-term hack version of this might be to use apt, perl or
perl-base as the something-native - because they are already installed
(and because apt is Important: yes and perl-base is Essential: yes),
hopefully apt will prefer to keep the existing version installed and take
the qemu-user alternative, in preference to attempting to crossgrade them
to a version that does not, in fact, work.

I don't know whether apt might become Multi-Arch: foreign in the future,
though. If it did, that would break our ability to use it like this.

This intersects with #1070773, in which users of amd64 + i386 multiarch
don't want to have to install qemu-user, and would prefer to have some
way to say "it's OK, my amd64 system can run i386 binaries directly".

If someone with more time available for cross-development
implemented the cross-exe-wrapper design that I sketched in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030223#37, then
gobject-introspection and libglib2.0-dev would be able to migrate to
depending on

    cross-exe-wrapper | can-run-arch,
    python3:any,

or for backwards-compat maybe

    cross-exe-wrapper | can-run-arch | qemu-user | qemu-user-static,
    python3:any,

and I think this would also solve the problem?

If that doesn't work, another option (but IMO a bad one) is that because
g-i functionality in GLib is a new feature in trixie, it would in
principle be possible to move gi-compile-repository and related tools to
a separate binary package, at the cost of breaking backwards compatibility
with older versions of trixie-as-testing (and also breaking compatibility
with Ubuntu 24.04, which would be unfortunate). We would probably need
*two* new binary packages:

- one Multi-Arch: same, analogous to gobject-introspection, to contain
  /usr/bin/$(DEB_HOST_GNU_ARCH)-gi-*, plus their implementations in a
  private directory (at the moment this is part of libglib2.0-dev);

- and one Multi-Arch: no containing the non-architecture-prefixed
  /usr/bin/gi-* which will be expected by simple non-cross-capable
  upstream scripts and build systems (at the moment this is part of
  libglib2.0-dev-bin, which is Multi-Arch: foreign, which is not 100%
  correct but is the best we can do to accommodate these naive build
  systems' expectations)

Hoever, I believe the package containing
/usr/bin/$(DEB_HOST_GNU_ARCH)-gi-* could cause the same installation
issue for your cross-build infrastructure that libglib2.0-dev now causes:
you'll see that gobject-introspection.deb also has this
"python3 | qemu-user, python3:any" dependency. So I'm not convinced that
would actually help, it would probably just move the problem around.

> On a technical level, python3 | qemu expresses what we want. The crossqa
> service sees this and has special handling for python3. It considers
> python3.*-minimal to not be installable for the host architecture (as
> that pratically fails postinst).
...
> and then apt looks at the dependencies. It sees python3 as the first
> alternative and is generally happy with installing the host one.

I think this is the real problem here. We say python3:any because that's
what the multiarch dependency syntax allows us to say, but what we
*mean* is "well *obviously* I need to be able to run the interpreter".
A better syntax, if it was allowed, would be python3:runnable or even
python3:native - but we can't say that in this context.

crossqa's dependency resolver has this special case to treat non-native
python3 as uninstallable, but apt does not, and it's that disagreement
that is breaking your test scenario.

> I note that this bug has a bit of urgency, because affected packages
> will be marked as bd-uninstallable and that is taken as a sign to stop
> trying to build the relevant architecture and makes crossqa.d.n stop
> doing builds until the next mirror push. So this bug breaks QA.

I'm sorry that GObject-Introspection is complicated, but that's just
the way it is. I am doing the best I can do make it work the way you
would want it to within my limited time and resources, and in some cases
against upstream hostility. If you would have preferred cross-g-i to
remain impossible, that would have been the other realistic option here.

I'm open to suggestions for how to resolve this, but I cannot always
drop everything to solve cross issues immediately, and I certainly
didn't intend to be Debian's single point of failure for all things
gobject-introspection.

    smcv

[toc] | [prev] | [next] | [standalone]

#1198117

Control: tags -1 - pending
Control: reassign -1 libglib2.0-dev,sbuild

Hi Simon,

I'm sorry for the delayed reply. Even though I dumped significant effort
into this, things turned out getting more complicated.

For one thing, my attempts to fix this on the sbuild side have resulted
in tagpending tagging this bug. sbuild's git no longer performs such
tagging for feature branches.

On Fri, May 17, 2024 at 10:42:11AM +0100, Simon McVittie wrote:
> This is happening because upstream GLib 2.80.x has taken over
> the lower-level parts of the GObject-Introspection toolchain
> from src:gobject-introspection. As a result, providing its full
> functionality requires the ability to run host-architecture binaries
> (for at least gi-compile-repository, which will eventually replace
> gobject-introspection's g-ir-compiler).
> 
> In older versions, src:gobject-introspection would likely have had
> the same issue; the only difference is that libglib2.0-dev affects
> more packages.
> 
> As a reminder of the design here, g-ir-scanner from in the
> gobject-introspection source package examines shared libraries and outputs
> GIR XML, which is analogous to C headers: usually, but not always,
> architecture-independent. g-ir-compiler or gi-compile-repository takes the
> GIR XML and compiles it into architecture-dependent binary typelibs.
> The GIR XML is used directly by some language bindings (typically at
> compile-time for static languages like Rust and C++), and the typelibs are
> used for others (typically at runtime for dynamic languages like Perl and
> Paython).
> 
> I had initially hoped that we could wrap the build-architecture
> g-ir-compiler/gi-compile-repository to compile GIR XML into typelibs,
> but that was broken and caused a RC bug (#1066900), so we cannot do that.

Yeah I know. Thanks for writing it down again. Not blaming you.

> In principle we could build one copy of gi-compile-repository per (build
> architecture, host architecture) pair, like cross-compilers, but that
> seems bad from a maintainabiity point of view. Upstream does not support
> this use-case (and in general has little interest in cross g-i) so it
> would be very easy for it to regress, it would require either GLib or
> some other package to know a complete list of all interesting Debian
> architectures in the same way that gcc-13-cross does, and running
> g-ir-scanner fundamentally requires executing code from the host
> architecture *anyway* because that's just how g-i works.
> 
> If the dependency was changed to:
> 
>     something-native | qemu-user | qemu-user-static,
>     python3:any,
> 
> where "something-native" is any package that is either M-A: no or
> M-A: allowed (but must not be M-A: foreign, and in practice is required
> to be installed from a runnable architecture and not a barbarian
> architecture) but is *not* Python, would that solve the problem?

I guess that it might solve the immediate problem, but I'm not sure this
is a good compromise. We've had the idea of restricting
Multi-Arch:foreign packages from satisfying dependencies with host
architecture instances since a while and crossqa's scheduler (but not
builder) actually enforces this, so I've been looking into also
enforcing it on the builder and that might be a better solution. I
cannot tell yet. That said, you can find it at:

https://salsa.debian.org/debian/sbuild/-/merge_requests/69

I note that my testing of this was not satisfactory. For one thing, it
didn't seem to be reliable (and I note that none of the python core
stack actually is Multi-Arch:foreign) and it also doesn't make any
package cross buildable. When I get past this problem, I run into a new
problem where libc6.postinst fails to understand that it is run in a
chroot (as it looks too much like a container) and thus systemd-sysv's
telinit kills the parent process of apt making sbuild unhappy
(#1071462). While I welcome feedback on this issue, it quite definitely
is a timesink.

> Most of the obvious Essential or Build-Essential packages like base-files
> are Multi-Arch: foreign, therefore not suitable for this purpose, but for
> example a short-term hack version of this might be to use apt, perl or
> perl-base as the something-native - because they are already installed
> (and because apt is Important: yes and perl-base is Essential: yes),
> hopefully apt will prefer to keep the existing version installed and take
> the qemu-user alternative, in preference to attempting to crossgrade them
> to a version that does not, in fact, work.

I appreciate your effort for a quick solution, but I question whether
the cure really is better than the disease. I also note that crossqa.d.n
was completely broken for a week by c-t-b being uninstallable, so maybe
we are better off acknowledging the problem and finding a solution we
can all live with.

Knowing the available options does help though. Thank you.

> I don't know whether apt might become Multi-Arch: foreign in the future,
> though. If it did, that would break our ability to use it like this.

apt also isn't technically build-essential and we are looking into
building inside apt-less chroots.

> This intersects with #1070773, in which users of amd64 + i386 multiarch
> don't want to have to install qemu-user, and would prefer to have some
> way to say "it's OK, my amd64 system can run i386 binaries directly".

Yes, absolutely.

> If someone with more time available for cross-development
> implemented the cross-exe-wrapper design that I sketched in
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030223#37, then
> gobject-introspection and libglib2.0-dev would be able to migrate to
> depending on
> 
>     cross-exe-wrapper | can-run-arch,
>     python3:any,
> 
> or for backwards-compat maybe
> 
>     cross-exe-wrapper | can-run-arch | qemu-user | qemu-user-static,
>     python3:any,
> 
> and I think this would also solve the problem?

I think this is technically unsolvable on the package relation level.
The "can-run-arch" property that you suggest here cannot technically be
expressed by a dependency relation, because the answer depends on things
apt doesn't know. Consider the following cases:

 * We cross build from amd64 to x32, but the kernel does not enable the
   x32 abi.
 * We cross build from amd64 to m68k and have externally installed
   qemu-user-static as we mostly care about speeding up builds rather
   than doing "pure" cross builds.
 * We cross build for i386 from amd64 and disable the i386 kernel ABI
   via seccomp-bpf to simulate a QA cross build.
 * Some arm64 CPUs can run arm32, but not all.

Whether the host architecture is runnable is to be defined by the
builder and the builder is supposed to set the cross build profile when
the host architecture is not runnable. So in regular, amd64->i386 cross
builds we'd want to remove the cross build profile unless we're
interested in QA.

Unfortunately, the dependency at hand is a runtime dependency and we
cannot use build profiles in runtime dependencies, so my whole
elaboration on the cross build profile gets us pretty much nothing here.

I don't have a better option.

> If that doesn't work, another option (but IMO a bad one) is that because
> g-i functionality in GLib is a new feature in trixie, it would in
> principle be possible to move gi-compile-repository and related tools to
> a separate binary package, at the cost of breaking backwards compatibility
> with older versions of trixie-as-testing (and also breaking compatibility
> with Ubuntu 24.04, which would be unfortunate). We would probably need
> *two* new binary packages:
> 
> - one Multi-Arch: same, analogous to gobject-introspection, to contain
>   /usr/bin/$(DEB_HOST_GNU_ARCH)-gi-*, plus their implementations in a
>   private directory (at the moment this is part of libglib2.0-dev);
> 
> - and one Multi-Arch: no containing the non-architecture-prefixed
>   /usr/bin/gi-* which will be expected by simple non-cross-capable
>   upstream scripts and build systems (at the moment this is part of
>   libglib2.0-dev-bin, which is Multi-Arch: foreign, which is not 100%
>   correct but is the best we can do to accommodate these naive build
>   systems' expectations)
> 
> Hoever, I believe the package containing
> /usr/bin/$(DEB_HOST_GNU_ARCH)-gi-* could cause the same installation
> issue for your cross-build infrastructure that libglib2.0-dev now causes:
> you'll see that gobject-introspection.deb also has this
> "python3 | qemu-user, python3:any" dependency. So I'm not convinced that
> would actually help, it would probably just move the problem around.

I fear that I don't quite understand the implications, but let me note
that it would move the problem around sufficiently for a relevant
package set. We've done without cross building gobject-introspection
stuff for a very long time. Regressing on libglib2.0-dev is relevant
here. So consider that "just move the problem around" provides
noticeable benefit from my point of view (without implying that doing so
is worth the downsides that you depict).

> > On a technical level, python3 | qemu expresses what we want. The crossqa
> > service sees this and has special handling for python3. It considers
> > python3.*-minimal to not be installable for the host architecture (as
> > that pratically fails postinst).
> ...
> > and then apt looks at the dependencies. It sees python3 as the first
> > alternative and is generally happy with installing the host one.
> 
> I think this is the real problem here. We say python3:any because that's
> what the multiarch dependency syntax allows us to say, but what we
> *mean* is "well *obviously* I need to be able to run the interpreter".
> A better syntax, if it was allowed, would be python3:runnable or even
> python3:native - but we can't say that in this context.

I concur.

> crossqa's dependency resolver has this special case to treat non-native
> python3 as uninstallable, but apt does not, and it's that disagreement
> that is breaking your test scenario.

That is exactly why I looked into configuring "barbarian" architectures
(see the sbuild MR above).

> I'm sorry that GObject-Introspection is complicated, but that's just
> the way it is. I am doing the best I can do make it work the way you
> would want it to within my limited time and resources, and in some cases
> against upstream hostility. If you would have preferred cross-g-i to
> remain impossible, that would have been the other realistic option here.

I very much appreciate your cooperation and feedback. I'm not meaning to
imply work on you here. To me, this is a technical problem and the most
responsible person is the porter (i.e. me), not you.

> I'm open to suggestions for how to resolve this, but I cannot always
> drop everything to solve cross issues immediately, and I certainly
> didn't intend to be Debian's single point of failure for all things
> gobject-introspection.

"some urgency" is very different from "drop everything" in my book. You
read it differently than it was meant. In the grand scheme of things,
cross building is a niche feature and it is up to cross build porters
(though the plural can be seen as a lie) to make it work. Your
supporting of the effort beyond what the average maintainer does is
appreciated.

Helmut

[toc] | [prev] | [next] | [standalone]

#1198155

On Mon, 20 May 2024 at 19:45:53 +0200, Helmut Grohne wrote:
> On Fri, May 17, 2024 at 10:42:11AM +0100, Simon McVittie wrote:
> > If someone with more time available for cross-development
> > implemented the cross-exe-wrapper design that I sketched in
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030223#37, then
> > gobject-introspection and libglib2.0-dev would be able to migrate to
> > depending on
> > 
> >     cross-exe-wrapper | can-run-arch,
> >     python3:any,
> > 
> > or for backwards-compat maybe
> > 
> >     cross-exe-wrapper | can-run-arch | qemu-user | qemu-user-static,
> >     python3:any,
> > 
> > and I think this would also solve the problem?
> 
> I think this is technically unsolvable on the package relation level.
> The "can-run-arch" property that you suggest here cannot technically be
> expressed by a dependency relation, because the answer depends on things
> apt doesn't know.

Oh, what I had assumed was that in most cases where there was any doubt,
the cross-exe-wrapper would have to depend on and run qemu-user (at least
after a runtime check for whether it can get away with avoiding qemu). The
cases where cross-exe-wrapper relied on direct execution instead of
qemu or equivalent would be build=foo, host=foo for all values of foo,
plus probably build=amd64, host=i386 as a common special case.

I'd been thinking of can-run-arch as an extension point that a sysadmin
could set up with equivs (for example to represent the use-case you
described where qemu-user-static is set up with binfmt_misc outside the
chroot), rather than something that would exist in the archive; I'd assumed
that cross-exe-wrapper would be the automatic interface.

Thinking more about this, though, I think the cross-exe-wrapper itself
could have the same problem that you've described in libglib2.0-dev:
on build=amd64 and (let's say) host=riscv64, if we had something like

    Package: libglib2.0-dev
    Architecture: riscv64
    Multi-Arch: same
    Depends: cross-exe-wrapper | can-run

    Package: cross-exe-wrapper
    Architecture: riscv64
    Multi-Arch: same
    Depends: cross-exe-wrapper-riscv64-linux-gnu

    Package: cross-exe-wrapper-riscv64-linux-gnu
    Architecture: riscv64
    Multi-Arch: ???    # no? foreign? allowed?
    # trivial wrapper, runs binary directly

    Package: cross-exe-wrapper-riscv64-linux-gnu
    Architecture: amd64
    Multi-Arch: foreign
    Depends: qemu-user
    # non-trivial wrapper, runs via qemu

then I think it would be valid (but definitely undesired!) for
apt to satisfy cross-exe-wrapper's dependency by installing
cross-exe-wrapper-riscv64-linux-gnu:riscv64, which of course we would
be unable to run. So this is another place where ideally we would like
to be able to depend on "cross-exe-wrapper-riscv64-linux-gnu:runnable"
or some similar syntax that doesn't yet exist.

>  * We cross build from amd64 to x32, but the kernel does not enable the
>    x32 abi.

It's common to do that (and I believe x32 is runtime-disabled on Debian
kernels by default to limit attack surface), so I would have assumed
we'd want to make cross-exe-wrapper pull in qemu-user when build=amd64,
host=x32; but if it can avoid actually needing to run it, with a runtime
check like the ones in the cross g-i tools, then so much the better.

>  * We cross build from amd64 to m68k and have externally installed
>    qemu-user-static as we mostly care about speeding up builds rather
>    than doing "pure" cross builds.

In this case it would be harmless to install qemu-user into the chroot
(a bit of a waste of I/O but no real harm), and it hardly matters whether
we use the host system's qemu or the chroot's, as long as all the necessary
tools are runnable.

>  * We cross build for i386 from amd64 and disable the i386 kernel ABI
>    via seccomp-bpf to simulate a QA cross build.

My inclination would be to say that's too artificial to be something
that we'd be willing to pay a price to support (the price is #1070773),
but you know better than I do.

>  * Some arm64 CPUs can run arm32, but not all.

As with x32 on amd64, the wrapper would have to pessimistically assume
that we do need qemu-user (optionally with a runtime check to shortcut
to running the binary directly if we can).

    smcv

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web