Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #84264 > unrolled thread

Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap

Back to article view | Back to linux.debian.kernel

Contents

  Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Helmut Grohne <helmut@subdivi.de> - 2024-10-10 22:20 +0200
    Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Bastian Blank <waldi@debian.org> - 2024-10-19 15:40 +0200
      Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Helmut Grohne <helmut@subdivi.de> - 2024-10-19 18:30 +0200
        Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Ben Hutchings <ben@decadent.org.uk> - 2024-10-24 01:40 +0200
          Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Helmut Grohne <helmut@subdivi.de> - 2024-11-29 08:00 +0100
        Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Bastian Blank <waldi@debian.org> - 2024-12-04 22:40 +0100
          Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Helmut Grohne <helmut@subdivi.de> - 2024-12-05 13:10 +0100
      Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Ben Hutchings <ben@decadent.org.uk> - 2024-10-24 01:40 +0200
        Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Bastian Blank <waldi@debian.org> - 2024-12-04 23:00 +0100
          Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Helmut Grohne <helmut@subdivi.de> - 2024-12-05 13:10 +0100
            Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Bastian Blank <waldi@debian.org> - 2024-12-05 15:40 +0100
              Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Helmut Grohne <helmut@subdivi.de> - 2024-12-05 22:30 +0100
                Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Stefano Rivera <stefanor@debian.org> - 2025-01-28 00:40 +0100
                  Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Bastian Blank <waldi@debian.org> - 2025-02-04 15:10 +0100
                    Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap stefanor@debian.org - 2025-02-04 18:30 +0100
                      Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Bastian Blank <waldi@debian.org> - 2025-02-04 20:40 +0100
                      Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Chris Hofstaedtler <zeha@debian.org> - 2025-02-05 00:00 +0100
                      Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Stefano Rivera <stefanor@debian.org> - 2025-03-04 18:00 +0100
                        Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Bastian Blank <waldi@debian.org> - 2025-03-13 07:40 +0100
                          Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Bastian Blank <waldi@debian.org> - 2025-03-15 11:40 +0100
      Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap Stefano Rivera <stefanor@debian.org> - 2024-11-29 14:20 +0100

Page 1 of 2  [1] 2  Next page →

#84264 — Bug#1084908: arch:all linux-libc-dev causes problems to architecture bootstrap

Package: linux-libc-dev
Version: 6.6.3-1~exp1

Hi,

Ben asked me to file a bug report with the linux package describing the
issues I see with linux-libc-dev being Arch:all.

A. Multi-Arch: foreign is a lie

linux-libc-dev declares being Multi-Arch: foreign, but it does not live
up to the involved promise and cannot. Earlier, a dependency on
linux-libc-dev:hurd-i386 would be considered unsatisfiable. With the
current setup, it is trivially satisfied even though that doesn't make
sense in any way. In a similar way, linux-libc-dev:sparc is now
considered satisfied. While that makes some sense, the actual package
does not ship any sparc files as 32bit sparc is dead. As such,
linux-libc-dev now makes dependencies look satisfied when they should
not be.

B. Inability to figure out what architectures linux-libc-dev supports

When linux-libc-dev was arch:any, architecture bootstrap would have to
cross build it. Now that it is arch:all, architecture bootstrap can use
the existing package. Sometimes. For release architectures and a few
more, it is good, but fore exotic and dead architectures (e.g. sparc),
one has to rebuild it with a modified configuration. In order to support
figuring out when a rebuild was required, Bastian added provides for
linux-libc-dev-$arch-cross due to my request. Neither of us saw how that
would break cross toolchains and these provides are now reverted as a
result. In any case, we're now back to the state where one cannot figure
out whether a rebuild is needed.

Ben asked me whether linux-libc-dev could simply support all relevant
architectures, but there often is a check&egg problem. The linux package
can only support architectures that dpkg knows about, but we want to
perform test bootstraps way before adding a triplet to dpkg and one of
the first things a test bootstrap does is building linux-libc-dev. So I
generally expect that linux-libc-dev can never provide support for all
architectures of interest and it shouldn't have to. Only once the
bootstrap is tested and the parameters are known do we want to file the
patches for a new architecture.

C. Inability to produce a linux-libc-dev-$arch-cross package

Now that linux-libc-dev-$arch-cross is no longer provided, dependencies
on it are no longer considered satisfied, but the toolchain packages
express dependencies on such packages. The architecture bootstrap
tooling does not use cross-toolchain-base and instead creates such
packages using dpkg-cross, but dpkg-cross cannot operate on arch:all
packages. Therefore, we can no longer turn linux-libc-dev:$arch into
linux-libc-dev-$arch-cross as is required by the cross toolchain.

I note that I do not have a good understanding why the change from
arch:any to arch:all was done. It does seem to reduce the cumulative
size consumed by binary .debs on mirrors and note that a similar effect
can be achieved by having a linux-libc-dev-common for the architecture
generic headers and a per-arch symlink farm in linux-libc-dev.

Ben, can you give some feedback on whether this report is useful for you
in moving things forward? I'm not sure how actionable this actually is
and welcome ideas on how architecture bootstrap should be adapted to
cope with the changed packages.

Helmut

[toc] | [next] | [standalone]

#84295

Hi

On Thu, Oct 10, 2024 at 06:03:24PM +0200, Helmut Grohne wrote:
> A. Multi-Arch: foreign is a lie
> 
> linux-libc-dev declares being Multi-Arch: foreign, but it does not live
> up to the involved promise and cannot. Earlier, a dependency on
> linux-libc-dev:hurd-i386 would be considered unsatisfiable. With the
> current setup, it is trivially satisfied even though that doesn't make
> sense in any way. In a similar way, linux-libc-dev:sparc is now
> considered satisfied. While that makes some sense, the actual package
> does not ship any sparc files as 32bit sparc is dead. As such,
> linux-libc-dev now makes dependencies look satisfied when they should
> not be.

What you describe here is a headers only library.  It is consensus
within Debian that those exist and are shipped as arch:all.  They don't
provide further architecture clues in the dependencies, but you need to
know or convey this information by failing to build.

Sparc is dead, so any Debian using Linux that does not support it any
longer will not provide the basic support to use it.  Sure, you could
hold on to an ancient linux-libc-dev, but this is then not longer
Debian.  Heck, we even consider running a mix of stable and unstable as
unsupported.  Why would we consider running a mix of
Debian-supporting-sparc and Debian-not-supporting-sparc a supportable
configuration?  Technically all of this can work, but still not
considered supported.

All the examples of yours will immediatelly fail to build anything with
it.  So what harm is done?

> B. Inability to figure out what architectures linux-libc-dev supports
> 
> When linux-libc-dev was arch:any, architecture bootstrap would have to
> cross build it. Now that it is arch:all, architecture bootstrap can use
> the existing package. Sometimes. For release architectures and a few
> more, it is good, but fore exotic and dead architectures (e.g. sparc),
> one has to rebuild it with a modified configuration. In order to support
> figuring out when a rebuild was required, Bastian added provides for
> linux-libc-dev-$arch-cross due to my request. Neither of us saw how that
> would break cross toolchains and these provides are now reverted as a
> result. In any case, we're now back to the state where one cannot figure
> out whether a rebuild is needed.

I have no idea what you are talking about.  I added the provides to
replace the existing cross packages with the functional identical
linux-libc-dev.  You where not involved in this at all.

> Ben asked me whether linux-libc-dev could simply support all relevant
> architectures, but there often is a check&egg problem. The linux package
> can only support architectures that dpkg knows about, but we want to
> perform test bootstraps way before adding a triplet to dpkg and one of
> the first things a test bootstrap does is building linux-libc-dev. So I
> generally expect that linux-libc-dev can never provide support for all
> architectures of interest and it shouldn't have to. Only once the
> bootstrap is tested and the parameters are known do we want to file the
> patches for a new architecture.

I have to disagree.  With this setup, we can support architectures that
are neither known by dpkg nor dak.  This was not possible before and
required patching, aka making a derivative of the package.  You can
still patch and rebuild it how you want and inject that modified package
into your workflow.

But now you can have a port without hand patched linux that is built in
a non-standard way and relying on internal and unstable package API,
just be talking and getting it added to the package first.

> C. Inability to produce a linux-libc-dev-$arch-cross package
> 
> Now that linux-libc-dev-$arch-cross is no longer provided, dependencies
> on it are no longer considered satisfied, but the toolchain packages
> express dependencies on such packages. The architecture bootstrap
> tooling does not use cross-toolchain-base and instead creates such
> packages using dpkg-cross, but dpkg-cross cannot operate on arch:all
> packages. Therefore, we can no longer turn linux-libc-dev:$arch into
> linux-libc-dev-$arch-cross as is required by the cross toolchain.

This sounds like a bug in dpkg-cross, as it uses a not agreed on
interface to do it's work.

> I note that I do not have a good understanding why the change from
> arch:any to arch:all was done. It does seem to reduce the cumulative
> size consumed by binary .debs on mirrors and note that a similar effect
> can be achieved by having a linux-libc-dev-common for the architecture
> generic headers and a per-arch symlink farm in linux-libc-dev.

No, it can not.  We can not have arch:any -> arch:all dependencies
within the same source package this deep in the toolchain, especially as
those require = dependencies.

If we talk about size, please go to gcc, which ships unstripped
binaries parts of the time.  We don't need to talk about a highly
compressible 10 MB installed (up from 7 or so before) package.

Bastian

-- 
Immortality consists largely of boredom.
		-- Zefrem Cochrane, "Metamorphosis", stardate 3219.8

[toc] | [prev] | [next] | [standalone]

#84298

Hi Bastian,

On Sat, Oct 19, 2024 at 03:35:21PM +0200, Bastian Blank wrote:
> On Thu, Oct 10, 2024 at 06:03:24PM +0200, Helmut Grohne wrote:
> > A. Multi-Arch: foreign is a lie
...
> All the examples of yours will immediatelly fail to build anything with
> it.  So what harm is done?

This is practically untrue. Even with current unstable packages, the
sparc (32) bootstrap still succeeds. But then the use of sparc here was
an example rather than a concrete issue. linux-libc-dev claims to
support an architecture that it technically does not. That problem works
the same way for obsolete architectures (such as sparc) as it does for
new ones. The harm is being done. Quite simply, linux-libc-dev should
ensure that dependencies on it are not considered satisfied when the
architecture of the dependency is not practically supported.

> > B. Inability to figure out what architectures linux-libc-dev supports
...
> I have no idea what you are talking about.  I added the provides to
> replace the existing cross packages with the functional identical
> linux-libc-dev.  You where not involved in this at all.

Ok, you may see it that way (except for the "functional identical" part
that evidently doesn't hold). I don't think it matters much who caused
the mess we find ourselves in now. Let's skip the history bit.

> I have to disagree.  With this setup, we can support architectures that
> are neither known by dpkg nor dak.  This was not possible before and
> required patching, aka making a derivative of the package.  You can
> still patch and rebuild it how you want and inject that modified package
> into your workflow.
> 
> But now you can have a port without hand patched linux that is built in
> a non-standard way and relying on internal and unstable package API,
> just be talking and getting it added to the package first.

Fair enough. Please add arc, cksy, mipsr6el, sh3, sparc, and
musl-linux-any. Thanks in advance.

> > C. Inability to produce a linux-libc-dev-$arch-cross package
...
> This sounds like a bug in dpkg-cross, as it uses a not agreed on
> interface to do it's work.

I stand corrected. dpkg-cross actually can convert arch:all packages in
a reasonable way. Please consider item C a non-issue.

> No, it can not.  We can not have arch:any -> arch:all dependencies
> within the same source package this deep in the toolchain, especially as
> those require = dependencies.
>
> If we talk about size, please go to gcc, which ships unstripped
> binaries parts of the time.  We don't need to talk about a highly
> compressible 10 MB installed (up from 7 or so before) package.

I read this as you saying that size was not a motivation to change
linux-libc-dev from Arch:any to Arch:all. But then, I do not understand
the benefits of the change and why it was done in the first place.
Obviously, we are still dealing with the fallout of an Arch:all
linux-libc-dev after more than half a year. This bug is evidence of that
dealing and demonstrates the significance of the downsides. Please
enlighten me about the benefits.

Helmut

[toc] | [prev] | [next] | [standalone]

#84345

[Multipart message — attachments visible in raw view] — view raw

On Sat, 2024-10-19 at 18:15 +0200, Helmut Grohne wrote:
> Hi Bastian,
> 
> On Sat, Oct 19, 2024 at 03:35:21PM +0200, Bastian Blank wrote:
[...]
> 
> > But now you can have a port without hand patched linux that is built in
> > a non-standard way and relying on internal and unstable package API,
> > just be talking and getting it added to the package first.
> 
> Fair enough. Please add arc, cksy, mipsr6el, sh3, sparc, and
> musl-linux-any. Thanks in advance.
[...]

We would need to know the multiarch triplet for csky.

Ben.


-- 
Ben Hutchings
Klipstein's 4th Law of Prototyping and Production:
                               A fail-safe circuit will destroy others.

[toc] | [prev] | [next] | [standalone]

#84691

Hi,

I skipped replying here earlier, because I felt like I would not add any
content. Stefano indicated that my impression was wrong. Hence replying
now.

On Thu, Oct 24, 2024 at 01:31:32AM +0200, Ben Hutchings wrote:
> On Sat, 2024-10-19 at 18:15 +0200, Helmut Grohne wrote:
> > Hi Bastian,
> > 
> > On Sat, Oct 19, 2024 at 03:35:21PM +0200, Bastian Blank wrote:
> [...]
> > 
> > > But now you can have a port without hand patched linux that is built in
> > > a non-standard way and relying on internal and unstable package API,
> > > just be talking and getting it added to the package first.
> > 
> > Fair enough. Please add arc, cksy, mipsr6el, sh3, sparc, and
> > musl-linux-any. Thanks in advance.

In retrospect, the use of irony was not the best of moves here as it did
not add clarity to a discussion that was already heated. I'm sorry about
that.

> We would need to know the multiarch triplet for csky.

Yes, exactly. Let me reiterate the reasons for why I prefer a different
solution here.

  * In general terms, the multiarch triplet is something that
    can change (and has changed) during the early phase of an
    architecture bootstrap.
  * When an architecture becomes available in toolchain packages, it is
    not clear whether it takes off. or1k very much looked like the next
    thing and now it is deleted. The story for tilegx is similar.
    Initially, riscv64 looked a lot like or1k. In not adding these to
    src:linux (and removing them later), we avoid a pile of unnecessary
    interaction. The excessive addition of architectures also interacts
    with #1081826.
  * It is very useful to locally add a new architecture and try a
    bootstrap to give feedback to upstreams as to what piece is missing.
    Not requiring src:linux to cooperate significantly speeds up this
    evaluation process.

I generally see src:linux as one of the later pieces relevant to
architecture cross bootstrap in the sense that while it is one of the
earliest packages needed for its headers, adding real support is often
delayed, because too many out-of-tree patches are required during the
initial bootstrap phase and thus everyone uses a self-built vendor
kernel there.

Automatically modifying src:linux is quite feasible. It has changed a
couple of times and now roughly looks like this:

    cat >debian/config.local/defines.toml >/dev/null <<EOF
    [[kernelarch]]
    name = '$kernel_arch'
      [[kernelarch.debianarch]]
      name = '$deb_arch'
    EOF
    ./debian/rules debian/rules.gen

This is quite mechanical and therein lies value. The difficulty now has
become that src:linux is the only package in the entire architecture
cross bootstrap that builds an architecture-dependent Arch:all package.
I have not yet found a good way to deal with this oxymoron.

Helmut

[toc] | [prev] | [next] | [standalone]

#84733

On Sat, Oct 19, 2024 at 06:15:57PM +0200, Helmut Grohne wrote:
> On Sat, Oct 19, 2024 at 03:35:21PM +0200, Bastian Blank wrote:
> > On Thu, Oct 10, 2024 at 06:03:24PM +0200, Helmut Grohne wrote:
> > > A. Multi-Arch: foreign is a lie
> > All the examples of yours will immediatelly fail to build anything with
> > it.  So what harm is done?
> This is practically untrue. Even with current unstable packages, the
> sparc (32) bootstrap still succeeds.

I don't understand what succeeds.  Because missing asm/* headers pretty
much makes it impossible to use anything from linux/*.  So the bootstrap
either have it's own headers or it will bail out if it tries to include
any kernel interface.

Can you please share the log for it?

> > But now you can have a port without hand patched linux that is built in
> > a non-standard way and relying on internal and unstable package API,
> > just be talking and getting it added to the package first.
> Fair enough. Please add arc, cksy, mipsr6el, sh3, sparc, and
> musl-linux-any. Thanks in advance.

Please make proper requests that are actionable.

- arc: in dpkg. bootstrap plan is where?
- cksy: not in dpkg, so not actionable, as triplet is not known.
  bootstrap plan is where?
- mipsr6el: which one do you mean?
  /usr/include/mipsisa64r6el-linux-gnuabi64?
- sh3: was never in debian. bootstrap plan?
- sparc: was removed years ago. bootstrap plan?
- musl-linux-any: how is that supposed to work?

Bastian

-- 
No one can guarantee the actions of another.
		-- Spock, "Day of the Dove", stardate unknown

[toc] | [prev] | [next] | [standalone]

#84736

Hi Bastian,

I think much of what you are asking here is already answered in my reply
to Ben's followup.

On Wed, Dec 04, 2024 at 10:33:28PM +0100, Bastian Blank wrote:
> On Sat, Oct 19, 2024 at 06:15:57PM +0200, Helmut Grohne wrote:
> > On Sat, Oct 19, 2024 at 03:35:21PM +0200, Bastian Blank wrote:
> > > On Thu, Oct 10, 2024 at 06:03:24PM +0200, Helmut Grohne wrote:
> > > > A. Multi-Arch: foreign is a lie
> > > All the examples of yours will immediatelly fail to build anything with
> > > it.  So what harm is done?
> > This is practically untrue. Even with current unstable packages, the
> > sparc (32) bootstrap still succeeds.
> 
> I don't understand what succeeds.  Because missing asm/* headers pretty
> much makes it impossible to use anything from linux/*.  So the bootstrap
> either have it's own headers or it will bail out if it tries to include
> any kernel interface.
> 
> Can you please share the log for it?

You may find all the logs at
https://jenkins.debian.net/view/rebootstrap. Indeed, I missed the fact
that sparc is being patched in. And with patching it in, the complete
sparc bootstrap is still successful at this time. So yeah, sparc
immediately fails, but the main reason for that is linux and not some
other component.

I'm not gonna argue about sparc in particular. I'll just keep it alive
in rebootstrap as long as it is low effort. Once actual porting work
becomes necessary, I'll poke John Paul Adrian Glaubitz and if nothing
happens, discard sparc from my bootstrapping efforts.

So these architectures only served as examples here. mipsel and armel
look like they could be the next sparc.

> > Fair enough. Please add arc, cksy, mipsr6el, sh3, sparc, and
> > musl-linux-any. Thanks in advance.
> 
> Please make proper requests that are actionable.

I think you accurately captured the point of the request: It is not
actionable. The architectures referenced are work in progress to
different extents. You'll find more detail in my other mail.

Helmut

[toc] | [prev] | [next] | [standalone]

#84346

[Multipart message — attachments visible in raw view] — view raw

On Sat, 2024-10-19 at 15:35 +0200, Bastian Blank wrote:
> Hi
> 
> On Thu, Oct 10, 2024 at 06:03:24PM +0200, Helmut Grohne wrote:
[...]
> 
> > Ben asked me whether linux-libc-dev could simply support all relevant
> > architectures, but there often is a check&egg problem. The linux package
> > can only support architectures that dpkg knows about, but we want to
> > perform test bootstraps way before adding a triplet to dpkg and one of
> > the first things a test bootstrap does is building linux-libc-dev. So I
> > generally expect that linux-libc-dev can never provide support for all
> > architectures of interest and it shouldn't have to. Only once the
> > bootstrap is tested and the parameters are known do we want to file the
> > patches for a new architecture.
> 
> I have to disagree.  With this setup, we can support architectures that
> are neither known by dpkg nor dak.  This was not possible before and
> required patching, aka making a derivative of the package.  You can
> still patch and rebuild it how you want and inject that modified package
> into your workflow.
[...]

Currently gencontrol.py runs dpkg-architecture to find the multiarch
triplet for each Debian architecture.  So it's not possible to add any
architecture that's unknown to dpkg in unstable.

Are you proposing to extend our config schema so we can define those
triplets within src:linux?

Do we really want to be on the critical path for the early stages of
defining a new architecture, including any renaming that might happen
before it's added to dpkg upstream?


Ben.

-- 
Ben Hutchings
Klipstein's 4th Law of Prototyping and Production:
                               A fail-safe circuit will destroy others.

[toc] | [prev] | [next] | [standalone]

#84734

On Thu, Oct 24, 2024 at 01:29:31AM +0200, Ben Hutchings wrote:
> On Sat, 2024-10-19 at 15:35 +0200, Bastian Blank wrote:
> > I have to disagree.  With this setup, we can support architectures that
> > are neither known by dpkg nor dak.  This was not possible before and
> > required patching, aka making a derivative of the package.  You can
> > still patch and rebuild it how you want and inject that modified package
> > into your workflow.
> Currently gencontrol.py runs dpkg-architecture to find the multiarch
> triplet for each Debian architecture.  So it's not possible to add any
> architecture that's unknown to dpkg in unstable.
> Are you proposing to extend our config schema so we can define those
> triplets within src:linux?

I did add that support first.

> Do we really want to be on the critical path for the early stages of
> defining a new architecture, including any renaming that might happen
> before it's added to dpkg upstream?

Which early stages?  We have three stages:
- Before the arch reaches anything related to Debian.  In this stage, it
  does not matter if the package it all or any, it can be patched.
- While it lives on -ports, but dak does not know about it.  In this
  stage, the package needs to be managed manually.
- While it lived on -ports and dak knows about.  In this stage, support
  can be added as arch:any.

Bastian

-- 
... The prejudices people feel about each other disappear when they get
to know each other.
		-- Kirk, "Elaan of Troyius", stardate 4372.5

[toc] | [prev] | [next] | [standalone]

#84737

Hi Bastian,

On Wed, Dec 04, 2024 at 10:40:59PM +0100, Bastian Blank wrote:
> On Thu, Oct 24, 2024 at 01:29:31AM +0200, Ben Hutchings wrote:
> > Do we really want to be on the critical path for the early stages of
> > defining a new architecture, including any renaming that might happen
> > before it's added to dpkg upstream?
> 
> Which early stages?  We have three stages:
> - Before the arch reaches anything related to Debian.  In this stage, it
>   does not matter if the package it all or any, it can be patched.

This is the stage I am concerned about. You claim that it would not
matter whether the package is all or any, but my experience is
otherwise. With the exception of linux, the cross bootstrap tooling does
not build any Arch:all packages at all. Adding that support has not been
trivial and the way it is implemented still is violating internal
principles making its code fragile. In particular, for each combination
of package name and architecture, there would only be one binary package
- with the exception of linux where the would now be two (the unpatched
one from the archive and the patched one). I'd have to somehow remove
linux-libc-dev from an unstable amd64 mirror.

The all versus any also poses the problem that it becomes unclear
whether it has to be rebuilt. When it was :any, it always had to be
built, but given that its M-A:foreign is a lie and that we had to remove
the Provides, it is difficult to tell whether a rebuild is needed.

In now find it all surprising that we (at least Ben, Bastian and me)
agree on patching linux at this stage being the right approach. My
earlier impression was that Bastian preferred src:linux to participate
in this early stage of the bootstrap and I have expended mails to argue
otherwise, but that seems unnecessary now.

Instead, the question becomes how we can make that patching easy and the
disagreement becomes whether Arch:all makes that sufficiently easy. Even
though I cite building Arch:all as difficult in the bootstrap context,
the more pressing issue from my side is the inability to tell whether a
rebuild is needed by looking at package relations. Dealing with
unsatisfiable dependencies is something, I've developed ways to deal
with. Dealing with wrongly satisfied dependencies is a totally different
thing to me. As it stands, linux currently is the only package involved
in architecture cross bootstrap that produces false positives in
dependency satisfiability tests used for scheduling builds.

> - While it lives on -ports, but dak does not know about it.  In this
>   stage, the package needs to be managed manually.
> - While it lived on -ports and dak knows about.  In this stage, support
>   can be added as arch:any.

As an architecture enters -ports for the first time, it must have its
triplet included in dpkg. At that time, I think it is reasonable to ask
src:linux to include its headers. linux also is uploaded so frequently
that it will not be practically blocking any port progress. The step
that usually takes much time and convincing is getting an architecture
into dpkg. I do not see an Arch:all linux-libc-dev as being a hindrance
to these later stages.

Helmut

[toc] | [prev] | [next] | [standalone]

#84739

On Thu, Dec 05, 2024 at 08:38:35AM +0100, Helmut Grohne wrote:
> This is the stage I am concerned about. You claim that it would not
> matter whether the package is all or any, but my experience is
> otherwise. With the exception of linux, the cross bootstrap tooling does
> not build any Arch:all packages at all. Adding that support has not been
> trivial and the way it is implemented still is violating internal
> principles making its code fragile. In particular, for each combination
> of package name and architecture, there would only be one binary package
> - with the exception of linux where the would now be two (the unpatched
> one from the archive and the patched one). I'd have to somehow remove
> linux-libc-dev from an unstable amd64 mirror.

So this tooling does
- patch the source,
- _not_ change the version,
- build non-all?

This sounds broken, as it already relies on those packages producing
compatible all/non-all packages from two different sources.  Yes, in
practice this will work.

To make the double package problem easier, just change the version with
a sufficiently high epoch?

> Instead, the question becomes how we can make that patching easy and the
> disagreement becomes whether Arch:all makes that sufficiently easy. Even
> though I cite building Arch:all as difficult in the bootstrap context,
> the more pressing issue from my side is the inability to tell whether a
> rebuild is needed by looking at package relations. Dealing with
> unsatisfiable dependencies is something, I've developed ways to deal
> with. Dealing with wrongly satisfied dependencies is a totally different
> thing to me. As it stands, linux currently is the only package involved
> in architecture cross bootstrap that produces false positives in
> dependency satisfiability tests used for scheduling builds.

What kind of information can this dependency satisfiability test use?

Something like this?

| Provides:
|   linux-libc-dev-supports (= amd64-0),
|   linux-libc-dev-supports (= arm64-0),
|   linux-libc-dev-supports-multiarch (= aarch64-linux-gnu-0),
|   linux-libc-dev-supports-multiarch (= x86-64-linux-gnu-0),

Bastian

-- 
Vulcans worship peace above all.
		-- McCoy, "Return to Tomorrow", stardate 4768.3

[toc] | [prev] | [next] | [standalone]

#84741

Hi Bastian,

On Thu, Dec 05, 2024 at 03:30:53PM +0100, Bastian Blank wrote:
> What kind of information can this dependency satisfiability test use?
> 
> Something like this?
> 
> | Provides:
> |   linux-libc-dev-supports (= amd64-0),
> |   linux-libc-dev-supports (= arm64-0),
> |   linux-libc-dev-supports-multiarch (= aarch64-linux-gnu-0),
> |   linux-libc-dev-supports-multiarch (= x86-64-linux-gnu-0),

Using Provides is the natural approach indeed. Encoding the architecture
into the version may technically work, but it feels really strange.
Earlier, I proposed encoding it into the provided package name like
linux-libc-dev-arm64-cross, but we all know how that went and I would
not have proposed it if I had seen how it broke other pieces. Still if
we were to just drop the "-cross" suffix and go for a very similar
version.

Provides: linux-libc-dev-amd64, linux-libc-dev-arm64, ...

In any of these cases, I can inject extra dependencies and have
dose-builddebcheck correctly determine whether the existing
linux-libc-dev satisfies what is needed.

As stated elsewhere, I still don't understand what we gained by
switching from Arch:any to Arch:all, but maybe I don't have to. A
solution that adds any of these Provides is what I'd call good enough in
practical terms for the purpose of bootstrapping.

Helmut

[toc] | [prev] | [next] | [standalone]

#85304

Hi Bastian:

> > Something like this?
> > 
> > | Provides:
> > |   linux-libc-dev-supports (= amd64-0),
> > |   linux-libc-dev-supports (= arm64-0),
> > |   linux-libc-dev-supports-multiarch (= aarch64-linux-gnu-0),
> > |   linux-libc-dev-supports-multiarch (= x86-64-linux-gnu-0),
> 
> Using Provides is the natural approach indeed. Encoding the architecture
> into the version may technically work, but it feels really strange.
> Earlier, I proposed encoding it into the provided package name like
> linux-libc-dev-arm64-cross, but we all know how that went and I would
> not have proposed it if I had seen how it broke other pieces. Still if
> we were to just drop the "-cross" suffix and go for a very similar
> version.
> 
> Provides: linux-libc-dev-amd64, linux-libc-dev-arm64, ...

We have two proposed provides schemes here, can we select one and add
it?

> As stated elsewhere, I still don't understand what we gained by
> switching from Arch:any to Arch:all, but maybe I don't have to. A
> solution that adds any of these Provides is what I'd call good enough in
> practical terms for the purpose of bootstrapping.

It would be helpful if this question could be answered. I think Matthias
was asking the same question in #1081826.

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#85378

Hi

On Mon, Jan 27, 2025 at 07:36:21PM -0400, Stefano Rivera wrote:
> > Provides: linux-libc-dev-amd64, linux-libc-dev-arm64, ...
> We have two proposed provides schemes here, can we select one and add
> it?

Something like simple providers is the easiest to do.

> > As stated elsewhere, I still don't understand what we gained by
> > switching from Arch:any to Arch:all, but maybe I don't have to.

And I currently have no idea what to say, except to repeat myself.

> It would be helpful if this question could be answered. I think Matthias
> was asking the same question in #1081826.

I read and re-read that.  Where is a question in this single message?
Just a: this is wrong.  Not, what is broken by it, which makes it wrong.

Bastian

-- 
Even historians fail to learn from history -- they repeat the same mistakes.
		-- John Gill, "Patterns of Force", stardate 2534.7

[toc] | [prev] | [next] | [standalone]

#85379

Hi Bastian (2025.02.04_14:05:07_+0000)
> > > Provides: linux-libc-dev-amd64, linux-libc-dev-arm64, ...
> > We have two proposed provides schemes here, can we select one and add
> > it?
>
> Something like simple providers is the easiest to do.

So, by implementing this Provides scheme, Helmut's cross-bootstrapping
toolchain can determine which architectures are supported, and have a
target to Depend on.

The resolution of the issue brought to the tech-ctte requires figuring
out whether linux-libc-dev will take over providing these headers for
cross toolchain building, or whether cross-toolchain-base continues to
own the linux-libc-dev-${arch}-cross virtual package names (and provides
symlinks or duplicates of headers).

Thanks again for temporarily releasing the -cross virtual package names
while this is being discussed.

Before we can come to a conclusion on that, the elephant in the room
seems to be the question of the motivation for the migration to an
architecture all linux-libc-dev package. Let me try to distil this
question:

What is being gained by the transition to an architecture independent
package?

I see several pros and cons:

1. Advantage: Reduced overall disk usage on mirrors. 1x 2MiB
   package replaces per-architecture 2MiB packages. For context, the
   linux source produces ~1GiB of binaries per arch.
2. Advantage: This allows taking over linux-libc-dev-{arch}-cross, as
   virtual packages, reducing the need for cross-toolchain-base to
   duplicate linux source. And avoiding the need for cross-building to
   create this architecture: all binary packages in bootstrap.
3. Disadvantage: Users get ~40 extra files (totalling 100kiB) per
   architecture in Debian installed when they install linux-libc-dev.
   (bug #1081826)
4. Disadvantage: linux-libc-dev is installable on architectures that it
   doesn't actually support. No other package involved in
   cross-bootstrapping exhibits this. This only applies to architectures
   that aren't supported in the linux source package (helmut's A)
5. Disadvantage: cross-build tooling needs special heuristics to
   determine which architectures linux-libc-dev actually supports.
   (helmut's B) The Provides scheme above supports that.
6. Disadvantage: cross-build tooling doesn't currently support building
   architecture independent packages. And that's quite understandable,
   they should never need to be built unless we hit a situation like
   this.

Is there anything that we're missing here?

Item 6 leads to more complex questions like how do we handle versions of
such packages? That's the fundamental issue here and it's where it gets
gnarly. It begs the question of whether it reasonable to patch sources
and build a binary package without a bumped source version.

When I get to this point, my feeling is: Yes, cross-building is breaking
assumptions that the rest of the archive is making. But it's (until now)
been working fairly successfully while keeping the approach as simple as
possible. That sounds like on balance a good thing.

Should some of those assumptions be codified into policy, if we want
them? Probably.

The change of linux-libc-dev to architecture: all has the effect of
causing a lot of non-trivial work for people doing cross-bootstrapping.

The balance of advantages vs disadvantages does depend a bit on how
valuable you see item 2 being. I imagine Matthias doesn't see any value
there.

With the pros and cons that I've seen, the weight of balance seems to go
against the arch all package. This is why people keep asking what the
motivation for the change was. If you have other motivations, please
share them.

Obviously, undoing the change would be a lot of work for you and the
kernel team, and we wouldn't want to put that on you. I'm not pushing
you to reverse the decision.

It would help if everyone saw the impact that they were having on each
other, and was able to weigh the costs of each option. Once we have some
common ground, we can come to conclusions on the questions raised.

Thanks for taking the time to understand people's questions and answer
them.

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#85380

Hi

On Tue, Feb 04, 2025 at 05:25:08PM +0000, stefanor@debian.org wrote:
> So, by implementing this Provides scheme, Helmut's cross-bootstrapping
> toolchain can determine which architectures are supported, and have a
> target to Depend on.

I just read through rebootstrap, which I think is what Helmut is always
talking about.  It uses dpkg-cross, so is already using undefined
behavior of linux-libc-dev to do it's work.

My current version of it does this now:
| Provides: linux-libc-dev+support-debian+alpha (= ${binary:Version})

> Before we can come to a conclusion on that, the elephant in the room
> seems to be the question of the motivation for the migration to an
> architecture all linux-libc-dev package. Let me try to distil this
> question:
> 
> What is being gained by the transition to an architecture independent
> package?

> 4. Disadvantage: linux-libc-dev is installable on architectures that it
>    doesn't actually support. No other package involved in
>    cross-bootstrapping exhibits this. This only applies to architectures
>    that aren't supported in the linux source package (helmut's A)

Yes.  It only supports architectures that are defined as Debian.

> 5. Disadvantage: cross-build tooling needs special heuristics to
>    determine which architectures linux-libc-dev actually supports.
>    (helmut's B) The Provides scheme above supports that.

You mean "bootstrap tooling".  And yes, that part I missed in my initial
assessment,

> 6. Disadvantage: cross-build tooling doesn't currently support building
>    architecture independent packages. And that's quite understandable,
>    they should never need to be built unless we hit a situation like
>    this.

The problem was:  dpkg-cross was not able to do it's work with an
arch:all linux-libc-dev.  But it might have failed with any change of
linux-libc-dev, because it relies of undefined behaviour.  So not a
disadvantage, but inherent instability of the dpkg-cross approach.

Or actually an advantage, because we can remove this instability and
reduce the complexity.

> Is there anything that we're missing here?

- It can provide headers for multiple architectures at the same time.
- It could even handle debian architectures mapping to multiple
  multiarch identifier and kernel arches (like hppa did kind of).
- debian-ports can import it without going through the manually handled
  "unreleased" part of their archive, even before ftp-master knows about
  an architecture.
- Reduces the part of the package API that is used in an undocumented
  way via dpkg-cross.

> Should some of those assumptions be codified into policy, if we want
> them? Probably.

I don't know.  Right now the whole cross-building stuff violates the
policy, how would we clean this up?

I'm not even sure about arch:all.  We currently have no problems with
such packages only working on some architectures, if by dependency or
actually failing to work.

> It would help if everyone saw the impact that they were having on each
> other, and was able to weigh the costs of each option. Once we have some
> common ground, we can come to conclusions on the questions raised.

The impact was rather small.  Only two other parts where actually
affected, and none of them was our endusers.

The bootstrap stuff was affected, which I missed at first.  But is all
contained within two scripts that control everything.  And the rest of
that script is patching other packages quite havily already.

The other is cross-toolchain-base.  But here the impact is still
completelly unknown or at least does not show up within Debian.

Bastian

-- 
It is a human characteristic to love little animals, especially if
they're attractive in some way.
		-- McCoy, "The Trouble with Tribbles", stardate 4525.6

[toc] | [prev] | [next] | [standalone]

#85382

Hi,

On Tue, Feb 04, 2025 at 05:25:08PM +0000, stefanor@debian.org wrote:
> Hi Bastian (2025.02.04_14:05:07_+0000)
> > > > Provides: linux-libc-dev-amd64, linux-libc-dev-arm64, ...
> > > We have two proposed provides schemes here, can we select one and add
> > > it?
> >
> > Something like simple providers is the easiest to do.
> 
> So, by implementing this Provides scheme, Helmut's cross-bootstrapping
> toolchain can determine which architectures are supported, and have a
> target to Depend on.
> 
> The resolution of the issue brought to the tech-ctte requires figuring
> out whether linux-libc-dev will take over providing these headers for
> cross toolchain building, or whether cross-toolchain-base continues to
> own the linux-libc-dev-${arch}-cross virtual package names (and provides
> symlinks or duplicates of headers).
> 
> Thanks again for temporarily releasing the -cross virtual package names
> while this is being discussed.
> 
> Before we can come to a conclusion on that, the elephant in the room
> seems to be the question of the motivation for the migration to an
> architecture all linux-libc-dev package. Let me try to distil this
> question:
> 
> What is being gained by the transition to an architecture independent
> package?

In addition to all listed points, it also temporarily breaks
creating chroots for archs when the src:linux build for a given arch
and for arch:all end up in different dinstall windows. In this time
linux-libc-dev will not be visible to debootstrap. This behaviour
was observed f.e. on 2025-02-03.

Chris

[toc] | [prev] | [next] | [standalone]

#86334

Hi Bastian (2025.02.04_13:25:08_-0400)
>The resolution of the issue brought to the tech-ctte requires figuring
>out whether linux-libc-dev will take over providing these headers for
>cross toolchain building, or whether cross-toolchain-base continues to
>own the linux-libc-dev-${arch}-cross virtual package names (and provides
>symlinks or duplicates of headers).

I've had a few IRC conversations with both Bastian and Matthias about 
what comes next. No conclusions yet, but the ground we've covered is:

Matthias is asserting that the -cross package namespace and 
/usr/<triplet> sysroot directories belong to cross-toolchain-base. He'd 
like to keep control of those packages.

Bastian is concerned that if both linux-libc-dev and 
linux-libc-dev-*-cross provide linux headers in different paths they 
should conflict with each other to avoid compilers accidentally sourcing 
the wrong headers. linux-libc-dev installs in the usual multiarch 
include directory (/usr/include/$ARCH/linux/). linux-libc-dev-*-cross 
installs in the sysroot directory (/usr/$ARCH/include/linux/)

I would imagine that any conflicts like that would make 
linux-libc-dev-*-cross practically uninstallable in any situation where 
you'd need them (libc6-dev depends on linux-libc-dev).

So, we're back to trying to unpick this knot. The Techncial Committee is 
not supposed to do design work, but I can see some possible ways out of 
this.

I could see a scenario where linux-libc-dev-*-cross are tiny wrapper 
packages, depending on linux-libc-dev and including symlinks from one 
include directory to the other. I assume that wouldn't need the 
Conflicts?

Or if the Conflicts were more version-targetted and allow linux-libc-dev 
and linux-libc-dev-*-cross to co-install if they were built from the 
same linux source version, would that be acceptable?

Do either of you have any thoughts on how we should figure this out?

Stefano

-- 
Stefano Rivera
   http://tumbleweed.org.za/
   +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#86434

On Tue, Mar 04, 2025 at 12:47:37PM -0400, Stefano Rivera wrote:
> Bastian is concerned that if both linux-libc-dev and linux-libc-dev-*-cross
> provide linux headers in different paths they should conflict with each
> other to avoid compilers accidentally sourcing the wrong headers.
> linux-libc-dev installs in the usual multiarch include directory
> (/usr/include/$ARCH/linux/). linux-libc-dev-*-cross installs in the sysroot
> directory (/usr/$ARCH/include/linux/)
> 
> I would imagine that any conflicts like that would make
> linux-libc-dev-*-cross practically uninstallable in any situation where
> you'd need them (libc6-dev depends on linux-libc-dev).

Yes.  An unversioned conflict would make the whole cross stuff
effectively uninstallable.

> I could see a scenario where linux-libc-dev-*-cross are tiny wrapper
> packages, depending on linux-libc-dev and including symlinks from one
> include directory to the other. I assume that wouldn't need the Conflicts?

This is what I implemented as well, a symlink farm.  But those symlinks
needs to be belong to the same source package, as the set of required
symlinks change over time.  Or we are back at: requires conflicts.

I only consider symlinks to files acceptable.  With symlinks to
directories we are back in aliased directories land, with all the same
problems as merged-/usr.

> Or if the Conflicts were more version-targetted and allow linux-libc-dev and
> linux-libc-dev-*-cross to co-install if they were built from the same linux
> source version, would that be acceptable?

Only if the *-cross maintainer is fine with updating it within a day,
every week or so for unstable, but even for stable or security.  So, not
possible, given that the existing conflict for the same reason against
glibc was only solved by NMU after a week or so.

Bastian

-- 
Where there's no emotion, there's no motive for violence.
		-- Spock, "Dagger of the Mind", stardate 2715.1

[toc] | [prev] | [next] | [standalone]

#86451

On Thu, Mar 13, 2025 at 07:34:44AM +0100, Bastian Blank wrote:
> Only if the *-cross maintainer is fine with updating it within a day,
> every week or so for unstable, but even for stable or security.  So, not
> possible, given that the existing conflict for the same reason against
> glibc was only solved by NMU after a week or so.

This conflict is currently escalating with the glibc side, see #1100544.
The glibc maintainers decided to not longer take up with it after the
release of trixie.

Bastian

-- 
Actual war is a very messy business.  Very, very messy business.
		-- Kirk, "A Taste of Armageddon", stardate 3193.0

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.kernel

csiph-web