Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1271058 > unrolled thread

Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted]

Back to article view | Back to linux.debian.bugs.dist

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Adrian Bunk <bunk@debian.org> - 2025-11-21 16:10 +0100
    Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Graham Inggs <ginggs@debian.org> - 2025-11-22 13:10 +0100
      Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Adrian Bunk <bunk@debian.org> - 2025-11-22 21:30 +0100
        Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Graham Inggs <ginggs@debian.org> - 2025-11-22 22:00 +0100
          Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Adrian Bunk <bunk@debian.org> - 2025-11-24 05:30 +0100
            Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Graham Inggs <ginggs@debian.org> - 2025-11-25 17:50 +0100
              Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Adrian Bunk <bunk@debian.org> - 2025-11-26 07:20 +0100
                Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Matthias Maier <tamiko+debian@43-1.org> - 2025-11-26 19:20 +0100
                  Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Graham Inggs <ginggs@debian.org> - 2025-12-14 13:50 +0100
                    Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Adrian Bunk <bunk@debian.org> - 2025-12-31 15:00 +0100
            Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted] Emanuele Rocca <ema@debian.org> - 2025-11-26 12:10 +0100

#1271058 — Bug#1112274: trilinos: FTBFS on arm64: collect2: fatal error: ld terminated with signal 6 [Aborted]

On Fri, Sep 05, 2025 at 05:42:31PM +0200, Emanuele Rocca wrote:
> Hi,
> 
> On 2025-09-05 11:58, Adrian Bunk wrote:
> > +ifneq (,$(filter $(DEB_HOST_ARCH),amd64 ppc64el riscv64 s390x sparc64))
> > +  MOLD=-DCMAKE_CXX_FLAGS="-fuse-ld=mold"
> > +endif
> 
> This unsets all CXXFLAGS!

Ouch, thanks for spotting.

> One could keep the flags with -DCMAKE_CXX_FLAGS="${CXXFLAGS} -fuse-ld=mold",
> but it seems cleaner to just set the following, without passing
> -DCMAKE_CXX_FLAGS at all.
> 
> DEB_CXXFLAGS_MAINT_APPEND='-fuse-ld=mold'

Updated patch below.

This got the build time on riscv64 down from 1 week to 1 day with GCC 14
(both are higher with GCC 15 for unrelated reasons).

cu
Adrian


diff -Nru trilinos-16.1.0/debian/control trilinos-16.1.0/debian/control
--- trilinos-16.1.0/debian/control	2025-08-28 11:44:10.000000000 +0300
+++ trilinos-16.1.0/debian/control	2025-08-28 11:45:06.000000000 +0300
@@ -19,6 +19,7 @@
  liblapack-dev,
  libmumps-dev (>= 4.10),
  libptscotch-dev (>= 6.0.3),
+ mold [amd64 ppc64el riscv64 s390x sparc64],
  openmpi-bin,
  zlib1g-dev
 Build-Depends-Indep: bc,
diff -Nru trilinos-16.1.0/debian/rules trilinos-16.1.0/debian/rules
--- trilinos-16.1.0/debian/rules	2025-08-28 11:44:30.000000000 +0300
+++ trilinos-16.1.0/debian/rules	2025-08-28 11:45:06.000000000 +0300
@@ -75,6 +75,11 @@
 override_dh_auto_configure: configure-stamp
 
 
+# mold is much faster but doesn't work everywhere, see #1112274
+ifneq (,$(filter $(DEB_HOST_ARCH),amd64 ppc64el riscv64 s390x sparc64))
+  export DEB_CXXFLAGS_MAINT_APPEND+=-fuse-ld=mold
+endif
+
 # The below configuration selectively enables packages instead of first
 # enabling all via
 # ```

[toc] | [next] | [standalone]

#1271157

Emanuele wrote on irc that it may be possible to use mold on arm64.
Maybe the same workaround could be used for other architectures?

However, I am hesitant to switch to mold in Debian now, as mold has
some RC bugs that would make trilinos eligible for removal from
testing.

For what it's worth, trilinos builds fine in Ubuntu on arm64 (without
the workaround), and using mold there is needed to avoid running out
of space on their buildds.


[00:00:00] - {Day changed to Monday, 15 September 2025}

[09:53:48] <ema> ginggs: while I have your ear, it seems that building
trilinos with -ffunction-sections is an effective workaround for
#1112274

[09:53:50] [zwiebelbot] Debian#1112274: trilinos: FTBFS on arm64:
collect2: fatal error: ld terminated with signal 6 [Aborted] -
https://bugs.debian.org/1112274

[09:54:03] <ema> apparently having smaller .text sections helps mold
not falling on its face

[09:55:45] <ema> I tried rebuilding both without mold and and with
mold (and -ffunction-sections), didn't see any performance improvement
really

[09:56:43] <ema> actually a slight degradation: 01:59:40 with mold and
01:52:19 without


[1] https://launchpad.net/ubuntu/+source/trilinos/16.1.0-2ubuntu1

[toc] | [prev] | [next] | [standalone]

#1271226

On Sat, Nov 22, 2025 at 11:03:24AM -0100, Graham Inggs wrote:
> Emanuele wrote on irc that it may be possible to use mold on arm64.
> Maybe the same workaround could be used for other architectures?

The mold upstream maintainer is active and responsive with fixing bugs,
an upstream bug report would be the proper way forward for architecture 
issues.

I have an item somewhere on my (long) TODO list that I want to check 
using mold for more packages, and into what architecture issues this 
might run. My impression so far is that mold might not yet be mature 
enough for widespread usage.

My short-term aim is to shorten the ongoing numerical stack transition 
by a week on riscv64, since the latest build of trilinos took 11 days.[1]

> However, I am hesitant to switch to mold in Debian now, as mold has
> some RC bugs that would make trilinos eligible for removal from
> testing.

I've just uploaded a fix for that.

Mid-term using mold likely runs into #1114210, but I am optimistic that 
upstream will fix this in the foreseeable future.

> For what it's worth, trilinos builds fine in Ubuntu on arm64 (without
> the workaround), and using mold there is needed to avoid running out
> of space on their buildds.

Ubuntu is still using mold 2.37.1+dfsg-1 due to #1119807,
it is possible that this is a recent arm64 regression in mold.

> [00:00:00] - {Day changed to Monday, 15 September 2025}
> 
> [09:53:48] <ema> ginggs: while I have your ear, it seems that building
> trilinos with -ffunction-sections is an effective workaround for
> #1112274
> 
> [09:53:50] [zwiebelbot] Debian#1112274: trilinos: FTBFS on arm64:
> collect2: fatal error: ld terminated with signal 6 [Aborted] -
> https://bugs.debian.org/1112274
> 
> [09:54:03] <ema> apparently having smaller .text sections helps mold
> not falling on its face
> 
> [09:55:45] <ema> I tried rebuilding both without mold and and with
> mold (and -ffunction-sections), didn't see any performance improvement
> really
> 
> [09:56:43] <ema> actually a slight degradation: 01:59:40 with mold and
> 01:52:19 without
>...

Perhaps that's due to -ffunction-sections?

A factor of 3-5 in the build time seems normal when comparing 16.1.0-1
with 16.1.0-2 on the other architectures:
https://buildd.debian.org/status/logs.php?pkg=trilinos&arch=amd64
https://buildd.debian.org/status/logs.php?pkg=trilinos&arch=ppc64el
https://buildd.debian.org/status/logs.php?pkg=trilinos&arch=s390x
https://buildd.debian.org/status/logs.php?pkg=trilinos&arch=sparc64

My "This got the build time on riscv64 down from 1 week to 1 day with 
GCC 14" was also something I did measure back then.[2]

Buildtime of trilinos on arm64 is not a huge issue even though it's
~ 18 hours on some buildds, your Ubuntu change plus -ffunction-sections
on arm64 might be a good short-term solution?

cu
Adrian

[1] GCC 15 brings additional build time regression here:
    https://lists.debian.org/debian-riscv/2025/09/msg00019.html
[2] I don't remember exact numbers, but with GCC 14 the build finished
    in < 1 day on the porterbox (same hardware as the buildds)

[toc] | [prev] | [next] | [standalone]

#1271230

Hi Adrian

On Sat, 22 Nov 2025 at 19:22, Adrian Bunk <bunk@debian.org> wrote:
> My short-term aim is to shorten the ongoing numerical stack transition
> by a week on riscv64, since the latest build of trilinos took 11 days.[1]

Please feel free to upload as you see fit.  You can commit to salsa
and/or NMU, whichever you prefer.

Regards
Graham

[toc] | [prev] | [next] | [standalone]

#1271381

On Sat, Nov 22, 2025 at 07:53:42PM -0100, Graham Inggs wrote:
> Hi Adrian

Hi Graham,

> On Sat, 22 Nov 2025 at 19:22, Adrian Bunk <bunk@debian.org> wrote:
> > My short-term aim is to shorten the ongoing numerical stack transition
> > by a week on riscv64, since the latest build of trilinos took 11 days.[1]
> 
> Please feel free to upload as you see fit.  You can commit to salsa
> and/or NMU, whichever you prefer.

done (NMU and git), but:

On Sat, Nov 22, 2025 at 10:22:36PM +0200, Adrian Bunk wrote:
> On Sat, Nov 22, 2025 at 11:03:24AM -0100, Graham Inggs wrote:
>...
> > [09:53:48] <ema> ginggs: while I have your ear, it seems that building
> > trilinos with -ffunction-sections is an effective workaround for
> > #1112274
> >
> > [09:53:50] [zwiebelbot] Debian#1112274: trilinos: FTBFS on arm64:
> > collect2: fatal error: ld terminated with signal 6 [Aborted] -
> > https://bugs.debian.org/1112274
> >
> > [09:54:03] <ema> apparently having smaller .text sections helps mold
> > not falling on its face
> >
> > [09:55:45] <ema> I tried rebuilding both without mold and and with
> > mold (and -ffunction-sections), didn't see any performance improvement
> > really
> >
> > [09:56:43] <ema> actually a slight degradation: 01:59:40 with mold and
> > 01:52:19 without
> >...
>
> Perhaps that's due to -ffunction-sections?
>
> A factor of 3-5 in the build time seems normal when comparing 16.1.0-1
> with 16.1.0-2 on the other architectures:
> https://buildd.debian.org/status/logs.php?pkg=trilinos&arch=amd64
> https://buildd.debian.org/status/logs.php?pkg=trilinos&arch=ppc64el
> https://buildd.debian.org/status/logs.php?pkg=trilinos&arch=s390x
> https://buildd.debian.org/status/logs.php?pkg=trilinos&arch=sparc64
>
> My "This got the build time on riscv64 down from 1 week to 1 day with
> GCC 14" was also something I did measure back then.[2]

I should really have double-checked that before uploading, because what 
is actually happening is rather unwanted - and I finally understood it 
after remembering:

On Fri, Sep 05, 2025 at 05:42:31PM +0200, Emanuele Rocca wrote:
>... 
> This unsets all CXXFLAGS!
>...

Already 16.1.0-1 did put mold in CXXFLAGS instead of LDFLAGS (using 
LDFLAGS seems to work), and this removed the -O2 resulting in building
without optimization - which made the build faster.

Emanuele, did you use a version built without optimization when 
discovering that -ffunction-sections helps?
Without -ffunction-sections it builds for me when optimization is not 
disabled, mold/arm64 seems to fail on the unoptimized code - which it 
shouldn't but that's not such a big issue.

Regarding 16.1.0-2ubuntu1, that fixed the Ubuntu build with the same 
"without optimization". Looking at the build logs of 16.1.0-2 in Ubuntu
it surprisingly failed due to lack of disk space (and not lack or RAM).
Maybe LTO requires more diskspace with -O2 than with -O0, and in that
case disabling LTO would help Ubuntu.

I would have a tendency to revert the mold change and instead do 
optimize=-lto, but whether the latter actually solves the problem
for Ubuntu is unproven.

I remembered that we also have a failure with mold on arm64 in deal.ii, 
and that is a variant of the same problem:
Manually reverting [1] fixes the deal.ii/arm64 build with mold,
but reverting mold usage might be the easiest fix.
Using -DCMAKE_BUILD_TYPE=Release for not additionally building and 
shipping a half GB debug version of the library would be another option.

> Regards
> Graham

cu
Adrian

[1] https://github.com/dealii/dealii/commit/b8d1c41fc07b9f0a759d60956613c27d753f0960 

[toc] | [prev] | [next] | [standalone]

#1271561

Hi Adrian

Thanks for digging into this!

On Mon, 24 Nov 2025 at 03:22, Adrian Bunk <bunk@debian.org> wrote:
> Already 16.1.0-1 did put mold in CXXFLAGS instead of LDFLAGS (using
> LDFLAGS seems to work), and this removed the -O2 resulting in building
> without optimization - which made the build faster.

So mold does not really speed things up?
Rather, the speed up is due to the accidental removal of the optimization flags?

> I would have a tendency to revert the mold change and instead do
> optimize=-lto, but whether the latter actually solves the problem
> for Ubuntu is unproven.

Do you mean reverting to the packaging state of 16.1.0-2?
If so, I am in favour.  The Ubuntu issues can be fixed in Ubuntu.

What about lowering optimization for riscv64?  Or just not building
there until the hardware is able to cope?

> I remembered that we also have a failure with mold on arm64 in deal.ii,
> and that is a variant of the same problem:
> Manually reverting [1] fixes the deal.ii/arm64 build with mold,
> but reverting mold usage might be the easiest fix.
> Using -DCMAKE_BUILD_TYPE=Release for not additionally building and
> shipping a half GB debug version of the library would be another option.

deal.ii has had ongoing problems with arm64 in Debian [2], which I
think predate the switch to mold.  I believe ftpmasters have removed
the arm64 binaries from testing at least three times in as many years.

Regards
Graham


[2] https://buildd.debian.org/status/logs.php?pkg=deal.ii&arch=arm64

[toc] | [prev] | [next] | [standalone]

#1271633

On Tue, Nov 25, 2025 at 03:41:38PM -0100, Graham Inggs wrote:
> Hi Adrian

Hi Graham,

> Thanks for digging into this!
> 
> On Mon, 24 Nov 2025 at 03:22, Adrian Bunk <bunk@debian.org> wrote:
> > Already 16.1.0-1 did put mold in CXXFLAGS instead of LDFLAGS (using
> > LDFLAGS seems to work), and this removed the -O2 resulting in building
> > without optimization - which made the build faster.
> 
> So mold does not really speed things up?
> Rather, the speed up is due to the accidental removal of the optimization flags?

yes.

> > I would have a tendency to revert the mold change and instead do
> > optimize=-lto, but whether the latter actually solves the problem
> > for Ubuntu is unproven.
> 
> Do you mean reverting to the packaging state of 16.1.0-2?

Yes.

> If so, I am in favour.  The Ubuntu issues can be fixed in Ubuntu.
> 
> What about lowering optimization for riscv64?  Or just not building
> there until the hardware is able to cope?
>...

I would rather try to avoid lowering optimization, and -O0 of huge C++ 
code would strike me as particularly bad.

Creating debug info is actually confirmed to be expensive for C++ code,
and GCC 15 on riscv64 even is significantly slower on that than GCC 14.[1]

For siconos (#1121341) the following brought the compilation of the 
problematic file from timeout after 12 hours on the buildd down to
~ 2 hours for me:
ifneq (,$(filter $(DEB_HOST_ARCH), riscv64))
  export DEB_CXXFLAGS_MAINT_APPEND += -g1
endif

I would guess that for trilinos this would get the compile time
from 11 days back down to ~ 3 days (untested).

> Regards
> Graham
>...

cu
Adrian

[1] https://lists.debian.org/debian-riscv/2025/09/msg00019.html

[toc] | [prev] | [next] | [standalone]

#1271721

Dear all,

Apologies that I caused all this mess by accidentally overriding CXX
flags.


On Wed, Nov 26, 2025, at 00:10 CST, Adrian Bunk <bunk@debian.org> wrote:

>> > LDFLAGS seems to work), and this removed the -O2 resulting in building
>> > without optimization - which made the build faster.
>> 
>> So mold does not really speed things up?
>> Rather, the speed up is due to the accidental removal of the optimization flags?
>
> yes.

I agree that the absence of optimization flags will speed up the
compilation process.

But, the whole reason why we switched linkage over to mold is that it
improved the link time for larg(er) shared library objects significantly
as opposed to ld.bfd - and this helped a lot with build times
(particularly for deal.ii) on some of the build daemons.

I am happy to quantify that for trilinos as well but here is a very
quick comparison (bfd 2.45.0 vs mold 2.40.4):

Link times for a small executable linking against Trilinos and a metric
ton of other libraries:

  -fuse-ld=mold:                 0.03s user 0.02s system 10% cpu  0.488 total
  -fuse-ld=mold -Wl,--nothreads: 0.05s user 0.02s system  7% cpu  1.024 total
  -fuse-ld=bfd:                  9.80s user 4.31s system 99% cpu 14.253 total

And for linking a very large shared library library (deal.ii):

  -fuse-ld=mold:                 0.05s user 0.04s system 15% cpu 0.534 total
  -fuse-ld=mold -Wl,--nothreads: 0.05s user 0.04s system  7% cpu 1.220 total
  -fuse-ld=bfd:                  2.52s user 1.38s system 99% cpu 3.915 total

(I have to say I am impressed by how much faster ld.bfd has become
compared to the last time I quantified this.)

The speed improvement for the over 60 shared libraries for Trilinos will
range somewhere in between.

Best,
Matthias

[toc] | [prev] | [next] | [standalone]

#1274333

Hi All

I uploaded trilinos 16.1.0-4, building without mold and also not
building the tests.  The tests seemed to be a major factor in the long
build times and disk usage.

Builds on arm64 (with no workaround), ppc64 and loong64 were
successful, where these had failed previously with mold.

Unfortunately, the build time on riscv64 seems to have increased from
about 32 hours to 52.

I'm open to taking a patch to improve the build time on riscv64, or
drop the architecture for now.

Regards
Graham

[toc] | [prev] | [next] | [standalone]

#1276498

On Sun, Dec 14, 2025 at 11:38:46AM -0100, Graham Inggs wrote:
> Hi All
> 
> I uploaded trilinos 16.1.0-4, building without mold and also not
> building the tests.  The tests seemed to be a major factor in the long
> build times and disk usage.
> 
> Builds on arm64 (with no workaround), ppc64 and loong64 were
> successful, where these had failed previously with mold.
> 
> Unfortunately, the build time on riscv64 seems to have increased from
> about 32 hours to 52.

This smaller difference might be due to mold.

> I'm open to taking a patch to improve the build time on riscv64, or
> drop the architecture for now.

Using mold and building with -g1 might get the build time under 1 day 
(untested), but even 52 hours is still less than the runtime of the 
testsuite of gcc (whose results are ignored) - and is in general not 
extraordinarily long.

I'm not saying this is good, but with the current buildds 11 days was 
too much but 2 days is OKish.

> Regards
> Graham

cu
Adrian

[toc] | [prev] | [next] | [standalone]

#1271665

Hi folks,

On 2025-11-24 06:22, Adrian Bunk wrote:
> Emanuele, did you use a version built without optimization when 
> discovering that -ffunction-sections helps?
> Without -ffunction-sections it builds for me when optimization is not 
> disabled, mold/arm64 seems to fail on the unoptimized code - which it 
> shouldn't but that's not such a big issue.

I did have -O2 in my flags. Looking through the old build logs:

-Wno-inline -Wno-deprecated-declarations -g -O2 -ffile-prefix-map=/build/reproducible-path/trilinos-16.1.0=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -mbranch-protection=standard -fuse-ld=mold -ffunction-sections -Wdate-time -D_FORTIFY_SOURCE=2 -std=c++17

Full logs here:
https://people.debian.org/~ema/trilinos_16.1.0-2_arm64-2025-09-09T09:52:25Z.build

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web