Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #83093 > unrolled thread

Bug#1076564: pahole BTF processing seems flaky on powerpc

Started byBen Hutchings <ben@decadent.org.uk>
First post2024-07-19 00:30 +0200
Last post2024-07-22 07:00 +0200
Articles 8 — 3 participants

Back to article view | Back to linux.debian.kernel

Contents

  Bug#1076564: pahole BTF processing seems flaky on powerpc Ben Hutchings <ben@decadent.org.uk> - 2024-07-19 00:30 +0200
    Re: Bug#1076564: pahole BTF processing seems flaky on powerpc Domenico Andreoli <cavok@debian.org> - 2024-07-19 23:30 +0200
      Re: Bug#1076564: pahole BTF processing seems flaky on powerpc John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> - 2024-07-20 09:20 +0200
        Re: Bug#1076564: pahole BTF processing seems flaky on powerpc Domenico Andreoli <cavok@debian.org> - 2024-07-20 09:50 +0200
        Re: Bug#1076564: pahole BTF processing seems flaky on powerpc Ben Hutchings <ben@decadent.org.uk> - 2024-07-20 21:30 +0200
          Re: Bug#1076564: pahole BTF processing seems flaky on powerpc John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> - 2024-07-20 21:50 +0200
            Re: Bug#1076564: pahole BTF processing seems flaky on powerpc Ben Hutchings <ben@decadent.org.uk> - 2024-07-22 00:10 +0200
              Re: Bug#1076564: pahole BTF processing seems flaky on powerpc John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> - 2024-07-22 07:00 +0200

#83093 — Bug#1076564: pahole BTF processing seems flaky on powerpc

FromBen Hutchings <ben@decadent.org.uk>
Date2024-07-19 00:30 +0200
SubjectBug#1076564: pahole BTF processing seems flaky on powerpc
Message-ID<J1HNn-34ej-5@gated-at.bofh.it>
Package: pahole
Version: 1.27-1
Severity: normal
X-Debbugs-Cc: debian-kernel@lists.debian.org

Several kernel builds on powerpc have failed recently:

6.8.12-1:    https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.8.12-1&stamp=1717234422&raw=1
6.9.9-1:     https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.9.9-1&stamp=1720906547&raw=1
6.10-1~exp1: https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1

Note, these log files are up to 270 MB in size and should be
downloaded; at least Firefox becomes unresponsive when trying to
display them.

For each of these, the failure seems to start with an error from
pahole such as:

    [102044] ARRAY (anon) type_id=99491 index_type_id=14 nr_elems=12 Error emitting BTF type
    Encountered error while encoding BTF.

This does *not* happen consistently.  Compare these successful
builds:

6.8.12-1:    https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.8.12-1&stamp=1717278092&raw=1
- This same version failed to build on the first try.
6.9.7-1:     https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.9.7-1&stamp=1719538806&raw=1
- Earlier and later 6.9.x versions failed to build.

Both pahole versions 1.26-1 and 1.27-1 have been used in both
successful and failing builds.

Ben.

-- System Information:
Debian Release: trixie/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'stable-updates'), (500, 'stable-security'), (500, 'oldstable-updates'), (500, 'oldstable-security'), (500, 'oldoldstable-updates'), (500, 'oldoldstable'), (500, 'unstable'), (500, 'stable'), (500, 'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.9.9-amd64 (SMP w/12 CPU threads; PREEMPT)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages pahole depends on:
ii  libbpf1     1:1.4.3-1
ii  libc6       2.39-4
ii  libdw1t64   0.191-2
ii  libelf1t64  0.191-2
ii  zlib1g      1:1.3.dfsg+really1.3.1-1

pahole recommends no packages.

pahole suggests no packages.

-- debconf-show failed

[toc] | [next] | [standalone]

#83121

FromDomenico Andreoli <cavok@debian.org>
Date2024-07-19 23:30 +0200
Message-ID<J23kR-3hh9-3@gated-at.bofh.it>
In reply to#83093

[Multipart message — attachments visible in raw view] — view raw

CCing debian-kernel and debian-powerpc

On Fri, Jul 19, 2024 at 04:13:00PM -0300, Arnaldo Carvalho de Melo wrote:
> Adding Alan and Jiri to the CC list.
> 
> On Fri, Jul 19, 2024 at 08:53:24AM +0200, Domenico Andreoli wrote:
> > Hi Arnaldo,
> > 
> > On Thu, Jul 18, 2024 at 11:56:09PM +0200, Ben Hutchings wrote:
> > > Package: pahole
> > > Version: 1.27-1
> > > Severity: normal
> > > X-Debbugs-Cc: debian-kernel@lists.debian.org
> > > 
> > > Several kernel builds on powerpc have failed recently:
> > > 
> > > 6.8.12-1:    https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.8.12-1&stamp=1717234422&raw=1
> > > 6.9.9-1:     https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.9.9-1&stamp=1720906547&raw=1
> > > 6.10-1~exp1: https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
> > > 
> > > Note, these log files are up to 270 MB in size and should be
> > > downloaded; at least Firefox becomes unresponsive when trying to
> > > display them.
> > > 
> > > For each of these, the failure seems to start with an error from
> > > pahole such as:
> > > 
> > >     [102044] ARRAY (anon) type_id=99491 index_type_id=14 nr_elems=12 Error emitting BTF type
> > >     Encountered error while encoding BTF.
> > 
> > Does the above error ring any bell?
> 
> Nope
>  
> > Is there anything I can do to help?
> 
> From the 6.10-1~exp1:
> https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
> file:
> 
> + LLVM_OBJCOPY=powerpc-linux-gnu-objcopy pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust .tmp_vmlinux.btf
> 
> Can I have access to that .tmp_vmlinux.btf file so that I can try to
> reproduce it here?

I don't have access to the build host (blaauw2) and I've some doubts
it would still have that file.

Maybe our kernel team and powerpc porters have suggestions?

Dom

-- 
rsa4096: 3B10 0CA1 8674 ACBA B4FE  FCD2 CE5B CF17 9960 DE13
ed25519: FFB4 0CC3 7F2E 091D F7DA  356E CC79 2832 ED38 CB05

[toc] | [prev] | [next] | [standalone]

#83129

FromJohn Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Date2024-07-20 09:20 +0200
Message-ID<J2cxP-3qdn-11@gated-at.bofh.it>
In reply to#83121
Hi Domenico,

On Fri, 2024-07-19 at 23:20 +0200, Domenico Andreoli wrote:
> > > Is there anything I can do to help?
> > 
> > From the 6.10-1~exp1:
> > https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
> > file:
> > 
> > + LLVM_OBJCOPY=powerpc-linux-gnu-objcopy pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust .tmp_vmlinux.btf
> > 
> > Can I have access to that .tmp_vmlinux.btf file so that I can try to
> > reproduce it here?
> 
> I don't have access to the build host (blaauw2) and I've some doubts
> it would still have that file.
> 
> Maybe our kernel team and powerpc porters have suggestions?

I have root access to all powerpc/ppc64 machines (buildds and porterbox).

I'm cleaning up the porterbox now, disk is quite full, then you can try
to build the kernel package on perotto.debian.net or I can try it myself.

I have seen the bug myself and I wanted to debug it, but the attempt was
foiled by the fact that the disk on perotto is full (again).

Will take care of it and let you know when it's (some hours).

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

[toc] | [prev] | [next] | [standalone]

#83130

FromDomenico Andreoli <cavok@debian.org>
Date2024-07-20 09:50 +0200
Message-ID<J2d0R-3qnl-5@gated-at.bofh.it>
In reply to#83129

[Multipart message — attachments visible in raw view] — view raw

On Sat, Jul 20, 2024 at 09:16:24AM +0200, John Paul Adrian Glaubitz wrote:
> Hi Domenico,

Hi!

> On Fri, 2024-07-19 at 23:20 +0200, Domenico Andreoli wrote:
> > > > Is there anything I can do to help?
> > > 
> > > From the 6.10-1~exp1:
> > > https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
> > > file:
> > > 
> > > + LLVM_OBJCOPY=powerpc-linux-gnu-objcopy pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust .tmp_vmlinux.btf
> > > 
> > > Can I have access to that .tmp_vmlinux.btf file so that I can try to
> > > reproduce it here?
> > 
> > I don't have access to the build host (blaauw2) and I've some doubts
> > it would still have that file.
> > 
> > Maybe our kernel team and powerpc porters have suggestions?
> 
> I have root access to all powerpc/ppc64 machines (buildds and porterbox).
> 
> I'm cleaning up the porterbox now, disk is quite full, then you can try
> to build the kernel package on perotto.debian.net or I can try it myself.
> 
> I have seen the bug myself and I wanted to debug it, but the attempt was
> foiled by the fact that the disk on perotto is full (again).
> 
> Will take care of it and let you know when it's (some hours).

That's great, thank you.

> 
> Adrian
> 
> -- 
>  .''`.  John Paul Adrian Glaubitz
> : :' :  Debian Developer
> `. `'   Physicist
>   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

-- 
rsa4096: 3B10 0CA1 8674 ACBA B4FE  FCD2 CE5B CF17 9960 DE13
ed25519: FFB4 0CC3 7F2E 091D F7DA  356E CC79 2832 ED38 CB05

[toc] | [prev] | [next] | [standalone]

#83136

FromBen Hutchings <ben@decadent.org.uk>
Date2024-07-20 21:30 +0200
Message-ID<J2nWh-3xqc-1@gated-at.bofh.it>
In reply to#83129

[Multipart message — attachments visible in raw view] — view raw

On Sat, 2024-07-20 at 09:16 +0200, John Paul Adrian Glaubitz wrote:
> Hi Domenico,
> 
> On Fri, 2024-07-19 at 23:20 +0200, Domenico Andreoli wrote:
> > > > Is there anything I can do to help?
> > > 
> > > From the 6.10-1~exp1:
> > > https://buildd.debian.org/status/fetch.php?pkg=linux&arch=powerpc&ver=6.10-1%7Eexp1&stamp=1721287862&raw=1
> > > file:
> > > 
> > > + LLVM_OBJCOPY=powerpc-linux-gnu-objcopy pahole -J -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func --lang_exclude=rust .tmp_vmlinux.btf
> > > 
> > > Can I have access to that .tmp_vmlinux.btf file so that I can try to
> > > reproduce it here?
> > 
> > I don't have access to the build host (blaauw2) and I've some doubts
> > it would still have that file.
> > 
> > Maybe our kernel team and powerpc porters have suggestions?
> 
> I have root access to all powerpc/ppc64 machines (buildds and porterbox).
> 
> I'm cleaning up the porterbox now, disk is quite full, then you can try
> to build the kernel package on perotto.debian.net or I can try it myself.
> 
> I have seen the bug myself and I wanted to debug it, but the attempt was
> foiled by the fact that the disk on perotto is full (again).

I had a go yesterday and ran into the same problem.  I couldn't
reproduce with a small kernel config (allnoconfig + BPF + DEBUG_INFO +
DEBUG_INFO_BTF) and there wasn't enough disk space to build even one of
the Debian kernel flavours.

> Will take care of it and let you know when it's (some hours).

Thank you!

Ben.

-- 
Ben Hutchings
This sentence contradicts itself - no actually it doesn't.

[toc] | [prev] | [next] | [standalone]

#83137

FromJohn Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Date2024-07-20 21:50 +0200
Message-ID<J2ofD-3xwL-1@gated-at.bofh.it>
In reply to#83136
On Sat, 2024-07-20 at 21:17 +0200, Ben Hutchings wrote:
> I had a go yesterday and ran into the same problem.  I couldn't
> reproduce with a small kernel config (allnoconfig + BPF + DEBUG_INFO +
> DEBUG_INFO_BTF) and there wasn't enough disk space to build even one of
> the Debian kernel flavours.
> 
> > Will take care of it and let you know when it's (some hours).
> 
> Thank you!

There are now 120 GB of free disk space. Let me know if that's sufficient
or whether I need to clean up more, probably asking others to clean up
their home directories.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

[toc] | [prev] | [next] | [standalone]

#83140

FromBen Hutchings <ben@decadent.org.uk>
Date2024-07-22 00:10 +0200
Message-ID<J2MUF-3ME1-1@gated-at.bofh.it>
In reply to#83137

[Multipart message — attachments visible in raw view] — view raw

On Sat, 2024-07-20 at 21:45 +0200, John Paul Adrian Glaubitz wrote:
> On Sat, 2024-07-20 at 21:17 +0200, Ben Hutchings wrote:
> > I had a go yesterday and ran into the same problem.  I couldn't
> > reproduce with a small kernel config (allnoconfig + BPF + DEBUG_INFO +
> > DEBUG_INFO_BTF) and there wasn't enough disk space to build even one of
> > the Debian kernel flavours.
> > 
> > > Will take care of it and let you know when it's (some hours).
> > 
> > Thank you!
> 
> There are now 120 GB of free disk space. Let me know if that's sufficient
> or whether I need to clean up more, probably asking others to clean up
> their home directories.

I've now done 10 kernel builds on perotto (4 builds of just the
"powerpc" flavour and then 2 builds of all 3 flavours) and not
reproduced this.  I'm thinking this may be machine-dependent in some
way.

Looking again at all the build logs since DEBUG_INFO_BTF was enabled
for powerpc, we have:

Successes:

Version        Builder
----------------------------------
6.9.10-1       debian-project-be-2
6.9.7-1        debian-project-be-1
6.8.12-1       debian-project-be-2
6.8.11-1       blaauw
6.8.9-1        debian-project-be-2
6.7.12-1       blaauw
6.7.9-2        debian-project-be-2
6.7.4-1~exp1   blaauw
6.7.1-1~exp1   debian-project-be-2
6.6.15-1       blaauw
6.6.13-1       blaauw
6.6.11-1       debian-project-be-1
6.6.9-1+b1     debian-project-be-1
6.6.8-1        debian-project-be-2
6.6.4-1~exp1   blaauw
6.5.13-1       debian-project-be-1
6.5.10-1       debian-project-be-2
6.5.8-1        blaauw
6.5.6-1        debian-project-be-2
6.5.3-1        blaauw
6.5~rc7-1~exp1 debian-project-be-1
6.5~rc6-1~exp1 blaauw
6.4.13-1       blaauw
6.4.11-1       blaauw
6.4.4-3        blaauw

Failures:

Version        Builder             Failure mode
----------------------------------------------------------------------------
6.10-1~exp1    blaauw              this bug
6.9.9-1        blaauw              this bug
6.9.8-1        kapitsa             this bug
6.9.2-1~exp1   blaauw              this bug
6.8.12-1       kapitsa             this bug
6.7-1~exp1     debian-project-be-2 compiler OOM; not powerpc-specific
6.6.3-1~exp1   blaauw              kernel-wedge failed; not powerpc-specific
6.4.4-2        blaauw              out of disk space

Ignoring the unrelated failures, kapitsa has a 0% success rate (but
with only 2 attempts), blaauw an 80% success rate, and debian-project-
be-{1,2} have 100% success rates.

I don't know what differences there are between these builders that
might be relevant.

Ben.

-- 
Ben Hutchings
One of the nice things about standards is that
there are so many of them.

[toc] | [prev] | [next] | [standalone]

#83142

FromJohn Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Date2024-07-22 07:00 +0200
Message-ID<J2Tjr-3Qqq-5@gated-at.bofh.it>
In reply to#83140
Hi Ben,

On Mon, 2024-07-22 at 00:07 +0200, Ben Hutchings wrote:
> I don't know what differences there are between these builders that
> might be relevant.

For kapitsa, the installed host system is powerpc while all the others
run the ppc64 port.

As for the hardware:

kapitsa runs bare-metal (inside an LPAR) on a POWER8 machine:

root@kapitsa:~# grep model /proc/cpuinfo 
model           : IBM,8284-22A
root@kapitsa:~#

Both blaauw and perotto are KVMs running on watson which runs
Debian's ppc64el port (little-endian):

root@watson:~# grep model /proc/cpuinfo 
model           : 8247-42L
root@watson:~#

root@blaauw:~# grep model /proc/cpuinfo 
model           : IBM pSeries (emulated by qemu)
root@blaauw:~#

root@perotto:~# grep model /proc/cpuinfo 
model           : IBM pSeries (emulated by qemu)
root@perotto:~#

Both debian-project-be-01 and debian-project-be-02 are KVMs running
on OpenStack at OSUOSL's OpenPOWER platform:

root@debian-project-be-1:~# grep model /proc/cpuinfo 
model           : IBM pSeries (emulated by qemu)
root@debian-project-be-1:~#

root@debian-project-be-2:~# grep model /proc/cpuinfo 
model           : IBM pSeries (emulated by qemu)
root@debian-project-be-2:~#

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web