Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #84281 > unrolled thread

Bug#1085178: linux-signed-amd64: Some BPF fentry hooks silently fail

Back to article view | Back to linux.debian.kernel

Contents

  Bug#1085178: linux-signed-amd64: Some BPF fentry hooks silently fail Martin Kelly <martin.kelly@crowdstrike.com> - 2024-10-15 21:50 +0200
    Bug#1085178: linux-signed-amd64: Some BPF fentry hooks silently fail Martin Kelly <martin.kelly@crowdstrike.com> - 2024-12-07 00:40 +0100
      Bug#1085178: linux-signed-amd64: Some BPF fentry hooks silently fail Salvatore Bonaccorso <carnil@debian.org> - 2025-02-21 21:10 +0100
    Processed: Re: Bug#1085178: linux-signed-amd64: Some BPF fentry  hooks silently fail "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-02-21 21:10 +0100
    Bug#1085178: Re: Bug#1085178: linux-signed-amd64: Some BPF fentry hooks silently fail Salvatore Bonaccorso <carnil@debian.org> - 2025-04-21 14:20 +0200
    Processed: Re: Bug#1085178: Re: Bug#1085178: linux-signed-amd64:  Some BPF fentry hooks silently fail "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-04-21 14:20 +0200
    Bug#1085178: closing 1085178 Salvatore Bonaccorso <carnil@debian.org> - 2025-10-22 08:00 +0200
      Processed: closing 1085178 "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-10-22 08:00 +0200

#84281 — Bug#1085178: linux-signed-amd64: Some BPF fentry hooks silently fail

Source: linux-signed-amd64
X-Debbugs-Cc: martin.kelly@crowdstrike.com
Version: 6.11.2+1
Severity: important
Tags: upstream

Some BPF fentry hooks in the 6.11.2-cloud-amd64 kernel successfully
load
but then are ignored when they should fire. This can be seen with the
following bpftrace command:

bpftrace -e 'kfunc:acct_process { printf("acct_process called\n"); }'

Which normally should trigger when running some short-lived process,
but
instead doesn't show output on the 6.11.2-cloud-amd64 kernel. The same
command works fine as a kprobe, showing this issue is unique to fentry.
So the below command shows output as expected:

bpftrace -e 'kprobe:acct_process { printf("acct_process called\n"); }'

Further, other related hooks in the same code path work fine, such as
this one:

bpftrace -e 'kfunc:acct_collect { printf("acct_collect called\n"); }'

Which shows that the issue has to do with only *some* fentry hooks.

This issue exists in upstream 6.11.2 as well and appears to be fixed in
upstream master. My git bisect pointed at commit 98f7e32f20d2 ("mm/x86:
implement arch_check_zapped_pud()"). I'm not sure why that commit fixes
it, but I manually cherry-picked that commit on top of 6.11.2 and it
indeed fixed the issue.

A few other things to note:
- This issue exists in 6.11.2-cloud-amd64 kernel but not 6.11.2-amd64,
  so it presumably depends on some collection of CONFIG options.
- There is an issue with identical symptoms on some kernels with
  CONFIG_X86_X32_ABI=y set (6.11.2-cloud-amd64 does *not* have x32 ABI
  set). I reported this upstream [1] but haven't gotten a response.
- This x32 ABI issue reproduces on Debian's 6.10.9-amd64 kernel and
  on upstream master but not the
  6.10.11-amd64 or 6.11.2-amd64 kernels, so it appears to be coming and
  going semi-randomly. Technically there is currently no latest Debian
  kernel hitting the x32 ABI issue, but since the issue is coming and
  going, it may come back in a later release.

[1]
https://lore.kernel.org/bpf/7136605d24de9b1fc62d02a355ef11c950a94153.camel@crowdstrike.com/T/#u

-- System Information:
Debian Release: 12.7
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500,
'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.11.2-cloud-amd64 (SMP w/20 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

[toc] | [next] | [standalone]

#84745

On Tue, 15 Oct 2024 18:55:11 +0000 Martin Kelly
<martin.kelly@crowdstrike.com> wrote:
> Source: linux-signed-amd64
> X-Debbugs-Cc: martin.kelly@crowdstrike.com
> Version: 6.11.2+1
> Severity: important
> Tags: upstream
> 
> Some BPF fentry hooks in the 6.11.2-cloud-amd64 kernel successfully
> load
> but then are ignored when they should fire. This can be seen with the
> following bpftrace command:
> 
> bpftrace -e 'kfunc:acct_process { printf("acct_process called\n"); }'
> 

This was root-caused to be the same issue as the one addressed in this
(unmerged) patch series:
https://lore.kernel.org/lkml/20240723063258.2240610-1-zhengyejian@huaweicloud.com/

Essentially this has to do with ftrace and weak functions.

[toc] | [prev] | [next] | [standalone]

#86034

Control: found -1 6.12.15-1

On Fri, Dec 06, 2024 at 11:10:33PM +0000, Martin Kelly wrote:
> On Tue, 15 Oct 2024 18:55:11 +0000 Martin Kelly
> <martin.kelly@crowdstrike.com> wrote:
> > Source: linux-signed-amd64
> > X-Debbugs-Cc: martin.kelly@crowdstrike.com
> > Version: 6.11.2+1
> > Severity: important
> > Tags: upstream
> > 
> > Some BPF fentry hooks in the 6.11.2-cloud-amd64 kernel successfully
> > load
> > but then are ignored when they should fire. This can be seen with the
> > following bpftrace command:
> > 
> > bpftrace -e 'kfunc:acct_process { printf("acct_process called\n"); }'
> > 
> 
> This was root-caused to be the same issue as the one addressed in this
> (unmerged) patch series:
> https://lore.kernel.org/lkml/20240723063258.2240610-1-zhengyejian@huaweicloud.com/
> 
> Essentially this has to do with ftrace and weak functions.

As I understand this is still an issue in 6.12.15-1.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#86036 — Processed: Re: Bug#1085178: linux-signed-amd64: Some BPF fentry hooks silently fail

Processing control commands:

> found -1 6.12.15-1
Bug #1085178 [src:linux] linux-signed-amd64: Some BPF fentry hooks silently fail
Marked as found in versions linux/6.12.15-1.

-- 
1085178: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085178
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#86937 — Bug#1085178: Re: Bug#1085178: linux-signed-amd64: Some BPF fentry hooks silently fail

Control: tags -1 + fixed-upstream

On Fri, Feb 21, 2025 at 09:43:34PM +0000, Martin Kelly wrote:
> On Fri, 2025-02-21 at 21:03 +0100, Salvatore Bonaccorso wrote:
> > 
> > > 
> > > Essentially this has to do with ftrace and weak functions.
> > 
> > As I understand this is still an issue in 6.12.15-1.
> > 
> 
> That's correct; 6.12.15 should still be impacted, as long as
> CONFIG_X86_KERNEL_IBT is set. CONFIG_X86_KERNEL_IBT changes the
> function prologue in a way that causes this issue when combined with an
> certain fentry bug.
> 
> I confirmed this issue is fixed with this patch series:
> https://lore.kernel.org/bpf/20250218195918.255228630@goodmis.org/
> Since that fixes the underlying fentry bug.
> 
> I would guess that series will merge into 6.15, but we'll have to see.

AFAICS, this has been merged for 6.15-rc1 with
https://git.kernel.org/linus/dd161f74f8198c62f9bcf893f72c64bbb0d68b25

Not sure they will be backported down to 6.12.y.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#86938 — Processed: Re: Bug#1085178: Re: Bug#1085178: linux-signed-amd64: Some BPF fentry hooks silently fail

Processing control commands:

> tags -1 + fixed-upstream
Bug #1085178 [src:linux] linux-signed-amd64: Some BPF fentry hooks silently fail
Added tag(s) fixed-upstream.

-- 
1085178: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085178
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#89711 — Bug#1085178: closing 1085178

close 1085178 6.15~rc7-1~exp1
thanks

[toc] | [prev] | [next] | [standalone]

#89712 — Processed: closing 1085178

Processing commands for control@bugs.debian.org:

> close 1085178 6.15~rc7-1~exp1
Bug #1085178 [src:linux] linux-signed-amd64: Some BPF fentry hooks silently fail
Marked as fixed in versions linux/6.15~rc7-1~exp1.
Bug #1085178 [src:linux] linux-signed-amd64: Some BPF fentry hooks silently fail
Marked Bug as done
> thanks
Stopping processing here.

Please contact me if you need assistance.
-- 
1085178: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085178
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web