Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #90858 > unrolled thread

Bug#1125405: Fwd: linux-image-6.12.63+deb13-amd64: Cannot mount cephfs

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1125405: Fwd: linux-image-6.12.63+deb13-amd64: Cannot mount cephfs Salvatore Bonaccorso <carnil@debian.org> - 2026-01-17 21:20 +0100
    Bug#1125405: Fwd: linux-image-6.12.63+deb13-amd64: Cannot mount cephfs Salvatore Bonaccorso <carnil@debian.org> - 2026-01-24 14:20 +0100
      Bug#1125405: Fwd: linux-image-6.12.63+deb13-amd64: Cannot mount cephfs Salvatore Bonaccorso <carnil@debian.org> - 2026-01-25 09:30 +0100
        Bug#1125405: Fwd: linux-image-6.12.63+deb13-amd64: Cannot mount cephfs Salvatore Bonaccorso <carnil@debian.org> - 2026-01-25 18:10 +0100

#90858 — Bug#1125405: Fwd: linux-image-6.12.63+deb13-amd64: Cannot mount cephfs

Control: tags -1 + moreinfo

Hi Reinhard,

On Tue, Jan 13, 2026 at 05:40:04PM +0100, Reinhard Eilmsteiner wrote:
> Package: src:linux
> Version: 6.12.63-1
> Severity: important
> 
> Dear Maintainer,
> 
> It used to work with kernel version 6.12.57+deb13-amd64 - after update
> it ceased working.
> 
> When doing
> 
> mount -t ceph -o name=... cn1,cn2,cn3:/directory /remote/mountpoint
> ls /remote/mountpoint results in
> 
> Killed
> 
> instead of
> 
> A normal directory listing
> 
> 
> -- Package-specific info:
> ** Version:
> Linux version 6.12.63+deb13-amd64 (debian-kernel@lists.debian.org)
> (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for
> Debian) 2.44) #1 SMP PREEMPT_DYNAMIC Debian 6.12.63-1 (2025-12-30)
> 
> ** Command line:
> BOOT_IMAGE=/boot/vmlinuz-6.12.63+deb13-amd64
> root=UUID=2f64ca89-8847-4677-bd7b-924c24d36004 ro quiet
> 
> ** Tainted: D (128)
> * kernel died recently, i.e. there was an OOPS or BUG
> 
> ** Kernel log:
> [269737.358350] nfs: server uru.eilm.at not responding, timed out
> [269742.478091] nfs: server uru.eilm.at not responding, timed out
> [269746.062291] nfs: server uru.eilm.at not responding, timed out
> [269751.185972] nfs: server uru.eilm.at not responding, timed out
> [269756.301995] nfs: server uru.eilm.at not responding, timed out
> [269761.422136] nfs: server uru.eilm.at not responding, timed out
> [269766.541884] nfs: server uru.eilm.at not responding, timed out
> [269771.617882] nfs: server uru.eilm.at not responding, timed out
> [269782.157826] nfs: server uru.eilm.at not responding, timed out
> [269792.397717] nfs: server uru.eilm.at not responding, timed out
> [269802.637736] nfs: server uru.eilm.at not responding, timed out
> [269806.221818] nfs: server uru.eilm.at not responding, timed out
> [269811.341919] nfs: server uru.eilm.at not responding, timed out
> [269816.465888] nfs: server uru.eilm.at not responding, timed out
> [269821.581608] nfs: server uru.eilm.at not responding, timed out
> [269836.941722] nfs: server uru.eilm.at not responding, timed out
> [269847.437601] nfs: server uru.eilm.at not responding, timed out
> [269857.677476] nfs: server uru.eilm.at not responding, timed out
> [269866.349444] nfs: server uru.eilm.at not responding, timed out
> [269871.501459] nfs: server uru.eilm.at not responding, timed out
> [269876.621299] nfs: server uru.eilm.at not responding, timed out
> [269886.862133] nfs: server uru.eilm.at not responding, timed out
> [269902.221980] nfs: server uru.eilm.at not responding, timed out
> [269912.749841] nfs: server uru.eilm.at not responding, timed out
> [269922.958017] nfs: server uru.eilm.at not responding, timed out
> [269926.481796] nfs: server uru.eilm.at not responding, timed out
> [269931.629699] nfs: server uru.eilm.at not responding, timed out
> [269936.813886] nfs: server uru.eilm.at not responding, timed out
> [269941.901703] nfs: server uru.eilm.at not responding, timed out
> [269952.141759] nfs: server uru.eilm.at not responding, timed out
> [269963.219324] libceph: mon1 (1)192.168.0.23:6789 session established
> [269963.219593] libceph: client27727483 fsid
> dae8b6ab-51f8-4d68-8539-9998ea8d83d3
> [269963.767779] BUG: kernel NULL pointer dereference, address:
> 0000000000000000
> [269963.767818] #PF: supervisor read access in kernel mode
> [269963.767963] #PF: error_code(0x0000) - not-present page
> [269963.768102] PGD 0 P4D 0 [269963.768279] Oops: Oops: 0000 [#1] PREEMPT
> SMP PTI
> [269963.768439] CPU: 0 UID: 2000 PID: 527382 Comm: ls Not tainted
> 6.12.63+deb13-amd64 #1 Debian 6.12.63-1
> [269963.768482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014
> [269963.768495] RIP: 0010:strcmp+0x1c/0x40
> [269963.771968] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa
> 31 c0 eb 10 0f 1f 84 00 00 00 00 00 48 83 c0 01 84 d2 74 13 0f b6 14 07 <3a>
> 14 06 74 ef 19 c0 83 c8 01 c3 cc cc cc cc 31 c0 c3 cc cc cc cc
> [269963.772140] RSP: 0018:ffffbe6dc8d07a80 EFLAGS: 00010246
> [269963.772158] RAX: 0000000000000000 RBX: ffff9a1eb73834c0 RCX:
> ffffffffb824c704
> [269963.772169] RDX: 0000000000000066 RSI: 0000000000000000 RDI:
> ffff9a1e87b34908
> [269963.772177] RBP: 00000000000007d0 R08: 0000000000000000 R09:
> 0000000000000000
> [269963.772184] R10: ffff9a1e80ac3000 R11: 0000000000000002 R12:
> 0000000000000000
> [269963.772192] R13: ffff9a1d83c72000 R14: ffff9a1ea1f8a000 R15:
> ffff9a1e8610b800
> [269963.772200] FS: 00007fc109279240(0000) GS:ffff9a1eb8c00000(0000)
> knlGS:0000000000000000
> [269963.772208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [269963.772215] CR2: 0000000000000000 CR3: 0000000051722000 CR4:
> 00000000000006f0
> [269963.772226] Call Trace:
> [269963.772425] <TASK>
> [269963.772449] ceph_mds_check_access+0x10c/0x790 [ceph]
> [269963.772496] ceph_open+0x171/0x5c0 [ceph]
> [269963.772654] ? __pfx_ceph_open+0x10/0x10 [ceph]
> [269963.772781] do_dentry_open+0x14f/0x440
> [269963.774839] vfs_open+0x2e/0xe0
> [269963.775250] path_openat+0x82e/0x12d0
> [269963.775849] ? __pte_offset_map+0x1b/0x180
> [269963.776581] do_filp_open+0xc4/0x170
> [269963.776692] do_sys_openat2+0xae/0xe0
> [269963.776798] __x64_sys_openat+0x55/0xa0
> [269963.776907] do_syscall_64+0x82/0x190
> [269963.777464] ? __count_memcg_events+0x53/0xf0
> [269963.777883] ? count_memcg_events.constprop.0+0x1a/0x30
> [269963.777993] ? handle_mm_fault+0x1bb/0x2c0
> [269963.778102] ? do_user_addr_fault+0x36c/0x620
> [269963.778711] ? arch_exit_to_user_mode_prepare.isra.0+0x16/0xa0
> [269963.779073] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [269963.779375] RIP: 0033:0x7fc10942db7c
> [269963.779509] Code: 4c 89 54 24 18 41 89 f2 41 83 e2 40 75 40 89 f0 f7 d0
> a9 00 00 41 00 74 35 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48>
> 3d 00 f0 ff ff 77 44 48 8b 54 24 18 64 48 2b 14 25 28 00 00 00
> [269963.779737] RSP: 002b:00007ffc5d911370 EFLAGS: 00000206 ORIG_RAX:
> 0000000000000101
> [269963.779846] RAX: ffffffffffffffda RBX: 0000000000000001 RCX:
> 00007fc10942db7c
> [269963.779956] RDX: 0000000000090800 RSI: 000055607001bc30 RDI:
> 00000000ffffff9c
> [269963.780066] RBP: 0000000000000000 R08: 0000000000000020 R09:
> 0000000000000001
> [269963.780172] R10: 0000000000000000 R11: 0000000000000206 R12:
> 00007fc1092790e0
> [269963.780281] R13: 00007ffc5d9117b0 R14: 000055607001bc30 R15:
> 00005560463e4700
> [269963.780584] </TASK>
> [269963.780796] Modules linked in: xt_REDIRECT dm_mod rpcsec_gss_krb5 nfsv4
> dns_resolver nfs ceph libceph netfs ip_vs_rr xt_ipvs ip_vs vxlan xt_policy
> xt_mark xt_bpf nf_conntrack_netlink xt_nat xt_tcpudp veth br_netfilter
> xt_conntrack xt_MASQUERADE bridge stp llc xfrm_user xfrm_algo xt_set ip_set
> nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype
> nft_compat nf_tables libcrc32c wireguard libchacha20poly1305 chacha_x86_64
> poly1305_x86_64 curve25519_x86_64 libcurve25519_generic libchacha
> ip6_udp_tunnel udp_tunnel overlay binfmt_misc sha512_ssse3 sha256_ssse3
> sha1_ssse3 aesni_intel gf128mul crypto_simd cryptd virtio_balloon pcspkr
> button joydev evdev sg nfsd nfs_acl lockd auth_rpcgss grace sunrpc
> efi_pstore configfs nfnetlink vsock_loopback
> vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci
> qemu_fw_cfg ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2
> crc32c_generic hid_generic usbhid hid bochs sr_mod cdrom drm_vram_helper
> ata_generic sd_mod drm_ttm_helper ttm uhci_hcd ata_piix
> [269963.780850] libata drm_kms_helper xhci_pci virtio_scsi ehci_pci xhci_hcd
> ehci_hcd virtio_net scsi_mod psmouse net_failover usbcore drm crc32c_intel
> i2c_piix4 failover serio_raw i2c_smbus scsi_common usb_common floppy
> [269963.783100] CR2: 0000000000000000
> [269963.783384] ---[ end trace 0000000000000000 ]---
> [269963.783778] RIP: 0010:strcmp+0x1c/0x40
> [269963.784087] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa
> 31 c0 eb 10 0f 1f 84 00 00 00 00 00 48 83 c0 01 84 d2 74 13 0f b6 14 07 <3a>
> 14 06 74 ef 19 c0 83 c8 01 c3 cc cc cc cc 31 c0 c3 cc cc cc cc
> [269963.784651] RSP: 0018:ffffbe6dc8d07a80 EFLAGS: 00010246
> [269963.785013] RAX: 0000000000000000 RBX: ffff9a1eb73834c0 RCX:
> ffffffffb824c704
> [269963.785427] RDX: 0000000000000066 RSI: 0000000000000000 RDI:
> ffff9a1e87b34908
> [269963.785882] RBP: 00000000000007d0 R08: 0000000000000000 R09:
> 0000000000000000
> [269963.786144] R10: ffff9a1e80ac3000 R11: 0000000000000002 R12:
> 0000000000000000
> [269963.786477] R13: ffff9a1d83c72000 R14: ffff9a1ea1f8a000 R15:
> ffff9a1e8610b800
> [269963.786834] FS: 00007fc109279240(0000) GS:ffff9a1eb8c00000(0000)
> knlGS:0000000000000000
> [269963.787275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [269963.787737] CR2: 0000000000000000 CR3: 0000000051722000 CR4:
> 00000000000006f0
> [269967.501681] nfs: server uru.eilm.at not responding, timed out
> [269977.997774] nfs: server uru.eilm.at not responding, timed out
> [269986.637442] nfs: server uru.eilm.at not responding, timed out
> [269996.909497] nfs: server uru.eilm.at not responding, timed out
> [270002.061444] nfs: server uru.eilm.at not responding, timed out
> [270007.181581] nfs: server uru.eilm.at not responding, timed out
> [270017.421335] nfs: server uru.eilm.at not responding, timed out

As I understand this is now a regression from 6.12.57-1 to 6.12.63-1.

Would it be possible that you bisect the issue to narrow down the
change which beaks? That would involve compiling and testing a few
kernels:

    git clone --single-branch -b linux-6.12.y https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
    cd linux-stable
    git checkout v6.12.57
    cp /boot/config-$(uname -r) .config
    yes '' | make localmodconfig
    make savedefconfig
    mv defconfig arch/x86/configs/my_defconfig

    # test 6.12.57 to ensure this is "good"
    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install the resulting .deb package and confirm problem does
    not exist.

    # test 6.12.63 to ensure this is "bad"
    git checkout v6.12.63
    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install the resulting .deb package and confirm poblem exists.

With that confirmed, the bisection can start:

    git bisect start
    git bisect good v6.12.57
    git bisect bad v6.12.63

In each bisection step git checks out a state between the oldest
known-bad and the newest known-good commit. In each step test using:

    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install, and verify if problem exists.

and if the problem is hit run:

    git bisect bad

and if the problem doesn't trigger run:

    git bisect good

. Please pay attention to always select the just built kernel for
booting, it won't always be the default kernel picked up by grub.

Iterate until git announces to have identified the first bad commit.

Then provide the output of

    git bisect log

In the course of the bisection you might have to uninstall previous
kernels again to not exhaust the disk space in /boot. Also in the end
uninstall all self-built kernels again.

Regards,
Salvatore

[toc] | [next] | [standalone]

#90942

Hi Reinhard,

On Wed, Jan 21, 2026 at 09:39:51AM +0100, Reinhard Eilmsteiner wrote:
> Hey Salvatore,
> 
> I managed to build a .66 and a .57 kernel, however when I try mounting the
> cephfs with this kernel, it complains that the required module is not
> available for this kernel. Unfortunately, I don't know where to get the
> sources for the cephfs kernel modules in the correct version to match the
> kernels I want to build.
> 
> Could you please give me "vectors" (directions)?

Before running the 'make savedefconfig' step, make sure to have the
ceph required configuration enabled, either by writing the config
options directly in the .config file or by running;

./scripts/config --module CEPH_FS
./scripts/config --module CEPH_LIB
./scripts/config --enable CEPH_FSCACHE
./scripts/config --enable CEPH_FS_POSIX_ACL

then the remaining

make savedefconfig
mv defconfig arch/x86/configs/my_defconfig

(make sure that they are configured before compiling the respective
ersions).

The rest remains the same in the procedure, but you need to make sure
the ceph support is enabled in the config you use.

Hope this help,

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#90961

Ciao Reinhard,

On Sat, Jan 24, 2026 at 08:05:00PM +0100, Reinhard Eilmteiner wrote:
> Ciao Salvatore,
> 
> git bisect start
> # status: waiting for both good and bad commits
> # good: [8a243ecde1f6447b8e237f2c1c67c0bb67d16d67] Linux 6.12.57
> git bisect good 8a243ecde1f6447b8e237f2c1c67c0bb67d16d67
> # status: waiting for bad commit, 1 good commit known
> # bad: [567bd8cbc2fe6b28b78864cbbbc41b0d405eb83c] Linux 6.12.63
> git bisect bad 567bd8cbc2fe6b28b78864cbbbc41b0d405eb83c
> # bad: [7475d784169c7df48b0c55525fb862e06674d63c] Linux 6.12.58
> git bisect bad 7475d784169c7df48b0c55525fb862e06674d63c
> # bad: [318a47068f7b88de838518897500d7509e3fe205] Linux 6.12.60
> git bisect bad 318a47068f7b88de838518897500d7509e3fe205
> 
> There you go.

Something went wrong in the above, because if you start the bisection
you should at the first commit to test land at:

$ git bisect good 8a243ecde1f6447b8e237f2c1c67c0bb67d16d67
status: waiting for bad commit, 1 good commit known
$ git bisect bad 567bd8cbc2fe6b28b78864cbbbc41b0d405eb83c
Bisecting: 698 revisions left to test after this (roughly 10 steps)
[850c7f0537cc5a37ed012592907920637cc548b6] x86/microcode/AMD: Add Zen5 model 0x44, stepping 0x1 minrev

So the next commit to test with the procedure was
850c7f0537cc5a37ed012592907920637cc548b6 and not as in the above log.

Can you reset the bisection and start freshly with the procedure?

Thanks for your time helping to identify the issue!

FWIW, I have so far not seen other ceph related regressions reports
upstream for 6.12.y, so this is puzzling me a bit.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#90964

Hi Reinhard,

On Sun, Jan 25, 2026 at 05:26:22PM +0100, Salvatore Bonaccorso wrote:
> Control: tags -1 + upstream 
> Control: forwarded -1 https://tracker.ceph.com/issues/73886 https://lore.kernel.org/ceph-devel/20260114195524.1025067-2-slava@dubeyko.com/
> 
> Hi Reinhard,
> 
> On Sun, Jan 25, 2026 at 11:22:52AM +0100, Reinhard Eilmsteiner wrote:
> > OK, I just restarted
> > 
> > FYI, I have create this script to build kernels:
> > #!/bin/bash
> > 
> > cd /home/eilmrein/code/linux-stable
> > git checkout $1
> > cp /boot/config-$(uname -r) .config
> > yes '' | make localmodconfig
> > ./scripts/config --module CEPH_FS
> > ./scripts/config --module CEPH_LIB
> > ./scripts/config --enable CEPH_FSCACHE
> > ./scripts/config --enable CEPH_FS_POSIX_ACL
> > make savedefconfig
> > mv defconfig arch/x86/configs/my_defconfig
> > make my_defconfig
> > make -j $(nproc) bindeb-pkg
> > 
> > After having run it with v6.12.58 as argument, I installed the resulting
> > kernel using
> > sudo dpkg -i linux-image-6.12.58_6.12.58-16_amd64.deb
> > 
> > Booted into it and did a quick ls on one of the cephfs, which I have systemd
> > automount for me - however it also fails when I try to mount it manually , I
> > get
> > natchez➜ codels /remote/vdl/files01
> > [1]    2647 killed     ls --color=tty /remote/vdl/files01
> > 
> > , did the requested git bisect bad
> > 
> > and as it is the immediate successor, I now ran this right after making the
> > current kernel as bad.
> > 
> > git bisect start
> > # status: waiting for both good and bad commits
> > # good: [8a243ecde1f6447b8e237f2c1c67c0bb67d16d67] Linux 6.12.57
> > git bisect good 8a243ecde1f6447b8e237f2c1c67c0bb67d16d67
> > # status: waiting for bad commit, 1 good commit known
> > # bad: [567bd8cbc2fe6b28b78864cbbbc41b0d405eb83c] Linux 6.12.63
> > git bisect bad 567bd8cbc2fe6b28b78864cbbbc41b0d405eb83c
> > # bad: [7475d784169c7df48b0c55525fb862e06674d63c] Linux 6.12.58
> > git bisect bad 7475d784169c7df48b0c55525fb862e06674d63c
> 
> This is still not the bisect procedure :).
> 
> The next ommit pointed out by the procedure to bisec would be
> 850c7f0537cc5a37ed012592907920637cc548b6 . You need to build with that
> commit up a kernel, boot into it, test if ceph mount works or not and
> based on the result mark it with 'git bisect bad' or 'git bisect
> good'. Git will then calculate you the next commit to test, and you
> interate until you get 'the first bad commit' result. We need to have
> that (or worst case a range, if you needed to skip one).
> 
> But in meahwhile I have found an upstream repot which matches your
> report:
> 
> https://tracker.ceph.com/issues/73886
> https://lore.kernel.org/ceph-devel/20260114195524.1025067-2-slava@dubeyko.com/
> 
> So it would be great if you instead could built the current kernel in
> trixie with this patch added on top.
> 
> The procedure for this one is documented in
> https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#id-1.6.6.4
> but let me know if you need help with that.

Let's shortcut that step, I have built *test* packages at
https://people.debian.org/~carnil/tmp/linux/1125405-cephfs/ (if you
want to trust them, unsigned, but I have put a sha256sum file for
those signed with my gpg key in the Debian keyring).

Please report back if that fixes the problem.

Regards,
Salvatore

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web