Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #83639 > unrolled thread

Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption

Back to article view | Back to linux.debian.kernel

Contents

  Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption Forest <forestix@nom.one> - 2024-08-23 05:20 +0200
    Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption Forest <forestix@nom.one> - 2024-08-23 05:50 +0200
    Processed: Re: Bug#1079394: linux-image-6.10.6-amd64: causes cifs  regression, flatpak & ostree signature corruption "Debian Bug Tracking System" <owner@bugs.debian.org> - 2024-08-23 05:50 +0200
    Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption Salvatore Bonaccorso <carnil@debian.org> - 2024-08-23 22:20 +0200
      Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption Salvatore Bonaccorso <carnil@debian.org> - 2024-08-24 09:50 +0200
      Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption Forest <forestix@nom.one> - 2024-08-24 10:20 +0200
      Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption Forest <forestix@nom.one> - 2024-08-26 10:40 +0200
    Processed: Re: Bug#1079394: linux-image-6.10.6-amd64: causes cifs  regression, flatpak & ostree signature corruption "Debian Bug Tracking System" <owner@bugs.debian.org> - 2024-08-23 22:20 +0200
    Bug#1079394: marked as done (linux-image-6.10.6-amd64: causes  cifs regression, flatpak & ostree signature corruption) "Debian Bug Tracking System" <owner@bugs.debian.org> - 2024-09-09 15:20 +0200

#83639 — Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption

Package: src:linux
Version: 6.10.6-1
Severity: important
X-Debbugs-Cc: forestix@nom.one

Dear Maintainer,

After upgrading from kernel 6.9.12 to 6.10.4, flatpak and ostree are now
writing corrupt gpg signatures when exporting signed packages or signing
their repository metadata/summary files, when the repository is on a cifs
mount. Instead of writing signature data, null bytes are written in its
place.

No error is reported by the application or the kernel when it happens.
The problem isn't revealed until something tries to use the repository,
and finds signatures full of null bytes. Of course, this completely
breaks affected flatpak repositories.

A kernel bisect reveals this:
3ee1a1fc39819906f04d6c62c180e760cd3a689d is the first bad commit
commit 3ee1a1fc39819906f04d6c62c180e760cd3a689d
Author: David Howells <dhowells@redhat.com>
Date:   Fri Oct 6 18:29:59 2023 +0100
    cifs: Cut over to using netfslib

I was unable to determine whether the problem is fixed in kernel
6.11.0-rc4, due to even worse cifs problems in that version.

An strace of flatpak (which uses libostree) hints that the problem might
be triggered by the following sequence of events:

- create a temp file
- write signature data to the temp file
- memory map the temp file
- close the temp file
- unlink the temp file
- read the previously written signature data from the memory mapping

My investigation so far can be found in these bug reports:
https://github.com/flatpak/flatpak/issues/5911
https://github.com/ostreedev/ostree/issues/3288

I am not familiar with those projects' code, so the triggering sequence
of events is merely a hypothesis for now.

However, I can consistently reproduce the problem by passing a path
located on a cifs mount (along with a gpg key ID) to this script:


#!/bin/sh
set -e

if [ "$#" -lt 2 ] || [ "$1" = "-h" ] ; then
    echo "usage: $(basename "$0") <repo-dir> <gpg-key-id>"
    exit 2
fi

repo=$1
keyid=$2
src="./foo"

echo "creating ostree repo at $repo"
ostree init --repo="$repo"

echo "creating test tree at $src"
mkdir -p "$src"
echo hi > "$src"/hello

ostree config --repo="$repo" set core.min-free-space-percent 1
ostree commit --repo="$repo" --branch=foo --gpg-sign="$keyid" "$src"

if ostree show --repo="$repo" foo; then
    echo ---
    echo success!
else
    echo ---
    ostree show --repo="$repo" --print-detached-metadata-key=ostree.gpgsigs foo
    echo failure!
    echo look for null bytes in the above commit signature
fi



-- Package-specific info:
** Version:
Linux version 6.10.6-amd64 (debian-kernel@lists.debian.org) (x86_64-linux-gnu-gcc-13 (Debian 13.3.0-5) 13.3.0, GNU ld (GNU Binutils for Debian) 2.43.1) #1 SMP PREEMPT_DYNAMIC Debian 6.10.6-1 (2024-08-19)

** Command line:
BOOT_IMAGE=/boot/vmlinuz-6.10.6-amd64 root=UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ro net.ifnames=0 quiet cryptdevice=UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX:xxxxxx root=/dev/mapper/xxxxxx splash

** Not tainted

** Kernel log:
Unable to read kernel log; any relevant messages should be attached

** Loaded modules:
tun
nls_utf8
cifs
cifs_arc4
nls_ucs2_utils
cifs_md4
dns_resolver
netfs
nft_masq
nft_chain_nat
nf_nat
nf_conntrack
nf_defrag_ipv6
nf_defrag_ipv4
bridge
stp
llc
nf_tables
nvme_fabrics
nfnetlink
rfcomm
essiv
authenc
crypto_null
snd_seq_dummy
snd_hrtimer
snd_seq
snd_seq_device
qrtr
cmac
algif_hash
algif_skcipher
af_alg
zstd
bnep
zram
binfmt_misc
nls_ascii
nls_cp437
vfat
fat
mt7921e
mt7921_common
mt792x_lib
mt76_connac_lib
snd_hda_codec_realtek
mt76
snd_hda_codec_generic
snd_hda_scodec_component
snd_hda_codec_hdmi
snd_hda_intel
amd_atl
intel_rapl_msr
mac80211
intel_rapl_common
snd_intel_dspcfg
amd64_edac
snd_intel_sdw_acpi
edac_mce_amd
btusb
snd_hda_codec
btrtl
btintel
kvm_amd
btbcm
snd_hda_core
btmtk
eeepc_wmi
asus_nb_wmi
libarc4
bluetooth
asus_wmi
snd_hwdep
kvm
cfg80211
snd_pcm
battery
sparse_keymap
platform_profile
snd_timer
wmi_bmof
sp5100_tco
rapl
snd
k10temp
watchdog
ccp
pcspkr
rfkill
soundcore
joydev
sg
evdev
nct6775
nct6775_core
hwmon_vid
msr
parport_pc
ppdev
lp
parport
loop
efi_pstore
configfs
ip_tables
x_tables
autofs4
ext4
crc16
mbcache
jbd2
btrfs
dm_crypt
dm_mod
efivarfs
raid10
raid456
async_raid6_recov
async_memcpy
async_pq
async_xor
async_tx
xor
raid6_pq
libcrc32c
crc32c_generic
raid1
raid0
md_mod
hid_generic
amdgpu
amdxcp
drm_exec
gpu_sched
drm_buddy
i2c_algo_bit
drm_suballoc_helper
drm_display_helper
cec
rc_core
usbhid
hid
drm_ttm_helper
sd_mod
ttm
ahci
nvme
drm_kms_helper
crc32_pclmul
libahci
xhci_pci
crc32c_intel
nvme_core
libata
xhci_hcd
r8169
ghash_clmulni_intel
t10_pi
realtek
drm
mdio_devres
crc64_rocksoft_generic
sha512_ssse3
usbcore
scsi_mod
crc64_rocksoft
sha256_ssse3
libphy
crc_t10dif
sha1_ssse3
crct10dif_generic
video
crct10dif_pclmul
scsi_common
usb_common
crc64
i2c_piix4
crct10dif_common
wmi
gpio_amdpt
gpio_generic
button
aesni_intel
crypto_simd
cryptd


-- System Information:
Debian Release: 12.6
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable'), (1, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.10.6-amd64 (SMP w/32 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages linux-image-6.10.6-amd64 depends on:
ii  initramfs-tools [linux-initramfs-tool]  0.142
ii  kmod                                    30+20221128-1
ii  linux-base                              4.9

Versions of packages linux-image-6.10.6-amd64 recommends:
ii  apparmor  3.0.8-3

Versions of packages linux-image-6.10.6-amd64 suggests:
pn  debian-kernel-handbook  <none>
ii  firmware-linux-free     20200122-1
ii  grub-efi-amd64          2.06-13+deb12u1
pn  linux-doc-6.10          <none>

Versions of packages linux-image-6.10.6-amd64 is related to:
ii  firmware-amd-graphics     20240220-1~forestix3
ii  firmware-atheros          20230210-5
ii  firmware-bnx2             20230210-5
ii  firmware-bnx2x            20230210-5
ii  firmware-brcm80211        20230210-5
ii  firmware-cavium           20230210-5
ii  firmware-intel-sound      20230210-5
pn  firmware-intelwimax       <none>
ii  firmware-ipw2x00          20230210-5
ii  firmware-ivtv             20230210-5
ii  firmware-iwlwifi          20230210-5
ii  firmware-libertas         20230210-5
ii  firmware-linux-nonfree    20240220-1~forestix3
ii  firmware-misc-nonfree     20240220-1~forestix3
ii  firmware-myricom          20230210-5
ii  firmware-netxen           20230210-5
ii  firmware-qlogic           20230210-5
ii  firmware-realtek          20230210-5
ii  firmware-samsung          20230210-5
ii  firmware-siano            20230210-5
ii  firmware-ti-connectivity  20230210-5
pn  xen-hypervisor            <none>

-- no debconf information

[toc] | [next] | [standalone]

#83642

Control: found -1 6.10.4-1
Control: found -1 6.10.3-1

[toc] | [prev] | [next] | [standalone]

#83643 — Processed: Re: Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption

Processing control commands:

> found -1 6.10.4-1
Bug #1079394 [src:linux] linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption
Marked as found in versions linux/6.10.4-1.
> found -1 6.10.3-1
Bug #1079394 [src:linux] linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption
Marked as found in versions linux/6.10.3-1.

-- 
1079394: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1079394
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#83652

Control: tags -1 + moreinfo upstream

hi,

On Thu, Aug 22, 2024 at 08:09:06PM -0700, Forest wrote:
> Package: src:linux
> Version: 6.10.6-1
> Severity: important
> X-Debbugs-Cc: forestix@nom.one
> 
> Dear Maintainer,
> 
> After upgrading from kernel 6.9.12 to 6.10.4, flatpak and ostree are now
> writing corrupt gpg signatures when exporting signed packages or signing
> their repository metadata/summary files, when the repository is on a cifs
> mount. Instead of writing signature data, null bytes are written in its
> place.
> 
> No error is reported by the application or the kernel when it happens.
> The problem isn't revealed until something tries to use the repository,
> and finds signatures full of null bytes. Of course, this completely
> breaks affected flatpak repositories.
> 
> A kernel bisect reveals this:
> 3ee1a1fc39819906f04d6c62c180e760cd3a689d is the first bad commit
> commit 3ee1a1fc39819906f04d6c62c180e760cd3a689d
> Author: David Howells <dhowells@redhat.com>
> Date:   Fri Oct 6 18:29:59 2023 +0100
>     cifs: Cut over to using netfslib
> 
> I was unable to determine whether the problem is fixed in kernel
> 6.11.0-rc4, due to even worse cifs problems in that version.
> 
> An strace of flatpak (which uses libostree) hints that the problem might
> be triggered by the following sequence of events:
> 
> - create a temp file
> - write signature data to the temp file
> - memory map the temp file
> - close the temp file
> - unlink the temp file
> - read the previously written signature data from the memory mapping
> 
> My investigation so far can be found in these bug reports:
> https://github.com/flatpak/flatpak/issues/5911
> https://github.com/ostreedev/ostree/issues/3288
> 
> I am not familiar with those projects' code, so the triggering sequence
> of events is merely a hypothesis for now.
> 
> However, I can consistently reproduce the problem by passing a path
> located on a cifs mount (along with a gpg key ID) to this script:
> 
> 
> #!/bin/sh
> set -e
> 
> if [ "$#" -lt 2 ] || [ "$1" = "-h" ] ; then
>     echo "usage: $(basename "$0") <repo-dir> <gpg-key-id>"
>     exit 2
> fi
> 
> repo=$1
> keyid=$2
> src="./foo"
> 
> echo "creating ostree repo at $repo"
> ostree init --repo="$repo"
> 
> echo "creating test tree at $src"
> mkdir -p "$src"
> echo hi > "$src"/hello
> 
> ostree config --repo="$repo" set core.min-free-space-percent 1
> ostree commit --repo="$repo" --branch=foo --gpg-sign="$keyid" "$src"
> 
> if ostree show --repo="$repo" foo; then
>     echo ---
>     echo success!
> else
>     echo ---
>     ostree show --repo="$repo" --print-detached-metadata-key=ostree.gpgsigs foo
>     echo failure!
>     echo look for null bytes in the above commit signature
> fi

Since you identified this as upstream regression including the
breaking commit, can you please report this upstream (and please do
keep us in the loop).

Since the breaking commit is 3ee1a1fc3981 ("cifs: Cut over to using
netfslib") so you can report it to:

David Howells <dhowells@redhat.com>
Steve French <sfrench@samba.org>
Shyam Prasad N <nspmangalore@gmail.com>
Rohith Surabattula <rohiths.msft@gmail.com>
Jeff Layton <jlayton@kernel.org>
linux-cifs@vger.kernel.org
netfs@lists.linux.dev
linux-fsdevel@vger.kernel.org
linux-mm@kvack.org

make sure to include stable@vger.kernel.org and ideally the regression
mailing list: regressions@lists.linux.dev

It does make little sense to have Debian relaying between.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#83659

Hi,

On Fri, Aug 23, 2024 at 10:13:07PM +0200, Salvatore Bonaccorso wrote:
> Control: tags -1 + moreinfo upstream
> 
> hi,
> 
> On Thu, Aug 22, 2024 at 08:09:06PM -0700, Forest wrote:
> > Package: src:linux
> > Version: 6.10.6-1
> > Severity: important
> > X-Debbugs-Cc: forestix@nom.one
> > 
> > Dear Maintainer,
> > 
> > After upgrading from kernel 6.9.12 to 6.10.4, flatpak and ostree are now
> > writing corrupt gpg signatures when exporting signed packages or signing
> > their repository metadata/summary files, when the repository is on a cifs
> > mount. Instead of writing signature data, null bytes are written in its
> > place.
> > 
> > No error is reported by the application or the kernel when it happens.
> > The problem isn't revealed until something tries to use the repository,
> > and finds signatures full of null bytes. Of course, this completely
> > breaks affected flatpak repositories.
> > 
> > A kernel bisect reveals this:
> > 3ee1a1fc39819906f04d6c62c180e760cd3a689d is the first bad commit
> > commit 3ee1a1fc39819906f04d6c62c180e760cd3a689d
> > Author: David Howells <dhowells@redhat.com>
> > Date:   Fri Oct 6 18:29:59 2023 +0100
> >     cifs: Cut over to using netfslib
> > 
> > I was unable to determine whether the problem is fixed in kernel
> > 6.11.0-rc4, due to even worse cifs problems in that version.
> > 
> > An strace of flatpak (which uses libostree) hints that the problem might
> > be triggered by the following sequence of events:
> > 
> > - create a temp file
> > - write signature data to the temp file
> > - memory map the temp file
> > - close the temp file
> > - unlink the temp file
> > - read the previously written signature data from the memory mapping
> > 
> > My investigation so far can be found in these bug reports:
> > https://github.com/flatpak/flatpak/issues/5911
> > https://github.com/ostreedev/ostree/issues/3288
> > 
> > I am not familiar with those projects' code, so the triggering sequence
> > of events is merely a hypothesis for now.
> > 
> > However, I can consistently reproduce the problem by passing a path
> > located on a cifs mount (along with a gpg key ID) to this script:
> > 
> > 
> > #!/bin/sh
> > set -e
> > 
> > if [ "$#" -lt 2 ] || [ "$1" = "-h" ] ; then
> >     echo "usage: $(basename "$0") <repo-dir> <gpg-key-id>"
> >     exit 2
> > fi
> > 
> > repo=$1
> > keyid=$2
> > src="./foo"
> > 
> > echo "creating ostree repo at $repo"
> > ostree init --repo="$repo"
> > 
> > echo "creating test tree at $src"
> > mkdir -p "$src"
> > echo hi > "$src"/hello
> > 
> > ostree config --repo="$repo" set core.min-free-space-percent 1
> > ostree commit --repo="$repo" --branch=foo --gpg-sign="$keyid" "$src"
> > 
> > if ostree show --repo="$repo" foo; then
> >     echo ---
> >     echo success!
> > else
> >     echo ---
> >     ostree show --repo="$repo" --print-detached-metadata-key=ostree.gpgsigs foo
> >     echo failure!
> >     echo look for null bytes in the above commit signature
> > fi
> 
> Since you identified this as upstream regression including the
> breaking commit, can you please report this upstream (and please do
> keep us in the loop).
> 
> Since the breaking commit is 3ee1a1fc3981 ("cifs: Cut over to using
> netfslib") so you can report it to:
> 
> David Howells <dhowells@redhat.com>
> Steve French <sfrench@samba.org>
> Shyam Prasad N <nspmangalore@gmail.com>
> Rohith Surabattula <rohiths.msft@gmail.com>
> Jeff Layton <jlayton@kernel.org>
> linux-cifs@vger.kernel.org
> netfs@lists.linux.dev
> linux-fsdevel@vger.kernel.org
> linux-mm@kvack.org
> 
> make sure to include stable@vger.kernel.org and ideally the regression
> mailing list: regressions@lists.linux.dev

One thing I forgot about the regressions list:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-regressions.html
can help making sure the regression tracking has as well the correct
metadata from start about where the issue is introduced, using the
'#regzbot introduced' command.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#83660

After further testing today, that commit also seems to be (intermittently)
causing ffmpeg and mkvmerge to write corrupt files to cifs mounts.


On Fri, 23 Aug 2024 22:13:07 +0200, Salvatore Bonaccorso wrote:

>Since you identified this as upstream regression including the
>breaking commit, can you please report this upstream (and please do
>keep us in the loop).

That is my plan. Last time I reported a bug to the cifs team, it was
ignored, but perhaps I'll have more luck through the additional email
addresses you provided. Thanks for those.


In the meantime, I'm in an awkward position on Debian: The Stable kernel is
too old for my GPU, and both Testing and Unstable now have kernels with this
data-corrupting bug.

Looks like I'm stuck with custom kernel builds until this is sorted out.
6.9.12 doesn't have the bug, but it's labeled EOL on kernel.org, which I
think means no more security updates. I guess I'll have to try the 6.6
branch.

[toc] | [prev] | [next] | [standalone]

#83696

Reported on the lists:

https://lore.kernel.org/linux-cifs/pv2lcjhveti4sfua95o0u6r4i73r39srra@sonic.net/T/#u

Tracked by regzbot:

https://linux-regtracking.leemhuis.info/regzbot/regression/lore/pv2lcjhveti4sfua95o0u6r4i73r39srra@sonic.net/


Related: a bug in recent 6.11 release candidates that prevents me from
running my reproducer for the present bug:

https://lore.kernel.org/linux-cifs/37fncjpgsq45becdf2pdju0idf3hj3dtmb@sonic.net/T/#u

https://linux-regtracking.leemhuis.info/regzbot/regression/lore/37fncjpgsq45becdf2pdju0idf3hj3dtmb@sonic.net/

[toc] | [prev] | [next] | [standalone]

#83654 — Processed: Re: Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption

Processing control commands:

> tags -1 + moreinfo upstream
Bug #1079394 [src:linux] linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption
Added tag(s) moreinfo and upstream.

-- 
1079394: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1079394
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#83884 — Bug#1079394: marked as done (linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption)

[Multipart message — attachments visible in raw view] — view raw

Your message dated Mon, 9 Sep 2024 15:15:40 +0200
with message-id <Zt70_LzVh4ioZndV@eldamar.lan>
and subject line Re: Bug#1079394: linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption
has caused the Debian Bug report #1079394,
regarding linux-image-6.10.6-amd64: causes cifs regression, flatpak & ostree signature corruption
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
1079394: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1079394
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web