Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #67907 > unrolled thread

Bug#969223: Can't rm directory on overlayfs in userns

Back to article view | Back to linux.debian.kernel

Contents

  Bug#969223: Can't rm directory on overlayfs in userns Shengjing Zhu <zhsj@debian.org> - 2020-08-29 16:20 +0200
    Bug#969223: Can't rm directory on overlayfs in userns Shengjing Zhu <zhsj@debian.org> - 2020-09-02 06:00 +0200
      Bug#969223: Can't rm directory on overlayfs in userns Nicolas Schier <nicolas@fjasle.eu> - 2020-09-16 10:10 +0200
        Bug#969223: Can't rm directory on overlayfs in userns Shengjing Zhu <zhsj@debian.org> - 2020-09-16 19:00 +0200
          Bug#969223: Can't rm directory on overlayfs in userns Nicolas Schier <nicolas@fjasle.eu> - 2020-09-16 21:00 +0200
            Bug#969223: Can't rm directory on overlayfs in userns Shengjing Zhu <zhsj@debian.org> - 2020-09-16 21:10 +0200
              Bug#969223: Can't rm directory on overlayfs in userns Nicolas Schier <nicolas@fjasle.eu> - 2020-12-11 09:30 +0100
                Bug#969223: Can't rm directory on overlayfs in userns Nicolas Schier <nicolas@fjasle.eu> - 2021-03-03 08:50 +0100
                  Bug#969223: Can't rm directory on overlayfs in userns Shengjing Zhu <zhsj@debian.org> - 2021-03-03 10:40 +0100
                    Bug#969223: Can't rm directory on overlayfs in userns Nicolas Schier <nicolas@fjasle.eu> - 2021-03-03 11:40 +0100
                      Bug#969223: Can't rm directory on overlayfs in userns Shengjing Zhu <zhsj@debian.org> - 2021-03-03 16:00 +0100
                        Bug#969223: Can't rm directory on overlayfs in userns Nicolas Schier <nicolas@fjasle.eu> - 2021-03-11 07:20 +0100

#67907 — Bug#969223: Can't rm directory on overlayfs in userns

Source: linux
Version: 5.7.10-1
Severity: normal

Hi,

After enabling overlayfs for userns, I find it doesn't work as expected.

$ cat /sys/module/overlay/parameters/permit_mounts_in_userns 
Y

zsj@debian:~/test$ pwd
/home/zsj/test
zsj@debian:~/test$ tree
.
├── lower
│   └── a
│       └── a
├── merged
├── upper
└── work

zsj@debian:~/test$ unshare -m -U -r
root@debian:~/test# mount -t overlay -o rw,lowerdir=/home/zsj/test/lower,upperdir=/home/zsj/test/upper,workdir=/home/zsj/test/work overlay /home/zsj/test/merged
root@debian:~/test# rm -rf merged/a
rm: cannot remove 'merged/a': Input/output error

-- System Information:
Debian Release: bullseye/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 5.7.0-2-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_USER, TAINT_FIRMWARE_WORKAROUND
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

[toc] | [next] | [standalone]

#67961

On Sat, Aug 29, 2020 at 10:13 PM Shengjing Zhu <zhsj@debian.org> wrote:
>
> Source: linux
> Version: 5.7.10-1
> Severity: normal
>
> Hi,
>
> After enabling overlayfs for userns, I find it doesn't work as expected.
>
> $ cat /sys/module/overlay/parameters/permit_mounts_in_userns
> Y
>
> zsj@debian:~/test$ pwd
> /home/zsj/test
> zsj@debian:~/test$ tree
> .
> ├── lower
> │   └── a
> │       └── a
> ├── merged
> ├── upper
> └── work
>
> zsj@debian:~/test$ unshare -m -U -r
> root@debian:~/test# mount -t overlay -o rw,lowerdir=/home/zsj/test/lower,upperdir=/home/zsj/test/upper,workdir=/home/zsj/test/work overlay /home/zsj/test/merged
> root@debian:~/test# rm -rf merged/a
> rm: cannot remove 'merged/a': Input/output error
>
> -- System Information:
> Debian Release: bullseye/sid
>   APT prefers testing
>   APT policy: (500, 'testing')
> Architecture: amd64 (x86_64)
>
> Kernel: Linux 5.7.0-2-amd64 (SMP w/4 CPU threads)
> Kernel taint flags: TAINT_USER, TAINT_FIRMWARE_WORKAROUND
> Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
> Shell: /bin/sh linked to /usr/bin/dash
> Init: systemd (via /run/systemd/system)

If I upgrade a debian10 VM to testing, it seems to work.
However if I boot a new debian testing VM, it seems not to work.
Both VMs are downloaded from http://cdimage.debian.org/cdimage/cloud/
What can be the difference here? I'm lost on debugging this..

-- 
Shengjing Zhu

[toc] | [prev] | [next] | [standalone]

#68143

On Wed, Sep 02, 2020 at 11:52:41AM +0800, Shengjing Zhu wrote:
> On Sat, Aug 29, 2020 at 10:13 PM Shengjing Zhu <zhsj@debian.org> wrote:
> >
> > Source: linux
> > Version: 5.7.10-1
> > Severity: normal
> >
> > Hi,
> >
> > After enabling overlayfs for userns, I find it doesn't work as expected.
> >
> > $ cat /sys/module/overlay/parameters/permit_mounts_in_userns
> > Y
> >
> > zsj@debian:~/test$ pwd
> > /home/zsj/test
> > zsj@debian:~/test$ tree
> > .
> > ├── lower
> > │   └── a
> > │       └── a
> > ├── merged
> > ├── upper
> > └── work
> >
> > zsj@debian:~/test$ unshare -m -U -r
> > root@debian:~/test# mount -t overlay -o rw,lowerdir=/home/zsj/test/lower,upperdir=/home/zsj/test/upper,workdir=/home/zsj/test/work overlay /home/zsj/test/merged
> > root@debian:~/test# rm -rf merged/a
> > rm: cannot remove 'merged/a': Input/output error
> >

Hi,

overlayfs uses filesystem xattrs to mark "whiteouts" and redirects of
directories, which are only accessable for root (CAP_SYS_ADMIN), thus,
not when overlay is mounted in a user namespace, cp. e.g. [1,2].

Ubuntu kernel "solves" this by skipping the "trusted."-xattr check, thus
allowing setting and removal of 'trusted.overlay.*' xattrs from within
user namespaces; but those are still visible in all other namespaces.  A
following overlayfs mount done by the real root user will use these
modified xattrs.

To me it would seem to be more adequate if overlayfs would use
'overlay.*' instead of 'trusted.overlay.*', if it is mounted in an
unpriviledged user namespace.  But this would make overlay mounts done
by root incompatible with those done in a user namespace.

Maybe you find #836211 to be related to this.


[1]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/fs/xattr.c?h=linux-5.7.y#n113
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/fs/xattr.c?h=linux-5.7.y#n1049
[3]: https://kernel.ubuntu.com/git/ubuntu/ubuntu-focal.git/commit/?id=111cd1a9840ce187e28b49fe4e77b9b5e84386b1

> If I upgrade a debian10 VM to testing, it seems to work.
> However if I boot a new debian testing VM, it seems not to work.
> Both VMs are downloaded from http://cdimage.debian.org/cdimage/cloud/
> What can be the difference here? I'm lost on debugging this..

This confuses me.  Are you sure, you used the same kernel version on
both VMs when mounting overlayfs in userns?

Kind regards,
Nicolas

-- 
epost: nicolas@fjasle.eu               irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --

[toc] | [prev] | [next] | [standalone]

#68144

On Wed, Sep 16, 2020 at 3:58 PM Nicolas Schier <nicolas@fjasle.eu> wrote:
> > If I upgrade a debian10 VM to testing, it seems to work.
> > However if I boot a new debian testing VM, it seems not to work.
> > Both VMs are downloaded from http://cdimage.debian.org/cdimage/cloud/
> > What can be the difference here? I'm lost on debugging this..
>
> This confuses me.  Are you sure, you used the same kernel version on
> both VMs when mounting overlayfs in userns?
>

I think I just mess up when debugging. It seems it never works.

Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
work. Buster is also affected.

-- 
Shengjing Zhu

[toc] | [prev] | [next] | [standalone]

#68145

> I think I just mess up when debugging. It seems it never works.
> 
> Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
> work. Buster is also affected.

Please, don't be too fast when thinking about a revert.  Several of my
colleagues (Debian users) cling to the feature since they need it for
using the company's LXC containers; if permit_mounts_in_userns is
removed again, they might be forced to switch to non-Debian kernels or
to live-patch the kernel with fragile stuff like [1], cp. #913880.


[1]: https://rocketgit.com/user/nicolas/overlay-userns-dkms

-- 
epost: nicolas@fjasle.eu               irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --

[toc] | [prev] | [next] | [standalone]

#68146

On Thu, Sep 17, 2020 at 2:52 AM Nicolas Schier <nicolas@fjasle.eu> wrote:
>
> > I think I just mess up when debugging. It seems it never works.
> >
> > Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
> > work. Buster is also affected.
>
> Please, don't be too fast when thinking about a revert.  Several of my
> colleagues (Debian users) cling to the feature since they need it for
> using the company's LXC containers; if permit_mounts_in_userns is
> removed again, they might be forced to switch to non-Debian kernels or
> to live-patch the kernel with fragile stuff like [1], cp. #913880.

I mean if you can't even remove a directory with files, it's too broken to use.
So your colleagues find the userns overlay works?
Or you mean we should take Ubuntu's patch to fix the issue?

-- 
Shengjing Zhu

[toc] | [prev] | [next] | [standalone]

#68809

[Multipart message — attachments visible in raw view] — view raw

On Tue, Sep 17, 2020 at 2:57 AM Shengjing Zhu wrote:
> 
> On Thu, Sep 17, 2020 at 2:52 AM Nicolas Schier <nicolas@fjasle.eu> wrote:
> >
> > > I think I just mess up when debugging. It seems it never works.
> > >
> > > Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
> > > work. Buster is also affected.
> >
> > Please, don't be too fast when thinking about a revert.  Several of my
> > colleagues (Debian users) cling to the feature since they need it for
> > using the company's LXC containers; if permit_mounts_in_userns is
> > removed again, they might be forced to switch to non-Debian kernels or
> > to live-patch the kernel with fragile stuff like [1], cp. #913880.
> 
> I mean if you can't even remove a directory with files, it's too broken to use.
> So your colleagues find the userns overlay works?
> Or you mean we should take Ubuntu's patch to fix the issue?

sorry for the long delay.  My colleagues are using unpriviledged LXC 
with overlay fs for building purposes only, thus, read-only access is 
sufficient and works.  (But yes, the incomplete write-support leads to 
annoyance.)

Currently, there is a patch on linux-unionfs that allows using user 
xattrs for overlay fs meta data [1].  If the related patchset [2] is 
going to be merged, the Debian patch will become obsolete; otherwise we 
could think about picking up the patch from [1].

As far as I have seen, the Ubuntu patch allows unpriviledged users to 
modify 'trusted.overlay.*' xattrs, which probably has security 
implications.  ("Probably" as just had a superficial look at it.)

I would prefer to keep a watch on [2] and dicuss further, if it has 
been merged or rejected.

Kind regards,
Nicolas



[1]: [PATCH v2 06/10] ovl: user xattr
     https://lore.kernel.org/linux-unionfs/20201207163255.564116-7-mszeredi@redhat.com/

[2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eYapsEUdmPcRBUiwQ@mail.gmail.com/T/#t

[toc] | [prev] | [next] | [standalone]

#69667

[Multipart message — attachments visible in raw view] — view raw

On Fri, Dec 11, 2020 09:10 AM Nicolas Schier wrote:
> 
> On Tue, Sep 17, 2020 at 2:57 AM Shengjing Zhu wrote:
> > 
> > On Thu, Sep 17, 2020 at 2:52 AM Nicolas Schier <nicolas@fjasle.eu> wrote:
> > >
> > > > I think I just mess up when debugging. It seems it never works.
> > > >
> > > > Maybe we should revert permit_mounts_in_userns? as it doesn't seem to
> > > > work. Buster is also affected.
> > >
> > > Please, don't be too fast when thinking about a revert.  Several of my
> > > colleagues (Debian users) cling to the feature since they need it for
> > > using the company's LXC containers; if permit_mounts_in_userns is
> > > removed again, they might be forced to switch to non-Debian kernels or
> > > to live-patch the kernel with fragile stuff like [1], cp. #913880.
> > 
> > I mean if you can't even remove a directory with files, it's too broken to use.
> > So your colleagues find the userns overlay works?
> > Or you mean we should take Ubuntu's patch to fix the issue?
> 
> sorry for the long delay.  My colleagues are using unpriviledged LXC 
> with overlay fs for building purposes only, thus, read-only access is 
> sufficient and works.  (But yes, the incomplete write-support leads to 
> annoyance.)
> 
> Currently, there is a patch on linux-unionfs that allows using user 
> xattrs for overlay fs meta data [1].  If the related patchset [2] is 
> going to be merged, the Debian patch will become obsolete; otherwise we 
> could think about picking up the patch from [1].
> 
> As far as I have seen, the Ubuntu patch allows unpriviledged users to 
> modify 'trusted.overlay.*' xattrs, which probably has security 
> implications.  ("Probably" as just had a superficial look at it.)
> 
> I would prefer to keep a watch on [2] and dicuss further, if it has 
> been merged or rejected.
> 
> Kind regards,
> Nicolas
> 
> 
> 
> [1]: [PATCH v2 06/10] ovl: user xattr
>      https://lore.kernel.org/linux-unionfs/20201207163255.564116-7-mszeredi@redhat.com/
> 
> [2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eYapsEUdmPcRBUiwQ@mail.gmail.com/T/#t

The overlay fs patchset [2] has been merged and with v5.10.13 (tested 
on linux-image-5.10.0-3-arm64) the issue is no more reproducible for 
me.  Might you want to re-check on your site?

Kind regards,
Nicolas


-- 
epost: nicolas@fjasle.eu               irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --

[toc] | [prev] | [next] | [standalone]

#69669

On Wed, Mar 3, 2021 at 3:40 PM Nicolas Schier <nicolas@fjasle.eu> wrote:
> > [2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eYapsEUdmPcRBUiwQ@mail.gmail.com/T/#t
>
> The overlay fs patchset [2] has been merged and with v5.10.13 (tested
> on linux-image-5.10.0-3-arm64) the issue is no more reproducible for
> me.  Might you want to re-check on your site?
>

If I understand correctly, the upstream patch is merged into the v5.11 tree.

And I still can reproduce the error on the Debian v5.10 kernel.

And another thing is that the upstream patch introduces a new mount
option, userxattr, instead of module parameter.

-- 
Shengjing Zhu

[toc] | [prev] | [next] | [standalone]

#69670

[Multipart message — attachments visible in raw view] — view raw

On Wed 03 Mar 2021 17:33:16 GMT Shengjing Zhu write:
> 
> On Wed, Mar 3, 2021 at 3:40 PM Nicolas Schier <nicolas@fjasle.eu> wrote:
> > > [2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eYapsEUdmPcRBUiwQ@mail.gmail.com/T/#t
> >
> > The overlay fs patchset [2] has been merged and with v5.10.13 (tested
> > on linux-image-5.10.0-3-arm64) the issue is no more reproducible for
> > me.  Might you want to re-check on your site?
> >
> 
> If I understand correctly, the upstream patch is merged into the v5.11 tree.

Sorry.  Yes, you're right.

> And I still can reproduce the error on the Debian v5.10 kernel.

That confuses me quite a bit.  I did it once again on an ext4 mount 
(still the 5.10.0-3-arm64 kernel):

  nsc@lillesand:/tmp$ cat /sys/module/overlay/parameters/permit_mounts_in_userns 
  Y
  nsc@lillesand:/tmp$ mkdir -p test/lower/a test/merged test/upper test/work
  nsc@lillesand:/tmp$ uname -a | tee test/lower/a/a
  Linux lillesand 5.10.0-3-arm64 #1 SMP Debian 5.10.13-1 (2021-02-06) aarch64 GNU/Linux
  nsc@lillesand:/tmp$ unshare -m -U -r
  root@lillesand:/tmp# mount -t overlay -o rw,lowerdir=/tmp/test/lower,upperdir=/tmp/test/upper,workdir=/tmp/test/work overlay /tmp/test/merged
  root@lillesand:/tmp# rm -rf test/merged/a
  root@lillesand:/tmp# find test -ls
    1597776      4 drwxr-xr-x   6 root     root         4096 mars  3 08:24 test
    1973978      4 drwxr-xr-x   2 root     root         4096 mars  3 08:27 test/upper
    2099881      0 c---------   1 root     root       0,   0 mars  3 08:27 test/upper/a
    1973978      4 drwxr-xr-x   1 root     root         4096 mars  3 08:27 test/merged
    1714388      4 drwxr-xr-x   3 root     root         4096 mars  3 08:24 test/lower
    1714389      4 drwxr-xr-x   2 root     root         4096 mars  3 08:27 test/lower/a
    1714393      4 -rw-r--r--   1 root     root           86 mars  3 10:48 test/lower/a/a
    1973979      4 drwxr-xr-x   3 root     root         4096 mars  3 10:48 test/work
    2099880      4 d---------   2 root     root         4096 mars  3 10:48 test/work/work
  root@lillesand:/tmp# 

Do you see any kernel log message from overlay fs?  Might it depend on 
the underlying filesystem? Can you create a white-out char dev node 
manually?

> And another thing is that the upstream patch introduces a new mount
> option, userxattr, instead of module parameter.

The 'permit_mounts_in_userns' module parameter becomes superfluous with 
v5.11 as overlay fs mounts will then always be enabled in userspace 
namespace.

Kind regards,
Nicolas

-- 
epost: nicolas@fjasle.eu               irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --

[toc] | [prev] | [next] | [standalone]

#69680

On Wed, Mar 03, 2021 at 11:30:20AM +0100, Nicolas Schier wrote:
> On Wed 03 Mar 2021 17:33:16 GMT Shengjing Zhu write:
> > 
> > On Wed, Mar 3, 2021 at 3:40 PM Nicolas Schier <nicolas@fjasle.eu> wrote:
> > > > [2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eYapsEUdmPcRBUiwQ@mail.gmail.com/T/#t
> > >
> > > The overlay fs patchset [2] has been merged and with v5.10.13 (tested
> > > on linux-image-5.10.0-3-arm64) the issue is no more reproducible for
> > > me.  Might you want to re-check on your site?
> > >
> > 
> > If I understand correctly, the upstream patch is merged into the v5.11 tree.
> 
> Sorry.  Yes, you're right.
> 
> > And I still can reproduce the error on the Debian v5.10 kernel.
> 
> That confuses me quite a bit.  I did it once again on an ext4 mount 
> (still the 5.10.0-3-arm64 kernel):
> 
>   nsc@lillesand:/tmp$ cat /sys/module/overlay/parameters/permit_mounts_in_userns 
>   Y
>   nsc@lillesand:/tmp$ mkdir -p test/lower/a test/merged test/upper test/work
>   nsc@lillesand:/tmp$ uname -a | tee test/lower/a/a
>   Linux lillesand 5.10.0-3-arm64 #1 SMP Debian 5.10.13-1 (2021-02-06) aarch64 GNU/Linux
>   nsc@lillesand:/tmp$ unshare -m -U -r
>   root@lillesand:/tmp# mount -t overlay -o rw,lowerdir=/tmp/test/lower,upperdir=/tmp/test/upper,workdir=/tmp/test/work overlay /tmp/test/merged
>   root@lillesand:/tmp# rm -rf test/merged/a
>   root@lillesand:/tmp# find test -ls
>     1597776      4 drwxr-xr-x   6 root     root         4096 mars  3 08:24 test
>     1973978      4 drwxr-xr-x   2 root     root         4096 mars  3 08:27 test/upper
>     2099881      0 c---------   1 root     root       0,   0 mars  3 08:27 test/upper/a
>     1973978      4 drwxr-xr-x   1 root     root         4096 mars  3 08:27 test/merged
>     1714388      4 drwxr-xr-x   3 root     root         4096 mars  3 08:24 test/lower
>     1714389      4 drwxr-xr-x   2 root     root         4096 mars  3 08:27 test/lower/a
>     1714393      4 -rw-r--r--   1 root     root           86 mars  3 10:48 test/lower/a/a
>     1973979      4 drwxr-xr-x   3 root     root         4096 mars  3 10:48 test/work
>     2099880      4 d---------   2 root     root         4096 mars  3 10:48 test/work/work
>   root@lillesand:/tmp# 
> 
zsj@debian:~$ cat /sys/module/overlay/parameters/permit_mounts_in_userns 
Y
zsj@debian:~/t$ mkdir -p test/lower/a test/merged test/upper test/work
zsj@debian:~/t$ uname -a | tee test/lower/a/a
Linux debian 5.10.0-3-amd64 #1 SMP Debian 5.10.13-1 (2021-02-06) x86_64 GNU/Linux
zsj@debian:~/t$ unshare -m -U -r
root@debian:~/t# mount -t overlay -o rw,lowerdir=./test/lower,upperdir=./test/upper,workdir=./test/work overlay ./test/merged/
root@debian:~/t# rm -rf ./test/merged/a
rm: cannot remove './test/merged/a': Input/output error
root@debian:~/t# find test -ls
  7350352      4 drwxr-xr-x   6 root     root         4096 Mar  3 22:44 test
  7351341      4 drwxr-xr-x   3 root     root         4096 Mar  3 22:44 test/lower
  7353492      4 drwxr-xr-x   2 root     root         4096 Mar  3 22:44 test/lower/a
  7356441      4 -rw-r--r--   1 root     root           82 Mar  3 22:44 test/lower/a/a
  7356069      4 drwxr-xr-x   3 root     root         4096 Mar  3 22:45 test/work
  7358324      4 d---------   2 root     root         4096 Mar  3 22:45 test/work/work
  7358564      0 c---------   2 root     root       0,   0 Mar  3 22:45 test/work/work/#4
  7354400      4 drwxr-xr-x   3 root     root         4096 Mar  3 22:44 test/upper
  7358563      4 drwxr-xr-x   2 root     root         4096 Mar  3 22:45 test/upper/a
  7358564      0 c---------   2 root     root       0,   0 Mar  3 22:45 test/upper/a/a
  7354400      4 drwxr-xr-x   1 root     root         4096 Mar  3 22:44 test/merged
  7353492      4 drwxr-xr-x   1 root     root         4096 Mar  3 22:45 test/merged/a

> Do you see any kernel log message from overlay fs?  Might it depend on 
> the underlying filesystem? Can you create a white-out char dev node 
> manually?
> 

[1215353.859717] Setting dangerous option permit_mounts_in_userns - tainting kernel
[1215353.859841] overlayfs: overlayfs: Allowing overlay mounts in user namespaces bears security risks
[1215425.416543] overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

The underlying fs is ext4.

zsj@debian:~/t$ mount|grep nvme
/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)
/dev/nvme0n1p4 on /home type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)

I don't know how to test "create a white-out char dev node manually".

Thanks

[toc] | [prev] | [next] | [standalone]

#69780

[Multipart message — attachments visible in raw view] — view raw

On Wed 03 Mar 2021 22:50:44 GMT Shengjing Zhu wrote:
> On Wed, Mar 03, 2021 at 11:30:20AM +0100, Nicolas Schier wrote:
> > On Wed 03 Mar 2021 17:33:16 GMT Shengjing Zhu write:
> > > 
> > > On Wed, Mar 3, 2021 at 3:40 PM Nicolas Schier <nicolas@fjasle.eu> wrote:
> > > > > [2]: https://lore.kernel.org/linux-unionfs/CAJfpegsiuf8ib5cvVrr=zHZ+Xu7BMMTT2eYapsEUdmPcRBUiwQ@mail.gmail.com/T/#t
> > > >
> > > > The overlay fs patchset [2] has been merged and with v5.10.13 (tested
> > > > on linux-image-5.10.0-3-arm64) the issue is no more reproducible for
> > > > me.  Might you want to re-check on your site?
> > > >
> > > 
> > > If I understand correctly, the upstream patch is merged into the v5.11 tree.
> > 
> > Sorry.  Yes, you're right.
> > 
> > > And I still can reproduce the error on the Debian v5.10 kernel.
> > 
> > That confuses me quite a bit.  I did it once again on an ext4 mount 
> > (still the 5.10.0-3-arm64 kernel):
> > 
> >   nsc@lillesand:/tmp$ cat /sys/module/overlay/parameters/permit_mounts_in_userns 
> >   Y
> >   nsc@lillesand:/tmp$ mkdir -p test/lower/a test/merged test/upper test/work
> >   nsc@lillesand:/tmp$ uname -a | tee test/lower/a/a
> >   Linux lillesand 5.10.0-3-arm64 #1 SMP Debian 5.10.13-1 (2021-02-06) aarch64 GNU/Linux
> >   nsc@lillesand:/tmp$ unshare -m -U -r
> >   root@lillesand:/tmp# mount -t overlay -o rw,lowerdir=/tmp/test/lower,upperdir=/tmp/test/upper,workdir=/tmp/test/work overlay /tmp/test/merged
> >   root@lillesand:/tmp# rm -rf test/merged/a
> >   root@lillesand:/tmp# find test -ls
> >     1597776      4 drwxr-xr-x   6 root     root         4096 mars  3 08:24 test
> >     1973978      4 drwxr-xr-x   2 root     root         4096 mars  3 08:27 test/upper
> >     2099881      0 c---------   1 root     root       0,   0 mars  3 08:27 test/upper/a
> >     1973978      4 drwxr-xr-x   1 root     root         4096 mars  3 08:27 test/merged
> >     1714388      4 drwxr-xr-x   3 root     root         4096 mars  3 08:24 test/lower
> >     1714389      4 drwxr-xr-x   2 root     root         4096 mars  3 08:27 test/lower/a
> >     1714393      4 -rw-r--r--   1 root     root           86 mars  3 10:48 test/lower/a/a
> >     1973979      4 drwxr-xr-x   3 root     root         4096 mars  3 10:48 test/work
> >     2099880      4 d---------   2 root     root         4096 mars  3 10:48 test/work/work
> >   root@lillesand:/tmp# 
> > 
> zsj@debian:~$ cat /sys/module/overlay/parameters/permit_mounts_in_userns 
> Y
> zsj@debian:~/t$ mkdir -p test/lower/a test/merged test/upper test/work
> zsj@debian:~/t$ uname -a | tee test/lower/a/a
> Linux debian 5.10.0-3-amd64 #1 SMP Debian 5.10.13-1 (2021-02-06) x86_64 GNU/Linux
> zsj@debian:~/t$ unshare -m -U -r
> root@debian:~/t# mount -t overlay -o rw,lowerdir=./test/lower,upperdir=./test/upper,workdir=./test/work overlay ./test/merged/
> root@debian:~/t# rm -rf ./test/merged/a
> rm: cannot remove './test/merged/a': Input/output error
> root@debian:~/t# find test -ls
>   7350352      4 drwxr-xr-x   6 root     root         4096 Mar  3 22:44 test
>   7351341      4 drwxr-xr-x   3 root     root         4096 Mar  3 22:44 test/lower
>   7353492      4 drwxr-xr-x   2 root     root         4096 Mar  3 22:44 test/lower/a
>   7356441      4 -rw-r--r--   1 root     root           82 Mar  3 22:44 test/lower/a/a
>   7356069      4 drwxr-xr-x   3 root     root         4096 Mar  3 22:45 test/work
>   7358324      4 d---------   2 root     root         4096 Mar  3 22:45 test/work/work
>   7358564      0 c---------   2 root     root       0,   0 Mar  3 22:45 test/work/work/#4
>   7354400      4 drwxr-xr-x   3 root     root         4096 Mar  3 22:44 test/upper
>   7358563      4 drwxr-xr-x   2 root     root         4096 Mar  3 22:45 test/upper/a
>   7358564      0 c---------   2 root     root       0,   0 Mar  3 22:45 test/upper/a/a
>   7354400      4 drwxr-xr-x   1 root     root         4096 Mar  3 22:44 test/merged
>   7353492      4 drwxr-xr-x   1 root     root         4096 Mar  3 22:45 test/merged/a
> 
> > Do you see any kernel log message from overlay fs?  Might it depend on 
> > the underlying filesystem? Can you create a white-out char dev node 
> > manually?
> > 
> 
> [1215353.859717] Setting dangerous option permit_mounts_in_userns - tainting kernel
> [1215353.859841] overlayfs: overlayfs: Allowing overlay mounts in user namespaces bears security risks
> [1215425.416543] overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
> 
> The underlying fs is ext4.
> 
> zsj@debian:~/t$ mount|grep nvme
> /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)
> /dev/nvme0n1p4 on /home type ext4 (rw,relatime)
> /dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
> 
> I don't know how to test "create a white-out char dev node manually".
> 
> Thanks

thanks for re-validation.  I can reproduce the issue exactly on amd64 
system, but not on arm64.  It's really strange, also enabling dynamic 
debug output does not reveal anything useful.  I guess it would be good 
to dig into the related kernel code to get some deeper understanding, I 
hope to get start examining a bit in the upcoming weekend.

Kind regards,
Nicolas


-- 
epost|xmpp: nicolas@fjasle.eu          irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web