Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1710588 > unrolled thread

Re: Do we really need d_weak_revalidate???

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Do we really need d_weak_revalidate??? NeilBrown <neilb@suse.com> - 2017-08-14 01:40 +0200
    Re: Do we really need d_weak_revalidate??? Jeff Layton <jlayton@redhat.com> - 2017-08-14 12:20 +0200
      Re: Do we really need d_weak_revalidate??? NeilBrown <neilb@suse.com> - 2017-08-16 04:50 +0200
        Re: Do we really need d_weak_revalidate??? Jeff Layton <jlayton@redhat.com> - 2017-08-16 13:40 +0200
          Re: Do we really need d_weak_revalidate??? NeilBrown <neilb@suse.com> - 2017-08-17 01:50 +0200
          Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-17 04:30 +0200
            Re: Do we really need d_weak_revalidate??? NeilBrown <neilb@suse.com> - 2017-08-18 07:30 +0200
              Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-18 08:50 +0200
                Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-18 09:00 +0200
                Re: Do we really need d_weak_revalidate??? NeilBrown <neilb@suse.com> - 2017-08-21 08:30 +0200
                  Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-21 08:40 +0200
                    Re: Do we really need d_weak_revalidate??? NeilBrown <neilb@suse.com> - 2017-08-21 09:50 +0200
                    Re: Do we really need d_weak_revalidate??? NeilBrown <neilb@suse.com> - 2017-08-23 03:10 +0200
                      Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-23 04:40 +0200
                        Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-23 04:50 +0200
                          Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-23 05:00 +0200
                            Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-23 10:00 +0200
                          Re: Do we really need d_weak_revalidate??? NeilBrown <neilb@suse.com> - 2017-08-24 05:30 +0200
                            Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-24 06:40 +0200
                        Re: Do we really need d_weak_revalidate??? NeilBrown <neilb@suse.com> - 2017-08-24 06:10 +0200
                          Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-24 06:50 +0200
                          Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-24 07:00 +0200
                          Re: Do we really need d_weak_revalidate??? "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> - 2017-08-24 13:10 +0200
                            Re: Do we really need d_weak_revalidate??? Ian Kent <ikent@redhat.com> - 2017-08-25 02:10 +0200
                            [PATCH manpages] stat.2: correct AT_NO_AUTOMOUNT text and general revisions. NeilBrown <neilb@suse.com> - 2017-08-25 07:40 +0200

Page 1 of 2  [1] 2  Next page →

#1710588 — Re: Do we really need d_weak_revalidate???

[Multipart message — attachments visible in raw view] — view raw

On Fri, Aug 11 2017, Jeff Layton wrote:

> On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>> On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>> > Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>> > flag and introduced the d_weak_revalidate dentry operation instead.
>> > We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>> > and added the new dentry operation to NFS dentries .... but not to
>> > NFSv4
>> > dentries.
>> > 
>> > And nobody noticed.
>> > 
>> > Until today.
>> > 
>> > A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>> > NFS
>> > filesystem hangs because the network has been deconfigured.  This
>> > makes
>> > perfect sense and I suggested a code change to fix the problem.
>> > However when a colleague was trying to reproduce the problem to
>> > validate
>> > the fix, he couldn't.  Then nor could I.
>> > 
>> > The problem is trivially reproducible with NFSv3, and not at all with
>> > NFSv4.  The reason is the missing d_weak_revalidate.
>> > 
>> > We could simply add d_weak_revalidate for NFSv4, but given that it
>> > has been missing for 4.5 years, and the only time anyone noticed was
>> > when the ommission resulted in a better user experience, I do wonder
>> > if
>> > we need to.  Can we just discard d_weak_revalidate?  What purpose
>> > does
>> > it serve?  I couldn't find one.
>> > 
>> > Thanks,
>> > NeilBrown
>> > 
>> > For reference, see
>> > Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>> > d_weak_revalidate dentry op")
>> > 
>> > 
>> > 
>> > To reproduce the problem at home, on a system that uses systemd:
>> > 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>> > 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>> > 3/ loop-mount the filesystem image read-only somewhere
>> > 4/ reboot
>> > 
>> > If you choose v4, the reboot will succeed, possibly after a 90second
>> > timeout.
>> > If you choose v3, the reboot will hang indefinitely in systemd-
>> > shutdown while
>> > remounting the nfs filesystem read-only.
>> > 
>> > If you don't use "noac" it can still hang, but only if something
>> > slows
>> > down the reboot enough that attributes have timed out by the time
>> > that
>> > systemd-shutdown runs.  This happens for our customer.
>> > 
>> > If the loop-mounted filesystem is not read-only, you get other
>> > problems.
>> > 
>> > We really want systemd to figure out that the loop-mount needs to be
>> > unmounted first.  I have ideas concerning that, but it is messy.  But
>> > that isn't the only bug here.
>> 
>> The main purpose of d_weak_revalidate() was to catch the issues that
>> arise when someone changes the contents of the current working
>> directory or its parent on the server. Since '.' and '..' are treated
>> specially in the lookup code, they would not be revalidated without
>> special treatment. That leads to issues when looking up files as
>> ./<filename> or ../<filename>, since the client won't detect that its
>> dcache is stale until it tries to use the cached dentry+inode.
>> 
>> The one thing that has changed since its introduction is, I believe,
>> the ESTALE handling in the VFS layer. That might fix a lot of the
>> dcache lookup bugs that were previously handled by d_weak_revalidate().
>> I haven't done an audit to figure out if it actually can handle all of
>> them.
>> 
>
> It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>
>     vfs: allow umount to handle mountpoints without revalidating them

You say in the comment for that commit:

     but there
    are cases where we do want to revalidate the root of the fs.

Do you happen to remember what those cases are?

>
> Possibly the fact that we no longer try to revalidate during unmount
> means that this is no longer necessary?
>
> The original patch that added d_weak_revalidate had a reproducer in the
> patch description. Have you verified that that problem is still not
> reproducible when you remove d_weak_revalidate?

I did try the reproducer and it works as expected both with and without
d_weak_revalidate.
On reflection, the problem it displayed was caused by d_revalidate()
being called when the dentry name was irrelevant.  We remove that
(fixing the problem) and introduce d_weak_revalidate because we thought
that minimum functionality was still useful.  I'm currently not
convinced that even that is needed.

If we discarded d_weak_revalidate(), we could get rid of the special
handling of umount....

Thanks,
NeilBrown

[toc] | [next] | [standalone]

#1710792

On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
> On Fri, Aug 11 2017, Jeff Layton wrote:
> 
> > On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
> > > On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
> > > > Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
> > > > flag and introduced the d_weak_revalidate dentry operation instead.
> > > > We duly removed the flag from NFS superblocks and NFSv4 superblocks,
> > > > and added the new dentry operation to NFS dentries .... but not to
> > > > NFSv4
> > > > dentries.
> > > > 
> > > > And nobody noticed.
> > > > 
> > > > Until today.
> > > > 
> > > > A customer reports a situation where mount(....,MS_REMOUNT,..) on an
> > > > NFS
> > > > filesystem hangs because the network has been deconfigured.  This
> > > > makes
> > > > perfect sense and I suggested a code change to fix the problem.
> > > > However when a colleague was trying to reproduce the problem to
> > > > validate
> > > > the fix, he couldn't.  Then nor could I.
> > > > 
> > > > The problem is trivially reproducible with NFSv3, and not at all with
> > > > NFSv4.  The reason is the missing d_weak_revalidate.
> > > > 
> > > > We could simply add d_weak_revalidate for NFSv4, but given that it
> > > > has been missing for 4.5 years, and the only time anyone noticed was
> > > > when the ommission resulted in a better user experience, I do wonder
> > > > if
> > > > we need to.  Can we just discard d_weak_revalidate?  What purpose
> > > > does
> > > > it serve?  I couldn't find one.
> > > > 
> > > > Thanks,
> > > > NeilBrown
> > > > 
> > > > For reference, see
> > > > Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
> > > > d_weak_revalidate dentry op")
> > > > 
> > > > 
> > > > 
> > > > To reproduce the problem at home, on a system that uses systemd:
> > > > 1/ place (or find) a filesystem image in a file on an NFS filesystem.
> > > > 2/ mount the nfs filesystem with "noac" - choose v3 or v4
> > > > 3/ loop-mount the filesystem image read-only somewhere
> > > > 4/ reboot
> > > > 
> > > > If you choose v4, the reboot will succeed, possibly after a 90second
> > > > timeout.
> > > > If you choose v3, the reboot will hang indefinitely in systemd-
> > > > shutdown while
> > > > remounting the nfs filesystem read-only.
> > > > 
> > > > If you don't use "noac" it can still hang, but only if something
> > > > slows
> > > > down the reboot enough that attributes have timed out by the time
> > > > that
> > > > systemd-shutdown runs.  This happens for our customer.
> > > > 
> > > > If the loop-mounted filesystem is not read-only, you get other
> > > > problems.
> > > > 
> > > > We really want systemd to figure out that the loop-mount needs to be
> > > > unmounted first.  I have ideas concerning that, but it is messy.  But
> > > > that isn't the only bug here.
> > > 
> > > The main purpose of d_weak_revalidate() was to catch the issues that
> > > arise when someone changes the contents of the current working
> > > directory or its parent on the server. Since '.' and '..' are treated
> > > specially in the lookup code, they would not be revalidated without
> > > special treatment. That leads to issues when looking up files as
> > > ./<filename> or ../<filename>, since the client won't detect that its
> > > dcache is stale until it tries to use the cached dentry+inode.
> > > 
> > > The one thing that has changed since its introduction is, I believe,
> > > the ESTALE handling in the VFS layer. That might fix a lot of the
> > > dcache lookup bugs that were previously handled by d_weak_revalidate().
> > > I haven't done an audit to figure out if it actually can handle all of
> > > them.
> > > 
> > 
> > It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
> > 
> >     vfs: allow umount to handle mountpoints without revalidating them
> 
> You say in the comment for that commit:
> 
>      but there
>     are cases where we do want to revalidate the root of the fs.
> 
> Do you happen to remember what those cases are?
> 

Not exactly, but I _think_ I might have been assuming that we needed to
ensure that the inode attrs on the root were up to date after the
pathwalk.

I think that was probably wrong. d_revalidate is really intended to
ensure that the dentry in question still points to the same inode. In
the case of the root of the mount though, we don't really care about the
dentry on the server at all. We're attaching the root of the mount to an
inode and don't care of the dentry name changes. If we do need to ensure
the inode attrs are updated, we'll just revalidate them at that point.

> > 
> > Possibly the fact that we no longer try to revalidate during unmount
> > means that this is no longer necessary?
> > 
> > The original patch that added d_weak_revalidate had a reproducer in the
> > patch description. Have you verified that that problem is still not
> > reproducible when you remove d_weak_revalidate?
> 
> I did try the reproducer and it works as expected both with and without
> d_weak_revalidate.
> On reflection, the problem it displayed was caused by d_revalidate()
> being called when the dentry name was irrelevant.  We remove that
> (fixing the problem) and introduce d_weak_revalidate because we thought
> that minimum functionality was still useful.  I'm currently not
> convinced that even that is needed.
> 
> If we discarded d_weak_revalidate(), we could get rid of the special
> handling of umount....

I like idea. I say go for it and we can see what (if anything) breaks?
-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1712616

[Multipart message — attachments visible in raw view] — view raw

On Mon, Aug 14 2017, Jeff Layton wrote:

> On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
>> On Fri, Aug 11 2017, Jeff Layton wrote:
>> 
>> > On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>> > > On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>> > > > Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>> > > > flag and introduced the d_weak_revalidate dentry operation instead.
>> > > > We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>> > > > and added the new dentry operation to NFS dentries .... but not to
>> > > > NFSv4
>> > > > dentries.
>> > > > 
>> > > > And nobody noticed.
>> > > > 
>> > > > Until today.
>> > > > 
>> > > > A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>> > > > NFS
>> > > > filesystem hangs because the network has been deconfigured.  This
>> > > > makes
>> > > > perfect sense and I suggested a code change to fix the problem.
>> > > > However when a colleague was trying to reproduce the problem to
>> > > > validate
>> > > > the fix, he couldn't.  Then nor could I.
>> > > > 
>> > > > The problem is trivially reproducible with NFSv3, and not at all with
>> > > > NFSv4.  The reason is the missing d_weak_revalidate.
>> > > > 
>> > > > We could simply add d_weak_revalidate for NFSv4, but given that it
>> > > > has been missing for 4.5 years, and the only time anyone noticed was
>> > > > when the ommission resulted in a better user experience, I do wonder
>> > > > if
>> > > > we need to.  Can we just discard d_weak_revalidate?  What purpose
>> > > > does
>> > > > it serve?  I couldn't find one.
>> > > > 
>> > > > Thanks,
>> > > > NeilBrown
>> > > > 
>> > > > For reference, see
>> > > > Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>> > > > d_weak_revalidate dentry op")
>> > > > 
>> > > > 
>> > > > 
>> > > > To reproduce the problem at home, on a system that uses systemd:
>> > > > 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>> > > > 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>> > > > 3/ loop-mount the filesystem image read-only somewhere
>> > > > 4/ reboot
>> > > > 
>> > > > If you choose v4, the reboot will succeed, possibly after a 90second
>> > > > timeout.
>> > > > If you choose v3, the reboot will hang indefinitely in systemd-
>> > > > shutdown while
>> > > > remounting the nfs filesystem read-only.
>> > > > 
>> > > > If you don't use "noac" it can still hang, but only if something
>> > > > slows
>> > > > down the reboot enough that attributes have timed out by the time
>> > > > that
>> > > > systemd-shutdown runs.  This happens for our customer.
>> > > > 
>> > > > If the loop-mounted filesystem is not read-only, you get other
>> > > > problems.
>> > > > 
>> > > > We really want systemd to figure out that the loop-mount needs to be
>> > > > unmounted first.  I have ideas concerning that, but it is messy.  But
>> > > > that isn't the only bug here.
>> > > 
>> > > The main purpose of d_weak_revalidate() was to catch the issues that
>> > > arise when someone changes the contents of the current working
>> > > directory or its parent on the server. Since '.' and '..' are treated
>> > > specially in the lookup code, they would not be revalidated without
>> > > special treatment. That leads to issues when looking up files as
>> > > ./<filename> or ../<filename>, since the client won't detect that its
>> > > dcache is stale until it tries to use the cached dentry+inode.
>> > > 
>> > > The one thing that has changed since its introduction is, I believe,
>> > > the ESTALE handling in the VFS layer. That might fix a lot of the
>> > > dcache lookup bugs that were previously handled by d_weak_revalidate().
>> > > I haven't done an audit to figure out if it actually can handle all of
>> > > them.
>> > > 
>> > 
>> > It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>> > 
>> >     vfs: allow umount to handle mountpoints without revalidating them
>> 
>> You say in the comment for that commit:
>> 
>>      but there
>>     are cases where we do want to revalidate the root of the fs.
>> 
>> Do you happen to remember what those cases are?
>> 
>
> Not exactly, but I _think_ I might have been assuming that we needed to
> ensure that the inode attrs on the root were up to date after the
> pathwalk.
>
> I think that was probably wrong. d_revalidate is really intended to
> ensure that the dentry in question still points to the same inode. In
> the case of the root of the mount though, we don't really care about the
> dentry on the server at all. We're attaching the root of the mount to an
> inode and don't care of the dentry name changes. If we do need to ensure
> the inode attrs are updated, we'll just revalidate them at that point.
>
>> > 
>> > Possibly the fact that we no longer try to revalidate during unmount
>> > means that this is no longer necessary?
>> > 
>> > The original patch that added d_weak_revalidate had a reproducer in the
>> > patch description. Have you verified that that problem is still not
>> > reproducible when you remove d_weak_revalidate?
>> 
>> I did try the reproducer and it works as expected both with and without
>> d_weak_revalidate.
>> On reflection, the problem it displayed was caused by d_revalidate()
>> being called when the dentry name was irrelevant.  We remove that
>> (fixing the problem) and introduce d_weak_revalidate because we thought
>> that minimum functionality was still useful.  I'm currently not
>> convinced that even that is needed.
>> 
>> If we discarded d_weak_revalidate(), we could get rid of the special
>> handling of umount....
>
> I like idea. I say go for it and we can see what (if anything) breaks?

Getting rid of d_weak_revalidate is easy enough - hardly any users.

Getting rid of filename_mountpoint() isn't so easy unfortunately.
autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
in autofs4_d_manage()?  It would be a shame to keep this infrastructure
around just so that one part of autofs4 can talk to another part of
autofs4.

Do you know if the fact that filename_mountpoint() skips ->d_manage is
important for sys_umount ??

Thanks,
NeilBrown


> -- 
> Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1712855

On Wed, 2017-08-16 at 12:43 +1000, NeilBrown wrote:
> On Mon, Aug 14 2017, Jeff Layton wrote:
> 
> > On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
> > > On Fri, Aug 11 2017, Jeff Layton wrote:
> > > 
> > > > On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
> > > > > On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
> > > > > > Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
> > > > > > flag and introduced the d_weak_revalidate dentry operation instead.
> > > > > > We duly removed the flag from NFS superblocks and NFSv4 superblocks,
> > > > > > and added the new dentry operation to NFS dentries .... but not to
> > > > > > NFSv4
> > > > > > dentries.
> > > > > > 
> > > > > > And nobody noticed.
> > > > > > 
> > > > > > Until today.
> > > > > > 
> > > > > > A customer reports a situation where mount(....,MS_REMOUNT,..) on an
> > > > > > NFS
> > > > > > filesystem hangs because the network has been deconfigured.  This
> > > > > > makes
> > > > > > perfect sense and I suggested a code change to fix the problem.
> > > > > > However when a colleague was trying to reproduce the problem to
> > > > > > validate
> > > > > > the fix, he couldn't.  Then nor could I.
> > > > > > 
> > > > > > The problem is trivially reproducible with NFSv3, and not at all with
> > > > > > NFSv4.  The reason is the missing d_weak_revalidate.
> > > > > > 
> > > > > > We could simply add d_weak_revalidate for NFSv4, but given that it
> > > > > > has been missing for 4.5 years, and the only time anyone noticed was
> > > > > > when the ommission resulted in a better user experience, I do wonder
> > > > > > if
> > > > > > we need to.  Can we just discard d_weak_revalidate?  What purpose
> > > > > > does
> > > > > > it serve?  I couldn't find one.
> > > > > > 
> > > > > > Thanks,
> > > > > > NeilBrown
> > > > > > 
> > > > > > For reference, see
> > > > > > Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
> > > > > > d_weak_revalidate dentry op")
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > To reproduce the problem at home, on a system that uses systemd:
> > > > > > 1/ place (or find) a filesystem image in a file on an NFS filesystem.
> > > > > > 2/ mount the nfs filesystem with "noac" - choose v3 or v4
> > > > > > 3/ loop-mount the filesystem image read-only somewhere
> > > > > > 4/ reboot
> > > > > > 
> > > > > > If you choose v4, the reboot will succeed, possibly after a 90second
> > > > > > timeout.
> > > > > > If you choose v3, the reboot will hang indefinitely in systemd-
> > > > > > shutdown while
> > > > > > remounting the nfs filesystem read-only.
> > > > > > 
> > > > > > If you don't use "noac" it can still hang, but only if something
> > > > > > slows
> > > > > > down the reboot enough that attributes have timed out by the time
> > > > > > that
> > > > > > systemd-shutdown runs.  This happens for our customer.
> > > > > > 
> > > > > > If the loop-mounted filesystem is not read-only, you get other
> > > > > > problems.
> > > > > > 
> > > > > > We really want systemd to figure out that the loop-mount needs to be
> > > > > > unmounted first.  I have ideas concerning that, but it is messy.  But
> > > > > > that isn't the only bug here.
> > > > > 
> > > > > The main purpose of d_weak_revalidate() was to catch the issues that
> > > > > arise when someone changes the contents of the current working
> > > > > directory or its parent on the server. Since '.' and '..' are treated
> > > > > specially in the lookup code, they would not be revalidated without
> > > > > special treatment. That leads to issues when looking up files as
> > > > > ./<filename> or ../<filename>, since the client won't detect that its
> > > > > dcache is stale until it tries to use the cached dentry+inode.
> > > > > 
> > > > > The one thing that has changed since its introduction is, I believe,
> > > > > the ESTALE handling in the VFS layer. That might fix a lot of the
> > > > > dcache lookup bugs that were previously handled by d_weak_revalidate().
> > > > > I haven't done an audit to figure out if it actually can handle all of
> > > > > them.
> > > > > 
> > > > 
> > > > It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
> > > > 
> > > >     vfs: allow umount to handle mountpoints without revalidating them
> > > 
> > > You say in the comment for that commit:
> > > 
> > >      but there
> > >     are cases where we do want to revalidate the root of the fs.
> > > 
> > > Do you happen to remember what those cases are?
> > > 
> > 
> > Not exactly, but I _think_ I might have been assuming that we needed to
> > ensure that the inode attrs on the root were up to date after the
> > pathwalk.
> > 
> > I think that was probably wrong. d_revalidate is really intended to
> > ensure that the dentry in question still points to the same inode. In
> > the case of the root of the mount though, we don't really care about the
> > dentry on the server at all. We're attaching the root of the mount to an
> > inode and don't care of the dentry name changes. If we do need to ensure
> > the inode attrs are updated, we'll just revalidate them at that point.
> > 
> > > > 
> > > > Possibly the fact that we no longer try to revalidate during unmount
> > > > means that this is no longer necessary?
> > > > 
> > > > The original patch that added d_weak_revalidate had a reproducer in the
> > > > patch description. Have you verified that that problem is still not
> > > > reproducible when you remove d_weak_revalidate?
> > > 
> > > I did try the reproducer and it works as expected both with and without
> > > d_weak_revalidate.
> > > On reflection, the problem it displayed was caused by d_revalidate()
> > > being called when the dentry name was irrelevant.  We remove that
> > > (fixing the problem) and introduce d_weak_revalidate because we thought
> > > that minimum functionality was still useful.  I'm currently not
> > > convinced that even that is needed.
> > > 
> > > If we discarded d_weak_revalidate(), we could get rid of the special
> > > handling of umount....
> > 
> > I like idea. I say go for it and we can see what (if anything) breaks?
> 
> Getting rid of d_weak_revalidate is easy enough - hardly any users.
> 
> Getting rid of filename_mountpoint() isn't so easy unfortunately.
> autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
> in autofs4_d_manage()?  It would be a shame to keep this infrastructure
> around just so that one part of autofs4 can talk to another part of
> autofs4.
> 
> Do you know if the fact that filename_mountpoint() skips ->d_manage is
> important for sys_umount ??
> 
> Thanks,
> NeilBrown
> 

(cc'ing David and Ian)

I'm less familiar with the automounting machinery, but I imagine you
don't really want to go triggering new mounts when your intent is to
unmount something. As long as that doesn't happen I'd think we'd be ok
here.
-- 
Jeff Layton <jlayton@redhat.com>

[toc] | [prev] | [next] | [standalone]

#1713416

[Multipart message — attachments visible in raw view] — view raw

On Wed, Aug 16 2017, Jeff Layton wrote:

> On Wed, 2017-08-16 at 12:43 +1000, NeilBrown wrote:
>> On Mon, Aug 14 2017, Jeff Layton wrote:
>> 
>> > On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
>> > > On Fri, Aug 11 2017, Jeff Layton wrote:
>> > > 
>> > > > On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>> > > > > On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>> > > > > > Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>> > > > > > flag and introduced the d_weak_revalidate dentry operation instead.
>> > > > > > We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>> > > > > > and added the new dentry operation to NFS dentries .... but not to
>> > > > > > NFSv4
>> > > > > > dentries.
>> > > > > > 
>> > > > > > And nobody noticed.
>> > > > > > 
>> > > > > > Until today.
>> > > > > > 
>> > > > > > A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>> > > > > > NFS
>> > > > > > filesystem hangs because the network has been deconfigured.  This
>> > > > > > makes
>> > > > > > perfect sense and I suggested a code change to fix the problem.
>> > > > > > However when a colleague was trying to reproduce the problem to
>> > > > > > validate
>> > > > > > the fix, he couldn't.  Then nor could I.
>> > > > > > 
>> > > > > > The problem is trivially reproducible with NFSv3, and not at all with
>> > > > > > NFSv4.  The reason is the missing d_weak_revalidate.
>> > > > > > 
>> > > > > > We could simply add d_weak_revalidate for NFSv4, but given that it
>> > > > > > has been missing for 4.5 years, and the only time anyone noticed was
>> > > > > > when the ommission resulted in a better user experience, I do wonder
>> > > > > > if
>> > > > > > we need to.  Can we just discard d_weak_revalidate?  What purpose
>> > > > > > does
>> > > > > > it serve?  I couldn't find one.
>> > > > > > 
>> > > > > > Thanks,
>> > > > > > NeilBrown
>> > > > > > 
>> > > > > > For reference, see
>> > > > > > Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>> > > > > > d_weak_revalidate dentry op")
>> > > > > > 
>> > > > > > 
>> > > > > > 
>> > > > > > To reproduce the problem at home, on a system that uses systemd:
>> > > > > > 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>> > > > > > 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>> > > > > > 3/ loop-mount the filesystem image read-only somewhere
>> > > > > > 4/ reboot
>> > > > > > 
>> > > > > > If you choose v4, the reboot will succeed, possibly after a 90second
>> > > > > > timeout.
>> > > > > > If you choose v3, the reboot will hang indefinitely in systemd-
>> > > > > > shutdown while
>> > > > > > remounting the nfs filesystem read-only.
>> > > > > > 
>> > > > > > If you don't use "noac" it can still hang, but only if something
>> > > > > > slows
>> > > > > > down the reboot enough that attributes have timed out by the time
>> > > > > > that
>> > > > > > systemd-shutdown runs.  This happens for our customer.
>> > > > > > 
>> > > > > > If the loop-mounted filesystem is not read-only, you get other
>> > > > > > problems.
>> > > > > > 
>> > > > > > We really want systemd to figure out that the loop-mount needs to be
>> > > > > > unmounted first.  I have ideas concerning that, but it is messy.  But
>> > > > > > that isn't the only bug here.
>> > > > > 
>> > > > > The main purpose of d_weak_revalidate() was to catch the issues that
>> > > > > arise when someone changes the contents of the current working
>> > > > > directory or its parent on the server. Since '.' and '..' are treated
>> > > > > specially in the lookup code, they would not be revalidated without
>> > > > > special treatment. That leads to issues when looking up files as
>> > > > > ./<filename> or ../<filename>, since the client won't detect that its
>> > > > > dcache is stale until it tries to use the cached dentry+inode.
>> > > > > 
>> > > > > The one thing that has changed since its introduction is, I believe,
>> > > > > the ESTALE handling in the VFS layer. That might fix a lot of the
>> > > > > dcache lookup bugs that were previously handled by d_weak_revalidate().
>> > > > > I haven't done an audit to figure out if it actually can handle all of
>> > > > > them.
>> > > > > 
>> > > > 
>> > > > It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>> > > > 
>> > > >     vfs: allow umount to handle mountpoints without revalidating them
>> > > 
>> > > You say in the comment for that commit:
>> > > 
>> > >      but there
>> > >     are cases where we do want to revalidate the root of the fs.
>> > > 
>> > > Do you happen to remember what those cases are?
>> > > 
>> > 
>> > Not exactly, but I _think_ I might have been assuming that we needed to
>> > ensure that the inode attrs on the root were up to date after the
>> > pathwalk.
>> > 
>> > I think that was probably wrong. d_revalidate is really intended to
>> > ensure that the dentry in question still points to the same inode. In
>> > the case of the root of the mount though, we don't really care about the
>> > dentry on the server at all. We're attaching the root of the mount to an
>> > inode and don't care of the dentry name changes. If we do need to ensure
>> > the inode attrs are updated, we'll just revalidate them at that point.
>> > 
>> > > > 
>> > > > Possibly the fact that we no longer try to revalidate during unmount
>> > > > means that this is no longer necessary?
>> > > > 
>> > > > The original patch that added d_weak_revalidate had a reproducer in the
>> > > > patch description. Have you verified that that problem is still not
>> > > > reproducible when you remove d_weak_revalidate?
>> > > 
>> > > I did try the reproducer and it works as expected both with and without
>> > > d_weak_revalidate.
>> > > On reflection, the problem it displayed was caused by d_revalidate()
>> > > being called when the dentry name was irrelevant.  We remove that
>> > > (fixing the problem) and introduce d_weak_revalidate because we thought
>> > > that minimum functionality was still useful.  I'm currently not
>> > > convinced that even that is needed.
>> > > 
>> > > If we discarded d_weak_revalidate(), we could get rid of the special
>> > > handling of umount....
>> > 
>> > I like idea. I say go for it and we can see what (if anything) breaks?
>> 
>> Getting rid of d_weak_revalidate is easy enough - hardly any users.
>> 
>> Getting rid of filename_mountpoint() isn't so easy unfortunately.
>> autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
>> in autofs4_d_manage()?  It would be a shame to keep this infrastructure
>> around just so that one part of autofs4 can talk to another part of
>> autofs4.
>> 
>> Do you know if the fact that filename_mountpoint() skips ->d_manage is
>> important for sys_umount ??
>> 
>> Thanks,
>> NeilBrown
>> 
>
> (cc'ing David and Ian)
>
> I'm less familiar with the automounting machinery, but I imagine you
> don't really want to go triggering new mounts when your intent is to
> unmount something. As long as that doesn't happen I'd think we'd be ok
> here.

New mounts don't get triggered unless asked for (LOOKUP_AUTOMOUNT) so
that isn't a problem.  But the d_manage() operation does get called, and
it can block.  I think the purpose of the interface is to avoid races
with expiring automounts.  Once the decision has been made to expire the
mount, you want to be able to stop anyone getting a new access.
d_manage() can do that.  umount() probably doesn't need d_manage() to be
skipped.

autofs *does* for certain ioctls which are used to re-establish
ownership of an automount (e.g. when restarting automountd).  The owner
never gets blocked, but if the new automountd isn't the owner yet, it
needs help.

The current code causes the lookup to never call d_manage(), which I
think is overkill.  There might be some other filesystem which uses
d_manage() (not actually possible at present) and those shouldn't be
ignored.

So I came up with this.  It isn't exactly elegant, but it should avoid
the deadlock that caused autofs4 to start using kern_path_mount().

If we apply this change, and get rid of d_weak_revalidate(), then I
think we can get rid of all the "mountpoint" pathname lookup code.

Ian/David - you do have any thoughts on this approach?

Thanks,
NeilBrown



diff --git a/fs/autofs4/autofs_i.h b/fs/autofs4/autofs_i.h
index beef981aa54f..478a9aafd6d3 100644
--- a/fs/autofs4/autofs_i.h
+++ b/fs/autofs4/autofs_i.h
@@ -271,3 +271,4 @@ static inline void autofs4_del_expiring(struct dentry *dentry)
 }
 
 void autofs4_kill_sb(struct super_block *);
+int is_autofs_finder(void);
diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
index dd9f1bebb5a3..f3e84bfed85a 100644
--- a/fs/autofs4/dev-ioctl.c
+++ b/fs/autofs4/dev-ioctl.c
@@ -200,6 +200,13 @@ static int autofs_dev_ioctl_protosubver(struct file *fp,
 	return 0;
 }
 
+static DEFINE_MUTEX(autofs_find);
+static struct task_struct *autofs_finder = NULL;
+int is_autofs_finder(void)
+{
+	return autofs_finder == current;
+}
+
 /* Find the topmost mount satisfying test() */
 static int find_autofs_mount(const char *pathname,
 			     struct path *res,
@@ -209,7 +216,12 @@ static int find_autofs_mount(const char *pathname,
 	struct path path;
 	int err;
 
-	err = kern_path_mountpoint(AT_FDCWD, pathname, &path, 0);
+	mutex_lock(&autofs_find);
+	autofs_finder = current;
+	err = kern_path(pathname, 0, &path);
+	autofs_finder = NULL;
+	mutex_unlock(&autofs_find);
+
 	if (err)
 		return err;
 	err = -ENOENT;
diff --git a/fs/autofs4/root.c b/fs/autofs4/root.c
index d79ced925861..e93a3112d60c 100644
--- a/fs/autofs4/root.c
+++ b/fs/autofs4/root.c
@@ -438,7 +438,7 @@ static int autofs4_d_manage(const struct path *path, bool rcu_walk)
 	pr_debug("dentry=%p %pd\n", dentry, dentry);
 
 	/* The daemon never waits. */
-	if (autofs4_oz_mode(sbi)) {
+	if (autofs4_oz_mode(sbi) || is_autofs_finder()) {
 		if (!path_is_mountpoint(path))
 			return -EISDIR;
 		return 0;

[toc] | [prev] | [next] | [standalone]

#1713482

On 16/08/17 19:34, Jeff Layton wrote:
> On Wed, 2017-08-16 at 12:43 +1000, NeilBrown wrote:
>> On Mon, Aug 14 2017, Jeff Layton wrote:
>>
>>> On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
>>>> On Fri, Aug 11 2017, Jeff Layton wrote:
>>>>
>>>>> On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>>>>>> On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>>>>>>> Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>>>>>>> flag and introduced the d_weak_revalidate dentry operation instead.
>>>>>>> We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>>>>>>> and added the new dentry operation to NFS dentries .... but not to
>>>>>>> NFSv4
>>>>>>> dentries.
>>>>>>>
>>>>>>> And nobody noticed.
>>>>>>>
>>>>>>> Until today.
>>>>>>>
>>>>>>> A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>>>>>>> NFS
>>>>>>> filesystem hangs because the network has been deconfigured.  This
>>>>>>> makes
>>>>>>> perfect sense and I suggested a code change to fix the problem.
>>>>>>> However when a colleague was trying to reproduce the problem to
>>>>>>> validate
>>>>>>> the fix, he couldn't.  Then nor could I.
>>>>>>>
>>>>>>> The problem is trivially reproducible with NFSv3, and not at all with
>>>>>>> NFSv4.  The reason is the missing d_weak_revalidate.
>>>>>>>
>>>>>>> We could simply add d_weak_revalidate for NFSv4, but given that it
>>>>>>> has been missing for 4.5 years, and the only time anyone noticed was
>>>>>>> when the ommission resulted in a better user experience, I do wonder
>>>>>>> if
>>>>>>> we need to.  Can we just discard d_weak_revalidate?  What purpose
>>>>>>> does
>>>>>>> it serve?  I couldn't find one.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> NeilBrown
>>>>>>>
>>>>>>> For reference, see
>>>>>>> Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>>>>>>> d_weak_revalidate dentry op")
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> To reproduce the problem at home, on a system that uses systemd:
>>>>>>> 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>>>>>>> 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>>>>>>> 3/ loop-mount the filesystem image read-only somewhere
>>>>>>> 4/ reboot
>>>>>>>
>>>>>>> If you choose v4, the reboot will succeed, possibly after a 90second
>>>>>>> timeout.
>>>>>>> If you choose v3, the reboot will hang indefinitely in systemd-
>>>>>>> shutdown while
>>>>>>> remounting the nfs filesystem read-only.
>>>>>>>
>>>>>>> If you don't use "noac" it can still hang, but only if something
>>>>>>> slows
>>>>>>> down the reboot enough that attributes have timed out by the time
>>>>>>> that
>>>>>>> systemd-shutdown runs.  This happens for our customer.
>>>>>>>
>>>>>>> If the loop-mounted filesystem is not read-only, you get other
>>>>>>> problems.
>>>>>>>
>>>>>>> We really want systemd to figure out that the loop-mount needs to be
>>>>>>> unmounted first.  I have ideas concerning that, but it is messy.  But
>>>>>>> that isn't the only bug here.
>>>>>>
>>>>>> The main purpose of d_weak_revalidate() was to catch the issues that
>>>>>> arise when someone changes the contents of the current working
>>>>>> directory or its parent on the server. Since '.' and '..' are treated
>>>>>> specially in the lookup code, they would not be revalidated without
>>>>>> special treatment. That leads to issues when looking up files as
>>>>>> ./<filename> or ../<filename>, since the client won't detect that its
>>>>>> dcache is stale until it tries to use the cached dentry+inode.
>>>>>>
>>>>>> The one thing that has changed since its introduction is, I believe,
>>>>>> the ESTALE handling in the VFS layer. That might fix a lot of the
>>>>>> dcache lookup bugs that were previously handled by d_weak_revalidate().
>>>>>> I haven't done an audit to figure out if it actually can handle all of
>>>>>> them.
>>>>>>
>>>>>
>>>>> It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>>>>>
>>>>>     vfs: allow umount to handle mountpoints without revalidating them
>>>>
>>>> You say in the comment for that commit:
>>>>
>>>>      but there
>>>>     are cases where we do want to revalidate the root of the fs.
>>>>
>>>> Do you happen to remember what those cases are?
>>>>
>>>
>>> Not exactly, but I _think_ I might have been assuming that we needed to
>>> ensure that the inode attrs on the root were up to date after the
>>> pathwalk.
>>>
>>> I think that was probably wrong. d_revalidate is really intended to
>>> ensure that the dentry in question still points to the same inode. In
>>> the case of the root of the mount though, we don't really care about the
>>> dentry on the server at all. We're attaching the root of the mount to an
>>> inode and don't care of the dentry name changes. If we do need to ensure
>>> the inode attrs are updated, we'll just revalidate them at that point.
>>>
>>>>>
>>>>> Possibly the fact that we no longer try to revalidate during unmount
>>>>> means that this is no longer necessary?
>>>>>
>>>>> The original patch that added d_weak_revalidate had a reproducer in the
>>>>> patch description. Have you verified that that problem is still not
>>>>> reproducible when you remove d_weak_revalidate?
>>>>
>>>> I did try the reproducer and it works as expected both with and without
>>>> d_weak_revalidate.
>>>> On reflection, the problem it displayed was caused by d_revalidate()
>>>> being called when the dentry name was irrelevant.  We remove that
>>>> (fixing the problem) and introduce d_weak_revalidate because we thought
>>>> that minimum functionality was still useful.  I'm currently not
>>>> convinced that even that is needed.
>>>>
>>>> If we discarded d_weak_revalidate(), we could get rid of the special
>>>> handling of umount....
>>>
>>> I like idea. I say go for it and we can see what (if anything) breaks?
>>
>> Getting rid of d_weak_revalidate is easy enough - hardly any users.
>>
>> Getting rid of filename_mountpoint() isn't so easy unfortunately.
>> autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
>> in autofs4_d_manage()?  It would be a shame to keep this infrastructure
>> around just so that one part of autofs4 can talk to another part of
>> autofs4.

When this was first implemented autofs didn't use kern_path_mountpoint()
(it didn't exist) it used a path lookup on the parent and a separate
lookup for the last component.

It's used for two operations, first to open a file handle to a (possibly)
covered autofs mount, and second to get mounted information about a path
without following past a (possibly covered) autofs mount.

It's less about not triggering an automount or getting stuck in ->d_manage()
and more about resolving paths that are not accessible via normal vfs walks.

I never thought about re-validation for either of these cases and altering
it to the way it was before filename_mountpoint() shouldn't be a problem.

>>
>> Do you know if the fact that filename_mountpoint() skips ->d_manage is
>> important for sys_umount ??

That's a lot harder to answer, d_manage() is needed by autofs to detect
if a path walk needs to block because of some concurrent activity, usually
to avoid concurrent expire (indirectly a umount) to mount races. But it
isn't the umount it's the autofs expire that sets the flag used to block
walks (sends it to the ref-walk path), the (potential) umount comes later
as a result of the expire and the flag is cleared after expire completion.

I can talk about why ->d_manage() isn't (shouldn't) be needed for other vfs
automount users such as NFS etc. if needed.

So filename_mountpoint() skiping ->d_manage shouldn't matter to autofs at
all. If it goes away then autofs will need to revert to an alternate way
(similar to the way it was originally done) of doing the lookup.

>>
>> Thanks,
>> NeilBrown
>>
> 
> (cc'ing David and Ian)
> 
> I'm less familiar with the automounting machinery, but I imagine you
> don't really want to go triggering new mounts when your intent is to
> unmount something. As long as that doesn't happen I'd think we'd be ok
> here.

Ian

[toc] | [prev] | [next] | [standalone]

#1714578

[Multipart message — attachments visible in raw view] — view raw

On Thu, Aug 17 2017, Ian Kent wrote:

> On 16/08/17 19:34, Jeff Layton wrote:
>> On Wed, 2017-08-16 at 12:43 +1000, NeilBrown wrote:
>>> On Mon, Aug 14 2017, Jeff Layton wrote:
>>>
>>>> On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
>>>>> On Fri, Aug 11 2017, Jeff Layton wrote:
>>>>>
>>>>>> On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>>>>>>> On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>>>>>>>> Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>>>>>>>> flag and introduced the d_weak_revalidate dentry operation instead.
>>>>>>>> We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>>>>>>>> and added the new dentry operation to NFS dentries .... but not to
>>>>>>>> NFSv4
>>>>>>>> dentries.
>>>>>>>>
>>>>>>>> And nobody noticed.
>>>>>>>>
>>>>>>>> Until today.
>>>>>>>>
>>>>>>>> A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>>>>>>>> NFS
>>>>>>>> filesystem hangs because the network has been deconfigured.  This
>>>>>>>> makes
>>>>>>>> perfect sense and I suggested a code change to fix the problem.
>>>>>>>> However when a colleague was trying to reproduce the problem to
>>>>>>>> validate
>>>>>>>> the fix, he couldn't.  Then nor could I.
>>>>>>>>
>>>>>>>> The problem is trivially reproducible with NFSv3, and not at all with
>>>>>>>> NFSv4.  The reason is the missing d_weak_revalidate.
>>>>>>>>
>>>>>>>> We could simply add d_weak_revalidate for NFSv4, but given that it
>>>>>>>> has been missing for 4.5 years, and the only time anyone noticed was
>>>>>>>> when the ommission resulted in a better user experience, I do wonder
>>>>>>>> if
>>>>>>>> we need to.  Can we just discard d_weak_revalidate?  What purpose
>>>>>>>> does
>>>>>>>> it serve?  I couldn't find one.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> NeilBrown
>>>>>>>>
>>>>>>>> For reference, see
>>>>>>>> Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>>>>>>>> d_weak_revalidate dentry op")
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> To reproduce the problem at home, on a system that uses systemd:
>>>>>>>> 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>>>>>>>> 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>>>>>>>> 3/ loop-mount the filesystem image read-only somewhere
>>>>>>>> 4/ reboot
>>>>>>>>
>>>>>>>> If you choose v4, the reboot will succeed, possibly after a 90second
>>>>>>>> timeout.
>>>>>>>> If you choose v3, the reboot will hang indefinitely in systemd-
>>>>>>>> shutdown while
>>>>>>>> remounting the nfs filesystem read-only.
>>>>>>>>
>>>>>>>> If you don't use "noac" it can still hang, but only if something
>>>>>>>> slows
>>>>>>>> down the reboot enough that attributes have timed out by the time
>>>>>>>> that
>>>>>>>> systemd-shutdown runs.  This happens for our customer.
>>>>>>>>
>>>>>>>> If the loop-mounted filesystem is not read-only, you get other
>>>>>>>> problems.
>>>>>>>>
>>>>>>>> We really want systemd to figure out that the loop-mount needs to be
>>>>>>>> unmounted first.  I have ideas concerning that, but it is messy.  But
>>>>>>>> that isn't the only bug here.
>>>>>>>
>>>>>>> The main purpose of d_weak_revalidate() was to catch the issues that
>>>>>>> arise when someone changes the contents of the current working
>>>>>>> directory or its parent on the server. Since '.' and '..' are treated
>>>>>>> specially in the lookup code, they would not be revalidated without
>>>>>>> special treatment. That leads to issues when looking up files as
>>>>>>> ./<filename> or ../<filename>, since the client won't detect that its
>>>>>>> dcache is stale until it tries to use the cached dentry+inode.
>>>>>>>
>>>>>>> The one thing that has changed since its introduction is, I believe,
>>>>>>> the ESTALE handling in the VFS layer. That might fix a lot of the
>>>>>>> dcache lookup bugs that were previously handled by d_weak_revalidate().
>>>>>>> I haven't done an audit to figure out if it actually can handle all of
>>>>>>> them.
>>>>>>>
>>>>>>
>>>>>> It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>>>>>>
>>>>>>     vfs: allow umount to handle mountpoints without revalidating them
>>>>>
>>>>> You say in the comment for that commit:
>>>>>
>>>>>      but there
>>>>>     are cases where we do want to revalidate the root of the fs.
>>>>>
>>>>> Do you happen to remember what those cases are?
>>>>>
>>>>
>>>> Not exactly, but I _think_ I might have been assuming that we needed to
>>>> ensure that the inode attrs on the root were up to date after the
>>>> pathwalk.
>>>>
>>>> I think that was probably wrong. d_revalidate is really intended to
>>>> ensure that the dentry in question still points to the same inode. In
>>>> the case of the root of the mount though, we don't really care about the
>>>> dentry on the server at all. We're attaching the root of the mount to an
>>>> inode and don't care of the dentry name changes. If we do need to ensure
>>>> the inode attrs are updated, we'll just revalidate them at that point.
>>>>
>>>>>>
>>>>>> Possibly the fact that we no longer try to revalidate during unmount
>>>>>> means that this is no longer necessary?
>>>>>>
>>>>>> The original patch that added d_weak_revalidate had a reproducer in the
>>>>>> patch description. Have you verified that that problem is still not
>>>>>> reproducible when you remove d_weak_revalidate?
>>>>>
>>>>> I did try the reproducer and it works as expected both with and without
>>>>> d_weak_revalidate.
>>>>> On reflection, the problem it displayed was caused by d_revalidate()
>>>>> being called when the dentry name was irrelevant.  We remove that
>>>>> (fixing the problem) and introduce d_weak_revalidate because we thought
>>>>> that minimum functionality was still useful.  I'm currently not
>>>>> convinced that even that is needed.
>>>>>
>>>>> If we discarded d_weak_revalidate(), we could get rid of the special
>>>>> handling of umount....
>>>>
>>>> I like idea. I say go for it and we can see what (if anything) breaks?
>>>
>>> Getting rid of d_weak_revalidate is easy enough - hardly any users.
>>>
>>> Getting rid of filename_mountpoint() isn't so easy unfortunately.
>>> autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
>>> in autofs4_d_manage()?  It would be a shame to keep this infrastructure
>>> around just so that one part of autofs4 can talk to another part of
>>> autofs4.
>
> When this was first implemented autofs didn't use kern_path_mountpoint()
> (it didn't exist) it used a path lookup on the parent and a separate
> lookup for the last component.

This was before commit 4e44b6852e03 ("Get rid of path_lookup in
autofs4").  This used kern_path().

I'm more interested in commit ac8387199656 ("autofs4 - fix device ioctl
mount lookup")  which replaced the use of kern_path() with
kern_path_mountpoint().

>
> It's used for two operations, first to open a file handle to a (possibly)
> covered autofs mount, and second to get mounted information about a path
> without following past a (possibly covered) autofs mount.
>
> It's less about not triggering an automount or getting stuck in ->d_manage()
> and more about resolving paths that are not accessible via normal vfs walks.
>
> I never thought about re-validation for either of these cases and altering
> it to the way it was before filename_mountpoint() shouldn't be a
> problem.

If it shouldn't be a problem, what justified ac8387199656??

Thanks,
NeilBrown

[toc] | [prev] | [next] | [standalone]

#1714612

On 18/08/17 13:24, NeilBrown wrote:
> On Thu, Aug 17 2017, Ian Kent wrote:
> 
>> On 16/08/17 19:34, Jeff Layton wrote:
>>> On Wed, 2017-08-16 at 12:43 +1000, NeilBrown wrote:
>>>> On Mon, Aug 14 2017, Jeff Layton wrote:
>>>>
>>>>> On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
>>>>>> On Fri, Aug 11 2017, Jeff Layton wrote:
>>>>>>
>>>>>>> On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>>>>>>>> On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>>>>>>>>> Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>>>>>>>>> flag and introduced the d_weak_revalidate dentry operation instead.
>>>>>>>>> We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>>>>>>>>> and added the new dentry operation to NFS dentries .... but not to
>>>>>>>>> NFSv4
>>>>>>>>> dentries.
>>>>>>>>>
>>>>>>>>> And nobody noticed.
>>>>>>>>>
>>>>>>>>> Until today.
>>>>>>>>>
>>>>>>>>> A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>>>>>>>>> NFS
>>>>>>>>> filesystem hangs because the network has been deconfigured.  This
>>>>>>>>> makes
>>>>>>>>> perfect sense and I suggested a code change to fix the problem.
>>>>>>>>> However when a colleague was trying to reproduce the problem to
>>>>>>>>> validate
>>>>>>>>> the fix, he couldn't.  Then nor could I.
>>>>>>>>>
>>>>>>>>> The problem is trivially reproducible with NFSv3, and not at all with
>>>>>>>>> NFSv4.  The reason is the missing d_weak_revalidate.
>>>>>>>>>
>>>>>>>>> We could simply add d_weak_revalidate for NFSv4, but given that it
>>>>>>>>> has been missing for 4.5 years, and the only time anyone noticed was
>>>>>>>>> when the ommission resulted in a better user experience, I do wonder
>>>>>>>>> if
>>>>>>>>> we need to.  Can we just discard d_weak_revalidate?  What purpose
>>>>>>>>> does
>>>>>>>>> it serve?  I couldn't find one.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> NeilBrown
>>>>>>>>>
>>>>>>>>> For reference, see
>>>>>>>>> Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>>>>>>>>> d_weak_revalidate dentry op")
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> To reproduce the problem at home, on a system that uses systemd:
>>>>>>>>> 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>>>>>>>>> 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>>>>>>>>> 3/ loop-mount the filesystem image read-only somewhere
>>>>>>>>> 4/ reboot
>>>>>>>>>
>>>>>>>>> If you choose v4, the reboot will succeed, possibly after a 90second
>>>>>>>>> timeout.
>>>>>>>>> If you choose v3, the reboot will hang indefinitely in systemd-
>>>>>>>>> shutdown while
>>>>>>>>> remounting the nfs filesystem read-only.
>>>>>>>>>
>>>>>>>>> If you don't use "noac" it can still hang, but only if something
>>>>>>>>> slows
>>>>>>>>> down the reboot enough that attributes have timed out by the time
>>>>>>>>> that
>>>>>>>>> systemd-shutdown runs.  This happens for our customer.
>>>>>>>>>
>>>>>>>>> If the loop-mounted filesystem is not read-only, you get other
>>>>>>>>> problems.
>>>>>>>>>
>>>>>>>>> We really want systemd to figure out that the loop-mount needs to be
>>>>>>>>> unmounted first.  I have ideas concerning that, but it is messy.  But
>>>>>>>>> that isn't the only bug here.
>>>>>>>>
>>>>>>>> The main purpose of d_weak_revalidate() was to catch the issues that
>>>>>>>> arise when someone changes the contents of the current working
>>>>>>>> directory or its parent on the server. Since '.' and '..' are treated
>>>>>>>> specially in the lookup code, they would not be revalidated without
>>>>>>>> special treatment. That leads to issues when looking up files as
>>>>>>>> ./<filename> or ../<filename>, since the client won't detect that its
>>>>>>>> dcache is stale until it tries to use the cached dentry+inode.
>>>>>>>>
>>>>>>>> The one thing that has changed since its introduction is, I believe,
>>>>>>>> the ESTALE handling in the VFS layer. That might fix a lot of the
>>>>>>>> dcache lookup bugs that were previously handled by d_weak_revalidate().
>>>>>>>> I haven't done an audit to figure out if it actually can handle all of
>>>>>>>> them.
>>>>>>>>
>>>>>>>
>>>>>>> It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>>>>>>>
>>>>>>>     vfs: allow umount to handle mountpoints without revalidating them
>>>>>>
>>>>>> You say in the comment for that commit:
>>>>>>
>>>>>>      but there
>>>>>>     are cases where we do want to revalidate the root of the fs.
>>>>>>
>>>>>> Do you happen to remember what those cases are?
>>>>>>
>>>>>
>>>>> Not exactly, but I _think_ I might have been assuming that we needed to
>>>>> ensure that the inode attrs on the root were up to date after the
>>>>> pathwalk.
>>>>>
>>>>> I think that was probably wrong. d_revalidate is really intended to
>>>>> ensure that the dentry in question still points to the same inode. In
>>>>> the case of the root of the mount though, we don't really care about the
>>>>> dentry on the server at all. We're attaching the root of the mount to an
>>>>> inode and don't care of the dentry name changes. If we do need to ensure
>>>>> the inode attrs are updated, we'll just revalidate them at that point.
>>>>>
>>>>>>>
>>>>>>> Possibly the fact that we no longer try to revalidate during unmount
>>>>>>> means that this is no longer necessary?
>>>>>>>
>>>>>>> The original patch that added d_weak_revalidate had a reproducer in the
>>>>>>> patch description. Have you verified that that problem is still not
>>>>>>> reproducible when you remove d_weak_revalidate?
>>>>>>
>>>>>> I did try the reproducer and it works as expected both with and without
>>>>>> d_weak_revalidate.
>>>>>> On reflection, the problem it displayed was caused by d_revalidate()
>>>>>> being called when the dentry name was irrelevant.  We remove that
>>>>>> (fixing the problem) and introduce d_weak_revalidate because we thought
>>>>>> that minimum functionality was still useful.  I'm currently not
>>>>>> convinced that even that is needed.
>>>>>>
>>>>>> If we discarded d_weak_revalidate(), we could get rid of the special
>>>>>> handling of umount....
>>>>>
>>>>> I like idea. I say go for it and we can see what (if anything) breaks?
>>>>
>>>> Getting rid of d_weak_revalidate is easy enough - hardly any users.
>>>>
>>>> Getting rid of filename_mountpoint() isn't so easy unfortunately.
>>>> autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
>>>> in autofs4_d_manage()?  It would be a shame to keep this infrastructure
>>>> around just so that one part of autofs4 can talk to another part of
>>>> autofs4.
>>
>> When this was first implemented autofs didn't use kern_path_mountpoint()
>> (it didn't exist) it used a path lookup on the parent and a separate
>> lookup for the last component.
> 
> This was before commit 4e44b6852e03 ("Get rid of path_lookup in
> autofs4").  This used kern_path().

I have to plead not guilty of this one.

IIRC it broke the requirement of "lookup parent then lookup last component"
rather it walked the whole path then followed up to find the mount point
struct path.

Like it says in the description of ac8387199656 the caller might not yet
"own" the autofs mount which causes a mount to be triggered during the
walk that can't be satisfied because of the deadlock that occurs.

> 
> I'm more interested in commit ac8387199656 ("autofs4 - fix device ioctl
> mount lookup")  which replaced the use of kern_path() with
> kern_path_mountpoint().

Probably should have had a Fixes: 4e44b6852e03 ...

> 
>>
>> It's used for two operations, first to open a file handle to a (possibly)
>> covered autofs mount, and second to get mounted information about a path
>> without following past a (possibly covered) autofs mount.
>>
>> It's less about not triggering an automount or getting stuck in ->d_manage()
>> and more about resolving paths that are not accessible via normal vfs walks.
>>
>> I never thought about re-validation for either of these cases and altering
>> it to the way it was before filename_mountpoint() shouldn't be a
>> problem.
> 
> If it shouldn't be a problem, what justified ac8387199656??
> 
> Thanks,
> NeilBrown
> 

[toc] | [prev] | [next] | [standalone]

#1714618

On 18/08/17 14:47, Ian Kent wrote:
> On 18/08/17 13:24, NeilBrown wrote:
>> On Thu, Aug 17 2017, Ian Kent wrote:
>>
>>> On 16/08/17 19:34, Jeff Layton wrote:
>>>> On Wed, 2017-08-16 at 12:43 +1000, NeilBrown wrote:
>>>>> On Mon, Aug 14 2017, Jeff Layton wrote:
>>>>>
>>>>>> On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
>>>>>>> On Fri, Aug 11 2017, Jeff Layton wrote:
>>>>>>>
>>>>>>>> On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>>>>>>>>> On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>>>>>>>>>> Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>>>>>>>>>> flag and introduced the d_weak_revalidate dentry operation instead.
>>>>>>>>>> We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>>>>>>>>>> and added the new dentry operation to NFS dentries .... but not to
>>>>>>>>>> NFSv4
>>>>>>>>>> dentries.
>>>>>>>>>>
>>>>>>>>>> And nobody noticed.
>>>>>>>>>>
>>>>>>>>>> Until today.
>>>>>>>>>>
>>>>>>>>>> A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>>>>>>>>>> NFS
>>>>>>>>>> filesystem hangs because the network has been deconfigured.  This
>>>>>>>>>> makes
>>>>>>>>>> perfect sense and I suggested a code change to fix the problem.
>>>>>>>>>> However when a colleague was trying to reproduce the problem to
>>>>>>>>>> validate
>>>>>>>>>> the fix, he couldn't.  Then nor could I.
>>>>>>>>>>
>>>>>>>>>> The problem is trivially reproducible with NFSv3, and not at all with
>>>>>>>>>> NFSv4.  The reason is the missing d_weak_revalidate.
>>>>>>>>>>
>>>>>>>>>> We could simply add d_weak_revalidate for NFSv4, but given that it
>>>>>>>>>> has been missing for 4.5 years, and the only time anyone noticed was
>>>>>>>>>> when the ommission resulted in a better user experience, I do wonder
>>>>>>>>>> if
>>>>>>>>>> we need to.  Can we just discard d_weak_revalidate?  What purpose
>>>>>>>>>> does
>>>>>>>>>> it serve?  I couldn't find one.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> NeilBrown
>>>>>>>>>>
>>>>>>>>>> For reference, see
>>>>>>>>>> Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>>>>>>>>>> d_weak_revalidate dentry op")
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> To reproduce the problem at home, on a system that uses systemd:
>>>>>>>>>> 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>>>>>>>>>> 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>>>>>>>>>> 3/ loop-mount the filesystem image read-only somewhere
>>>>>>>>>> 4/ reboot
>>>>>>>>>>
>>>>>>>>>> If you choose v4, the reboot will succeed, possibly after a 90second
>>>>>>>>>> timeout.
>>>>>>>>>> If you choose v3, the reboot will hang indefinitely in systemd-
>>>>>>>>>> shutdown while
>>>>>>>>>> remounting the nfs filesystem read-only.
>>>>>>>>>>
>>>>>>>>>> If you don't use "noac" it can still hang, but only if something
>>>>>>>>>> slows
>>>>>>>>>> down the reboot enough that attributes have timed out by the time
>>>>>>>>>> that
>>>>>>>>>> systemd-shutdown runs.  This happens for our customer.
>>>>>>>>>>
>>>>>>>>>> If the loop-mounted filesystem is not read-only, you get other
>>>>>>>>>> problems.
>>>>>>>>>>
>>>>>>>>>> We really want systemd to figure out that the loop-mount needs to be
>>>>>>>>>> unmounted first.  I have ideas concerning that, but it is messy.  But
>>>>>>>>>> that isn't the only bug here.
>>>>>>>>>
>>>>>>>>> The main purpose of d_weak_revalidate() was to catch the issues that
>>>>>>>>> arise when someone changes the contents of the current working
>>>>>>>>> directory or its parent on the server. Since '.' and '..' are treated
>>>>>>>>> specially in the lookup code, they would not be revalidated without
>>>>>>>>> special treatment. That leads to issues when looking up files as
>>>>>>>>> ./<filename> or ../<filename>, since the client won't detect that its
>>>>>>>>> dcache is stale until it tries to use the cached dentry+inode.
>>>>>>>>>
>>>>>>>>> The one thing that has changed since its introduction is, I believe,
>>>>>>>>> the ESTALE handling in the VFS layer. That might fix a lot of the
>>>>>>>>> dcache lookup bugs that were previously handled by d_weak_revalidate().
>>>>>>>>> I haven't done an audit to figure out if it actually can handle all of
>>>>>>>>> them.
>>>>>>>>>
>>>>>>>>
>>>>>>>> It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>>>>>>>>
>>>>>>>>     vfs: allow umount to handle mountpoints without revalidating them
>>>>>>>
>>>>>>> You say in the comment for that commit:
>>>>>>>
>>>>>>>      but there
>>>>>>>     are cases where we do want to revalidate the root of the fs.
>>>>>>>
>>>>>>> Do you happen to remember what those cases are?
>>>>>>>
>>>>>>
>>>>>> Not exactly, but I _think_ I might have been assuming that we needed to
>>>>>> ensure that the inode attrs on the root were up to date after the
>>>>>> pathwalk.
>>>>>>
>>>>>> I think that was probably wrong. d_revalidate is really intended to
>>>>>> ensure that the dentry in question still points to the same inode. In
>>>>>> the case of the root of the mount though, we don't really care about the
>>>>>> dentry on the server at all. We're attaching the root of the mount to an
>>>>>> inode and don't care of the dentry name changes. If we do need to ensure
>>>>>> the inode attrs are updated, we'll just revalidate them at that point.
>>>>>>
>>>>>>>>
>>>>>>>> Possibly the fact that we no longer try to revalidate during unmount
>>>>>>>> means that this is no longer necessary?
>>>>>>>>
>>>>>>>> The original patch that added d_weak_revalidate had a reproducer in the
>>>>>>>> patch description. Have you verified that that problem is still not
>>>>>>>> reproducible when you remove d_weak_revalidate?
>>>>>>>
>>>>>>> I did try the reproducer and it works as expected both with and without
>>>>>>> d_weak_revalidate.
>>>>>>> On reflection, the problem it displayed was caused by d_revalidate()
>>>>>>> being called when the dentry name was irrelevant.  We remove that
>>>>>>> (fixing the problem) and introduce d_weak_revalidate because we thought
>>>>>>> that minimum functionality was still useful.  I'm currently not
>>>>>>> convinced that even that is needed.
>>>>>>>
>>>>>>> If we discarded d_weak_revalidate(), we could get rid of the special
>>>>>>> handling of umount....
>>>>>>
>>>>>> I like idea. I say go for it and we can see what (if anything) breaks?
>>>>>
>>>>> Getting rid of d_weak_revalidate is easy enough - hardly any users.
>>>>>
>>>>> Getting rid of filename_mountpoint() isn't so easy unfortunately.
>>>>> autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
>>>>> in autofs4_d_manage()?  It would be a shame to keep this infrastructure
>>>>> around just so that one part of autofs4 can talk to another part of
>>>>> autofs4.
>>>
>>> When this was first implemented autofs didn't use kern_path_mountpoint()
>>> (it didn't exist) it used a path lookup on the parent and a separate
>>> lookup for the last component.
>>
>> This was before commit 4e44b6852e03 ("Get rid of path_lookup in
>> autofs4").  This used kern_path().
> 
> I have to plead not guilty of this one.
> 
> IIRC it broke the requirement of "lookup parent then lookup last component"
> rather it walked the whole path then followed up to find the mount point
> struct path.
> 
> Like it says in the description of ac8387199656 the caller might not yet
> "own" the autofs mount which causes a mount to be triggered during the
> walk that can't be satisfied because of the deadlock that occurs.

Also, 4e44b6852e03 fixed another mistake I had made with:

-               if (nd.path.mnt->mnt_mountpoint != nd.path.mnt->mnt_root) {
-                       if (follow_down(&nd.path.mnt, &nd.path.dentry)) {
-                               struct inode *inode = nd.path.dentry->d_inode;
-                               magic = inode->i_sb->s_magic;
-                       }
+               if (path.mnt->mnt_mountpoint != path.mnt->mnt_root) {
+                       if (follow_down(&path.mnt, &path.dentry))
+                               magic = path.mnt->mnt_sb->s_magic;
                }

> 
>>
>> I'm more interested in commit ac8387199656 ("autofs4 - fix device ioctl
>> mount lookup")  which replaced the use of kern_path() with
>> kern_path_mountpoint().
> 
> Probably should have had a Fixes: 4e44b6852e03 ...
> 
>>
>>>
>>> It's used for two operations, first to open a file handle to a (possibly)
>>> covered autofs mount, and second to get mounted information about a path
>>> without following past a (possibly covered) autofs mount.
>>>
>>> It's less about not triggering an automount or getting stuck in ->d_manage()
>>> and more about resolving paths that are not accessible via normal vfs walks.
>>>
>>> I never thought about re-validation for either of these cases and altering
>>> it to the way it was before filename_mountpoint() shouldn't be a
>>> problem.
>>
>> If it shouldn't be a problem, what justified ac8387199656??
>>
>> Thanks,
>> NeilBrown
>>
> 

[toc] | [prev] | [next] | [standalone]

#1716111

[Multipart message — attachments visible in raw view] — view raw

On Fri, Aug 18 2017, Ian Kent wrote:

> On 18/08/17 13:24, NeilBrown wrote:
>> On Thu, Aug 17 2017, Ian Kent wrote:
>> 
>>> On 16/08/17 19:34, Jeff Layton wrote:
>>>> On Wed, 2017-08-16 at 12:43 +1000, NeilBrown wrote:
>>>>> On Mon, Aug 14 2017, Jeff Layton wrote:
>>>>>
>>>>>> On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
>>>>>>> On Fri, Aug 11 2017, Jeff Layton wrote:
>>>>>>>
>>>>>>>> On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>>>>>>>>> On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>>>>>>>>>> Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>>>>>>>>>> flag and introduced the d_weak_revalidate dentry operation instead.
>>>>>>>>>> We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>>>>>>>>>> and added the new dentry operation to NFS dentries .... but not to
>>>>>>>>>> NFSv4
>>>>>>>>>> dentries.
>>>>>>>>>>
>>>>>>>>>> And nobody noticed.
>>>>>>>>>>
>>>>>>>>>> Until today.
>>>>>>>>>>
>>>>>>>>>> A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>>>>>>>>>> NFS
>>>>>>>>>> filesystem hangs because the network has been deconfigured.  This
>>>>>>>>>> makes
>>>>>>>>>> perfect sense and I suggested a code change to fix the problem.
>>>>>>>>>> However when a colleague was trying to reproduce the problem to
>>>>>>>>>> validate
>>>>>>>>>> the fix, he couldn't.  Then nor could I.
>>>>>>>>>>
>>>>>>>>>> The problem is trivially reproducible with NFSv3, and not at all with
>>>>>>>>>> NFSv4.  The reason is the missing d_weak_revalidate.
>>>>>>>>>>
>>>>>>>>>> We could simply add d_weak_revalidate for NFSv4, but given that it
>>>>>>>>>> has been missing for 4.5 years, and the only time anyone noticed was
>>>>>>>>>> when the ommission resulted in a better user experience, I do wonder
>>>>>>>>>> if
>>>>>>>>>> we need to.  Can we just discard d_weak_revalidate?  What purpose
>>>>>>>>>> does
>>>>>>>>>> it serve?  I couldn't find one.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> NeilBrown
>>>>>>>>>>
>>>>>>>>>> For reference, see
>>>>>>>>>> Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>>>>>>>>>> d_weak_revalidate dentry op")
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> To reproduce the problem at home, on a system that uses systemd:
>>>>>>>>>> 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>>>>>>>>>> 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>>>>>>>>>> 3/ loop-mount the filesystem image read-only somewhere
>>>>>>>>>> 4/ reboot
>>>>>>>>>>
>>>>>>>>>> If you choose v4, the reboot will succeed, possibly after a 90second
>>>>>>>>>> timeout.
>>>>>>>>>> If you choose v3, the reboot will hang indefinitely in systemd-
>>>>>>>>>> shutdown while
>>>>>>>>>> remounting the nfs filesystem read-only.
>>>>>>>>>>
>>>>>>>>>> If you don't use "noac" it can still hang, but only if something
>>>>>>>>>> slows
>>>>>>>>>> down the reboot enough that attributes have timed out by the time
>>>>>>>>>> that
>>>>>>>>>> systemd-shutdown runs.  This happens for our customer.
>>>>>>>>>>
>>>>>>>>>> If the loop-mounted filesystem is not read-only, you get other
>>>>>>>>>> problems.
>>>>>>>>>>
>>>>>>>>>> We really want systemd to figure out that the loop-mount needs to be
>>>>>>>>>> unmounted first.  I have ideas concerning that, but it is messy.  But
>>>>>>>>>> that isn't the only bug here.
>>>>>>>>>
>>>>>>>>> The main purpose of d_weak_revalidate() was to catch the issues that
>>>>>>>>> arise when someone changes the contents of the current working
>>>>>>>>> directory or its parent on the server. Since '.' and '..' are treated
>>>>>>>>> specially in the lookup code, they would not be revalidated without
>>>>>>>>> special treatment. That leads to issues when looking up files as
>>>>>>>>> ./<filename> or ../<filename>, since the client won't detect that its
>>>>>>>>> dcache is stale until it tries to use the cached dentry+inode.
>>>>>>>>>
>>>>>>>>> The one thing that has changed since its introduction is, I believe,
>>>>>>>>> the ESTALE handling in the VFS layer. That might fix a lot of the
>>>>>>>>> dcache lookup bugs that were previously handled by d_weak_revalidate().
>>>>>>>>> I haven't done an audit to figure out if it actually can handle all of
>>>>>>>>> them.
>>>>>>>>>
>>>>>>>>
>>>>>>>> It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>>>>>>>>
>>>>>>>>     vfs: allow umount to handle mountpoints without revalidating them
>>>>>>>
>>>>>>> You say in the comment for that commit:
>>>>>>>
>>>>>>>      but there
>>>>>>>     are cases where we do want to revalidate the root of the fs.
>>>>>>>
>>>>>>> Do you happen to remember what those cases are?
>>>>>>>
>>>>>>
>>>>>> Not exactly, but I _think_ I might have been assuming that we needed to
>>>>>> ensure that the inode attrs on the root were up to date after the
>>>>>> pathwalk.
>>>>>>
>>>>>> I think that was probably wrong. d_revalidate is really intended to
>>>>>> ensure that the dentry in question still points to the same inode. In
>>>>>> the case of the root of the mount though, we don't really care about the
>>>>>> dentry on the server at all. We're attaching the root of the mount to an
>>>>>> inode and don't care of the dentry name changes. If we do need to ensure
>>>>>> the inode attrs are updated, we'll just revalidate them at that point.
>>>>>>
>>>>>>>>
>>>>>>>> Possibly the fact that we no longer try to revalidate during unmount
>>>>>>>> means that this is no longer necessary?
>>>>>>>>
>>>>>>>> The original patch that added d_weak_revalidate had a reproducer in the
>>>>>>>> patch description. Have you verified that that problem is still not
>>>>>>>> reproducible when you remove d_weak_revalidate?
>>>>>>>
>>>>>>> I did try the reproducer and it works as expected both with and without
>>>>>>> d_weak_revalidate.
>>>>>>> On reflection, the problem it displayed was caused by d_revalidate()
>>>>>>> being called when the dentry name was irrelevant.  We remove that
>>>>>>> (fixing the problem) and introduce d_weak_revalidate because we thought
>>>>>>> that minimum functionality was still useful.  I'm currently not
>>>>>>> convinced that even that is needed.
>>>>>>>
>>>>>>> If we discarded d_weak_revalidate(), we could get rid of the special
>>>>>>> handling of umount....
>>>>>>
>>>>>> I like idea. I say go for it and we can see what (if anything) breaks?
>>>>>
>>>>> Getting rid of d_weak_revalidate is easy enough - hardly any users.
>>>>>
>>>>> Getting rid of filename_mountpoint() isn't so easy unfortunately.
>>>>> autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
>>>>> in autofs4_d_manage()?  It would be a shame to keep this infrastructure
>>>>> around just so that one part of autofs4 can talk to another part of
>>>>> autofs4.
>>>
>>> When this was first implemented autofs didn't use kern_path_mountpoint()
>>> (it didn't exist) it used a path lookup on the parent and a separate
>>> lookup for the last component.
>> 
>> This was before commit 4e44b6852e03 ("Get rid of path_lookup in
>> autofs4").  This used kern_path().
>
> I have to plead not guilty of this one.
>
> IIRC it broke the requirement of "lookup parent then lookup last component"
> rather it walked the whole path then followed up to find the mount point
> struct path.
>
> Like it says in the description of ac8387199656 the caller might not yet
> "own" the autofs mount which causes a mount to be triggered during the
> walk that can't be satisfied because of the deadlock that occurs.

A mount isn't triggered by kern_path(pathname, 0, &path).
That '0' would need to include one of
  LOOKUP_PARENT | LOOKUP_DIRECTORY |
  LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT

to trigger an automount (otherwise you just get -EISDIR).

That is why I assumed that ->d_managed was the problem.

>
>> 
>> I'm more interested in commit ac8387199656 ("autofs4 - fix device ioctl
>> mount lookup")  which replaced the use of kern_path() with
>> kern_path_mountpoint().
>
> Probably should have had a Fixes: 4e44b6852e03 ...

Still a bit confused as to exactly what was fixed...

Thanks,
NeilBrown

[toc] | [prev] | [next] | [standalone]

#1716112

On 21/08/17 14:23, NeilBrown wrote:
> On Fri, Aug 18 2017, Ian Kent wrote:
> 
>> On 18/08/17 13:24, NeilBrown wrote:
>>> On Thu, Aug 17 2017, Ian Kent wrote:
>>>
>>>> On 16/08/17 19:34, Jeff Layton wrote:
>>>>> On Wed, 2017-08-16 at 12:43 +1000, NeilBrown wrote:
>>>>>> On Mon, Aug 14 2017, Jeff Layton wrote:
>>>>>>
>>>>>>> On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
>>>>>>>> On Fri, Aug 11 2017, Jeff Layton wrote:
>>>>>>>>
>>>>>>>>> On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>>>>>>>>>> On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>>>>>>>>>>> Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>>>>>>>>>>> flag and introduced the d_weak_revalidate dentry operation instead.
>>>>>>>>>>> We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>>>>>>>>>>> and added the new dentry operation to NFS dentries .... but not to
>>>>>>>>>>> NFSv4
>>>>>>>>>>> dentries.
>>>>>>>>>>>
>>>>>>>>>>> And nobody noticed.
>>>>>>>>>>>
>>>>>>>>>>> Until today.
>>>>>>>>>>>
>>>>>>>>>>> A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>>>>>>>>>>> NFS
>>>>>>>>>>> filesystem hangs because the network has been deconfigured.  This
>>>>>>>>>>> makes
>>>>>>>>>>> perfect sense and I suggested a code change to fix the problem.
>>>>>>>>>>> However when a colleague was trying to reproduce the problem to
>>>>>>>>>>> validate
>>>>>>>>>>> the fix, he couldn't.  Then nor could I.
>>>>>>>>>>>
>>>>>>>>>>> The problem is trivially reproducible with NFSv3, and not at all with
>>>>>>>>>>> NFSv4.  The reason is the missing d_weak_revalidate.
>>>>>>>>>>>
>>>>>>>>>>> We could simply add d_weak_revalidate for NFSv4, but given that it
>>>>>>>>>>> has been missing for 4.5 years, and the only time anyone noticed was
>>>>>>>>>>> when the ommission resulted in a better user experience, I do wonder
>>>>>>>>>>> if
>>>>>>>>>>> we need to.  Can we just discard d_weak_revalidate?  What purpose
>>>>>>>>>>> does
>>>>>>>>>>> it serve?  I couldn't find one.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> NeilBrown
>>>>>>>>>>>
>>>>>>>>>>> For reference, see
>>>>>>>>>>> Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>>>>>>>>>>> d_weak_revalidate dentry op")
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> To reproduce the problem at home, on a system that uses systemd:
>>>>>>>>>>> 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>>>>>>>>>>> 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>>>>>>>>>>> 3/ loop-mount the filesystem image read-only somewhere
>>>>>>>>>>> 4/ reboot
>>>>>>>>>>>
>>>>>>>>>>> If you choose v4, the reboot will succeed, possibly after a 90second
>>>>>>>>>>> timeout.
>>>>>>>>>>> If you choose v3, the reboot will hang indefinitely in systemd-
>>>>>>>>>>> shutdown while
>>>>>>>>>>> remounting the nfs filesystem read-only.
>>>>>>>>>>>
>>>>>>>>>>> If you don't use "noac" it can still hang, but only if something
>>>>>>>>>>> slows
>>>>>>>>>>> down the reboot enough that attributes have timed out by the time
>>>>>>>>>>> that
>>>>>>>>>>> systemd-shutdown runs.  This happens for our customer.
>>>>>>>>>>>
>>>>>>>>>>> If the loop-mounted filesystem is not read-only, you get other
>>>>>>>>>>> problems.
>>>>>>>>>>>
>>>>>>>>>>> We really want systemd to figure out that the loop-mount needs to be
>>>>>>>>>>> unmounted first.  I have ideas concerning that, but it is messy.  But
>>>>>>>>>>> that isn't the only bug here.
>>>>>>>>>>
>>>>>>>>>> The main purpose of d_weak_revalidate() was to catch the issues that
>>>>>>>>>> arise when someone changes the contents of the current working
>>>>>>>>>> directory or its parent on the server. Since '.' and '..' are treated
>>>>>>>>>> specially in the lookup code, they would not be revalidated without
>>>>>>>>>> special treatment. That leads to issues when looking up files as
>>>>>>>>>> ./<filename> or ../<filename>, since the client won't detect that its
>>>>>>>>>> dcache is stale until it tries to use the cached dentry+inode.
>>>>>>>>>>
>>>>>>>>>> The one thing that has changed since its introduction is, I believe,
>>>>>>>>>> the ESTALE handling in the VFS layer. That might fix a lot of the
>>>>>>>>>> dcache lookup bugs that were previously handled by d_weak_revalidate().
>>>>>>>>>> I haven't done an audit to figure out if it actually can handle all of
>>>>>>>>>> them.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>>>>>>>>>
>>>>>>>>>     vfs: allow umount to handle mountpoints without revalidating them
>>>>>>>>
>>>>>>>> You say in the comment for that commit:
>>>>>>>>
>>>>>>>>      but there
>>>>>>>>     are cases where we do want to revalidate the root of the fs.
>>>>>>>>
>>>>>>>> Do you happen to remember what those cases are?
>>>>>>>>
>>>>>>>
>>>>>>> Not exactly, but I _think_ I might have been assuming that we needed to
>>>>>>> ensure that the inode attrs on the root were up to date after the
>>>>>>> pathwalk.
>>>>>>>
>>>>>>> I think that was probably wrong. d_revalidate is really intended to
>>>>>>> ensure that the dentry in question still points to the same inode. In
>>>>>>> the case of the root of the mount though, we don't really care about the
>>>>>>> dentry on the server at all. We're attaching the root of the mount to an
>>>>>>> inode and don't care of the dentry name changes. If we do need to ensure
>>>>>>> the inode attrs are updated, we'll just revalidate them at that point.
>>>>>>>
>>>>>>>>>
>>>>>>>>> Possibly the fact that we no longer try to revalidate during unmount
>>>>>>>>> means that this is no longer necessary?
>>>>>>>>>
>>>>>>>>> The original patch that added d_weak_revalidate had a reproducer in the
>>>>>>>>> patch description. Have you verified that that problem is still not
>>>>>>>>> reproducible when you remove d_weak_revalidate?
>>>>>>>>
>>>>>>>> I did try the reproducer and it works as expected both with and without
>>>>>>>> d_weak_revalidate.
>>>>>>>> On reflection, the problem it displayed was caused by d_revalidate()
>>>>>>>> being called when the dentry name was irrelevant.  We remove that
>>>>>>>> (fixing the problem) and introduce d_weak_revalidate because we thought
>>>>>>>> that minimum functionality was still useful.  I'm currently not
>>>>>>>> convinced that even that is needed.
>>>>>>>>
>>>>>>>> If we discarded d_weak_revalidate(), we could get rid of the special
>>>>>>>> handling of umount....
>>>>>>>
>>>>>>> I like idea. I say go for it and we can see what (if anything) breaks?
>>>>>>
>>>>>> Getting rid of d_weak_revalidate is easy enough - hardly any users.
>>>>>>
>>>>>> Getting rid of filename_mountpoint() isn't so easy unfortunately.
>>>>>> autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
>>>>>> in autofs4_d_manage()?  It would be a shame to keep this infrastructure
>>>>>> around just so that one part of autofs4 can talk to another part of
>>>>>> autofs4.
>>>>
>>>> When this was first implemented autofs didn't use kern_path_mountpoint()
>>>> (it didn't exist) it used a path lookup on the parent and a separate
>>>> lookup for the last component.
>>>
>>> This was before commit 4e44b6852e03 ("Get rid of path_lookup in
>>> autofs4").  This used kern_path().
>>
>> I have to plead not guilty of this one.
>>
>> IIRC it broke the requirement of "lookup parent then lookup last component"
>> rather it walked the whole path then followed up to find the mount point
>> struct path.
>>
>> Like it says in the description of ac8387199656 the caller might not yet
>> "own" the autofs mount which causes a mount to be triggered during the
>> walk that can't be satisfied because of the deadlock that occurs.
> 
> A mount isn't triggered by kern_path(pathname, 0, &path).
> That '0' would need to include one of
>   LOOKUP_PARENT | LOOKUP_DIRECTORY |
>   LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT
> 
> to trigger an automount (otherwise you just get -EISDIR).

It's perfectly sensible to think that but there is a case where a
a mount is triggered when using kern_path().

The EISDIR return occurs for positive dentrys, negative dentrys
will still trigger an automount (which is autofs specific,
indirect mount map using nobrowse option, the install default).

> 
> That is why I assumed that ->d_managed was the problem.
> 
>>
>>>
>>> I'm more interested in commit ac8387199656 ("autofs4 - fix device ioctl
>>> mount lookup")  which replaced the use of kern_path() with
>>> kern_path_mountpoint().
>>
>> Probably should have had a Fixes: 4e44b6852e03 ...
> 
> Still a bit confused as to exactly what was fixed...

Hopefully also considering the negative dentry case will clear that up.

Ian

[toc] | [prev] | [next] | [standalone]

#1716136

[Multipart message — attachments visible in raw view] — view raw

On Mon, Aug 21 2017, Ian Kent wrote:

> On 21/08/17 14:23, NeilBrown wrote:
>> On Fri, Aug 18 2017, Ian Kent wrote:
>> 
>>> On 18/08/17 13:24, NeilBrown wrote:
>>>> On Thu, Aug 17 2017, Ian Kent wrote:
>>>>
>>>>> On 16/08/17 19:34, Jeff Layton wrote:
>>>>>> On Wed, 2017-08-16 at 12:43 +1000, NeilBrown wrote:
>>>>>>> On Mon, Aug 14 2017, Jeff Layton wrote:
>>>>>>>
>>>>>>>> On Mon, 2017-08-14 at 09:36 +1000, NeilBrown wrote:
>>>>>>>>> On Fri, Aug 11 2017, Jeff Layton wrote:
>>>>>>>>>
>>>>>>>>>> On Fri, 2017-08-11 at 05:55 +0000, Trond Myklebust wrote:
>>>>>>>>>>> On Fri, 2017-08-11 at 14:31 +1000, NeilBrown wrote:
>>>>>>>>>>>> Funny story.  4.5 years ago we discarded the FS_REVAL_DOT superblock
>>>>>>>>>>>> flag and introduced the d_weak_revalidate dentry operation instead.
>>>>>>>>>>>> We duly removed the flag from NFS superblocks and NFSv4 superblocks,
>>>>>>>>>>>> and added the new dentry operation to NFS dentries .... but not to
>>>>>>>>>>>> NFSv4
>>>>>>>>>>>> dentries.
>>>>>>>>>>>>
>>>>>>>>>>>> And nobody noticed.
>>>>>>>>>>>>
>>>>>>>>>>>> Until today.
>>>>>>>>>>>>
>>>>>>>>>>>> A customer reports a situation where mount(....,MS_REMOUNT,..) on an
>>>>>>>>>>>> NFS
>>>>>>>>>>>> filesystem hangs because the network has been deconfigured.  This
>>>>>>>>>>>> makes
>>>>>>>>>>>> perfect sense and I suggested a code change to fix the problem.
>>>>>>>>>>>> However when a colleague was trying to reproduce the problem to
>>>>>>>>>>>> validate
>>>>>>>>>>>> the fix, he couldn't.  Then nor could I.
>>>>>>>>>>>>
>>>>>>>>>>>> The problem is trivially reproducible with NFSv3, and not at all with
>>>>>>>>>>>> NFSv4.  The reason is the missing d_weak_revalidate.
>>>>>>>>>>>>
>>>>>>>>>>>> We could simply add d_weak_revalidate for NFSv4, but given that it
>>>>>>>>>>>> has been missing for 4.5 years, and the only time anyone noticed was
>>>>>>>>>>>> when the ommission resulted in a better user experience, I do wonder
>>>>>>>>>>>> if
>>>>>>>>>>>> we need to.  Can we just discard d_weak_revalidate?  What purpose
>>>>>>>>>>>> does
>>>>>>>>>>>> it serve?  I couldn't find one.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> NeilBrown
>>>>>>>>>>>>
>>>>>>>>>>>> For reference, see
>>>>>>>>>>>> Commit: ecf3d1f1aa74 ("vfs: kill FS_REVAL_DOT by adding a
>>>>>>>>>>>> d_weak_revalidate dentry op")
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> To reproduce the problem at home, on a system that uses systemd:
>>>>>>>>>>>> 1/ place (or find) a filesystem image in a file on an NFS filesystem.
>>>>>>>>>>>> 2/ mount the nfs filesystem with "noac" - choose v3 or v4
>>>>>>>>>>>> 3/ loop-mount the filesystem image read-only somewhere
>>>>>>>>>>>> 4/ reboot
>>>>>>>>>>>>
>>>>>>>>>>>> If you choose v4, the reboot will succeed, possibly after a 90second
>>>>>>>>>>>> timeout.
>>>>>>>>>>>> If you choose v3, the reboot will hang indefinitely in systemd-
>>>>>>>>>>>> shutdown while
>>>>>>>>>>>> remounting the nfs filesystem read-only.
>>>>>>>>>>>>
>>>>>>>>>>>> If you don't use "noac" it can still hang, but only if something
>>>>>>>>>>>> slows
>>>>>>>>>>>> down the reboot enough that attributes have timed out by the time
>>>>>>>>>>>> that
>>>>>>>>>>>> systemd-shutdown runs.  This happens for our customer.
>>>>>>>>>>>>
>>>>>>>>>>>> If the loop-mounted filesystem is not read-only, you get other
>>>>>>>>>>>> problems.
>>>>>>>>>>>>
>>>>>>>>>>>> We really want systemd to figure out that the loop-mount needs to be
>>>>>>>>>>>> unmounted first.  I have ideas concerning that, but it is messy.  But
>>>>>>>>>>>> that isn't the only bug here.
>>>>>>>>>>>
>>>>>>>>>>> The main purpose of d_weak_revalidate() was to catch the issues that
>>>>>>>>>>> arise when someone changes the contents of the current working
>>>>>>>>>>> directory or its parent on the server. Since '.' and '..' are treated
>>>>>>>>>>> specially in the lookup code, they would not be revalidated without
>>>>>>>>>>> special treatment. That leads to issues when looking up files as
>>>>>>>>>>> ./<filename> or ../<filename>, since the client won't detect that its
>>>>>>>>>>> dcache is stale until it tries to use the cached dentry+inode.
>>>>>>>>>>>
>>>>>>>>>>> The one thing that has changed since its introduction is, I believe,
>>>>>>>>>>> the ESTALE handling in the VFS layer. That might fix a lot of the
>>>>>>>>>>> dcache lookup bugs that were previously handled by d_weak_revalidate().
>>>>>>>>>>> I haven't done an audit to figure out if it actually can handle all of
>>>>>>>>>>> them.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It may also be related to 8033426e6bdb2690d302872ac1e1fadaec1a5581:
>>>>>>>>>>
>>>>>>>>>>     vfs: allow umount to handle mountpoints without revalidating them
>>>>>>>>>
>>>>>>>>> You say in the comment for that commit:
>>>>>>>>>
>>>>>>>>>      but there
>>>>>>>>>     are cases where we do want to revalidate the root of the fs.
>>>>>>>>>
>>>>>>>>> Do you happen to remember what those cases are?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Not exactly, but I _think_ I might have been assuming that we needed to
>>>>>>>> ensure that the inode attrs on the root were up to date after the
>>>>>>>> pathwalk.
>>>>>>>>
>>>>>>>> I think that was probably wrong. d_revalidate is really intended to
>>>>>>>> ensure that the dentry in question still points to the same inode. In
>>>>>>>> the case of the root of the mount though, we don't really care about the
>>>>>>>> dentry on the server at all. We're attaching the root of the mount to an
>>>>>>>> inode and don't care of the dentry name changes. If we do need to ensure
>>>>>>>> the inode attrs are updated, we'll just revalidate them at that point.
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Possibly the fact that we no longer try to revalidate during unmount
>>>>>>>>>> means that this is no longer necessary?
>>>>>>>>>>
>>>>>>>>>> The original patch that added d_weak_revalidate had a reproducer in the
>>>>>>>>>> patch description. Have you verified that that problem is still not
>>>>>>>>>> reproducible when you remove d_weak_revalidate?
>>>>>>>>>
>>>>>>>>> I did try the reproducer and it works as expected both with and without
>>>>>>>>> d_weak_revalidate.
>>>>>>>>> On reflection, the problem it displayed was caused by d_revalidate()
>>>>>>>>> being called when the dentry name was irrelevant.  We remove that
>>>>>>>>> (fixing the problem) and introduce d_weak_revalidate because we thought
>>>>>>>>> that minimum functionality was still useful.  I'm currently not
>>>>>>>>> convinced that even that is needed.
>>>>>>>>>
>>>>>>>>> If we discarded d_weak_revalidate(), we could get rid of the special
>>>>>>>>> handling of umount....
>>>>>>>>
>>>>>>>> I like idea. I say go for it and we can see what (if anything) breaks?
>>>>>>>
>>>>>>> Getting rid of d_weak_revalidate is easy enough - hardly any users.
>>>>>>>
>>>>>>> Getting rid of filename_mountpoint() isn't so easy unfortunately.
>>>>>>> autofs4 uses kern_path_mountpoint() - presumably to avoid getting stuck
>>>>>>> in autofs4_d_manage()?  It would be a shame to keep this infrastructure
>>>>>>> around just so that one part of autofs4 can talk to another part of
>>>>>>> autofs4.
>>>>>
>>>>> When this was first implemented autofs didn't use kern_path_mountpoint()
>>>>> (it didn't exist) it used a path lookup on the parent and a separate
>>>>> lookup for the last component.
>>>>
>>>> This was before commit 4e44b6852e03 ("Get rid of path_lookup in
>>>> autofs4").  This used kern_path().
>>>
>>> I have to plead not guilty of this one.
>>>
>>> IIRC it broke the requirement of "lookup parent then lookup last component"
>>> rather it walked the whole path then followed up to find the mount point
>>> struct path.
>>>
>>> Like it says in the description of ac8387199656 the caller might not yet
>>> "own" the autofs mount which causes a mount to be triggered during the
>>> walk that can't be satisfied because of the deadlock that occurs.
>> 
>> A mount isn't triggered by kern_path(pathname, 0, &path).
>> That '0' would need to include one of
>>   LOOKUP_PARENT | LOOKUP_DIRECTORY |
>>   LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT
>> 
>> to trigger an automount (otherwise you just get -EISDIR).
>
> It's perfectly sensible to think that but there is a case where a
> a mount is triggered when using kern_path().
>
> The EISDIR return occurs for positive dentrys, negative dentrys
> will still trigger an automount (which is autofs specific,
> indirect mount map using nobrowse option, the install default).

Ahh  -  light dawns.  Thanks :-)

NeilBrown


>
>> 
>> That is why I assumed that ->d_managed was the problem.
>> 
>>>
>>>>
>>>> I'm more interested in commit ac8387199656 ("autofs4 - fix device ioctl
>>>> mount lookup")  which replaced the use of kern_path() with
>>>> kern_path_mountpoint().
>>>
>>> Probably should have had a Fixes: 4e44b6852e03 ...
>> 
>> Still a bit confused as to exactly what was fixed...
>
> Hopefully also considering the negative dentry case will clear that up.
>
> Ian

[toc] | [prev] | [next] | [standalone]

#1717937

[Multipart message — attachments visible in raw view] — view raw

On Mon, Aug 21 2017, Ian Kent wrote:

>> 
>> A mount isn't triggered by kern_path(pathname, 0, &path).
>> That '0' would need to include one of
>>   LOOKUP_PARENT | LOOKUP_DIRECTORY |
>>   LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT
>> 
>> to trigger an automount (otherwise you just get -EISDIR).
>
> It's perfectly sensible to think that but there is a case where a
> a mount is triggered when using kern_path().
>
> The EISDIR return occurs for positive dentrys, negative dentrys
> will still trigger an automount (which is autofs specific,
> indirect mount map using nobrowse option, the install default).

Ok, I understand this better now.  This difference between direct and
indirect mounts is slightly awkward. It is visible from user-space, but
not elegant to document.
When you use O_PATH to open a direct automount that has not already been
triggered, the open returns the underlying directory (and fstatfs
confirms that it is AUTOFS_SUPER_MAGIC).  When you use O_PATH on
an indirect automount, it *will* trigger the automount when "nobrowse" is
in effect, but it won't when "browse" is in effect.

So we cannot just say "O_PATH doesn't trigger automounts", which is
essentially what I said in

https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=97a45d02e6671482e8b2cdcce3951930bf6bdb94

It might be possible to modify automount so that it was more consistent
- i.e. if the point is triggered by a mkdir has been done, just to the
mkdir.  If it is triggered after a mkdir has been done, do the mount.  I
guess that might be racy, and in any case is hard to justify.

Maybe I should change it to be about "direct automounts", and add a note
that indirect automounts aren't so predictable.

But back to my original issue of wanting to discard
kern_path_mountpoint, what would you think of the following approach -
slight revised from before.

Thanks,
NeilBrown

diff --git a/fs/autofs4/autofs_i.h b/fs/autofs4/autofs_i.h
index beef981aa54f..7663ea82e68d 100644
--- a/fs/autofs4/autofs_i.h
+++ b/fs/autofs4/autofs_i.h
@@ -135,10 +135,13 @@ static inline struct autofs_info *autofs4_dentry_ino(struct dentry *dentry)
 /* autofs4_oz_mode(): do we see the man behind the curtain?  (The
  * processes which do manipulations for us in user space sees the raw
  * filesystem without "magic".)
+ * A process performing certain ioctls can get temporary oz status.
  */
+extern struct task_struct *autofs_tmp_oz;
 static inline int autofs4_oz_mode(struct autofs_sb_info *sbi)
 {
-	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp;
+	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp ||
+		autofs_tmp_oz == current;
 }
 
 struct inode *autofs4_get_inode(struct super_block *, umode_t);
diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
index dd9f1bebb5a3..d76401669a20 100644
--- a/fs/autofs4/dev-ioctl.c
+++ b/fs/autofs4/dev-ioctl.c
@@ -200,6 +200,20 @@ static int autofs_dev_ioctl_protosubver(struct file *fp,
 	return 0;
 }
 
+struct task_struct *autofs_tmp_oz;
+int kern_path_oz(const char *pathname, int flags, struct path *path)
+{
+	static DEFINE_MUTEX(autofs_oz);
+	int err;
+
+	mutex_lock(&autofs_oz);
+	autofs_tmp_oz = current;
+	err = kern_path(pathname, flags, path);
+	autofs_tmp_oz = NULL;
+	mutex_unlock(&autofs_oz);
+	return err;
+}
+
 /* Find the topmost mount satisfying test() */
 static int find_autofs_mount(const char *pathname,
 			     struct path *res,
@@ -209,7 +223,8 @@ static int find_autofs_mount(const char *pathname,
 	struct path path;
 	int err;
 
-	err = kern_path_mountpoint(AT_FDCWD, pathname, &path, 0);
+	err = kern_path_oz(pathname, 0, &path);
+
 	if (err)
 		return err;
 	err = -ENOENT;
@@ -552,8 +567,7 @@ static int autofs_dev_ioctl_ismountpoint(struct file *fp,
 
 	if (!fp || param->ioctlfd == -1) {
 		if (autofs_type_any(type))
-			err = kern_path_mountpoint(AT_FDCWD,
-						   name, &path, LOOKUP_FOLLOW);
+			err = kern_path_oz(name, LOOKUP_FOLLOW, &path);
 		else
 			err = find_autofs_mount(name, &path,
 						test_by_type, &type);

[toc] | [prev] | [next] | [standalone]

#1717988

On 23/08/17 09:06, NeilBrown wrote:
> On Mon, Aug 21 2017, Ian Kent wrote:
> 
>>>
>>> A mount isn't triggered by kern_path(pathname, 0, &path).
>>> That '0' would need to include one of
>>>   LOOKUP_PARENT | LOOKUP_DIRECTORY |
>>>   LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT
>>>
>>> to trigger an automount (otherwise you just get -EISDIR).
>>
>> It's perfectly sensible to think that but there is a case where a
>> a mount is triggered when using kern_path().
>>
>> The EISDIR return occurs for positive dentrys, negative dentrys
>> will still trigger an automount (which is autofs specific,
>> indirect mount map using nobrowse option, the install default).
> 
> Ok, I understand this better now.  This difference between direct and
> indirect mounts is slightly awkward. It is visible from user-space, but
> not elegant to document.
> When you use O_PATH to open a direct automount that has not already been
> triggered, the open returns the underlying directory (and fstatfs
> confirms that it is AUTOFS_SUPER_MAGIC).  When you use O_PATH on
> an indirect automount, it *will* trigger the automount when "nobrowse" is
> in effect, but it won't when "browse" is in effect.

That inconsistency has bothered me for quite a while now.

It was carried over from the autofs module behavior when automounting
support was added to the VFS. What's worse is it prevents the use of
the AT_NO_AUTOMOUNT flag from working properly with fstatat(2) and with
statx().

There is some risk in changing that so it does work but it really does
need to work to enable userspace to not trigger an automount by using
this flag.

So that's (hopefully) going to change soonish, see:
http://ozlabs.org/~akpm/mmotm/broken-out/autofs-fix-at_no_automount-not-being-honored.patch

The result should be that stat family calls don't trigger automounts except
for fstatat(2) and statx() which will require the AT_NO_AUTOMOUNT flag.

> 
> So we cannot just say "O_PATH doesn't trigger automounts", which is
> essentially what I said in
> 
> https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=97a45d02e6671482e8b2cdcce3951930bf6bdb94
> 
> It might be possible to modify automount so that it was more consistent
> - i.e. if the point is triggered by a mkdir has been done, just to the
> mkdir.  If it is triggered after a mkdir has been done, do the mount.  I
> guess that might be racy, and in any case is hard to justify.
> 
> Maybe I should change it to be about "direct automounts", and add a note
> that indirect automounts aren't so predictable.

Right and the semantics should be much more consistent in the near future.
I hope (and expect) this semantic change won't cause problems.

> 
> But back to my original issue of wanting to discard
> kern_path_mountpoint, what would you think of the following approach -
> slight revised from before.
> 
> Thanks,
> NeilBrown
> 
> diff --git a/fs/autofs4/autofs_i.h b/fs/autofs4/autofs_i.h
> index beef981aa54f..7663ea82e68d 100644
> --- a/fs/autofs4/autofs_i.h
> +++ b/fs/autofs4/autofs_i.h
> @@ -135,10 +135,13 @@ static inline struct autofs_info *autofs4_dentry_ino(struct dentry *dentry)
>  /* autofs4_oz_mode(): do we see the man behind the curtain?  (The
>   * processes which do manipulations for us in user space sees the raw
>   * filesystem without "magic".)
> + * A process performing certain ioctls can get temporary oz status.
>   */
> +extern struct task_struct *autofs_tmp_oz;
>  static inline int autofs4_oz_mode(struct autofs_sb_info *sbi)
>  {
> -	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp;
> +	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp ||
> +		autofs_tmp_oz == current;
>  }
>  
>  struct inode *autofs4_get_inode(struct super_block *, umode_t);
> diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
> index dd9f1bebb5a3..d76401669a20 100644
> --- a/fs/autofs4/dev-ioctl.c
> +++ b/fs/autofs4/dev-ioctl.c
> @@ -200,6 +200,20 @@ static int autofs_dev_ioctl_protosubver(struct file *fp,
>  	return 0;
>  }
>  
> +struct task_struct *autofs_tmp_oz;
> +int kern_path_oz(const char *pathname, int flags, struct path *path)
> +{
> +	static DEFINE_MUTEX(autofs_oz);
> +	int err;
> +
> +	mutex_lock(&autofs_oz);
> +	autofs_tmp_oz = current;
> +	err = kern_path(pathname, flags, path);
> +	autofs_tmp_oz = NULL;
> +	mutex_unlock(&autofs_oz);
> +	return err;
> +}
> +

It's simple enough but does look like it will attract criticism as being
a hack!

The kern_path_locked() function is very similar to what was originally
done, along with code to look down the mount stack (rather than up the
way it does now) to get the mount point. In this case, to be valid the
dentry can't be a symlink so that fits kern_path_locked() too.

So maybe it is worth going back to the way it was in the beginning and
be done with it .... OTOH Al must have had a reason for changing the
way it was done that I didn't get.

>  /* Find the topmost mount satisfying test() */
>  static int find_autofs_mount(const char *pathname,
>  			     struct path *res,
> @@ -209,7 +223,8 @@ static int find_autofs_mount(const char *pathname,
>  	struct path path;
>  	int err;
>  
> -	err = kern_path_mountpoint(AT_FDCWD, pathname, &path, 0);
> +	err = kern_path_oz(pathname, 0, &path);
> +
>  	if (err)
>  		return err;
>  	err = -ENOENT;
> @@ -552,8 +567,7 @@ static int autofs_dev_ioctl_ismountpoint(struct file *fp,
>  
>  	if (!fp || param->ioctlfd == -1) {
>  		if (autofs_type_any(type))
> -			err = kern_path_mountpoint(AT_FDCWD,
> -						   name, &path, LOOKUP_FOLLOW);
> +			err = kern_path_oz(name, LOOKUP_FOLLOW, &path);
>  		else
>  			err = find_autofs_mount(name, &path,
>  						test_by_type, &type);
> 

[toc] | [prev] | [next] | [standalone]

#1717993

On 23/08/17 10:32, Ian Kent wrote:
> On 23/08/17 09:06, NeilBrown wrote:
>> On Mon, Aug 21 2017, Ian Kent wrote:
>>
>>>>
>>>> A mount isn't triggered by kern_path(pathname, 0, &path).
>>>> That '0' would need to include one of
>>>>   LOOKUP_PARENT | LOOKUP_DIRECTORY |
>>>>   LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT
>>>>
>>>> to trigger an automount (otherwise you just get -EISDIR).
>>>
>>> It's perfectly sensible to think that but there is a case where a
>>> a mount is triggered when using kern_path().
>>>
>>> The EISDIR return occurs for positive dentrys, negative dentrys
>>> will still trigger an automount (which is autofs specific,
>>> indirect mount map using nobrowse option, the install default).
>>
>> Ok, I understand this better now.  This difference between direct and
>> indirect mounts is slightly awkward. It is visible from user-space, but
>> not elegant to document.
>> When you use O_PATH to open a direct automount that has not already been
>> triggered, the open returns the underlying directory (and fstatfs
>> confirms that it is AUTOFS_SUPER_MAGIC).  When you use O_PATH on
>> an indirect automount, it *will* trigger the automount when "nobrowse" is
>> in effect, but it won't when "browse" is in effect.
> 
> That inconsistency has bothered me for quite a while now.
> 
> It was carried over from the autofs module behavior when automounting
> support was added to the VFS. What's worse is it prevents the use of
> the AT_NO_AUTOMOUNT flag from working properly with fstatat(2) and with
> statx().
> 
> There is some risk in changing that so it does work but it really does
> need to work to enable userspace to not trigger an automount by using
> this flag.
> 
> So that's (hopefully) going to change soonish, see:
> http://ozlabs.org/~akpm/mmotm/broken-out/autofs-fix-at_no_automount-not-being-honored.patch
> 
> The result should be that stat family calls don't trigger automounts except
> for fstatat(2) and statx() which will require the AT_NO_AUTOMOUNT flag.
> 
>>
>> So we cannot just say "O_PATH doesn't trigger automounts", which is
>> essentially what I said in
>>
>> https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=97a45d02e6671482e8b2cdcce3951930bf6bdb94
>>
>> It might be possible to modify automount so that it was more consistent
>> - i.e. if the point is triggered by a mkdir has been done, just to the
>> mkdir.  If it is triggered after a mkdir has been done, do the mount.  I
>> guess that might be racy, and in any case is hard to justify.
>>
>> Maybe I should change it to be about "direct automounts", and add a note
>> that indirect automounts aren't so predictable.
> 
> Right and the semantics should be much more consistent in the near future.
> I hope (and expect) this semantic change won't cause problems.
> 
>>
>> But back to my original issue of wanting to discard
>> kern_path_mountpoint, what would you think of the following approach -
>> slight revised from before.
>>
>> Thanks,
>> NeilBrown
>>
>> diff --git a/fs/autofs4/autofs_i.h b/fs/autofs4/autofs_i.h
>> index beef981aa54f..7663ea82e68d 100644
>> --- a/fs/autofs4/autofs_i.h
>> +++ b/fs/autofs4/autofs_i.h
>> @@ -135,10 +135,13 @@ static inline struct autofs_info *autofs4_dentry_ino(struct dentry *dentry)
>>  /* autofs4_oz_mode(): do we see the man behind the curtain?  (The
>>   * processes which do manipulations for us in user space sees the raw
>>   * filesystem without "magic".)
>> + * A process performing certain ioctls can get temporary oz status.
>>   */
>> +extern struct task_struct *autofs_tmp_oz;
>>  static inline int autofs4_oz_mode(struct autofs_sb_info *sbi)
>>  {
>> -	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp;
>> +	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp ||
>> +		autofs_tmp_oz == current;
>>  }
>>  
>>  struct inode *autofs4_get_inode(struct super_block *, umode_t);
>> diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
>> index dd9f1bebb5a3..d76401669a20 100644
>> --- a/fs/autofs4/dev-ioctl.c
>> +++ b/fs/autofs4/dev-ioctl.c
>> @@ -200,6 +200,20 @@ static int autofs_dev_ioctl_protosubver(struct file *fp,
>>  	return 0;
>>  }
>>  
>> +struct task_struct *autofs_tmp_oz;
>> +int kern_path_oz(const char *pathname, int flags, struct path *path)
>> +{
>> +	static DEFINE_MUTEX(autofs_oz);
>> +	int err;
>> +
>> +	mutex_lock(&autofs_oz);
>> +	autofs_tmp_oz = current;
>> +	err = kern_path(pathname, flags, path);
>> +	autofs_tmp_oz = NULL;
>> +	mutex_unlock(&autofs_oz);
>> +	return err;
>> +}
>> +
> 
> It's simple enough but does look like it will attract criticism as being
> a hack!
> 
> The kern_path_locked() function is very similar to what was originally
> done, along with code to look down the mount stack (rather than up the
> way it does now) to get the mount point. In this case, to be valid the
> dentry can't be a symlink so that fits kern_path_locked() too.

Oh wait, that __lookup_hash() tries too hard to resolve the dentry,
that won't quite work, and maybe d_lookup() can't be used safely in
this context either ....

> 
> So maybe it is worth going back to the way it was in the beginning and
> be done with it .... OTOH Al must have had a reason for changing the
> way it was done that I didn't get.
> 
>>  /* Find the topmost mount satisfying test() */
>>  static int find_autofs_mount(const char *pathname,
>>  			     struct path *res,
>> @@ -209,7 +223,8 @@ static int find_autofs_mount(const char *pathname,
>>  	struct path path;
>>  	int err;
>>  
>> -	err = kern_path_mountpoint(AT_FDCWD, pathname, &path, 0);
>> +	err = kern_path_oz(pathname, 0, &path);
>> +
>>  	if (err)
>>  		return err;
>>  	err = -ENOENT;
>> @@ -552,8 +567,7 @@ static int autofs_dev_ioctl_ismountpoint(struct file *fp,
>>  
>>  	if (!fp || param->ioctlfd == -1) {
>>  		if (autofs_type_any(type))
>> -			err = kern_path_mountpoint(AT_FDCWD,
>> -						   name, &path, LOOKUP_FOLLOW);
>> +			err = kern_path_oz(name, LOOKUP_FOLLOW, &path);
>>  		else
>>  			err = find_autofs_mount(name, &path,
>>  						test_by_type, &type);
>>
> 

[toc] | [prev] | [next] | [standalone]

#1717998

On 23/08/17 10:40, Ian Kent wrote:
> On 23/08/17 10:32, Ian Kent wrote:
>> On 23/08/17 09:06, NeilBrown wrote:
>>> On Mon, Aug 21 2017, Ian Kent wrote:
>>>
>>>>>
>>>>> A mount isn't triggered by kern_path(pathname, 0, &path).
>>>>> That '0' would need to include one of
>>>>>   LOOKUP_PARENT | LOOKUP_DIRECTORY |
>>>>>   LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT
>>>>>
>>>>> to trigger an automount (otherwise you just get -EISDIR).
>>>>
>>>> It's perfectly sensible to think that but there is a case where a
>>>> a mount is triggered when using kern_path().
>>>>
>>>> The EISDIR return occurs for positive dentrys, negative dentrys
>>>> will still trigger an automount (which is autofs specific,
>>>> indirect mount map using nobrowse option, the install default).
>>>
>>> Ok, I understand this better now.  This difference between direct and
>>> indirect mounts is slightly awkward. It is visible from user-space, but
>>> not elegant to document.
>>> When you use O_PATH to open a direct automount that has not already been
>>> triggered, the open returns the underlying directory (and fstatfs
>>> confirms that it is AUTOFS_SUPER_MAGIC).  When you use O_PATH on
>>> an indirect automount, it *will* trigger the automount when "nobrowse" is
>>> in effect, but it won't when "browse" is in effect.
>>
>> That inconsistency has bothered me for quite a while now.
>>
>> It was carried over from the autofs module behavior when automounting
>> support was added to the VFS. What's worse is it prevents the use of
>> the AT_NO_AUTOMOUNT flag from working properly with fstatat(2) and with
>> statx().
>>
>> There is some risk in changing that so it does work but it really does
>> need to work to enable userspace to not trigger an automount by using
>> this flag.
>>
>> So that's (hopefully) going to change soonish, see:
>> http://ozlabs.org/~akpm/mmotm/broken-out/autofs-fix-at_no_automount-not-being-honored.patch
>>
>> The result should be that stat family calls don't trigger automounts except
>> for fstatat(2) and statx() which will require the AT_NO_AUTOMOUNT flag.
>>
>>>
>>> So we cannot just say "O_PATH doesn't trigger automounts", which is
>>> essentially what I said in
>>>
>>> https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=97a45d02e6671482e8b2cdcce3951930bf6bdb94
>>>
>>> It might be possible to modify automount so that it was more consistent
>>> - i.e. if the point is triggered by a mkdir has been done, just to the
>>> mkdir.  If it is triggered after a mkdir has been done, do the mount.  I
>>> guess that might be racy, and in any case is hard to justify.
>>>
>>> Maybe I should change it to be about "direct automounts", and add a note
>>> that indirect automounts aren't so predictable.
>>
>> Right and the semantics should be much more consistent in the near future.
>> I hope (and expect) this semantic change won't cause problems.
>>
>>>
>>> But back to my original issue of wanting to discard
>>> kern_path_mountpoint, what would you think of the following approach -
>>> slight revised from before.
>>>
>>> Thanks,
>>> NeilBrown
>>>
>>> diff --git a/fs/autofs4/autofs_i.h b/fs/autofs4/autofs_i.h
>>> index beef981aa54f..7663ea82e68d 100644
>>> --- a/fs/autofs4/autofs_i.h
>>> +++ b/fs/autofs4/autofs_i.h
>>> @@ -135,10 +135,13 @@ static inline struct autofs_info *autofs4_dentry_ino(struct dentry *dentry)
>>>  /* autofs4_oz_mode(): do we see the man behind the curtain?  (The
>>>   * processes which do manipulations for us in user space sees the raw
>>>   * filesystem without "magic".)
>>> + * A process performing certain ioctls can get temporary oz status.
>>>   */
>>> +extern struct task_struct *autofs_tmp_oz;
>>>  static inline int autofs4_oz_mode(struct autofs_sb_info *sbi)
>>>  {
>>> -	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp;
>>> +	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp ||
>>> +		autofs_tmp_oz == current;
>>>  }
>>>  
>>>  struct inode *autofs4_get_inode(struct super_block *, umode_t);
>>> diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
>>> index dd9f1bebb5a3..d76401669a20 100644
>>> --- a/fs/autofs4/dev-ioctl.c
>>> +++ b/fs/autofs4/dev-ioctl.c
>>> @@ -200,6 +200,20 @@ static int autofs_dev_ioctl_protosubver(struct file *fp,
>>>  	return 0;
>>>  }
>>>  
>>> +struct task_struct *autofs_tmp_oz;
>>> +int kern_path_oz(const char *pathname, int flags, struct path *path)
>>> +{
>>> +	static DEFINE_MUTEX(autofs_oz);
>>> +	int err;
>>> +
>>> +	mutex_lock(&autofs_oz);
>>> +	autofs_tmp_oz = current;
>>> +	err = kern_path(pathname, flags, path);
>>> +	autofs_tmp_oz = NULL;
>>> +	mutex_unlock(&autofs_oz);
>>> +	return err;
>>> +}
>>> +
>>
>> It's simple enough but does look like it will attract criticism as being
>> a hack!
>>
>> The kern_path_locked() function is very similar to what was originally
>> done, along with code to look down the mount stack (rather than up the
>> way it does now) to get the mount point. In this case, to be valid the
>> dentry can't be a symlink so that fits kern_path_locked() too.
> 
> Oh wait, that __lookup_hash() tries too hard to resolve the dentry,
> that won't quite work, and maybe d_lookup() can't be used safely in
> this context either ....

Umm .. d_lookup() does look ok so maybe path_parentat() + d_lookup()
would be ok.

> 
>>
>> So maybe it is worth going back to the way it was in the beginning and
>> be done with it .... OTOH Al must have had a reason for changing the
>> way it was done that I didn't get.
>>
>>>  /* Find the topmost mount satisfying test() */
>>>  static int find_autofs_mount(const char *pathname,
>>>  			     struct path *res,
>>> @@ -209,7 +223,8 @@ static int find_autofs_mount(const char *pathname,
>>>  	struct path path;
>>>  	int err;
>>>  
>>> -	err = kern_path_mountpoint(AT_FDCWD, pathname, &path, 0);
>>> +	err = kern_path_oz(pathname, 0, &path);
>>> +
>>>  	if (err)
>>>  		return err;
>>>  	err = -ENOENT;
>>> @@ -552,8 +567,7 @@ static int autofs_dev_ioctl_ismountpoint(struct file *fp,
>>>  
>>>  	if (!fp || param->ioctlfd == -1) {
>>>  		if (autofs_type_any(type))
>>> -			err = kern_path_mountpoint(AT_FDCWD,
>>> -						   name, &path, LOOKUP_FOLLOW);
>>> +			err = kern_path_oz(name, LOOKUP_FOLLOW, &path);
>>>  		else
>>>  			err = find_autofs_mount(name, &path,
>>>  						test_by_type, &type);
>>>
>>
> 

[toc] | [prev] | [next] | [standalone]

#1718127

On 23/08/17 10:54, Ian Kent wrote:
> On 23/08/17 10:40, Ian Kent wrote:
>> On 23/08/17 10:32, Ian Kent wrote:
>>> On 23/08/17 09:06, NeilBrown wrote:
>>>> On Mon, Aug 21 2017, Ian Kent wrote:
>>>>
>>>>>>
>>>>>> A mount isn't triggered by kern_path(pathname, 0, &path).
>>>>>> That '0' would need to include one of
>>>>>>   LOOKUP_PARENT | LOOKUP_DIRECTORY |
>>>>>>   LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT
>>>>>>
>>>>>> to trigger an automount (otherwise you just get -EISDIR).
>>>>>
>>>>> It's perfectly sensible to think that but there is a case where a
>>>>> a mount is triggered when using kern_path().
>>>>>
>>>>> The EISDIR return occurs for positive dentrys, negative dentrys
>>>>> will still trigger an automount (which is autofs specific,
>>>>> indirect mount map using nobrowse option, the install default).
>>>>
>>>> Ok, I understand this better now.  This difference between direct and
>>>> indirect mounts is slightly awkward. It is visible from user-space, but
>>>> not elegant to document.
>>>> When you use O_PATH to open a direct automount that has not already been
>>>> triggered, the open returns the underlying directory (and fstatfs
>>>> confirms that it is AUTOFS_SUPER_MAGIC).  When you use O_PATH on
>>>> an indirect automount, it *will* trigger the automount when "nobrowse" is
>>>> in effect, but it won't when "browse" is in effect.
>>>
>>> That inconsistency has bothered me for quite a while now.
>>>
>>> It was carried over from the autofs module behavior when automounting
>>> support was added to the VFS. What's worse is it prevents the use of
>>> the AT_NO_AUTOMOUNT flag from working properly with fstatat(2) and with
>>> statx().
>>>
>>> There is some risk in changing that so it does work but it really does
>>> need to work to enable userspace to not trigger an automount by using
>>> this flag.
>>>
>>> So that's (hopefully) going to change soonish, see:
>>> http://ozlabs.org/~akpm/mmotm/broken-out/autofs-fix-at_no_automount-not-being-honored.patch
>>>
>>> The result should be that stat family calls don't trigger automounts except
>>> for fstatat(2) and statx() which will require the AT_NO_AUTOMOUNT flag.
>>>
>>>>
>>>> So we cannot just say "O_PATH doesn't trigger automounts", which is
>>>> essentially what I said in
>>>>
>>>> https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=97a45d02e6671482e8b2cdcce3951930bf6bdb94
>>>>
>>>> It might be possible to modify automount so that it was more consistent
>>>> - i.e. if the point is triggered by a mkdir has been done, just to the
>>>> mkdir.  If it is triggered after a mkdir has been done, do the mount.  I
>>>> guess that might be racy, and in any case is hard to justify.
>>>>
>>>> Maybe I should change it to be about "direct automounts", and add a note
>>>> that indirect automounts aren't so predictable.
>>>
>>> Right and the semantics should be much more consistent in the near future.
>>> I hope (and expect) this semantic change won't cause problems.
>>>
>>>>
>>>> But back to my original issue of wanting to discard
>>>> kern_path_mountpoint, what would you think of the following approach -
>>>> slight revised from before.
>>>>
>>>> Thanks,
>>>> NeilBrown
>>>>
>>>> diff --git a/fs/autofs4/autofs_i.h b/fs/autofs4/autofs_i.h
>>>> index beef981aa54f..7663ea82e68d 100644
>>>> --- a/fs/autofs4/autofs_i.h
>>>> +++ b/fs/autofs4/autofs_i.h
>>>> @@ -135,10 +135,13 @@ static inline struct autofs_info *autofs4_dentry_ino(struct dentry *dentry)
>>>>  /* autofs4_oz_mode(): do we see the man behind the curtain?  (The
>>>>   * processes which do manipulations for us in user space sees the raw
>>>>   * filesystem without "magic".)
>>>> + * A process performing certain ioctls can get temporary oz status.
>>>>   */
>>>> +extern struct task_struct *autofs_tmp_oz;
>>>>  static inline int autofs4_oz_mode(struct autofs_sb_info *sbi)
>>>>  {
>>>> -	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp;
>>>> +	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp ||
>>>> +		autofs_tmp_oz == current;
>>>>  }
>>>>  
>>>>  struct inode *autofs4_get_inode(struct super_block *, umode_t);
>>>> diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
>>>> index dd9f1bebb5a3..d76401669a20 100644
>>>> --- a/fs/autofs4/dev-ioctl.c
>>>> +++ b/fs/autofs4/dev-ioctl.c
>>>> @@ -200,6 +200,20 @@ static int autofs_dev_ioctl_protosubver(struct file *fp,
>>>>  	return 0;
>>>>  }
>>>>  
>>>> +struct task_struct *autofs_tmp_oz;
>>>> +int kern_path_oz(const char *pathname, int flags, struct path *path)
>>>> +{
>>>> +	static DEFINE_MUTEX(autofs_oz);
>>>> +	int err;
>>>> +
>>>> +	mutex_lock(&autofs_oz);
>>>> +	autofs_tmp_oz = current;
>>>> +	err = kern_path(pathname, flags, path);
>>>> +	autofs_tmp_oz = NULL;
>>>> +	mutex_unlock(&autofs_oz);
>>>> +	return err;
>>>> +}
>>>> +
>>>
>>> It's simple enough but does look like it will attract criticism as being
>>> a hack!
>>>
>>> The kern_path_locked() function is very similar to what was originally
>>> done, along with code to look down the mount stack (rather than up the
>>> way it does now) to get the mount point. In this case, to be valid the
>>> dentry can't be a symlink so that fits kern_path_locked() too.
>>
>> Oh wait, that __lookup_hash() tries too hard to resolve the dentry,
>> that won't quite work, and maybe d_lookup() can't be used safely in
>> this context either ....
> 
> Umm .. d_lookup() does look ok so maybe path_parentat() + d_lookup()
> would be ok.

Double Umm ... with the patch above kern_path() with flags 0 or
LOOKUP_FOLLOW should get either EISDIR or ENOENT ... maybe I should
think occasionally !!  

> 
>>
>>>
>>> So maybe it is worth going back to the way it was in the beginning and
>>> be done with it .... OTOH Al must have had a reason for changing the
>>> way it was done that I didn't get.
>>>
>>>>  /* Find the topmost mount satisfying test() */
>>>>  static int find_autofs_mount(const char *pathname,
>>>>  			     struct path *res,
>>>> @@ -209,7 +223,8 @@ static int find_autofs_mount(const char *pathname,
>>>>  	struct path path;
>>>>  	int err;
>>>>  
>>>> -	err = kern_path_mountpoint(AT_FDCWD, pathname, &path, 0);
>>>> +	err = kern_path_oz(pathname, 0, &path);
>>>> +
>>>>  	if (err)
>>>>  		return err;
>>>>  	err = -ENOENT;
>>>> @@ -552,8 +567,7 @@ static int autofs_dev_ioctl_ismountpoint(struct file *fp,
>>>>  
>>>>  	if (!fp || param->ioctlfd == -1) {
>>>>  		if (autofs_type_any(type))
>>>> -			err = kern_path_mountpoint(AT_FDCWD,
>>>> -						   name, &path, LOOKUP_FOLLOW);
>>>> +			err = kern_path_oz(name, LOOKUP_FOLLOW, &path);
>>>>  		else
>>>>  			err = find_autofs_mount(name, &path,
>>>>  						test_by_type, &type);
>>>>
>>>
>>
> 

[toc] | [prev] | [next] | [standalone]

#1718799

[Multipart message — attachments visible in raw view] — view raw

On Wed, Aug 23 2017, Ian Kent wrote:

> On 23/08/17 10:32, Ian Kent wrote:
>> On 23/08/17 09:06, NeilBrown wrote:
>>> On Mon, Aug 21 2017, Ian Kent wrote:
>>>
>>>>>
>>>>> A mount isn't triggered by kern_path(pathname, 0, &path).
>>>>> That '0' would need to include one of
>>>>>   LOOKUP_PARENT | LOOKUP_DIRECTORY |
>>>>>   LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT
>>>>>
>>>>> to trigger an automount (otherwise you just get -EISDIR).
>>>>
>>>> It's perfectly sensible to think that but there is a case where a
>>>> a mount is triggered when using kern_path().
>>>>
>>>> The EISDIR return occurs for positive dentrys, negative dentrys
>>>> will still trigger an automount (which is autofs specific,
>>>> indirect mount map using nobrowse option, the install default).
>>>
>>> Ok, I understand this better now.  This difference between direct and
>>> indirect mounts is slightly awkward. It is visible from user-space, but
>>> not elegant to document.
>>> When you use O_PATH to open a direct automount that has not already been
>>> triggered, the open returns the underlying directory (and fstatfs
>>> confirms that it is AUTOFS_SUPER_MAGIC).  When you use O_PATH on
>>> an indirect automount, it *will* trigger the automount when "nobrowse" is
>>> in effect, but it won't when "browse" is in effect.
>> 
>> That inconsistency has bothered me for quite a while now.
>> 
>> It was carried over from the autofs module behavior when automounting
>> support was added to the VFS. What's worse is it prevents the use of
>> the AT_NO_AUTOMOUNT flag from working properly with fstatat(2) and with
>> statx().
>> 
>> There is some risk in changing that so it does work but it really does
>> need to work to enable userspace to not trigger an automount by using
>> this flag.
>> 
>> So that's (hopefully) going to change soonish, see:
>> http://ozlabs.org/~akpm/mmotm/broken-out/autofs-fix-at_no_automount-not-being-honored.patch
>> 
>> The result should be that stat family calls don't trigger automounts except
>> for fstatat(2) and statx() which will require the AT_NO_AUTOMOUNT flag.
>> 
>>>
>>> So we cannot just say "O_PATH doesn't trigger automounts", which is
>>> essentially what I said in
>>>
>>> https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=97a45d02e6671482e8b2cdcce3951930bf6bdb94
>>>
>>> It might be possible to modify automount so that it was more consistent
>>> - i.e. if the point is triggered by a mkdir has been done, just to the
>>> mkdir.  If it is triggered after a mkdir has been done, do the mount.  I
>>> guess that might be racy, and in any case is hard to justify.
>>>
>>> Maybe I should change it to be about "direct automounts", and add a note
>>> that indirect automounts aren't so predictable.
>> 
>> Right and the semantics should be much more consistent in the near future.
>> I hope (and expect) this semantic change won't cause problems.
>> 
>>>
>>> But back to my original issue of wanting to discard
>>> kern_path_mountpoint, what would you think of the following approach -
>>> slight revised from before.
>>>
>>> Thanks,
>>> NeilBrown
>>>
>>> diff --git a/fs/autofs4/autofs_i.h b/fs/autofs4/autofs_i.h
>>> index beef981aa54f..7663ea82e68d 100644
>>> --- a/fs/autofs4/autofs_i.h
>>> +++ b/fs/autofs4/autofs_i.h
>>> @@ -135,10 +135,13 @@ static inline struct autofs_info *autofs4_dentry_ino(struct dentry *dentry)
>>>  /* autofs4_oz_mode(): do we see the man behind the curtain?  (The
>>>   * processes which do manipulations for us in user space sees the raw
>>>   * filesystem without "magic".)
>>> + * A process performing certain ioctls can get temporary oz status.
>>>   */
>>> +extern struct task_struct *autofs_tmp_oz;
>>>  static inline int autofs4_oz_mode(struct autofs_sb_info *sbi)
>>>  {
>>> -	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp;
>>> +	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp ||
>>> +		autofs_tmp_oz == current;
>>>  }
>>>  
>>>  struct inode *autofs4_get_inode(struct super_block *, umode_t);
>>> diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
>>> index dd9f1bebb5a3..d76401669a20 100644
>>> --- a/fs/autofs4/dev-ioctl.c
>>> +++ b/fs/autofs4/dev-ioctl.c
>>> @@ -200,6 +200,20 @@ static int autofs_dev_ioctl_protosubver(struct file *fp,
>>>  	return 0;
>>>  }
>>>  
>>> +struct task_struct *autofs_tmp_oz;
>>> +int kern_path_oz(const char *pathname, int flags, struct path *path)
>>> +{
>>> +	static DEFINE_MUTEX(autofs_oz);
>>> +	int err;
>>> +
>>> +	mutex_lock(&autofs_oz);
>>> +	autofs_tmp_oz = current;
>>> +	err = kern_path(pathname, flags, path);
>>> +	autofs_tmp_oz = NULL;
>>> +	mutex_unlock(&autofs_oz);
>>> +	return err;
>>> +}
>>> +
>> 
>> It's simple enough but does look like it will attract criticism as being
>> a hack!
>> 
>> The kern_path_locked() function is very similar to what was originally
>> done, along with code to look down the mount stack (rather than up the
>> way it does now) to get the mount point. In this case, to be valid the
>> dentry can't be a symlink so that fits kern_path_locked() too.
>
> Oh wait, that __lookup_hash() tries too hard to resolve the dentry,
> that won't quite work, and maybe d_lookup() can't be used safely in
> this context either ....
>

Why do you think that __look_hash() tries too hard?
It does call into the filesystem ->lookup() if the name isn't in the
cache, which probably isn't strictly needed, but that isn't harmful and
the current code does that.

Some the following seems sensible to me (though I haven't tested it).

Thanks,
NeilBrown


diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
index dd9f1bebb5a3..859c198d0163 100644
--- a/fs/autofs4/dev-ioctl.c
+++ b/fs/autofs4/dev-ioctl.c
@@ -208,12 +208,16 @@ static int find_autofs_mount(const char *pathname,
 {
 	struct path path;
 	int err;
+	struct dentry *de, *parent;
+
+	de = kern_path_locked(pathname, &path);
+	if (IS_ERR(de))
+		return PTR_ERR(de);
+	parent = path.dentry;
+	path.dentry = de;
 
-	err = kern_path_mountpoint(AT_FDCWD, pathname, &path, 0);
-	if (err)
-		return err;
 	err = -ENOENT;
-	while (path.dentry == path.mnt->mnt_root) {
+	do {
 		if (path.dentry->d_sb->s_magic == AUTOFS_SUPER_MAGIC) {
 			if (test(&path, data)) {
 				path_get(&path);
@@ -222,10 +226,11 @@ static int find_autofs_mount(const char *pathname,
 				break;
 			}
 		}
-		if (!follow_up(&path))
-			break;
-	}
+	} while (follow_down_one(&path));
+
 	path_put(&path);
+	inode_unlock(d_inode(parent));
+	dput(parent);
 	return err;
 }
 

[toc] | [prev] | [next] | [standalone]

#1718815

On 24/08/17 11:21, NeilBrown wrote:
> On Wed, Aug 23 2017, Ian Kent wrote:
> 
>> On 23/08/17 10:32, Ian Kent wrote:
>>> On 23/08/17 09:06, NeilBrown wrote:
>>>> On Mon, Aug 21 2017, Ian Kent wrote:
>>>>
>>>>>>
>>>>>> A mount isn't triggered by kern_path(pathname, 0, &path).
>>>>>> That '0' would need to include one of
>>>>>>   LOOKUP_PARENT | LOOKUP_DIRECTORY |
>>>>>>   LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT
>>>>>>
>>>>>> to trigger an automount (otherwise you just get -EISDIR).
>>>>>
>>>>> It's perfectly sensible to think that but there is a case where a
>>>>> a mount is triggered when using kern_path().
>>>>>
>>>>> The EISDIR return occurs for positive dentrys, negative dentrys
>>>>> will still trigger an automount (which is autofs specific,
>>>>> indirect mount map using nobrowse option, the install default).
>>>>
>>>> Ok, I understand this better now.  This difference between direct and
>>>> indirect mounts is slightly awkward. It is visible from user-space, but
>>>> not elegant to document.
>>>> When you use O_PATH to open a direct automount that has not already been
>>>> triggered, the open returns the underlying directory (and fstatfs
>>>> confirms that it is AUTOFS_SUPER_MAGIC).  When you use O_PATH on
>>>> an indirect automount, it *will* trigger the automount when "nobrowse" is
>>>> in effect, but it won't when "browse" is in effect.
>>>
>>> That inconsistency has bothered me for quite a while now.
>>>
>>> It was carried over from the autofs module behavior when automounting
>>> support was added to the VFS. What's worse is it prevents the use of
>>> the AT_NO_AUTOMOUNT flag from working properly with fstatat(2) and with
>>> statx().
>>>
>>> There is some risk in changing that so it does work but it really does
>>> need to work to enable userspace to not trigger an automount by using
>>> this flag.
>>>
>>> So that's (hopefully) going to change soonish, see:
>>> http://ozlabs.org/~akpm/mmotm/broken-out/autofs-fix-at_no_automount-not-being-honored.patch
>>>
>>> The result should be that stat family calls don't trigger automounts except
>>> for fstatat(2) and statx() which will require the AT_NO_AUTOMOUNT flag.
>>>
>>>>
>>>> So we cannot just say "O_PATH doesn't trigger automounts", which is
>>>> essentially what I said in
>>>>
>>>> https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=97a45d02e6671482e8b2cdcce3951930bf6bdb94
>>>>
>>>> It might be possible to modify automount so that it was more consistent
>>>> - i.e. if the point is triggered by a mkdir has been done, just to the
>>>> mkdir.  If it is triggered after a mkdir has been done, do the mount.  I
>>>> guess that might be racy, and in any case is hard to justify.
>>>>
>>>> Maybe I should change it to be about "direct automounts", and add a note
>>>> that indirect automounts aren't so predictable.
>>>
>>> Right and the semantics should be much more consistent in the near future.
>>> I hope (and expect) this semantic change won't cause problems.
>>>
>>>>
>>>> But back to my original issue of wanting to discard
>>>> kern_path_mountpoint, what would you think of the following approach -
>>>> slight revised from before.
>>>>
>>>> Thanks,
>>>> NeilBrown
>>>>
>>>> diff --git a/fs/autofs4/autofs_i.h b/fs/autofs4/autofs_i.h
>>>> index beef981aa54f..7663ea82e68d 100644
>>>> --- a/fs/autofs4/autofs_i.h
>>>> +++ b/fs/autofs4/autofs_i.h
>>>> @@ -135,10 +135,13 @@ static inline struct autofs_info *autofs4_dentry_ino(struct dentry *dentry)
>>>>  /* autofs4_oz_mode(): do we see the man behind the curtain?  (The
>>>>   * processes which do manipulations for us in user space sees the raw
>>>>   * filesystem without "magic".)
>>>> + * A process performing certain ioctls can get temporary oz status.
>>>>   */
>>>> +extern struct task_struct *autofs_tmp_oz;
>>>>  static inline int autofs4_oz_mode(struct autofs_sb_info *sbi)
>>>>  {
>>>> -	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp;
>>>> +	return sbi->catatonic || task_pgrp(current) == sbi->oz_pgrp ||
>>>> +		autofs_tmp_oz == current;
>>>>  }
>>>>  
>>>>  struct inode *autofs4_get_inode(struct super_block *, umode_t);
>>>> diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
>>>> index dd9f1bebb5a3..d76401669a20 100644
>>>> --- a/fs/autofs4/dev-ioctl.c
>>>> +++ b/fs/autofs4/dev-ioctl.c
>>>> @@ -200,6 +200,20 @@ static int autofs_dev_ioctl_protosubver(struct file *fp,
>>>>  	return 0;
>>>>  }
>>>>  
>>>> +struct task_struct *autofs_tmp_oz;
>>>> +int kern_path_oz(const char *pathname, int flags, struct path *path)
>>>> +{
>>>> +	static DEFINE_MUTEX(autofs_oz);
>>>> +	int err;
>>>> +
>>>> +	mutex_lock(&autofs_oz);
>>>> +	autofs_tmp_oz = current;
>>>> +	err = kern_path(pathname, flags, path);
>>>> +	autofs_tmp_oz = NULL;
>>>> +	mutex_unlock(&autofs_oz);
>>>> +	return err;
>>>> +}
>>>> +
>>>
>>> It's simple enough but does look like it will attract criticism as being
>>> a hack!
>>>
>>> The kern_path_locked() function is very similar to what was originally
>>> done, along with code to look down the mount stack (rather than up the
>>> way it does now) to get the mount point. In this case, to be valid the
>>> dentry can't be a symlink so that fits kern_path_locked() too.
>>
>> Oh wait, that __lookup_hash() tries too hard to resolve the dentry,
>> that won't quite work, and maybe d_lookup() can't be used safely in
>> this context either ....
>>
> 
> Why do you think that __look_hash() tries too hard?
> It does call into the filesystem ->lookup() if the name isn't in the
> cache, which probably isn't strictly needed, but that isn't harmful and
> the current code does that.

Only that the execution path can encounter the negative dentry case in
follow_automount().

Also, my saying that the dentry can't be a symlink is wrong.

For the covered fd case it's reasonable but for the mount point check
ioctl the path could be anything, eg. a symlink to an automount dentry,
and I want that ioctl to be callable by anyone.

> 
> Some the following seems sensible to me (though I haven't tested it).
> 
> Thanks,
> NeilBrown
> 
> 
> diff --git a/fs/autofs4/dev-ioctl.c b/fs/autofs4/dev-ioctl.c
> index dd9f1bebb5a3..859c198d0163 100644
> --- a/fs/autofs4/dev-ioctl.c
> +++ b/fs/autofs4/dev-ioctl.c
> @@ -208,12 +208,16 @@ static int find_autofs_mount(const char *pathname,
>  {
>  	struct path path;
>  	int err;
> +	struct dentry *de, *parent;
> +
> +	de = kern_path_locked(pathname, &path);
> +	if (IS_ERR(de))
> +		return PTR_ERR(de);
> +	parent = path.dentry;
> +	path.dentry = de;
>  
> -	err = kern_path_mountpoint(AT_FDCWD, pathname, &path, 0);
> -	if (err)
> -		return err;
>  	err = -ENOENT;
> -	while (path.dentry == path.mnt->mnt_root) {
> +	do {
>  		if (path.dentry->d_sb->s_magic == AUTOFS_SUPER_MAGIC) {
>  			if (test(&path, data)) {
>  				path_get(&path);
> @@ -222,10 +226,11 @@ static int find_autofs_mount(const char *pathname,
>  				break;
>  			}
>  		}
> -		if (!follow_up(&path))
> -			break;
> -	}
> +	} while (follow_down_one(&path));
> +
>  	path_put(&path);
> +	inode_unlock(d_inode(parent));
> +	dput(parent);
>  	return err;
>  }
>  
> 

[toc] | [prev] | [next] | [standalone]

#1718807

[Multipart message — attachments visible in raw view] — view raw

On Wed, Aug 23 2017, Ian Kent wrote:

>
> That inconsistency has bothered me for quite a while now.
>
> It was carried over from the autofs module behavior when automounting
> support was added to the VFS. What's worse is it prevents the use of
> the AT_NO_AUTOMOUNT flag from working properly with fstatat(2) and with
> statx().
>
> There is some risk in changing that so it does work but it really does
> need to work to enable userspace to not trigger an automount by using
> this flag.
>
> So that's (hopefully) going to change soonish, see:
> http://ozlabs.org/~akpm/mmotm/broken-out/autofs-fix-at_no_automount-not-being-honored.patch
>
> The result should be that stat family calls don't trigger automounts except
> for fstatat(2) and statx() which will require the AT_NO_AUTOMOUNT flag.
>

oooh, yes.  That's much better - thanks.

We should make sure that change gets into the man pages...

First however, we should probably correct the man page!
stat.2 says:


  NOTES
       On Linux, lstat() will generally not trigger  automounter
       action,  whereas  stat()  will  (but  see  the description of
       fstatat() AT_NO_AUTOMOUNT fag, above).

which is wrong: lstat and stat treat automounts the same.
@Michael: do you recall why you inserted that text?  The commit message
in commit 1ef5b2805471 ("stat.2: Cosmetic reworking of timestamp
discussion in NOTES") is not very helpful.

I propose correcting to

  NOTES:
      On Linux, lstat() nor stat() act as though AT_NO_AUTOMOUNT was set
      and will not trigger automounter action for direct automount
      points, though they may (prior to 4.14) for indirect automount
      points.


The more precise details, that automount action for indirect automount
points is not triggered when the 'browse' option is used, is probably
not necessary.

Ian: if you agree with that text, and Michael doesn't provide alternate
evidence, I'll submit a formal patch for the man page.... or should we
just wait until the patch actually lands?

Thanks,
NeilBrown

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web