Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount completion

From	Jaegeuk Kim <jaegeuk@kernel.org>
Newsgroups	linux.kernel
Subject	Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount completion
Date	2017-09-15 02:20 +0200
Message-ID	<upMwp-3AN-7@gated-at.bofh.it> (permalink)
References	(3 earlier) <uppzP-5zs-1@gated-at.bofh.it> <upqYV-6zf-1@gated-at.bofh.it> <uprih-6FS-3@gated-at.bofh.it> <upHdo-aN-23@gated-at.bofh.it> <upHQ5-Dd-5@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On 09/14, Jaegeuk Kim wrote:
> On 09/14, Al Viro wrote:
> > On Thu, Sep 14, 2017 at 02:30:17AM +0100, Al Viro wrote:
> > > On Wed, Sep 13, 2017 at 06:10:48PM -0700, Jaegeuk Kim wrote:
> > > 
> > > > Android triggers umount(2) by init process, which is definitely not a kernel
> > > > thread. But, we've seen some kernel panics which say umount(2) was succeeded,
> > > > but ext4 triggered a kernel panic due to EIO after then like below. I'm also
> > > > not sure task_work_run() would be also safe enoughly. May I ask where I can
> > > > find sys_umount() calls task_work_run()?
> > > 
> > > ret_{fast,slow}_syscall ->
> > > 	slow_work_pending ->
> > > 		do_work_pending() ->
> > > 			tracehook_notify_resume() ->
> > > 				task_work_run()
> > > 
> > > It's not sys_umount() (or any other sys_...()) - it's syscall dispatcher after
> > > having called one of those and before returning to userland.  What is guaranteed
> > > is that after successful task_work_add() the damn thing will be run in context
> > > of originating process before it returns from syscall.  So any subsequent
> > > syscalls from that process are guaranteed to happen after the work has run.
> > > The same happens if the process exits rather than returns to userland (do_exit() ->
> > > exit_task_work() -> task_work_run()), but for that you would need it to die in
> > > umount(2) (e.g. get kill -9 delivered on the way out).
> > > 
> > > Please, check if you are seeing task_work_add() failure in there and if you do,
> > > I would like to see a stack trace.  IOW, slap WARN_ON(1); right after
> > >                         if (!task_work_add(task, &mnt->mnt_rcu, true))
> > >                                 return;
> > > and see what (if anything) gets printed.
> > 
> > AFAICS, for task_work_add() to fail here we need a final mntput() to be run
> > in context of a thread that already had exit_signals() run *and* subsequent
> > task_work_run() run to completion (with all pending callbacks executed, along
> > with all callbacks added by those, etc.)
> > 
> > For that to have happened during umount(2) we would've needed
> > 	* killing signal delivered while going through the syscall
> > 	* final mntput() to have been done *NOT* from sys_umount() (otherwise
> > the work would've been added before we got to exit_signals())
> > 	* final mntput() to have been done *NOT* from any task_work callbacks
> > (otherwise it would've been added before we'd observed a combination of empty
> > list of pending work with PF_EXITING)
> > 
> > I really want to see the stack trace of that failing task_work_add(), if that's
> > what actually happens there.  What kind of a reproducer do you have for that?
> 
> I've got this error from Android user, so there's no reproducer unfortunately.
> So, I wrote a script capturing WARN_ON after reboot running at every minute, but
> couldn't have got the error since yesterday so far.

Instead, I put more traces in the reboot procedure, and got a clue to suspect
the below flow.

delayed_fput()                 init
                               - umount
 - mntput()
 - mntput_no_expire()            - mntput_no_expire()
                                 - mnt_add_count(-1);
                                 - mnt_get_count() return;
                                 - return 0;
 - mnt_add_count(-1);
 - delayed_mntput_work
                               - device_shutdown
 - ext4_put_super()
 - EIO

Does this make any sense?

Thanks,

Thread

[PATCH] vfs: introduce UMOUNT_WAIT which waits for umount completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-13 22:20 +0200
  Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Al Viro <viro@ZenIV.linux.org.uk> - 2017-09-14 01:10 +0200
    Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-14 01:40 +0200
      Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Al Viro <viro@ZenIV.linux.org.uk> - 2017-09-14 01:50 +0200
        Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-14 03:20 +0200
          Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Al Viro <viro@ZenIV.linux.org.uk> - 2017-09-14 03:40 +0200
            Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Al Viro <viro@ZenIV.linux.org.uk> - 2017-09-14 20:40 +0200
              Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-14 21:20 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-15 02:20 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Al Viro <viro@ZenIV.linux.org.uk> - 2017-09-15 04:10 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-15 05:50 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Al Viro <viro@ZenIV.linux.org.uk> - 2017-09-15 06:30 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-15 20:50 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Theodore Ts'o <tytso@mit.edu> - 2017-09-16 00:20 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-16 01:30 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Al Viro <viro@ZenIV.linux.org.uk> - 2017-09-16 01:50 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-19 18:00 +0200
                Re: [PATCH] vfs: introduce UMOUNT_WAIT which waits for umount completion Amir Goldstein <amir73il@gmail.com> - 2017-09-16 09:20 +0200
  Re: [PATCH v2] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-20 19:40 +0200
    Re: [PATCH v2] vfs: introduce UMOUNT_WAIT which waits for umount  completion Al Viro <viro@ZenIV.linux.org.uk> - 2017-09-20 20:40 +0200
      Re: [PATCH v2] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-21 02:40 +0200
        Re: [PATCH v2] vfs: introduce UMOUNT_WAIT which waits for umount  completion Al Viro <viro@ZenIV.linux.org.uk> - 2017-09-21 04:50 +0200
          Re: [PATCH v2] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-21 07:10 +0200
            Re: [PATCH v2] vfs: introduce UMOUNT_WAIT which waits for umount  completion Theodore Ts'o <tytso@mit.edu> - 2017-09-21 17:00 +0200
              Re: [PATCH v2] vfs: introduce UMOUNT_WAIT which waits for umount  completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-21 19:20 +0200
    Re: [PATCH v3] vfs: introduce UMOUNT_WAIT to wait for  delayed_fput/mntput completion Jaegeuk Kim <jaegeuk@kernel.org> - 2017-09-21 20:30 +0200

csiph-web