Re: Threads stuck in zap_pid_ns_processes()

From	Guenter Roeck <linux@roeck-us.net>
Newsgroups	linux.kernel
Subject	Re: Threads stuck in zap_pid_ns_processes()
Date	2017-05-11 22:30 +0200
Message-ID	<tG2SJ-2Ew-1@gated-at.bofh.it> (permalink)
References	<tFZUS-Oe-19@gated-at.bofh.it> <tG0ed-V3-5@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Thu, May 11, 2017 at 12:31:21PM -0500, Eric W. Biederman wrote:
> Guenter Roeck <linux@roeck-us.net> writes:
> 
> > Hi all,
> >
> > the test program attached below almost always results in one of the child
> > processes being stuck in zap_pid_ns_processes(). When this happens, I can
> > see from test logs that nr_hashed == 2 and init_pids==1, but there is only
> > a single thread left in the pid namespace (the one that is stuck).
> > Traceback from /proc/<pid>/stack is
> >
> > [<ffffffff811c385e>] zap_pid_ns_processes+0x1ee/0x2a0
> > [<ffffffff810c1ba4>] do_exit+0x10d4/0x1330
> > [<ffffffff810c1ee6>] do_group_exit+0x86/0x130
> > [<ffffffff810d4347>] get_signal+0x367/0x8a0
> > [<ffffffff81046e73>] do_signal+0x83/0xb90
> > [<ffffffff81004475>] exit_to_usermode_loop+0x75/0xc0
> > [<ffffffff810055b6>] syscall_return_slowpath+0xc6/0xd0
> > [<ffffffff81ced488>] entry_SYSCALL_64_fastpath+0xab/0xad
> > [<ffffffffffffffff>] 0xffffffffffffffff
> >
> > After 120 seconds, I get the "hung task" message.
> >
> > Example from v4.11:
> >
> > ...
> > [ 3263.379545] INFO: task clone:27910 blocked for more than 120 seconds.
> > [ 3263.379561]       Not tainted 4.11.0+ #1
> > [ 3263.379569] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 3263.379577] clone           D    0 27910  27909 0x00000000
> > [ 3263.379587] Call Trace:
> > [ 3263.379608]  __schedule+0x677/0xda0
> > [ 3263.379621]  ? pci_mmcfg_check_reserved+0xc0/0xc0
> > [ 3263.379634]  ? task_stopped_code+0x70/0x70
> > [ 3263.379643]  schedule+0x4d/0xd0
> > [ 3263.379653]  zap_pid_ns_processes+0x1ee/0x2a0
> > [ 3263.379659]  ? copy_pid_ns+0x4d0/0x4d0
> > [ 3263.379670]  do_exit+0x10d4/0x1330
> > ...
> >
> > The problem is seen in all kernels up to v4.11.
> >
> > Any idea what might be going on and how to fix the problem ?
> 
> Let me see.  Reading the code it looks like we have three tasks
> let's call them main, child1, and child2.
> 
> child1 and child2 are started using CLONE_THREAD and are
> thus clones of one another.
> 
> child2 exits first but is ptraced by main so is not reaped.
>        Further child2 calls do_group_exit forcing child1 to
>        exit making for fun races.
> 
>        A ptread_exit() or syscall(SYS_exit, 0); would skip
>        the group exit and make the window larger.
> 
> child1 exits next and calls zap_pid_ns_processes and is
>        waiting for child2 to be reaped by main.
> 
> main is just sitting around doing nothing for 3600 seconds
> not reaping anyone.
> 
> I would expect that when main exits everything would be cleaned up
> and the only real issue is that we have a hung task warning.
> 
> Does everything cleanup when main exits?
> 

As an add-on to my previous mail: I added a function to count
the number of threads in the pid namespace, using next_pidmap().
Even though nr_hashed == 2, only the hanging thread is still
present.

Is there maybe a better way to terminate the wait loop than
with "if (pid_ns->nr_hashed == init_pids)" ?

Thanks,
Guenter

Thread

Threads stuck in zap_pid_ns_processes() Guenter Roeck <linux@roeck-us.net> - 2017-05-11 19:20 +0200
  Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-11 19:40 +0200
    Re: Threads stuck in zap_pid_ns_processes() Guenter Roeck <linux@roeck-us.net> - 2017-05-11 20:40 +0200
      Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-11 22:30 +0200
        Re: Threads stuck in zap_pid_ns_processes() Guenter Roeck <linux@roeck-us.net> - 2017-05-11 22:50 +0200
          Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-11 23:50 +0200
    Re: Threads stuck in zap_pid_ns_processes() Guenter Roeck <linux@roeck-us.net> - 2017-05-11 22:30 +0200
      Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-11 23:40 +0200
        Re: Threads stuck in zap_pid_ns_processes() Guenter Roeck <linux@roeck-us.net> - 2017-05-12 00:50 +0200
          Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-12 01:30 +0200
            Re: Threads stuck in zap_pid_ns_processes() Vovo Yang <vovoy@google.com> - 2017-05-12 11:40 +0200
              Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-12 15:40 +0200
                Re: Threads stuck in zap_pid_ns_processes() Guenter Roeck <linux@roeck-us.net> - 2017-05-12 19:00 +0200
                Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-12 19:40 +0200
                [REVIEW][PATCH] pid_ns: Sleep in TASK_INTERRUPTIBLE in zap_pid_ns_processes ebiederm@xmission.com (Eric W. Biederman) - 2017-05-12 20:10 +0200
                Re: [REVIEW][PATCH] pid_ns: Sleep in TASK_INTERRUPTIBLE in  zap_pid_ns_processes Guenter Roeck <linux@roeck-us.net> - 2017-05-12 21:40 +0200
                Re: Threads stuck in zap_pid_ns_processes() Guenter Roeck <linux@roeck-us.net> - 2017-05-12 21:50 +0200
                Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-12 22:20 +0200
                Re: Threads stuck in zap_pid_ns_processes() Guenter Roeck <linux@roeck-us.net> - 2017-05-13 16:40 +0200
                Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-13 20:30 +0200
          Re: Threads stuck in zap_pid_ns_processes() ebiederm@xmission.com (Eric W. Biederman) - 2017-05-12 05:50 +0200

csiph-web