Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1309190 > unrolled thread

regression 4.4: deadlock in with cgroup percpu_rwsem

Back to article view | Back to linux.kernel

Contents

  regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-14 12:30 +0100
    Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-14 14:40 +0100
    Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-14 15:10 +0100
      Re: regression 4.4: deadlock in with cgroup percpu_rwsem Nikolay Borisov <kernel@kyup.com> - 2016-01-14 15:30 +0100
        Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-14 18:20 +0100
    Re: regression 4.4: deadlock in with cgroup percpu_rwsem Nikolay Borisov <kernel@kyup.com> - 2016-01-14 15:10 +0100
    Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-14 21:00 +0100
      Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-15 08:40 +0100
        Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-15 16:20 +0100
          Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-18 19:40 +0100
            Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-18 19:50 +0100
              Re: regression 4.4: deadlock in with cgroup percpu_rwsem Heiko Carstens <heiko.carstens@de.ibm.com> - 2016-01-19 11:00 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-19 20:40 +0100
                  Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-19 20:40 +0100
                    Re: regression 4.4: deadlock in with cgroup percpu_rwsem Heiko Carstens <heiko.carstens@de.ibm.com> - 2016-01-20 08:10 +0100
                      Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-20 11:20 +0100
                        Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-20 11:40 +0100
                          Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-20 11:50 +0100
                            Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-20 16:40 +0100
                              Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-20 17:10 +0100
                                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-20 17:50 +0100
                                  Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-20 18:00 +0100
                              Re: regression 4.4: deadlock in with cgroup percpu_rwsem "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-01-23 03:10 +0100
                                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christoph Hellwig <hch@lst.de> - 2016-01-25 09:50 +0100
                                  Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-25 20:40 +0100
                                    Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christoph Hellwig <hch@lst.de> - 2016-01-26 16:00 +0100
                                      Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-26 16:30 +0100
                                        Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christoph Hellwig <hch@lst.de> - 2016-01-26 17:50 +0100
                          Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-20 12:00 +0100
                            Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-21 09:30 +0100
                              Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-21 10:30 +0100
        Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-15 17:50 +0100
          [PATCH cgroup/for-4.5-fixes] cpuset: make mm migration asynchronous Tejun Heo <tj@kernel.org> - 2016-01-19 18:20 +0100
            Re: [PATCH cgroup/for-4.5-fixes] cpuset: make mm migration  asynchronous Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-22 15:30 +0100
              Re: [PATCH cgroup/for-4.5-fixes] cpuset: make mm migration  asynchronous Tejun Heo <tj@kernel.org> - 2016-01-22 16:30 +0100
                Re: [PATCH cgroup/for-4.5-fixes] cpuset: make mm migration  asynchronous Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-22 16:50 +0100
                  Re: [PATCH cgroup/for-4.5-fixes] cpuset: make mm migration  asynchronous Tejun Heo <tj@kernel.org> - 2016-01-22 16:50 +0100
            Re: [PATCH cgroup/for-4.5-fixes] cpuset: make mm migration  asynchronous Tejun Heo <tj@kernel.org> - 2016-01-22 16:30 +0100
        [PATCH 1/2] cgroup: make sure a parent css isn't offlined before its  children Tejun Heo <tj@kernel.org> - 2016-01-21 21:40 +0100
          [PATCH 2/2] cgroup: make sure a parent css isn't freed before its  children Tejun Heo <tj@kernel.org> - 2016-01-21 21:40 +0100
            [PATCH v2 2/2] cgroup: make sure a parent css isn't freed before its  children Tejun Heo <tj@kernel.org> - 2016-01-22 16:50 +0100
          Re: [PATCH 1/2] cgroup: make sure a parent css isn't offlined before  its children Tejun Heo <tj@kernel.org> - 2016-01-21 22:30 +0100
            Re: [PATCH 1/2] cgroup: make sure a parent css isn't offlined before  its children Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-22 09:20 +0100
          Re: [PATCH 1/2] cgroup: make sure a parent css isn't offlined before  its children Peter Zijlstra <peterz@infradead.org> - 2016-01-21 22:30 +0100
          [PATCH v2 1/2] cgroup: make sure a parent css isn't offlined before  its children Tejun Heo <tj@kernel.org> - 2016-01-22 16:50 +0100

Page 1 of 3  [1] 2 3  Next page →

#1309190 — regression 4.4: deadlock in with cgroup percpu_rwsem

Folks,

With 4.4 I can easily bring the system into a hang like situation by
putting stress on the cgroup_threadgroup rwsem. (e.g. starting/stopping
kvm guests via libvirt and many vCPUs). Here is my preliminary analysis:

When the hang happens, the system is idle for all CPUs. There are some
processes waiting for the cgroup_thread_rwsem, e.g.

crash> bt 87399
PID: 87399  TASK: faef084998        CPU: 59  COMMAND: "systemd-udevd"
 #0 [f9e762fc88] __schedule at 83b2cc
 #1 [f9e762fcf0] schedule at 83ba26
 #2 [f9e762fd08] rwsem_down_read_failed at 83fb64
 #3 [f9e762fd68] percpu_down_read at 1bdf56
 #4 [f9e762fdd0] exit_signals at 1742ae
 #5 [f9e762fe00] do_exit at 163be0
 #6 [f9e762fe60] do_group_exit at 165c62
 #7 [f9e762fe90] __wake_up_parent at 165d00
 #8 [f9e762fea8] system_call at 842386

of course, any new process would wait for the same lock during fork.

Looking at the rwsem, while all CPUs are idle, it appears that the lock
is taken for write:

crash> print /x cgroup_threadgroup_rwsem.rw_sem
$8 = {
  count = 0xfffffffe00000001, 
[..]
  owner = 0xfabf28c998, 
}

Looking at the owner field:

crash> bt 0xfabf28c998
PID: 11867  TASK: fabf28c998        CPU: 42  COMMAND: "libvirtd"
 #0 [fadeccb5e8] __schedule at 83b2cc
 #1 [fadeccb650] schedule at 83ba26
 #2 [fadeccb668] schedule_timeout at 8403c6
 #3 [fadeccb748] wait_for_common at 83c850
 #4 [fadeccb7b8] flush_work at 18064a
 #5 [fadeccb8d8] lru_add_drain_all at 2abd10
 #6 [fadeccb938] migrate_prep at 309ed2
 #7 [fadeccb950] do_migrate_pages at 2f7644
 #8 [fadeccb9f0] cpuset_migrate_mm at 220848
 #9 [fadeccba58] cpuset_attach at 223248
#10 [fadeccbaa0] cgroup_taskset_migrate at 21a678
#11 [fadeccbaf8] cgroup_migrate at 21a942
#12 [fadeccbba0] cgroup_attach_task at 21ab8a
#13 [fadeccbc18] __cgroup_procs_write at 21affa
#14 [fadeccbc98] cgroup_file_write at 216be0
#15 [fadeccbd08] kernfs_fop_write at 3aa088
#16 [fadeccbd50] __vfs_write at 319782
#17 [fadeccbe08] vfs_write at 31a1ac
#18 [fadeccbe68] sys_write at 31af06
#19 [fadeccbea8] system_call at 842386
 PSW:  0705100180000000 000003ff9438f9f0 (user space)

it appears that the write holder scheduled away and waits
for a completion. Now what happens is, that the write lock
holder finally calls flush_work for the lru_add_drain_all
work. 

As far as I can see, this work is now tries to create a new kthread
and waits for that, as the backtrace for the kworker on that cpu has:

PID: 81913  TASK: fab5356220        CPU: 42  COMMAND: "kworker/42:2"
 #0 [fadd6d7998] __schedule at 83b2cc
 #1 [fadd6d7a00] schedule at 83ba26
 #2 [fadd6d7a18] schedule_timeout at 8403c6
 #3 [fadd6d7af8] wait_for_common at 83c850
 #4 [fadd6d7b68] wait_for_completion_killable at 83c996
 #5 [fadd6d7b88] kthread_create_on_node at 1876a4
 #6 [fadd6d7cc0] create_worker at 17d7fa
 #7 [fadd6d7d30] worker_thread at 17fff0
 #8 [fadd6d7da0] kthread at 187884
 #9 [fadd6d7ea8] kernel_thread_starter at 842552

Problem is that kthreadd then needs the cgroup lock for reading,
while libvirtd still has the lock for writing.

crash> bt 0xfaf031e220
PID: 2      TASK: faf031e220        CPU: 40  COMMAND: "kthreadd"
 #0 [faf034bad8] __schedule at 83b2cc
 #1 [faf034bb40] schedule at 83ba26
 #2 [faf034bb58] rwsem_down_read_failed at 83fb64
 #3 [faf034bbb8] percpu_down_read at 1bdf56
 #4 [faf034bc20] copy_process at 15eab6
 #5 [faf034bd08] _do_fork at 160430
 #6 [faf034bdd0] kernel_thread at 160a82
 #7 [faf034be30] kthreadd at 188580
 #8 [faf034bea8] kernel_thread_starter at 842552

BANG.kthreadd waits for the lock that libvirtd hold, and libvirtd waits
for kthreadd to finish some task

Reverting 001dac627ff374 ("locking/percpu-rwsem: Make use of the rcu_sync 
infrastructure") does not help, so it does not seem to be related to the
rcu_sync rework.

Any ideas, questions (dump is still available)

PS: not sure if lockdep could detect such a situation. it is running but silent.


Christian

[toc] | [next] | [standalone]

#1309286

On 01/14/2016 12:19 PM, Christian Borntraeger wrote:
> Folks,


FWIW, it _LOOKS_ like it was introduced between 4.4-rc4 and 4.4-rc5


> 
> With 4.4 I can easily bring the system into a hang like situation by
> putting stress on the cgroup_threadgroup rwsem. (e.g. starting/stopping
> kvm guests via libvirt and many vCPUs). Here is my preliminary analysis:
> 
> When the hang happens, the system is idle for all CPUs. There are some
> processes waiting for the cgroup_thread_rwsem, e.g.
> 
> crash> bt 87399
> PID: 87399  TASK: faef084998        CPU: 59  COMMAND: "systemd-udevd"
>  #0 [f9e762fc88] __schedule at 83b2cc
>  #1 [f9e762fcf0] schedule at 83ba26
>  #2 [f9e762fd08] rwsem_down_read_failed at 83fb64
>  #3 [f9e762fd68] percpu_down_read at 1bdf56
>  #4 [f9e762fdd0] exit_signals at 1742ae
>  #5 [f9e762fe00] do_exit at 163be0
>  #6 [f9e762fe60] do_group_exit at 165c62
>  #7 [f9e762fe90] __wake_up_parent at 165d00
>  #8 [f9e762fea8] system_call at 842386
> 
> of course, any new process would wait for the same lock during fork.
> 
> Looking at the rwsem, while all CPUs are idle, it appears that the lock
> is taken for write:
> 
> crash> print /x cgroup_threadgroup_rwsem.rw_sem
> $8 = {
>   count = 0xfffffffe00000001, 
> [..]
>   owner = 0xfabf28c998, 
> }
> 
> Looking at the owner field:
> 
> crash> bt 0xfabf28c998
> PID: 11867  TASK: fabf28c998        CPU: 42  COMMAND: "libvirtd"
>  #0 [fadeccb5e8] __schedule at 83b2cc
>  #1 [fadeccb650] schedule at 83ba26
>  #2 [fadeccb668] schedule_timeout at 8403c6
>  #3 [fadeccb748] wait_for_common at 83c850
>  #4 [fadeccb7b8] flush_work at 18064a
>  #5 [fadeccb8d8] lru_add_drain_all at 2abd10
>  #6 [fadeccb938] migrate_prep at 309ed2
>  #7 [fadeccb950] do_migrate_pages at 2f7644
>  #8 [fadeccb9f0] cpuset_migrate_mm at 220848
>  #9 [fadeccba58] cpuset_attach at 223248
> #10 [fadeccbaa0] cgroup_taskset_migrate at 21a678
> #11 [fadeccbaf8] cgroup_migrate at 21a942
> #12 [fadeccbba0] cgroup_attach_task at 21ab8a
> #13 [fadeccbc18] __cgroup_procs_write at 21affa
> #14 [fadeccbc98] cgroup_file_write at 216be0
> #15 [fadeccbd08] kernfs_fop_write at 3aa088
> #16 [fadeccbd50] __vfs_write at 319782
> #17 [fadeccbe08] vfs_write at 31a1ac
> #18 [fadeccbe68] sys_write at 31af06
> #19 [fadeccbea8] system_call at 842386
>  PSW:  0705100180000000 000003ff9438f9f0 (user space)
> 
> it appears that the write holder scheduled away and waits
> for a completion. Now what happens is, that the write lock
> holder finally calls flush_work for the lru_add_drain_all
> work. 
> 
> As far as I can see, this work is now tries to create a new kthread
> and waits for that, as the backtrace for the kworker on that cpu has:
> 
> PID: 81913  TASK: fab5356220        CPU: 42  COMMAND: "kworker/42:2"
>  #0 [fadd6d7998] __schedule at 83b2cc
>  #1 [fadd6d7a00] schedule at 83ba26
>  #2 [fadd6d7a18] schedule_timeout at 8403c6
>  #3 [fadd6d7af8] wait_for_common at 83c850
>  #4 [fadd6d7b68] wait_for_completion_killable at 83c996
>  #5 [fadd6d7b88] kthread_create_on_node at 1876a4
>  #6 [fadd6d7cc0] create_worker at 17d7fa
>  #7 [fadd6d7d30] worker_thread at 17fff0
>  #8 [fadd6d7da0] kthread at 187884
>  #9 [fadd6d7ea8] kernel_thread_starter at 842552
> 
> Problem is that kthreadd then needs the cgroup lock for reading,
> while libvirtd still has the lock for writing.
> 
> crash> bt 0xfaf031e220
> PID: 2      TASK: faf031e220        CPU: 40  COMMAND: "kthreadd"
>  #0 [faf034bad8] __schedule at 83b2cc
>  #1 [faf034bb40] schedule at 83ba26
>  #2 [faf034bb58] rwsem_down_read_failed at 83fb64
>  #3 [faf034bbb8] percpu_down_read at 1bdf56
>  #4 [faf034bc20] copy_process at 15eab6
>  #5 [faf034bd08] _do_fork at 160430
>  #6 [faf034bdd0] kernel_thread at 160a82
>  #7 [faf034be30] kthreadd at 188580
>  #8 [faf034bea8] kernel_thread_starter at 842552
> 
> BANG.kthreadd waits for the lock that libvirtd hold, and libvirtd waits
> for kthreadd to finish some task
> 
> Reverting 001dac627ff374 ("locking/percpu-rwsem: Make use of the rcu_sync 
> infrastructure") does not help, so it does not seem to be related to the
> rcu_sync rework.
> 
> Any ideas, questions (dump is still available)
> 
> PS: not sure if lockdep could detect such a situation. it is running but silent.
> 
> 
> Christian
> 

[toc] | [prev] | [next] | [standalone]

#1309300

On 01/14/2016 03:04 PM, Nikolay Borisov wrote:
> 
> 
> On 01/14/2016 01:19 PM, Christian Borntraeger wrote:
>> Folks,
>>
>> With 4.4 I can easily bring the system into a hang like situation by
>> putting stress on the cgroup_threadgroup rwsem. (e.g. starting/stopping
>> kvm guests via libvirt and many vCPUs). Here is my preliminary analysis:
>>
>> When the hang happens, the system is idle for all CPUs. There are some
>> processes waiting for the cgroup_thread_rwsem, e.g.
>>
>> crash> bt 87399
>> PID: 87399  TASK: faef084998        CPU: 59  COMMAND: "systemd-udevd"
>>  #0 [f9e762fc88] __schedule at 83b2cc
>>  #1 [f9e762fcf0] schedule at 83ba26
>>  #2 [f9e762fd08] rwsem_down_read_failed at 83fb64
>>  #3 [f9e762fd68] percpu_down_read at 1bdf56
>>  #4 [f9e762fdd0] exit_signals at 1742ae
>>  #5 [f9e762fe00] do_exit at 163be0
>>  #6 [f9e762fe60] do_group_exit at 165c62
>>  #7 [f9e762fe90] __wake_up_parent at 165d00
>>  #8 [f9e762fea8] system_call at 842386
>>
>> of course, any new process would wait for the same lock during fork.
>>
>> Looking at the rwsem, while all CPUs are idle, it appears that the lock
>> is taken for write:
>>
>> crash> print /x cgroup_threadgroup_rwsem.rw_sem
>> $8 = {
>>   count = 0xfffffffe00000001, 
>> [..]
>>   owner = 0xfabf28c998, 
>> }
>>
>> Looking at the owner field:
>>
>> crash> bt 0xfabf28c998
>> PID: 11867  TASK: fabf28c998        CPU: 42  COMMAND: "libvirtd"
>>  #0 [fadeccb5e8] __schedule at 83b2cc
>>  #1 [fadeccb650] schedule at 83ba26
>>  #2 [fadeccb668] schedule_timeout at 8403c6
>>  #3 [fadeccb748] wait_for_common at 83c850
>>  #4 [fadeccb7b8] flush_work at 18064a
>>  #5 [fadeccb8d8] lru_add_drain_all at 2abd10
>>  #6 [fadeccb938] migrate_prep at 309ed2
>>  #7 [fadeccb950] do_migrate_pages at 2f7644
>>  #8 [fadeccb9f0] cpuset_migrate_mm at 220848
>>  #9 [fadeccba58] cpuset_attach at 223248
>> #10 [fadeccbaa0] cgroup_taskset_migrate at 21a678
>> #11 [fadeccbaf8] cgroup_migrate at 21a942
>> #12 [fadeccbba0] cgroup_attach_task at 21ab8a
>> #13 [fadeccbc18] __cgroup_procs_write at 21affa
>> #14 [fadeccbc98] cgroup_file_write at 216be0
>> #15 [fadeccbd08] kernfs_fop_write at 3aa088
>> #16 [fadeccbd50] __vfs_write at 319782
>> #17 [fadeccbe08] vfs_write at 31a1ac
>> #18 [fadeccbe68] sys_write at 31af06
>> #19 [fadeccbea8] system_call at 842386
>>  PSW:  0705100180000000 000003ff9438f9f0 (user space)
>>
>> it appears that the write holder scheduled away and waits
>> for a completion. Now what happens is, that the write lock
>> holder finally calls flush_work for the lru_add_drain_all
>> work. 
> 
> So what's happening is that libvirtd wants to move some processes in the
> cgroup subtree and it to the respective cgroup file. So
> cgroup_threadgroup_rwsem is acquired in __cgroup_procs_write, then as
> part of this process the pages for that process have to be migrated,
> hence the do_migrate_pages. And this call chain boils down to calling
> lru_add_drain_cpu on every cpu.
> 
> 
>>
>> As far as I can see, this work is now tries to create a new kthread
>> and waits for that, as the backtrace for the kworker on that cpu has:
>>
>> PID: 81913  TASK: fab5356220        CPU: 42  COMMAND: "kworker/42:2"
>>  #0 [fadd6d7998] __schedule at 83b2cc
>>  #1 [fadd6d7a00] schedule at 83ba26
>>  #2 [fadd6d7a18] schedule_timeout at 8403c6
>>  #3 [fadd6d7af8] wait_for_common at 83c850
>>  #4 [fadd6d7b68] wait_for_completion_killable at 83c996
>>  #5 [fadd6d7b88] kthread_create_on_node at 1876a4
>>  #6 [fadd6d7cc0] create_worker at 17d7fa
>>  #7 [fadd6d7d30] worker_thread at 17fff0
>>  #8 [fadd6d7da0] kthread at 187884
>>  #9 [fadd6d7ea8] kernel_thread_starter at 842552
>>
>> Problem is that kthreadd then needs the cgroup lock for reading,
>> while libvirtd still has the lock for writing.
>>
>> crash> bt 0xfaf031e220
>> PID: 2      TASK: faf031e220        CPU: 40  COMMAND: "kthreadd"
>>  #0 [faf034bad8] __schedule at 83b2cc
>>  #1 [faf034bb40] schedule at 83ba26
>>  #2 [faf034bb58] rwsem_down_read_failed at 83fb64
>>  #3 [faf034bbb8] percpu_down_read at 1bdf56
>>  #4 [faf034bc20] copy_process at 15eab6
>>  #5 [faf034bd08] _do_fork at 160430
>>  #6 [faf034bdd0] kernel_thread at 160a82
>>  #7 [faf034be30] kthreadd at 188580
>>  #8 [faf034bea8] kernel_thread_starter at 842552
>>
>> BANG.kthreadd waits for the lock that libvirtd hold, and libvirtd waits
>> for kthreadd to finish some task
> 
> I don't see percpu_down_read being invoked from copy_process. According
> to LXR, this semaphore is used only in __cgroup_procs_write and
> cgroup_update_dfl_csses. And cgroup_update_dfl_csses is invoked when
> cgroup.subtree_control is written to. And I don't see this happening in
> this call chain.

The callchain is inlined and as follows:


_do_fork
copy_process
threadgroup_change_begin
cgroup_threadgroup_change_begin

[toc] | [prev] | [next] | [standalone]

#1309325

On 01/14/2016 04:08 PM, Christian Borntraeger wrote:
> On 01/14/2016 03:04 PM, Nikolay Borisov wrote:
>>
>>
>> On 01/14/2016 01:19 PM, Christian Borntraeger wrote:
>>> Folks,
>>>
>>> With 4.4 I can easily bring the system into a hang like situation by
>>> putting stress on the cgroup_threadgroup rwsem. (e.g. starting/stopping
>>> kvm guests via libvirt and many vCPUs). Here is my preliminary analysis:
>>>
>>> When the hang happens, the system is idle for all CPUs. There are some
>>> processes waiting for the cgroup_thread_rwsem, e.g.
>>>
>>> crash> bt 87399
>>> PID: 87399  TASK: faef084998        CPU: 59  COMMAND: "systemd-udevd"
>>>  #0 [f9e762fc88] __schedule at 83b2cc
>>>  #1 [f9e762fcf0] schedule at 83ba26
>>>  #2 [f9e762fd08] rwsem_down_read_failed at 83fb64
>>>  #3 [f9e762fd68] percpu_down_read at 1bdf56
>>>  #4 [f9e762fdd0] exit_signals at 1742ae
>>>  #5 [f9e762fe00] do_exit at 163be0
>>>  #6 [f9e762fe60] do_group_exit at 165c62
>>>  #7 [f9e762fe90] __wake_up_parent at 165d00
>>>  #8 [f9e762fea8] system_call at 842386
>>>
>>> of course, any new process would wait for the same lock during fork.
>>>
>>> Looking at the rwsem, while all CPUs are idle, it appears that the lock
>>> is taken for write:
>>>
>>> crash> print /x cgroup_threadgroup_rwsem.rw_sem
>>> $8 = {
>>>   count = 0xfffffffe00000001, 
>>> [..]
>>>   owner = 0xfabf28c998, 
>>> }
>>>
>>> Looking at the owner field:
>>>
>>> crash> bt 0xfabf28c998
>>> PID: 11867  TASK: fabf28c998        CPU: 42  COMMAND: "libvirtd"
>>>  #0 [fadeccb5e8] __schedule at 83b2cc
>>>  #1 [fadeccb650] schedule at 83ba26
>>>  #2 [fadeccb668] schedule_timeout at 8403c6
>>>  #3 [fadeccb748] wait_for_common at 83c850
>>>  #4 [fadeccb7b8] flush_work at 18064a
>>>  #5 [fadeccb8d8] lru_add_drain_all at 2abd10
>>>  #6 [fadeccb938] migrate_prep at 309ed2
>>>  #7 [fadeccb950] do_migrate_pages at 2f7644
>>>  #8 [fadeccb9f0] cpuset_migrate_mm at 220848
>>>  #9 [fadeccba58] cpuset_attach at 223248
>>> #10 [fadeccbaa0] cgroup_taskset_migrate at 21a678
>>> #11 [fadeccbaf8] cgroup_migrate at 21a942
>>> #12 [fadeccbba0] cgroup_attach_task at 21ab8a
>>> #13 [fadeccbc18] __cgroup_procs_write at 21affa
>>> #14 [fadeccbc98] cgroup_file_write at 216be0
>>> #15 [fadeccbd08] kernfs_fop_write at 3aa088
>>> #16 [fadeccbd50] __vfs_write at 319782
>>> #17 [fadeccbe08] vfs_write at 31a1ac
>>> #18 [fadeccbe68] sys_write at 31af06
>>> #19 [fadeccbea8] system_call at 842386
>>>  PSW:  0705100180000000 000003ff9438f9f0 (user space)
>>>
>>> it appears that the write holder scheduled away and waits
>>> for a completion. Now what happens is, that the write lock
>>> holder finally calls flush_work for the lru_add_drain_all
>>> work. 
>>
>> So what's happening is that libvirtd wants to move some processes in the
>> cgroup subtree and it to the respective cgroup file. So
>> cgroup_threadgroup_rwsem is acquired in __cgroup_procs_write, then as
>> part of this process the pages for that process have to be migrated,
>> hence the do_migrate_pages. And this call chain boils down to calling
>> lru_add_drain_cpu on every cpu.
>>
>>
>>>
>>> As far as I can see, this work is now tries to create a new kthread
>>> and waits for that, as the backtrace for the kworker on that cpu has:
>>>
>>> PID: 81913  TASK: fab5356220        CPU: 42  COMMAND: "kworker/42:2"
>>>  #0 [fadd6d7998] __schedule at 83b2cc
>>>  #1 [fadd6d7a00] schedule at 83ba26
>>>  #2 [fadd6d7a18] schedule_timeout at 8403c6
>>>  #3 [fadd6d7af8] wait_for_common at 83c850
>>>  #4 [fadd6d7b68] wait_for_completion_killable at 83c996
>>>  #5 [fadd6d7b88] kthread_create_on_node at 1876a4
>>>  #6 [fadd6d7cc0] create_worker at 17d7fa
>>>  #7 [fadd6d7d30] worker_thread at 17fff0
>>>  #8 [fadd6d7da0] kthread at 187884
>>>  #9 [fadd6d7ea8] kernel_thread_starter at 842552
>>>
>>> Problem is that kthreadd then needs the cgroup lock for reading,
>>> while libvirtd still has the lock for writing.
>>>
>>> crash> bt 0xfaf031e220
>>> PID: 2      TASK: faf031e220        CPU: 40  COMMAND: "kthreadd"
>>>  #0 [faf034bad8] __schedule at 83b2cc
>>>  #1 [faf034bb40] schedule at 83ba26
>>>  #2 [faf034bb58] rwsem_down_read_failed at 83fb64
>>>  #3 [faf034bbb8] percpu_down_read at 1bdf56
>>>  #4 [faf034bc20] copy_process at 15eab6
>>>  #5 [faf034bd08] _do_fork at 160430
>>>  #6 [faf034bdd0] kernel_thread at 160a82
>>>  #7 [faf034be30] kthreadd at 188580
>>>  #8 [faf034bea8] kernel_thread_starter at 842552
>>>
>>> BANG.kthreadd waits for the lock that libvirtd hold, and libvirtd waits
>>> for kthreadd to finish some task
>>
>> I don't see percpu_down_read being invoked from copy_process. According
>> to LXR, this semaphore is used only in __cgroup_procs_write and
>> cgroup_update_dfl_csses. And cgroup_update_dfl_csses is invoked when
>> cgroup.subtree_control is written to. And I don't see this happening in
>> this call chain.
> 
> The callchain is inlined and as follows:
> 
> 
> _do_fork
> copy_process
> threadgroup_change_begin
> cgroup_threadgroup_change_begin

Ah, I see I have missed that one. So essentially what's happening is
that while migrating processes using a gobal rw semaphore essentially
"disables" forking, but in this case in order to finish the migration a
task has to be spawned (the workqueue worker) and this causes the lock.
Such problems were non-existent before the percpu_rwsem rework since the
lock used was a per-threadgroup. Bummer...

[toc] | [prev] | [next] | [standalone]

#1309496

On 01/14/2016 03:27 PM, Nikolay Borisov wrote:
> 
> 
> On 01/14/2016 04:08 PM, Christian Borntraeger wrote:
>> On 01/14/2016 03:04 PM, Nikolay Borisov wrote:
>>>
>>>
>>> On 01/14/2016 01:19 PM, Christian Borntraeger wrote:
>>>> Folks,
>>>>
>>>> With 4.4 I can easily bring the system into a hang like situation by
>>>> putting stress on the cgroup_threadgroup rwsem. (e.g. starting/stopping
>>>> kvm guests via libvirt and many vCPUs). Here is my preliminary analysis:
>>>>
>>>> When the hang happens, the system is idle for all CPUs. There are some
>>>> processes waiting for the cgroup_thread_rwsem, e.g.
>>>>
>>>> crash> bt 87399
>>>> PID: 87399  TASK: faef084998        CPU: 59  COMMAND: "systemd-udevd"
>>>>  #0 [f9e762fc88] __schedule at 83b2cc
>>>>  #1 [f9e762fcf0] schedule at 83ba26
>>>>  #2 [f9e762fd08] rwsem_down_read_failed at 83fb64
>>>>  #3 [f9e762fd68] percpu_down_read at 1bdf56
>>>>  #4 [f9e762fdd0] exit_signals at 1742ae
>>>>  #5 [f9e762fe00] do_exit at 163be0
>>>>  #6 [f9e762fe60] do_group_exit at 165c62
>>>>  #7 [f9e762fe90] __wake_up_parent at 165d00
>>>>  #8 [f9e762fea8] system_call at 842386
>>>>
>>>> of course, any new process would wait for the same lock during fork.
>>>>
>>>> Looking at the rwsem, while all CPUs are idle, it appears that the lock
>>>> is taken for write:
>>>>
>>>> crash> print /x cgroup_threadgroup_rwsem.rw_sem
>>>> $8 = {
>>>>   count = 0xfffffffe00000001, 
>>>> [..]
>>>>   owner = 0xfabf28c998, 
>>>> }
>>>>
>>>> Looking at the owner field:
>>>>
>>>> crash> bt 0xfabf28c998
>>>> PID: 11867  TASK: fabf28c998        CPU: 42  COMMAND: "libvirtd"
>>>>  #0 [fadeccb5e8] __schedule at 83b2cc
>>>>  #1 [fadeccb650] schedule at 83ba26
>>>>  #2 [fadeccb668] schedule_timeout at 8403c6
>>>>  #3 [fadeccb748] wait_for_common at 83c850
>>>>  #4 [fadeccb7b8] flush_work at 18064a
>>>>  #5 [fadeccb8d8] lru_add_drain_all at 2abd10
>>>>  #6 [fadeccb938] migrate_prep at 309ed2
>>>>  #7 [fadeccb950] do_migrate_pages at 2f7644
>>>>  #8 [fadeccb9f0] cpuset_migrate_mm at 220848
>>>>  #9 [fadeccba58] cpuset_attach at 223248
>>>> #10 [fadeccbaa0] cgroup_taskset_migrate at 21a678
>>>> #11 [fadeccbaf8] cgroup_migrate at 21a942
>>>> #12 [fadeccbba0] cgroup_attach_task at 21ab8a
>>>> #13 [fadeccbc18] __cgroup_procs_write at 21affa
>>>> #14 [fadeccbc98] cgroup_file_write at 216be0
>>>> #15 [fadeccbd08] kernfs_fop_write at 3aa088
>>>> #16 [fadeccbd50] __vfs_write at 319782
>>>> #17 [fadeccbe08] vfs_write at 31a1ac
>>>> #18 [fadeccbe68] sys_write at 31af06
>>>> #19 [fadeccbea8] system_call at 842386
>>>>  PSW:  0705100180000000 000003ff9438f9f0 (user space)
>>>>
>>>> it appears that the write holder scheduled away and waits
>>>> for a completion. Now what happens is, that the write lock
>>>> holder finally calls flush_work for the lru_add_drain_all
>>>> work. 
>>>
>>> So what's happening is that libvirtd wants to move some processes in the
>>> cgroup subtree and it to the respective cgroup file. So
>>> cgroup_threadgroup_rwsem is acquired in __cgroup_procs_write, then as
>>> part of this process the pages for that process have to be migrated,
>>> hence the do_migrate_pages. And this call chain boils down to calling
>>> lru_add_drain_cpu on every cpu.
>>>
>>>
>>>>
>>>> As far as I can see, this work is now tries to create a new kthread
>>>> and waits for that, as the backtrace for the kworker on that cpu has:
>>>>
>>>> PID: 81913  TASK: fab5356220        CPU: 42  COMMAND: "kworker/42:2"
>>>>  #0 [fadd6d7998] __schedule at 83b2cc
>>>>  #1 [fadd6d7a00] schedule at 83ba26
>>>>  #2 [fadd6d7a18] schedule_timeout at 8403c6
>>>>  #3 [fadd6d7af8] wait_for_common at 83c850
>>>>  #4 [fadd6d7b68] wait_for_completion_killable at 83c996
>>>>  #5 [fadd6d7b88] kthread_create_on_node at 1876a4
>>>>  #6 [fadd6d7cc0] create_worker at 17d7fa
>>>>  #7 [fadd6d7d30] worker_thread at 17fff0
>>>>  #8 [fadd6d7da0] kthread at 187884
>>>>  #9 [fadd6d7ea8] kernel_thread_starter at 842552
>>>>
>>>> Problem is that kthreadd then needs the cgroup lock for reading,
>>>> while libvirtd still has the lock for writing.
>>>>
>>>> crash> bt 0xfaf031e220
>>>> PID: 2      TASK: faf031e220        CPU: 40  COMMAND: "kthreadd"
>>>>  #0 [faf034bad8] __schedule at 83b2cc
>>>>  #1 [faf034bb40] schedule at 83ba26
>>>>  #2 [faf034bb58] rwsem_down_read_failed at 83fb64
>>>>  #3 [faf034bbb8] percpu_down_read at 1bdf56
>>>>  #4 [faf034bc20] copy_process at 15eab6
>>>>  #5 [faf034bd08] _do_fork at 160430
>>>>  #6 [faf034bdd0] kernel_thread at 160a82
>>>>  #7 [faf034be30] kthreadd at 188580
>>>>  #8 [faf034bea8] kernel_thread_starter at 842552
>>>>
>>>> BANG.kthreadd waits for the lock that libvirtd hold, and libvirtd waits
>>>> for kthreadd to finish some task
>>>
>>> I don't see percpu_down_read being invoked from copy_process. According
>>> to LXR, this semaphore is used only in __cgroup_procs_write and
>>> cgroup_update_dfl_csses. And cgroup_update_dfl_csses is invoked when
>>> cgroup.subtree_control is written to. And I don't see this happening in
>>> this call chain.
>>
>> The callchain is inlined and as follows:
>>
>>
>> _do_fork
>> copy_process
>> threadgroup_change_begin
>> cgroup_threadgroup_change_begin
> 
> Ah, I see I have missed that one. So essentially what's happening is
> that while migrating processes using a gobal rw semaphore essentially
> "disables" forking, but in this case in order to finish the migration a
> task has to be spawned (the workqueue worker) and this causes the lock.
> Such problems were non-existent before the percpu_rwsem rework since the
> lock used was a per-threadgroup. Bummer...

I think the problem was not caused by the percpu_rwsem rework,

instead by

commit c9e75f0492b248aeaa7af8991a6fc9a21506bc96
    cgroup: pids: fix race between cgroup_post_fork() and cgroup_migrate()


which did changes like

-       if (clone_flags & CLONE_THREAD)
-               threadgroup_change_begin(current);
+       threadgroup_change_begin(current);


So we now ALWAYS take the lock, even for new kernel threads, while before
spawning kernel threads ignored cgroups.

Maybe something like (untested, incomplete, white space damaged)

--- a/include/uapi/linux/sched.h
+++ b/include/uapi/linux/sched.h
@@ -21,8 +21,7 @@
 #define CLONE_DETACHED         0x00400000      /* Unused, ignored */
 #define CLONE_UNTRACED         0x00800000      /* set if the tracing process can't force CLONE_PTRACE on this clone */
 #define CLONE_CHILD_SETTID     0x01000000      /* set the TID in the child */
-/* 0x02000000 was previously the unused CLONE_STOPPED (Start in stopped state)
-   and is now available for re-use. */
+#define CLONE_KERNEL           0x02000000      /* Clone kernel thread */
 #define CLONE_NEWUTS           0x04000000      /* New utsname namespace */
 #define CLONE_NEWIPC           0x08000000      /* New ipc namespace */
 #define CLONE_NEWUSER          0x10000000      /* New user namespace */
diff --git a/kernel/fork.c b/kernel/fork.c
index fce002e..c061b5d 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1368,7 +1368,8 @@ static struct task_struct *copy_process(unsigned long clone_flags,
        p->real_start_time = ktime_get_boot_ns();
        p->io_context = NULL;
        p->audit_context = NULL;
-       threadgroup_change_begin(current);
+       if (!(clone_flags & CLONE_KERNEL))
+               threadgroup_change_begin(current);
        cgroup_fork(p);
 #ifdef CONFIG_NUMA
        p->mempolicy = mpol_dup(p->mempolicy);


Oleg?

[toc] | [prev] | [next] | [standalone]

#1309301

On 01/14/2016 01:19 PM, Christian Borntraeger wrote:
> Folks,
> 
> With 4.4 I can easily bring the system into a hang like situation by
> putting stress on the cgroup_threadgroup rwsem. (e.g. starting/stopping
> kvm guests via libvirt and many vCPUs). Here is my preliminary analysis:
> 
> When the hang happens, the system is idle for all CPUs. There are some
> processes waiting for the cgroup_thread_rwsem, e.g.
> 
> crash> bt 87399
> PID: 87399  TASK: faef084998        CPU: 59  COMMAND: "systemd-udevd"
>  #0 [f9e762fc88] __schedule at 83b2cc
>  #1 [f9e762fcf0] schedule at 83ba26
>  #2 [f9e762fd08] rwsem_down_read_failed at 83fb64
>  #3 [f9e762fd68] percpu_down_read at 1bdf56
>  #4 [f9e762fdd0] exit_signals at 1742ae
>  #5 [f9e762fe00] do_exit at 163be0
>  #6 [f9e762fe60] do_group_exit at 165c62
>  #7 [f9e762fe90] __wake_up_parent at 165d00
>  #8 [f9e762fea8] system_call at 842386
> 
> of course, any new process would wait for the same lock during fork.
> 
> Looking at the rwsem, while all CPUs are idle, it appears that the lock
> is taken for write:
> 
> crash> print /x cgroup_threadgroup_rwsem.rw_sem
> $8 = {
>   count = 0xfffffffe00000001, 
> [..]
>   owner = 0xfabf28c998, 
> }
> 
> Looking at the owner field:
> 
> crash> bt 0xfabf28c998
> PID: 11867  TASK: fabf28c998        CPU: 42  COMMAND: "libvirtd"
>  #0 [fadeccb5e8] __schedule at 83b2cc
>  #1 [fadeccb650] schedule at 83ba26
>  #2 [fadeccb668] schedule_timeout at 8403c6
>  #3 [fadeccb748] wait_for_common at 83c850
>  #4 [fadeccb7b8] flush_work at 18064a
>  #5 [fadeccb8d8] lru_add_drain_all at 2abd10
>  #6 [fadeccb938] migrate_prep at 309ed2
>  #7 [fadeccb950] do_migrate_pages at 2f7644
>  #8 [fadeccb9f0] cpuset_migrate_mm at 220848
>  #9 [fadeccba58] cpuset_attach at 223248
> #10 [fadeccbaa0] cgroup_taskset_migrate at 21a678
> #11 [fadeccbaf8] cgroup_migrate at 21a942
> #12 [fadeccbba0] cgroup_attach_task at 21ab8a
> #13 [fadeccbc18] __cgroup_procs_write at 21affa
> #14 [fadeccbc98] cgroup_file_write at 216be0
> #15 [fadeccbd08] kernfs_fop_write at 3aa088
> #16 [fadeccbd50] __vfs_write at 319782
> #17 [fadeccbe08] vfs_write at 31a1ac
> #18 [fadeccbe68] sys_write at 31af06
> #19 [fadeccbea8] system_call at 842386
>  PSW:  0705100180000000 000003ff9438f9f0 (user space)
> 
> it appears that the write holder scheduled away and waits
> for a completion. Now what happens is, that the write lock
> holder finally calls flush_work for the lru_add_drain_all
> work. 

So what's happening is that libvirtd wants to move some processes in the
cgroup subtree and it to the respective cgroup file. So
cgroup_threadgroup_rwsem is acquired in __cgroup_procs_write, then as
part of this process the pages for that process have to be migrated,
hence the do_migrate_pages. And this call chain boils down to calling
lru_add_drain_cpu on every cpu.


> 
> As far as I can see, this work is now tries to create a new kthread
> and waits for that, as the backtrace for the kworker on that cpu has:
> 
> PID: 81913  TASK: fab5356220        CPU: 42  COMMAND: "kworker/42:2"
>  #0 [fadd6d7998] __schedule at 83b2cc
>  #1 [fadd6d7a00] schedule at 83ba26
>  #2 [fadd6d7a18] schedule_timeout at 8403c6
>  #3 [fadd6d7af8] wait_for_common at 83c850
>  #4 [fadd6d7b68] wait_for_completion_killable at 83c996
>  #5 [fadd6d7b88] kthread_create_on_node at 1876a4
>  #6 [fadd6d7cc0] create_worker at 17d7fa
>  #7 [fadd6d7d30] worker_thread at 17fff0
>  #8 [fadd6d7da0] kthread at 187884
>  #9 [fadd6d7ea8] kernel_thread_starter at 842552
> 
> Problem is that kthreadd then needs the cgroup lock for reading,
> while libvirtd still has the lock for writing.
> 
> crash> bt 0xfaf031e220
> PID: 2      TASK: faf031e220        CPU: 40  COMMAND: "kthreadd"
>  #0 [faf034bad8] __schedule at 83b2cc
>  #1 [faf034bb40] schedule at 83ba26
>  #2 [faf034bb58] rwsem_down_read_failed at 83fb64
>  #3 [faf034bbb8] percpu_down_read at 1bdf56
>  #4 [faf034bc20] copy_process at 15eab6
>  #5 [faf034bd08] _do_fork at 160430
>  #6 [faf034bdd0] kernel_thread at 160a82
>  #7 [faf034be30] kthreadd at 188580
>  #8 [faf034bea8] kernel_thread_starter at 842552
> 
> BANG.kthreadd waits for the lock that libvirtd hold, and libvirtd waits
> for kthreadd to finish some task

I don't see percpu_down_read being invoked from copy_process. According
to LXR, this semaphore is used only in __cgroup_procs_write and
cgroup_update_dfl_csses. And cgroup_update_dfl_csses is invoked when
cgroup.subtree_control is written to. And I don't see this happening in
this call chain.

Going from there I'm questioning whether the failure to fork the from
kthreadd is indeed related to the cgroup semaphore. Can you try and
inspect the stacks for process 0xfaf031e220 to see if the address of the
cgroup rwsemaphore can be found there?

> 
> Reverting 001dac627ff374 ("locking/percpu-rwsem: Make use of the rcu_sync 
> infrastructure") does not help, so it does not seem to be related to the
> rcu_sync rework.
> 
> Any ideas, questions (dump is still available)
> 
> PS: not sure if lockdep could detect such a situation. it is running but silent.
> 
> 
> Christian
> 

[toc] | [prev] | [next] | [standalone]

#1309618

Hello,

Thanks a lot for the report and detailed analysis.  Can you please
test whether the following patch fixes the issue?

Thanks.

---
 include/linux/cpuset.h |    6 ++++++
 kernel/cgroup.c        |    2 ++
 kernel/cpuset.c        |   48 +++++++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 51 insertions(+), 5 deletions(-)

--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -137,6 +137,8 @@ static inline void set_mems_allowed(node
 	task_unlock(current);
 }
 
+extern void cpuset_post_attach_flush(void);
+
 #else /* !CONFIG_CPUSETS */
 
 static inline bool cpusets_enabled(void) { return false; }
@@ -243,6 +245,10 @@ static inline bool read_mems_allowed_ret
 	return false;
 }
 
+static inline void cpuset_post_attach_flush(void)
+{
+}
+
 #endif /* !CONFIG_CPUSETS */
 
 #endif /* _LINUX_CPUSET_H */
--- a/kernel/cgroup.c
+++ b/kernel/cgroup.c
@@ -57,6 +57,7 @@
 #include <linux/vmalloc.h> /* TODO: replace with more sophisticated array */
 #include <linux/kthread.h>
 #include <linux/delay.h>
+#include <linux/cpuset.h>
 
 #include <linux/atomic.h>
 
@@ -2739,6 +2740,7 @@ out_unlock_rcu:
 out_unlock_threadgroup:
 	percpu_up_write(&cgroup_threadgroup_rwsem);
 	cgroup_kn_unlock(of->kn);
+	cpuset_post_attach_flush();
 	return ret ?: nbytes;
 }
 
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -287,6 +287,8 @@ static struct cpuset top_cpuset = {
 static DEFINE_MUTEX(cpuset_mutex);
 static DEFINE_SPINLOCK(callback_lock);
 
+static struct workqueue_struct *cpuset_migrate_mm_wq;
+
 /*
  * CPU / memory hotplug is handled asynchronously.
  */
@@ -971,6 +973,23 @@ static int update_cpumask(struct cpuset
 	return 0;
 }
 
+struct cpuset_migrate_mm_work {
+	struct work_struct	work;
+	struct mm_struct	*mm;
+	nodemask_t		from;
+	nodemask_t		to;
+};
+
+static void cpuset_migrate_mm_workfn(struct work_struct *work)
+{
+	struct cpuset_migrate_mm_work *mwork =
+		container_of(work, struct cpuset_migrate_mm_work, work);
+
+	do_migrate_pages(mwork->mm, &mwork->from, &mwork->to, MPOL_MF_MOVE_ALL);
+	mmput(mwork->mm);
+	kfree(mwork);
+}
+
 /*
  * cpuset_migrate_mm
  *
@@ -989,16 +1008,31 @@ static void cpuset_migrate_mm(struct mm_
 							const nodemask_t *to)
 {
 	struct task_struct *tsk = current;
+	struct cpuset_migrate_mm_work *mwork;
 
 	tsk->mems_allowed = *to;
 
-	do_migrate_pages(mm, from, to, MPOL_MF_MOVE_ALL);
+	mwork = kzalloc(sizeof(*mwork), GFP_KERNEL);
+	if (mwork) {
+		mwork->mm = mm;
+		mwork->from = *from;
+		mwork->to = *to;
+		INIT_WORK(&mwork->work, cpuset_migrate_mm_workfn);
+		queue_work(cpuset_migrate_mm_wq, &mwork->work);
+	} else {
+		mmput(mm);
+	}
 
 	rcu_read_lock();
 	guarantee_online_mems(task_cs(tsk), &tsk->mems_allowed);
 	rcu_read_unlock();
 }
 
+void cpuset_post_attach_flush(void)
+{
+	flush_workqueue(cpuset_migrate_mm_wq);
+}
+
 /*
  * cpuset_change_task_nodemask - change task's mems_allowed and mempolicy
  * @tsk: the task to change
@@ -1097,7 +1131,8 @@ static void update_tasks_nodemask(struct
 		mpol_rebind_mm(mm, &cs->mems_allowed);
 		if (migrate)
 			cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
-		mmput(mm);
+		else
+			mmput(mm);
 	}
 	css_task_iter_end(&it);
 
@@ -1545,11 +1580,11 @@ static void cpuset_attach(struct cgroup_
 			 * @old_mems_allowed is the right nodesets that we
 			 * migrate mm from.
 			 */
-			if (is_memory_migrate(cs)) {
+			if (is_memory_migrate(cs))
 				cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
 						  &cpuset_attach_nodemask_to);
-			}
-			mmput(mm);
+			else
+				mmput(mm);
 		}
 	}
 
@@ -2359,6 +2394,9 @@ void __init cpuset_init_smp(void)
 	top_cpuset.effective_mems = node_states[N_MEMORY];
 
 	register_hotmemory_notifier(&cpuset_track_online_nodes_nb);
+
+	cpuset_migrate_mm_wq = alloc_ordered_workqueue("cpuset_migrate_mm", 0);
+	BUG_ON(!cpuset_migrate_mm_wq);
 }
 
 /**

[toc] | [prev] | [next] | [standalone]

#1309929

On 01/14/2016 08:56 PM, Tejun Heo wrote:
> Hello,
> 
> Thanks a lot for the report and detailed analysis.  Can you please
> test whether the following patch fixes the issue?
> 
> Thanks.
> 


Yes, the deadlock is gone and the system is still running.
After some time I had the following WARN in the logs, though.
Not sure yet if that is related.

[25331.763607] DEBUG_LOCKS_WARN_ON(lock->owner != current)
[25331.763630] ------------[ cut here ]------------
[25331.763634] WARNING: at kernel/locking/mutex-debug.c:80
[25331.763637] Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc btrfs xor raid6_pq ghash_s390 prng ecb aes_s390 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common eadm_sch nfsd auth_rpcgss oid_registry nfs_acl lockd vhost_net tun vhost macvtap macvlan grace sunrpc dm_service_time dm_multipath dm_mod autofs4
[25331.763708] CPU: 56 PID: 114657 Comm: systemd-udevd Not tainted 4.4.0+ #91
[25331.763711] task: 000000fadc79de40 ti: 000000f95e7f8000 task.ti: 000000f95e7f8000
[25331.763715] Krnl PSW : 0404c00180000000 00000000001b7f32 (debug_mutex_unlock+0x16a/0x188)
[25331.763726]            R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 EA:3
Krnl GPRS: 0000004c00000037 000000fadc79de40 000000000000002b 0000000000000000
[25331.763732]            000000000028da3c 0000000000000000 000000f95e7fbf08 000000fab8e10df0
[25331.763735]            000000000000005c 000000facc0dc000 000000000000005c 000000000033e14a
[25331.763738]            0700000000000000 000000fab8e10df0 00000000001b7f2e 000000f95e7fbc80
[25331.763746] Krnl Code: 00000000001b7f22: c0200042784c	larl	%r2,a06fba
           00000000001b7f28: c0e50006ad50	brasl	%r14,28d9c8
          #00000000001b7f2e: a7f40001		brc	15,1b7f30
          >00000000001b7f32: a7f4ffe1		brc	15,1b7ef4
           00000000001b7f36: c03000429c9f	larl	%r3,a0b874
           00000000001b7f3c: c0200042783f	larl	%r2,a06fba
           00000000001b7f42: c0e50006ad43	brasl	%r14,28d9c8
           00000000001b7f48: a7f40001		brc	15,1b7f4a
[25331.763795] Call Trace:
[25331.763798] ([<00000000001b7f2e>] debug_mutex_unlock+0x166/0x188)
[25331.763804]  [<0000000000836a08>] __mutex_unlock_slowpath+0xa8/0x190
[25331.763808]  [<000000000033e14a>] seq_read+0x1c2/0x450
[25331.763813]  [<0000000000311e72>] __vfs_read+0x42/0x100
[25331.763818]  [<000000000031284e>] vfs_read+0x76/0x130
[25331.763821]  [<000000000031361e>] SyS_read+0x66/0xd8
[25331.763826]  [<000000000083af06>] system_call+0xd6/0x270
[25331.763829]  [<000003ffae1f19c8>] 0x3ffae1f19c8
[25331.763831] INFO: lockdep is turned off.
[25331.763833] Last Breaking-Event-Address:
[25331.763836]  [<00000000001b7f2e>] debug_mutex_unlock+0x166/0x188
[25331.763839] ---[ end trace 45177640eb39ef44 ]---





> ---
>  include/linux/cpuset.h |    6 ++++++
>  kernel/cgroup.c        |    2 ++
>  kernel/cpuset.c        |   48 +++++++++++++++++++++++++++++++++++++++++++-----
>  3 files changed, 51 insertions(+), 5 deletions(-)
> 
> --- a/include/linux/cpuset.h
> +++ b/include/linux/cpuset.h
> @@ -137,6 +137,8 @@ static inline void set_mems_allowed(node
>  	task_unlock(current);
>  }
> 
> +extern void cpuset_post_attach_flush(void);
> +
>  #else /* !CONFIG_CPUSETS */
> 
>  static inline bool cpusets_enabled(void) { return false; }
> @@ -243,6 +245,10 @@ static inline bool read_mems_allowed_ret
>  	return false;
>  }
> 
> +static inline void cpuset_post_attach_flush(void)
> +{
> +}
> +
>  #endif /* !CONFIG_CPUSETS */
> 
>  #endif /* _LINUX_CPUSET_H */
> --- a/kernel/cgroup.c
> +++ b/kernel/cgroup.c
> @@ -57,6 +57,7 @@
>  #include <linux/vmalloc.h> /* TODO: replace with more sophisticated array */
>  #include <linux/kthread.h>
>  #include <linux/delay.h>
> +#include <linux/cpuset.h>
> 
>  #include <linux/atomic.h>
> 
> @@ -2739,6 +2740,7 @@ out_unlock_rcu:
>  out_unlock_threadgroup:
>  	percpu_up_write(&cgroup_threadgroup_rwsem);
>  	cgroup_kn_unlock(of->kn);
> +	cpuset_post_attach_flush();
>  	return ret ?: nbytes;
>  }
> 
> --- a/kernel/cpuset.c
> +++ b/kernel/cpuset.c
> @@ -287,6 +287,8 @@ static struct cpuset top_cpuset = {
>  static DEFINE_MUTEX(cpuset_mutex);
>  static DEFINE_SPINLOCK(callback_lock);
> 
> +static struct workqueue_struct *cpuset_migrate_mm_wq;
> +
>  /*
>   * CPU / memory hotplug is handled asynchronously.
>   */
> @@ -971,6 +973,23 @@ static int update_cpumask(struct cpuset
>  	return 0;
>  }
> 
> +struct cpuset_migrate_mm_work {
> +	struct work_struct	work;
> +	struct mm_struct	*mm;
> +	nodemask_t		from;
> +	nodemask_t		to;
> +};
> +
> +static void cpuset_migrate_mm_workfn(struct work_struct *work)
> +{
> +	struct cpuset_migrate_mm_work *mwork =
> +		container_of(work, struct cpuset_migrate_mm_work, work);
> +
> +	do_migrate_pages(mwork->mm, &mwork->from, &mwork->to, MPOL_MF_MOVE_ALL);
> +	mmput(mwork->mm);
> +	kfree(mwork);
> +}
> +
>  /*
>   * cpuset_migrate_mm
>   *
> @@ -989,16 +1008,31 @@ static void cpuset_migrate_mm(struct mm_
>  							const nodemask_t *to)
>  {
>  	struct task_struct *tsk = current;
> +	struct cpuset_migrate_mm_work *mwork;
> 
>  	tsk->mems_allowed = *to;
> 
> -	do_migrate_pages(mm, from, to, MPOL_MF_MOVE_ALL);
> +	mwork = kzalloc(sizeof(*mwork), GFP_KERNEL);
> +	if (mwork) {
> +		mwork->mm = mm;
> +		mwork->from = *from;
> +		mwork->to = *to;
> +		INIT_WORK(&mwork->work, cpuset_migrate_mm_workfn);
> +		queue_work(cpuset_migrate_mm_wq, &mwork->work);
> +	} else {
> +		mmput(mm);
> +	}
> 
>  	rcu_read_lock();
>  	guarantee_online_mems(task_cs(tsk), &tsk->mems_allowed);
>  	rcu_read_unlock();
>  }
> 
> +void cpuset_post_attach_flush(void)
> +{
> +	flush_workqueue(cpuset_migrate_mm_wq);
> +}
> +
>  /*
>   * cpuset_change_task_nodemask - change task's mems_allowed and mempolicy
>   * @tsk: the task to change
> @@ -1097,7 +1131,8 @@ static void update_tasks_nodemask(struct
>  		mpol_rebind_mm(mm, &cs->mems_allowed);
>  		if (migrate)
>  			cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
> -		mmput(mm);
> +		else
> +			mmput(mm);
>  	}
>  	css_task_iter_end(&it);
> 
> @@ -1545,11 +1580,11 @@ static void cpuset_attach(struct cgroup_
>  			 * @old_mems_allowed is the right nodesets that we
>  			 * migrate mm from.
>  			 */
> -			if (is_memory_migrate(cs)) {
> +			if (is_memory_migrate(cs))
>  				cpuset_migrate_mm(mm, &oldcs->old_mems_allowed,
>  						  &cpuset_attach_nodemask_to);
> -			}
> -			mmput(mm);
> +			else
> +				mmput(mm);
>  		}
>  	}
> 
> @@ -2359,6 +2394,9 @@ void __init cpuset_init_smp(void)
>  	top_cpuset.effective_mems = node_states[N_MEMORY];
> 
>  	register_hotmemory_notifier(&cpuset_track_online_nodes_nb);
> +
> +	cpuset_migrate_mm_wq = alloc_ordered_workqueue("cpuset_migrate_mm", 0);
> +	BUG_ON(!cpuset_migrate_mm_wq);
>  }
> 
>  /**
> 

[toc] | [prev] | [next] | [standalone]

#1310215

On 01/15/2016 08:30 AM, Christian Borntraeger wrote:
> On 01/14/2016 08:56 PM, Tejun Heo wrote:
>> Hello,
>>
>> Thanks a lot for the report and detailed analysis.  Can you please
>> test whether the following patch fixes the issue?
>>
>> Thanks.
>>
> 
> 
> Yes, the deadlock is gone and the system is still running.
> After some time I had the following WARN in the logs, though.
> Not sure yet if that is related.
> 
> [25331.763607] DEBUG_LOCKS_WARN_ON(lock->owner != current)
> [25331.763630] ------------[ cut here ]------------
> [25331.763634] WARNING: at kernel/locking/mutex-debug.c:80
> [25331.763637] Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc btrfs xor raid6_pq ghash_s390 prng ecb aes_s390 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common eadm_sch nfsd auth_rpcgss oid_registry nfs_acl lockd vhost_net tun vhost macvtap macvlan grace sunrpc dm_service_time dm_multipath dm_mod autofs4
> [25331.763708] CPU: 56 PID: 114657 Comm: systemd-udevd Not tainted 4.4.0+ #91
> [25331.763711] task: 000000fadc79de40 ti: 000000f95e7f8000 task.ti: 000000f95e7f8000
> [25331.763715] Krnl PSW : 0404c00180000000 00000000001b7f32 (debug_mutex_unlock+0x16a/0x188)
> [25331.763726]            R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 EA:3
> Krnl GPRS: 0000004c00000037 000000fadc79de40 000000000000002b 0000000000000000
> [25331.763732]            000000000028da3c 0000000000000000 000000f95e7fbf08 000000fab8e10df0
> [25331.763735]            000000000000005c 000000facc0dc000 000000000000005c 000000000033e14a
> [25331.763738]            0700000000000000 000000fab8e10df0 00000000001b7f2e 000000f95e7fbc80
> [25331.763746] Krnl Code: 00000000001b7f22: c0200042784c	larl	%r2,a06fba
>            00000000001b7f28: c0e50006ad50	brasl	%r14,28d9c8
>           #00000000001b7f2e: a7f40001		brc	15,1b7f30
>           >00000000001b7f32: a7f4ffe1		brc	15,1b7ef4
>            00000000001b7f36: c03000429c9f	larl	%r3,a0b874
>            00000000001b7f3c: c0200042783f	larl	%r2,a06fba
>            00000000001b7f42: c0e50006ad43	brasl	%r14,28d9c8
>            00000000001b7f48: a7f40001		brc	15,1b7f4a
> [25331.763795] Call Trace:
> [25331.763798] ([<00000000001b7f2e>] debug_mutex_unlock+0x166/0x188)
> [25331.763804]  [<0000000000836a08>] __mutex_unlock_slowpath+0xa8/0x190
> [25331.763808]  [<000000000033e14a>] seq_read+0x1c2/0x450
> [25331.763813]  [<0000000000311e72>] __vfs_read+0x42/0x100
> [25331.763818]  [<000000000031284e>] vfs_read+0x76/0x130
> [25331.763821]  [<000000000031361e>] SyS_read+0x66/0xd8
> [25331.763826]  [<000000000083af06>] system_call+0xd6/0x270
> [25331.763829]  [<000003ffae1f19c8>] 0x3ffae1f19c8
> [25331.763831] INFO: lockdep is turned off.
> [25331.763833] Last Breaking-Event-Address:
> [25331.763836]  [<00000000001b7f2e>] debug_mutex_unlock+0x166/0x188
> [25331.763839] ---[ end trace 45177640eb39ef44 ]---
> 

I restarted the test with panic_on_warn. Hopefully I can get a dump to check
which mutex this was.

Christian

[toc] | [prev] | [next] | [standalone]

#1311740

On Fri, Jan 15, 2016 at 04:13:34PM +0100, Christian Borntraeger wrote:
> > Yes, the deadlock is gone and the system is still running.
> > After some time I had the following WARN in the logs, though.
> > Not sure yet if that is related.
> > 
> > [25331.763607] DEBUG_LOCKS_WARN_ON(lock->owner != current)
> > [25331.763630] ------------[ cut here ]------------
> > [25331.763634] WARNING: at kernel/locking/mutex-debug.c:80

> I restarted the test with panic_on_warn. Hopefully I can get a dump to check
> which mutex this was.

Hard to reproduce warnings like this tend to point towards memory
corruption. Someone stepped on the mutex value and tickles the sanity
check.

With lockdep and debugging enabled the mutex gets quite a bit bigger, so
it gets more likely to be hit by 'random' corruption.

The locking in seq_read() seems rather straight forward.

[toc] | [prev] | [next] | [standalone]

#1311744

On 01/18/2016 07:32 PM, Peter Zijlstra wrote:
> On Fri, Jan 15, 2016 at 04:13:34PM +0100, Christian Borntraeger wrote:
>>> Yes, the deadlock is gone and the system is still running.
>>> After some time I had the following WARN in the logs, though.
>>> Not sure yet if that is related.
>>>
>>> [25331.763607] DEBUG_LOCKS_WARN_ON(lock->owner != current)
>>> [25331.763630] ------------[ cut here ]------------
>>> [25331.763634] WARNING: at kernel/locking/mutex-debug.c:80
> 
>> I restarted the test with panic_on_warn. Hopefully I can get a dump to check
>> which mutex this was.
> 
> Hard to reproduce warnings like this tend to point towards memory
> corruption. Someone stepped on the mutex value and tickles the sanity
> check.
> 
> With lockdep and debugging enabled the mutex gets quite a bit bigger, so
> it gets more likely to be hit by 'random' corruption.
> 
> The locking in seq_read() seems rather straight forward.

I was able to reproduce. The dump shows a mutex that has an owner field, which
does not exists as a task so this all looks fishy. The good thing is, that I
can reproduce the issue within some hours. (exact same backtrace). Will add some
more debug data to get a handle where we come from.

[toc] | [prev] | [next] | [standalone]

#1312056

On Mon, Jan 18, 2016 at 07:48:16PM +0100, Christian Borntraeger wrote:
> On 01/18/2016 07:32 PM, Peter Zijlstra wrote:
> > On Fri, Jan 15, 2016 at 04:13:34PM +0100, Christian Borntraeger wrote:
> >>> Yes, the deadlock is gone and the system is still running.
> >>> After some time I had the following WARN in the logs, though.
> >>> Not sure yet if that is related.
> >>>
> >>> [25331.763607] DEBUG_LOCKS_WARN_ON(lock->owner != current)
> >>> [25331.763630] ------------[ cut here ]------------
> >>> [25331.763634] WARNING: at kernel/locking/mutex-debug.c:80
> > 
> >> I restarted the test with panic_on_warn. Hopefully I can get a dump to check
> >> which mutex this was.
> > 
> > Hard to reproduce warnings like this tend to point towards memory
> > corruption. Someone stepped on the mutex value and tickles the sanity
> > check.
> > 
> > With lockdep and debugging enabled the mutex gets quite a bit bigger, so
> > it gets more likely to be hit by 'random' corruption.
> > 
> > The locking in seq_read() seems rather straight forward.
> 
> I was able to reproduce. The dump shows a mutex that has an owner field, which
> does not exists as a task so this all looks fishy. The good thing is, that I
> can reproduce the issue within some hours. (exact same backtrace). Will add some
> more debug data to get a handle where we come from.

Did the owner field show to something that still looks like a task_struct?

[toc] | [prev] | [next] | [standalone]

#1312466

On 01/19/2016 10:55 AM, Heiko Carstens wrote:
> On Mon, Jan 18, 2016 at 07:48:16PM +0100, Christian Borntraeger wrote:
>> On 01/18/2016 07:32 PM, Peter Zijlstra wrote:
>>> On Fri, Jan 15, 2016 at 04:13:34PM +0100, Christian Borntraeger wrote:
>>>>> Yes, the deadlock is gone and the system is still running.
>>>>> After some time I had the following WARN in the logs, though.
>>>>> Not sure yet if that is related.
>>>>>
>>>>> [25331.763607] DEBUG_LOCKS_WARN_ON(lock->owner != current)
>>>>> [25331.763630] ------------[ cut here ]------------
>>>>> [25331.763634] WARNING: at kernel/locking/mutex-debug.c:80
>>>
>>>> I restarted the test with panic_on_warn. Hopefully I can get a dump to check
>>>> which mutex this was.
>>>
>>> Hard to reproduce warnings like this tend to point towards memory
>>> corruption. Someone stepped on the mutex value and tickles the sanity
>>> check.
>>>
>>> With lockdep and debugging enabled the mutex gets quite a bit bigger, so
>>> it gets more likely to be hit by 'random' corruption.
>>>
>>> The locking in seq_read() seems rather straight forward.
>>
>> I was able to reproduce. The dump shows a mutex that has an owner field, which
>> does not exists as a task so this all looks fishy. The good thing is, that I
>> can reproduce the issue within some hours. (exact same backtrace). Will add some
>> more debug data to get a handle where we come from.
> 
> Did the owner field show to something that still looks like a task_struct?

No, its not a task_struct. Activating some more debug information did indeed 
revealed several other issues (overwritten redzones etc). Unfortunately I 
only saw the broken things after the facts, so I do not know which code did that.
When I disabled the cgroup controllers in libvirt I was no longer able to trigger
the bugs. Still trying to narrow things down.

Christian

[toc] | [prev] | [next] | [standalone]

#1312468

Hello,

On Tue, Jan 19, 2016 at 08:36:18PM +0100, Christian Borntraeger wrote:
> No, its not a task_struct. Activating some more debug information did indeed 
> revealed several other issues (overwritten redzones etc). Unfortunately I 
> only saw the broken things after the facts, so I do not know which code did that.
> When I disabled the cgroup controllers in libvirt I was no longer able to trigger
> the bugs. Still trying to narrow things down.

Hmmm... that's worrying.  CONFIG_DEBUG_PAGEALLOC sometimes can catch
these sort of bugs red-handed.  Might worth trying.

Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1312940

On Tue, Jan 19, 2016 at 02:38:45PM -0500, Tejun Heo wrote:
> Hello,
> 
> On Tue, Jan 19, 2016 at 08:36:18PM +0100, Christian Borntraeger wrote:
> > No, its not a task_struct. Activating some more debug information did indeed 
> > revealed several other issues (overwritten redzones etc). Unfortunately I 
> > only saw the broken things after the facts, so I do not know which code did that.
> > When I disabled the cgroup controllers in libvirt I was no longer able to trigger
> > the bugs. Still trying to narrow things down.
> 
> Hmmm... that's worrying.  CONFIG_DEBUG_PAGEALLOC sometimes can catch
> these sort of bugs red-handed.  Might worth trying.

Christian, just to avoid that you get surprised like I did:
CONFIG_DEBUG_PAGEALLOC requires in the meantime an additional kernel
parameter "debug_pagealloc=on" to be active.

That change was introduced a year ago, so it was probably only me who
wasn't aware of that change :)

[toc] | [prev] | [next] | [standalone]

#1313056

On 01/20/2016 08:07 AM, Heiko Carstens wrote:
> On Tue, Jan 19, 2016 at 02:38:45PM -0500, Tejun Heo wrote:
>> Hello,
>>
>> On Tue, Jan 19, 2016 at 08:36:18PM +0100, Christian Borntraeger wrote:
>>> No, its not a task_struct. Activating some more debug information did indeed 
>>> revealed several other issues (overwritten redzones etc). Unfortunately I 
>>> only saw the broken things after the facts, so I do not know which code did that.
>>> When I disabled the cgroup controllers in libvirt I was no longer able to trigger
>>> the bugs. Still trying to narrow things down.
>>
>> Hmmm... that's worrying.  CONFIG_DEBUG_PAGEALLOC sometimes can catch
>> these sort of bugs red-handed.  Might worth trying.
> 
> Christian, just to avoid that you get surprised like I did:
> CONFIG_DEBUG_PAGEALLOC requires in the meantime an additional kernel
> parameter "debug_pagealloc=on" to be active.
> 
> That change was introduced a year ago, so it was probably only me who
> wasn't aware of that change :)

I had CONFIG_DEBUG_PAGEALLOC, but not the command line. :-(

With that enabled I now have:

[  561.043895] Unable to handle kernel pointer dereference in virtual kernel address space
[  561.043902] failing address: 000000fa14b30000 TEID: 000000fa14b30803
[  561.043905] Fault in home space mode while using kernel ASCE.
[  561.043911] AS:0000000000fa5007 R3:000000ff627ff007 S:000000ff62759800 P:000000fa14b30400 
[  561.043953] Oops: 0011 ilc:3 [#1] SMP DEBUG_PAGEALLOC
[  561.043964] Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc btrfs xor raid6_pq ghash_s390 prng ecb aes_s390 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common eadm_sch nfsd auth_rpcgss vhost_net tun oid_registry nfs_acl lockd vhost macvtap macvlan grace sunrpc dm_service_time dm_multipath dm_mod autofs4
[  561.044057] CPU: 52 PID: 215 Comm: ksoftirqd/52 Not tainted 4.4.0+ #94
[  561.044062] task: 000000fa5bc48000 ti: 000000fa5bc50000 task.ti: 000000fa5bc50000
[  561.044066] Krnl PSW : 0704e00180000000 00000000001aa1ee (remove_entity_load_avg+0x1e/0x1b8)
[  561.044080]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 EA:3
Krnl GPRS: 0000000000000000 000000fa0933b3d8 000000fa0b411860 000000fa14b30000
[  561.044087]            00000000001ad750 0000000000000001 0000000000000000 000000000000000a
[  561.044093]            0000000000d28b0c 0000000000c4ba28 0000000000000028 0000000000000140
[  561.044095]            000000fa389f0348 000000000084cfb0 00000000001ad774 000000fa5bc53b88
[  561.044105] Krnl Code: 00000000001aa1dc: c0d0003516ea	larl	%r13,84cfb0
           00000000001aa1e2: e33020780004	lg	%r3,120(%r2)
          #00000000001aa1e8: e30020880004	lg	%r0,136(%r2)
          >00000000001aa1ee: e34030580004	lg	%r4,88(%r3)
           00000000001aa1f4: b9e90014		sgrk	%r1,%r4,%r0
           00000000001aa1f8: ec140095007c	cgij	%r1,0,4,1aa322
           00000000001aa1fe: eb11000a000c	srlg	%r1,%r1,10
           00000000001aa204: ec160013007c	cgij	%r1,0,6,1aa22a
[  561.044170] Call Trace:
[  561.044176] ([<00000000001ad750>] free_fair_sched_group+0x80/0xf8)
[  561.044181]  [<0000000000192656>] free_sched_group+0x2e/0x58
[  561.044187]  [<00000000001ded82>] rcu_process_callbacks+0x3fa/0x928
[  561.044194]  [<00000000001676a4>] __do_softirq+0xd4/0x4b0
[  561.044199]  [<0000000000167abe>] run_ksoftirqd+0x3e/0xa8
[  561.044204]  [<000000000018d5bc>] smpboot_thread_fn+0x16c/0x2a0
[  561.044210]  [<0000000000188704>] kthread+0x10c/0x128
[  561.044216]  [<000000000083d8a2>] kernel_thread_starter+0x6/0xc
[  561.044220]  [<000000000083d89c>] kernel_thread_starter+0x0/0xc
[  561.044223] INFO: lockdep is turned off.
[  561.044225] Last Breaking-Event-Address:
[  561.044230]  [<00000000001ad76e>] free_fair_sched_group+0x9e/0xf8
[  561.044237]  
[  561.044241] Kernel panic - not syncing: Fatal exception in interrupt


Will look into that and see if fixing this makes the problem go away.
(unless somebody else has a quick idea)

Christian

[toc] | [prev] | [next] | [standalone]

#1313092

On Wed, Jan 20, 2016 at 11:15:05AM +0100, Christian Borntraeger wrote:
> [  561.044066] Krnl PSW : 0704e00180000000 00000000001aa1ee (remove_entity_load_avg+0x1e/0x1b8)

> [  561.044176] ([<00000000001ad750>] free_fair_sched_group+0x80/0xf8)
> [  561.044181]  [<0000000000192656>] free_sched_group+0x2e/0x58
> [  561.044187]  [<00000000001ded82>] rcu_process_callbacks+0x3fa/0x928

Urgh,.. lemme stare at that.

[toc] | [prev] | [next] | [standalone]

#1313104

On Wed, Jan 20, 2016 at 11:30:36AM +0100, Peter Zijlstra wrote:
> On Wed, Jan 20, 2016 at 11:15:05AM +0100, Christian Borntraeger wrote:
> > [  561.044066] Krnl PSW : 0704e00180000000 00000000001aa1ee (remove_entity_load_avg+0x1e/0x1b8)
> 
> > [  561.044176] ([<00000000001ad750>] free_fair_sched_group+0x80/0xf8)
> > [  561.044181]  [<0000000000192656>] free_sched_group+0x2e/0x58
> > [  561.044187]  [<00000000001ded82>] rcu_process_callbacks+0x3fa/0x928
> 
> Urgh,.. lemme stare at that.

TJ, is css_offline guaranteed to be called in hierarchical order? I
got properly lost in the whole cgroup destroy code. There's endless
workqueues and rcu callbacks there.

So the current place in free_fair_sched_group() is far too late to be
calling remove_entity_load_avg(). But I'm not sure where I should put
it, it needs to be in a place where we know the group is going to die
but its parent is guaranteed to still exist.

Would offline be that place?

[toc] | [prev] | [next] | [standalone]

#1313308

Hello,

On Wed, Jan 20, 2016 at 11:47:58AM +0100, Peter Zijlstra wrote:
> TJ, is css_offline guaranteed to be called in hierarchical order? I

No, they aren't.  The ancestors of a css are guaranteed to stay around
until css_free is called on the css and that's the only ordering
guarantee.

> got properly lost in the whole cgroup destroy code. There's endless
> workqueues and rcu callbacks there.

Yeah, it's hairy.  I wondered about adding support for bouncing to
workqueue in both percpu_ref and rcu which would make things easier to
follow.  Not sure how often this pattern happens tho.

> So the current place in free_fair_sched_group() is far too late to be
> calling remove_entity_load_avg(). But I'm not sure where I should put
> it, it needs to be in a place where we know the group is going to die
> but its parent is guaranteed to still exist.
> 
> Would offline be that place?

Hmmm... css_free would be with the following patch.

diff -u b/kernel/cgroup.c work/kernel/cgroup.c
--- b/kernel/cgroup.c
+++ work/kernel/cgroup.c
@@ -4725,14 +4725,14 @@
 
 	if (ss) {
 		/* css free path */
+		struct cgroup_subsys_state *parent = css->parent;
 		int id = css->id;
 
-		if (css->parent)
-			css_put(css->parent);
-
 		ss->css_free(css);
 		cgroup_idr_remove(&ss->css_idr, id);
 		cgroup_put(cgrp);
+		if (parent)
+			css_put(parent);
 	} else {
 		/* cgroup free path */
 		atomic_dec(&cgrp->root->nr_cgrps);


Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1313325

On Wed, Jan 20, 2016 at 10:30:07AM -0500, Tejun Heo wrote:
> > So the current place in free_fair_sched_group() is far too late to be
> > calling remove_entity_load_avg(). But I'm not sure where I should put
> > it, it needs to be in a place where we know the group is going to die
> > but its parent is guaranteed to still exist.
> > 
> > Would offline be that place?
> 
> Hmmm... css_free would be with the following patch.

I thought a bit more about this and I think the right thing to do here
is making both css_offline and css_free follow the ancestry order.
I'll post a patch to do that soon.  offline is called at the head of
destruction when the css is made invisble and draining of existing
refs starts.  free at the end of that process.  Tree ordering
shouldn't be where the two differ.

Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.kernel

csiph-web