Re: regression 4.4: deadlock in with cgroup percpu_rwsem

From	Christian Borntraeger <borntraeger@de.ibm.com>
Newsgroups	linux.kernel
Subject	Re: regression 4.4: deadlock in with cgroup percpu_rwsem
Date	2016-01-18 19:50 +0100
Message-ID	<qSn2h-4CH-1@gated-at.bofh.it> (permalink)
References	<qQOgi-6eV-7@gated-at.bofh.it> <qQWdP-3jT-7@gated-at.bofh.it> <qR79g-2OU-11@gated-at.bofh.it> <qRekq-7Xi-21@gated-at.bofh.it> <qSmSC-4zc-17@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On 01/18/2016 07:32 PM, Peter Zijlstra wrote:
> On Fri, Jan 15, 2016 at 04:13:34PM +0100, Christian Borntraeger wrote:
>>> Yes, the deadlock is gone and the system is still running.
>>> After some time I had the following WARN in the logs, though.
>>> Not sure yet if that is related.
>>>
>>> [25331.763607] DEBUG_LOCKS_WARN_ON(lock->owner != current)
>>> [25331.763630] ------------[ cut here ]------------
>>> [25331.763634] WARNING: at kernel/locking/mutex-debug.c:80
> 
>> I restarted the test with panic_on_warn. Hopefully I can get a dump to check
>> which mutex this was.
> 
> Hard to reproduce warnings like this tend to point towards memory
> corruption. Someone stepped on the mutex value and tickles the sanity
> check.
> 
> With lockdep and debugging enabled the mutex gets quite a bit bigger, so
> it gets more likely to be hit by 'random' corruption.
> 
> The locking in seq_read() seems rather straight forward.

I was able to reproduce. The dump shows a mutex that has an owner field, which
does not exists as a task so this all looks fishy. The good thing is, that I
can reproduce the issue within some hours. (exact same backtrace). Will add some
more debug data to get a handle where we come from.

Thread

Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-15 16:20 +0100
  Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-18 19:40 +0100
    Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-18 19:50 +0100
      Re: regression 4.4: deadlock in with cgroup percpu_rwsem Heiko Carstens <heiko.carstens@de.ibm.com> - 2016-01-19 11:00 +0100
        Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-19 20:40 +0100
          Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-19 20:40 +0100
            Re: regression 4.4: deadlock in with cgroup percpu_rwsem Heiko Carstens <heiko.carstens@de.ibm.com> - 2016-01-20 08:10 +0100
              Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-20 11:20 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-20 11:40 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-20 11:50 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-20 16:40 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-20 17:10 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-20 17:50 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-20 18:00 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-01-23 03:10 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christoph Hellwig <hch@lst.de> - 2016-01-25 09:50 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-25 20:40 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christoph Hellwig <hch@lst.de> - 2016-01-26 16:00 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Tejun Heo <tj@kernel.org> - 2016-01-26 16:30 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christoph Hellwig <hch@lst.de> - 2016-01-26 17:50 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-20 12:00 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Christian Borntraeger <borntraeger@de.ibm.com> - 2016-01-21 09:30 +0100
                Re: regression 4.4: deadlock in with cgroup percpu_rwsem Peter Zijlstra <peterz@infradead.org> - 2016-01-21 10:30 +0100

csiph-web