Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1473249

Re: [RFC][PATCH] Fix a race between rwsem and the scheduler

Show all headers | View raw

On 31/08/16 17:28, Peter Zijlstra wrote:
> On Wed, Aug 31, 2016 at 01:41:33PM +1000, Balbir Singh wrote:
>> On 30/08/16 22:19, Peter Zijlstra wrote:
>>> On Tue, Aug 30, 2016 at 06:49:37PM +1000, Balbir Singh wrote:
>>>>
>>>>
>>>> The origin of the issue I've seen seems to be related to
>>>> rwsem spin lock stealing. Basically I see the system deadlock'd in the
>>>> following state
>>>
>>> As Nick says (good to see you're back Nick!), this is unrelated to
>>> rwsems.
>>>
>>> This is true for pretty much every blocking wait loop out there, they
>>> all do:
>>>
>>> 	for (;;) {
>>> 		current->state = UNINTERRUPTIBLE;
>>> 		smp_mb();
>>> 		if (cond)
>>> 			break;
>>> 		schedule();
>>> 	}
>>> 	current->state = RUNNING;
>>>
>>> Which, if the wakeup is spurious, is just the pattern you need.
>>
>> Yes True! My bad Alexey had seen the same basic pattern, I should have been clearer
>> in my commit log. Should I resend the patch?
> 
> Yes please.
> 

Done, just now. I've tried to generalize the issue, but I've kept the
example

>>> There isn't an MB there. The best I can do is UNLOCK+LOCK, which, thanks
>>> to PPC, is _not_ MB. It is however sufficient for this case.
>>>
>>
>> The MB comes from the __switch_to() in schedule(). Ben mentioned it in a 
>> different thread.
> 
> Right, although even without that, there is sufficient ordering, as the
> rq unlock from the wakeup, coupled with the rq lock from the schedule
> already form a load-store barrier.
> 
>>> Now, this has been present for a fair while, I suspect ever since we
>>> reworked the wakeup path to not use rq->lock twice. Curious you only now
>>> hit it.
>>>
>>
>> Yes, I just hit it a a week or two back and I needed to collect data to
>> explain why p->on_rq got to 0. Hitting it requires extreme stress -- for me
>> I needed a system with large threads and less memory running stress-ng.
>> Reproducing the problem takes an unpredictable amount of time.
> 
> What hardware do you see this on, is it shiny new Power8 chips which
> have never before seen deep queues or something. Or is it 'regular' old
> Power7 like stuff?
> 

I don't think the issue is processor specific, it is probabilistic, but
I've not tested on Power7. I am seeing it on a Power8 system


Balbir Singh

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[RFC][PATCH] Fix a race between rwsem and the scheduler Balbir Singh <bsingharora@gmail.com> - 2016-08-30 10:50 +0200
  Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Nicholas Piggin <npiggin@gmail.com> - 2016-08-30 11:20 +0200
  Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Peter Zijlstra <peterz@infradead.org> - 2016-08-30 14:20 +0200
    Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Oleg Nesterov <oleg@redhat.com> - 2016-08-30 15:10 +0200
      Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Peter Zijlstra <peterz@infradead.org> - 2016-08-30 16:20 +0200
        Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Oleg Nesterov <oleg@redhat.com> - 2016-08-30 19:00 +0200
          Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Peter Zijlstra <peterz@infradead.org> - 2016-08-30 20:40 +0200
            Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Benjamin Herrenschmidt <benh@kernel.crashing.org> - 2016-08-30 23:30 +0200
              Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Peter Zijlstra <peterz@infradead.org> - 2016-08-31 09:20 +0200
                Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Benjamin Herrenschmidt <benh@kernel.crashing.org> - 2016-08-31 13:40 +0200
              Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Peter Zijlstra <peterz@infradead.org> - 2016-08-31 15:40 +0200
                Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Benjamin Herrenschmidt <benh@kernel.crashing.org> - 2016-08-31 23:50 +0200
                Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Balbir Singh <bsingharora@gmail.com> - 2016-09-01 08:50 +0200
                Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Peter Zijlstra <peterz@infradead.org> - 2016-09-01 09:00 +0200
                Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Boqun Feng <boqun.feng@gmail.com> - 2016-09-01 16:20 +0200
                Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Peter Zijlstra <peterz@infradead.org> - 2016-09-01 17:40 +0200
      Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Benjamin Herrenschmidt <benh@kernel.crashing.org> - 2016-08-30 23:30 +0200
        Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Peter Zijlstra <peterz@infradead.org> - 2016-08-31 09:30 +0200
          Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Benjamin Herrenschmidt <benh@kernel.crashing.org> - 2016-08-31 13:40 +0200
    Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Balbir Singh <bsingharora@gmail.com> - 2016-08-31 05:50 +0200
      Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Peter Zijlstra <peterz@infradead.org> - 2016-08-31 09:30 +0200
        Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Balbir Singh <bsingharora@gmail.com> - 2016-08-31 12:20 +0200
        Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Benjamin Herrenschmidt <benh@kernel.crashing.org> - 2016-08-31 13:00 +0200
        Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-09-01 03:50 +0200
          Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-09-01 14:20 +0200
  Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Oleg Nesterov <oleg@redhat.com> - 2016-08-30 15:00 +0200
    Re: [RFC][PATCH] Fix a race between rwsem and the scheduler Balbir Singh <bsingharora@gmail.com> - 2016-08-31 05:50 +0200

csiph-web