Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1712182 > unrolled thread

Re: RCU stall when using function_graph

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: RCU stall when using function_graph Steven Rostedt <rostedt@goodmis.org> - 2017-08-15 15:30 +0200
    Re: RCU stall when using function_graph Daniel Lezcano <daniel.lezcano@linaro.org> - 2017-08-16 10:50 +0200
      Re: RCU stall when using function_graph Steven Rostedt <rostedt@goodmis.org> - 2017-08-16 16:10 +0200
        Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-16 18:40 +0200
          Re: RCU stall when using function_graph Steven Rostedt <rostedt@goodmis.org> - 2017-08-16 18:50 +0200
            Re: RCU stall when using function_graph "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-16 20:00 +0200

#1712182 — Re: RCU stall when using function_graph

[ I'm back from vacation! ]

On Wed, 9 Aug 2017 17:51:33 +0200
Daniel Lezcano <daniel.lezcano@linaro.org> wrote:

> Well, may be the instruction pointer thing is not a good idea.
> 
> I learnt from this experience, an overloaded kernel with a lot of
> interrupts can hang the console and issue RCU stall.
> 
> However, someone else can face the same situation. Even if he reads the
> RCU/stallwarn.txt documentation, it will be hard to figure out the issue.
> 
> A message telling the grace period can't be reached because we are too
> busy processing interrupts would have helped but I understand it is not
> easy to implement.

What if the stall code triggered an irqwork first? The irqwork would
trigger as soon as interrupts were enabled again (or at the next tick,
depending on the arch), and then it would know that RCU stalled due to
an irq storm if the irqwork is being hit.

-- Steve

> 
> Perhaps, adding a new bullet in the documentation can help:
> 
> "If the interrupt processing time is longer than the interval between
> each interrupt, the CPU will keep processing the interrupts without
> allowing the RCU's grace period kthread. This situation can happen if
> there is a highly rated number of interrupts and the function_graph
> tracer is enabled".
> 
> 
> 
> 

[toc] | [next] | [standalone]

#1712769

Hi Steven,


On 15/08/2017 15:29, Steven Rostedt wrote:
> 
> [ I'm back from vacation! ]

Did you get the tapes? :)

> On Wed, 9 Aug 2017 17:51:33 +0200
> Daniel Lezcano <daniel.lezcano@linaro.org> wrote:
> 
>> Well, may be the instruction pointer thing is not a good idea.
>>
>> I learnt from this experience, an overloaded kernel with a lot of
>> interrupts can hang the console and issue RCU stall.
>>
>> However, someone else can face the same situation. Even if he reads the
>> RCU/stallwarn.txt documentation, it will be hard to figure out the issue.
>>
>> A message telling the grace period can't be reached because we are too
>> busy processing interrupts would have helped but I understand it is not
>> easy to implement.
> 
> What if the stall code triggered an irqwork first? The irqwork would
> trigger as soon as interrupts were enabled again (or at the next tick,
> depending on the arch), and then it would know that RCU stalled due to
> an irq storm if the irqwork is being hit.

Is that condition enough to tell the CPU is over utilized by the
interrupts handling?

And I'm wondering if it wouldn't make sense to have this detection in
the irq code. With or without the RCU stall warning kernel option set,
the irq framework will be warning about this situation. If the RCU stall
option is set, that will issue a second message. It will be easy to do
the connection between the first message and the second one, no ?




-- 
 <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

[toc] | [prev] | [next] | [standalone]

#1712989

On Wed, 16 Aug 2017 10:42:15 +0200
Daniel Lezcano <daniel.lezcano@linaro.org> wrote:

> Hi Steven,
> 
> 
> On 15/08/2017 15:29, Steven Rostedt wrote:
> > 
> > [ I'm back from vacation! ]  
> 
> Did you get the tapes? :)

Yes, but nothing in them would cause the reputation of the POTUS to
become any worse than it already is.

> 
> > On Wed, 9 Aug 2017 17:51:33 +0200
> > Daniel Lezcano <daniel.lezcano@linaro.org> wrote:
> >   
> >> Well, may be the instruction pointer thing is not a good idea.
> >>
> >> I learnt from this experience, an overloaded kernel with a lot of
> >> interrupts can hang the console and issue RCU stall.
> >>
> >> However, someone else can face the same situation. Even if he reads the
> >> RCU/stallwarn.txt documentation, it will be hard to figure out the issue.
> >>
> >> A message telling the grace period can't be reached because we are too
> >> busy processing interrupts would have helped but I understand it is not
> >> easy to implement.  
> > 
> > What if the stall code triggered an irqwork first? The irqwork would
> > trigger as soon as interrupts were enabled again (or at the next tick,
> > depending on the arch), and then it would know that RCU stalled due to
> > an irq storm if the irqwork is being hit.  
> 
> Is that condition enough to tell the CPU is over utilized by the
> interrupts handling?
> 
> And I'm wondering if it wouldn't make sense to have this detection in
> the irq code. With or without the RCU stall warning kernel option set,
> the irq framework will be warning about this situation. If the RCU stall
> option is set, that will issue a second message. It will be easy to do
> the connection between the first message and the second one, no ?

The thing is, the RCU code keeps track of the state of progress, I
don't believe the interrupt code does. It just worries about handling
interrupts. I'm not excited about adding infrastructure to the
interrupt code to do accounting of IRQ storms.

On the other hand, the RCU code already does this. If it notices a
stall, it can trigger a irq_work and wait a little more. If the
irq_work doesn't fire, then it can do the normal RCU stall message. But
if the irq_work does fire, and the RCU progress still hasn't moved
forward, then it would be able to say this is due to an IRQ storm and
produce a better error message.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1713109

On Wed, Aug 16, 2017 at 10:04:21AM -0400, Steven Rostedt wrote:
> On Wed, 16 Aug 2017 10:42:15 +0200
> Daniel Lezcano <daniel.lezcano@linaro.org> wrote:
> 
> > Hi Steven,
> > 
> > 
> > On 15/08/2017 15:29, Steven Rostedt wrote:
> > > 
> > > [ I'm back from vacation! ]  
> > 
> > Did you get the tapes? :)
> 
> Yes, but nothing in them would cause the reputation of the POTUS to
> become any worse than it already is.
> 
> > 
> > > On Wed, 9 Aug 2017 17:51:33 +0200
> > > Daniel Lezcano <daniel.lezcano@linaro.org> wrote:
> > >   
> > >> Well, may be the instruction pointer thing is not a good idea.
> > >>
> > >> I learnt from this experience, an overloaded kernel with a lot of
> > >> interrupts can hang the console and issue RCU stall.
> > >>
> > >> However, someone else can face the same situation. Even if he reads the
> > >> RCU/stallwarn.txt documentation, it will be hard to figure out the issue.
> > >>
> > >> A message telling the grace period can't be reached because we are too
> > >> busy processing interrupts would have helped but I understand it is not
> > >> easy to implement.  
> > > 
> > > What if the stall code triggered an irqwork first? The irqwork would
> > > trigger as soon as interrupts were enabled again (or at the next tick,
> > > depending on the arch), and then it would know that RCU stalled due to
> > > an irq storm if the irqwork is being hit.  
> > 
> > Is that condition enough to tell the CPU is over utilized by the
> > interrupts handling?
> > 
> > And I'm wondering if it wouldn't make sense to have this detection in
> > the irq code. With or without the RCU stall warning kernel option set,
> > the irq framework will be warning about this situation. If the RCU stall
> > option is set, that will issue a second message. It will be easy to do
> > the connection between the first message and the second one, no ?
> 
> The thing is, the RCU code keeps track of the state of progress, I
> don't believe the interrupt code does. It just worries about handling
> interrupts. I'm not excited about adding infrastructure to the
> interrupt code to do accounting of IRQ storms.
> 
> On the other hand, the RCU code already does this. If it notices a
> stall, it can trigger a irq_work and wait a little more. If the
> irq_work doesn't fire, then it can do the normal RCU stall message. But
> if the irq_work does fire, and the RCU progress still hasn't moved
> forward, then it would be able to say this is due to an IRQ storm and
> produce a better error message.

Let me see if I understand you...  About halfway to the stall limit,
RCU triggers an irq_work (on each CPU that has not yet passed through
a quiescent state, IPIing them in turn?), and if the irq_work has
not completed by the end of the stall limit, RCU adds that to its
stall-warning message.

Or am I missing something here?

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1713112

On Wed, 16 Aug 2017 09:32:28 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> Let me see if I understand you...  About halfway to the stall limit,
> RCU triggers an irq_work (on each CPU that has not yet passed through
> a quiescent state, IPIing them in turn?), and if the irq_work has
> not completed by the end of the stall limit, RCU adds that to its
> stall-warning message.

Doesn't even have to be half way through. It could be done at the
limit, and then wait a little more.

> 
> Or am I missing something here?

No, I think that's what I was suggesting.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1713198

On Wed, Aug 16, 2017 at 12:41:40PM -0400, Steven Rostedt wrote:
> On Wed, 16 Aug 2017 09:32:28 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > Let me see if I understand you...  About halfway to the stall limit,
> > RCU triggers an irq_work (on each CPU that has not yet passed through
> > a quiescent state, IPIing them in turn?), and if the irq_work has
> > not completed by the end of the stall limit, RCU adds that to its
> > stall-warning message.
> 
> Doesn't even have to be half way through. It could be done at the
> limit, and then wait a little more.

Agreed.  However, I take other evasive action halfway through, so there
is a nice place to put the code.

> > Or am I missing something here?
> 
> No, I think that's what I was suggesting.

I don't see a way of canceling an irq_work.

Ah, but there is an irq_work_queue_on() that fails if still pending.
I should be able to use that instead of cancel.  Plus, -I- don't have
to send the IPIs, I can just let irq_work_queue_on() do my dirty work!

This should provide sufficient entertainment for a bit!  ;-)

							Thanx, Paul

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web