Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1360146 > unrolled thread

Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace() methods

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods Peter Zijlstra <peterz@infradead.org> - 2016-03-17 20:40 +0100
    Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods Chris Metcalf <cmetcalf@mellanox.com> - 2016-03-17 23:40 +0100
      Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods Peter Zijlstra <peterz@infradead.org> - 2016-03-17 23:40 +0100
        Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods Peter Zijlstra <peterz@infradead.org> - 2016-03-18 00:20 +0100
    Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-03-18 00:00 +0100
      Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods Peter Zijlstra <peterz@infradead.org> - 2016-03-18 00:10 +0100
      Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods Peter Zijlstra <peterz@infradead.org> - 2016-03-18 00:20 +0100
        Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-03-18 01:40 +0100
      Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods Chris Metcalf <cmetcalf@mellanox.com> - 2016-03-18 01:20 +0100
        Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-03-18 01:50 +0100
          Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods Daniel Thompson <daniel.thompson@linaro.org> - 2016-03-18 10:50 +0100
            Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace()  methods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-03-19 01:00 +0100

#1360146 — Re: [PATCH v2 1/4] nmi_backtrace: add more trigger_*_cpu_backtrace() methods

On Wed, Mar 16, 2016 at 01:02:10PM -0400, Chris Metcalf wrote:
> Currently you can only request a backtrace of either all cpus, or
> all cpus but yourself.  It can also be helpful to request a remote
> backtrace of a single cpu, and since we want that, the logical
> extension is to support a cpumask as the underlying primitive.
> 
> This change modifies the existing lib/nmi_backtrace.c code to take
> a cpumask as its basic primitive, and modifies the linux/nmi.h code
> to use either the old "all/all_but_self" arch methods, or the new
> "cpumask" method, depending on which is available.
> 
> The existing clients of nmi_backtrace (arm and x86) are converted
> to using the new cpumask approach in this change.

So the past days I've been staring at RCU stall warns, and they can use
a little of this. Their remote stack unwinds are less than useful.

[toc] | [next] | [standalone]

#1360236

On 3/17/2016 3:36 PM, Peter Zijlstra wrote:
> On Wed, Mar 16, 2016 at 01:02:10PM -0400, Chris Metcalf wrote:
>> Currently you can only request a backtrace of either all cpus, or
>> all cpus but yourself.  It can also be helpful to request a remote
>> backtrace of a single cpu, and since we want that, the logical
>> extension is to support a cpumask as the underlying primitive.
>>
>> This change modifies the existing lib/nmi_backtrace.c code to take
>> a cpumask as its basic primitive, and modifies the linux/nmi.h code
>> to use either the old "all/all_but_self" arch methods, or the new
>> "cpumask" method, depending on which is available.
>>
>> The existing clients of nmi_backtrace (arm and x86) are converted
>> to using the new cpumask approach in this change.
> So the past days I've been staring at RCU stall warns, and they can use
> a little of this. Their remote stack unwinds are less than useful.

Were you suggesting this as an improvement for a possible v3, or just a
kind of implicit ack of the patch series?  Thanks!

-- 
Chris Metcalf, Mellanox Technologies
http://www.mellanox.com

[toc] | [prev] | [next] | [standalone]

#1360240

On Thu, Mar 17, 2016 at 06:31:44PM -0400, Chris Metcalf wrote:
> On 3/17/2016 3:36 PM, Peter Zijlstra wrote:
> >On Wed, Mar 16, 2016 at 01:02:10PM -0400, Chris Metcalf wrote:
> >>Currently you can only request a backtrace of either all cpus, or
> >>all cpus but yourself.  It can also be helpful to request a remote
> >>backtrace of a single cpu, and since we want that, the logical
> >>extension is to support a cpumask as the underlying primitive.
> >>
> >>This change modifies the existing lib/nmi_backtrace.c code to take
> >>a cpumask as its basic primitive, and modifies the linux/nmi.h code
> >>to use either the old "all/all_but_self" arch methods, or the new
> >>"cpumask" method, depending on which is available.
> >>
> >>The existing clients of nmi_backtrace (arm and x86) are converted
> >>to using the new cpumask approach in this change.
> >So the past days I've been staring at RCU stall warns, and they can use
> >a little of this. Their remote stack unwinds are less than useful.
> 
> Were you suggesting this as an improvement for a possible v3, or just a
> kind of implicit ack of the patch series?  Thanks!

A suggestion more like. I've not actually looked at the 4th patch.

I'll try and fold the patches into the runs I do tomorrow, I'm sure to
trigger lots of fail. Maybe I'll even do that RCU patch.

[toc] | [prev] | [next] | [standalone]

#1360253

On Thu, Mar 17, 2016 at 06:41:42PM -0400, Chris Metcalf wrote:
> The build bot caught the fact that I missed arch/xtensa since it doesn't use
> LOCK_TEXT, so if you're testing on that (ok maybe unlikely) you can add this:

Ha!, no. regular boring x86_64.

[toc] | [prev] | [next] | [standalone]

#1360246

On Thu, Mar 17, 2016 at 08:36:00PM +0100, Peter Zijlstra wrote:
> On Wed, Mar 16, 2016 at 01:02:10PM -0400, Chris Metcalf wrote:
> > Currently you can only request a backtrace of either all cpus, or
> > all cpus but yourself.  It can also be helpful to request a remote
> > backtrace of a single cpu, and since we want that, the logical
> > extension is to support a cpumask as the underlying primitive.
> > 
> > This change modifies the existing lib/nmi_backtrace.c code to take
> > a cpumask as its basic primitive, and modifies the linux/nmi.h code
> > to use either the old "all/all_but_self" arch methods, or the new
> > "cpumask" method, depending on which is available.
> > 
> > The existing clients of nmi_backtrace (arm and x86) are converted
> > to using the new cpumask approach in this change.
> 
> So the past days I've been staring at RCU stall warns, and they can use
> a little of this. Their remote stack unwinds are less than useful.

The RCU stall-warn stack traces can be ugly, agreed.

That said, RCU used to use NMI-based stack traces, but switched to the
current scheme due to the NMIs having the unfortunate habit of locking
things up, which IIRC often meant no stack traces at all.  If I recall
correctly, one of the problems was self-deadlock in printk().

Have these problems been fixed?

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1360251

On Thu, Mar 17, 2016 at 03:55:57PM -0700, Paul E. McKenney wrote:
> That said, RCU used to use NMI-based stack traces, but switched to the
> current scheme due to the NMIs having the unfortunate habit of locking
> things up, which IIRC often meant no stack traces at all.  If I recall
> correctly, one of the problems was self-deadlock in printk().
> 
> Have these problems been fixed?

Improved is I think the word.

Although I've butchered my printk() into absolute submission :-)

[toc] | [prev] | [next] | [standalone]

#1360254

On Thu, Mar 17, 2016 at 03:55:57PM -0700, Paul E. McKenney wrote:
> The RCU stall-warn stack traces can be ugly, agreed.

Ugly isn't the problem, completely random bollocks that puts you on the
wrong path was more the problem.

It uses a stack pointer saved at some random time in the past to start
unwinding an active stack from. Completely and utter misery.

[toc] | [prev] | [next] | [standalone]

#1360276

On Fri, Mar 18, 2016 at 12:11:28AM +0100, Peter Zijlstra wrote:
> On Thu, Mar 17, 2016 at 03:55:57PM -0700, Paul E. McKenney wrote:
> > The RCU stall-warn stack traces can be ugly, agreed.
> 
> Ugly isn't the problem, completely random bollocks that puts you on the
> wrong path was more the problem.
> 
> It uses a stack pointer saved at some random time in the past to start
> unwinding an active stack from. Completely and utter misery.

Yep, its accuracy does depend on what is going on, which was also my
experience with the NMI-based approach's reliablity.

Perhaps a boot-time parameter enabling the sysadm to pick the desired
flavor of poison?

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1360271

On 3/17/2016 6:55 PM, Paul E. McKenney wrote:
> The RCU stall-warn stack traces can be ugly, agreed.
>
> That said, RCU used to use NMI-based stack traces, but switched to the
> current scheme due to the NMIs having the unfortunate habit of locking
> things up, which IIRC often meant no stack traces at all.  If I recall
> correctly, one of the problems was self-deadlock in printk().

Steven Rostedt enabled the per_cpu printk func support in June 2014, and
the nmi_backtrace code uses it to just capture printk output to percpu
buffers, so I think it's going to be a lot more robust than earlier attempts.

-- 
Chris Metcalf, Mellanox Technologies
http://www.mellanox.com

[toc] | [prev] | [next] | [standalone]

#1360279

On Thu, Mar 17, 2016 at 08:17:59PM -0400, Chris Metcalf wrote:
> On 3/17/2016 6:55 PM, Paul E. McKenney wrote:
> >The RCU stall-warn stack traces can be ugly, agreed.
> >
> >That said, RCU used to use NMI-based stack traces, but switched to the
> >current scheme due to the NMIs having the unfortunate habit of locking
> >things up, which IIRC often meant no stack traces at all.  If I recall
> >correctly, one of the problems was self-deadlock in printk().
> 
> Steven Rostedt enabled the per_cpu printk func support in June 2014, and
> the nmi_backtrace code uses it to just capture printk output to percpu
> buffers, so I think it's going to be a lot more robust than earlier attempts.

That would be a very good thing, give or take the "I think" qualifier.
And assuming that the target CPU is healthy enough to find its way back
to some place that can dump the per-CPU printk buffer.  I might well
be overly paranoid, but I have to suspect that the probability of that
buffer getting dumped is reduced greatly on a CPU that isn't healthy
enough to respond to RCU, though.

But it seems like enabling the experiment might be useful.

"Try enabling the NMI version.  If that doesn't get you your RCU CPU
stall warning stack trace, try the remote-print variant."

Or I suppose we could just do both in succession, just in case their
console was a serial port.  ;-)

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1360475

On 18/03/16 00:33, Paul E. McKenney wrote:
> On Thu, Mar 17, 2016 at 08:17:59PM -0400, Chris Metcalf wrote:
>> On 3/17/2016 6:55 PM, Paul E. McKenney wrote:
>>> The RCU stall-warn stack traces can be ugly, agreed.
>>>
>>> That said, RCU used to use NMI-based stack traces, but switched to the
>>> current scheme due to the NMIs having the unfortunate habit of locking
>>> things up, which IIRC often meant no stack traces at all.  If I recall
>>> correctly, one of the problems was self-deadlock in printk().
>>
>> Steven Rostedt enabled the per_cpu printk func support in June 2014, and
>> the nmi_backtrace code uses it to just capture printk output to percpu
>> buffers, so I think it's going to be a lot more robust than earlier attempts.
>
> That would be a very good thing, give or take the "I think" qualifier.
> And assuming that the target CPU is healthy enough to find its way back
> to some place that can dump the per-CPU printk buffer.  I might well
> be overly paranoid, but I have to suspect that the probability of that
> buffer getting dumped is reduced greatly on a CPU that isn't healthy
> enough to respond to RCU, though.

The target CPU doesn't dump the buffer. It "just" fields the NMI, stores 
the backtrace and sets a flag.

The buffer is dumped to console by the requesting CPU, either when all 
backtraces have come back or when a timeout is reached.


> But it seems like enabling the experiment might be useful.
>
> "Try enabling the NMI version.  If that doesn't get you your RCU CPU
> stall warning stack trace, try the remote-print variant."
>
> Or I suppose we could just do both in succession, just in case their
> console was a serial port.  ;-)

I guess both might be needed but only when the target CPU is dead enough 
to fail to respond to NMI. In principle, we could exploit the timeout in 
the NMI backtrace logic and only issue the missing backtraces.


Daniel.

[toc] | [prev] | [next] | [standalone]

#1361014

On Fri, Mar 18, 2016 at 09:40:25AM +0000, Daniel Thompson wrote:
> On 18/03/16 00:33, Paul E. McKenney wrote:
> >On Thu, Mar 17, 2016 at 08:17:59PM -0400, Chris Metcalf wrote:
> >>On 3/17/2016 6:55 PM, Paul E. McKenney wrote:
> >>>The RCU stall-warn stack traces can be ugly, agreed.
> >>>
> >>>That said, RCU used to use NMI-based stack traces, but switched to the
> >>>current scheme due to the NMIs having the unfortunate habit of locking
> >>>things up, which IIRC often meant no stack traces at all.  If I recall
> >>>correctly, one of the problems was self-deadlock in printk().
> >>
> >>Steven Rostedt enabled the per_cpu printk func support in June 2014, and
> >>the nmi_backtrace code uses it to just capture printk output to percpu
> >>buffers, so I think it's going to be a lot more robust than earlier attempts.
> >
> >That would be a very good thing, give or take the "I think" qualifier.
> >And assuming that the target CPU is healthy enough to find its way back
> >to some place that can dump the per-CPU printk buffer.  I might well
> >be overly paranoid, but I have to suspect that the probability of that
> >buffer getting dumped is reduced greatly on a CPU that isn't healthy
> >enough to respond to RCU, though.
> 
> The target CPU doesn't dump the buffer. It "just" fields the NMI,
> stores the backtrace and sets a flag.
> 
> The buffer is dumped to console by the requesting CPU, either when
> all backtraces have come back or when a timeout is reached.

That does sound a bit more robust, good!

> >But it seems like enabling the experiment might be useful.
> >
> >"Try enabling the NMI version.  If that doesn't get you your RCU CPU
> >stall warning stack trace, try the remote-print variant."
> >
> >Or I suppose we could just do both in succession, just in case their
> >console was a serial port.  ;-)
> 
> I guess both might be needed but only when the target CPU is dead
> enough to fail to respond to NMI. In principle, we could exploit the
> timeout in the NMI backtrace logic and only issue the missing
> backtraces.

It would be really nice if I could call one function that used the
best strategy for getting information (including stack trace) about a
specified CPU.  Ditto for getting information about a specified task,
which might be running or might be preempted at the time.

							Thanx, Paul

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web