Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1671736 > unrolled thread

[PATCH V2] kernel/watchdog: fix spurious hard lockups

Started bykan.liang@intel.com
First post2017-06-21 16:50 +0200
Last post2017-06-29 18:40 +0200
Articles 5 on this page of 25 — 6 participants

Back to article view | Back to linux.kernel

Contents

  [PATCH V2] kernel/watchdog: fix spurious hard lockups kan.liang@intel.com - 2017-06-21 16:50 +0200
    Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Thomas Gleixner <tglx@linutronix.de> - 2017-06-21 17:20 +0200
      RE: [PATCH V2] kernel/watchdog: fix spurious hard lockups "Liang, Kan" <kan.liang@intel.com> - 2017-06-21 19:10 +0200
        Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Prarit Bhargava <prarit@redhat.com> - 2017-06-21 19:50 +0200
      Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Andi Kleen <ak@linux.intel.com> - 2017-06-21 19:10 +0200
        Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Thomas Gleixner <tglx@linutronix.de> - 2017-06-21 22:10 +0200
    Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Thomas Gleixner <tglx@linutronix.de> - 2017-06-22 00:00 +0200
      Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Thomas Gleixner <tglx@linutronix.de> - 2017-06-22 17:40 +0200
      Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Don Zickus <dzickus@redhat.com> - 2017-06-22 17:50 +0200
        RE: [PATCH V2] kernel/watchdog: fix spurious hard lockups "Liang, Kan" <kan.liang@intel.com> - 2017-06-22 18:00 +0200
        Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Thomas Gleixner <tglx@linutronix.de> - 2017-06-23 10:10 +0200
          Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Don Zickus <dzickus@redhat.com> - 2017-06-23 18:30 +0200
            Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Thomas Gleixner <tglx@linutronix.de> - 2017-06-24 00:00 +0200
              Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Don Zickus <dzickus@redhat.com> - 2017-06-26 22:30 +0200
                Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Thomas Gleixner <tglx@linutronix.de> - 2017-06-26 22:40 +0200
                Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Don Zickus <dzickus@redhat.com> - 2017-06-27 22:20 +0200
                  RE: [PATCH V2] kernel/watchdog: fix spurious hard lockups "Liang, Kan" <kan.liang@intel.com> - 2017-06-27 22:50 +0200
                    Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Don Zickus <dzickus@redhat.com> - 2017-06-27 23:10 +0200
                    Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Andi Kleen <ak@linux.intel.com> - 2017-06-28 01:50 +0200
                      Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Don Zickus <dzickus@redhat.com> - 2017-06-28 21:10 +0200
                        Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Andi Kleen <ak@linux.intel.com> - 2017-06-28 22:20 +0200
                          Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Don Zickus <dzickus@redhat.com> - 2017-06-29 17:50 +0200
                            Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Andi Kleen <ak@linux.intel.com> - 2017-06-29 18:20 +0200
                              Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Don Zickus <dzickus@redhat.com> - 2017-06-29 18:30 +0200
                                Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups Andi Kleen <ak@linux.intel.com> - 2017-06-29 18:40 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1677132

FromAndi Kleen <ak@linux.intel.com>
Date2017-06-28 22:20 +0200
Message-ID<tXrBo-237-23@gated-at.bofh.it>
In reply to#1677087
On Wed, Jun 28, 2017 at 03:00:08PM -0400, Don Zickus wrote:
> On Tue, Jun 27, 2017 at 04:48:22PM -0700, Andi Kleen wrote:
> > > I haven't heard back any test result yet.
> > > 
> > > The above patch looks good to me.
> > 
> > This needs performance testing.  It may slow down performance or latency sensitive workloads.
> 
> More motivation to work through the issues with the proposed real fix? :-)
> 
> > 
> > > Which workaround do you prefer, the above one or the one checking timestamp?
> > 
> > I prefer the earlier patch, it has far less risk of performance issues.
> 
> But now you are slowing down the nmi_watchdog so much that the
> watchdog_thresh hold becomes meaningless, no? (granted the turbo-mode blows
> it out of the water too)  So now folks who depend on the 10/5/1/whatever second
> reliability lose that.  I think that might be unfair too.

What do you mean with reliability? If you need guarantees of resetting
you always need another separate hardware watchdog (like the TCO watchdog),
as the CPU could be hung up enough that even the NMI watchdog is not 
functional anymore.

So relying solely on the NMI watchdog doesn't make any sense.

It can be a useful debugging tool for a specific class of bugs: 
when kernel software is looping forever.

But if that happens does it really matter how many iterations the
loop does before it is stopped?

Even the current timeout is essentially eternity in CPU time, and 3x
eternity is still eternity.

> The hrtimer increase maintains that and just adds a few more
> interrupts/second.

Interruptions are a big deal for many people.

-Andi

[toc] | [prev] | [next] | [standalone]

#1677882

FromDon Zickus <dzickus@redhat.com>
Date2017-06-29 17:50 +0200
Message-ID<tXJRE-XD-25@gated-at.bofh.it>
In reply to#1677132
On Wed, Jun 28, 2017 at 01:14:04PM -0700, Andi Kleen wrote:
> It can be a useful debugging tool for a specific class of bugs: 
> when kernel software is looping forever.
> 
> But if that happens does it really matter how many iterations the
> loop does before it is stopped?
> 
> Even the current timeout is essentially eternity in CPU time, and 3x
> eternity is still eternity.

That isn't true.  We have customers that test the accuracy and file bugs.  I
had to write a RHEL whitepaper a number of years ago explaining why the
softlockup took 62 seconds to fire instead of 60.

Customers were changing the watchdog_thresh when the system slowed down to
purposely trigger a panic (which exposed race conditions leading Uli to
redesign the sysctl interface).

I don't feel like explaining to our customers how we regressed on our
watchdog accuracy.  It is exhausting, especially if it is a debug feature.

> 
> > The hrtimer increase maintains that and just adds a few more
> > interrupts/second.
> 
> Interruptions are a big deal for many people.

Yes, and we probably have customers that will complain on that too.

Either solution is a lose/lose.  And yes, we will probably get bit by the
false NMI problems on those Intel boxes.  This is why I was preferring a
real solution.

The question is, if the real solution is going to take a while, what is the
least sucky solution for now?  Or how do we minimize it to a specific class
of Intel boxes.

Cheers,
Don

[toc] | [prev] | [next] | [standalone]

#1677914

FromAndi Kleen <ak@linux.intel.com>
Date2017-06-29 18:20 +0200
Message-ID<tXKkG-1nd-43@gated-at.bofh.it>
In reply to#1677882
On Thu, Jun 29, 2017 at 11:44:06AM -0400, Don Zickus wrote:
> On Wed, Jun 28, 2017 at 01:14:04PM -0700, Andi Kleen wrote:
> > It can be a useful debugging tool for a specific class of bugs: 
> > when kernel software is looping forever.
> > 
> > But if that happens does it really matter how many iterations the
> > loop does before it is stopped?
> > 
> > Even the current timeout is essentially eternity in CPU time, and 3x
> > eternity is still eternity.
> 
> That isn't true.  We have customers that test the accuracy and file bugs.  I
> had to write a RHEL whitepaper a number of years ago explaining why the
> softlockup took 62 seconds to fire instead of 60.

Ok that makes sense.

It seems like a broken QA test from your customer, not a real issue, 
but yes explaining and documenting that can be difficult.

> 
> The question is, if the real solution is going to take a while, what is the
> least sucky solution for now?  Or how do we minimize it to a specific class
> of Intel boxes.

You can't minimize it because there's no forward looking solution
to detect a large turbo range, and also whatever issue you have in the
generic case would apply to them too.

Thomas' patch to modulate the frequency seemed reasonable to me.
It made the NMI watchdog depend on accurate ktime, but that's probably ok.

-Andi

[toc] | [prev] | [next] | [standalone]

#1677920

FromDon Zickus <dzickus@redhat.com>
Date2017-06-29 18:30 +0200
Message-ID<tXKum-1qg-13@gated-at.bofh.it>
In reply to#1677914
On Thu, Jun 29, 2017 at 09:12:20AM -0700, Andi Kleen wrote:
> On Thu, Jun 29, 2017 at 11:44:06AM -0400, Don Zickus wrote:
> > On Wed, Jun 28, 2017 at 01:14:04PM -0700, Andi Kleen wrote:
> > > It can be a useful debugging tool for a specific class of bugs: 
> > > when kernel software is looping forever.
> > > 
> > > But if that happens does it really matter how many iterations the
> > > loop does before it is stopped?
> > > 
> > > Even the current timeout is essentially eternity in CPU time, and 3x
> > > eternity is still eternity.
> > 
> > That isn't true.  We have customers that test the accuracy and file bugs.  I
> > had to write a RHEL whitepaper a number of years ago explaining why the
> > softlockup took 62 seconds to fire instead of 60.
> 
> Ok that makes sense.
> 
> It seems like a broken QA test from your customer, not a real issue, 

Agreed.

> but yes explaining and documenting that can be difficult.

Yes.

> 
> > 
> > The question is, if the real solution is going to take a while, what is the
> > least sucky solution for now?  Or how do we minimize it to a specific class
> > of Intel boxes.
> 
> You can't minimize it because there's no forward looking solution
> to detect a large turbo range, and also whatever issue you have in the
> generic case would apply to them too.
> 
> Thomas' patch to modulate the frequency seemed reasonable to me.
> It made the NMI watchdog depend on accurate ktime, but that's probably ok.

Ok, did Kan finish testing this patch (with the small fix on top)?

Cheers,
Don

[toc] | [prev] | [next] | [standalone]

#1677934

FromAndi Kleen <ak@linux.intel.com>
Date2017-06-29 18:40 +0200
Message-ID<tXKE2-1tl-17@gated-at.bofh.it>
In reply to#1677920
> > Thomas' patch to modulate the frequency seemed reasonable to me.
> > It made the NMI watchdog depend on accurate ktime, but that's probably ok.
> 
> Ok, did Kan finish testing this patch (with the small fix on top)?

Kan doesn't have the specific hardware to test it. We've been waiting
for another team to do the testing. I'll ping them.

-Andi

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web