Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1189654 > unrolled thread

Re: [PATCH] soft lockup: kill realtime threads before panic

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] soft lockup: kill realtime threads before panic Jörn Engel <joern@purestorage.com> - 2015-07-22 07:20 +0200
    Re: [PATCH] soft lockup: kill realtime threads before panic Mike Galbraith <umgwanakikbuti@gmail.com> - 2015-07-22 07:50 +0200
      Re: [PATCH] soft lockup: kill realtime threads before panic Jörn Engel <joern@purestorage.com> - 2015-07-22 08:40 +0200
        Re: [PATCH] soft lockup: kill realtime threads before panic Mike Galbraith <umgwanakikbuti@gmail.com> - 2015-07-22 09:40 +0200
          Re: [PATCH] soft lockup: kill realtime threads before panic Don Zickus <dzickus@redhat.com> - 2015-07-22 16:00 +0200
          Re: [PATCH] soft lockup: kill realtime threads before panic Jörn Engel <joern@purestorage.com> - 2015-07-22 18:40 +0200

#1189654 — Re: [PATCH] soft lockup: kill realtime threads before panic

On Wed, Jul 22, 2015 at 06:36:30AM +0200, Mike Galbraith wrote:
> On Tue, 2015-07-21 at 15:07 -0700, Spencer Baugh wrote:
> 
> > We have observed cases where the soft lockup detector triggered, but no
> > kernel bug existed.  Instead we had a buggy realtime thread that
> > monopolized a cpu.  So let's kill the responsible party and not panic
> > the entire system.
> 
> If you don't tell the kernel to panic, it won't, and if you don't remove
> its leash (the throttle), your not so tame rt beast won't maul you.

Not sure if this patch is something for mainline, but those two
alternatives have problems of their own.  Not panicking on lockups can
leave a system disabled until some human come around.  In many cases
that human will do no better than power-cycle.  A panic reduces the
downtime.

And the realtime throttling gives non-realtime threads some minimum
runtime, but does nothing to help low-priority realtime threads.  It
also introduces latencies, often when workloads are high and you would
like any available cpu to get through that rough spot.

I don't think we have a good answer to this problem in the mainline
kernel yet.

Jörn

--
To announce that there must be no criticism of the President, or that we
are to stand by the President, right or wrong, is not only unpatriotic
and servile, but is morally treasonable to the American public.
-- Theodore Roosevelt, Kansas City Star, 1918
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1189663

On Tue, 2015-07-21 at 22:18 -0700, Jörn Engel wrote:
> On Wed, Jul 22, 2015 at 06:36:30AM +0200, Mike Galbraith wrote:
> > On Tue, 2015-07-21 at 15:07 -0700, Spencer Baugh wrote:
> > 
> > > We have observed cases where the soft lockup detector triggered, but no
> > > kernel bug existed.  Instead we had a buggy realtime thread that
> > > monopolized a cpu.  So let's kill the responsible party and not panic
> > > the entire system.
> > 
> > If you don't tell the kernel to panic, it won't, and if you don't remove
> > its leash (the throttle), your not so tame rt beast won't maul you.
> 
> Not sure if this patch is something for mainline, but those two
> alternatives have problems of their own.  Not panicking on lockups can
> leave a system disabled until some human come around.  In many cases
> that human will do no better than power-cycle.  A panic reduces the
> downtime.

If a realtime task goes bonkers, the realtime game is over, you're down.
 
> And the realtime throttling gives non-realtime threads some minimum
> runtime, but does nothing to help low-priority realtime threads.  It
> also introduces latencies, often when workloads are high and you would
> like any available cpu to get through that rough spot.

You can use group scheduling as a debug crutch until the little beasts
are housebroken.

> I don't think we have a good answer to this problem in the mainline
> kernel yet.

IMHO, there's no point in trying to make rt warm/fuzzy/cuddly.  Just
don't stuff a Hells Angel into a super-suit, that gets real ugly ;-)

	-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1189674

On Wed, Jul 22, 2015 at 07:41:48AM +0200, Mike Galbraith wrote:
> On Tue, 2015-07-21 at 22:18 -0700, Jörn Engel wrote:
> > 
> > Not sure if this patch is something for mainline, but those two
> > alternatives have problems of their own.  Not panicking on lockups can
> > leave a system disabled until some human come around.  In many cases
> > that human will do no better than power-cycle.  A panic reduces the
> > downtime.
> 
> If a realtime task goes bonkers, the realtime game is over, you're down.

Agreed.  But a reboot will often solve the issue.  So the automatic
panic will repair the system within minutes, while no panic will leave
the system broken for days, depending on human response time.  Automatic
panic is a great way to minimize downtime - or vulnerable time if you
have HA.

One could argue that killing the realtime thread is even better than
panic, as things can restart with a blank slate even faster.  But the
real benefit is that we get better debug data for the failing component.
If we had a kernel bug, the backtrace would usually be sufficient to
point fingers.  With a bonkers realtime thread, not so much.

Anyway, this patch has been useful to us once.  If someone deems it
merge-worthy, great.  If not, I won't lose any sleep either.

Jörn

--
The key to performance is elegance, not battalions of special cases.
-- Jon Bentley and Doug McIlroy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1189711

On Tue, 2015-07-21 at 23:33 -0700, Jörn Engel wrote:

> One could argue that killing the realtime thread is even better than
> panic, as things can restart with a blank slate even faster.  But the
> real benefit is that we get better debug data for the failing component.
> If we had a kernel bug, the backtrace would usually be sufficient to
> point fingers.  With a bonkers realtime thread, not so much.

If userspace wants a watchdog, it should train a userspace dog, not turn
the kernel watchdog into a userspace attack dog.

	-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1189960

On Wed, Jul 22, 2015 at 09:35:28AM +0200, Mike Galbraith wrote:
> On Tue, 2015-07-21 at 23:33 -0700, Jörn Engel wrote:
> 
> > One could argue that killing the realtime thread is even better than
> > panic, as things can restart with a blank slate even faster.  But the
> > real benefit is that we get better debug data for the failing component.
> > If we had a kernel bug, the backtrace would usually be sufficient to
> > point fingers.  With a bonkers realtime thread, not so much.
> 
> If userspace wants a watchdog, it should train a userspace dog, not turn
> the kernel watchdog into a userspace attack dog.

I agree.  The spirit of the watchdog was detection and panic (if configured
that way).  I don't think adding policy like this works well in the long
run.

Cheers,
Don
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1190097

On Wed, Jul 22, 2015 at 09:35:28AM +0200, Mike Galbraith wrote:
> On Tue, 2015-07-21 at 23:33 -0700, Jörn Engel wrote:
> 
> > One could argue that killing the realtime thread is even better than
> > panic, as things can restart with a blank slate even faster.  But the
> > real benefit is that we get better debug data for the failing component.
> > If we had a kernel bug, the backtrace would usually be sufficient to
> > point fingers.  With a bonkers realtime thread, not so much.
> 
> If userspace wants a watchdog, it should train a userspace dog, not turn
> the kernel watchdog into a userspace attack dog.

Fair point.  Let's drop this patch then.

Jörn

--
Money can buy bandwidth, but latency is forever.
-- John R. Mashey
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web