Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1190346 > unrolled thread

Re: [PATCH] soft lockup: kill realtime threads before panic

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] soft lockup: kill realtime threads before panic Andrew Morton <akpm@linux-foundation.org> - 2015-07-23 01:00 +0200
    Re: [PATCH] soft lockup: kill realtime threads before panic Jörn Engel <joern@purestorage.com> - 2015-07-23 01:30 +0200

#1190346 — Re: [PATCH] soft lockup: kill realtime threads before panic

On Tue, 21 Jul 2015 15:07:57 -0700 Spencer Baugh <sbaugh@catern.com> wrote:

> From: Joern Engel <joern@logfs.org>
> 
> We have observed cases where the soft lockup detector triggered, but no
> kernel bug existed.  Instead we had a buggy realtime thread that
> monopolized a cpu.  So let's kill the responsible party and not panic
> the entire system.
> 
> ...
>
> --- a/kernel/watchdog.c
> +++ b/kernel/watchdog.c
> @@ -428,7 +428,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
>  		}
>  
>  		add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK);
> -		if (softlockup_panic)
> +		if (rt_prio(current->prio)) {
> +			pr_emerg("killing realtime thread\n");
> +			send_sig(SIGILL, current, 0);

Why choose SIGILL?

> +		} else if (softlockup_panic)
>  			panic("softlockup: hung tasks");
>  		__this_cpu_write(soft_watchdog_warn, true);

But what about a non-buggy realtime thread which happens to
occasionally spend 15 seconds doing stuff?

Old behaviour: kernel blurts a softlockup message, everything keeps running.

New behaviour: thread gets killed, plane crashes.


Possibly a better approach would be to only kill the thread if
softlockup_panic was set, because the system is going down anyway.

Also, perhaps some users would prefer that the kernel simply suppress
the softlockup warning in this situation, rather than killing stuff!




Really, what you're trying to implement here is a watchdog for runaway
realtime threads.  And that sounds a worthy project but it's a rather
separate thing from the softlockup detector.  A realtime thread
watchdog feature might have things as

- timeout duration separately configurable from softlockup

- enabled independently from sotflockup: people might want one and
  not the other.

- configurable signal, perhaps?

Now, the *implementation* of the realtime thread watchdog may well
share code with the softlockup detector.  But from a
conceptual/configuration/documentation point of view, it's a separate
thing, no?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1190354

On Wed, Jul 22, 2015 at 03:54:36PM -0700, Andrew Morton wrote:
> On Tue, 21 Jul 2015 15:07:57 -0700 Spencer Baugh <sbaugh@catern.com> wrote:
> 
> > From: Joern Engel <joern@logfs.org>
> > 
> > We have observed cases where the soft lockup detector triggered, but no
> > kernel bug existed.  Instead we had a buggy realtime thread that
> > monopolized a cpu.  So let's kill the responsible party and not panic
> > the entire system.
> > 
> > ...
> >
> > --- a/kernel/watchdog.c
> > +++ b/kernel/watchdog.c
> > @@ -428,7 +428,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
> >  		}
> >  
> >  		add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK);
> > -		if (softlockup_panic)
> > +		if (rt_prio(current->prio)) {
> > +			pr_emerg("killing realtime thread\n");
> > +			send_sig(SIGILL, current, 0);
> 
> Why choose SIGILL?

It is a random signal that happens to generate a stacktrace in
userspace.

> > +		} else if (softlockup_panic)
> >  			panic("softlockup: hung tasks");
> >  		__this_cpu_write(soft_watchdog_warn, true);
> 
> But what about a non-buggy realtime thread which happens to
> occasionally spend 15 seconds doing stuff?
> 
> Old behaviour: kernel blurts a softlockup message, everything keeps running.
> 
> New behaviour: thread gets killed, plane crashes.
> 
> 
> Possibly a better approach would be to only kill the thread if
> softlockup_panic was set, because the system is going down anyway.
> 
> Also, perhaps some users would prefer that the kernel simply suppress
> the softlockup warning in this situation, rather than killing stuff!
> 
> 
> 
> 
> Really, what you're trying to implement here is a watchdog for runaway
> realtime threads.  And that sounds a worthy project but it's a rather
> separate thing from the softlockup detector.  A realtime thread
> watchdog feature might have things as
> 
> - timeout duration separately configurable from softlockup
> 
> - enabled independently from sotflockup: people might want one and
>   not the other.
> 
> - configurable signal, perhaps?
> 
> Now, the *implementation* of the realtime thread watchdog may well
> share code with the softlockup detector.  But from a
> conceptual/configuration/documentation point of view, it's a separate
> thing, no?

Agreed.  We needed this patch exactly once and it is a rather quick hack
that yielded the necessary results.  Realtime threads were well-behaved
since and the patch has seen zero polish as a result.

I think it is better to drop the patch for now.  If someone else keeps
running into the same issue, it might be a starting point for a better
implementation.  They will find it in list archives.

Jörn

--
I can say that I spend most of my time fixing bugs even if I have lots
of new features to implement in mind, but I give bugs more priority.
-- Andrea Arcangeli, 2000
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web