Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1293836 > unrolled thread

[PATCH] kernel/hung_task.c: use timeout diff when timeout is updated

Back to article view | Back to linux.kernel

Contents

  [PATCH] kernel/hung_task.c: use timeout diff when timeout is updated Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> - 2015-12-17 13:30 +0100
    Re: [PATCH] kernel/hung_task.c: use timeout diff when timeout is  updated Andrew Morton <akpm@linux-foundation.org> - 2015-12-17 23:20 +0100
      Re: [PATCH] kernel/hung_task.c: use timeout diff when timeout is updated Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> - 2015-12-21 12:50 +0100
        Re: [PATCH] kernel/hung_task.c: use timeout diff when timeout is  updated Andrew Morton <akpm@linux-foundation.org> - 2015-12-21 22:50 +0100

#1293836 — [PATCH] kernel/hung_task.c: use timeout diff when timeout is updated

>From 529ff00b556e110c6e801c39e94b06f559307136 Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Date: Thu, 17 Dec 2015 16:27:08 +0900
Subject: [PATCH] kernel/hung_task.c: use timeout diff when timeout is updated

When new timeout is written to /proc/sys/kernel/hung_task_timeout_secs,
khungtaskd is interrupted and again sleeps for full timeout duration.

This means that hang task will not be checked if new timeout is written
periodically within old timeout duration and/or checking of hang task
will be delayed for up to previous timeout duration.
Fix this by remembering last time khungtaskd checked hang task.

This change will allow other watchdog tasks (if any) to share khungtaskd
by sleeping for minimal timeout diff of all watchdog tasks. Doing more
watchdog tasks from khungtaskd will reduce the possibility of printk()
collisions by multiple watchdog threads.

Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
---
 kernel/hung_task.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index e0f90c2..d234022 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -185,10 +185,12 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	rcu_read_unlock();
 }
 
-static unsigned long timeout_jiffies(unsigned long timeout)
+static long hung_timeout_jiffies(unsigned long last_checked,
+				 unsigned long timeout)
 {
 	/* timeout of 0 will disable the watchdog */
-	return timeout ? timeout * HZ : MAX_SCHEDULE_TIMEOUT;
+	return timeout ? last_checked - jiffies + timeout * HZ :
+		MAX_SCHEDULE_TIMEOUT;
 }
 
 /*
@@ -224,18 +226,21 @@ EXPORT_SYMBOL_GPL(reset_hung_task_detector);
  */
 static int watchdog(void *dummy)
 {
+	unsigned long hung_last_checked = jiffies;
+
 	set_user_nice(current, 0);
 
 	for ( ; ; ) {
 		unsigned long timeout = sysctl_hung_task_timeout_secs;
+		long t = hung_timeout_jiffies(hung_last_checked, timeout);
 
-		while (schedule_timeout_interruptible(timeout_jiffies(timeout)))
-			timeout = sysctl_hung_task_timeout_secs;
-
-		if (atomic_xchg(&reset_hung_task, 0))
+		if (t <= 0) {
+			if (!atomic_xchg(&reset_hung_task, 0))
+				check_hung_uninterruptible_tasks(timeout);
+			hung_last_checked = jiffies;
 			continue;
-
-		check_hung_uninterruptible_tasks(timeout);
+		}
+		schedule_timeout_interruptible(t);
 	}
 
 	return 0;
-- 
1.8.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1294301 — Re: [PATCH] kernel/hung_task.c: use timeout diff when timeout is updated

On Thu, 17 Dec 2015 21:23:03 +0900 Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> wrote:

> >From 529ff00b556e110c6e801c39e94b06f559307136 Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Date: Thu, 17 Dec 2015 16:27:08 +0900
> Subject: [PATCH] kernel/hung_task.c: use timeout diff when timeout is updated
> 
> When new timeout is written to /proc/sys/kernel/hung_task_timeout_secs,
> khungtaskd is interrupted and again sleeps for full timeout duration.
> 
> This means that hang task will not be checked if new timeout is written
> periodically within old timeout duration and/or checking of hang task
> will be delayed for up to previous timeout duration.
> Fix this by remembering last time khungtaskd checked hang task.
> 
> This change will allow other watchdog tasks (if any) to share khungtaskd
> by sleeping for minimal timeout diff of all watchdog tasks. Doing more
> watchdog tasks from khungtaskd will reduce the possibility of printk()
> collisions by multiple watchdog threads.

This seems like reasonable behaviour, but it is a
non-backward-compatible change.  I don't know how important that is -
probably "not very".

Please let's fully describe this behaviour in the documentation. 
Documentation/sysctl/kernel.txt.

It appears that if userspace writes a new timeout which has already
expired, a check is triggered immediately, correct?  Let's ensure that
this is documented as well.   And tested!

And it would be helpful to add a comment to hung_timeout_jiffies()
which describes the behaviour and explains the reasons for it.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1295934

Andrew Morton wrote:
> On Thu, 17 Dec 2015 21:23:03 +0900 Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> wrote:
> 
> > >From 529ff00b556e110c6e801c39e94b06f559307136 Mon Sep 17 00:00:00 2001
> > From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> > Date: Thu, 17 Dec 2015 16:27:08 +0900
> > Subject: [PATCH] kernel/hung_task.c: use timeout diff when timeout is updated
> > 
> > When new timeout is written to /proc/sys/kernel/hung_task_timeout_secs,
> > khungtaskd is interrupted and again sleeps for full timeout duration.
> > 
> > This means that hang task will not be checked if new timeout is written
> > periodically within old timeout duration and/or checking of hang task
> > will be delayed for up to previous timeout duration.
> > Fix this by remembering last time khungtaskd checked hang task.
> > 
> > This change will allow other watchdog tasks (if any) to share khungtaskd
> > by sleeping for minimal timeout diff of all watchdog tasks. Doing more
> > watchdog tasks from khungtaskd will reduce the possibility of printk()
> > collisions by multiple watchdog threads.
> 
> This seems like reasonable behaviour, but it is a
> non-backward-compatible change.  I don't know how important that is -
> probably "not very".
> 
> Please let's fully describe this behaviour in the documentation. 
> Documentation/sysctl/kernel.txt.
> 
> It appears that if userspace writes a new timeout which has already
> expired, a check is triggered immediately, correct?  Let's ensure that
> this is documented as well.   And tested!

Yes.

> 
> And it would be helpful to add a comment to hung_timeout_jiffies()
> which describes the behaviour and explains the reasons for it.

But before doing it, I'd like to confirm hung task maintainer's will.

The reason I proposed this patch is that I want to add a watchdog task
which emits warning messages when memory allocations are stalling.
http://lkml.kernel.org/r/201512130033.ABH90650.FtFOMOFLVOJHQS@I-love.SAKURA.ne.jp

But concurrently emitting multiple backtraces is problematic. Concurrent
emitting by hung task watchdog and memory allocation stall watchdog is very
likely to occur, for it is likely that other task is also stuck in
uninterruptible sleep when one task got stuck at memory allocation.

Therefore, I started trying to use same thread for both watchdogs.
A draft patch is at
http://lkml.kernel.org/r/201512170011.IAC73451.FLtFMSJHOQFVOO@I-love.SAKURA.ne.jp .

If you prefer current hang task behavior, I'll try to preseve current
behavior. Instead, I might use two threads and try to mutex both watchdogs
using console_lock() or something like that.

So, may I ask what your preference is?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1296234 — Re: [PATCH] kernel/hung_task.c: use timeout diff when timeout is updated

On Mon, 21 Dec 2015 20:45:23 +0900 Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> wrote:
> > 
> > And it would be helpful to add a comment to hung_timeout_jiffies()
> > which describes the behaviour and explains the reasons for it.
> 
> But before doing it, I'd like to confirm hung task maintainer's will.
> 
> The reason I proposed this patch is that I want to add a watchdog task
> which emits warning messages when memory allocations are stalling.
> http://lkml.kernel.org/r/201512130033.ABH90650.FtFOMOFLVOJHQS@I-love.SAKURA.ne.jp
> 
> But concurrently emitting multiple backtraces is problematic. Concurrent
> emitting by hung task watchdog and memory allocation stall watchdog is very
> likely to occur, for it is likely that other task is also stuck in
> uninterruptible sleep when one task got stuck at memory allocation.
> 
> Therefore, I started trying to use same thread for both watchdogs.
> A draft patch is at
> http://lkml.kernel.org/r/201512170011.IAC73451.FLtFMSJHOQFVOO@I-love.SAKURA.ne.jp .
> 
> If you prefer current hang task behavior, I'll try to preseve current
> behavior. Instead, I might use two threads and try to mutex both watchdogs
> using console_lock() or something like that.
> 
> So, may I ask what your preference is?

I've added linux-mm to Cc.  Please never forget that.

The general topic here is "add more diagnostics around an out-of-memory
event".  Clearly we need this, but Michal is working on the same thing
as part of his "OOM detection rework v4" work, so can we please do the
appropriate coordination and review there?

Preventing watchdog-triggered backtraces from messing each other up is
of course a good idea.  Your malloc watchdog patch adds a surprising
amount of code and adding yet another kernel thread is painful but
perhaps it's all worth it.  It's a matter of people reviewing, testing
and using the code in realistic situations and that process has hardly
begun, alas.

Sorry, that was waffly but I don't feel able to be more definite at
this time.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web