Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1656392 > unrolled thread

[PATCH RT] sched: Prevent task state corruption by spurious lock wakeup

Back to article view | Back to linux.kernel

Contents

  [PATCH RT] sched: Prevent task state corruption by spurious lock  wakeup Thomas Gleixner <tglx@linutronix.de> - 2017-06-02 18:50 +0200
    RE: [PATCH RT] sched: Prevent task state corruption by spurious  lock wakeup David Hauck <davidh@netacquire.com> - 2017-06-02 19:40 +0200
    Re: [PATCH RT] sched: Prevent task state corruption by spurious lock  wakeup Thomas Gleixner <tglx@linutronix.de> - 2017-06-03 09:50 +0200
      Re: [PATCH RT] sched: Prevent task state corruption by spurious lock  wakeup Thomas Gleixner <tglx@linutronix.de> - 2017-06-06 14:20 +0200
        [PATCH RT V2] sched: Prevent task state corruption by spurious lock  wakeup Thomas Gleixner <tglx@linutronix.de> - 2017-06-06 14:30 +0200
          Re: [PATCH RT V2] sched: Prevent task state corruption by spurious  lock wakeup Peter Zijlstra <peterz@infradead.org> - 2017-06-06 18:30 +0200
            Re: [PATCH RT V2] sched: Prevent task state corruption by spurious  lock wakeup Thomas Gleixner <tglx@linutronix.de> - 2017-06-06 20:40 +0200

#1656392 — [PATCH RT] sched: Prevent task state corruption by spurious lock wakeup

Mathias and some others reported GDB failures on RT.

The following scenario leads to task state corruption:

CPU0						CPU1

T1->state = TASK_XXX;
spin_lock(&lock)
  rt_spin_lock_slowlock(&lock->rtmutex)
    raw_spin_lock(&rtm->wait_lock);
    T1->saved_state = current->state;
    T1->state = TASK_UNINTERRUPTIBLE;
						spin_unlock(&lock)
    task_blocks_on_rt_mutex(rtm)  		  rt_spin_lock_slowunlock(&lock->rtmutex)
      queue_waiter(rtm)				    raw_spin_lock(&rtm->wait_lock);
      pi_chain_walk(rtm)
        raw_spin_unlock(&rtm->wait_lock);
						    mark_top_waiter_for_wakeup(T1)
						    raw_spin_unlock(&rtm->wait_lock);
      raw_spin_lock(&rtm->wait_lock);
						    wake_up_top_waiter()    

    for (;;) {
      if (__try_to_take_rt_mutex())  <- Succeeds
        break;
      ...
    }
						     try_to_wake_up(T1)
    T1->state = T1->saved_state;
    ==> T1->state == TASK_XXX
						       ttwu_do_wakeup(T1)
				FAIL ---->               T1->state = TASK_RUNNING;


In most cases this is harmless because waiting for some event, which is the
usual reason for TASK_[UN]INTERRUPTIBLE, has to be safe against other forms
of spurious wakeups anyway.

But in case of TASK_TRACED this is actually fatal, because the task loses
the TASK_TRACED state. In consequence it fails to consume SIGSTOP which was
sent from the debugger and actually delivers SIGSTOP to the task which
breaks the ptrace mechanics and brings the debugger into an unexpected
state.

The cure is way simpler as figuring it out:

In a lock wakeup, check whether the task is actually blocked on a lock. If
yes, deliver it. If not, consider the wakeup spurious and exit the wake up
code without touching tasks state.

Reported-by: Mathias Koehrer <mathias.koehrer@etas.com>
Reported-by: David Hauck <davidh@netacquire.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: stable-rt@vger.kernel.org
---
 kernel/sched/core.c |   12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

Index: b/kernel/sched/core.c
===================================================================
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2174,8 +2174,18 @@ try_to_wake_up(struct task_struct *p, un
 	 * If this is a regular wakeup, then we can unconditionally
 	 * clear the saved state of a "lock sleeper".
 	 */
-	if (!(wake_flags & WF_LOCK_SLEEPER))
+	if (!(wake_flags & WF_LOCK_SLEEPER)) {
 		p->saved_state = TASK_RUNNING;
+	} else {
+		/*
+		 * The task might not yet have reached schedule() and has
+		 * taken over the lock already and restored the saved
+		 * state. Prevent that this spurious wakeup destroys the saved
+		 * state.
+		 */
+		if (!tsk_is_pi_blocked(p))
+			goto out;
+	}
 
 	trace_sched_waking(p);
 

[toc] | [next] | [standalone]

#1656412 — RE: [PATCH RT] sched: Prevent task state corruption by spurious lock wakeup

Hi Thomas,
 
On Friday, June 02, 2017 9:40 AM, Thomas Gleixner wrote:
> Mathias and some others reported GDB failures on RT.

I've run an initial test of this on our build/application and the debugger is working *way* better now. So far no immediate issues - I'll continue testing. Thanks so much for finding, fixing, and sending this along.

FYI, my target:
	- kernel: v4.9.30-rt20
	- GDB/gdbserver: v7.11.1
	- GLibC: v2.25

Regards,
-David Hauck

PS: Mathias, my runs of your gdb-issue test utility also appear to run fine.

[toc] | [prev] | [next] | [standalone]

#1656748

On Fri, 2 Jun 2017, Thomas Gleixner wrote:

> Mathias and some others reported GDB failures on RT.
> 
> The following scenario leads to task state corruption:
> 
> CPU0						CPU1
> 
> T1->state = TASK_XXX;
> spin_lock(&lock)
>   rt_spin_lock_slowlock(&lock->rtmutex)
>     raw_spin_lock(&rtm->wait_lock);
>     T1->saved_state = current->state;
>     T1->state = TASK_UNINTERRUPTIBLE;
> 						spin_unlock(&lock)
>     task_blocks_on_rt_mutex(rtm)  		  rt_spin_lock_slowunlock(&lock->rtmutex)
>       queue_waiter(rtm)				    raw_spin_lock(&rtm->wait_lock);
>       pi_chain_walk(rtm)
>         raw_spin_unlock(&rtm->wait_lock);
> 						    mark_top_waiter_for_wakeup(T1)
> 						    raw_spin_unlock(&rtm->wait_lock);
>       raw_spin_lock(&rtm->wait_lock);
> 						    wake_up_top_waiter()    
> 
>     for (;;) {
>       if (__try_to_take_rt_mutex())  <- Succeeds
>         break;
>       ...
>     }
> 						     try_to_wake_up(T1)
>     T1->state = T1->saved_state;
>     ==> T1->state == TASK_XXX
> 						       ttwu_do_wakeup(T1)
> 				FAIL ---->               T1->state = TASK_RUNNING;
> 
> 
> In most cases this is harmless because waiting for some event, which is the
> usual reason for TASK_[UN]INTERRUPTIBLE, has to be safe against other forms
> of spurious wakeups anyway.
> 
> But in case of TASK_TRACED this is actually fatal, because the task loses
> the TASK_TRACED state. In consequence it fails to consume SIGSTOP which was
> sent from the debugger and actually delivers SIGSTOP to the task which
> breaks the ptrace mechanics and brings the debugger into an unexpected
> state.
> 
> The cure is way simpler as figuring it out:
> 
> In a lock wakeup, check whether the task is actually blocked on a lock. If
> yes, deliver it. If not, consider the wakeup spurious and exit the wake up
> code without touching tasks state.

After recovering from heat and ptrace induced brain melt, I have to say
that the fix is actually working, but it's not fixing the root cause of the
problem.

If the to be woken task has state TASK_TRACED then the wakeup state check
should not match. But for some stupid reason wake_up_lock_sleeper() should
not use TASK_ALL. The lock sleepers are in state UNINTERRUPTIBLE, so the
wake state should be UNINTERRUPTIBLE as well.

The extra check in try_to_wake_up() should stay though as it prevents
spurious wake ups in general.

Delta patch below.

Thanks,

	tglx

8<------------------
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2220,7 +2220,7 @@ EXPORT_SYMBOL(wake_up_process);
  */
 int wake_up_lock_sleeper(struct task_struct *p)
 {
-	return try_to_wake_up(p, TASK_ALL, WF_LOCK_SLEEPER);
+	return try_to_wake_up(p, TASK_UNINTERRUPTIBLE, WF_LOCK_SLEEPER);
 }
 
 int wake_up_state(struct task_struct *p, unsigned int state)

[toc] | [prev] | [next] | [standalone]

#1658695

On Sat, 3 Jun 2017, Thomas Gleixner wrote:
> If the to be woken task has state TASK_TRACED then the wakeup state check
> should not match. But for some stupid reason wake_up_lock_sleeper() should
> not use TASK_ALL. The lock sleepers are in state UNINTERRUPTIBLE, so the
> wake state should be UNINTERRUPTIBLE as well.
> 
> The extra check in try_to_wake_up() should stay though as it prevents
> spurious wake ups in general.

Further experimentation shows that it's not worth it. I'll send a V2 of
that patch with the proper fix.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1658719 — [PATCH RT V2] sched: Prevent task state corruption by spurious lock wakeup

Mathias and others reported GDB failures on RT.

The following scenario leads to task state corruption:

CPU0						CPU1

T1->state = TASK_XXX;
spin_lock(&lock)
  rt_spin_lock_slowlock(&lock->rtmutex)
    raw_spin_lock(&rtm->wait_lock);
    T1->saved_state = current->state;
    T1->state = TASK_UNINTERRUPTIBLE;
						spin_unlock(&lock)
    task_blocks_on_rt_mutex(rtm)  		  rt_spin_lock_slowunlock(&lock->rtmutex)
      queue_waiter(rtm)				    raw_spin_lock(&rtm->wait_lock);
      pi_chain_walk(rtm)
        raw_spin_unlock(&rtm->wait_lock);
						    wake_top_waiter(T1)

      raw_spin_lock(&rtm->wait_lock);
    

    for (;;) {
      if (__try_to_take_rt_mutex())  <- Succeeds
        break;
      ...
    }
	
    T1->state = T1->saved_state;
						     try_to_wake_up(T1)
						       ttwu_do_wakeup(T1)
						         T1->state = TASK_RUNNING;

In most cases this is harmless because waiting for some event, which is the
usual reason for TASK_[UN]INTERRUPTIBLE has to be safe against other forms
of spurious wakeups anyway.

But in case of TASK_TRACED this is actually fatal, because the task loses
the TASK_TRACED state. In consequence it fails to consume SIGSTOP which was
sent from the debugger and actually delivers SIGSTOP to the task which
breaks the ptrace mechanics and brings the debugger into an unexpected
state.

The TASK_TRACED state should prevent getting there due to the state
matching logic in try_to_wake_up(). But that's not true because
wake_up_lock_sleeper() uses TASK_ALL as state mask. That's bogus because
lock sleepers always use TASK_UNINTERRUPTIBLE, so the wakeup should use
that as well.

The cure is way simpler as figuring it out:

Change the mask used in wake_up_lock_sleeper() from TASK_ALL to
TASK_UNINTERRUPTIBLE.

Reported-by: Mathias Koehrer <mathias.koehrer@etas.com>
Reported-by: David Hauck <davidh@netacquire.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: stable-rt@vger.kernel.org
---
 kernel/sched/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2210,7 +2210,7 @@ EXPORT_SYMBOL(wake_up_process);
  */
 int wake_up_lock_sleeper(struct task_struct *p)
 {
-	return try_to_wake_up(p, TASK_ALL, WF_LOCK_SLEEPER);
+	return try_to_wake_up(p, TASK_UNINTERRUPTIBLE, WF_LOCK_SLEEPER);
 }
 
 int wake_up_state(struct task_struct *p, unsigned int state)

[toc] | [prev] | [next] | [standalone]

#1658923 — Re: [PATCH RT V2] sched: Prevent task state corruption by spurious lock wakeup

On Tue, Jun 06, 2017 at 02:20:37PM +0200, Thomas Gleixner wrote:
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -2210,7 +2210,7 @@ EXPORT_SYMBOL(wake_up_process);
>   */
>  int wake_up_lock_sleeper(struct task_struct *p)
>  {
> -	return try_to_wake_up(p, TASK_ALL, WF_LOCK_SLEEPER);
> +	return try_to_wake_up(p, TASK_UNINTERRUPTIBLE, WF_LOCK_SLEEPER);
>  }
>  
>  int wake_up_state(struct task_struct *p, unsigned int state)

Do we want this?

---
Subject: sched: Remove TASK_ALL

It's unused:

$ git grep "\<TASK_ALL\>" | wc -l
1

And dangerous, kill the bugger.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 3dfa5f99d6ee..e88aadbe2071 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -94,7 +94,6 @@ struct task_group;
 
 /* Convenience macros for the sake of wake_up(): */
 #define TASK_NORMAL			(TASK_INTERRUPTIBLE | TASK_UNINTERRUPTIBLE)
-#define TASK_ALL			(TASK_NORMAL | __TASK_STOPPED | __TASK_TRACED)
 
 /* get_task_state(): */
 #define TASK_REPORT			(TASK_RUNNING | TASK_INTERRUPTIBLE | \

[toc] | [prev] | [next] | [standalone]

#1659047 — Re: [PATCH RT V2] sched: Prevent task state corruption by spurious lock wakeup

On Tue, 6 Jun 2017, Peter Zijlstra wrote:

> On Tue, Jun 06, 2017 at 02:20:37PM +0200, Thomas Gleixner wrote:
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -2210,7 +2210,7 @@ EXPORT_SYMBOL(wake_up_process);
> >   */
> >  int wake_up_lock_sleeper(struct task_struct *p)
> >  {
> > -	return try_to_wake_up(p, TASK_ALL, WF_LOCK_SLEEPER);
> > +	return try_to_wake_up(p, TASK_UNINTERRUPTIBLE, WF_LOCK_SLEEPER);
> >  }
> >  
> >  int wake_up_state(struct task_struct *p, unsigned int state)
> 
> Do we want this?

Definitely.

> ---
> Subject: sched: Remove TASK_ALL
> 
> It's unused:
> 
> $ git grep "\<TASK_ALL\>" | wc -l
> 1
> 
> And dangerous, kill the bugger.
> 
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>

Acked-by: Thomas Gleixner <tglx@linutronix.de>

> ---
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 3dfa5f99d6ee..e88aadbe2071 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -94,7 +94,6 @@ struct task_group;
>  
>  /* Convenience macros for the sake of wake_up(): */
>  #define TASK_NORMAL			(TASK_INTERRUPTIBLE | TASK_UNINTERRUPTIBLE)
> -#define TASK_ALL			(TASK_NORMAL | __TASK_STOPPED | __TASK_TRACED)
>  
>  /* get_task_state(): */
>  #define TASK_REPORT			(TASK_RUNNING | TASK_INTERRUPTIBLE | \
> 

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web