Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1497218

[PATCH -v4 6/8] locking/mutex: Restructure wait loop

Show all headers | View raw

Doesn't really matter yet, but pull the HANDOFF and trylock out from
under the wait_lock.

The intention is to add an optimistic spin loop here, which requires
we do not hold the wait_lock, so shuffle code around in preparation.

Also clarify the purpose of taking the wait_lock in the wait loop, its
tempting to want to avoid it altogether, but the cancellation cases
need to to avoid losing wakeups.

Suggested-by: Waiman Long <waiman.long@hpe.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/locking/mutex.c |   30 +++++++++++++++++++++++++-----
 1 file changed, 25 insertions(+), 5 deletions(-)

--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -631,13 +631,21 @@ __mutex_lock_common(struct mutex *lock,
 
 	lock_contended(&lock->dep_map, ip);
 
+	set_task_state(task, state);
 	for (;;) {
+		/*
+		 * Once we hold wait_lock, we're serialized against
+		 * mutex_unlock() handing the lock off to us, do a trylock
+		 * before testing the error conditions to make sure we pick up
+		 * the handoff.
+		 */
 		if (__mutex_trylock(lock, first))
-			break;
+			goto acquired;
 
 		/*
-		 * got a signal? (This code gets eliminated in the
-		 * TASK_UNINTERRUPTIBLE case.)
+		 * Check for signals and wound conditions while holding
+		 * wait_lock. This ensures the lock cancellation is ordered
+		 * against mutex_unlock() and wake-ups do not go missing.
 		 */
 		if (unlikely(signal_pending_state(state, task))) {
 			ret = -EINTR;
@@ -650,16 +658,27 @@ __mutex_lock_common(struct mutex *lock,
 				goto err;
 		}
 
-		__set_task_state(task, state);
 		spin_unlock_mutex(&lock->wait_lock, flags);
 		schedule_preempt_disabled();
-		spin_lock_mutex(&lock->wait_lock, flags);
 
 		if (!first && __mutex_waiter_is_first(lock, &waiter)) {
 			first = true;
 			__mutex_set_flag(lock, MUTEX_FLAG_HANDOFF);
 		}
+
+		set_task_state(task, state);
+		/*
+		 * Here we order against unlock; we must either see it change
+		 * state back to RUNNING and fall through the next schedule(),
+		 * or we must see its unlock and acquire.
+		 */
+		if (__mutex_trylock(lock, first))
+			break;
+
+		spin_lock_mutex(&lock->wait_lock, flags);
 	}
+	spin_lock_mutex(&lock->wait_lock, flags);
+acquired:
 	__set_task_state(task, TASK_RUNNING);
 
 	mutex_remove_waiter(lock, &waiter, task);
@@ -682,6 +701,7 @@ __mutex_lock_common(struct mutex *lock,
 	return 0;
 
 err:
+	__set_task_state(task, TASK_RUNNING);
 	mutex_remove_waiter(lock, &waiter, task);
 	spin_unlock_mutex(&lock->wait_lock, flags);
 	debug_mutex_free_waiter(&waiter);

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH -v4 0/8] locking/mutex: Rewrite basic mutex Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:10 +0200
  [PATCH -v4 4/8] locking/mutex: Allow MUTEX_SPIN_ON_OWNER when DEBUG_MUTEXES Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:10 +0200
  [PATCH -v4 7/8] locking/mutex: Simplify some ww_mutex code in __mutex_lock_common() Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:10 +0200
  [PATCH -v4 6/8] locking/mutex: Restructure wait loop Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:30 +0200
  [PATCH -v4 8/8] locking/mutex: Enable optimistic spinning of woken waiter Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:30 +0200
  [PATCH -v4 2/8] locking/mutex: Rework mutex::owner Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:30 +0200
    Re: [PATCH -v4 2/8] locking/mutex: Rework mutex::owner Davidlohr Bueso <dave@stgolabs.net> - 2016-10-12 20:40 +0200
      Re: [PATCH -v4 2/8] locking/mutex: Rework mutex::owner Jason Low <jason.low2@hpe.com> - 2016-10-12 22:00 +0200
  [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid starvation Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:40 +0200
  [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:40 +0200
    Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:50 +0200
      Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Linus Torvalds <torvalds@linux-foundation.org> - 2016-10-07 18:00 +0200
        Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-07 18:20 +0200
      Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Thomas Gleixner <tglx@linutronix.de> - 2016-10-08 14:10 +0200
        Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Thomas Gleixner <tglx@linutronix.de> - 2016-10-08 16:20 +0200
          Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-08 18:50 +0200
        Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-08 16:20 +0200
  [PATCH -v4 3/8] locking/mutex: Kill arch specific code Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:40 +0200
  Re: [PATCH -v4 0/8] locking/mutex: Rewrite basic mutex Linus Torvalds <torvalds@linux-foundation.org> - 2016-10-07 17:50 +0200
  Re: [PATCH -v4 0/8] locking/mutex: Rewrite basic mutex Jason Low <jason.low2@hpe.com> - 2016-10-11 21:10 +0200

csiph-web