Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1534132

[PATCH v2 06/11] locking/ww_mutex: Notify waiters that have to back off while adding tasks to wait list

Show all headers | View raw

From: Nicolai Hähnle <Nicolai.Haehnle@amd.com>

While adding our task as a waiter, detect if another task should back off
because of us.

With this patch, we establish the invariant that the wait list contains
at most one (sleeping) waiter with ww_ctx->acquired > 0, and this waiter
will be the first waiter with a context.

Since only waiters with ww_ctx->acquired > 0 have to back off, this allows
us to be much more economical with wakeups.

v2: rebase on v2 of earlier patches

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Maarten Lankhorst <dev@mblankhorst.nl>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: Nicolai Hähnle <Nicolai.Haehnle@amd.com>
---
 kernel/locking/mutex.c | 42 ++++++++++++++++++++++++++++++++----------
 1 file changed, 32 insertions(+), 10 deletions(-)

diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index d63eebb..01e9438 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -609,23 +609,34 @@ void __sched ww_mutex_unlock(struct ww_mutex *lock)
 EXPORT_SYMBOL(ww_mutex_unlock);
 
 static inline int __sched
-__ww_mutex_lock_check_stamp(struct mutex *lock, struct ww_acquire_ctx *ctx)
+__ww_mutex_lock_check_stamp(struct mutex *lock, struct mutex_waiter *waiter,
+			    struct ww_acquire_ctx *ctx)
 {
 	struct ww_mutex *ww = container_of(lock, struct ww_mutex, base);
 	struct ww_acquire_ctx *hold_ctx = READ_ONCE(ww->ctx);
+	struct mutex_waiter *cur;
 
-	if (!hold_ctx)
-		return 0;
+	if (hold_ctx && __ww_mutex_stamp_after(ctx, hold_ctx))
+		goto deadlock;
 
-	if (__ww_mutex_stamp_after(ctx, hold_ctx)) {
-#ifdef CONFIG_DEBUG_MUTEXES
-		DEBUG_LOCKS_WARN_ON(ctx->contending_lock);
-		ctx->contending_lock = ww;
-#endif
-		return -EDEADLK;
+	/*
+	 * If there is a waiter in front of us that has a context, then its
+	 * stamp is earlier than ours and we must back off.
+	 */
+	cur = waiter;
+	list_for_each_entry_continue_reverse(cur, &lock->wait_list, list) {
+		if (cur->ww_ctx)
+			goto deadlock;
 	}
 
 	return 0;
+
+deadlock:
+#ifdef CONFIG_DEBUG_MUTEXES
+	DEBUG_LOCKS_WARN_ON(ctx->contending_lock);
+	ctx->contending_lock = ww;
+#endif
+	return -EDEADLK;
 }
 
 static inline int __sched
@@ -666,6 +677,16 @@ __ww_mutex_add_waiter(struct mutex_waiter *waiter,
 		}
 
 		list_add_tail(&waiter->list, &cur->list);
+
+		/*
+		 * Wake up the waiter so that it gets a chance to back
+		 * off.
+		 */
+		if (cur->ww_ctx->acquired > 0) {
+			debug_mutex_wake_waiter(lock, cur);
+			wake_up_process(cur->task);
+		}
+
 		return 0;
 	}
 
@@ -767,7 +788,8 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass,
 		}
 
 		if (use_ww_ctx && ww_ctx && ww_ctx->acquired > 0) {
-			ret = __ww_mutex_lock_check_stamp(lock, ww_ctx);
+			ret = __ww_mutex_lock_check_stamp(lock, &waiter,
+							  ww_ctx);
 			if (ret)
 				goto err;
 		}
-- 
2.7.4

Back to linux.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

[PATCH v2 00/11] locking/ww_mutex: Keep sorted wait list to avoid stampedes Nicolai Hähnle <nhaehnle@gmail.com> - 2016-12-01 15:10 +0100
  [PATCH v2 05/11] locking/ww_mutex: Add waiters in stamp order Nicolai Hähnle <nhaehnle@gmail.com> - 2016-12-01 15:10 +0100
    Re: [PATCH v2 05/11] locking/ww_mutex: Add waiters in stamp order Chris Wilson <chris@chris-wilson.co.uk> - 2016-12-01 17:10 +0100
    Re: [PATCH v2 05/11] locking/ww_mutex: Add waiters in stamp order Peter Zijlstra <peterz@infradead.org> - 2016-12-06 16:40 +0100
    Re: [PATCH v2 05/11] locking/ww_mutex: Add waiters in stamp order Peter Zijlstra <peterz@infradead.org> - 2016-12-06 18:00 +0100
  [PATCH v2 07/11] locking/ww_mutex: Wake at most one waiter for back off when acquiring the lock Nicolai Hähnle <nhaehnle@gmail.com> - 2016-12-01 15:10 +0100
  [PATCH v2 10/11] Documentation/locking/ww_mutex: Update the design document Nicolai Hähnle <nhaehnle@gmail.com> - 2016-12-01 15:10 +0100
  [PATCH v2 09/11] locking/mutex: Initialize mutex_waiter::ww_ctx with poison when debugging Nicolai Hähnle <nhaehnle@gmail.com> - 2016-12-01 15:10 +0100
  [PATCH v2 02/11] locking/ww_mutex: Re-check ww->ctx in the inner optimistic spin loop Nicolai Hähnle <nhaehnle@gmail.com> - 2016-12-01 15:10 +0100
    Re: [PATCH v2 02/11] locking/ww_mutex: Re-check ww->ctx in the inner  optimistic spin loop Chris Wilson <chris@chris-wilson.co.uk> - 2016-12-01 15:40 +0100
    Re: [PATCH v2 02/11] locking/ww_mutex: Re-check ww->ctx in the inner  optimistic spin loop Peter Zijlstra <peterz@infradead.org> - 2016-12-06 16:10 +0100
      Re: [PATCH v2 02/11] locking/ww_mutex: Re-check ww->ctx in the inner  optimistic spin loop Waiman Long <longman@redhat.com> - 2016-12-06 17:10 +0100
        Re: [PATCH v2 02/11] locking/ww_mutex: Re-check ww->ctx in the inner  optimistic spin loop Waiman Long <longman@redhat.com> - 2016-12-06 19:50 +0100
        Re: [PATCH v2 02/11] locking/ww_mutex: Re-check ww->ctx in the inner  optimistic spin loop Peter Zijlstra <peterz@infradead.org> - 2016-12-06 20:10 +0100
  [PATCH v2 01/11] drm/vgem: Use ww_mutex_(un)lock even with a NULL context Nicolai Hähnle <nhaehnle@gmail.com> - 2016-12-01 15:10 +0100
    Re: [PATCH v2 01/11] drm/vgem: Use ww_mutex_(un)lock even with a  NULL context Chris Wilson <chris@chris-wilson.co.uk> - 2016-12-01 15:20 +0100
      Re: [PATCH v2 01/11] drm/vgem: Use ww_mutex_(un)lock even with a  NULL context Daniel Vetter <daniel@ffwll.ch> - 2016-12-01 16:20 +0100
    Re: [PATCH v2 01/11] drm/vgem: Use ww_mutex_(un)lock even with a  NULL context Peter Zijlstra <peterz@infradead.org> - 2016-12-01 17:30 +0100
  [PATCH v2 11/11] [rfc] locking/ww_mutex: Always spin optimistically for the first waiter Nicolai Hähnle <nhaehnle@gmail.com> - 2016-12-01 15:10 +0100
  [PATCH v2 06/11] locking/ww_mutex: Notify waiters that have to back off while adding tasks to wait list Nicolai Hähnle <nhaehnle@gmail.com> - 2016-12-01 15:10 +0100

csiph-web