Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1497206 > unrolled thread

[PATCH -v4 0/8] locking/mutex: Rewrite basic mutex

Back to article view | Back to linux.kernel

Contents

  [PATCH -v4 0/8] locking/mutex: Rewrite basic mutex Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:10 +0200
    [PATCH -v4 4/8] locking/mutex: Allow MUTEX_SPIN_ON_OWNER when DEBUG_MUTEXES Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:10 +0200
    [PATCH -v4 7/8] locking/mutex: Simplify some ww_mutex code in __mutex_lock_common() Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:10 +0200
    [PATCH -v4 6/8] locking/mutex: Restructure wait loop Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:30 +0200
      Re: [PATCH -v4 6/8] locking/mutex: Restructure wait loop Will Deacon <will.deacon@arm.com> - 2016-10-13 17:20 +0200
        Re: [PATCH -v4 6/8] locking/mutex: Restructure wait loop Peter Zijlstra <peterz@infradead.org> - 2016-10-17 12:50 +0200
          Re: [PATCH -v4 6/8] locking/mutex: Restructure wait loop Peter Zijlstra <peterz@infradead.org> - 2016-10-17 15:30 +0200
            Re: [PATCH -v4 6/8] locking/mutex: Restructure wait loop Boqun Feng <boqun.feng@gmail.com> - 2016-10-17 15:50 +0200
              Re: [PATCH -v4 6/8] locking/mutex: Restructure wait loop Peter Zijlstra <peterz@infradead.org> - 2016-10-17 18:00 +0200
        Re: [PATCH -v4 6/8] locking/mutex: Restructure wait loop Peter Zijlstra <peterz@infradead.org> - 2016-10-19 19:40 +0200
          ciao set_task_state() (was Re: [PATCH -v4 6/8] locking/mutex:  Restructure wait loop) Davidlohr Bueso <dave@stgolabs.net> - 2016-10-24 04:00 +0200
            Re: ciao set_task_state() (was Re: [PATCH -v4 6/8] locking/mutex:  Restructure wait loop) Kent Overstreet <kent.overstreet@gmail.com> - 2016-10-24 15:30 +0200
            Re: ciao set_task_state() (was Re: [PATCH -v4 6/8] locking/mutex:  Restructure wait loop) Kent Overstreet <kent.overstreet@gmail.com> - 2016-10-24 16:30 +0200
              Re: ciao set_task_state() (was Re: [PATCH -v4 6/8] locking/mutex:  Restructure wait loop) Eric Wheeler <bcache@lists.ewheeler.net> - 2016-10-25 19:00 +0200
                Re: ciao set_task_state() (was Re: [PATCH -v4 6/8] locking/mutex:  Restructure wait loop) Kent Overstreet <kent.overstreet@gmail.com> - 2016-10-25 19:50 +0200
      Re: [PATCH -v4 6/8] locking/mutex: Restructure wait loop Peter Zijlstra <peterz@infradead.org> - 2016-10-18 15:20 +0200
    [PATCH -v4 8/8] locking/mutex: Enable optimistic spinning of woken waiter Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:30 +0200
      Re: [PATCH -v4 8/8] locking/mutex: Enable optimistic spinning of  woken waiter Will Deacon <will.deacon@arm.com> - 2016-10-13 17:30 +0200
        Re: [PATCH -v4 8/8] locking/mutex: Enable optimistic spinning of  woken waiter Peter Zijlstra <peterz@infradead.org> - 2016-10-17 11:40 +0200
      Re: [PATCH -v4 8/8] locking/mutex: Enable optimistic spinning of  woken waiter Peter Zijlstra <peterz@infradead.org> - 2016-10-18 14:30 +0200
    [PATCH -v4 2/8] locking/mutex: Rework mutex::owner Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:30 +0200
      Re: [PATCH -v4 2/8] locking/mutex: Rework mutex::owner Davidlohr Bueso <dave@stgolabs.net> - 2016-10-12 20:40 +0200
        Re: [PATCH -v4 2/8] locking/mutex: Rework mutex::owner Jason Low <jason.low2@hpe.com> - 2016-10-12 22:00 +0200
      Re: [PATCH -v4 2/8] locking/mutex: Rework mutex::owner Will Deacon <will.deacon@arm.com> - 2016-10-13 17:20 +0200
    [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid starvation Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:40 +0200
      Re: [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid  starvation Will Deacon <will.deacon@arm.com> - 2016-10-13 17:20 +0200
        Re: [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid  starvation Peter Zijlstra <peterz@infradead.org> - 2016-10-17 11:30 +0200
      Re: [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid  starvation Peter Zijlstra <peterz@infradead.org> - 2016-10-18 14:40 +0200
      Re: [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid  starvation Peter Zijlstra <peterz@infradead.org> - 2016-10-18 15:10 +0200
    [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:40 +0200
      Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:50 +0200
        Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Linus Torvalds <torvalds@linux-foundation.org> - 2016-10-07 18:00 +0200
          Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-07 18:20 +0200
        Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Thomas Gleixner <tglx@linutronix.de> - 2016-10-08 14:10 +0200
          Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Thomas Gleixner <tglx@linutronix.de> - 2016-10-08 16:20 +0200
            Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-08 18:50 +0200
          Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-08 16:20 +0200
      Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-18 14:50 +0200
        Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Peter Zijlstra <peterz@infradead.org> - 2016-10-18 15:00 +0200
        Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery Chris Wilson <chris@chris-wilson.co.uk> - 2016-10-18 15:00 +0200
    [PATCH -v4 3/8] locking/mutex: Kill arch specific code Peter Zijlstra <peterz@infradead.org> - 2016-10-07 17:40 +0200
    Re: [PATCH -v4 0/8] locking/mutex: Rewrite basic mutex Linus Torvalds <torvalds@linux-foundation.org> - 2016-10-07 17:50 +0200
    Re: [PATCH -v4 0/8] locking/mutex: Rewrite basic mutex Jason Low <jason.low2@hpe.com> - 2016-10-11 21:10 +0200

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#1497254 — [PATCH -v4 2/8] locking/mutex: Rework mutex::owner

The current mutex implementation has an atomic lock word and a
non-atomic owner field.

This disparity leads to a number of issues with the current mutex code
as it means that we can have a locked mutex without an explicit owner
(because the owner field has not been set, or already cleared).

This leads to a number of weird corner cases, esp. between the
optimistic spinning and debug code. Where the optimistic spinning
code needs the owner field updated inside the lock region, the debug
code is more relaxed because the whole lock is serialized by the
wait_lock.

Also, the spinning code itself has a few corner cases where we need to
deal with a held lock without an owner field.

Furthermore, it becomes even more of a problem when trying to fix
starvation cases in the current code. We end up stacking special case
on special case.

To solve this rework the basic mutex implementation to be a single
atomic word that contains the owner and uses the low bits for extra
state.

This matches how PI futexes and rt_mutex already work. By having the
owner an integral part of the lock state a lot of the problems
dissapear and we get a better option to deal with starvation cases,
direct owner handoff.

Changing the basic mutex does however invalidate all the arch specific
mutex code; this patch leaves that unused in-place, a later patch will
remove that.


Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 include/linux/mutex-debug.h  |   24 --
 include/linux/mutex.h        |   46 +++--
 kernel/locking/mutex-debug.c |   13 -
 kernel/locking/mutex-debug.h |   10 -
 kernel/locking/mutex.c       |  371 ++++++++++++++++++-------------------------
 kernel/locking/mutex.h       |   26 ---
 kernel/sched/core.c          |    2 
 7 files changed, 187 insertions(+), 305 deletions(-)

--- a/include/linux/mutex-debug.h
+++ /dev/null
@@ -1,24 +0,0 @@
-#ifndef __LINUX_MUTEX_DEBUG_H
-#define __LINUX_MUTEX_DEBUG_H
-
-#include <linux/linkage.h>
-#include <linux/lockdep.h>
-#include <linux/debug_locks.h>
-
-/*
- * Mutexes - debugging helpers:
- */
-
-#define __DEBUG_MUTEX_INITIALIZER(lockname)				\
-	, .magic = &lockname
-
-#define mutex_init(mutex)						\
-do {									\
-	static struct lock_class_key __key;				\
-									\
-	__mutex_init((mutex), #mutex, &__key);				\
-} while (0)
-
-extern void mutex_destroy(struct mutex *lock);
-
-#endif
--- a/include/linux/mutex.h
+++ b/include/linux/mutex.h
@@ -18,6 +18,7 @@
 #include <linux/atomic.h>
 #include <asm/processor.h>
 #include <linux/osq_lock.h>
+#include <linux/debug_locks.h>
 
 /*
  * Simple, straightforward mutexes with strict semantics:
@@ -48,16 +49,12 @@
  *   locks and tasks (and only those tasks)
  */
 struct mutex {
-	/* 1: unlocked, 0: locked, negative: locked, possible waiters */
-	atomic_t		count;
+	atomic_long_t		owner;
 	spinlock_t		wait_lock;
-	struct list_head	wait_list;
-#if defined(CONFIG_DEBUG_MUTEXES) || defined(CONFIG_MUTEX_SPIN_ON_OWNER)
-	struct task_struct	*owner;
-#endif
 #ifdef CONFIG_MUTEX_SPIN_ON_OWNER
 	struct optimistic_spin_queue osq; /* Spinner MCS lock */
 #endif
+	struct list_head	wait_list;
 #ifdef CONFIG_DEBUG_MUTEXES
 	void			*magic;
 #endif
@@ -66,6 +63,11 @@ struct mutex {
 #endif
 };
 
+static inline struct task_struct *__mutex_owner(struct mutex *lock)
+{
+	return (struct task_struct *)(atomic_long_read(&lock->owner) & ~0x03);
+}
+
 /*
  * This is the control structure for tasks blocked on mutex,
  * which resides on the blocked task's kernel stack:
@@ -79,9 +81,20 @@ struct mutex_waiter {
 };
 
 #ifdef CONFIG_DEBUG_MUTEXES
-# include <linux/mutex-debug.h>
+
+#define __DEBUG_MUTEX_INITIALIZER(lockname)				\
+	, .magic = &lockname
+
+extern void mutex_destroy(struct mutex *lock);
+
 #else
+
 # define __DEBUG_MUTEX_INITIALIZER(lockname)
+
+static inline void mutex_destroy(struct mutex *lock) {}
+
+#endif
+
 /**
  * mutex_init - initialize the mutex
  * @mutex: the mutex to be initialized
@@ -90,14 +103,12 @@ struct mutex_waiter {
  *
  * It is not allowed to initialize an already locked mutex.
  */
-# define mutex_init(mutex) \
-do {							\
-	static struct lock_class_key __key;		\
-							\
-	__mutex_init((mutex), #mutex, &__key);		\
+#define mutex_init(mutex)						\
+do {									\
+	static struct lock_class_key __key;				\
+									\
+	__mutex_init((mutex), #mutex, &__key);				\
 } while (0)
-static inline void mutex_destroy(struct mutex *lock) {}
-#endif
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 # define __DEP_MAP_MUTEX_INITIALIZER(lockname) \
@@ -107,7 +118,7 @@ static inline void mutex_destroy(struct
 #endif
 
 #define __MUTEX_INITIALIZER(lockname) \
-		{ .count = ATOMIC_INIT(1) \
+		{ .owner = ATOMIC_LONG_INIT(0) \
 		, .wait_lock = __SPIN_LOCK_UNLOCKED(lockname.wait_lock) \
 		, .wait_list = LIST_HEAD_INIT(lockname.wait_list) \
 		__DEBUG_MUTEX_INITIALIZER(lockname) \
@@ -127,7 +138,10 @@ extern void __mutex_init(struct mutex *l
  */
 static inline int mutex_is_locked(struct mutex *lock)
 {
-	return atomic_read(&lock->count) != 1;
+	/*
+	 * XXX think about spin_is_locked
+	 */
+	return __mutex_owner(lock) != NULL;
 }
 
 /*
--- a/kernel/locking/mutex-debug.c
+++ b/kernel/locking/mutex-debug.c
@@ -73,21 +73,8 @@ void debug_mutex_unlock(struct mutex *lo
 {
 	if (likely(debug_locks)) {
 		DEBUG_LOCKS_WARN_ON(lock->magic != lock);
-
-		if (!lock->owner)
-			DEBUG_LOCKS_WARN_ON(!lock->owner);
-		else
-			DEBUG_LOCKS_WARN_ON(lock->owner != current);
-
 		DEBUG_LOCKS_WARN_ON(!lock->wait_list.prev && !lock->wait_list.next);
 	}
-
-	/*
-	 * __mutex_slowpath_needs_to_unlock() is explicitly 0 for debug
-	 * mutexes so that we can do it here after we've verified state.
-	 */
-	mutex_clear_owner(lock);
-	atomic_set(&lock->count, 1);
 }
 
 void debug_mutex_init(struct mutex *lock, const char *name,
--- a/kernel/locking/mutex-debug.h
+++ b/kernel/locking/mutex-debug.h
@@ -27,16 +27,6 @@ extern void debug_mutex_unlock(struct mu
 extern void debug_mutex_init(struct mutex *lock, const char *name,
 			     struct lock_class_key *key);
 
-static inline void mutex_set_owner(struct mutex *lock)
-{
-	WRITE_ONCE(lock->owner, current);
-}
-
-static inline void mutex_clear_owner(struct mutex *lock)
-{
-	WRITE_ONCE(lock->owner, NULL);
-}
-
 #define spin_lock_mutex(lock, flags)			\
 	do {						\
 		struct mutex *l = container_of(lock, struct mutex, wait_lock); \
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -27,41 +27,113 @@
 #include <linux/debug_locks.h>
 #include <linux/osq_lock.h>
 
-/*
- * In the DEBUG case we are using the "NULL fastpath" for mutexes,
- * which forces all calls into the slowpath:
- */
 #ifdef CONFIG_DEBUG_MUTEXES
 # include "mutex-debug.h"
-# include <asm-generic/mutex-null.h>
-/*
- * Must be 0 for the debug case so we do not do the unlock outside of the
- * wait_lock region. debug_mutex_unlock() will do the actual unlock in this
- * case.
- */
-# undef __mutex_slowpath_needs_to_unlock
-# define  __mutex_slowpath_needs_to_unlock()	0
 #else
 # include "mutex.h"
-# include <asm/mutex.h>
 #endif
 
 void
 __mutex_init(struct mutex *lock, const char *name, struct lock_class_key *key)
 {
-	atomic_set(&lock->count, 1);
+	atomic_long_set(&lock->owner, 0);
 	spin_lock_init(&lock->wait_lock);
 	INIT_LIST_HEAD(&lock->wait_list);
-	mutex_clear_owner(lock);
 #ifdef CONFIG_MUTEX_SPIN_ON_OWNER
 	osq_lock_init(&lock->osq);
 #endif
 
 	debug_mutex_init(lock, name, key);
 }
-
 EXPORT_SYMBOL(__mutex_init);
 
+/*
+ * @owner: contains: 'struct task_struct *' to the current lock owner,
+ * NULL means not owned. Since task_struct pointers are aligned at
+ * ARCH_MIN_TASKALIGN (which is at least sizeof(void *)), we have low
+ * bits to store extra state.
+ *
+ * Bit0 indicates a non-empty waiter list; unlock must issue a wakeup.
+ */
+#define MUTEX_FLAG_WAITERS	0x01
+
+#define MUTEX_FLAGS		0x03
+
+static inline struct task_struct *__owner_task(unsigned long owner)
+{
+	return (struct task_struct *)(owner & ~MUTEX_FLAGS);
+}
+
+static inline unsigned long __owner_flags(unsigned long owner)
+{
+	return owner & MUTEX_FLAGS;
+}
+
+/*
+ * Actual trylock that will work on any unlocked state.
+ */
+static inline bool __mutex_trylock(struct mutex *lock)
+{
+	unsigned long owner, curr = (unsigned long)current;
+
+	owner = atomic_long_read(&lock->owner);
+	for (;;) { /* must loop, can race against a flag */
+		unsigned long old;
+
+		if (__owner_task(owner))
+			return false;
+
+		old = atomic_long_cmpxchg_acquire(&lock->owner, owner,
+						  curr | __owner_flags(owner));
+		if (old == owner)
+			return true;
+
+		owner = old;
+	}
+}
+
+#ifndef CONFIG_DEBUG_LOCK_ALLOC
+/*
+ * Lockdep annotations are contained to the slow paths for simplicity.
+ * There is nothing that would stop spreading the lockdep annotations outwards
+ * except more code.
+ */
+
+/*
+ * Optimistic trylock that only works in the uncontended case. Make sure to
+ * follow with a __mutex_trylock() before failing.
+ */
+static __always_inline bool __mutex_trylock_fast(struct mutex *lock)
+{
+	unsigned long curr = (unsigned long)current;
+
+	if (!atomic_long_cmpxchg_acquire(&lock->owner, 0UL, curr))
+		return true;
+
+	return false;
+}
+
+static __always_inline bool __mutex_unlock_fast(struct mutex *lock)
+{
+	unsigned long curr = (unsigned long)current;
+
+	if (atomic_long_cmpxchg_release(&lock->owner, curr, 0UL) == curr)
+		return true;
+
+	return false;
+}
+#endif
+
+static inline void __mutex_set_flag(struct mutex *lock, unsigned long flag)
+{
+	atomic_long_or(flag, &lock->owner);
+}
+
+static inline void __mutex_clear_flag(struct mutex *lock, unsigned long flag)
+{
+	atomic_long_andnot(flag, &lock->owner);
+}
+
 #ifndef CONFIG_DEBUG_LOCK_ALLOC
 /*
  * We split the mutex lock/unlock logic into separate fastpath and
@@ -69,7 +141,7 @@ EXPORT_SYMBOL(__mutex_init);
  * We also put the fastpath first in the kernel image, to make sure the
  * branch is predicted by the CPU as default-untaken.
  */
-__visible void __sched __mutex_lock_slowpath(atomic_t *lock_count);
+static void __sched __mutex_lock_slowpath(struct mutex *lock);
 
 /**
  * mutex_lock - acquire the mutex
@@ -95,14 +167,10 @@ __visible void __sched __mutex_lock_slow
 void __sched mutex_lock(struct mutex *lock)
 {
 	might_sleep();
-	/*
-	 * The locking fastpath is the 1->0 transition from
-	 * 'unlocked' into 'locked' state.
-	 */
-	__mutex_fastpath_lock(&lock->count, __mutex_lock_slowpath);
-	mutex_set_owner(lock);
-}
 
+	if (!__mutex_trylock_fast(lock))
+		__mutex_lock_slowpath(lock);
+}
 EXPORT_SYMBOL(mutex_lock);
 #endif
 
@@ -149,9 +217,6 @@ static __always_inline void ww_mutex_loc
 /*
  * After acquiring lock with fastpath or when we lost out in contested
  * slowpath, set ctx and wake up any waiters so they can recheck.
- *
- * This function is never called when CONFIG_DEBUG_LOCK_ALLOC is set,
- * as the fastpath and opportunistic spinning are disabled in that case.
  */
 static __always_inline void
 ww_mutex_set_context_fastpath(struct ww_mutex *lock,
@@ -176,7 +241,7 @@ ww_mutex_set_context_fastpath(struct ww_
 	/*
 	 * Check if lock is contended, if not there is nobody to wake up
 	 */
-	if (likely(atomic_read(&lock->base.count) == 0))
+	if (likely(!(atomic_long_read(&lock->base.owner) & MUTEX_FLAG_WAITERS)))
 		return;
 
 	/*
@@ -227,7 +292,7 @@ bool mutex_spin_on_owner(struct mutex *l
 	bool ret = true;
 
 	rcu_read_lock();
-	while (lock->owner == owner) {
+	while (__mutex_owner(lock) == owner) {
 		/*
 		 * Ensure we emit the owner->on_cpu, dereference _after_
 		 * checking lock->owner still matches owner. If that fails,
@@ -260,27 +325,20 @@ static inline int mutex_can_spin_on_owne
 		return 0;
 
 	rcu_read_lock();
-	owner = READ_ONCE(lock->owner);
+	owner = __mutex_owner(lock);
 	if (owner)
 		retval = owner->on_cpu;
 	rcu_read_unlock();
+
 	/*
-	 * if lock->owner is not set, the mutex owner may have just acquired
-	 * it and not set the owner yet or the mutex has been released.
+	 * If lock->owner is not set, the mutex has been released. Return true
+	 * such that we'll trylock in the spin path, which is a faster option
+	 * than the blocking slow path.
 	 */
 	return retval;
 }
 
 /*
- * Atomically try to take the lock when it is available
- */
-static inline bool mutex_try_to_acquire(struct mutex *lock)
-{
-	return !mutex_is_locked(lock) &&
-		(atomic_cmpxchg_acquire(&lock->count, 1, 0) == 1);
-}
-
-/*
  * Optimistic spinning.
  *
  * We try to spin for acquisition when we find that the lock owner
@@ -288,13 +346,6 @@ static inline bool mutex_try_to_acquire(
  * need to reschedule. The rationale is that if the lock owner is
  * running, it is likely to release the lock soon.
  *
- * Since this needs the lock owner, and this mutex implementation
- * doesn't track the owner atomically in the lock field, we need to
- * track it non-atomically.
- *
- * We can't do this for DEBUG_MUTEXES because that relies on wait_lock
- * to serialize everything.
- *
  * The mutex spinners are queued up using MCS lock so that only one
  * spinner can compete for the mutex. However, if mutex spinning isn't
  * going to happen, there is no point in going through the lock/unlock
@@ -342,36 +393,17 @@ static bool mutex_optimistic_spin(struct
 		 * If there's an owner, wait for it to either
 		 * release the lock or go to sleep.
 		 */
-		owner = READ_ONCE(lock->owner);
+		owner = __mutex_owner(lock);
 		if (owner && !mutex_spin_on_owner(lock, owner))
 			break;
 
 		/* Try to acquire the mutex if it is unlocked. */
-		if (mutex_try_to_acquire(lock)) {
-			lock_acquired(&lock->dep_map, ip);
-
-			if (use_ww_ctx) {
-				struct ww_mutex *ww;
-				ww = container_of(lock, struct ww_mutex, base);
-
-				ww_mutex_set_context_fastpath(ww, ww_ctx);
-			}
-
-			mutex_set_owner(lock);
+		if (__mutex_trylock(lock)) {
 			osq_unlock(&lock->osq);
 			return true;
 		}
 
 		/*
-		 * When there's no owner, we might have preempted between the
-		 * owner acquiring the lock and setting the owner field. If
-		 * we're an RT task that will live-lock because we won't let
-		 * the owner complete.
-		 */
-		if (!owner && (need_resched() || rt_task(task)))
-			break;
-
-		/*
 		 * The cpu_relax() call is a compiler barrier which forces
 		 * everything in this loop to be re-loaded. We don't need
 		 * memory barriers as we'll eventually observe the right
@@ -406,8 +438,7 @@ static bool mutex_optimistic_spin(struct
 }
 #endif
 
-__visible __used noinline
-void __sched __mutex_unlock_slowpath(atomic_t *lock_count);
+static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigned long ip);
 
 /**
  * mutex_unlock - release the mutex
@@ -422,21 +453,12 @@ void __sched __mutex_unlock_slowpath(ato
  */
 void __sched mutex_unlock(struct mutex *lock)
 {
-	/*
-	 * The unlocking fastpath is the 0->1 transition from 'locked'
-	 * into 'unlocked' state:
-	 */
-#ifndef CONFIG_DEBUG_MUTEXES
-	/*
-	 * When debugging is enabled we must not clear the owner before time,
-	 * the slow path will always be taken, and that clears the owner field
-	 * after verifying that it was indeed current.
-	 */
-	mutex_clear_owner(lock);
+#ifndef CONFIG_DEBUG_LOCK_ALLOC
+	if (__mutex_unlock_fast(lock))
+		return;
 #endif
-	__mutex_fastpath_unlock(&lock->count, __mutex_unlock_slowpath);
+	__mutex_unlock_slowpath(lock, _RET_IP_);
 }
-
 EXPORT_SYMBOL(mutex_unlock);
 
 /**
@@ -465,15 +487,7 @@ void __sched ww_mutex_unlock(struct ww_m
 		lock->ctx = NULL;
 	}
 
-#ifndef CONFIG_DEBUG_MUTEXES
-	/*
-	 * When debugging is enabled we must not clear the owner before time,
-	 * the slow path will always be taken, and that clears the owner field
-	 * after verifying that it was indeed current.
-	 */
-	mutex_clear_owner(&lock->base);
-#endif
-	__mutex_fastpath_unlock(&lock->base.count, __mutex_unlock_slowpath);
+	mutex_unlock(&lock->base);
 }
 EXPORT_SYMBOL(ww_mutex_unlock);
 
@@ -520,20 +534,24 @@ __mutex_lock_common(struct mutex *lock,
 	preempt_disable();
 	mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, ip);
 
-	if (mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx)) {
+	if (__mutex_trylock(lock) || mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx)) {
 		/* got the lock, yay! */
+		lock_acquired(&lock->dep_map, ip);
+		if (use_ww_ctx) {
+			struct ww_mutex *ww;
+			ww = container_of(lock, struct ww_mutex, base);
+
+			ww_mutex_set_context_fastpath(ww, ww_ctx);
+		}
 		preempt_enable();
 		return 0;
 	}
 
 	spin_lock_mutex(&lock->wait_lock, flags);
-
 	/*
-	 * Once more, try to acquire the lock. Only try-lock the mutex if
-	 * it is unlocked to reduce unnecessary xchg() operations.
+	 * After waiting to acquire the wait_lock, try again.
 	 */
-	if (!mutex_is_locked(lock) &&
-	    (atomic_xchg_acquire(&lock->count, 0) == 1))
+	if (__mutex_trylock(lock))
 		goto skip_wait;
 
 	debug_mutex_lock_common(lock, &waiter);
@@ -543,21 +561,13 @@ __mutex_lock_common(struct mutex *lock,
 	list_add_tail(&waiter.list, &lock->wait_list);
 	waiter.task = task;
 
+	if (list_first_entry(&lock->wait_list, struct mutex_waiter, list) == &waiter)
+		__mutex_set_flag(lock, MUTEX_FLAG_WAITERS);
+
 	lock_contended(&lock->dep_map, ip);
 
 	for (;;) {
-		/*
-		 * Lets try to take the lock again - this is needed even if
-		 * we get here for the first time (shortly after failing to
-		 * acquire the lock), to make sure that we get a wakeup once
-		 * it's unlocked. Later on, if we sleep, this is the
-		 * operation that gives us the lock. We xchg it to -1, so
-		 * that when we release the lock, we properly wake up the
-		 * other waiters. We only attempt the xchg if the count is
-		 * non-negative in order to avoid unnecessary xchg operations:
-		 */
-		if (atomic_read(&lock->count) >= 0 &&
-		    (atomic_xchg_acquire(&lock->count, -1) == 1))
+		if (__mutex_trylock(lock))
 			break;
 
 		/*
@@ -585,15 +595,14 @@ __mutex_lock_common(struct mutex *lock,
 	__set_task_state(task, TASK_RUNNING);
 
 	mutex_remove_waiter(lock, &waiter, task);
-	/* set it to 0 if there are no waiters left: */
 	if (likely(list_empty(&lock->wait_list)))
-		atomic_set(&lock->count, 0);
+		__mutex_clear_flag(lock, MUTEX_FLAG_WAITERS);
+
 	debug_mutex_free_waiter(&waiter);
 
 skip_wait:
 	/* got the lock - cleanup and rejoice! */
 	lock_acquired(&lock->dep_map, ip);
-	mutex_set_owner(lock);
 
 	if (use_ww_ctx) {
 		struct ww_mutex *ww = container_of(lock, struct ww_mutex, base);
@@ -631,7 +640,6 @@ _mutex_lock_nest_lock(struct mutex *lock
 	__mutex_lock_common(lock, TASK_UNINTERRUPTIBLE,
 			    0, nest, _RET_IP_, NULL, 0);
 }
-
 EXPORT_SYMBOL_GPL(_mutex_lock_nest_lock);
 
 int __sched
@@ -650,7 +658,6 @@ mutex_lock_interruptible_nested(struct m
 	return __mutex_lock_common(lock, TASK_INTERRUPTIBLE,
 				   subclass, NULL, _RET_IP_, NULL, 0);
 }
-
 EXPORT_SYMBOL_GPL(mutex_lock_interruptible_nested);
 
 static inline int
@@ -715,29 +722,22 @@ EXPORT_SYMBOL_GPL(__ww_mutex_lock_interr
 /*
  * Release the lock, slowpath:
  */
-static inline void
-__mutex_unlock_common_slowpath(struct mutex *lock, int nested)
+static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigned long ip)
 {
-	unsigned long flags;
+	unsigned long owner, flags;
 	WAKE_Q(wake_q);
 
+	mutex_release(&lock->dep_map, 1, ip);
+
 	/*
-	 * As a performance measurement, release the lock before doing other
-	 * wakeup related duties to follow. This allows other tasks to acquire
-	 * the lock sooner, while still handling cleanups in past unlock calls.
-	 * This can be done as we do not enforce strict equivalence between the
-	 * mutex counter and wait_list.
-	 *
-	 *
-	 * Some architectures leave the lock unlocked in the fastpath failure
-	 * case, others need to leave it locked. In the later case we have to
-	 * unlock it here - as the lock counter is currently 0 or negative.
+	 * Release the lock before (potentially) taking the spinlock
+	 * such that other contenders can get on with things ASAP.
 	 */
-	if (__mutex_slowpath_needs_to_unlock())
-		atomic_set(&lock->count, 1);
+	owner = atomic_long_fetch_and_release(MUTEX_FLAGS, &lock->owner);
+	if (!__owner_flags(owner))
+		return;
 
 	spin_lock_mutex(&lock->wait_lock, flags);
-	mutex_release(&lock->dep_map, nested, _RET_IP_);
 	debug_mutex_unlock(lock);
 
 	if (!list_empty(&lock->wait_list)) {
@@ -754,17 +754,6 @@ __mutex_unlock_common_slowpath(struct mu
 	wake_up_q(&wake_q);
 }
 
-/*
- * Release the lock, slowpath:
- */
-__visible void
-__mutex_unlock_slowpath(atomic_t *lock_count)
-{
-	struct mutex *lock = container_of(lock_count, struct mutex, count);
-
-	__mutex_unlock_common_slowpath(lock, 1);
-}
-
 #ifndef CONFIG_DEBUG_LOCK_ALLOC
 /*
  * Here come the less common (and hence less performance-critical) APIs:
@@ -789,38 +778,30 @@ __mutex_lock_interruptible_slowpath(stru
  */
 int __sched mutex_lock_interruptible(struct mutex *lock)
 {
-	int ret;
-
 	might_sleep();
-	ret =  __mutex_fastpath_lock_retval(&lock->count);
-	if (likely(!ret)) {
-		mutex_set_owner(lock);
+
+	if (__mutex_trylock_fast(lock))
 		return 0;
-	} else
-		return __mutex_lock_interruptible_slowpath(lock);
+
+	return __mutex_lock_interruptible_slowpath(lock);
 }
 
 EXPORT_SYMBOL(mutex_lock_interruptible);
 
 int __sched mutex_lock_killable(struct mutex *lock)
 {
-	int ret;
-
 	might_sleep();
-	ret = __mutex_fastpath_lock_retval(&lock->count);
-	if (likely(!ret)) {
-		mutex_set_owner(lock);
+
+	if (__mutex_trylock_fast(lock))
 		return 0;
-	} else
-		return __mutex_lock_killable_slowpath(lock);
+
+	return __mutex_lock_killable_slowpath(lock);
 }
 EXPORT_SYMBOL(mutex_lock_killable);
 
-__visible void __sched
-__mutex_lock_slowpath(atomic_t *lock_count)
+static noinline void __sched
+__mutex_lock_slowpath(struct mutex *lock)
 {
-	struct mutex *lock = container_of(lock_count, struct mutex, count);
-
 	__mutex_lock_common(lock, TASK_UNINTERRUPTIBLE, 0,
 			    NULL, _RET_IP_, NULL, 0);
 }
@@ -856,37 +837,6 @@ __ww_mutex_lock_interruptible_slowpath(s
 
 #endif
 
-/*
- * Spinlock based trylock, we take the spinlock and check whether we
- * can get the lock:
- */
-static inline int __mutex_trylock_slowpath(atomic_t *lock_count)
-{
-	struct mutex *lock = container_of(lock_count, struct mutex, count);
-	unsigned long flags;
-	int prev;
-
-	/* No need to trylock if the mutex is locked. */
-	if (mutex_is_locked(lock))
-		return 0;
-
-	spin_lock_mutex(&lock->wait_lock, flags);
-
-	prev = atomic_xchg_acquire(&lock->count, -1);
-	if (likely(prev == 1)) {
-		mutex_set_owner(lock);
-		mutex_acquire(&lock->dep_map, 0, 1, _RET_IP_);
-	}
-
-	/* Set it back to 0 if there are no waiters: */
-	if (likely(list_empty(&lock->wait_list)))
-		atomic_set(&lock->count, 0);
-
-	spin_unlock_mutex(&lock->wait_lock, flags);
-
-	return prev == 1;
-}
-
 /**
  * mutex_trylock - try to acquire the mutex, without waiting
  * @lock: the mutex to be acquired
@@ -903,13 +853,12 @@ static inline int __mutex_trylock_slowpa
  */
 int __sched mutex_trylock(struct mutex *lock)
 {
-	int ret;
+	bool locked = __mutex_trylock(lock);
 
-	ret = __mutex_fastpath_trylock(&lock->count, __mutex_trylock_slowpath);
-	if (ret)
-		mutex_set_owner(lock);
+	if (locked)
+		mutex_acquire(&lock->dep_map, 0, 1, _RET_IP_);
 
-	return ret;
+	return locked;
 }
 EXPORT_SYMBOL(mutex_trylock);
 
@@ -917,36 +866,28 @@ EXPORT_SYMBOL(mutex_trylock);
 int __sched
 __ww_mutex_lock(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)
 {
-	int ret;
-
 	might_sleep();
 
-	ret = __mutex_fastpath_lock_retval(&lock->base.count);
-
-	if (likely(!ret)) {
+	if (__mutex_trylock_fast(&lock->base)) {
 		ww_mutex_set_context_fastpath(lock, ctx);
-		mutex_set_owner(&lock->base);
-	} else
-		ret = __ww_mutex_lock_slowpath(lock, ctx);
-	return ret;
+		return 0;
+	}
+
+	return __ww_mutex_lock_slowpath(lock, ctx);
 }
 EXPORT_SYMBOL(__ww_mutex_lock);
 
 int __sched
 __ww_mutex_lock_interruptible(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)
 {
-	int ret;
-
 	might_sleep();
 
-	ret = __mutex_fastpath_lock_retval(&lock->base.count);
-
-	if (likely(!ret)) {
+	if (__mutex_trylock_fast(&lock->base)) {
 		ww_mutex_set_context_fastpath(lock, ctx);
-		mutex_set_owner(&lock->base);
-	} else
-		ret = __ww_mutex_lock_interruptible_slowpath(lock, ctx);
-	return ret;
+		return 0;
+	}
+
+	return __ww_mutex_lock_interruptible_slowpath(lock, ctx);
 }
 EXPORT_SYMBOL(__ww_mutex_lock_interruptible);
 
--- a/kernel/locking/mutex.h
+++ b/kernel/locking/mutex.h
@@ -16,32 +16,6 @@
 #define mutex_remove_waiter(lock, waiter, task) \
 		__list_del((waiter)->list.prev, (waiter)->list.next)
 
-#ifdef CONFIG_MUTEX_SPIN_ON_OWNER
-/*
- * The mutex owner can get read and written to locklessly.
- * We should use WRITE_ONCE when writing the owner value to
- * avoid store tearing, otherwise, a thread could potentially
- * read a partially written and incomplete owner value.
- */
-static inline void mutex_set_owner(struct mutex *lock)
-{
-	WRITE_ONCE(lock->owner, current);
-}
-
-static inline void mutex_clear_owner(struct mutex *lock)
-{
-	WRITE_ONCE(lock->owner, NULL);
-}
-#else
-static inline void mutex_set_owner(struct mutex *lock)
-{
-}
-
-static inline void mutex_clear_owner(struct mutex *lock)
-{
-}
-#endif
-
 #define debug_mutex_wake_waiter(lock, waiter)		do { } while (0)
 #define debug_mutex_free_waiter(waiter)			do { } while (0)
 #define debug_mutex_add_waiter(lock, waiter, ti)	do { } while (0)
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -75,11 +75,11 @@
 #include <linux/compiler.h>
 #include <linux/frame.h>
 #include <linux/prefetch.h>
+#include <linux/mutex.h>
 
 #include <asm/switch_to.h>
 #include <asm/tlb.h>
 #include <asm/irq_regs.h>
-#include <asm/mutex.h>
 #ifdef CONFIG_PARAVIRT
 #include <asm/paravirt.h>
 #endif

[toc] | [prev] | [next] | [standalone]

#1499880 — Re: [PATCH -v4 2/8] locking/mutex: Rework mutex::owner

On Fri, 07 Oct 2016, Peter Zijlstra wrote:
>+/*
>+ * Optimistic trylock that only works in the uncontended case. Make sure to
>+ * follow with a __mutex_trylock() before failing.
>+ */
>+static __always_inline bool __mutex_trylock_fast(struct mutex *lock)
>+{
>+	unsigned long curr = (unsigned long)current;
>+
>+	if (!atomic_long_cmpxchg_acquire(&lock->owner, 0UL, curr))
>+		return true;

Do we want to do a ccas check for !lock->owner? Although I can see a possible
case of 'optimizing for the contended' reasons for nay.

Thanks,
Davidlohr

[toc] | [prev] | [next] | [standalone]

#1499906 — Re: [PATCH -v4 2/8] locking/mutex: Rework mutex::owner

On Wed, 2016-10-12 at 10:59 -0700, Davidlohr Bueso wrote:
> On Fri, 07 Oct 2016, Peter Zijlstra wrote:
> >+/*
> >+ * Optimistic trylock that only works in the uncontended case. Make sure to
> >+ * follow with a __mutex_trylock() before failing.
> >+ */
> >+static __always_inline bool __mutex_trylock_fast(struct mutex *lock)
> >+{
> >+	unsigned long curr = (unsigned long)current;
> >+
> >+	if (!atomic_long_cmpxchg_acquire(&lock->owner, 0UL, curr))
> >+		return true;
> 
> Do we want to do a ccas check for !lock->owner? Although I can see a possible
> case of 'optimizing for the contended' reasons for nay.

Since this is the fast path version that gets used in mutex_lock(),
ect..., I think it would make sense to keep it like it is so that we
optimize it for the "common" case. This trylock function is more likely
to succeed as it is used for the initial attempt to get the mutex, so I
think we could avoid the ccas in the trylock_fast().

Jason

[toc] | [prev] | [next] | [standalone]

#1500338 — Re: [PATCH -v4 2/8] locking/mutex: Rework mutex::owner

On Fri, Oct 07, 2016 at 04:52:45PM +0200, Peter Zijlstra wrote:
> The current mutex implementation has an atomic lock word and a
> non-atomic owner field.
> 
> This disparity leads to a number of issues with the current mutex code
> as it means that we can have a locked mutex without an explicit owner
> (because the owner field has not been set, or already cleared).
> 
> This leads to a number of weird corner cases, esp. between the
> optimistic spinning and debug code. Where the optimistic spinning
> code needs the owner field updated inside the lock region, the debug
> code is more relaxed because the whole lock is serialized by the
> wait_lock.
> 
> Also, the spinning code itself has a few corner cases where we need to
> deal with a held lock without an owner field.
> 
> Furthermore, it becomes even more of a problem when trying to fix
> starvation cases in the current code. We end up stacking special case
> on special case.
> 
> To solve this rework the basic mutex implementation to be a single
> atomic word that contains the owner and uses the low bits for extra
> state.
> 
> This matches how PI futexes and rt_mutex already work. By having the
> owner an integral part of the lock state a lot of the problems
> dissapear and we get a better option to deal with starvation cases,
> direct owner handoff.
> 
> Changing the basic mutex does however invalidate all the arch specific
> mutex code; this patch leaves that unused in-place, a later patch will
> remove that.
> 
> 
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
>  include/linux/mutex-debug.h  |   24 --
>  include/linux/mutex.h        |   46 +++--
>  kernel/locking/mutex-debug.c |   13 -
>  kernel/locking/mutex-debug.h |   10 -
>  kernel/locking/mutex.c       |  371 ++++++++++++++++++-------------------------
>  kernel/locking/mutex.h       |   26 ---
>  kernel/sched/core.c          |    2 
>  7 files changed, 187 insertions(+), 305 deletions(-)

Looks good to me:

Reviewed-by: Will Deacon <will.deacon@arm.com>

Will

[toc] | [prev] | [next] | [standalone]

#1497261 — [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid starvation

Implement lock handoff to avoid lock starvation.

Lock starvation is possible because mutex_lock() allows lock stealing,
where a running (or optimistic spinning) task beats the woken waiter
to the acquire.

Lock stealing is an important performance optimization because waiting
for a waiter to wake up and get runtime can take a significant time,
during which everyboy would stall on the lock.

The down-side is of course that it allows for starvation.

This patch has the waiter requesting a handoff if it fails to acquire
the lock upon waking. This re-introduces some of the wait time,
because once we do a handoff we have to wait for the waiter to wake up
again.

A future patch will add a round of optimistic spinning to attempt to
alleviate this penalty, but if that turns out to not be enough, we can
add a counter and only request handoff after multiple failed wakeups.

There are a few tricky implementation details:

 - accepting a handoff must only be done in the wait-loop. Since the
   handoff condition is owner == current, it can easily cause
   recursive locking trouble.

 - accepting the handoff must be careful to provide the ACQUIRE
   semantics.

 - having the HANDOFF bit set on unlock requires care, we must not
   clear the owner.

 - we must be careful to not leave HANDOFF set after we've acquired
   the lock. The tricky scenario is setting the HANDOFF bit on an
   unlocked mutex.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/locking/mutex.c |  142 +++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 119 insertions(+), 23 deletions(-)

--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -54,8 +54,10 @@ EXPORT_SYMBOL(__mutex_init);
  * bits to store extra state.
  *
  * Bit0 indicates a non-empty waiter list; unlock must issue a wakeup.
+ * Bit1 indicates unlock needs to hand the lock to the top-waiter
  */
 #define MUTEX_FLAG_WAITERS	0x01
+#define MUTEX_FLAG_HANDOFF	0x02
 
 #define MUTEX_FLAGS		0x03
 
@@ -71,20 +73,48 @@ static inline unsigned long __owner_flag
 
 /*
  * Actual trylock that will work on any unlocked state.
+ *
+ * When setting the owner field, we must preserve the low flag bits.
+ *
+ * Be careful with @handoff, only set that in a wait-loop (where you set
+ * HANDOFF) to avoid recursive lock attempts.
  */
-static inline bool __mutex_trylock(struct mutex *lock)
+static inline bool __mutex_trylock(struct mutex *lock, const bool handoff)
 {
 	unsigned long owner, curr = (unsigned long)current;
 
 	owner = atomic_long_read(&lock->owner);
 	for (;;) { /* must loop, can race against a flag */
-		unsigned long old;
+		unsigned long old, flags = __owner_flags(owner);
+
+		if (__owner_task(owner)) {
+			if (handoff && unlikely(__owner_task(owner) == current)) {
+				/*
+				 * Provide ACQUIRE semantics for the lock-handoff.
+				 *
+				 * We cannot easily use load-acquire here, since
+				 * the actual load is a failed cmpxchg, which
+				 * doesn't imply any barriers.
+				 *
+				 * Also, this is a fairly unlikely scenario, and
+				 * this contains the cost.
+				 */
+				smp_mb(); /* ACQUIRE */
+				return true;
+			}
 
-		if (__owner_task(owner))
 			return false;
+		}
 
-		old = atomic_long_cmpxchg_acquire(&lock->owner, owner,
-						  curr | __owner_flags(owner));
+		/*
+		 * We set the HANDOFF bit, we must make sure it doesn't live
+		 * past the point where we acquire it. This would be possible
+		 * if we (accidentally) set the bit on an unlocked mutex.
+		 */
+		if (handoff)
+			flags &= ~MUTEX_FLAG_HANDOFF;
+
+		old = atomic_long_cmpxchg_acquire(&lock->owner, owner, curr | flags);
 		if (old == owner)
 			return true;
 
@@ -134,6 +164,39 @@ static inline void __mutex_clear_flag(st
 	atomic_long_andnot(flag, &lock->owner);
 }
 
+static inline bool __mutex_waiter_is_first(struct mutex *lock, struct mutex_waiter *waiter)
+{
+	return list_first_entry(&lock->wait_list, struct mutex_waiter, list) == waiter;
+}
+
+/*
+ * Give up ownership to a specific task, when @task = NULL, this is equivalent
+ * to a regular unlock. Clears HANDOFF, preserves WAITERS. Provides RELEASE
+ * semantics like a regular unlock, the __mutex_trylock() provides matching
+ * ACQUIRE semantics for the handoff.
+ */
+static void __mutex_handoff(struct mutex *lock, struct task_struct *task)
+{
+	unsigned long owner = atomic_long_read(&lock->owner);
+
+	for (;;) {
+		unsigned long old, new;
+
+#ifdef CONFIG_DEBUG_MUTEXES
+		DEBUG_LOCKS_WARN_ON(__owner_task(owner) != current);
+#endif
+
+		new = (owner & MUTEX_FLAG_WAITERS);
+		new |= (unsigned long)task;
+
+		old = atomic_long_cmpxchg_release(&lock->owner, owner, new);
+		if (old == owner)
+			break;
+
+		owner = old;
+	}
+}
+
 #ifndef CONFIG_DEBUG_LOCK_ALLOC
 /*
  * We split the mutex lock/unlock logic into separate fastpath and
@@ -398,7 +461,7 @@ static bool mutex_optimistic_spin(struct
 			break;
 
 		/* Try to acquire the mutex if it is unlocked. */
-		if (__mutex_trylock(lock)) {
+		if (__mutex_trylock(lock, false)) {
 			osq_unlock(&lock->osq);
 			return true;
 		}
@@ -523,6 +586,7 @@ __mutex_lock_common(struct mutex *lock,
 	struct task_struct *task = current;
 	struct mutex_waiter waiter;
 	unsigned long flags;
+	bool first = false;
 	int ret;
 
 	if (use_ww_ctx) {
@@ -534,7 +598,8 @@ __mutex_lock_common(struct mutex *lock,
 	preempt_disable();
 	mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, ip);
 
-	if (__mutex_trylock(lock) || mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx)) {
+	if (__mutex_trylock(lock, false) ||
+	    mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx)) {
 		/* got the lock, yay! */
 		lock_acquired(&lock->dep_map, ip);
 		if (use_ww_ctx) {
@@ -551,7 +616,7 @@ __mutex_lock_common(struct mutex *lock,
 	/*
 	 * After waiting to acquire the wait_lock, try again.
 	 */
-	if (__mutex_trylock(lock))
+	if (__mutex_trylock(lock, false))
 		goto skip_wait;
 
 	debug_mutex_lock_common(lock, &waiter);
@@ -561,13 +626,13 @@ __mutex_lock_common(struct mutex *lock,
 	list_add_tail(&waiter.list, &lock->wait_list);
 	waiter.task = task;
 
-	if (list_first_entry(&lock->wait_list, struct mutex_waiter, list) == &waiter)
+	if (__mutex_waiter_is_first(lock, &waiter))
 		__mutex_set_flag(lock, MUTEX_FLAG_WAITERS);
 
 	lock_contended(&lock->dep_map, ip);
 
 	for (;;) {
-		if (__mutex_trylock(lock))
+		if (__mutex_trylock(lock, first))
 			break;
 
 		/*
@@ -586,17 +651,20 @@ __mutex_lock_common(struct mutex *lock,
 		}
 
 		__set_task_state(task, state);
-
-		/* didn't get the lock, go to sleep: */
 		spin_unlock_mutex(&lock->wait_lock, flags);
 		schedule_preempt_disabled();
 		spin_lock_mutex(&lock->wait_lock, flags);
+
+		if (!first && __mutex_waiter_is_first(lock, &waiter)) {
+			first = true;
+			__mutex_set_flag(lock, MUTEX_FLAG_HANDOFF);
+		}
 	}
 	__set_task_state(task, TASK_RUNNING);
 
 	mutex_remove_waiter(lock, &waiter, task);
 	if (likely(list_empty(&lock->wait_list)))
-		__mutex_clear_flag(lock, MUTEX_FLAG_WAITERS);
+		__mutex_clear_flag(lock, MUTEX_FLAGS);
 
 	debug_mutex_free_waiter(&waiter);
 
@@ -724,33 +792,61 @@ EXPORT_SYMBOL_GPL(__ww_mutex_lock_interr
  */
 static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigned long ip)
 {
+	struct task_struct *next = NULL;
 	unsigned long owner, flags;
 	WAKE_Q(wake_q);
 
 	mutex_release(&lock->dep_map, 1, ip);
 
 	/*
-	 * Release the lock before (potentially) taking the spinlock
-	 * such that other contenders can get on with things ASAP.
+	 * Release the lock before (potentially) taking the spinlock such that
+	 * other contenders can get on with things ASAP.
+	 *
+	 * Except when HANDOFF, in that case we must not clear the owner field,
+	 * but instead set it to the top waiter.
 	 */
-	owner = atomic_long_fetch_and_release(MUTEX_FLAGS, &lock->owner);
-	if (!__owner_flags(owner))
-		return;
+	owner = atomic_long_read(&lock->owner);
+	for (;;) {
+		unsigned long old;
+
+#ifdef CONFIG_DEBUG_MUTEXES
+		DEBUG_LOCKS_WARN_ON(__owner_task(owner) != current);
+#endif
+
+		if (owner & MUTEX_FLAG_HANDOFF)
+			break;
+
+		old = atomic_long_cmpxchg_release(&lock->owner, owner,
+						  __owner_flags(owner));
+		if (old == owner) {
+			if (owner & MUTEX_FLAG_WAITERS)
+				break;
+
+			return;
+		}
+
+		owner = old;
+	}
 
 	spin_lock_mutex(&lock->wait_lock, flags);
 	debug_mutex_unlock(lock);
-
 	if (!list_empty(&lock->wait_list)) {
 		/* get the first entry from the wait-list: */
 		struct mutex_waiter *waiter =
-				list_entry(lock->wait_list.next,
-					   struct mutex_waiter, list);
+			list_first_entry(&lock->wait_list,
+					 struct mutex_waiter, list);
+
+		next = waiter->task;
 
 		debug_mutex_wake_waiter(lock, waiter);
-		wake_q_add(&wake_q, waiter->task);
+		wake_q_add(&wake_q, next);
 	}
 
+	if (owner & MUTEX_FLAG_HANDOFF)
+		__mutex_handoff(lock, next);
+
 	spin_unlock_mutex(&lock->wait_lock, flags);
+
 	wake_up_q(&wake_q);
 }
 
@@ -853,7 +949,7 @@ __ww_mutex_lock_interruptible_slowpath(s
  */
 int __sched mutex_trylock(struct mutex *lock)
 {
-	bool locked = __mutex_trylock(lock);
+	bool locked = __mutex_trylock(lock, false);
 
 	if (locked)
 		mutex_acquire(&lock->dep_map, 0, 1, _RET_IP_);

[toc] | [prev] | [next] | [standalone]

#1500349 — Re: [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid starvation

Hi Peter,

Just one comment below.

On Fri, Oct 07, 2016 at 04:52:48PM +0200, Peter Zijlstra wrote:
> --- a/kernel/locking/mutex.c
> +++ b/kernel/locking/mutex.c
> @@ -54,8 +54,10 @@ EXPORT_SYMBOL(__mutex_init);
>   * bits to store extra state.
>   *
>   * Bit0 indicates a non-empty waiter list; unlock must issue a wakeup.
> + * Bit1 indicates unlock needs to hand the lock to the top-waiter
>   */
>  #define MUTEX_FLAG_WAITERS	0x01
> +#define MUTEX_FLAG_HANDOFF	0x02
>  
>  #define MUTEX_FLAGS		0x03
>  
> @@ -71,20 +73,48 @@ static inline unsigned long __owner_flag
>  
>  /*
>   * Actual trylock that will work on any unlocked state.
> + *
> + * When setting the owner field, we must preserve the low flag bits.
> + *
> + * Be careful with @handoff, only set that in a wait-loop (where you set
> + * HANDOFF) to avoid recursive lock attempts.
>   */
> -static inline bool __mutex_trylock(struct mutex *lock)
> +static inline bool __mutex_trylock(struct mutex *lock, const bool handoff)
>  {
>  	unsigned long owner, curr = (unsigned long)current;
>  
>  	owner = atomic_long_read(&lock->owner);
>  	for (;;) { /* must loop, can race against a flag */
> -		unsigned long old;
> +		unsigned long old, flags = __owner_flags(owner);
> +
> +		if (__owner_task(owner)) {
> +			if (handoff && unlikely(__owner_task(owner) == current)) {
> +				/*
> +				 * Provide ACQUIRE semantics for the lock-handoff.
> +				 *
> +				 * We cannot easily use load-acquire here, since
> +				 * the actual load is a failed cmpxchg, which
> +				 * doesn't imply any barriers.
> +				 *
> +				 * Also, this is a fairly unlikely scenario, and
> +				 * this contains the cost.
> +				 */
> +				smp_mb(); /* ACQUIRE */

As we discussed on another thread recently, a failed cmpxchg_acquire
will always give you ACQUIRE semantics in practice. Maybe we should update
the documentation to allow this? The only special case is the full-barrier
version.

Will

[toc] | [prev] | [next] | [standalone]

#1501812 — Re: [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid starvation

On Thu, Oct 13, 2016 at 04:14:47PM +0100, Will Deacon wrote:

> > +		if (__owner_task(owner)) {
> > +			if (handoff && unlikely(__owner_task(owner) == current)) {
> > +				/*
> > +				 * Provide ACQUIRE semantics for the lock-handoff.
> > +				 *
> > +				 * We cannot easily use load-acquire here, since
> > +				 * the actual load is a failed cmpxchg, which
> > +				 * doesn't imply any barriers.
> > +				 *
> > +				 * Also, this is a fairly unlikely scenario, and
> > +				 * this contains the cost.
> > +				 */
> > +				smp_mb(); /* ACQUIRE */
> 
> As we discussed on another thread recently, a failed cmpxchg_acquire
> will always give you ACQUIRE semantics in practice. Maybe we should update
> the documentation to allow this? The only special case is the full-barrier
> version.

So on PPC we do:

static __always_inline unsigned long
__cmpxchg_u32_acquire(u32 *p, unsigned long old, unsigned long new)
{
        unsigned long prev;

        __asm__ __volatile__ (
"1:     lwarx   %0,0,%2         # __cmpxchg_u32_acquire\n"
"       cmpw    0,%0,%3\n"
"       bne-    2f\n"
        PPC405_ERR77(0, %2)
"       stwcx.  %4,0,%2\n"
"       bne-    1b\n"
        PPC_ACQUIRE_BARRIER
        "\n"
"2:"
        : "=&r" (prev), "+m" (*p)
        : "r" (p), "r" (old), "r" (new)
        : "cc", "memory");

        return prev;
}

which I read to skip over the ACQUIRE_BARRIER on fail.


Similarly, we _could_ make the generic version skip the barrier entirely
(we currently do not it seems).


And while I agree that it makes semantic sense, in that we always issue
the LOAD, and since we defined the ACQUIRE to apply to the LOADs only,
and we always issue the LOAD, we should also always provide ACQUIRE
semantics. I'm not entirely convinced we should go there just yet. It
would make failed cmpxchg_acquire()'s more expensive, and this really is
the only place we care about those.


So I would propose for now we keep these explicit barriers; both here
and the other place you mentioned, but keep this in mind.

Also, I don't feel we need more complexity in this patch set just now.

[toc] | [prev] | [next] | [standalone]

#1502954 — Re: [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid starvation

On Mon, Oct 17, 2016 at 02:45:50PM -0400, Waiman Long wrote:
> On 10/07/2016 10:52 AM, Peter Zijlstra wrote:
> >  /*
> >   * Actual trylock that will work on any unlocked state.
> >+ *
> >+ * When setting the owner field, we must preserve the low flag bits.
> >+ *
> >+ * Be careful with @handoff, only set that in a wait-loop (where you set
> >+ * HANDOFF) to avoid recursive lock attempts.
> >   */
> >-static inline bool __mutex_trylock(struct mutex *lock)
> >+static inline bool __mutex_trylock(struct mutex *lock, const bool handoff)
> >  {
> >  	unsigned long owner, curr = (unsigned long)current;
> >
> >  	owner = atomic_long_read(&lock->owner);
> >  	for (;;) { /* must loop, can race against a flag */
> >-		unsigned long old;
> >+		unsigned long old, flags = __owner_flags(owner);
> >+
> >+		if (__owner_task(owner)) {
> >+			if (handoff&&  unlikely(__owner_task(owner) == current)) {
> >+				/*
> >+				 * Provide ACQUIRE semantics for the lock-handoff.
> >+				 *
> >+				 * We cannot easily use load-acquire here, since
> >+				 * the actual load is a failed cmpxchg, which
> >+				 * doesn't imply any barriers.
> >+				 *
> >+				 * Also, this is a fairly unlikely scenario, and
> >+				 * this contains the cost.
> >+				 */
> 
> I am not so sure about your comment here. I guess you are referring to the
> atomic_long_cmpxchg_acquire below for the failed cmpxchg. However, it is
> also possible that the path can be triggered on the first round without
> cmpxchg. Maybe we can do a load_acquire on the owner again to satisfy this
> requirement without a smp_mb().

Yes, I refer to the atomic_long_cmpxchg_acquire() below. If that cmpxchg
fails, no barriers are implied.

Yes we could fix that, but that would make all cmpxchg_acquire() loops
more expensive. And only fixing the initial load doesn't help, since we
still need to deal with the cmpxchg case failing.

We _could_ re-issue the load I suppose, because since if owner==current,
nobody else is going to change it, but I feel slightly uneasy with that.
Also, its a relative slow path, so for now, lets keep it simple and
explicit like so.

> 
> >+				smp_mb(); /* ACQUIRE */
> >+				return true;
> >+			}
> >
> >-		if (__owner_task(owner))
> >  			return false;
> >+		}
> >
> >-		old = atomic_long_cmpxchg_acquire(&lock->owner, owner,
> >-						  curr | __owner_flags(owner));
> >+		/*
> >+		 * We set the HANDOFF bit, we must make sure it doesn't live
> >+		 * past the point where we acquire it. This would be possible
> >+		 * if we (accidentally) set the bit on an unlocked mutex.
> >+		 */
> >+		if (handoff)
> >+			flags&= ~MUTEX_FLAG_HANDOFF;
> >+
> >+		old = atomic_long_cmpxchg_acquire(&lock->owner, owner, curr | flags);
> >  		if (old == owner)
> >  			return true;
> >
> >
> 
> Other than that, the code is fine.
> 
> Reviewed-by: Waiman Long <Waiman.Long@hpe.com>

Thanks!

[toc] | [prev] | [next] | [standalone]

#1502976 — Re: [PATCH -v4 5/8] locking/mutex: Add lock handoff to avoid starvation

On Mon, Oct 17, 2016 at 03:07:54PM -0400, Waiman Long wrote:
> One more thing, I think it may be worthwhile to add another comment about
> what happens when the HANDOFF bit was set while we take the error path (goto
> err). As the actual handoff is serialized by the wait_lock, the code will
> still do the right thing. Either the next one in the queue will be handed
> off or it will be unlocked if the queue is empty.

Doesn't the next patch add just such a comment?

[toc] | [prev] | [next] | [standalone]

#1497263 — [PATCH -v4 1/8] locking/drm: Kill mutex trickery

Poking at lock internals is not cool. Since I'm going to change the
implementation this will break, take it out.

Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Rob Clark <robdclark@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 drivers/gpu/drm/i915/i915_gem_shrinker.c |   26 +++-----------------------
 drivers/gpu/drm/msm/msm_gem_shrinker.c   |   23 +++--------------------
 2 files changed, 6 insertions(+), 43 deletions(-)

--- a/drivers/gpu/drm/i915/i915_gem_shrinker.c
+++ b/drivers/gpu/drm/i915/i915_gem_shrinker.c
@@ -35,19 +35,6 @@
 #include "i915_drv.h"
 #include "i915_trace.h"
 
-static bool mutex_is_locked_by(struct mutex *mutex, struct task_struct *task)
-{
-	if (!mutex_is_locked(mutex))
-		return false;
-
-#if defined(CONFIG_DEBUG_MUTEXES) || defined(CONFIG_MUTEX_SPIN_ON_OWNER)
-	return mutex->owner == task;
-#else
-	/* Since UP may be pre-empted, we cannot assume that we own the lock */
-	return false;
-#endif
-}
-
 static int num_vma_bound(struct drm_i915_gem_object *obj)
 {
 	struct i915_vma *vma;
@@ -238,17 +225,10 @@ unsigned long i915_gem_shrink_all(struct
 
 static bool i915_gem_shrinker_lock(struct drm_device *dev, bool *unlock)
 {
-	if (!mutex_trylock(&dev->struct_mutex)) {
-		if (!mutex_is_locked_by(&dev->struct_mutex, current))
-			return false;
-
-		if (to_i915(dev)->mm.shrinker_no_lock_stealing)
-			return false;
-
-		*unlock = false;
-	} else
-		*unlock = true;
+	if (!mutex_trylock(&dev->struct_mutex))
+		return false;
 
+	*unlock = true;
 	return true;
 }
 
--- a/drivers/gpu/drm/msm/msm_gem_shrinker.c
+++ b/drivers/gpu/drm/msm/msm_gem_shrinker.c
@@ -18,29 +18,12 @@
 #include "msm_drv.h"
 #include "msm_gem.h"
 
-static bool mutex_is_locked_by(struct mutex *mutex, struct task_struct *task)
-{
-	if (!mutex_is_locked(mutex))
-		return false;
-
-#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_MUTEXES)
-	return mutex->owner == task;
-#else
-	/* Since UP may be pre-empted, we cannot assume that we own the lock */
-	return false;
-#endif
-}
-
 static bool msm_gem_shrinker_lock(struct drm_device *dev, bool *unlock)
 {
-	if (!mutex_trylock(&dev->struct_mutex)) {
-		if (!mutex_is_locked_by(&dev->struct_mutex, current))
-			return false;
-		*unlock = false;
-	} else {
-		*unlock = true;
-	}
+	if (!mutex_trylock(&dev->struct_mutex))
+		return false;
 
+	*unlock = true;
 	return true;
 }
 

[toc] | [prev] | [next] | [standalone]

#1497293 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Fri, Oct 07, 2016 at 04:52:44PM +0200, Peter Zijlstra wrote:
> Poking at lock internals is not cool. Since I'm going to change the
> implementation this will break, take it out.


So something like the below would serve as a replacement for your
previous hacks. Is this API something acceptable to people? Ingo,
Thomas?

---
 include/linux/mutex.h | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/include/linux/mutex.h b/include/linux/mutex.h
index 4d3bccabbea5..afcff2c85957 100644
--- a/include/linux/mutex.h
+++ b/include/linux/mutex.h
@@ -189,4 +189,29 @@ extern void mutex_unlock(struct mutex *lock);
 
 extern int atomic_dec_and_mutex_lock(atomic_t *cnt, struct mutex *lock);
 
+enum mutex_trylock_recursive_enum {
+	mutex_trylock_failed = 0,
+	mutex_trylock_success = 1,
+	mutex_trylock_recursive,
+};
+
+/**
+ * mutex_trylock_recursive - trylock variant that allows recursive locking
+ * @lock: mutex to be locked
+ *
+ *
+ * Returns:
+ *  mutex_trylock_failed    - trylock failed,
+ *  mutex_trylock_success   - lock acquired,
+ *  mutex_trylock_recursive - we already owned the lock.
+ */
+static inline enum mutex_trylock_recursive_enum
+mutex_trylock_recursive(struct mutex *lock)
+{
+	if (unlikely(__mutex_owner(lock) == current))
+		return mutex_trylock_recursive;
+
+	return mutex_trylock(lock);
+}
+
 #endif /* __LINUX_MUTEX_H */

[toc] | [prev] | [next] | [standalone]

#1497310 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Fri, Oct 7, 2016 at 8:43 AM, Peter Zijlstra <peterz@infradead.org> wrote:
>
> So something like the below would serve as a replacement for your
> previous hacks. Is this API something acceptable to people? Ingo,
> Thomas?

Ugh. I think the concept is fine, but can we place make these enum's
be all upper case or something to make them really stand out visually.

Also, it's a bit subtle how this depends on the success/failure enums
to just have the same values as the return value of the non-try mutex.
So a comment about why that particular numbering is important above
the enum definition, please?

You _kind_ of imply that magic numbering requirementby explicitly
using " = 0" and " = 1", but quite frankly, in many ways that's really
nasty, because I can see somebody looking at that enum definition and
saying "that makes no sense: it sets *two* of the three values
explicitly, and the values it sets explicitly are the same that it
would get implicitly _anyway_, so I'll just clean this up".

So I really think that enum needs a comment on the forced choice of
values, and making them all caps would make the users more obvious, I
think.

The other choice would be to just make the choices be negative (==
recursive), zero (== failed) or positive (== got lock), which allows
for the same value re-use for the non-recursive case, and you could
avoid the enum entirely.

                  Linus

[toc] | [prev] | [next] | [standalone]

#1497314 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Fri, Oct 07, 2016 at 08:58:43AM -0700, Linus Torvalds wrote:
> Ugh. I think the concept is fine, but can we place make these enum's
> be all upper case or something to make them really stand out visually.

OK.

> The other choice would be to just make the choices be negative (==
> recursive), zero (== failed) or positive (== got lock), which allows
> for the same value re-use for the non-recursive case, and you could
> avoid the enum entirely.

I thought about that, but liked the enum better for having to then spell
it out.

I'll go make the enum shout and add comment as you suggest.

[toc] | [prev] | [next] | [standalone]

#1497723 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Fri, 7 Oct 2016, Peter Zijlstra wrote:
> On Fri, Oct 07, 2016 at 04:52:44PM +0200, Peter Zijlstra wrote:
> > Poking at lock internals is not cool. Since I'm going to change the
> > implementation this will break, take it out.
> 
> 
> So something like the below would serve as a replacement for your
> previous hacks. Is this API something acceptable to people? Ingo,
> Thomas?
> 
> ---
>  include/linux/mutex.h | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/include/linux/mutex.h b/include/linux/mutex.h
> index 4d3bccabbea5..afcff2c85957 100644
> --- a/include/linux/mutex.h
> +++ b/include/linux/mutex.h
> @@ -189,4 +189,29 @@ extern void mutex_unlock(struct mutex *lock);
>  
>  extern int atomic_dec_and_mutex_lock(atomic_t *cnt, struct mutex *lock);
>  
> +enum mutex_trylock_recursive_enum {
> +	mutex_trylock_failed = 0,
> +	mutex_trylock_success = 1,
> +	mutex_trylock_recursive,

Upper case enum symbols please, if at all.

> +};
> +
> +/**
> + * mutex_trylock_recursive - trylock variant that allows recursive locking
> + * @lock: mutex to be locked
> + *
> + *
> + * Returns:
> + *  mutex_trylock_failed    - trylock failed,
> + *  mutex_trylock_success   - lock acquired,
> + *  mutex_trylock_recursive - we already owned the lock.
> + */
> +static inline enum mutex_trylock_recursive_enum
> +mutex_trylock_recursive(struct mutex *lock)
> +{
> +	if (unlikely(__mutex_owner(lock) == current))
> +		return mutex_trylock_recursive;
> +
> +	return mutex_trylock(lock);
> +}

Hmm. I'm not a great fan of this, because that requires an conditional
unlock mechanism.

       res = trylock_recursive(lock);
       if (res == FAILED)
       	       goto out;
       .....

       if (res == SUCCESS)
       	       unlock(lock);

While if you actually keep track of recursion you can do:
  
      if (!trylock_recursive(lock))
		goto out;

      ....

      unlock_recursive(lock);

or even:

     lock_recursive(lock);

     unlock_recursive(lock);

That's making lock/trylock and unlock symetric, so its obvious in the
source what's going on and the recursion tracking allows for better
debugability.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1497754 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Sat, 8 Oct 2016, Peter Zijlstra wrote:
> On Sat, Oct 08, 2016 at 01:58:07PM +0200, Thomas Gleixner wrote:
> > Hmm. I'm not a great fan of this, because that requires an conditional
> > unlock mechanism.
> > 
> >        res = trylock_recursive(lock);
> >        if (res == FAILED)
> >        	       goto out;
> >        .....
> > 
> >        if (res == SUCCESS)
> >        	       unlock(lock);
> > 
> > While if you actually keep track of recursion you can do:
> >   
> >       if (!trylock_recursive(lock))
> > 		goto out;
> > 
> >       ....
> > 
> >       unlock_recursive(lock);
> > 
> > or even:
> > 
> >      lock_recursive(lock);
> > 
> >      unlock_recursive(lock);
> > 
> > That's making lock/trylock and unlock symetric, so its obvious in the
> > source what's going on and the recursion tracking allows for better
> > debugability.
> 
> Hurm,. so I thought that in general we disliked recursive locking
> because it quickly turns in to a horrible mess.
> 
> Adding such primitives makes it 'easy' to use recursive locking and then
> where does it stop?

Well, when you add just trylock_recursive then people are going to use it
anyway no matter whether it is easy or not.

So if we decide to provide something which supports recursive locking for
mutexes then we are better off doing it with a proper set of functions and
not just a single undebugable wrapper.

Thanks,

	tglx

 

[toc] | [prev] | [next] | [standalone]

#1497775 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Sat, Oct 08, 2016 at 04:11:25PM +0200, Thomas Gleixner wrote:

> Well, when you add just trylock_recursive then people are going to use it
> anyway no matter whether it is easy or not.
> 
> So if we decide to provide something which supports recursive locking for
> mutexes then we are better off doing it with a proper set of functions and
> not just a single undebugable wrapper.

So ideally I'd say, no recursive stuff at all. But that means the GEM
people need to either do custom hacks (and you know that will spread),
or need to be convinced to rework their locking somehow.

[toc] | [prev] | [next] | [standalone]

#1497757 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Sat, Oct 08, 2016 at 01:58:07PM +0200, Thomas Gleixner wrote:
> Hmm. I'm not a great fan of this, because that requires an conditional
> unlock mechanism.
> 
>        res = trylock_recursive(lock);
>        if (res == FAILED)
>        	       goto out;
>        .....
> 
>        if (res == SUCCESS)
>        	       unlock(lock);
> 
> While if you actually keep track of recursion you can do:
>   
>       if (!trylock_recursive(lock))
> 		goto out;
> 
>       ....
> 
>       unlock_recursive(lock);
> 
> or even:
> 
>      lock_recursive(lock);
> 
>      unlock_recursive(lock);
> 
> That's making lock/trylock and unlock symetric, so its obvious in the
> source what's going on and the recursion tracking allows for better
> debugability.

Hurm,. so I thought that in general we disliked recursive locking
because it quickly turns in to a horrible mess.

Adding such primitives makes it 'easy' to use recursive locking and then
where does it stop?

[toc] | [prev] | [next] | [standalone]

#1502960 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Fri, Oct 07, 2016 at 04:52:44PM +0200, Peter Zijlstra wrote:
> Poking at lock internals is not cool. Since I'm going to change the
> implementation this will break, take it out.
> 
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Cc: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Rob Clark <robdclark@gmail.com>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
>  drivers/gpu/drm/i915/i915_gem_shrinker.c |   26 +++-----------------------
>  drivers/gpu/drm/msm/msm_gem_shrinker.c   |   23 +++--------------------
>  2 files changed, 6 insertions(+), 43 deletions(-)

OK, so it appears that i915 changed their locking around and got rid of
this thing entirely. Much appreciated Chris!!

Rob, is there any chance you can do the same for msm?

Not having to provide a replacement function and opening the door to
recursive locking would be ever so good.

[toc] | [prev] | [next] | [standalone]

#1502964 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Tue, Oct 18, 2016 at 02:48:41PM +0200, Peter Zijlstra wrote:
> On Fri, Oct 07, 2016 at 04:52:44PM +0200, Peter Zijlstra wrote:
> > Poking at lock internals is not cool. Since I'm going to change the
> > implementation this will break, take it out.
> > 
> > Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> > Cc: Chris Wilson <chris@chris-wilson.co.uk>
> > Cc: Rob Clark <robdclark@gmail.com>
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > ---
> >  drivers/gpu/drm/i915/i915_gem_shrinker.c |   26 +++-----------------------
> >  drivers/gpu/drm/msm/msm_gem_shrinker.c   |   23 +++--------------------
> >  2 files changed, 6 insertions(+), 43 deletions(-)
> 
> OK, so it appears that i915 changed their locking around and got rid of
> this thing entirely. Much appreciated Chris!!

Hmm, I might have spoken too soon. My patch conflicted and I seem to
have read too much in the Changelog of 3b4e896f14b1 ("drm/i915: Remove
unused no-shrinker-steal").

[toc] | [prev] | [next] | [standalone]

#1502967 — Re: [PATCH -v4 1/8] locking/drm: Kill mutex trickery

On Tue, Oct 18, 2016 at 02:48:41PM +0200, Peter Zijlstra wrote:
> On Fri, Oct 07, 2016 at 04:52:44PM +0200, Peter Zijlstra wrote:
> > Poking at lock internals is not cool. Since I'm going to change the
> > implementation this will break, take it out.
> > 
> > Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> > Cc: Chris Wilson <chris@chris-wilson.co.uk>
> > Cc: Rob Clark <robdclark@gmail.com>
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > ---
> >  drivers/gpu/drm/i915/i915_gem_shrinker.c |   26 +++-----------------------
> >  drivers/gpu/drm/msm/msm_gem_shrinker.c   |   23 +++--------------------
> >  2 files changed, 6 insertions(+), 43 deletions(-)
> 
> OK, so it appears that i915 changed their locking around and got rid of
> this thing entirely. Much appreciated Chris!!

It's not dead yet! Sorry.

It's close though, in the next cycle we may be at a point where we don't
rely on recursion locking in the shrinker.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | linux.kernel

csiph-web