Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1548158

[PATCH v4 15/20] TP-futex: Enable kernel reader lock stealing

Show all headers | View raw

The TP-futexes prefer writers as the code path is much simpler for
them. This may not be good for overall system performance especially
if there is a fair amount of reader activities.

This patch enables kernel reader to steal the lock when the futex is
currently reader-owned and the lock handoff mechanism hasn't been
enabled yet. A new field locksteal_disabled is added to the futex
state object for controlling reader lock stealing. So the waiting
reader must retrieve the futex state object first before doing it.

Signed-off-by: Waiman Long <longman@redhat.com>
---
 kernel/futex.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/kernel/futex.c b/kernel/futex.c
index ee35264..b690a79 100644
--- a/kernel/futex.c
+++ b/kernel/futex.c
@@ -237,6 +237,7 @@ struct futex_state {
 	atomic_t refcount;
 
 	u32 handoff_pid;			/* For TP futexes only */
+	int locksteal_disabled;			/* For TP futexes only */
 
 	enum futex_type type;
 	union futex_key key;
@@ -3436,6 +3437,7 @@ void exit_robust_list(struct task_struct *curr)
 	state = alloc_futex_state();
 	state->type = TYPE_TP;
 	state->key = *key;
+	state->locksteal_disabled = false;
 	list_add(&state->fs_list, &hb->fs_head);
 	WARN_ON(atomic_read(&state->refcount) != 1);
 
@@ -3781,9 +3783,10 @@ static int futex_spin_on_owner(u32 __user *uaddr, const u32 vpid,
 				WRITE_ONCE(state->handoff_pid, vpid);
 
 				/*
-				 * Disable handoff check.
+				 * Disable handoff check and reader lock
+				 * stealing.
 				 */
-				handoff_set = true;
+				state->locksteal_disabled = handoff_set = true;
 			}
 		}
 
@@ -3886,6 +3889,7 @@ static int futex_spin_on_owner(u32 __user *uaddr, const u32 vpid,
 	 */
 	WRITE_ONCE(state->mutex_owner, NULL);
 	WRITE_ONCE(state->handoff_pid, 0);
+	state->locksteal_disabled = false;
 
 	preempt_enable();
 	return (ret < 0) ? ret : TP_STATUS_SLEEP(ret, nsleep);
@@ -3966,6 +3970,21 @@ static noinline int futex_lock(u32 __user *uaddr, unsigned int flags,
 		goto out_put_state_key;
 	}
 
+	/*
+	 * For reader, we will try to steal the lock here as if it is the
+	 * top waiter without taking the serialization mutex if reader
+	 * lock stealing isn't disabled even though the FUTEX_WAITERS bit
+	 * is set.
+	 */
+	if (shared && !state->locksteal_disabled) {
+		ret = futex_trylock(uaddr, vpid, &uval, true);
+		if (ret) {
+			if (ret > 0)
+				ret = TP_LOCK_STOLEN;
+			goto out_put_state_key;
+		}
+	}
+
 	if (to)
 		hrtimer_start_expires(&to->timer, HRTIMER_MODE_ABS);
 
-- 
1.8.3.1

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH v4 00/20] futex: Introducing throughput-optimized (TP) futexes Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
  [PATCH v4 07/20] futex: Introduce throughput-optimized (TP) futexes Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
    Re: [PATCH v4 07/20] futex: Introduce throughput-optimized (TP)  futexes kbuild test robot <lkp@intel.com> - 2016-12-29 22:20 +0100
  [PATCH v4 08/20] TP-futex: Enable robust handling Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
  [PATCH v4 06/20] futex: Allow direct attachment of futex_state objects to hash bucket Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
  [PATCH v4 20/20] futex: Dump internal futex state via debugfs Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
    Re: [PATCH v4 20/20] futex: Dump internal futex state via debugfs kbuild test robot <lkp@intel.com> - 2016-12-29 20:00 +0100
  [PATCH v4 16/20] TP-futex: Group readers together in wait queue Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
    Re: [PATCH v4 16/20] TP-futex: Group readers together in wait queue kbuild test robot <lkp@intel.com> - 2016-12-29 21:00 +0100
    Re: [PATCH v4 16/20] TP-futex: Group readers together in wait queue kbuild test robot <lkp@intel.com> - 2016-12-29 21:20 +0100
  [PATCH v4 15/20] TP-futex: Enable kernel reader lock stealing Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
  [PATCH v4 01/20] futex: Consolidate duplicated timer setup code Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
  [PATCH v4 11/20] TP-futex: Add timeout support Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
  [PATCH v4 04/20] futex: Consolidate pure pi_state_list add & delete codes to helpers Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
  [PATCH v4 14/20] TP-futex: Support userspace reader/writer locks Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
    Re: [PATCH v4 14/20] TP-futex: Support userspace reader/writer locks kbuild test robot <lkp@intel.com> - 2016-12-29 19:40 +0100
  [PATCH v4 09/20] TP-futex: Implement lock handoff to prevent lock starvation Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100
  [PATCH v4 18/20] perf bench: New microbenchmark for userspace rwlock performance Waiman Long <longman@redhat.com> - 2016-12-29 17:20 +0100

csiph-web