Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1593610 > unrolled thread

[PATCH-tip v2 0/3] locking/rwsem: Minor twists to improve rwsem performance

Back to article view | Back to linux.kernel

Contents

  [PATCH-tip v2 0/3] locking/rwsem: Minor twists to improve rwsem performance Waiman Long <longman@redhat.com> - 2017-03-06 20:10 +0100
    [PATCH-tip v2 1/3] locking/rwsem: Check wait_list without lock if spinner present Waiman Long <longman@redhat.com> - 2017-03-06 20:10 +0100

#1593610 — [PATCH-tip v2 0/3] locking/rwsem: Minor twists to improve rwsem performance

v1->v2:
 - Replace trylock with a more light-weight raw_spin_is_locked()
   call to reduce overhead.
 - Run fio test in addition to rwsem microbenchmark.

This patch set introduces minor changes to the rwsem code path to
provide minor performance improvement especially with short critical
sections.

Patch 1 checks wait_list without lock when spinners are present.

Patch 2 moves down the rwsem_down_read_failed() function after the
optimistic spinning section so that functions in that section can
be used.

Patch 3 undoes active read lock ASAP when either the spinners are
present or the wait_lock isn't free.

Waiman Long (3):
  locking/rwsem: Check wait_list without lock if spinner present
  locking/rwsem: relocate rwsem_down_read_failed()
  locking/rwsem: Stop active read lock ASAP

 kernel/locking/rwsem-xadd.c | 129 ++++++++++++++++++++++++++------------------
 1 file changed, 76 insertions(+), 53 deletions(-)

-- 
1.8.3.1

[toc] | [next] | [standalone]

#1593611 — [PATCH-tip v2 1/3] locking/rwsem: Check wait_list without lock if spinner present

In rwsem_wake(), we can safely check the wait_list to see if waiters
are present without lock when there are spinners to fall back on in
case we miss a waiter.  The advantage is that we can save a pair of
spin_lock/unlock calls when the wait_list is empty. This translates
to a reduction in latency and hence slightly better performance.

The raw_spin_trylock_irqsave() call is also being replaced by a
raw_spin_is_locked() to reduce the overhead of irqsave/irqrestore
especially when the trylock fails.

On a 2-socket 36-core x86-64 E5-2699 v3 system, a rwsem microbenchmark
was run with 36 locking threads doing 1 million writer lock/unlock
operations each, the resulting locking rates (avg of 3 runs) on a
4.11-rc1 based kernel were 4,923 Mop/s and 5,136 Mop/s without and
with the patch respectively. That was an increase of about 4%.

On the same system, a 36-thread fio direct IO random write test to
the same 2GB file on a XFS formatted ramdisk was run. The aggregated
bandwidth was 1564.4 MB/s and 1593.0 MB/s before and after the
patch. That was an increase of about 2%.

Signed-off-by: Waiman Long <longman@redhat.com>
---
 kernel/locking/rwsem-xadd.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 34e727f..f31dd61b 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -587,7 +587,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
 
 	/*
 	 * If a spinner is present, it is not necessary to do the wakeup.
-	 * Try to do wakeup only if the trylock succeeds to minimize
+	 * Try to do wakeup only if the wait_lock is free to minimize
 	 * spinlock contention which may introduce too much delay in the
 	 * unlock operation.
 	 *
@@ -595,7 +595,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
 	 *    ---------------		-----------------------
 	 * [S]   osq_unlock()		[L]   osq
 	 *	 MB			      RMB
-	 * [RmW] rwsem_try_write_lock() [RmW] spin_trylock(wait_lock)
+	 * [RmW] rwsem_try_write_lock() [RmW] spin_is_locked(wait_lock)
 	 *
 	 * Here, it is important to make sure that there won't be a missed
 	 * wakeup while the rwsem is free and the only spinning writer goes
@@ -611,12 +611,18 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
 		 * state is consulted before reading the wait_lock.
 		 */
 		smp_rmb();
-		if (!raw_spin_trylock_irqsave(&sem->wait_lock, flags))
+
+		/*
+		 * Normally checking wait_list without wait_lock isn't safe
+		 * as we may miss an incoming waiter. With spinners present,
+		 * however, we have someone to fall back on in case that
+		 * happens.
+		 */
+		if (list_empty(&sem->wait_list) ||
+		    raw_spin_is_locked(&sem->wait_lock))
 			return sem;
-		goto locked;
 	}
 	raw_spin_lock_irqsave(&sem->wait_lock, flags);
-locked:
 
 	if (!list_empty(&sem->wait_list))
 		__rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
-- 
1.8.3.1

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web