Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1586361 > unrolled thread

[PATCH-tip 3/3] locking/rwsem: Stop active read lock ASAP

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH-tip 3/3] locking/rwsem: Stop active read lock ASAP Waiman Long <longman@redhat.com> - 2017-02-22 19:10 +0100
    Re: [PATCH-tip 3/3] locking/rwsem: Stop active read lock ASAP Davidlohr Bueso <dave@stgolabs.net> - 2017-02-26 20:00 +0100
      Re: [PATCH-tip 3/3] locking/rwsem: Stop active read lock ASAP Waiman Long <longman@redhat.com> - 2017-02-27 16:10 +0100

#1586361 — [PATCH-tip 3/3] locking/rwsem: Stop active read lock ASAP

Currently, when down_read() fails, the active read locking isn't undone
until the rwsem_down_read_failed() function grabs the wait_lock. If the
wait_lock is contended, it may takes a while to get the lock. During
that period, writer lock stealing will be disabled because of the
active read lock.

This patch will release the active read lock ASAP when either the
optimisitic spinners are present or the trylock fails so that writer
lock stealing can happen sooner.

On a 2-socket 36-core 72-thread x86-64 E5-2699 v3 system, a rwsem
microbenchmark was run with 36 locking threads (one/core) doing 100k
reader and writer lock/unlock operations each, the resulting locking
rates (avg of 3 runs) on a 4.10 kernel were 561.4 Mop/s and 588.8
Mop/s without and with the patch respectively. That was an increase
of about 5%.

Signed-off-by: Waiman Long <longman@redhat.com>
---
 kernel/locking/rwsem-xadd.c | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index fb6d6f3..6080afc 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -418,20 +418,40 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem)
 __visible
 struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem)
 {
-	long count, adjustment = -RWSEM_ACTIVE_READ_BIAS;
+	bool first_in_queue = false;
+	long count, adjustment;
 	struct rwsem_waiter waiter;
 	DEFINE_WAKE_Q(wake_q);
 
 	waiter.task = current;
 	waiter.type = RWSEM_WAITING_FOR_READ;
 
-	raw_spin_lock_irq(&sem->wait_lock);
-	if (list_empty(&sem->wait_list))
+	/*
+	 * Undo read bias from down_read operation to stop active locking
+	 * if the lock isn't free which means it may take a while to acquire
+	 * the lock or when spinners are present. Doing that after taking the
+	 * wait_lock may block writer lock stealing for too long impacting
+	 * performance.
+	 */
+	if (rwsem_has_spinner(sem) || !raw_spin_trylock_irq(&sem->wait_lock)) {
+		atomic_long_add(-RWSEM_ACTIVE_READ_BIAS, &sem->count);
+		adjustment = 0;
+		raw_spin_lock_irq(&sem->wait_lock);
+	} else {
+		adjustment = -RWSEM_ACTIVE_READ_BIAS;
+	}
+
+	if (list_empty(&sem->wait_list)) {
 		adjustment += RWSEM_WAITING_BIAS;
+		first_in_queue = true;
+	}
 	list_add_tail(&waiter.list, &sem->wait_list);
 
-	/* we're now waiting on the lock, but no longer actively locking */
-	count = atomic_long_add_return(adjustment, &sem->count);
+	/* we're now waiting on the lock */
+	if (adjustment)
+		count = atomic_long_add_return(adjustment, &sem->count);
+	else
+		count = atomic_long_read(&sem->count);
 
 	/*
 	 * If there are no active locks, wake the front queued process(es).
@@ -440,8 +460,7 @@ struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem)
 	 * wake our own waiter to join the existing active readers !
 	 */
 	if (count == RWSEM_WAITING_BIAS ||
-	    (count > RWSEM_WAITING_BIAS &&
-	     adjustment != -RWSEM_ACTIVE_READ_BIAS))
+	    (count > RWSEM_WAITING_BIAS && first_in_queue))
 		__rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
 
 	raw_spin_unlock_irq(&sem->wait_lock);
-- 
1.8.3.1

[toc] | [next] | [standalone]

#1588378

On Wed, 22 Feb 2017, Waiman Long wrote:

>On a 2-socket 36-core 72-thread x86-64 E5-2699 v3 system, a rwsem
>microbenchmark was run with 36 locking threads (one/core) doing 100k
>reader and writer lock/unlock operations each, the resulting locking
>rates (avg of 3 runs) on a 4.10 kernel were 561.4 Mop/s and 588.8
>Mop/s without and with the patch respectively. That was an increase
>of about 5%.

iirc this patch is a repost, no? If so, did you get a chance to measure
single file access with direct io as dchinner suggested?

Thanks,
Davidlohr

[toc] | [prev] | [next] | [standalone]

#1588718

On 02/26/2017 01:58 PM, Davidlohr Bueso wrote:
> On Wed, 22 Feb 2017, Waiman Long wrote:
>
>> On a 2-socket 36-core 72-thread x86-64 E5-2699 v3 system, a rwsem
>> microbenchmark was run with 36 locking threads (one/core) doing 100k
>> reader and writer lock/unlock operations each, the resulting locking
>> rates (avg of 3 runs) on a 4.10 kernel were 561.4 Mop/s and 588.8
>> Mop/s without and with the patch respectively. That was an increase
>> of about 5%.
>
> iirc this patch is a repost, no? If so, did you get a chance to measure
> single file access with direct io as dchinner suggested?
>
> Thanks,
> Davidlohr

Yes, this patch is a derivative of part of the rwsem reader spinning
patch set that I posted before. The major change here in this patch is
from the unconditional spitting into 2 atomic adds to a conditional
spitting depending the on OSQ state and the ability to acquire the
wait_lock immediately without waiting. This should reduce the concern
that you have with the original patch. I will run a single-file direct
I/O test as well. 

Cheers,
Longman

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web