Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1211275

[PATCH v2 6/8] percpu-rwsem: change it to rely on rss_sync infrastructure

From Oleg Nesterov <oleg@redhat.com>
Newsgroups linux.kernel
Subject [PATCH v2 6/8] percpu-rwsem: change it to rely on rss_sync infrastructure
Date 2015-08-21 19:50 +0200
Message-ID <pZYBY-2dA-15@gated-at.bofh.it> (permalink)
References <pZYBY-2dA-11@gated-at.bofh.it>
Organization linux.* mail to news gateway

Show all headers | View raw

Currently down_write/up_write calls synchronize_sched_expedited()
twice which is evil. Change this code to rely on rcu-sync primitives.
This avoids the _expedited "big hammer", and this can be faster in
the contended case or even in the case when a single thread does
down_write/up_write in a loop.

Of course, a single down_write() will take more time, but otoh it
will be much more friendly to the whole system.

To simplify the review this patch doesn't update the comments, fixed
by the next change.

Signed-off-by: Oleg Nesterov <oleg@redhat.com>
---
 include/linux/percpu-rwsem.h  |    3 ++-
 kernel/locking/percpu-rwsem.c |   18 +++++++-----------
 2 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/include/linux/percpu-rwsem.h b/include/linux/percpu-rwsem.h
index 834c4e5..06af654 100644
--- a/include/linux/percpu-rwsem.h
+++ b/include/linux/percpu-rwsem.h
@@ -5,11 +5,12 @@
 #include <linux/rwsem.h>
 #include <linux/percpu.h>
 #include <linux/wait.h>
+#include <linux/rcusync.h>
 #include <linux/lockdep.h>
 
 struct percpu_rw_semaphore {
+	struct rcu_sync_struct	rss;
 	unsigned int __percpu	*fast_read_ctr;
-	atomic_t		write_ctr;
 	struct rw_semaphore	rw_sem;
 	atomic_t		slow_read_ctr;
 	wait_queue_head_t	write_waitq;
diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c
index 3bb2dc9..d3103da 100644
--- a/kernel/locking/percpu-rwsem.c
+++ b/kernel/locking/percpu-rwsem.c
@@ -17,7 +17,7 @@ int __percpu_init_rwsem(struct percpu_rw_semaphore *brw,
 
 	/* ->rw_sem represents the whole percpu_rw_semaphore for lockdep */
 	__init_rwsem(&brw->rw_sem, name, rwsem_key);
-	atomic_set(&brw->write_ctr, 0);
+	rcu_sync_init(&brw->rss, RCU_SCHED_SYNC);
 	atomic_set(&brw->slow_read_ctr, 0);
 	init_waitqueue_head(&brw->write_waitq);
 	return 0;
@@ -32,6 +32,7 @@ void percpu_free_rwsem(struct percpu_rw_semaphore *brw)
 	if (!brw->fast_read_ctr)
 		return;
 
+	rcu_sync_dtor(&brw->rss);
 	free_percpu(brw->fast_read_ctr);
 	brw->fast_read_ctr = NULL; /* catch use after free bugs */
 }
@@ -61,13 +62,12 @@ void percpu_free_rwsem(struct percpu_rw_semaphore *brw)
  */
 static bool update_fast_ctr(struct percpu_rw_semaphore *brw, unsigned int val)
 {
-	bool success = false;
+	bool success;
 
 	preempt_disable();
-	if (likely(!atomic_read(&brw->write_ctr))) {
+	success = rcu_sync_is_idle(&brw->rss);
+	if (likely(success))
 		__this_cpu_add(*brw->fast_read_ctr, val);
-		success = true;
-	}
 	preempt_enable();
 
 	return success;
@@ -146,8 +146,6 @@ static int clear_fast_ctr(struct percpu_rw_semaphore *brw)
  */
 void percpu_down_write(struct percpu_rw_semaphore *brw)
 {
-	/* tell update_fast_ctr() there is a pending writer */
-	atomic_inc(&brw->write_ctr);
 	/*
 	 * 1. Ensures that write_ctr != 0 is visible to any down_read/up_read
 	 *    so that update_fast_ctr() can't succeed.
@@ -159,7 +157,7 @@ void percpu_down_write(struct percpu_rw_semaphore *brw)
 	 *    fast-path, it executes a full memory barrier before we return.
 	 *    See R_W case in the comment above update_fast_ctr().
 	 */
-	synchronize_sched_expedited();
+	rcu_sync_enter(&brw->rss);
 
 	/* exclude other writers, and block the new readers completely */
 	down_write(&brw->rw_sem);
@@ -179,7 +177,5 @@ void percpu_up_write(struct percpu_rw_semaphore *brw)
 	 * Insert the barrier before the next fast-path in down_read,
 	 * see W_R case in the comment above update_fast_ctr().
 	 */
-	synchronize_sched_expedited();
-	/* the last writer unblocks update_fast_ctr() */
-	atomic_dec(&brw->write_ctr);
+	rcu_sync_exit(&brw->rss);
 }
-- 
1.5.5.1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH v2 0/8] Add rcu_sync infrastructure to avoid _expedited()  in percpu-rwsem Oleg Nesterov <oleg@redhat.com> - 2015-08-21 19:50 +0200
  [PATCH v2 3/8] rcusync: Add the CONFIG_PROVE_RCU checks Oleg Nesterov <oleg@redhat.com> - 2015-08-21 19:50 +0200
  [PATCH v2 6/8] percpu-rwsem: change it to rely on rss_sync  infrastructure Oleg Nesterov <oleg@redhat.com> - 2015-08-21 19:50 +0200
  [PATCH v2 5/8] percpu-rwsem: make percpu_free_rwsem() after  kzalloc() safe Oleg Nesterov <oleg@redhat.com> - 2015-08-21 19:50 +0200
  [PATCH v2 8/8] percpu-rwsem: cleanup the lockdep annotations in  percpu_down_read() Oleg Nesterov <oleg@redhat.com> - 2015-08-21 19:50 +0200
  [PATCH v2 4/8] rcusync: Introduce rcu_sync_dtor() Oleg Nesterov <oleg@redhat.com> - 2015-08-21 19:50 +0200
  [PATCH v2 7/8] percpu-rwsem: fix the comments outdated by rcu_sync Oleg Nesterov <oleg@redhat.com> - 2015-08-21 19:50 +0200
  [PATCH v2 2/8] rcusync: Introduce struct rcu_sync_ops Oleg Nesterov <oleg@redhat.com> - 2015-08-21 19:50 +0200
  [PATCH v2 1/8] rcu: Create rcu_sync infrastructure Oleg Nesterov <oleg@redhat.com> - 2015-08-21 19:50 +0200
  Re: [PATCH v2 0/8] Add rcu_sync infrastructure to avoid _expedited()  in percpu-rwsem "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-08-22 18:40 +0200
    Re: [PATCH v2 0/8] Add rcu_sync infrastructure to avoid  _expedited() in percpu-rwsem Oleg Nesterov <oleg@redhat.com> - 2015-08-24 17:40 +0200
      parse_args() is too unforgivable? Oleg Nesterov <oleg@redhat.com> - 2015-08-24 20:40 +0200
        Re: parse_args() is too unforgivable? Rusty Russell <rusty@rustcorp.com.au> - 2015-08-25 03:40 +0200
          [PATCH 0/1] params: don't ignore the rest of cmdline if  parse_one() fails Oleg Nesterov <oleg@redhat.com> - 2015-08-25 17:30 +0200
            [PATCH 1/1] params: don't ignore the rest of cmdline if  parse_one() fails Oleg Nesterov <oleg@redhat.com> - 2015-08-25 17:30 +0200
              Re: [PATCH 1/1] params: don't ignore the rest of cmdline if parse_one() fails Rusty Russell <rusty@rustcorp.com.au> - 2015-08-26 03:20 +0200
      Re: [PATCH v2 0/8] Add rcu_sync infrastructure to avoid  _expedited() in percpu-rwsem "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-08-26 02:30 +0200
        Re: [PATCH v2 0/8] Add rcu_sync infrastructure to avoid  _expedited() in percpu-rwsem Oleg Nesterov <oleg@redhat.com> - 2015-08-26 14:20 +0200
          Re: [PATCH v2 0/8] Add rcu_sync infrastructure to avoid  _expedited() in percpu-rwsem Oleg Nesterov <oleg@redhat.com> - 2015-08-26 15:00 +0200
            Re: [PATCH v2 0/8] Add rcu_sync infrastructure to avoid  _expedited() in percpu-rwsem "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-08-26 16:40 +0200

csiph-web