Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1443672
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob |
| Date | 2016-07-14 20:40 +0200 |
| Message-ID | <rUTIe-4Mw-17@gated-at.bofh.it> (permalink) |
| References | <rUTIe-4Mw-13@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
The current percpu-rwsem read side is entirely free of serializing
instructions at the cost of having a synchronize_sched() in the write
path.
The latency of the synchronize_sched() is too high for some users
(cgroups), so provide a __percpu_init_rwsem(.bias) argument to forgot
this synchronize_sched() at the cost of forcing all readers into the
slow path, which has serializing instructions.
Cc: Tejun Heo <tj@kernel.org>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Paul McKenney <paulmck@linux.vnet.ibm.com>
Reported-by: John Stultz <john.stultz@linaro.org>
Reported-by: Dmitry Shmidt <dimitrysh@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
fs/super.c | 3 ++-
include/linux/percpu-rwsem.h | 15 +++++++++++++--
kernel/cgroup.c | 2 +-
kernel/locking/percpu-rwsem.c | 10 +++++++++-
4 files changed, 25 insertions(+), 5 deletions(-)
--- a/fs/super.c
+++ b/fs/super.c
@@ -195,7 +195,8 @@ static struct super_block *alloc_super(s
for (i = 0; i < SB_FREEZE_LEVELS; i++) {
if (__percpu_init_rwsem(&s->s_writers.rw_sem[i],
sb_writers_name[i],
- &type->s_writers_key[i]))
+ &type->s_writers_key[i],
+ PERCPU_RWSEM_READER))
goto fail;
}
init_waitqueue_head(&s->s_writers.wait_unfrozen);
--- a/include/linux/percpu-rwsem.h
+++ b/include/linux/percpu-rwsem.h
@@ -90,15 +90,26 @@ static inline void percpu_up_read(struct
extern void percpu_down_write(struct percpu_rw_semaphore *);
extern void percpu_up_write(struct percpu_rw_semaphore *);
+enum percpu_rwsem_bias { PERCPU_RWSEM_READER, PERCPU_RWSEM_WRITER };
+
extern int __percpu_init_rwsem(struct percpu_rw_semaphore *,
- const char *, struct lock_class_key *);
+ const char *, struct lock_class_key *,
+ enum percpu_rwsem_bias bias);
extern void percpu_free_rwsem(struct percpu_rw_semaphore *);
#define percpu_init_rwsem(sem) \
({ \
static struct lock_class_key rwsem_key; \
- __percpu_init_rwsem(sem, #sem, &rwsem_key); \
+ __percpu_init_rwsem(sem, #sem, &rwsem_key, \
+ PERCPU_RWSEM_READER); \
+})
+
+#define percpu_init_rwsem_writer(sem) \
+({ \
+ static struct lock_class_key rwsem_key; \
+ __percpu_init_rwsem(sem, #sem, &rwsem_key, \
+ PERCPU_RWSEM_WRITER); \
})
#define percpu_rwsem_is_held(sem) lockdep_is_held(&(sem)->rw_sem)
--- a/kernel/cgroup.c
+++ b/kernel/cgroup.c
@@ -5605,7 +5605,7 @@ int __init cgroup_init(void)
int ssid;
BUILD_BUG_ON(CGROUP_SUBSYS_COUNT > 16);
- BUG_ON(percpu_init_rwsem(&cgroup_threadgroup_rwsem));
+ BUG_ON(percpu_init_rwsem_writer(&cgroup_threadgroup_rwsem));
BUG_ON(cgroup_init_cftypes(NULL, cgroup_dfl_base_files));
BUG_ON(cgroup_init_cftypes(NULL, cgroup_legacy_base_files));
--- a/kernel/locking/percpu-rwsem.c
+++ b/kernel/locking/percpu-rwsem.c
@@ -11,7 +11,8 @@
enum { readers_slow, readers_block };
int __percpu_init_rwsem(struct percpu_rw_semaphore *sem,
- const char *name, struct lock_class_key *rwsem_key)
+ const char *name, struct lock_class_key *rwsem_key,
+ enum percpu_rwsem_bias bias)
{
sem->read_count = alloc_percpu(int);
if (unlikely(!sem->read_count))
@@ -19,6 +20,13 @@ int __percpu_init_rwsem(struct percpu_rw
/* ->rw_sem represents the whole percpu_rw_semaphore for lockdep */
rcu_sync_init(&sem->rss, RCU_SCHED_SYNC);
+ if (bias == PERCPU_RWSEM_WRITER) {
+ /*
+ * Disable rcu_sync() and force slow path.
+ */
+ sem->rss.gp_count++;
+ sem->rss.gp_state = !0;
+ }
__init_rwsem(&sem->rw_sem, name, rwsem_key);
init_waitqueue_head(&sem->writer);
sem->state = readers_slow;
Back to linux.kernel | Previous | Next — Next in thread | Find similar | Unroll thread
[PATCH 2/2] locking/percpu-rwsem: Introduce bias knob Peter Zijlstra <peterz@infradead.org> - 2016-07-14 20:40 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob Oleg Nesterov <oleg@redhat.com> - 2016-07-14 20:50 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob Peter Zijlstra <peterz@infradead.org> - 2016-07-14 21:00 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob Peter Zijlstra <peterz@infradead.org> - 2016-07-14 21:30 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-14 21:30 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob Peter Zijlstra <peterz@infradead.org> - 2016-07-14 21:40 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-14 22:00 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob Oleg Nesterov <oleg@redhat.com> - 2016-07-15 15:30 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-15 15:40 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob Oleg Nesterov <oleg@redhat.com> - 2016-07-15 15:50 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-15 17:40 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob Oleg Nesterov <oleg@redhat.com> - 2016-07-15 18:50 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-15 20:10 +0200
[PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Oleg Nesterov <oleg@redhat.com> - 2016-07-16 19:20 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Oleg Nesterov <oleg@redhat.com> - 2016-07-16 20:50 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Peter Zijlstra <peterz@infradead.org> - 2016-07-18 14:00 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Oleg Nesterov <oleg@redhat.com> - 2016-07-18 15:50 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-19 23:00 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Oleg Nesterov <oleg@redhat.com> - 2016-07-20 17:20 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-20 23:00 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Oleg Nesterov <oleg@redhat.com> - 2016-07-21 19:40 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Oleg Nesterov <oleg@redhat.com> - 2016-07-20 19:20 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-20 23:40 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Oleg Nesterov <oleg@redhat.com> - 2016-07-21 19:40 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-22 05:30 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() John Stultz <john.stultz@linaro.org> - 2016-07-25 19:10 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Oleg Nesterov <oleg@redhat.com> - 2016-07-25 19:30 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() Oleg Nesterov <oleg@redhat.com> - 2016-07-25 19:10 +0200
Re: [PATCH] rcu_sync: simplify the state machine, introduce __rcu_sync_enter() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-07-25 19:50 +0200
Re: [PATCH 2/2] locking/percpu-rwsem: Introduce bias knob Oleg Nesterov <oleg@redhat.com> - 2016-07-15 15:50 +0200
csiph-web