Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1260658
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCH 4/4] locking: Introduce smp_cond_acquire() |
| Date | 2015-11-02 15:00 +0100 |
| Message-ID | <qqnOq-5Zc-9@gated-at.bofh.it> (permalink) |
| References | <qqnOq-5Zc-11@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
Introduce smp_cond_acquire() which combines a control dependency and a
read barrier to form acquire semantics.
This primitive has two benefits:
- it documents control dependencies,
- its typically cheaper than using smp_load_acquire() in a loop.
Note that while smp_cond_acquire() has an explicit
smp_read_barrier_depends() for Alpha, neither sites it gets used in
were actually buggy on Alpha for their lack of it. The first uses
smp_rmb(), which on Alpha is a full barrier too and therefore serves
its purpose. The second had an explicit full barrier.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
include/linux/compiler.h | 18 ++++++++++++++++++
kernel/sched/core.c | 8 +-------
kernel/task_work.c | 4 ++--
3 files changed, 21 insertions(+), 9 deletions(-)
--- a/include/linux/compiler.h
+++ b/include/linux/compiler.h
@@ -275,6 +275,24 @@ static __always_inline void __write_once
__val; \
})
+/**
+ * smp_cond_acquire() - Spin wait for cond with ACQUIRE ordering
+ * @cond: boolean expression to wait for
+ *
+ * Equivalent to using smp_load_acquire() on the condition variable but employs
+ * the control dependency of the wait to reduce the barrier on many platforms.
+ *
+ * The control dependency provides a LOAD->STORE order, the additional RMB
+ * provides LOAD->LOAD order, together they provide LOAD->{LOAD,STORE} order,
+ * aka. ACQUIRE.
+ */
+#define smp_cond_acquire(cond) do { \
+ while (!(cond)) \
+ cpu_relax(); \
+ smp_read_barrier_depends(); /* ctrl */ \
+ smp_rmb(); /* ctrl + rmb := acquire */ \
+} while (0)
+
#endif /* __KERNEL__ */
#endif /* __ASSEMBLY__ */
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2111,19 +2111,13 @@ try_to_wake_up(struct task_struct *p, un
/*
* If the owning (remote) cpu is still in the middle of schedule() with
* this task as prev, wait until its done referencing the task.
- */
- while (p->on_cpu)
- cpu_relax();
- /*
- * Combined with the control dependency above, we have an effective
- * smp_load_acquire() without the need for full barriers.
*
* Pairs with the smp_store_release() in finish_lock_switch().
*
* This ensures that tasks getting woken will be fully ordered against
* their previous state and preserve Program Order.
*/
- smp_rmb();
+ smp_cond_acquire(!p->on_cpu);
p->sched_contributes_to_load = !!task_contributes_to_load(p);
p->state = TASK_WAKING;
--- a/kernel/task_work.c
+++ b/kernel/task_work.c
@@ -102,13 +102,13 @@ void task_work_run(void)
if (!work)
break;
+
/*
* Synchronize with task_work_cancel(). It can't remove
* the first entry == work, cmpxchg(task_works) should
* fail, but it can play with *work and other entries.
*/
- raw_spin_unlock_wait(&task->pi_lock);
- smp_mb();
+ smp_cond_acquire(!raw_spin_is_locked(&task->pi_lock));
do {
next = work->next;
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCH 0/4] scheduler ordering bits Peter Zijlstra <peterz@infradead.org> - 2015-11-02 15:00 +0100
[PATCH 2/4] sched: Document Program-Order guarantees Peter Zijlstra <peterz@infradead.org> - 2015-11-02 15:00 +0100
Re: [PATCH 2/4] sched: Document Program-Order guarantees Paul Turner <pjt@google.com> - 2015-11-02 21:30 +0100
Re: [PATCH 2/4] sched: Document Program-Order guarantees Peter Zijlstra <peterz@infradead.org> - 2015-11-02 21:40 +0100
Re: [PATCH 2/4] sched: Document Program-Order guarantees Paul Turner <pjt@google.com> - 2015-11-02 23:10 +0100
Re: [PATCH 2/4] sched: Document Program-Order guarantees Peter Zijlstra <peterz@infradead.org> - 2015-11-02 23:20 +0100
[PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-02 15:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-02 15:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Will Deacon <will.deacon@arm.com> - 2015-11-02 18:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-03 02:20 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Linus Torvalds <torvalds@linux-foundation.org> - 2015-11-03 02:30 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Will Deacon <will.deacon@arm.com> - 2015-11-02 18:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Linus Torvalds <torvalds@linux-foundation.org> - 2015-11-02 19:10 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Will Deacon <will.deacon@arm.com> - 2015-11-02 19:40 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Linus Torvalds <torvalds@linux-foundation.org> - 2015-11-02 20:20 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Will Deacon <will.deacon@arm.com> - 2015-11-02 21:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-02 21:30 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-02 23:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-03 03:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Linus Torvalds <torvalds@linux-foundation.org> - 2015-11-03 20:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-04 05:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Linus Torvalds <torvalds@linux-foundation.org> - 2015-11-04 05:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-04 14:10 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() David Howells <dhowells@redhat.com> - 2015-11-02 21:40 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-02 21:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Linus Torvalds <torvalds@linux-foundation.org> - 2015-11-02 22:20 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Oleg Nesterov <oleg@redhat.com> - 2015-11-03 18:10 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-03 19:30 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Boqun Feng <boqun.feng@gmail.com> - 2015-11-11 10:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Boqun Feng <boqun.feng@gmail.com> - 2015-11-11 11:40 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Oleg Nesterov <oleg@redhat.com> - 2015-11-11 20:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-12 15:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-11 13:20 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Oleg Nesterov <oleg@redhat.com> - 2015-11-11 19:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Linus Torvalds <torvalds@linux-foundation.org> - 2015-11-11 22:30 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Boqun Feng <boqun.feng@gmail.com> - 2015-11-12 08:20 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-12 11:30 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Oleg Nesterov <oleg@redhat.com> - 2015-11-12 15:10 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Boqun Feng <boqun.feng@gmail.com> - 2015-11-12 15:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-12 16:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Will Deacon <will.deacon@arm.com> - 2015-11-12 23:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-12 15:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-12 16:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-12 16:10 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-12 16:10 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-12 16:30 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Will Deacon <will.deacon@arm.com> - 2015-11-12 22:30 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Boqun Feng <boqun.feng@gmail.com> - 2015-11-12 16:20 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Oleg Nesterov <oleg@redhat.com> - 2015-11-12 18:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra <peterz@infradead.org> - 2015-11-12 19:10 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Oleg Nesterov <oleg@redhat.com> - 2015-11-12 19:40 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-12 20:00 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Will Deacon <will.deacon@arm.com> - 2015-11-12 22:40 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-11-13 00:50 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Linus Torvalds <torvalds@linux-foundation.org> - 2015-11-12 19:30 +0100
Re: [PATCH 4/4] locking: Introduce smp_cond_acquire() Will Deacon <will.deacon@arm.com> - 2015-11-12 23:10 +0100
[PATCH 1/4] sched: Better document the try_to_wake_up() barriers Peter Zijlstra <peterz@infradead.org> - 2015-11-02 15:00 +0100
[PATCH 3/4] sched: Fix a race in try_to_wake_up() vs schedule() Peter Zijlstra <peterz@infradead.org> - 2015-11-02 15:00 +0100
csiph-web