Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1259851
| From | Waiman Long <Waiman.Long@hpe.com> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCH tip/locking/core v9 2/6] locking/qspinlock: prefetch next node cacheline |
| Date | 2015-10-31 00:30 +0100 |
| Message-ID | <qprhn-3F0-11@gated-at.bofh.it> (permalink) |
| References | <qprhn-3F0-1@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
A queue head CPU, after acquiring the lock, will have to notify
the next CPU in the wait queue that it has became the new queue
head. This involves loading a new cacheline from the MCS node of the
next CPU. That operation can be expensive and add to the latency of
locking operation.
This patch addes code to optmistically prefetch the next MCS node
cacheline if the next pointer is defined and it has been spinning
for the MCS lock for a while. This reduces the locking latency and
improves the system throughput.
Using a locking microbenchmark on a Haswell-EX system, this patch
can improve throughput by about 5%.
Signed-off-by: Waiman Long <Waiman.Long@hpe.com>
---
kernel/locking/qspinlock.c | 21 +++++++++++++++++++++
1 files changed, 21 insertions(+), 0 deletions(-)
diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c
index 7868418..c1c8a1a 100644
--- a/kernel/locking/qspinlock.c
+++ b/kernel/locking/qspinlock.c
@@ -396,6 +396,7 @@ queue:
* p,*,* -> n,*,*
*/
old = xchg_tail(lock, tail);
+ next = NULL;
/*
* if there was a previous node; link it and wait until reaching the
@@ -407,6 +408,16 @@ queue:
pv_wait_node(node);
arch_mcs_spin_lock_contended(&node->locked);
+
+ /*
+ * While waiting for the MCS lock, the next pointer may have
+ * been set by another lock waiter. We optimistically load
+ * the next pointer & prefetch the cacheline for writing
+ * to reduce latency in the upcoming MCS unlock operation.
+ */
+ next = READ_ONCE(node->next);
+ if (next)
+ prefetchw(next);
}
/*
@@ -426,6 +437,15 @@ queue:
cpu_relax();
/*
+ * If the next pointer is defined, we are not tail anymore.
+ * In this case, claim the spinlock & release the MCS lock.
+ */
+ if (next) {
+ set_locked(lock);
+ goto mcs_unlock;
+ }
+
+ /*
* claim the lock:
*
* n,0,0 -> 0,0,1 : lock, uncontended
@@ -458,6 +478,7 @@ queue:
while (!(next = READ_ONCE(node->next)))
cpu_relax();
+mcs_unlock:
arch_mcs_spin_unlock_contended(&next->locked);
pv_kick_node(lock, next);
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCH tip/locking/core v9 0/6] locking/qspinlock: Enhance pvqspinlock Waiman Long <Waiman.Long@hpe.com> - 2015-10-31 00:30 +0100
[PATCH tip/locking/core v9 1/6] locking/qspinlock: Use _acquire/_release versions of cmpxchg & xchg Waiman Long <Waiman.Long@hpe.com> - 2015-10-31 00:30 +0100
[PATCH tip/locking/core v9 2/6] locking/qspinlock: prefetch next node cacheline Waiman Long <Waiman.Long@hpe.com> - 2015-10-31 00:30 +0100
Re: [PATCH tip/locking/core v9 2/6] locking/qspinlock: prefetch next node cacheline Peter Zijlstra <peterz@infradead.org> - 2015-11-02 17:40 +0100
Re: [PATCH tip/locking/core v9 2/6] locking/qspinlock: prefetch next node cacheline Peter Zijlstra <peterz@infradead.org> - 2015-11-03 00:00 +0100
Re: [PATCH tip/locking/core v9 2/6] locking/qspinlock: prefetch next node cacheline Waiman Long <waiman.long@hpe.com> - 2015-11-05 17:50 +0100
Re: [PATCH tip/locking/core v9 2/6] locking/qspinlock: prefetch next node cacheline Peter Zijlstra <peterz@infradead.org> - 2015-11-05 18:00 +0100
Re: [PATCH tip/locking/core v9 2/6] locking/qspinlock: prefetch next node cacheline Waiman Long <waiman.long@hpe.com> - 2015-11-05 17:10 +0100
Re: [PATCH tip/locking/core v9 2/6] locking/qspinlock: prefetch next node cacheline Peter Zijlstra <peterz@infradead.org> - 2015-11-05 17:40 +0100
Re: [PATCH tip/locking/core v9 2/6] locking/qspinlock: prefetch next node cacheline Waiman Long <waiman.long@hpe.com> - 2015-11-05 18:00 +0100
[PATCH tip/locking/core v9 5/6] locking/pvqspinlock: Allow 1 lock stealing attempt Waiman Long <Waiman.Long@hpe.com> - 2015-10-31 00:30 +0100
Re: [PATCH tip/locking/core v9 5/6] locking/pvqspinlock: Allow 1 lock stealing attempt Peter Zijlstra <peterz@infradead.org> - 2015-11-06 16:00 +0100
Re: [PATCH tip/locking/core v9 5/6] locking/pvqspinlock: Allow 1 lock stealing attempt Waiman Long <waiman.long@hpe.com> - 2015-11-06 18:50 +0100
Re: [PATCH tip/locking/core v9 5/6] locking/pvqspinlock: Allow 1 lock stealing attempt Peter Zijlstra <peterz@infradead.org> - 2015-11-09 18:40 +0100
Re: [PATCH tip/locking/core v9 5/6] locking/pvqspinlock: Allow 1 lock stealing attempt Waiman Long <waiman.long@hpe.com> - 2015-11-09 21:00 +0100
[PATCH tip/locking/core v9 6/6] locking/pvqspinlock: Queue node adaptive spinning Waiman Long <Waiman.Long@hpe.com> - 2015-10-31 00:30 +0100
Re: [PATCH tip/locking/core v9 6/6] locking/pvqspinlock: Queue node adaptive spinning Peter Zijlstra <peterz@infradead.org> - 2015-11-06 16:10 +0100
Re: [PATCH tip/locking/core v9 6/6] locking/pvqspinlock: Queue node adaptive spinning Waiman Long <waiman.long@hpe.com> - 2015-11-06 19:00 +0100
Re: [PATCH tip/locking/core v9 6/6] locking/pvqspinlock: Queue node adaptive spinning Peter Zijlstra <peterz@infradead.org> - 2015-11-06 21:40 +0100
Re: [PATCH tip/locking/core v9 6/6] locking/pvqspinlock: Queue node adaptive spinning Waiman Long <waiman.long@hpe.com> - 2015-11-09 18:00 +0100
Re: [PATCH tip/locking/core v9 6/6] locking/pvqspinlock: Queue node adaptive spinning Peter Zijlstra <peterz@infradead.org> - 2015-11-09 18:40 +0100
[PATCH tip/locking/core v9 3/6] locking/pvqspinlock, x86: Optimize PV unlock code path Waiman Long <Waiman.Long@hpe.com> - 2015-10-31 00:30 +0100
[PATCH tip/locking/core v9 4/6] locking/pvqspinlock: Collect slowpath lock statistics Waiman Long <Waiman.Long@hpe.com> - 2015-10-31 00:30 +0100
Re: [PATCH tip/locking/core v9 4/6] locking/pvqspinlock: Collect slowpath lock statistics Peter Zijlstra <peterz@infradead.org> - 2015-11-02 17:50 +0100
Re: [PATCH tip/locking/core v9 4/6] locking/pvqspinlock: Collect slowpath lock statistics Waiman Long <waiman.long@hpe.com> - 2015-11-05 17:30 +0100
Re: [PATCH tip/locking/core v9 4/6] locking/pvqspinlock: Collect slowpath lock statistics Peter Zijlstra <peterz@infradead.org> - 2015-11-05 17:50 +0100
Re: [PATCH tip/locking/core v9 4/6] locking/pvqspinlock: Collect slowpath lock statistics Waiman Long <waiman.long@hpe.com> - 2015-11-05 18:00 +0100
Re: [PATCH tip/locking/core v9 4/6] locking/pvqspinlock: Collect slowpath lock statistics Peter Zijlstra <peterz@infradead.org> - 2015-11-05 18:10 +0100
Re: [PATCH tip/locking/core v9 4/6] locking/pvqspinlock: Collect slowpath lock statistics Waiman Long <waiman.long@hpe.com> - 2015-11-05 18:40 +0100
csiph-web