Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1622453 > unrolled thread
| Started by | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| First post | 2017-04-12 19:50 +0200 |
| Last post | 2017-04-21 04:20 +0200 |
| Articles | 20 on this page of 93 — 6 participants |
Back to article view | Back to linux.kernel
[PATCH tip/core/rcu 0/40] SRCU callback parallelization for 4.12 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 19:50 +0200
[PATCH tip/core/rcu 03/40] srcu: Consolidate batch checking into rcu_all_batches_empty() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 20:00 +0200
[PATCH tip/core/rcu 05/40] rcu: Semicolon inside RCU_TRACE() for rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 20:00 +0200
[PATCH tip/core/rcu 11/40] rcu: Pull rcu_sched_qs_mask into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 20:00 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling Peter Zijlstra <peterz@infradead.org> - 2017-04-13 12:00 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-13 18:40 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-13 19:50 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling Peter Zijlstra <peterz@infradead.org> - 2017-04-13 12:00 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-13 19:00 +0200
[PATCH v2 tip/core/rcu 0/40] SRCU callback parallelization for 4.12 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 02/39] rcu: Make arch select smp_mb__after_unlock_lock() strength "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
Re: [PATCH v2 tip/core/rcu 02/39] rcu: Make arch select smp_mb__after_unlock_lock() strength Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:20 +0200
[PATCH v2 tip/core/rcu 10/39] rcu: Eliminate flavor scan in rcu_momentary_dyntick_idle() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 24/39] srcu: Move combining-tree definitions for SRCU's benefit "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 31/39] srcu: Allow a second bit in rcu_seq for SRCU state "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 15/39] srcu: Allow early boot use of synchronize_srcu() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 14/39] srcu: Allow SRCU to access rcu_scheduler_active "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 05/39] rcu: Semicolon inside RCU_TRACE() for rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 33/39] srcu: Crude control of expedited grace periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 29/39] srcu: Fix bogus try_check_zero() comment "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 25/39] srcu: Move rcu_init_levelspread() to rcu_tree_node.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 27/39] srcu: Move rcu_node traversal macros to rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 06/39] rcu: Semicolon inside RCU_TRACE() for Tiny RCU "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 39/39] rcu: Make non-preemptive schedule be Tasks RCU quiescent state "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 07/39] rcu: Semicolon inside RCU_TRACE() for tree.c "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 18/39] rcu: Expedited wakeups need to be fully ordered "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 21/39] srcu: Move to state-based grace-period sequencing "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 22/39] srcu: Add grace-period sequence numbers "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 34/39] mm: Use static initialization for "srcu" "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 35/39] srcu: Create a tiny SRCU "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 16/39] rcu: Add single-element dequeue functions to rcu_segcblist "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 19/39] rcu: Fix warning in rcu_seq_end() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 03/39] srcu: Consolidate batch checking into rcu_all_batches_empty() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
Re: [PATCH v2 tip/core/rcu 03/39] srcu: Consolidate batch checking into rcu_all_batches_empty() Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:40 +0200
[PATCH v2 tip/core/rcu 08/39] rcu: Pull rcu_sched_qs_mask into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 09/39] rcu: Pull rcu_qs_ctr into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 30/39] srcu: Improve rcu_seq grace-period-counter abstraction "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 17/39] srcu: Move rcu_seq_start() and friends to rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 11/39] rcu: Place guard on rcu_all_qs() and rcu_note_context_switch() actions "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 28/39] srcu: Make num_rcu_lvl[] array be external "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 26/39] rcu: Remove redundant levelcnt[] array from rcu_init_one() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 32/39] srcu: Merge ->srcu_state into ->srcu_gp_seq "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 36/39] srcutorture: Print Tiny SRCU reader statistics "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 01/39] rcu: Maintain special bits at bottom of ->dynticks counter "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
Re: [PATCH v2 tip/core/rcu 01/39] rcu: Maintain special bits at bottom of ->dynticks counter Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:10 +0200
Re: [PATCH v2 tip/core/rcu 01/39] rcu: Maintain special bits at bottom of ->dynticks counter "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 20:30 +0200
[PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:40 +0200
Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 20:40 +0200
Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() Josh Triplett <josh@joshtriplett.org> - 2017-04-18 21:50 +0200
Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:40 +0200
[PATCH v3 tip/core/rcu 0/40] SRCU callback parallelization for 4.12 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:00 +0200
[PATCH v3 tip/core/rcu 35/40] srcu: Create a tiny SRCU "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:00 +0200
[PATCH v3 tip/core/rcu 12/40] rcu: Default RCU_FANOUT_LEAF to 16 unless explicitly changed "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:00 +0200
[PATCH v3 tip/core/rcu 05/40] rcu: Semicolon inside RCU_TRACE() for rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
Re: [PATCH v3 tip/core/rcu 05/40] rcu: Semicolon inside RCU_TRACE() for rcu.h Joe Perches <joe@perches.com> - 2017-04-19 19:50 +0200
[PATCH v3 tip/core/rcu 26/40] rcu: Remove redundant levelcnt[] array from rcu_init_one() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 14/40] srcu: Allow SRCU to access rcu_scheduler_active "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 27/40] srcu: Move rcu_node traversal macros to rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 19/40] rcu: Fix warning in rcu_seq_end() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 29/40] srcu: Fix bogus try_check_zero() comment "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 28/40] srcu: Make num_rcu_lvl[] array be external "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 22/40] srcu: Add grace-period sequence numbers "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 39/40] srcu: Expedite srcu_schedule_cbs_snp() callback invocation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 30/40] srcu: Improve rcu_seq grace-period-counter abstraction "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 01/40] rcu: Maintain special bits at bottom of ->dynticks counter "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 20/40] srcu: Push srcu_advance_batches() fastpath into common case "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 34/40] mm: Use static initialization for "srcu" "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 17/40] srcu: Move rcu_seq_start() and friends to rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 08/40] rcu: Pull rcu_sched_qs_mask into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 36/40] srcutorture: Print Tiny SRCU reader statistics "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 09/40] rcu: Pull rcu_qs_ctr into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 04/40] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 07/40] rcu: Semicolon inside RCU_TRACE() for tree.c "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 10/40] rcu: Eliminate flavor scan in rcu_momentary_dyntick_idle() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 02/40] rcu: Make arch select smp_mb__after_unlock_lock() strength "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 24/40] srcu: Move combining-tree definitions for SRCU's benefit "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 15/40] srcu: Allow early boot use of synchronize_srcu() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 33/40] srcu: Crude control of expedited grace periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 06/40] rcu: Semicolon inside RCU_TRACE() for Tiny RCU "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 03/40] srcu: Consolidate batch checking into rcu_all_batches_empty() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 32/40] srcu: Merge ->srcu_state into ->srcu_gp_seq "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 25/40] srcu: Move rcu_init_levelspread() to rcu_tree_node.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Michael Ellerman <michaele@au1.ibm.com> - 2017-04-20 05:50 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-20 16:30 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Paolo Bonzini <pbonzini@redhat.com> - 2017-04-20 17:30 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-21 02:40 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Michael Ellerman <michaele@au1.ibm.com> - 2017-04-21 03:50 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-21 06:20 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Paolo Bonzini <pbonzini@redhat.com> - 2017-04-21 09:30 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-21 15:00 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Michael Ellerman <michaele@au1.ibm.com> - 2017-04-22 08:20 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Michael Ellerman <michaele@au1.ibm.com> - 2017-04-21 04:20 +0200
Page 3 of 5 — ← Prev page 1 2 [3] 4 5 Next page →
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 02:00 +0200 |
| Subject | [PATCH v2 tip/core/rcu 26/39] rcu: Remove redundant levelcnt[] array from rcu_init_one() |
| Message-ID | <txoIO-7c0-23@gated-at.bofh.it> |
| In reply to | #1624925 |
The levelcnt[] array is identical to num_rcu_lvl[], so this commit
removes levelcnt[].
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/tree.c | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 436eaa3047e6..21140192fd76 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3961,7 +3961,6 @@ static void __init rcu_init_one(struct rcu_state *rsp)
static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS];
- int levelcnt[RCU_NUM_LVLS]; /* # nodes in each level. */
int levelspread[RCU_NUM_LVLS]; /* kids/node in each level. */
int cpustride = 1;
int i;
@@ -3976,18 +3975,16 @@ static void __init rcu_init_one(struct rcu_state *rsp)
/* Initialize the level-tracking arrays. */
- for (i = 0; i < rcu_num_lvls; i++)
- levelcnt[i] = num_rcu_lvl[i];
for (i = 1; i < rcu_num_lvls; i++)
- rsp->level[i] = rsp->level[i - 1] + levelcnt[i - 1];
- rcu_init_levelspread(levelspread, levelcnt);
+ rsp->level[i] = rsp->level[i - 1] + num_rcu_lvl[i - 1];
+ rcu_init_levelspread(levelspread, num_rcu_lvl);
/* Initialize the elements themselves, starting from the leaves. */
for (i = rcu_num_lvls - 1; i >= 0; i--) {
cpustride *= levelspread[i];
rnp = rsp->level[i];
- for (j = 0; j < levelcnt[i]; j++, rnp++) {
+ for (j = 0; j < num_rcu_lvl[i]; j++, rnp++) {
raw_spin_lock_init(&ACCESS_PRIVATE(rnp, lock));
lockdep_set_class_and_name(&ACCESS_PRIVATE(rnp, lock),
&rcu_node_class[i], buf[i]);
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 02:00 +0200 |
| Subject | [PATCH v2 tip/core/rcu 32/39] srcu: Merge ->srcu_state into ->srcu_gp_seq |
| Message-ID | <txoIO-7c0-27@gated-at.bofh.it> |
| In reply to | #1624925 |
Updating ->srcu_state and ->srcu_gp_seq will lead to extremely complex
race conditions given multiple callback queues, so this commit takes
advantage of the two-bit state now available in rcu_seq counters to
store the state in the bottom two bits of ->srcu_gp_seq.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/srcu.h | 5 +----
kernel/rcu/rcu.h | 10 ++++++++++
kernel/rcu/srcu.c | 55 +++++++++++++++++++++++++++++++++-------------------
3 files changed, 46 insertions(+), 24 deletions(-)
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index ad154a7bc114..e7dbc01b61a1 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -43,8 +43,7 @@ struct srcu_struct {
unsigned long completed;
unsigned long srcu_gp_seq;
struct srcu_array __percpu *per_cpu_ref;
- spinlock_t queue_lock; /* protect ->srcu_cblist, ->srcu_state */
- int srcu_state;
+ spinlock_t queue_lock; /* protect ->srcu_cblist */
struct rcu_segcblist srcu_cblist;
struct delayed_work work;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
@@ -56,7 +55,6 @@ struct srcu_struct {
#define SRCU_STATE_IDLE 0
#define SRCU_STATE_SCAN1 1
#define SRCU_STATE_SCAN2 2
-#define SRCU_STATE_DONE 3
#ifdef CONFIG_DEBUG_LOCK_ALLOC
@@ -85,7 +83,6 @@ void process_srcu(struct work_struct *work);
.completed = -300, \
.per_cpu_ref = &name##_srcu_array, \
.queue_lock = __SPIN_LOCK_UNLOCKED(name.queue_lock), \
- .srcu_state = SRCU_STATE_IDLE, \
.srcu_cblist = RCU_SEGCBLIST_INITIALIZER(name.srcu_cblist),\
.work = __DELAYED_WORK_INITIALIZER(name.work, process_srcu, 0),\
__SRCU_DEP_MAP_INIT(name) \
diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
index 87a0ac95b551..73e16ec4054b 100644
--- a/kernel/rcu/rcu.h
+++ b/kernel/rcu/rcu.h
@@ -82,6 +82,16 @@ static inline int rcu_seq_state(unsigned long s)
return s & RCU_SEQ_STATE_MASK;
}
+/*
+ * Set the state portion of the pointed-to sequence number.
+ * The caller is responsible for preventing conflicting updates.
+ */
+static inline void rcu_seq_set_state(unsigned long *sp, int newstate)
+{
+ WARN_ON_ONCE(newstate & ~RCU_SEQ_STATE_MASK);
+ WRITE_ONCE(*sp, (*sp & ~RCU_SEQ_STATE_MASK) + newstate);
+}
+
/* Adjust sequence number for start of update-side operation. */
static inline void rcu_seq_start(unsigned long *sp)
{
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index 5aeeaecfb673..97aec5d7b316 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -44,7 +44,6 @@ static int init_srcu_struct_fields(struct srcu_struct *sp)
sp->completed = 0;
sp->srcu_gp_seq = 0;
spin_lock_init(&sp->queue_lock);
- sp->srcu_state = SRCU_STATE_IDLE;
rcu_segcblist_init(&sp->srcu_cblist);
INIT_DELAYED_WORK(&sp->work, process_srcu);
sp->per_cpu_ref = alloc_percpu(struct srcu_array);
@@ -180,6 +179,9 @@ static bool srcu_readers_active(struct srcu_struct *sp)
return sum;
}
+#define SRCU_CALLBACK_BATCH 10
+#define SRCU_INTERVAL 1
+
/**
* cleanup_srcu_struct - deconstruct a sleep-RCU structure
* @sp: structure to clean up.
@@ -194,8 +196,10 @@ void cleanup_srcu_struct(struct srcu_struct *sp)
if (WARN_ON(!rcu_segcblist_empty(&sp->srcu_cblist)))
return; /* Leakage unless caller handles error. */
flush_delayed_work(&sp->work);
- if (WARN_ON(READ_ONCE(sp->srcu_state) != SRCU_STATE_IDLE))
+ if (WARN_ON(rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)) != SRCU_STATE_IDLE)) {
+ pr_info("cleanup_srcu_struct: Active srcu_struct %lu CBs %c state: %d\n", rcu_segcblist_n_cbs(&sp->srcu_cblist), ".E"[rcu_segcblist_empty(&sp->srcu_cblist)], rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)));
return; /* Caller forgot to stop doing call_srcu()? */
+ }
free_percpu(sp->per_cpu_ref);
sp->per_cpu_ref = NULL;
}
@@ -247,10 +251,13 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);
*/
static void srcu_gp_start(struct srcu_struct *sp)
{
+ int state;
+
rcu_segcblist_accelerate(&sp->srcu_cblist,
rcu_seq_snap(&sp->srcu_gp_seq));
- WRITE_ONCE(sp->srcu_state, SRCU_STATE_SCAN1);
rcu_seq_start(&sp->srcu_gp_seq);
+ state = rcu_seq_state(READ_ONCE(sp->srcu_gp_seq));
+ WARN_ON_ONCE(state != SRCU_STATE_SCAN1);
}
/*
@@ -294,7 +301,6 @@ static void srcu_flip(struct srcu_struct *sp)
static void srcu_gp_end(struct srcu_struct *sp)
{
rcu_seq_end(&sp->srcu_gp_seq);
- WRITE_ONCE(sp->srcu_state, SRCU_STATE_DONE);
spin_lock_irq(&sp->queue_lock);
rcu_segcblist_advance(&sp->srcu_cblist,
@@ -339,7 +345,7 @@ void call_srcu(struct srcu_struct *sp, struct rcu_head *head,
spin_lock_irqsave(&sp->queue_lock, flags);
smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
rcu_segcblist_enqueue(&sp->srcu_cblist, head, false);
- if (READ_ONCE(sp->srcu_state) == SRCU_STATE_IDLE) {
+ if (rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)) == SRCU_STATE_IDLE) {
srcu_gp_start(sp);
queue_delayed_work(system_power_efficient_wq, &sp->work, 0);
}
@@ -372,7 +378,7 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
head->func = wakeme_after_rcu;
spin_lock_irq(&sp->queue_lock);
smp_mb__after_unlock_lock(); /* Caller's prior accesses before GP. */
- if (READ_ONCE(sp->srcu_state) == SRCU_STATE_IDLE) {
+ if (rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)) == SRCU_STATE_IDLE) {
/* steal the processing owner */
rcu_segcblist_enqueue(&sp->srcu_cblist, head, false);
srcu_gp_start(sp);
@@ -474,9 +480,6 @@ unsigned long srcu_batches_completed(struct srcu_struct *sp)
}
EXPORT_SYMBOL_GPL(srcu_batches_completed);
-#define SRCU_CALLBACK_BATCH 10
-#define SRCU_INTERVAL 1
-
/*
* Core SRCU state machine. Advance callbacks from ->batch_check0 to
* ->batch_check1 and then to ->batch_done as readers drain.
@@ -485,28 +488,40 @@ static void srcu_advance_batches(struct srcu_struct *sp, int trycount)
{
int idx;
- WARN_ON_ONCE(sp->srcu_state == SRCU_STATE_IDLE);
-
/*
* Because readers might be delayed for an extended period after
* fetching ->completed for their index, at any point in time there
* might well be readers using both idx=0 and idx=1. We therefore
* need to wait for readers to clear from both index values before
* invoking a callback.
+ *
+ * The load-acquire ensures that we see the accesses performed
+ * by the prior grace period.
*/
+ idx = rcu_seq_state(smp_load_acquire(&sp->srcu_gp_seq)); /* ^^^ */
+ if (idx == SRCU_STATE_IDLE) {
+ spin_lock_irq(&sp->queue_lock);
+ if (rcu_segcblist_empty(&sp->srcu_cblist)) {
+ spin_unlock_irq(&sp->queue_lock);
+ return;
+ }
+ idx = rcu_seq_state(READ_ONCE(sp->srcu_gp_seq));
+ if (idx == SRCU_STATE_IDLE)
+ srcu_gp_start(sp);
+ spin_unlock_irq(&sp->queue_lock);
+ if (idx != SRCU_STATE_IDLE)
+ return; /* Someone else started the grace period. */
+ }
- if (sp->srcu_state == SRCU_STATE_DONE)
- srcu_gp_start(sp);
-
- if (sp->srcu_state == SRCU_STATE_SCAN1) {
+ if (rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)) == SRCU_STATE_SCAN1) {
idx = 1 ^ (sp->completed & 1);
if (!try_check_zero(sp, idx, trycount))
return; /* readers present, retry after SRCU_INTERVAL */
srcu_flip(sp);
- WRITE_ONCE(sp->srcu_state, SRCU_STATE_SCAN2);
+ rcu_seq_set_state(&sp->srcu_gp_seq, SRCU_STATE_SCAN2);
}
- if (sp->srcu_state == SRCU_STATE_SCAN2) {
+ if (rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)) == SRCU_STATE_SCAN2) {
/*
* SRCU read-side critical sections are normally short,
@@ -557,14 +572,14 @@ static void srcu_invoke_callbacks(struct srcu_struct *sp)
static void srcu_reschedule(struct srcu_struct *sp, unsigned long delay)
{
bool pending = true;
+ int state;
if (rcu_segcblist_empty(&sp->srcu_cblist)) {
spin_lock_irq(&sp->queue_lock);
+ state = rcu_seq_state(READ_ONCE(sp->srcu_gp_seq));
if (rcu_segcblist_empty(&sp->srcu_cblist) &&
- READ_ONCE(sp->srcu_state) == SRCU_STATE_DONE) {
- WRITE_ONCE(sp->srcu_state, SRCU_STATE_IDLE);
+ state == SRCU_STATE_IDLE)
pending = false;
- }
spin_unlock_irq(&sp->queue_lock);
}
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 02:00 +0200 |
| Subject | [PATCH v2 tip/core/rcu 36/39] srcutorture: Print Tiny SRCU reader statistics |
| Message-ID | <txoIP-7c0-43@gated-at.bofh.it> |
| In reply to | #1624925 |
The srcu_torture_stats() function is adapted to the specific srcu_struct
layout traditionally used by SRCU. This commit therefore adds support
for Tiny SRCU.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/rcutorture.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 98591e16db1a..9cbb8a7b909d 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -559,11 +559,12 @@ static void srcu_torture_barrier(void)
static void srcu_torture_stats(void)
{
-#ifdef CONFIG_TREE_SRCU
- int cpu;
- int idx = srcu_ctlp->completed & 0x1;
+ int __maybe_unused cpu;
+ int idx;
- pr_alert("%s%s per-CPU(idx=%d):",
+#ifdef CONFIG_TREE_SRCU
+ idx = srcu_ctlp->completed & 0x1;
+ pr_alert("%s%s Tree SRCU per-CPU(idx=%d):",
torture_type, TORTURE_FLAG, idx);
for_each_possible_cpu(cpu) {
unsigned long l0, l1;
@@ -588,6 +589,12 @@ static void srcu_torture_stats(void)
pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
}
pr_cont("\n");
+#elif defined(CONFIG_TINY_SRCU)
+ idx = READ_ONCE(srcu_ctlp->srcu_idx) & 0x1;
+ pr_alert("%s%s Tiny SRCU per-CPU(idx=%d): (%d,%d)\n",
+ torture_type, TORTURE_FLAG, idx,
+ READ_ONCE(srcu_ctlp->srcu_lock_nesting[!idx]),
+ READ_ONCE(srcu_ctlp->srcu_lock_nesting[idx]));
#endif
}
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 02:00 +0200 |
| Subject | [PATCH v2 tip/core/rcu 01/39] rcu: Maintain special bits at bottom of ->dynticks counter |
| Message-ID | <txoIP-7c0-35@gated-at.bofh.it> |
| In reply to | #1624925 |
Currently, IPIs are used to force other CPUs to invalidate their TLBs
in response to a kernel virtual-memory mapping change. This works, but
degrades both battery lifetime (for idle CPUs) and real-time response
(for nohz_full CPUs), and in addition results in unnecessary IPIs due to
the fact that CPUs executing in usermode are unaffected by stale kernel
mappings. It would be better to cause a CPU executing in usermode to
wait until it is entering kernel mode to do the flush, first to avoid
interrupting usemode tasks and second to handle multiple flush requests
with a single flush in the case of a long-running user task.
This commit therefore reserves a bit at the bottom of the ->dynticks
counter, which is checked upon exit from extended quiescent states.
If it is set, it is cleared and then a new rcu_eqs_special_exit() macro is
invoked, which, if not supplied, is an empty single-pass do-while loop.
If this bottom bit is set on -entry- to an extended quiescent state,
then a WARN_ON_ONCE() triggers.
This bottom bit may be set using a new rcu_eqs_special_set() function,
which returns true if the bit was set, or false if the CPU turned
out to not be in an extended quiescent state. Please note that this
function refuses to set the bit for a non-nohz_full CPU when that CPU
is executing in usermode because usermode execution is tracked by RCU
as a dyntick-idle extended quiescent state only for nohz_full CPUs.
Reported-by: Andy Lutomirski <luto@amacapital.net>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcutiny.h | 5 ++++
kernel/rcu/tree.c | 73 +++++++++++++++++++++++++++++++++++++++----------
kernel/rcu/tree.h | 1 +
3 files changed, 65 insertions(+), 14 deletions(-)
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index b452953e21c8..6c9d941e3962 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -33,6 +33,11 @@ static inline int rcu_dynticks_snap(struct rcu_dynticks *rdtp)
return 0;
}
+static inline bool rcu_eqs_special_set(int cpu)
+{
+ return false; /* Never flag non-existent other CPUs! */
+}
+
static inline unsigned long get_state_synchronize_rcu(void)
{
return 0;
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 50fee7689e71..df8f6401f2d5 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -274,9 +274,19 @@ void rcu_bh_qs(void)
static DEFINE_PER_CPU(int, rcu_sched_qs_mask);
+/*
+ * Steal a bit from the bottom of ->dynticks for idle entry/exit
+ * control. Initially this is for TLB flushing.
+ */
+#define RCU_DYNTICK_CTRL_MASK 0x1
+#define RCU_DYNTICK_CTRL_CTR (RCU_DYNTICK_CTRL_MASK + 1)
+#ifndef rcu_eqs_special_exit
+#define rcu_eqs_special_exit() do { } while (0)
+#endif
+
static DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = {
.dynticks_nesting = DYNTICK_TASK_EXIT_IDLE,
- .dynticks = ATOMIC_INIT(1),
+ .dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR),
#ifdef CONFIG_NO_HZ_FULL_SYSIDLE
.dynticks_idle_nesting = DYNTICK_TASK_NEST_VALUE,
.dynticks_idle = ATOMIC_INIT(1),
@@ -290,15 +300,20 @@ static DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = {
static void rcu_dynticks_eqs_enter(void)
{
struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
- int special;
+ int seq;
/*
* CPUs seeing atomic_inc_return() must see prior RCU read-side
* critical sections, and we also must force ordering with the
* next idle sojourn.
*/
- special = atomic_inc_return(&rdtp->dynticks);
- WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && special & 0x1);
+ seq = atomic_add_return(RCU_DYNTICK_CTRL_CTR, &rdtp->dynticks);
+ /* Better be in an extended quiescent state! */
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) &&
+ (seq & RCU_DYNTICK_CTRL_CTR));
+ /* Better not have special action (TLB flush) pending! */
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) &&
+ (seq & RCU_DYNTICK_CTRL_MASK));
}
/*
@@ -308,15 +323,22 @@ static void rcu_dynticks_eqs_enter(void)
static void rcu_dynticks_eqs_exit(void)
{
struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
- int special;
+ int seq;
/*
* CPUs seeing atomic_inc_return() must see prior idle sojourns,
* and we also must force ordering with the next RCU read-side
* critical section.
*/
- special = atomic_inc_return(&rdtp->dynticks);
- WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(special & 0x1));
+ seq = atomic_add_return(RCU_DYNTICK_CTRL_CTR, &rdtp->dynticks);
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) &&
+ !(seq & RCU_DYNTICK_CTRL_CTR));
+ if (seq & RCU_DYNTICK_CTRL_MASK) {
+ atomic_andnot(RCU_DYNTICK_CTRL_MASK, &rdtp->dynticks);
+ smp_mb__after_atomic(); /* _exit after clearing mask. */
+ /* Prefer duplicate flushes to losing a flush. */
+ rcu_eqs_special_exit();
+ }
}
/*
@@ -333,9 +355,9 @@ static void rcu_dynticks_eqs_online(void)
{
struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
- if (atomic_read(&rdtp->dynticks) & 0x1)
+ if (atomic_read(&rdtp->dynticks) & RCU_DYNTICK_CTRL_CTR)
return;
- atomic_add(0x1, &rdtp->dynticks);
+ atomic_add(RCU_DYNTICK_CTRL_CTR, &rdtp->dynticks);
}
/*
@@ -347,7 +369,7 @@ bool rcu_dynticks_curr_cpu_in_eqs(void)
{
struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
- return !(atomic_read(&rdtp->dynticks) & 0x1);
+ return !(atomic_read(&rdtp->dynticks) & RCU_DYNTICK_CTRL_CTR);
}
/*
@@ -358,7 +380,7 @@ int rcu_dynticks_snap(struct rcu_dynticks *rdtp)
{
int snap = atomic_add_return(0, &rdtp->dynticks);
- return snap;
+ return snap & ~RCU_DYNTICK_CTRL_MASK;
}
/*
@@ -367,7 +389,7 @@ int rcu_dynticks_snap(struct rcu_dynticks *rdtp)
*/
static bool rcu_dynticks_in_eqs(int snap)
{
- return !(snap & 0x1);
+ return !(snap & RCU_DYNTICK_CTRL_CTR);
}
/*
@@ -387,10 +409,33 @@ static bool rcu_dynticks_in_eqs_since(struct rcu_dynticks *rdtp, int snap)
static void rcu_dynticks_momentary_idle(void)
{
struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
- int special = atomic_add_return(2, &rdtp->dynticks);
+ int special = atomic_add_return(2 * RCU_DYNTICK_CTRL_CTR,
+ &rdtp->dynticks);
/* It is illegal to call this from idle state. */
- WARN_ON_ONCE(!(special & 0x1));
+ WARN_ON_ONCE(!(special & RCU_DYNTICK_CTRL_CTR));
+}
+
+/*
+ * Set the special (bottom) bit of the specified CPU so that it
+ * will take special action (such as flushing its TLB) on the
+ * next exit from an extended quiescent state. Returns true if
+ * the bit was successfully set, or false if the CPU was not in
+ * an extended quiescent state.
+ */
+bool rcu_eqs_special_set(int cpu)
+{
+ int old;
+ int new;
+ struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
+
+ do {
+ old = atomic_read(&rdtp->dynticks);
+ if (old & RCU_DYNTICK_CTRL_CTR)
+ return false;
+ new = old | RCU_DYNTICK_CTRL_MASK;
+ } while (atomic_cmpxchg(&rdtp->dynticks, old, new) != old);
+ return true;
}
DEFINE_PER_CPU_SHARED_ALIGNED(unsigned long, rcu_qs_ctr);
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index ec62a05bfdb3..7468b4de7e0c 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -596,6 +596,7 @@ extern struct rcu_state rcu_preempt_state;
#endif /* #ifdef CONFIG_PREEMPT_RCU */
int rcu_dynticks_snap(struct rcu_dynticks *rdtp);
+bool rcu_eqs_special_set(int cpu);
#ifdef CONFIG_RCU_BOOST
DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_status);
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | Josh Triplett <josh@joshtriplett.org> |
|---|---|
| Date | 2017-04-18 02:10 +0200 |
| Subject | Re: [PATCH v2 tip/core/rcu 01/39] rcu: Maintain special bits at bottom of ->dynticks counter |
| Message-ID | <txoSu-7uv-9@gated-at.bofh.it> |
| In reply to | #1624961 |
On Mon, Apr 17, 2017 at 04:44:48PM -0700, Paul E. McKenney wrote:
> Currently, IPIs are used to force other CPUs to invalidate their TLBs
> in response to a kernel virtual-memory mapping change. This works, but
> degrades both battery lifetime (for idle CPUs) and real-time response
> (for nohz_full CPUs), and in addition results in unnecessary IPIs due to
> the fact that CPUs executing in usermode are unaffected by stale kernel
> mappings. It would be better to cause a CPU executing in usermode to
> wait until it is entering kernel mode to do the flush, first to avoid
> interrupting usemode tasks and second to handle multiple flush requests
> with a single flush in the case of a long-running user task.
>
> This commit therefore reserves a bit at the bottom of the ->dynticks
> counter, which is checked upon exit from extended quiescent states.
> If it is set, it is cleared and then a new rcu_eqs_special_exit() macro is
> invoked, which, if not supplied, is an empty single-pass do-while loop.
> If this bottom bit is set on -entry- to an extended quiescent state,
> then a WARN_ON_ONCE() triggers.
>
> This bottom bit may be set using a new rcu_eqs_special_set() function,
> which returns true if the bit was set, or false if the CPU turned
> out to not be in an extended quiescent state. Please note that this
> function refuses to set the bit for a non-nohz_full CPU when that CPU
> is executing in usermode because usermode execution is tracked by RCU
> as a dyntick-idle extended quiescent state only for nohz_full CPUs.
>
> Reported-by: Andy Lutomirski <luto@amacapital.net>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Comments below. With those fixed:
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -290,15 +300,20 @@ static DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = {
> static void rcu_dynticks_eqs_enter(void)
> {
> struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
> - int special;
> + int seq;
>
> /*
> * CPUs seeing atomic_inc_return() must see prior RCU read-side
> * critical sections, and we also must force ordering with the
> * next idle sojourn.
> */
> - special = atomic_inc_return(&rdtp->dynticks);
> - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && special & 0x1);
> + seq = atomic_add_return(RCU_DYNTICK_CTRL_CTR, &rdtp->dynticks);
You changed atomic_inc_return to atomic_add_return here, but the comment
above still says atomic_inc_return.
> @@ -308,15 +323,22 @@ static void rcu_dynticks_eqs_enter(void)
> static void rcu_dynticks_eqs_exit(void)
> {
> struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
> - int special;
> + int seq;
>
> /*
> * CPUs seeing atomic_inc_return() must see prior idle sojourns,
> * and we also must force ordering with the next RCU read-side
> * critical section.
> */
> - special = atomic_inc_return(&rdtp->dynticks);
> - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(special & 0x1));
> + seq = atomic_add_return(RCU_DYNTICK_CTRL_CTR, &rdtp->dynticks);
Likewise.
- Josh Triplett
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 20:30 +0200 |
| Subject | Re: [PATCH v2 tip/core/rcu 01/39] rcu: Maintain special bits at bottom of ->dynticks counter |
| Message-ID | <txG30-10r-21@gated-at.bofh.it> |
| In reply to | #1624968 |
On Mon, Apr 17, 2017 at 05:07:55PM -0700, Josh Triplett wrote:
> On Mon, Apr 17, 2017 at 04:44:48PM -0700, Paul E. McKenney wrote:
> > Currently, IPIs are used to force other CPUs to invalidate their TLBs
> > in response to a kernel virtual-memory mapping change. This works, but
> > degrades both battery lifetime (for idle CPUs) and real-time response
> > (for nohz_full CPUs), and in addition results in unnecessary IPIs due to
> > the fact that CPUs executing in usermode are unaffected by stale kernel
> > mappings. It would be better to cause a CPU executing in usermode to
> > wait until it is entering kernel mode to do the flush, first to avoid
> > interrupting usemode tasks and second to handle multiple flush requests
> > with a single flush in the case of a long-running user task.
> >
> > This commit therefore reserves a bit at the bottom of the ->dynticks
> > counter, which is checked upon exit from extended quiescent states.
> > If it is set, it is cleared and then a new rcu_eqs_special_exit() macro is
> > invoked, which, if not supplied, is an empty single-pass do-while loop.
> > If this bottom bit is set on -entry- to an extended quiescent state,
> > then a WARN_ON_ONCE() triggers.
> >
> > This bottom bit may be set using a new rcu_eqs_special_set() function,
> > which returns true if the bit was set, or false if the CPU turned
> > out to not be in an extended quiescent state. Please note that this
> > function refuses to set the bit for a non-nohz_full CPU when that CPU
> > is executing in usermode because usermode execution is tracked by RCU
> > as a dyntick-idle extended quiescent state only for nohz_full CPUs.
> >
> > Reported-by: Andy Lutomirski <luto@amacapital.net>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>
> Comments below. With those fixed:
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>
Good eyes, fixed!
Thanx, Paul
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -290,15 +300,20 @@ static DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = {
> > static void rcu_dynticks_eqs_enter(void)
> > {
> > struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
> > - int special;
> > + int seq;
> >
> > /*
> > * CPUs seeing atomic_inc_return() must see prior RCU read-side
> > * critical sections, and we also must force ordering with the
> > * next idle sojourn.
> > */
> > - special = atomic_inc_return(&rdtp->dynticks);
> > - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && special & 0x1);
> > + seq = atomic_add_return(RCU_DYNTICK_CTRL_CTR, &rdtp->dynticks);
>
> You changed atomic_inc_return to atomic_add_return here, but the comment
> above still says atomic_inc_return.
>
> > @@ -308,15 +323,22 @@ static void rcu_dynticks_eqs_enter(void)
> > static void rcu_dynticks_eqs_exit(void)
> > {
> > struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
> > - int special;
> > + int seq;
> >
> > /*
> > * CPUs seeing atomic_inc_return() must see prior idle sojourns,
> > * and we also must force ordering with the next RCU read-side
> > * critical section.
> > */
> > - special = atomic_inc_return(&rdtp->dynticks);
> > - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(special & 0x1));
> > + seq = atomic_add_return(RCU_DYNTICK_CTRL_CTR, &rdtp->dynticks);
>
> Likewise.
>
> - Josh Triplett
>
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 02:00 +0200 |
| Subject | [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() |
| Message-ID | <txoIP-7c0-41@gated-at.bofh.it> |
| In reply to | #1624925 |
Users of SRCU are obliged to complete all grace-period activity before
invoking cleanup_srcu_struct(). This means that all calls to either
synchronize_srcu() or synchronize_srcu_expedited() must have returned,
and all calls to call_srcu() must have returned, and the last call to
call_srcu() must have been followed by a call to srcu_barrier().
Furthermore, the caller must have done something to prevent any
further calls to synchronize_srcu(), synchronize_srcu_expedited(),
and call_srcu().
Therefore, if there has ever been an invocation of call_srcu() on
the srcu_struct in question, the sequence of events must be as
follows:
1. Prevent any further calls to call_srcu().
2. Wait for any pre-existing call_srcu() invocations to return.
3. Invoke srcu_barrier().
4. It is now safe to invoke cleanup_srcu_struct().
On the other hand, if there has ever been a call to synchronize_srcu()
or synchronize_srcu_expedited(), the sequence of events must be as
follows:
1. Prevent any further calls to synchronize_srcu() or
synchronize_srcu_expedited().
2. Wait for any pre-existing synchronize_srcu() or
synchronize_srcu_expedited() invocations to return.
3. It is now safe to invoke cleanup_srcu_struct().
If there have been calls to all both types of functions (call_srcu()
and either of synchronize_srcu() and synchronize_srcu_expedited()), then
the caller must do the first three steps of the call_srcu() procedure
above and the first two steps of the synchronize_s*() procedure above,
and only then invoke cleanup_srcu_struct().
Reported-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/srcu.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index ba41a5d04b49..6beeba7b0b67 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -261,6 +261,11 @@ void cleanup_srcu_struct(struct srcu_struct *sp)
{
if (WARN_ON(srcu_readers_active(sp)))
return; /* Leakage unless caller handles error. */
+ if (WARN_ON(!rcu_all_batches_empty(sp)))
+ return; /* Leakage unless caller handles error. */
+ flush_delayed_work(&sp->work);
+ if (WARN_ON(sp->running))
+ return; /* Caller forgot to stop doing call_srcu()? */
free_percpu(sp->per_cpu_ref);
sp->per_cpu_ref = NULL;
}
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | Josh Triplett <josh@joshtriplett.org> |
|---|---|
| Date | 2017-04-18 02:40 +0200 |
| Subject | Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() |
| Message-ID | <txplv-7EF-3@gated-at.bofh.it> |
| In reply to | #1624963 |
On Mon, Apr 17, 2017 at 05:33:32PM -0700, Josh Triplett wrote:
> On Mon, Apr 17, 2017 at 04:44:51PM -0700, Paul E. McKenney wrote:
> > Users of SRCU are obliged to complete all grace-period activity before
> > invoking cleanup_srcu_struct(). This means that all calls to either
> > synchronize_srcu() or synchronize_srcu_expedited() must have returned,
> > and all calls to call_srcu() must have returned, and the last call to
> > call_srcu() must have been followed by a call to srcu_barrier().
> > Furthermore, the caller must have done something to prevent any
> > further calls to synchronize_srcu(), synchronize_srcu_expedited(),
> > and call_srcu().
> >
> > Therefore, if there has ever been an invocation of call_srcu() on
> > the srcu_struct in question, the sequence of events must be as
> > follows:
> >
> > 1. Prevent any further calls to call_srcu().
> > 2. Wait for any pre-existing call_srcu() invocations to return.
> > 3. Invoke srcu_barrier().
> > 4. It is now safe to invoke cleanup_srcu_struct().
> >
> > On the other hand, if there has ever been a call to synchronize_srcu()
> > or synchronize_srcu_expedited(), the sequence of events must be as
> > follows:
> >
> > 1. Prevent any further calls to synchronize_srcu() or
> > synchronize_srcu_expedited().
> > 2. Wait for any pre-existing synchronize_srcu() or
> > synchronize_srcu_expedited() invocations to return.
> > 3. It is now safe to invoke cleanup_srcu_struct().
> >
> > If there have been calls to all both types of functions (call_srcu()
> > and either of synchronize_srcu() and synchronize_srcu_expedited()), then
> > the caller must do the first three steps of the call_srcu() procedure
> > above and the first two steps of the synchronize_s*() procedure above,
> > and only then invoke cleanup_srcu_struct().
>
> This commit message clearly explains the correct sequence for the
> client, but not which aspects of this the change now enforces. Some of
> the steps above remain the responsibility of the caller, while the
> callee now checks more of them. Could you add something at the end
> explaining the change and what it enforces?
More importantly, perhaps this explanation could find its way into the
documentation of cleanup_srcu_struct?
> > Reported-by: Paolo Bonzini <pbonzini@redhat.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>
> With the above change:
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>
>
> > kernel/rcu/srcu.c | 5 +++++
> > 1 file changed, 5 insertions(+)
> >
> > diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
> > index ba41a5d04b49..6beeba7b0b67 100644
> > --- a/kernel/rcu/srcu.c
> > +++ b/kernel/rcu/srcu.c
> > @@ -261,6 +261,11 @@ void cleanup_srcu_struct(struct srcu_struct *sp)
> > {
> > if (WARN_ON(srcu_readers_active(sp)))
> > return; /* Leakage unless caller handles error. */
> > + if (WARN_ON(!rcu_all_batches_empty(sp)))
> > + return; /* Leakage unless caller handles error. */
> > + flush_delayed_work(&sp->work);
> > + if (WARN_ON(sp->running))
> > + return; /* Caller forgot to stop doing call_srcu()? */
> > free_percpu(sp->per_cpu_ref);
> > sp->per_cpu_ref = NULL;
> > }
> > --
> > 2.5.2
> >
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 20:40 +0200 |
| Subject | Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() |
| Message-ID | <txGcF-140-7@gated-at.bofh.it> |
| In reply to | #1624977 |
On Mon, Apr 17, 2017 at 05:34:30PM -0700, Josh Triplett wrote:
> On Mon, Apr 17, 2017 at 05:33:32PM -0700, Josh Triplett wrote:
> > On Mon, Apr 17, 2017 at 04:44:51PM -0700, Paul E. McKenney wrote:
> > > Users of SRCU are obliged to complete all grace-period activity before
> > > invoking cleanup_srcu_struct(). This means that all calls to either
> > > synchronize_srcu() or synchronize_srcu_expedited() must have returned,
> > > and all calls to call_srcu() must have returned, and the last call to
> > > call_srcu() must have been followed by a call to srcu_barrier().
> > > Furthermore, the caller must have done something to prevent any
> > > further calls to synchronize_srcu(), synchronize_srcu_expedited(),
> > > and call_srcu().
> > >
> > > Therefore, if there has ever been an invocation of call_srcu() on
> > > the srcu_struct in question, the sequence of events must be as
> > > follows:
> > >
> > > 1. Prevent any further calls to call_srcu().
> > > 2. Wait for any pre-existing call_srcu() invocations to return.
> > > 3. Invoke srcu_barrier().
> > > 4. It is now safe to invoke cleanup_srcu_struct().
> > >
> > > On the other hand, if there has ever been a call to synchronize_srcu()
> > > or synchronize_srcu_expedited(), the sequence of events must be as
> > > follows:
> > >
> > > 1. Prevent any further calls to synchronize_srcu() or
> > > synchronize_srcu_expedited().
> > > 2. Wait for any pre-existing synchronize_srcu() or
> > > synchronize_srcu_expedited() invocations to return.
> > > 3. It is now safe to invoke cleanup_srcu_struct().
> > >
> > > If there have been calls to all both types of functions (call_srcu()
> > > and either of synchronize_srcu() and synchronize_srcu_expedited()), then
> > > the caller must do the first three steps of the call_srcu() procedure
> > > above and the first two steps of the synchronize_s*() procedure above,
> > > and only then invoke cleanup_srcu_struct().
> >
> > This commit message clearly explains the correct sequence for the
> > client, but not which aspects of this the change now enforces. Some of
> > the steps above remain the responsibility of the caller, while the
> > callee now checks more of them. Could you add something at the end
> > explaining the change and what it enforces?
>
> More importantly, perhaps this explanation could find its way into the
> documentation of cleanup_srcu_struct?
Like this?
/**
* cleanup_srcu_struct - deconstruct a sleep-RCU structure
* @sp: structure to clean up.
*
* Must invoke this only after you are finished using a given srcu_struct
* that was initialized via init_srcu_struct(). This code does some
* probabalistic checking, spotting late uses of srcu_read_lock(),
* synchronize_srcu(), synchronize_srcu_expedited(), and call_srcu().
* If any such late uses are detected, the per-CPU memory associated with
* the srcu_struct is simply leaked and WARN_ON() is invoked. If the
* caller frees the srcu_struct itself, a use-after-free crash will likely
* ensue, but at least there will be a warning printed.
*/
I added the following paragraph to the commit log:
Note that cleanup_srcu_struct() does some probabilistic checks
for the caller failing to follow these procedures, in which
case cleanup_srcu_struct() does WARN_ON() and avoids freeing
the per-CPU structures associated with the specified srcu_struct
structure.
And added your Reviewed-by, but please let me if more is needed.
Thanx, Paul
> > > Reported-by: Paolo Bonzini <pbonzini@redhat.com>
> > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> >
> > With the above change:
> > Reviewed-by: Josh Triplett <josh@joshtriplett.org>
> >
> > > kernel/rcu/srcu.c | 5 +++++
> > > 1 file changed, 5 insertions(+)
> > >
> > > diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
> > > index ba41a5d04b49..6beeba7b0b67 100644
> > > --- a/kernel/rcu/srcu.c
> > > +++ b/kernel/rcu/srcu.c
> > > @@ -261,6 +261,11 @@ void cleanup_srcu_struct(struct srcu_struct *sp)
> > > {
> > > if (WARN_ON(srcu_readers_active(sp)))
> > > return; /* Leakage unless caller handles error. */
> > > + if (WARN_ON(!rcu_all_batches_empty(sp)))
> > > + return; /* Leakage unless caller handles error. */
> > > + flush_delayed_work(&sp->work);
> > > + if (WARN_ON(sp->running))
> > > + return; /* Caller forgot to stop doing call_srcu()? */
> > > free_percpu(sp->per_cpu_ref);
> > > sp->per_cpu_ref = NULL;
> > > }
> > > --
> > > 2.5.2
> > >
>
[toc] | [prev] | [next] | [standalone]
| From | Josh Triplett <josh@joshtriplett.org> |
|---|---|
| Date | 2017-04-18 21:50 +0200 |
| Subject | Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() |
| Message-ID | <txHip-1Gh-9@gated-at.bofh.it> |
| In reply to | #1625526 |
On Tue, Apr 18, 2017 at 11:34:53AM -0700, Paul E. McKenney wrote: > On Mon, Apr 17, 2017 at 05:34:30PM -0700, Josh Triplett wrote: > > On Mon, Apr 17, 2017 at 05:33:32PM -0700, Josh Triplett wrote: > > > On Mon, Apr 17, 2017 at 04:44:51PM -0700, Paul E. McKenney wrote: > > > > Users of SRCU are obliged to complete all grace-period activity before > > > > invoking cleanup_srcu_struct(). This means that all calls to either > > > > synchronize_srcu() or synchronize_srcu_expedited() must have returned, > > > > and all calls to call_srcu() must have returned, and the last call to > > > > call_srcu() must have been followed by a call to srcu_barrier(). > > > > Furthermore, the caller must have done something to prevent any > > > > further calls to synchronize_srcu(), synchronize_srcu_expedited(), > > > > and call_srcu(). > > > > > > > > Therefore, if there has ever been an invocation of call_srcu() on > > > > the srcu_struct in question, the sequence of events must be as > > > > follows: > > > > > > > > 1. Prevent any further calls to call_srcu(). > > > > 2. Wait for any pre-existing call_srcu() invocations to return. > > > > 3. Invoke srcu_barrier(). > > > > 4. It is now safe to invoke cleanup_srcu_struct(). > > > > > > > > On the other hand, if there has ever been a call to synchronize_srcu() > > > > or synchronize_srcu_expedited(), the sequence of events must be as > > > > follows: > > > > > > > > 1. Prevent any further calls to synchronize_srcu() or > > > > synchronize_srcu_expedited(). > > > > 2. Wait for any pre-existing synchronize_srcu() or > > > > synchronize_srcu_expedited() invocations to return. > > > > 3. It is now safe to invoke cleanup_srcu_struct(). > > > > > > > > If there have been calls to all both types of functions (call_srcu() > > > > and either of synchronize_srcu() and synchronize_srcu_expedited()), then > > > > the caller must do the first three steps of the call_srcu() procedure > > > > above and the first two steps of the synchronize_s*() procedure above, > > > > and only then invoke cleanup_srcu_struct(). > > > > > > This commit message clearly explains the correct sequence for the > > > client, but not which aspects of this the change now enforces. Some of > > > the steps above remain the responsibility of the caller, while the > > > callee now checks more of them. Could you add something at the end > > > explaining the change and what it enforces? > > > > More importantly, perhaps this explanation could find its way into the > > documentation of cleanup_srcu_struct? > > Like this? > > /** > * cleanup_srcu_struct - deconstruct a sleep-RCU structure > * @sp: structure to clean up. > * > * Must invoke this only after you are finished using a given srcu_struct > * that was initialized via init_srcu_struct(). This code does some > * probabalistic checking, spotting late uses of srcu_read_lock(), > * synchronize_srcu(), synchronize_srcu_expedited(), and call_srcu(). > * If any such late uses are detected, the per-CPU memory associated with > * the srcu_struct is simply leaked and WARN_ON() is invoked. If the > * caller frees the srcu_struct itself, a use-after-free crash will likely > * ensue, but at least there will be a warning printed. > */ > > I added the following paragraph to the commit log: > > Note that cleanup_srcu_struct() does some probabilistic checks > for the caller failing to follow these procedures, in which > case cleanup_srcu_struct() does WARN_ON() and avoids freeing > the per-CPU structures associated with the specified srcu_struct > structure. > > And added your Reviewed-by, but please let me if more is needed. Looks good to me, thanks for improving the documentation.
[toc] | [prev] | [next] | [standalone]
| From | Josh Triplett <josh@joshtriplett.org> |
|---|---|
| Date | 2017-04-18 02:40 +0200 |
| Subject | Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() |
| Message-ID | <txplv-7EF-5@gated-at.bofh.it> |
| In reply to | #1624963 |
On Mon, Apr 17, 2017 at 04:44:51PM -0700, Paul E. McKenney wrote:
> Users of SRCU are obliged to complete all grace-period activity before
> invoking cleanup_srcu_struct(). This means that all calls to either
> synchronize_srcu() or synchronize_srcu_expedited() must have returned,
> and all calls to call_srcu() must have returned, and the last call to
> call_srcu() must have been followed by a call to srcu_barrier().
> Furthermore, the caller must have done something to prevent any
> further calls to synchronize_srcu(), synchronize_srcu_expedited(),
> and call_srcu().
>
> Therefore, if there has ever been an invocation of call_srcu() on
> the srcu_struct in question, the sequence of events must be as
> follows:
>
> 1. Prevent any further calls to call_srcu().
> 2. Wait for any pre-existing call_srcu() invocations to return.
> 3. Invoke srcu_barrier().
> 4. It is now safe to invoke cleanup_srcu_struct().
>
> On the other hand, if there has ever been a call to synchronize_srcu()
> or synchronize_srcu_expedited(), the sequence of events must be as
> follows:
>
> 1. Prevent any further calls to synchronize_srcu() or
> synchronize_srcu_expedited().
> 2. Wait for any pre-existing synchronize_srcu() or
> synchronize_srcu_expedited() invocations to return.
> 3. It is now safe to invoke cleanup_srcu_struct().
>
> If there have been calls to all both types of functions (call_srcu()
> and either of synchronize_srcu() and synchronize_srcu_expedited()), then
> the caller must do the first three steps of the call_srcu() procedure
> above and the first two steps of the synchronize_s*() procedure above,
> and only then invoke cleanup_srcu_struct().
This commit message clearly explains the correct sequence for the
client, but not which aspects of this the change now enforces. Some of
the steps above remain the responsibility of the caller, while the
callee now checks more of them. Could you add something at the end
explaining the change and what it enforces?
> Reported-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
With the above change:
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
> kernel/rcu/srcu.c | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
> index ba41a5d04b49..6beeba7b0b67 100644
> --- a/kernel/rcu/srcu.c
> +++ b/kernel/rcu/srcu.c
> @@ -261,6 +261,11 @@ void cleanup_srcu_struct(struct srcu_struct *sp)
> {
> if (WARN_ON(srcu_readers_active(sp)))
> return; /* Leakage unless caller handles error. */
> + if (WARN_ON(!rcu_all_batches_empty(sp)))
> + return; /* Leakage unless caller handles error. */
> + flush_delayed_work(&sp->work);
> + if (WARN_ON(sp->running))
> + return; /* Caller forgot to stop doing call_srcu()? */
> free_percpu(sp->per_cpu_ref);
> sp->per_cpu_ref = NULL;
> }
> --
> 2.5.2
>
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-19 19:00 +0200 |
| Subject | [PATCH v3 tip/core/rcu 0/40] SRCU callback parallelization for 4.12 |
| Message-ID | <ty17r-5PA-11@gated-at.bofh.it> |
| In reply to | #1624925 |
Hello!
This v3 series moves SRCU from its traditional single per-srcu_struct
callback queue to per-srcu_struct/per-CPU callback queues. This involves
abstracting functionality from Tree RCU, which results in a large
conflict footprint, which in turn results in some otherwise unrelated
patches coming along for the ride.
1. Maintain special bits at bottom of ->dynticks counter.
This is for some upcoming MM work. My intent was to hold
it until that work was ready, but merge conflicts dictated
otherwise. If the MM work does not appear soonish, I will
manually revert this patch.
2. Make arch select smp_mb__after_unlock_lock() strength, which
gets rid of an arch-specific #ifdef.
3. Consolidate SRCU batch checking into rcu_all_batches_empty().
4. Check for tardy grace-period activity in cleanup_srcu_struct().
5-7. Semicolon inside RCU_TRACE() for various parts of RCU.
8. Pull rcu_sched_qs_mask into rcu_dynticks structure in order to
eliminate an isolated per-CPU variable.
9. Pull rcu_qs_ctr into rcu_dynticks structure.
10. Eliminate flavor scan in rcu_momentary_dyntick_idle() to
reduce semi-common-case context-switch overhead.
11. Place guard on rcu_all_qs() and rcu_note_context_switch()
actions to reduce common-case scheduler-fastpath overhead.
12. Default RCU_FANOUT_LEAF to 16 unless explicitly changed.
13. Abstract multi-tail callback list handling for SRCU.
14. Allow SRCU to access rcu_scheduler_active.
15. Allow early boot use of synchronize_srcu(), though not yet
mid-boot use.
16. Add single-element dequeue functions to rcu_segcblist for
debug use.
17. Move rcu_seq_start() and friends to rcu.h for SRCU's benefit.
18. Expedited wakeups need to be fully ordered.
19. Fix warning in rcu_seq_end().
20. Push srcu_advance_batches() fastpath into common case as a
step towards callback parallelization.
21. Move to state-based grace-period sequencing, also as a step
towards callback parallelization.
22. Add grace-period sequence numbers to SRCU.
23. Use rcu_segcblist to track SRCU callbacks.
24. Move combining-tree definitions for SRCU's benefit.
25. Move rcu_init_levelspread() to rcu_tree_node.h for SRCU's benefit.
26. Remove redundant levelcnt[] array from rcu_init_one().
27. Move rcu_node traversal macros to rcu.h for SRCU's benefit.
28. Make num_rcu_lvl[] array be external for SRCU's benefit.
29. Fix bogus try_check_zero() comment.
30. Improve rcu_seq grace-period-counter abstraction for SRCU's
benefit.
31. Allow a second bit in rcu_seq for SRCU state.
32. Merge ->srcu_state into ->srcu_gp_seq to allow atomic updates.
33. Provide crude control of expedited SRCU grace periods.
34. Use static initialization for "srcu" in mm/mmu_notifier.c.
35. Create a tiny SRCU for bloatwatch/tinification.
36. Print Tiny SRCU reader statistics in rcutorture.
37. Introduce CLASSIC_SRCU Kconfig option for those who do not
wish to help debug Tree SRCU.
38. Parallelize SRCU callback handling.
39. Make expedited parallel SRCU callback handling really be fully
expedited.
40. Make non-preemptive schedule be Tasks RCU quiescent state.
Updates since v2:
o Apply Josh Triplett feedback.
o Fix a performance regression found by Marc Zyngier.
Updates since v1:
o Incorporate feedback from Peter Zijlstra.
o Dropped v1 patches 8-10 ("Make various parts of RCU do deferred
NOCB wakeups in order to prevent callback blockages, and thus
hangs"). These patches turned out to be papering over a no-CBs
CPU design flaw. There will be patches in v4.13 to fix the design
flaw directly.
o Added v2 patch #34 ("Use static initialization for "srcu" in
mm/mmu_notifier.c"), moving it from its v1 location in the
fixes series.
o Added v2 patch #39 ("Make non-preemptive schedule be Tasks RCU
quiescent state") for the benefit of upcoming ftrace work at
Steve Rostedt's request.
Thanx, Paul
------------------------------------------------------------------------
/kernel/rcu/rcu_segcblist.h | 670 -----
b/Documentation/RCU/Design/Data-Structures/Data-Structures.html | 36
b/arch/Kconfig | 3
b/arch/powerpc/Kconfig | 1
b/include/linux/rcu_node_tree.h | 105
b/include/linux/rcu_segcblist.h | 720 +++++
b/include/linux/rcupdate.h | 17
b/include/linux/rcutiny.h | 24
b/include/linux/rcutree.h | 5
b/include/linux/srcu.h | 112
b/include/linux/srcuclassic.h | 101
b/include/linux/srcutiny.h | 81
b/include/linux/srcutree.h | 171 +
b/init/Kconfig | 33
b/kernel/rcu/Makefile | 6
b/kernel/rcu/rcu.h | 165 +
b/kernel/rcu/rcu_segcblist.h | 670 +++++
b/kernel/rcu/rcutorture.c | 39
b/kernel/rcu/srcu.c | 856 +++---
b/kernel/rcu/srcutiny.c | 215 +
b/kernel/rcu/srcutree.c | 1255 ++++++++--
b/kernel/rcu/tiny.c | 20
b/kernel/rcu/tiny_plugin.h | 13
b/kernel/rcu/tree.c | 661 ++---
b/kernel/rcu/tree.h | 174 -
b/kernel/rcu/tree_exp.h | 25
b/kernel/rcu/tree_plugin.h | 62
b/kernel/rcu/tree_trace.c | 26
b/kernel/rcu/update.c | 53
b/kernel/sched/core.c | 2
b/mm/mmu_notifier.c | 14
31 files changed, 4299 insertions(+), 2036 deletions(-)
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-19 19:00 +0200 |
| Subject | [PATCH v3 tip/core/rcu 35/40] srcu: Create a tiny SRCU |
| Message-ID | <ty17t-5PA-67@gated-at.bofh.it> |
| In reply to | #1626562 |
In response to automated complaints about modifications to SRCU
increasing its size, this commit creates a tiny SRCU that is
used in SMP=n && PREEMPT=n builds.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/srcu.h | 69 ++-------------
include/linux/srcutiny.h | 81 ++++++++++++++++++
include/linux/srcutree.h | 91 ++++++++++++++++++++
init/Kconfig | 12 +++
kernel/rcu/Makefile | 3 +-
kernel/rcu/rcutorture.c | 2 +
kernel/rcu/srcutiny.c | 215 +++++++++++++++++++++++++++++++++++++++++++++++
7 files changed, 411 insertions(+), 62 deletions(-)
create mode 100644 include/linux/srcutiny.h
create mode 100644 include/linux/srcutree.h
create mode 100644 kernel/rcu/srcutiny.c
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index 73a1b6296224..907f09b14eda 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -34,28 +34,7 @@
#include <linux/workqueue.h>
#include <linux/rcu_segcblist.h>
-struct srcu_array {
- unsigned long lock_count[2];
- unsigned long unlock_count[2];
-};
-
-struct srcu_struct {
- unsigned long completed;
- unsigned long srcu_gp_seq;
- atomic_t srcu_exp_cnt;
- struct srcu_array __percpu *per_cpu_ref;
- spinlock_t queue_lock; /* protect ->srcu_cblist */
- struct rcu_segcblist srcu_cblist;
- struct delayed_work work;
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
- struct lockdep_map dep_map;
-#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
-};
-
-/* Values for -> state variable. */
-#define SRCU_STATE_IDLE 0
-#define SRCU_STATE_SCAN1 1
-#define SRCU_STATE_SCAN2 2
+struct srcu_struct;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
@@ -77,42 +56,13 @@ int init_srcu_struct(struct srcu_struct *sp);
#define __SRCU_DEP_MAP_INIT(srcu_name)
#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
-void process_srcu(struct work_struct *work);
-
-#define __SRCU_STRUCT_INIT(name) \
- { \
- .completed = -300, \
- .per_cpu_ref = &name##_srcu_array, \
- .queue_lock = __SPIN_LOCK_UNLOCKED(name.queue_lock), \
- .srcu_cblist = RCU_SEGCBLIST_INITIALIZER(name.srcu_cblist),\
- .work = __DELAYED_WORK_INITIALIZER(name.work, process_srcu, 0),\
- __SRCU_DEP_MAP_INIT(name) \
- }
-
-/*
- * Define and initialize a srcu struct at build time.
- * Do -not- call init_srcu_struct() nor cleanup_srcu_struct() on it.
- *
- * Note that although DEFINE_STATIC_SRCU() hides the name from other
- * files, the per-CPU variable rules nevertheless require that the
- * chosen name be globally unique. These rules also prohibit use of
- * DEFINE_STATIC_SRCU() within a function. If these rules are too
- * restrictive, declare the srcu_struct manually. For example, in
- * each file:
- *
- * static struct srcu_struct my_srcu;
- *
- * Then, before the first use of each my_srcu, manually initialize it:
- *
- * init_srcu_struct(&my_srcu);
- *
- * See include/linux/percpu-defs.h for the rules on per-CPU variables.
- */
-#define __DEFINE_SRCU(name, is_static) \
- static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
- is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
-#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */)
-#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
+#ifdef CONFIG_TINY_SRCU
+#include <linux/srcutiny.h>
+#elif defined(CONFIG_TREE_SRCU)
+#include <linux/srcutree.h>
+#else
+#error "Unknown SRCU implementation specified to kernel configuration"
+#endif
/**
* call_srcu() - Queue a callback for invocation after an SRCU grace period
@@ -138,9 +88,6 @@ void cleanup_srcu_struct(struct srcu_struct *sp);
int __srcu_read_lock(struct srcu_struct *sp) __acquires(sp);
void __srcu_read_unlock(struct srcu_struct *sp, int idx) __releases(sp);
void synchronize_srcu(struct srcu_struct *sp);
-void synchronize_srcu_expedited(struct srcu_struct *sp);
-unsigned long srcu_batches_completed(struct srcu_struct *sp);
-void srcu_barrier(struct srcu_struct *sp);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h
new file mode 100644
index 000000000000..4f284e4f4d8c
--- /dev/null
+++ b/include/linux/srcutiny.h
@@ -0,0 +1,81 @@
+/*
+ * Sleepable Read-Copy Update mechanism for mutual exclusion,
+ * tiny variant.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you can access it online at
+ * http://www.gnu.org/licenses/gpl-2.0.html.
+ *
+ * Copyright (C) IBM Corporation, 2017
+ *
+ * Author: Paul McKenney <paulmck@us.ibm.com>
+ */
+
+#ifndef _LINUX_SRCU_TINY_H
+#define _LINUX_SRCU_TINY_H
+
+#include <linux/swait.h>
+
+struct srcu_struct {
+ int srcu_lock_nesting[2]; /* srcu_read_lock() nesting depth. */
+ struct swait_queue_head srcu_wq;
+ /* Last srcu_read_unlock() wakes GP. */
+ unsigned long srcu_gp_seq; /* GP seq # for callback tagging. */
+ struct rcu_segcblist srcu_cblist;
+ /* Pending SRCU callbacks. */
+ int srcu_idx; /* Current reader array element. */
+ bool srcu_gp_running; /* GP workqueue running? */
+ bool srcu_gp_waiting; /* GP waiting for readers? */
+ struct work_struct srcu_work; /* For driving grace periods. */
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+ struct lockdep_map dep_map;
+#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+};
+
+void srcu_drive_gp(struct work_struct *wp);
+
+#define __SRCU_STRUCT_INIT(name) \
+{ \
+ .srcu_wq = __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \
+ .srcu_cblist = RCU_SEGCBLIST_INITIALIZER(name.srcu_cblist), \
+ .srcu_work = __WORK_INITIALIZER(name.srcu_work, srcu_drive_gp), \
+ __SRCU_DEP_MAP_INIT(name) \
+}
+
+/*
+ * This odd _STATIC_ arrangement is needed for API compatibility with
+ * Tree SRCU, which needs some per-CPU data.
+ */
+#define DEFINE_SRCU(name) \
+ struct srcu_struct name = __SRCU_STRUCT_INIT(name)
+#define DEFINE_STATIC_SRCU(name) \
+ static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
+
+void synchronize_srcu(struct srcu_struct *sp);
+
+static inline void synchronize_srcu_expedited(struct srcu_struct *sp)
+{
+ synchronize_srcu(sp);
+}
+
+static inline void srcu_barrier(struct srcu_struct *sp)
+{
+ synchronize_srcu(sp);
+}
+
+static inline unsigned long srcu_batches_completed(struct srcu_struct *sp)
+{
+ return 0;
+}
+
+#endif
diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h
new file mode 100644
index 000000000000..f2b3bd6c6bc2
--- /dev/null
+++ b/include/linux/srcutree.h
@@ -0,0 +1,91 @@
+/*
+ * Sleepable Read-Copy Update mechanism for mutual exclusion,
+ * tree variant.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you can access it online at
+ * http://www.gnu.org/licenses/gpl-2.0.html.
+ *
+ * Copyright (C) IBM Corporation, 2017
+ *
+ * Author: Paul McKenney <paulmck@us.ibm.com>
+ */
+
+#ifndef _LINUX_SRCU_TREE_H
+#define _LINUX_SRCU_TREE_H
+
+struct srcu_array {
+ unsigned long lock_count[2];
+ unsigned long unlock_count[2];
+};
+
+struct srcu_struct {
+ unsigned long completed;
+ unsigned long srcu_gp_seq;
+ atomic_t srcu_exp_cnt;
+ struct srcu_array __percpu *per_cpu_ref;
+ spinlock_t queue_lock; /* protect ->srcu_cblist */
+ struct rcu_segcblist srcu_cblist;
+ struct delayed_work work;
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+ struct lockdep_map dep_map;
+#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+};
+
+/* Values for -> state variable. */
+#define SRCU_STATE_IDLE 0
+#define SRCU_STATE_SCAN1 1
+#define SRCU_STATE_SCAN2 2
+
+void process_srcu(struct work_struct *work);
+
+#define __SRCU_STRUCT_INIT(name) \
+ { \
+ .completed = -300, \
+ .per_cpu_ref = &name##_srcu_array, \
+ .queue_lock = __SPIN_LOCK_UNLOCKED(name.queue_lock), \
+ .srcu_cblist = RCU_SEGCBLIST_INITIALIZER(name.srcu_cblist),\
+ .work = __DELAYED_WORK_INITIALIZER(name.work, process_srcu, 0),\
+ __SRCU_DEP_MAP_INIT(name) \
+ }
+
+/*
+ * Define and initialize a srcu struct at build time.
+ * Do -not- call init_srcu_struct() nor cleanup_srcu_struct() on it.
+ *
+ * Note that although DEFINE_STATIC_SRCU() hides the name from other
+ * files, the per-CPU variable rules nevertheless require that the
+ * chosen name be globally unique. These rules also prohibit use of
+ * DEFINE_STATIC_SRCU() within a function. If these rules are too
+ * restrictive, declare the srcu_struct manually. For example, in
+ * each file:
+ *
+ * static struct srcu_struct my_srcu;
+ *
+ * Then, before the first use of each my_srcu, manually initialize it:
+ *
+ * init_srcu_struct(&my_srcu);
+ *
+ * See include/linux/percpu-defs.h for the rules on per-CPU variables.
+ */
+#define __DEFINE_SRCU(name, is_static) \
+ static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
+ is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
+#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */)
+#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
+
+void synchronize_srcu_expedited(struct srcu_struct *sp);
+void srcu_barrier(struct srcu_struct *sp);
+unsigned long srcu_batches_completed(struct srcu_struct *sp);
+
+#endif
diff --git a/init/Kconfig b/init/Kconfig
index a92f27da4a27..d269f2ca17b8 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -526,6 +526,18 @@ config SRCU
permits arbitrary sleeping or blocking within RCU read-side critical
sections.
+config TINY_SRCU
+ bool
+ default y if TINY_RCU
+ help
+ This option selects the single-CPU non-preemptible version of SRCU.
+
+config TREE_SRCU
+ bool
+ default y if !TINY_RCU
+ help
+ This option selects the full-fledged version of SRCU.
+
config TASKS_RCU
bool
default n
diff --git a/kernel/rcu/Makefile b/kernel/rcu/Makefile
index 18dfc485225c..b853214a2b99 100644
--- a/kernel/rcu/Makefile
+++ b/kernel/rcu/Makefile
@@ -3,7 +3,8 @@
KCOV_INSTRUMENT := n
obj-y += update.o sync.o
-obj-$(CONFIG_SRCU) += srcu.o
+obj-$(CONFIG_TREE_SRCU) += srcu.o
+obj-$(CONFIG_TINY_SRCU) += srcutiny.o
obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o
obj-$(CONFIG_RCU_PERF_TEST) += rcuperf.o
obj-$(CONFIG_TREE_RCU) += tree.o
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index cccc417a8135..98591e16db1a 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -559,6 +559,7 @@ static void srcu_torture_barrier(void)
static void srcu_torture_stats(void)
{
+#ifdef CONFIG_TREE_SRCU
int cpu;
int idx = srcu_ctlp->completed & 0x1;
@@ -587,6 +588,7 @@ static void srcu_torture_stats(void)
pr_cont(" %d(%ld,%ld)", cpu, c0, c1);
}
pr_cont("\n");
+#endif
}
static void srcu_torture_synchronize_expedited(void)
diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c
new file mode 100644
index 000000000000..b8293527ee18
--- /dev/null
+++ b/kernel/rcu/srcutiny.c
@@ -0,0 +1,215 @@
+/*
+ * Sleepable Read-Copy Update mechanism for mutual exclusion,
+ * tiny version for non-preemptible single-CPU use.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you can access it online at
+ * http://www.gnu.org/licenses/gpl-2.0.html.
+ *
+ * Copyright (C) IBM Corporation, 2017
+ *
+ * Author: Paul McKenney <paulmck@us.ibm.com>
+ */
+
+#include <linux/export.h>
+#include <linux/mutex.h>
+#include <linux/preempt.h>
+#include <linux/rcupdate_wait.h>
+#include <linux/sched.h>
+#include <linux/delay.h>
+#include <linux/srcu.h>
+
+#include <linux/rcu_node_tree.h>
+#include "rcu.h"
+
+static int init_srcu_struct_fields(struct srcu_struct *sp)
+{
+ sp->srcu_lock_nesting[0] = 0;
+ sp->srcu_lock_nesting[1] = 0;
+ init_swait_queue_head(&sp->srcu_wq);
+ sp->srcu_gp_seq = 0;
+ rcu_segcblist_init(&sp->srcu_cblist);
+ sp->srcu_gp_running = false;
+ sp->srcu_gp_waiting = false;
+ sp->srcu_idx = 0;
+ INIT_WORK(&sp->srcu_work, srcu_drive_gp);
+ return 0;
+}
+
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+
+int __init_srcu_struct(struct srcu_struct *sp, const char *name,
+ struct lock_class_key *key)
+{
+ /* Don't re-initialize a lock while it is held. */
+ debug_check_no_locks_freed((void *)sp, sizeof(*sp));
+ lockdep_init_map(&sp->dep_map, name, key, 0);
+ return init_srcu_struct_fields(sp);
+}
+EXPORT_SYMBOL_GPL(__init_srcu_struct);
+
+#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+
+/*
+ * init_srcu_struct - initialize a sleep-RCU structure
+ * @sp: structure to initialize.
+ *
+ * Must invoke this on a given srcu_struct before passing that srcu_struct
+ * to any other function. Each srcu_struct represents a separate domain
+ * of SRCU protection.
+ */
+int init_srcu_struct(struct srcu_struct *sp)
+{
+ return init_srcu_struct_fields(sp);
+}
+EXPORT_SYMBOL_GPL(init_srcu_struct);
+
+#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+
+/*
+ * cleanup_srcu_struct - deconstruct a sleep-RCU structure
+ * @sp: structure to clean up.
+ *
+ * Must invoke this after you are finished using a given srcu_struct that
+ * was initialized via init_srcu_struct(), else you leak memory.
+ */
+void cleanup_srcu_struct(struct srcu_struct *sp)
+{
+ WARN_ON(sp->srcu_lock_nesting[0] || sp->srcu_lock_nesting[1]);
+ flush_work(&sp->srcu_work);
+ WARN_ON(rcu_seq_state(sp->srcu_gp_seq));
+ WARN_ON(sp->srcu_gp_running);
+ WARN_ON(sp->srcu_gp_waiting);
+ WARN_ON(!rcu_segcblist_empty(&sp->srcu_cblist));
+}
+EXPORT_SYMBOL_GPL(cleanup_srcu_struct);
+
+/*
+ * Counts the new reader in the appropriate per-CPU element of the
+ * srcu_struct. Must be called from process context.
+ * Returns an index that must be passed to the matching srcu_read_unlock().
+ */
+int __srcu_read_lock(struct srcu_struct *sp)
+{
+ int idx;
+
+ idx = READ_ONCE(sp->srcu_idx);
+ WRITE_ONCE(sp->srcu_lock_nesting[idx], sp->srcu_lock_nesting[idx] + 1);
+ return idx;
+}
+EXPORT_SYMBOL_GPL(__srcu_read_lock);
+
+/*
+ * Removes the count for the old reader from the appropriate element of
+ * the srcu_struct. Must be called from process context.
+ */
+void __srcu_read_unlock(struct srcu_struct *sp, int idx)
+{
+ int newval = sp->srcu_lock_nesting[idx] - 1;
+
+ WRITE_ONCE(sp->srcu_lock_nesting[idx], newval);
+ if (!newval && READ_ONCE(sp->srcu_gp_waiting))
+ swake_up(&sp->srcu_wq);
+}
+EXPORT_SYMBOL_GPL(__srcu_read_unlock);
+
+/*
+ * Workqueue handler to drive one grace period and invoke any callbacks
+ * that become ready as a result. Single-CPU and !PREEMPT operation
+ * means that we get away with murder on synchronization. ;-)
+ */
+void srcu_drive_gp(struct work_struct *wp)
+{
+ int idx;
+ struct rcu_cblist ready_cbs;
+ struct srcu_struct *sp;
+ struct rcu_head *rhp;
+
+ sp = container_of(wp, struct srcu_struct, srcu_work);
+ if (sp->srcu_gp_running || rcu_segcblist_empty(&sp->srcu_cblist))
+ return; /* Already running or nothing to do. */
+
+ /* Tag recently arrived callbacks and wait for readers. */
+ WRITE_ONCE(sp->srcu_gp_running, true);
+ rcu_segcblist_accelerate(&sp->srcu_cblist,
+ rcu_seq_snap(&sp->srcu_gp_seq));
+ rcu_seq_start(&sp->srcu_gp_seq);
+ idx = sp->srcu_idx;
+ WRITE_ONCE(sp->srcu_idx, !sp->srcu_idx);
+ WRITE_ONCE(sp->srcu_gp_waiting, true); /* srcu_read_unlock() wakes! */
+ swait_event(sp->srcu_wq, !READ_ONCE(sp->srcu_lock_nesting[idx]));
+ WRITE_ONCE(sp->srcu_gp_waiting, false); /* srcu_read_unlock() cheap. */
+ rcu_seq_end(&sp->srcu_gp_seq);
+
+ /* Update callback list based on GP, and invoke ready callbacks. */
+ rcu_segcblist_advance(&sp->srcu_cblist,
+ rcu_seq_current(&sp->srcu_gp_seq));
+ if (rcu_segcblist_ready_cbs(&sp->srcu_cblist)) {
+ rcu_cblist_init(&ready_cbs);
+ local_irq_disable();
+ rcu_segcblist_extract_done_cbs(&sp->srcu_cblist, &ready_cbs);
+ local_irq_enable();
+ rhp = rcu_cblist_dequeue(&ready_cbs);
+ for (; rhp != NULL; rhp = rcu_cblist_dequeue(&ready_cbs)) {
+ local_bh_disable();
+ rhp->func(rhp);
+ local_bh_enable();
+ }
+ local_irq_disable();
+ rcu_segcblist_insert_count(&sp->srcu_cblist, &ready_cbs);
+ local_irq_enable();
+ }
+ WRITE_ONCE(sp->srcu_gp_running, false);
+
+ /*
+ * If more callbacks, reschedule ourselves. This can race with
+ * a call_srcu() at interrupt level, but the ->srcu_gp_running
+ * checks will straighten that out.
+ */
+ if (!rcu_segcblist_empty(&sp->srcu_cblist))
+ schedule_work(&sp->srcu_work);
+}
+EXPORT_SYMBOL_GPL(srcu_drive_gp);
+
+/*
+ * Enqueue an SRCU callback on the specified srcu_struct structure,
+ * initiating grace-period processing if it is not already running.
+ */
+void call_srcu(struct srcu_struct *sp, struct rcu_head *head,
+ rcu_callback_t func)
+{
+ unsigned long flags;
+
+ head->func = func;
+ local_irq_save(flags);
+ rcu_segcblist_enqueue(&sp->srcu_cblist, head, false);
+ local_irq_restore(flags);
+ if (!READ_ONCE(sp->srcu_gp_running))
+ schedule_work(&sp->srcu_work);
+}
+EXPORT_SYMBOL_GPL(call_srcu);
+
+/*
+ * synchronize_srcu - wait for prior SRCU read-side critical-section completion
+ */
+void synchronize_srcu(struct srcu_struct *sp)
+{
+ struct rcu_synchronize rs;
+
+ init_rcu_head_on_stack(&rs.head);
+ init_completion(&rs.completion);
+ call_srcu(sp, &rs.head, wakeme_after_rcu);
+ wait_for_completion(&rs.completion);
+ destroy_rcu_head_on_stack(&rs.head);
+}
+EXPORT_SYMBOL_GPL(synchronize_srcu);
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-19 19:00 +0200 |
| Subject | [PATCH v3 tip/core/rcu 12/40] rcu: Default RCU_FANOUT_LEAF to 16 unless explicitly changed |
| Message-ID | <ty17t-5PA-71@gated-at.bofh.it> |
| In reply to | #1626562 |
If the RCU_EXPERT Kconfig option is not set (the default), then the RCU_FANOUT_LEAF Kconfig option will not be defined, which will cause the leaf-level rcu_node tree fanout to default to 32 on 32-bit systems and 64 on 64-bit systems. This can result in excessive lock contention. This commit therefore changes the computation of the leaf-level rcu_node tree fanout so that the result will be 16 unless an explicit Kconfig or kernel-boot setting says otherwise. Reported-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> --- kernel/rcu/tree.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index d2f276fc2edc..376c01e539c7 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -52,11 +52,7 @@ #ifdef CONFIG_RCU_FANOUT_LEAF #define RCU_FANOUT_LEAF CONFIG_RCU_FANOUT_LEAF #else /* #ifdef CONFIG_RCU_FANOUT_LEAF */ -# ifdef CONFIG_64BIT -# define RCU_FANOUT_LEAF 64 -# else -# define RCU_FANOUT_LEAF 32 -# endif +#define RCU_FANOUT_LEAF 16 #endif /* #else #ifdef CONFIG_RCU_FANOUT_LEAF */ #define RCU_FANOUT_1 (RCU_FANOUT_LEAF) -- 2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-19 19:10 +0200 |
| Subject | [PATCH v3 tip/core/rcu 05/40] rcu: Semicolon inside RCU_TRACE() for rcu.h |
| Message-ID | <ty1h7-68e-1@gated-at.bofh.it> |
| In reply to | #1626562 |
The current use of "RCU_TRACE(statement);" can cause odd bugs, especially
where "statement" is a local-variable declaration, as it can leave a
misplaced ";" in the source code. This commit therefore converts these
to "RCU_TRACE(statement;)", which avoids the misplaced ";".
Reported-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/rcu.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
index 0d6ff3e471be..8700a81daf56 100644
--- a/kernel/rcu/rcu.h
+++ b/kernel/rcu/rcu.h
@@ -109,12 +109,12 @@ static inline bool __rcu_reclaim(const char *rn, struct rcu_head *head)
rcu_lock_acquire(&rcu_callback_map);
if (__is_kfree_rcu_offset(offset)) {
- RCU_TRACE(trace_rcu_invoke_kfree_callback(rn, head, offset));
+ RCU_TRACE(trace_rcu_invoke_kfree_callback(rn, head, offset);)
kfree((void *)head - offset);
rcu_lock_release(&rcu_callback_map);
return true;
} else {
- RCU_TRACE(trace_rcu_invoke_callback(rn, head));
+ RCU_TRACE(trace_rcu_invoke_callback(rn, head);)
head->func(head);
rcu_lock_release(&rcu_callback_map);
return false;
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | Joe Perches <joe@perches.com> |
|---|---|
| Date | 2017-04-19 19:50 +0200 |
| Subject | Re: [PATCH v3 tip/core/rcu 05/40] rcu: Semicolon inside RCU_TRACE() for rcu.h |
| Message-ID | <ty1TP-6kT-7@gated-at.bofh.it> |
| In reply to | #1626572 |
On Wed, 2017-04-19 at 09:58 -0700, Paul E. McKenney wrote:
> The current use of "RCU_TRACE(statement);" can cause odd bugs, especially
> where "statement" is a local-variable declaration, as it can leave a
> misplaced ";" in the source code. This commit therefore converts these
> to "RCU_TRACE(statement;)", which avoids the misplaced ";".
Perhaps instead the single RCU_TRACE macro should be split
into RCU_TRACE_DECLARE and RCU_TRACE, and then RCU_TRACE
should use do {} while (0)
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-19 19:10 +0200 |
| Subject | [PATCH v3 tip/core/rcu 26/40] rcu: Remove redundant levelcnt[] array from rcu_init_one() |
| Message-ID | <ty1h7-68e-17@gated-at.bofh.it> |
| In reply to | #1626562 |
The levelcnt[] array is identical to num_rcu_lvl[], so this commit
removes levelcnt[].
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/tree.c | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index df3527744af8..9397a6693d70 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3961,7 +3961,6 @@ static void __init rcu_init_one(struct rcu_state *rsp)
static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS];
- int levelcnt[RCU_NUM_LVLS]; /* # nodes in each level. */
int levelspread[RCU_NUM_LVLS]; /* kids/node in each level. */
int cpustride = 1;
int i;
@@ -3976,18 +3975,16 @@ static void __init rcu_init_one(struct rcu_state *rsp)
/* Initialize the level-tracking arrays. */
- for (i = 0; i < rcu_num_lvls; i++)
- levelcnt[i] = num_rcu_lvl[i];
for (i = 1; i < rcu_num_lvls; i++)
- rsp->level[i] = rsp->level[i - 1] + levelcnt[i - 1];
- rcu_init_levelspread(levelspread, levelcnt);
+ rsp->level[i] = rsp->level[i - 1] + num_rcu_lvl[i - 1];
+ rcu_init_levelspread(levelspread, num_rcu_lvl);
/* Initialize the elements themselves, starting from the leaves. */
for (i = rcu_num_lvls - 1; i >= 0; i--) {
cpustride *= levelspread[i];
rnp = rsp->level[i];
- for (j = 0; j < levelcnt[i]; j++, rnp++) {
+ for (j = 0; j < num_rcu_lvl[i]; j++, rnp++) {
raw_spin_lock_init(&ACCESS_PRIVATE(rnp, lock));
lockdep_set_class_and_name(&ACCESS_PRIVATE(rnp, lock),
&rcu_node_class[i], buf[i]);
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-19 19:10 +0200 |
| Subject | [PATCH v3 tip/core/rcu 14/40] srcu: Allow SRCU to access rcu_scheduler_active |
| Message-ID | <ty1h7-68e-9@gated-at.bofh.it> |
| In reply to | #1626562 |
This is primarily a code-movement commit in preparation for allowing
SRCU to handle early-boot SRCU grace periods.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcutiny.h | 6 +++---
kernel/rcu/tiny_plugin.h | 9 +++++----
kernel/rcu/tree.c | 2 +-
kernel/rcu/tree_exp.h | 12 -----------
kernel/rcu/update.c | 52 +++++++++++++++++++++++++++++++-----------------
5 files changed, 43 insertions(+), 38 deletions(-)
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index 6c9d941e3962..5219be250f00 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -217,14 +217,14 @@ static inline void exit_rcu(void)
{
}
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
+#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU)
extern int rcu_scheduler_active __read_mostly;
void rcu_scheduler_starting(void);
-#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+#else /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
static inline void rcu_scheduler_starting(void)
{
}
-#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+#endif /* #else #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE)
diff --git a/kernel/rcu/tiny_plugin.h b/kernel/rcu/tiny_plugin.h
index df3a60e19f07..371034e77f87 100644
--- a/kernel/rcu/tiny_plugin.h
+++ b/kernel/rcu/tiny_plugin.h
@@ -52,7 +52,7 @@ static struct rcu_ctrlblk rcu_bh_ctrlblk = {
RCU_TRACE(.name = "rcu_bh")
};
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
+#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU)
#include <linux/kernel_stat.h>
int rcu_scheduler_active __read_mostly;
@@ -65,15 +65,16 @@ EXPORT_SYMBOL_GPL(rcu_scheduler_active);
* to RCU_SCHEDULER_RUNNING, skipping the RCU_SCHEDULER_INIT stage.
* The reason for this is that Tiny RCU does not need kthreads, so does
* not have to care about the fact that the scheduler is half-initialized
- * at a certain phase of the boot process.
+ * at a certain phase of the boot process. Unless SRCU is in the mix.
*/
void __init rcu_scheduler_starting(void)
{
WARN_ON(nr_context_switches() > 0);
- rcu_scheduler_active = RCU_SCHEDULER_RUNNING;
+ rcu_scheduler_active = IS_ENABLED(CONFIG_SRCU)
+ ? RCU_SCHEDULER_INIT : RCU_SCHEDULER_RUNNING;
}
-#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+#endif /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
#ifdef CONFIG_RCU_TRACE
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 8cc9d40b41ea..2380d1e3dfb8 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3974,7 +3974,7 @@ early_initcall(rcu_spawn_gp_kthread);
* task is booting the system, and such primitives are no-ops). After this
* function is called, any synchronous grace-period primitives are run as
* expedited, with the requesting task driving the grace period forward.
- * A later core_initcall() rcu_exp_runtime_mode() will switch to full
+ * A later core_initcall() rcu_set_runtime_mode() will switch to full
* runtime RCU functionality.
*/
void rcu_scheduler_starting(void)
diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index a1f52bbe9db6..51ca287828a2 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -737,15 +737,3 @@ void synchronize_rcu_expedited(void)
EXPORT_SYMBOL_GPL(synchronize_rcu_expedited);
#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
-
-/*
- * Switch to run-time mode once Tree RCU has fully initialized.
- */
-static int __init rcu_exp_runtime_mode(void)
-{
- rcu_test_sync_prims();
- rcu_scheduler_active = RCU_SCHEDULER_RUNNING;
- rcu_test_sync_prims();
- return 0;
-}
-core_initcall(rcu_exp_runtime_mode);
diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index 55c8530316c7..c5df0d756900 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -124,7 +124,7 @@ EXPORT_SYMBOL(rcu_read_lock_sched_held);
* non-expedited counterparts? Intended for use within RCU. Note
* that if the user specifies both rcu_expedited and rcu_normal, then
* rcu_normal wins. (Except during the time period during boot from
- * when the first task is spawned until the rcu_exp_runtime_mode()
+ * when the first task is spawned until the rcu_set_runtime_mode()
* core_initcall() is invoked, at which point everything is expedited.)
*/
bool rcu_gp_is_normal(void)
@@ -190,6 +190,39 @@ void rcu_end_inkernel_boot(void)
#endif /* #ifndef CONFIG_TINY_RCU */
+/*
+ * Test each non-SRCU synchronous grace-period wait API. This is
+ * useful just after a change in mode for these primitives, and
+ * during early boot.
+ */
+void rcu_test_sync_prims(void)
+{
+ if (!IS_ENABLED(CONFIG_PROVE_RCU))
+ return;
+ synchronize_rcu();
+ synchronize_rcu_bh();
+ synchronize_sched();
+ synchronize_rcu_expedited();
+ synchronize_rcu_bh_expedited();
+ synchronize_sched_expedited();
+}
+
+#if !defined(CONFIG_TINY_RCU) || defined(CONFIG_SRCU)
+
+/*
+ * Switch to run-time mode once RCU has fully initialized.
+ */
+static int __init rcu_set_runtime_mode(void)
+{
+ rcu_test_sync_prims();
+ rcu_scheduler_active = RCU_SCHEDULER_RUNNING;
+ rcu_test_sync_prims();
+ return 0;
+}
+core_initcall(rcu_set_runtime_mode);
+
+#endif /* #if !defined(CONFIG_TINY_RCU) || defined(CONFIG_SRCU) */
+
#ifdef CONFIG_PREEMPT_RCU
/*
@@ -817,23 +850,6 @@ static void rcu_spawn_tasks_kthread(void)
#endif /* #ifdef CONFIG_TASKS_RCU */
-/*
- * Test each non-SRCU synchronous grace-period wait API. This is
- * useful just after a change in mode for these primitives, and
- * during early boot.
- */
-void rcu_test_sync_prims(void)
-{
- if (!IS_ENABLED(CONFIG_PROVE_RCU))
- return;
- synchronize_rcu();
- synchronize_rcu_bh();
- synchronize_sched();
- synchronize_rcu_expedited();
- synchronize_rcu_bh_expedited();
- synchronize_sched_expedited();
-}
-
#ifdef CONFIG_PROVE_RCU
/*
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-19 19:10 +0200 |
| Subject | [PATCH v3 tip/core/rcu 27/40] srcu: Move rcu_node traversal macros to rcu.h |
| Message-ID | <ty1h7-68e-15@gated-at.bofh.it> |
| In reply to | #1626562 |
This commit moves rcu_for_each_node_breadth_first(),
rcu_for_each_nonleaf_node_breadth_first(), and
rcu_for_each_leaf_node() from kernel/rcu/tree.h to
kernel/rcu/rcu.h so that SRCU can access them.
This commit is code-movement only.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/rcu.h | 35 +++++++++++++++++++++++++++++++++++
kernel/rcu/tree.h | 35 -----------------------------------
2 files changed, 35 insertions(+), 35 deletions(-)
diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
index 87326479b39a..5c1798ec32f7 100644
--- a/kernel/rcu/rcu.h
+++ b/kernel/rcu/rcu.h
@@ -224,6 +224,41 @@ static inline void rcu_init_levelspread(int *levelspread, const int *levelcnt)
}
}
+/*
+ * Do a full breadth-first scan of the rcu_node structures for the
+ * specified rcu_state structure.
+ */
+#define rcu_for_each_node_breadth_first(rsp, rnp) \
+ for ((rnp) = &(rsp)->node[0]; \
+ (rnp) < &(rsp)->node[rcu_num_nodes]; (rnp)++)
+
+/*
+ * Do a breadth-first scan of the non-leaf rcu_node structures for the
+ * specified rcu_state structure. Note that if there is a singleton
+ * rcu_node tree with but one rcu_node structure, this loop is a no-op.
+ */
+#define rcu_for_each_nonleaf_node_breadth_first(rsp, rnp) \
+ for ((rnp) = &(rsp)->node[0]; \
+ (rnp) < (rsp)->level[rcu_num_lvls - 1]; (rnp)++)
+
+/*
+ * Scan the leaves of the rcu_node hierarchy for the specified rcu_state
+ * structure. Note that if there is a singleton rcu_node tree with but
+ * one rcu_node structure, this loop -will- visit the rcu_node structure.
+ * It is still a leaf node, even if it is also the root node.
+ */
+#define rcu_for_each_leaf_node(rsp, rnp) \
+ for ((rnp) = (rsp)->level[rcu_num_lvls - 1]; \
+ (rnp) < &(rsp)->node[rcu_num_nodes]; (rnp)++)
+
+/*
+ * Iterate over all possible CPUs in a leaf RCU node.
+ */
+#define for_each_leaf_node_possible_cpu(rnp, cpu) \
+ for ((cpu) = cpumask_next(rnp->grplo - 1, cpu_possible_mask); \
+ cpu <= rnp->grphi; \
+ cpu = cpumask_next((cpu), cpu_possible_mask))
+
#endif /* #if defined(SRCU) || !defined(TINY_RCU) */
#endif /* __LINUX_RCU_H */
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 1bec3958d44f..a2a45cb629d6 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -193,41 +193,6 @@ struct rcu_node {
#define leaf_node_cpu_bit(rnp, cpu) (1UL << ((cpu) - (rnp)->grplo))
/*
- * Do a full breadth-first scan of the rcu_node structures for the
- * specified rcu_state structure.
- */
-#define rcu_for_each_node_breadth_first(rsp, rnp) \
- for ((rnp) = &(rsp)->node[0]; \
- (rnp) < &(rsp)->node[rcu_num_nodes]; (rnp)++)
-
-/*
- * Do a breadth-first scan of the non-leaf rcu_node structures for the
- * specified rcu_state structure. Note that if there is a singleton
- * rcu_node tree with but one rcu_node structure, this loop is a no-op.
- */
-#define rcu_for_each_nonleaf_node_breadth_first(rsp, rnp) \
- for ((rnp) = &(rsp)->node[0]; \
- (rnp) < (rsp)->level[rcu_num_lvls - 1]; (rnp)++)
-
-/*
- * Scan the leaves of the rcu_node hierarchy for the specified rcu_state
- * structure. Note that if there is a singleton rcu_node tree with but
- * one rcu_node structure, this loop -will- visit the rcu_node structure.
- * It is still a leaf node, even if it is also the root node.
- */
-#define rcu_for_each_leaf_node(rsp, rnp) \
- for ((rnp) = (rsp)->level[rcu_num_lvls - 1]; \
- (rnp) < &(rsp)->node[rcu_num_nodes]; (rnp)++)
-
-/*
- * Iterate over all possible CPUs in a leaf RCU node.
- */
-#define for_each_leaf_node_possible_cpu(rnp, cpu) \
- for ((cpu) = cpumask_next(rnp->grplo - 1, cpu_possible_mask); \
- cpu <= rnp->grphi; \
- cpu = cpumask_next((cpu), cpu_possible_mask))
-
-/*
* Union to allow "aggregate OR" operation on the need for a quiescent
* state by the normal and expedited grace periods.
*/
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-19 19:10 +0200 |
| Subject | [PATCH v3 tip/core/rcu 19/40] rcu: Fix warning in rcu_seq_end() |
| Message-ID | <ty1h7-68e-13@gated-at.bofh.it> |
| In reply to | #1626562 |
From: Dmitry Vyukov <dvyukov@google.com>
The rcu_seq_end() function increments seq signifying completion
of a grace period, after that checks that the seq is even and wakes
_synchronize_rcu_expedited(). The _synchronize_rcu_expedited() function
uses wait_event() to wait for even seq. The problem is that wait_event()
can return as soon as seq becomes even without waiting for the wakeup.
In such case the warning in rcu_seq_end() can falsely fire if the next
expedited grace period starts before the check.
Check that seq has good value before incrementing it.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: syzkaller@googlegroups.com
Cc: linux-kernel@vger.kernel.org
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: josh@joshtriplett.org
Cc: jiangshanlai@gmail.com
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
syzkaller-triggered warning:
WARNING: CPU: 0 PID: 4832 at kernel/rcu/tree.c:3533
rcu_seq_end+0x110/0x140 kernel/rcu/tree.c:3533
CPU: 0 PID: 4832 Comm: kworker/0:3 Not tainted 4.10.0+ #276
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Workqueue: events wait_rcu_exp_gp
Call Trace:
__dump_stack lib/dump_stack.c:15 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
panic+0x1fb/0x412 kernel/panic.c:179
__warn+0x1c4/0x1e0 kernel/panic.c:540
warn_slowpath_null+0x2c/0x40 kernel/panic.c:583
rcu_seq_end+0x110/0x140 kernel/rcu/tree.c:3533
rcu_exp_gp_seq_end kernel/rcu/tree_exp.h:36 [inline]
rcu_exp_wait_wake+0x8a9/0x1330 kernel/rcu/tree_exp.h:517
rcu_exp_sel_wait_wake kernel/rcu/tree_exp.h:559 [inline]
wait_rcu_exp_gp+0x83/0xc0 kernel/rcu/tree_exp.h:570
process_one_work+0xc06/0x1c20 kernel/workqueue.c:2096
worker_thread+0x223/0x19c0 kernel/workqueue.c:2230
kthread+0x326/0x3f0 kernel/kthread.c:227
ret_from_fork+0x31/0x40 arch/x86/entry/entry_64.S:430
---
---
kernel/rcu/rcu.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
index 91e0bf31f6ce..0bc1313c49e2 100644
--- a/kernel/rcu/rcu.h
+++ b/kernel/rcu/rcu.h
@@ -73,8 +73,8 @@ static inline void rcu_seq_start(unsigned long *sp)
static inline void rcu_seq_end(unsigned long *sp)
{
smp_mb(); /* Ensure update-side operation before counter increment. */
+ WARN_ON_ONCE(!(*sp & 0x1));
WRITE_ONCE(*sp, *sp + 1);
- WARN_ON_ONCE(*sp & 0x1);
}
/* Take a snapshot of the update side's sequence number. */
--
2.5.2
[toc] | [prev] | [next] | [standalone]
Page 3 of 5 — ← Prev page 1 2 [3] 4 5 Next page →
Back to top | Article view | linux.kernel
csiph-web