Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1622453 > unrolled thread
| Started by | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| First post | 2017-04-12 19:50 +0200 |
| Last post | 2017-04-21 04:20 +0200 |
| Articles | 20 on this page of 93 — 6 participants |
Back to article view | Back to linux.kernel
[PATCH tip/core/rcu 0/40] SRCU callback parallelization for 4.12 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 19:50 +0200
[PATCH tip/core/rcu 03/40] srcu: Consolidate batch checking into rcu_all_batches_empty() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 20:00 +0200
[PATCH tip/core/rcu 05/40] rcu: Semicolon inside RCU_TRACE() for rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 20:00 +0200
[PATCH tip/core/rcu 11/40] rcu: Pull rcu_sched_qs_mask into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-12 20:00 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling Peter Zijlstra <peterz@infradead.org> - 2017-04-13 12:00 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-13 18:40 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-13 19:50 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling Peter Zijlstra <peterz@infradead.org> - 2017-04-13 12:00 +0200
Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-13 19:00 +0200
[PATCH v2 tip/core/rcu 0/40] SRCU callback parallelization for 4.12 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 02/39] rcu: Make arch select smp_mb__after_unlock_lock() strength "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
Re: [PATCH v2 tip/core/rcu 02/39] rcu: Make arch select smp_mb__after_unlock_lock() strength Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:20 +0200
[PATCH v2 tip/core/rcu 10/39] rcu: Eliminate flavor scan in rcu_momentary_dyntick_idle() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 24/39] srcu: Move combining-tree definitions for SRCU's benefit "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 31/39] srcu: Allow a second bit in rcu_seq for SRCU state "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 15/39] srcu: Allow early boot use of synchronize_srcu() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 14/39] srcu: Allow SRCU to access rcu_scheduler_active "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 05/39] rcu: Semicolon inside RCU_TRACE() for rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 33/39] srcu: Crude control of expedited grace periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 29/39] srcu: Fix bogus try_check_zero() comment "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 25/39] srcu: Move rcu_init_levelspread() to rcu_tree_node.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 27/39] srcu: Move rcu_node traversal macros to rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 06/39] rcu: Semicolon inside RCU_TRACE() for Tiny RCU "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 39/39] rcu: Make non-preemptive schedule be Tasks RCU quiescent state "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 07/39] rcu: Semicolon inside RCU_TRACE() for tree.c "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 18/39] rcu: Expedited wakeups need to be fully ordered "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 21/39] srcu: Move to state-based grace-period sequencing "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 22/39] srcu: Add grace-period sequence numbers "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 34/39] mm: Use static initialization for "srcu" "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 35/39] srcu: Create a tiny SRCU "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 16/39] rcu: Add single-element dequeue functions to rcu_segcblist "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 01:50 +0200
[PATCH v2 tip/core/rcu 19/39] rcu: Fix warning in rcu_seq_end() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 03/39] srcu: Consolidate batch checking into rcu_all_batches_empty() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
Re: [PATCH v2 tip/core/rcu 03/39] srcu: Consolidate batch checking into rcu_all_batches_empty() Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:40 +0200
[PATCH v2 tip/core/rcu 08/39] rcu: Pull rcu_sched_qs_mask into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 09/39] rcu: Pull rcu_qs_ctr into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 30/39] srcu: Improve rcu_seq grace-period-counter abstraction "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 17/39] srcu: Move rcu_seq_start() and friends to rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 11/39] rcu: Place guard on rcu_all_qs() and rcu_note_context_switch() actions "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 28/39] srcu: Make num_rcu_lvl[] array be external "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 26/39] rcu: Remove redundant levelcnt[] array from rcu_init_one() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 32/39] srcu: Merge ->srcu_state into ->srcu_gp_seq "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 36/39] srcutorture: Print Tiny SRCU reader statistics "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
[PATCH v2 tip/core/rcu 01/39] rcu: Maintain special bits at bottom of ->dynticks counter "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
Re: [PATCH v2 tip/core/rcu 01/39] rcu: Maintain special bits at bottom of ->dynticks counter Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:10 +0200
Re: [PATCH v2 tip/core/rcu 01/39] rcu: Maintain special bits at bottom of ->dynticks counter "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 20:30 +0200
[PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 02:00 +0200
Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:40 +0200
Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-18 20:40 +0200
Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() Josh Triplett <josh@joshtriplett.org> - 2017-04-18 21:50 +0200
Re: [PATCH v2 tip/core/rcu 04/39] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() Josh Triplett <josh@joshtriplett.org> - 2017-04-18 02:40 +0200
[PATCH v3 tip/core/rcu 0/40] SRCU callback parallelization for 4.12 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:00 +0200
[PATCH v3 tip/core/rcu 35/40] srcu: Create a tiny SRCU "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:00 +0200
[PATCH v3 tip/core/rcu 12/40] rcu: Default RCU_FANOUT_LEAF to 16 unless explicitly changed "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:00 +0200
[PATCH v3 tip/core/rcu 05/40] rcu: Semicolon inside RCU_TRACE() for rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
Re: [PATCH v3 tip/core/rcu 05/40] rcu: Semicolon inside RCU_TRACE() for rcu.h Joe Perches <joe@perches.com> - 2017-04-19 19:50 +0200
[PATCH v3 tip/core/rcu 26/40] rcu: Remove redundant levelcnt[] array from rcu_init_one() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 14/40] srcu: Allow SRCU to access rcu_scheduler_active "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 27/40] srcu: Move rcu_node traversal macros to rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 19/40] rcu: Fix warning in rcu_seq_end() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 29/40] srcu: Fix bogus try_check_zero() comment "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 28/40] srcu: Make num_rcu_lvl[] array be external "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 22/40] srcu: Add grace-period sequence numbers "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 39/40] srcu: Expedite srcu_schedule_cbs_snp() callback invocation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 30/40] srcu: Improve rcu_seq grace-period-counter abstraction "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 01/40] rcu: Maintain special bits at bottom of ->dynticks counter "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 20/40] srcu: Push srcu_advance_batches() fastpath into common case "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 34/40] mm: Use static initialization for "srcu" "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 17/40] srcu: Move rcu_seq_start() and friends to rcu.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 08/40] rcu: Pull rcu_sched_qs_mask into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 36/40] srcutorture: Print Tiny SRCU reader statistics "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 09/40] rcu: Pull rcu_qs_ctr into rcu_dynticks structure "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 04/40] srcu: Check for tardy grace-period activity in cleanup_srcu_struct() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 07/40] rcu: Semicolon inside RCU_TRACE() for tree.c "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 10/40] rcu: Eliminate flavor scan in rcu_momentary_dyntick_idle() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 02/40] rcu: Make arch select smp_mb__after_unlock_lock() strength "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 24/40] srcu: Move combining-tree definitions for SRCU's benefit "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 15/40] srcu: Allow early boot use of synchronize_srcu() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 33/40] srcu: Crude control of expedited grace periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 06/40] rcu: Semicolon inside RCU_TRACE() for Tiny RCU "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 03/40] srcu: Consolidate batch checking into rcu_all_batches_empty() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 32/40] srcu: Merge ->srcu_state into ->srcu_gp_seq "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
[PATCH v3 tip/core/rcu 25/40] srcu: Move rcu_init_levelspread() to rcu_tree_node.h "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-19 19:10 +0200
powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Michael Ellerman <michaele@au1.ibm.com> - 2017-04-20 05:50 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-20 16:30 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Paolo Bonzini <pbonzini@redhat.com> - 2017-04-20 17:30 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-21 02:40 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Michael Ellerman <michaele@au1.ibm.com> - 2017-04-21 03:50 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-21 06:20 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Paolo Bonzini <pbonzini@redhat.com> - 2017-04-21 09:30 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-04-21 15:00 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Michael Ellerman <michaele@au1.ibm.com> - 2017-04-22 08:20 +0200
Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Michael Ellerman <michaele@au1.ibm.com> - 2017-04-21 04:20 +0200
Page 1 of 5 [1] 2 3 4 5 Next page →
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-12 19:50 +0200 |
| Subject | [PATCH tip/core/rcu 0/40] SRCU callback parallelization for 4.12 |
| Message-ID | <tvuyZ-7rV-5@gated-at.bofh.it> |
Hello! This series moves SRCU from its traditional single per-srcu_struct callback queue to per-srcu_struct/per-CPU callback queues. This involves abstracting functionality from Tree RCU, which results in a large conflict footprint, which in turn results in some otherwise unrelated patches coming along for the ride. 1. Maintain special bits at bottom of ->dynticks counter. This is for some upcoming MM work. My intent was to hold it until that work was ready, but merge conflicts dictated otherwise. If the MM work does not appear soonish, I will manually revert this patch. 2. Make arch select smp_mb__after_unlock_lock() strength, which gets rid of an arch-specific #ifdef. 3. Consolidate SRCU batch checking into rcu_all_batches_empty(). 4. Check for tardy grace-period activity in cleanup_srcu_struct(). 5-7. Semicolon inside RCU_TRACE() for various parts of RCU. 8-10. Make various parts of RCU do deferred NOCB wakeups in order to prevent callback blockages, and thus hangs. 11. Pull rcu_sched_qs_mask into rcu_dynticks structure in order to eliminate an isolated per-CPU variable. 12. Pull rcu_qs_ctr into rcu_dynticks structure. 13. Eliminate flavor scan in rcu_momentary_dyntick_idle() to reduce semi-common-case context-switch overhead. 14. Place guard on rcu_all_qs() and rcu_note_context_switch() actions to reduce common-case scheduler-fastpath overhead. 15. Default RCU_FANOUT_LEAF to 16 unless explicitly changed. 16. Abstract multi-tail callback list handling for SRCU. 17. Allow SRCU to access rcu_scheduler_active. 18. Allow early boot use of synchronize_srcu(), though not yet mid-boot use. 19. Add single-element dequeue functions to rcu_segcblist for debug use. 20. Move rcu_seq_start() and friends to rcu.h for SRCU's benefit. 21. Expedited wakeups need to be fully ordered. 22. Fix warning in rcu_seq_end(). 23. Push srcu_advance_batches() fastpath into common case as a step towards callback parallelization. 24. Move to state-based grace-period sequencing, also as a step towards callback parallelization. 25. Add grace-period sequence numbers to SRCU. 26. Use rcu_segcblist to track SRCU callbacks. 27. Move combining-tree definitions for SRCU's benefit. 28. Move rcu_init_levelspread() to rcu_tree_node.h for SRCU's benefit. 29. Remove redundant levelcnt[] array from rcu_init_one(). 30. Move rcu_node traversal macros to rcu.h for SRCU's benefit. 31. Make num_rcu_lvl[] array be external for SRCU's benefit. 32. Fix bogus try_check_zero() comment. 33. Improve rcu_seq grace-period-counter abstraction for SRCU's benefit. 34. Allow a second bit in rcu_seq for SRCU state. 35. Merge ->srcu_state into ->srcu_gp_seq to allow atomic updates. 36. Provide crude control of expedited SRCU grace periods. 37. Create a tiny SRCU for bloatwatch/tinification. 38. Print Tiny SRCU reader statistics in rcutorture. 39. Introduce CLASSIC_SRCU Kconfig option for those who do not wish to help debug Tree SRCU. 40. Parallelize SRCU callback handling. Thanx, Paul ------------------------------------------------------------------------ /kernel/rcu/rcu_segcblist.h | 671 ----- b/Documentation/RCU/Design/Data-Structures/Data-Structures.html | 36 b/arch/Kconfig | 3 b/arch/powerpc/Kconfig | 1 b/include/linux/rcu_node_tree.h | 105 b/include/linux/rcu_segcblist.h | 720 +++++ b/include/linux/rcupdate.h | 6 b/include/linux/rcutiny.h | 11 b/include/linux/srcu.h | 112 b/include/linux/srcuclassic.h | 101 b/include/linux/srcutiny.h | 81 b/include/linux/srcutree.h | 171 + b/init/Kconfig | 33 b/kernel/rcu/Makefile | 6 b/kernel/rcu/rcu.h | 165 + b/kernel/rcu/rcu_segcblist.h | 671 +++++ b/kernel/rcu/rcutorture.c | 39 b/kernel/rcu/srcu.c | 846 +++--- b/kernel/rcu/srcutiny.c | 215 + b/kernel/rcu/srcutree.c | 1252 ++++++++-- b/kernel/rcu/tiny.c | 20 b/kernel/rcu/tiny_plugin.h | 13 b/kernel/rcu/tree.c | 650 +---- b/kernel/rcu/tree.h | 174 - b/kernel/rcu/tree_exp.h | 25 b/kernel/rcu/tree_plugin.h | 70 b/kernel/rcu/tree_trace.c | 26 b/kernel/rcu/update.c | 52 28 files changed, 4261 insertions(+), 2014 deletions(-)
[toc] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-12 20:00 +0200 |
| Subject | [PATCH tip/core/rcu 03/40] srcu: Consolidate batch checking into rcu_all_batches_empty() |
| Message-ID | <tvuIF-7vr-1@gated-at.bofh.it> |
| In reply to | #1622453 |
The srcu_reschedule() function invokes rcu_batch_empty() on each of
the four rcu_batch structures in the srcu_struct in question twice.
Given that this check will also be needed in cleanup_srcu_struct(), this
commit consolidates these four checks into a new rcu_all_batches_empty()
function.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/srcu.c | 21 +++++++++++++--------
1 file changed, 13 insertions(+), 8 deletions(-)
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index ef3bcfb15b39..ba41a5d04b49 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -65,6 +65,17 @@ static inline bool rcu_batch_empty(struct rcu_batch *b)
}
/*
+ * Are all batches empty for the specified srcu_struct?
+ */
+static inline bool rcu_all_batches_empty(struct srcu_struct *sp)
+{
+ return rcu_batch_empty(&sp->batch_done) &&
+ rcu_batch_empty(&sp->batch_check1) &&
+ rcu_batch_empty(&sp->batch_check0) &&
+ rcu_batch_empty(&sp->batch_queue);
+}
+
+/*
* Remove the callback at the head of the specified rcu_batch structure
* and return a pointer to it, or return NULL if the structure is empty.
*/
@@ -619,15 +630,9 @@ static void srcu_reschedule(struct srcu_struct *sp)
{
bool pending = true;
- if (rcu_batch_empty(&sp->batch_done) &&
- rcu_batch_empty(&sp->batch_check1) &&
- rcu_batch_empty(&sp->batch_check0) &&
- rcu_batch_empty(&sp->batch_queue)) {
+ if (rcu_all_batches_empty(sp)) {
spin_lock_irq(&sp->queue_lock);
- if (rcu_batch_empty(&sp->batch_done) &&
- rcu_batch_empty(&sp->batch_check1) &&
- rcu_batch_empty(&sp->batch_check0) &&
- rcu_batch_empty(&sp->batch_queue)) {
+ if (rcu_all_batches_empty(sp)) {
sp->running = false;
pending = false;
}
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-12 20:00 +0200 |
| Subject | [PATCH tip/core/rcu 05/40] rcu: Semicolon inside RCU_TRACE() for rcu.h |
| Message-ID | <tvuIG-7vr-21@gated-at.bofh.it> |
| In reply to | #1622453 |
The current use of "RCU_TRACE(statement);" can cause odd bugs, especially
where "statement" is a local-variable declaration, as it can leave a
misplaced ";" in the source code. This commit therefore converts these
to "RCU_TRACE(statement;)", which avoids the misplaced ";".
Reported-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/rcu.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
index 0d6ff3e471be..8700a81daf56 100644
--- a/kernel/rcu/rcu.h
+++ b/kernel/rcu/rcu.h
@@ -109,12 +109,12 @@ static inline bool __rcu_reclaim(const char *rn, struct rcu_head *head)
rcu_lock_acquire(&rcu_callback_map);
if (__is_kfree_rcu_offset(offset)) {
- RCU_TRACE(trace_rcu_invoke_kfree_callback(rn, head, offset));
+ RCU_TRACE(trace_rcu_invoke_kfree_callback(rn, head, offset);)
kfree((void *)head - offset);
rcu_lock_release(&rcu_callback_map);
return true;
} else {
- RCU_TRACE(trace_rcu_invoke_callback(rn, head));
+ RCU_TRACE(trace_rcu_invoke_callback(rn, head);)
head->func(head);
rcu_lock_release(&rcu_callback_map);
return false;
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-12 20:00 +0200 |
| Subject | [PATCH tip/core/rcu 11/40] rcu: Pull rcu_sched_qs_mask into rcu_dynticks structure |
| Message-ID | <tvuIG-7vr-27@gated-at.bofh.it> |
| In reply to | #1622453 |
The rcu_sched_qs_mask variable is yet another isolated per-CPU variable,
so this commit pulls it into the pre-existing rcu_dynticks per-CPU
structure.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
.../RCU/Design/Data-Structures/Data-Structures.html | 9 ++++++++-
kernel/rcu/tree.c | 12 +++++-------
kernel/rcu/tree.h | 1 +
3 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/Documentation/RCU/Design/Data-Structures/Data-Structures.html b/Documentation/RCU/Design/Data-Structures/Data-Structures.html
index d583c653a703..bf7f266e8888 100644
--- a/Documentation/RCU/Design/Data-Structures/Data-Structures.html
+++ b/Documentation/RCU/Design/Data-Structures/Data-Structures.html
@@ -1104,6 +1104,7 @@ Its fields are as follows:
1 int dynticks_nesting;
2 int dynticks_nmi_nesting;
3 atomic_t dynticks;
+ 4 int rcu_sched_qs_mask;
</pre>
<p>The <tt>->dynticks_nesting</tt> field counts the
@@ -1117,11 +1118,17 @@ NMIs are counted by the <tt>->dynticks_nmi_nesting</tt>
field, except that NMIs that interrupt non-dyntick-idle execution
are not counted.
-</p><p>Finally, the <tt>->dynticks</tt> field counts the corresponding
+</p><p>The <tt>->dynticks</tt> field counts the corresponding
CPU's transitions to and from dyntick-idle mode, so that this counter
has an even value when the CPU is in dyntick-idle mode and an odd
value otherwise.
+</p><p>Finally, the <tt>->rcu_sched_qs_mask</tt> field is used
+to record the fact that the RCU core code would really like to
+see a quiescent state from the corresponding CPU.
+This flag is checked by RCU's context-switch and <tt>cond_resched()</tt>
+code, which provide a momentary idle sojourn in response.
+
<table>
<tr><th> </th></tr>
<tr><th align="left">Quick Quiz:</th></tr>
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 7fa46967021f..315647d4e4cd 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -272,8 +272,6 @@ void rcu_bh_qs(void)
}
}
-static DEFINE_PER_CPU(int, rcu_sched_qs_mask);
-
/*
* Steal a bit from the bottom of ->dynticks for idle entry/exit
* control. Initially this is for TLB flushing.
@@ -464,8 +462,8 @@ static void rcu_momentary_dyntick_idle(void)
* Yes, we can lose flag-setting operations. This is OK, because
* the flag will be set again after some delay.
*/
- resched_mask = raw_cpu_read(rcu_sched_qs_mask);
- raw_cpu_write(rcu_sched_qs_mask, 0);
+ resched_mask = raw_cpu_read(rcu_dynticks.rcu_sched_qs_mask);
+ raw_cpu_write(rcu_dynticks.rcu_sched_qs_mask, 0);
/* Find the flavor that needs a quiescent state. */
for_each_rcu_flavor(rsp) {
@@ -501,7 +499,7 @@ void rcu_note_context_switch(void)
trace_rcu_utilization(TPS("Start context switch"));
rcu_sched_qs();
rcu_preempt_note_context_switch();
- if (unlikely(raw_cpu_read(rcu_sched_qs_mask)))
+ if (unlikely(raw_cpu_read(rcu_dynticks.rcu_sched_qs_mask)))
rcu_momentary_dyntick_idle();
for_each_rcu_flavor(rsp)
do_nocb_deferred_wakeup(this_cpu_ptr(rsp->rda));
@@ -529,7 +527,7 @@ void rcu_all_qs(void)
struct rcu_state *rsp;
barrier(); /* Avoid RCU read-side critical sections leaking down. */
- if (unlikely(raw_cpu_read(rcu_sched_qs_mask))) {
+ if (unlikely(raw_cpu_read(rcu_dynticks.rcu_sched_qs_mask))) {
local_irq_save(flags);
rcu_momentary_dyntick_idle();
local_irq_restore(flags);
@@ -1361,7 +1359,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
* is set too high, we override with half of the RCU CPU stall
* warning delay.
*/
- rcrmp = &per_cpu(rcu_sched_qs_mask, rdp->cpu);
+ rcrmp = &per_cpu(rcu_dynticks.rcu_sched_qs_mask, rdp->cpu);
if (time_after(jiffies, rdp->rsp->gp_start + jtsq) ||
time_after(jiffies, rdp->rsp->jiffies_resched)) {
if (!(READ_ONCE(*rcrmp) & rdp->rsp->flavor_mask)) {
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 7468b4de7e0c..e298281984dc 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -113,6 +113,7 @@ struct rcu_dynticks {
/* Process level is worth LLONG_MAX/2. */
int dynticks_nmi_nesting; /* Track NMI nesting level. */
atomic_t dynticks; /* Even value for idle, else odd. */
+ int rcu_sched_qs_mask; /* GP old, need quiescent state. */
#ifdef CONFIG_NO_HZ_FULL_SYSIDLE
long long dynticks_idle_nesting;
/* irq/process nesting level from idle. */
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2017-04-13 12:00 +0200 |
| Subject | Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling |
| Message-ID | <tvJHH-18N-5@gated-at.bofh.it> |
| In reply to | #1622453 |
On Wed, Apr 12, 2017 at 10:40:25AM -0700, Paul E. McKenney wrote: > Peter Zijlstra proposed using SRCU to reduce mmap_sem contention [1], Bugger, now you're making me feel bad for not having updated those patches in ages.. I'll try and bump it on the todo list.
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-13 18:40 +0200 |
| Subject | Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling |
| Message-ID | <tvPWN-5wS-7@gated-at.bofh.it> |
| In reply to | #1622873 |
On Thu, Apr 13, 2017 at 11:50:29AM +0200, Peter Zijlstra wrote: > On Wed, Apr 12, 2017 at 10:40:25AM -0700, Paul E. McKenney wrote: > > Peter Zijlstra proposed using SRCU to reduce mmap_sem contention [1], > > Bugger, now you're making me feel bad for not having updated those > patches in ages.. I'll try and bump it on the todo list. Apologies, I wasn't trying to make you feel bad. But I did get an old version of the patch. A much more recent one is here, which I have added to the commit log: https://patchwork.kernel.org/patch/5108281/ For my part, I feel bad that I didn't realize much earlier that parallel SRCU callbacks were needed. As it was, someone had to tell me late last year. But yes, it would be really cool to have lockless VMA lookup!!! Thanx, Paul
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-13 19:50 +0200 |
| Subject | Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling |
| Message-ID | <tvR2y-6iE-9@gated-at.bofh.it> |
| In reply to | #1623169 |
On Thu, Apr 13, 2017 at 09:37:10AM -0700, Paul E. McKenney wrote: > On Thu, Apr 13, 2017 at 11:50:29AM +0200, Peter Zijlstra wrote: > > On Wed, Apr 12, 2017 at 10:40:25AM -0700, Paul E. McKenney wrote: > > > Peter Zijlstra proposed using SRCU to reduce mmap_sem contention [1], > > > > Bugger, now you're making me feel bad for not having updated those > > patches in ages.. I'll try and bump it on the todo list. > > Apologies, I wasn't trying to make you feel bad. But I did get an old > version of the patch. A much more recent one is here, which I have > added to the commit log: > > https://patchwork.kernel.org/patch/5108281/ > > For my part, I feel bad that I didn't realize much earlier that parallel > SRCU callbacks were needed. As it was, someone had to tell me late > last year. And it turns out that Laurent Dufour (CCed) forward-ported your patch from the above patchworks URL to 4.11. He is chasing down some mmseq bugs on a best-effort basis. Thanx, Paul > But yes, it would be really cool to have lockless VMA lookup!!! > > Thanx, Paul
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2017-04-13 12:00 +0200 |
| Subject | Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling |
| Message-ID | <tvJHI-18N-19@gated-at.bofh.it> |
| In reply to | #1622453 |
On Wed, Apr 12, 2017 at 10:40:25AM -0700, Paul E. McKenney wrote: > Peter Zijlstra proposed using SRCU to reduce mmap_sem contention [1], > however, there are workloads that could result in a high volume of > concurrent invocations of call_srcu(), which with current SRCU would > result in excessive lock contention on the srcu_struct structure's > ->queue_lock, which protects SRCU's callback lists. This commit therefore > moves SRCU to per-CPU callback lists, thus greatly reducing contention. > > Because a given SRCU instance no longer has a single centralized callback > list, starting grace periods and invoking callbacks each require a bit > more work. These are handled using an srcu_node tree that is in some ways > similar to the rcu_node trees used by RCU-bh, RCU-preempt, and RCU-sched > (for example, the srcu_node tree shape is controlled by exactly the > same Kconfig options and boot parameters that control the shape of the > rcu_node tree). > > In addition, the old per-CPU srcu_array structure is now named srcu_data > and contains an rcu_segcblist structure named ->srcu_cblist for its > callbacks (and a spinlock to protect this). The srcu_struct gets > an srcu_gp_seq that is used to associate callback segments with the > corresponding completion-time grace-period number. These completion-time > grace-period numbers are propagated up the srcu_node tree so that the > grace-period workqueue handler can determine whether additional grace > periods are needed on the one hand and where to look for callbacks that > are ready to be invoked. > > The srcu_barrier() function must now wait on all instances of the > per-CPU ->srcu_cblist. Because each ->srcu_cblist is protected > by ->lock, srcu_barrier() can remotely add the needed callbacks. > In theory, it could also remotely start grace periods, but this gets > complex and racy. And interestingly enough, it is never necessary to > start a grace period in this case because srcu_barrier() only enqueues > a callback when a callback is already present. And a grace period has > to have already been started for this pre-existing callback. And it is > only the callback that srcu_barrier() needs to wait on, not any particular > grace period. Therefore, a new rcu_segcblist_entrain() function enqueues > the srcu_barrier() function's callback into the same segment occupied by > the pre-existing callback. The special case where all the pre-existing > callbacks are on a different list being invoked is handled by enqueuing > srcu_barrier()'s callback into the RCU_DONE_TAIL segment, relying on > the done-callbacks check that takes place after all callbacks are inovked. > > Note that the readers use the same algorithm as before. Note that there > is a separate srcu_idx that tells the readers what counter to increment. > This unfortunately cannot be combined with srcu_gp_seq because they > need to be incremented at different times. So one thing I've asked before I think, would it not be possible to abstract PREEMPT_RCU and use the exact same code for PREEMPT_RCU and SRCU ?
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-13 19:00 +0200 |
| Subject | Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling |
| Message-ID | <tvQg9-5H7-5@gated-at.bofh.it> |
| In reply to | #1622877 |
On Thu, Apr 13, 2017 at 11:54:20AM +0200, Peter Zijlstra wrote: > On Wed, Apr 12, 2017 at 10:40:25AM -0700, Paul E. McKenney wrote: > > Peter Zijlstra proposed using SRCU to reduce mmap_sem contention [1], > > however, there are workloads that could result in a high volume of > > concurrent invocations of call_srcu(), which with current SRCU would > > result in excessive lock contention on the srcu_struct structure's > > ->queue_lock, which protects SRCU's callback lists. This commit therefore > > moves SRCU to per-CPU callback lists, thus greatly reducing contention. > > > > Because a given SRCU instance no longer has a single centralized callback > > list, starting grace periods and invoking callbacks each require a bit > > more work. These are handled using an srcu_node tree that is in some ways > > similar to the rcu_node trees used by RCU-bh, RCU-preempt, and RCU-sched > > (for example, the srcu_node tree shape is controlled by exactly the > > same Kconfig options and boot parameters that control the shape of the > > rcu_node tree). > > > > In addition, the old per-CPU srcu_array structure is now named srcu_data > > and contains an rcu_segcblist structure named ->srcu_cblist for its > > callbacks (and a spinlock to protect this). The srcu_struct gets > > an srcu_gp_seq that is used to associate callback segments with the > > corresponding completion-time grace-period number. These completion-time > > grace-period numbers are propagated up the srcu_node tree so that the > > grace-period workqueue handler can determine whether additional grace > > periods are needed on the one hand and where to look for callbacks that > > are ready to be invoked. > > > > The srcu_barrier() function must now wait on all instances of the > > per-CPU ->srcu_cblist. Because each ->srcu_cblist is protected > > by ->lock, srcu_barrier() can remotely add the needed callbacks. > > In theory, it could also remotely start grace periods, but this gets > > complex and racy. And interestingly enough, it is never necessary to > > start a grace period in this case because srcu_barrier() only enqueues > > a callback when a callback is already present. And a grace period has > > to have already been started for this pre-existing callback. And it is > > only the callback that srcu_barrier() needs to wait on, not any particular > > grace period. Therefore, a new rcu_segcblist_entrain() function enqueues > > the srcu_barrier() function's callback into the same segment occupied by > > the pre-existing callback. The special case where all the pre-existing > > callbacks are on a different list being invoked is handled by enqueuing > > srcu_barrier()'s callback into the RCU_DONE_TAIL segment, relying on > > the done-callbacks check that takes place after all callbacks are inovked. > > > > Note that the readers use the same algorithm as before. Note that there > > is a separate srcu_idx that tells the readers what counter to increment. > > This unfortunately cannot be combined with srcu_gp_seq because they > > need to be incremented at different times. > > So one thing I've asked before I think, would it not be possible to > abstract PREEMPT_RCU and use the exact same code for PREEMPT_RCU and > SRCU ? I took a hard look at that some time ago, and it gets pretty ugly pretty quickly. Much of the PREEMPT_RCU code has the idea that there is only one global PREEMPT_RCU implementation baked deeply into it. For but one example, the handling of an arbitrarily large number of ->blkd_tasks lists at context-switch time would not be pretty, especially if the task in question blocked while in both a PREEMPT_RCU and in an SRCU read-side critical section. Or, worse yet, if it blocked while in several different SRCU read-side critical sections. It might be easier to go the other way and implement PREEMPT_RCU in terms of SRCU, but I don't believe that the read-side smp_mb() calls would make people happy. Plus there are use cases that would not be well-served by idle no longer being an extended quiescent state. And SRCU currently has inconvenient restrictions about use in interrupt and NMI handlers. It might well be that there is a global solution for all this, but in the meantime I am instead sharing common code and doing a bit of consolidation. Thanx, Paul
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 0/40] SRCU callback parallelization for 4.12 |
| Message-ID | <txoz7-789-5@gated-at.bofh.it> |
| In reply to | #1622453 |
Hello!
This v2 series moves SRCU from its traditional single per-srcu_struct
callback queue to per-srcu_struct/per-CPU callback queues. This involves
abstracting functionality from Tree RCU, which results in a large
conflict footprint, which in turn results in some otherwise unrelated
patches coming along for the ride.
1. Maintain special bits at bottom of ->dynticks counter.
This is for some upcoming MM work. My intent was to hold
it until that work was ready, but merge conflicts dictated
otherwise. If the MM work does not appear soonish, I will
manually revert this patch.
2. Make arch select smp_mb__after_unlock_lock() strength, which
gets rid of an arch-specific #ifdef.
3. Consolidate SRCU batch checking into rcu_all_batches_empty().
4. Check for tardy grace-period activity in cleanup_srcu_struct().
5-7. Semicolon inside RCU_TRACE() for various parts of RCU.
8. Pull rcu_sched_qs_mask into rcu_dynticks structure in order to
eliminate an isolated per-CPU variable.
9. Pull rcu_qs_ctr into rcu_dynticks structure.
10. Eliminate flavor scan in rcu_momentary_dyntick_idle() to
reduce semi-common-case context-switch overhead.
11. Place guard on rcu_all_qs() and rcu_note_context_switch()
actions to reduce common-case scheduler-fastpath overhead.
12. Default RCU_FANOUT_LEAF to 16 unless explicitly changed.
13. Abstract multi-tail callback list handling for SRCU.
14. Allow SRCU to access rcu_scheduler_active.
15. Allow early boot use of synchronize_srcu(), though not yet
mid-boot use.
16. Add single-element dequeue functions to rcu_segcblist for
debug use.
17. Move rcu_seq_start() and friends to rcu.h for SRCU's benefit.
18. Expedited wakeups need to be fully ordered.
19. Fix warning in rcu_seq_end().
20. Push srcu_advance_batches() fastpath into common case as a
step towards callback parallelization.
21. Move to state-based grace-period sequencing, also as a step
towards callback parallelization.
22. Add grace-period sequence numbers to SRCU.
23. Use rcu_segcblist to track SRCU callbacks.
24. Move combining-tree definitions for SRCU's benefit.
25. Move rcu_init_levelspread() to rcu_tree_node.h for SRCU's benefit.
26. Remove redundant levelcnt[] array from rcu_init_one().
27. Move rcu_node traversal macros to rcu.h for SRCU's benefit.
28. Make num_rcu_lvl[] array be external for SRCU's benefit.
29. Fix bogus try_check_zero() comment.
30. Improve rcu_seq grace-period-counter abstraction for SRCU's
benefit.
31. Allow a second bit in rcu_seq for SRCU state.
32. Merge ->srcu_state into ->srcu_gp_seq to allow atomic updates.
33. Provide crude control of expedited SRCU grace periods.
34. Use static initialization for "srcu" in mm/mmu_notifier.c.
35. Create a tiny SRCU for bloatwatch/tinification.
36. Print Tiny SRCU reader statistics in rcutorture.
37. Introduce CLASSIC_SRCU Kconfig option for those who do not
wish to help debug Tree SRCU.
38. Parallelize SRCU callback handling.
39. Make non-preemptive schedule be Tasks RCU quiescent state.
Updates since v1:
o Incorporate feedback from Peter Zijlstra.
o Dropped v1 patches 8-10 ("Make various parts of RCU do deferred
NOCB wakeups in order to prevent callback blockages, and thus
hangs"). These patches turned out to be papering over a no-CBs
CPU design flaw. There will be patches in v4.13 to fix the design
flaw directly.
o Added v2 patch #34 ("Use static initialization for "srcu" in
mm/mmu_notifier.c"), moving it from its v1 location in the
fixes series.
o Added v2 patch #39 ("Make non-preemptive schedule be Tasks RCU
quiescent state") for the benefit of upcoming ftrace work at
Steve Rostedt's request.
Thanx, Paul
------------------------------------------------------------------------
/kernel/rcu/rcu_segcblist.h | 670 -----
b/Documentation/RCU/Design/Data-Structures/Data-Structures.html | 36
b/arch/Kconfig | 3
b/arch/powerpc/Kconfig | 1
b/include/linux/rcu_node_tree.h | 105
b/include/linux/rcu_segcblist.h | 720 +++++
b/include/linux/rcupdate.h | 17
b/include/linux/rcutiny.h | 24
b/include/linux/rcutree.h | 5
b/include/linux/srcu.h | 112
b/include/linux/srcuclassic.h | 101
b/include/linux/srcutiny.h | 81
b/include/linux/srcutree.h | 171 +
b/init/Kconfig | 33
b/kernel/rcu/Makefile | 6
b/kernel/rcu/rcu.h | 165 +
b/kernel/rcu/rcu_segcblist.h | 670 +++++
b/kernel/rcu/rcutorture.c | 39
b/kernel/rcu/srcu.c | 846 +++---
b/kernel/rcu/srcutiny.c | 215 +
b/kernel/rcu/srcutree.c | 1252 ++++++++--
b/kernel/rcu/tiny.c | 20
b/kernel/rcu/tiny_plugin.h | 13
b/kernel/rcu/tree.c | 657 ++---
b/kernel/rcu/tree.h | 174 -
b/kernel/rcu/tree_exp.h | 25
b/kernel/rcu/tree_plugin.h | 62
b/kernel/rcu/tree_trace.c | 26
b/kernel/rcu/update.c | 53
b/kernel/sched/core.c | 2
b/mm/mmu_notifier.c | 14
31 files changed, 4287 insertions(+), 2031 deletions(-)
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 02/39] rcu: Make arch select smp_mb__after_unlock_lock() strength |
| Message-ID | <txoz7-789-13@gated-at.bofh.it> |
| In reply to | #1624925 |
The definition of smp_mb__after_unlock_lock() is currently smp_mb()
for CONFIG_PPC and a no-op otherwise. It would be better to instead
provide an architecture-selectable Kconfig option, and select the
strength of smp_mb__after_unlock_lock() based on that option. This
commit therefore creates ARCH_WEAK_RELEASE_ACQUIRE, has PPC select it,
and bases the definition of smp_mb__after_unlock_lock() on this new
ARCH_WEAK_RELEASE_ACQUIRE Kconfig option.
Reported-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Boqun Feng <boqun.feng@linux.vnet.ibm.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Cc: <linuxppc-dev@lists.ozlabs.org>
---
arch/Kconfig | 3 +++
arch/powerpc/Kconfig | 1 +
include/linux/rcupdate.h | 6 +++---
3 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/arch/Kconfig b/arch/Kconfig
index cd211a14a88f..adefaf344239 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -320,6 +320,9 @@ config HAVE_CMPXCHG_LOCAL
config HAVE_CMPXCHG_DOUBLE
bool
+config ARCH_WEAK_RELEASE_ACQUIRE
+ bool
+
config ARCH_WANT_IPC_PARSE_VERSION
bool
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 97a8bc8a095c..7a5c9b764cd2 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -99,6 +99,7 @@ config PPC
select ARCH_USE_BUILTIN_BSWAP
select ARCH_USE_CMPXCHG_LOCKREF if PPC64
select ARCH_WANT_IPC_PARSE_VERSION
+ select ARCH_WEAK_RELEASE_ACQUIRE
select BINFMT_ELF
select BUILDTIME_EXTABLE_SORT
select CLONE_BACKWARDS
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index de88b33c0974..e6146d0074f8 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1127,11 +1127,11 @@ do { \
* if the UNLOCK and LOCK are executed by the same CPU or if the
* UNLOCK and LOCK operate on the same lock variable.
*/
-#ifdef CONFIG_PPC
+#ifdef CONFIG_ARCH_WEAK_RELEASE_ACQUIRE
#define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
-#else /* #ifdef CONFIG_PPC */
+#else /* #ifdef CONFIG_ARCH_WEAK_RELEASE_ACQUIRE */
#define smp_mb__after_unlock_lock() do { } while (0)
-#endif /* #else #ifdef CONFIG_PPC */
+#endif /* #else #ifdef CONFIG_ARCH_WEAK_RELEASE_ACQUIRE */
#endif /* __LINUX_RCUPDATE_H */
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | Josh Triplett <josh@joshtriplett.org> |
|---|---|
| Date | 2017-04-18 02:20 +0200 |
| Subject | Re: [PATCH v2 tip/core/rcu 02/39] rcu: Make arch select smp_mb__after_unlock_lock() strength |
| Message-ID | <txp2a-7y8-13@gated-at.bofh.it> |
| In reply to | #1624926 |
On Mon, Apr 17, 2017 at 04:44:49PM -0700, Paul E. McKenney wrote:
> The definition of smp_mb__after_unlock_lock() is currently smp_mb()
> for CONFIG_PPC and a no-op otherwise. It would be better to instead
> provide an architecture-selectable Kconfig option, and select the
> strength of smp_mb__after_unlock_lock() based on that option. This
> commit therefore creates ARCH_WEAK_RELEASE_ACQUIRE, has PPC select it,
> and bases the definition of smp_mb__after_unlock_lock() on this new
> ARCH_WEAK_RELEASE_ACQUIRE Kconfig option.
>
> Reported-by: Ingo Molnar <mingo@kernel.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Boqun Feng <boqun.feng@linux.vnet.ibm.com>
> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Acked-by: Michael Ellerman <mpe@ellerman.id.au>
> Cc: <linuxppc-dev@lists.ozlabs.org>
Seems sensible.
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
> arch/Kconfig | 3 +++
> arch/powerpc/Kconfig | 1 +
> include/linux/rcupdate.h | 6 +++---
> 3 files changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/arch/Kconfig b/arch/Kconfig
> index cd211a14a88f..adefaf344239 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -320,6 +320,9 @@ config HAVE_CMPXCHG_LOCAL
> config HAVE_CMPXCHG_DOUBLE
> bool
>
> +config ARCH_WEAK_RELEASE_ACQUIRE
> + bool
> +
> config ARCH_WANT_IPC_PARSE_VERSION
> bool
>
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index 97a8bc8a095c..7a5c9b764cd2 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -99,6 +99,7 @@ config PPC
> select ARCH_USE_BUILTIN_BSWAP
> select ARCH_USE_CMPXCHG_LOCKREF if PPC64
> select ARCH_WANT_IPC_PARSE_VERSION
> + select ARCH_WEAK_RELEASE_ACQUIRE
> select BINFMT_ELF
> select BUILDTIME_EXTABLE_SORT
> select CLONE_BACKWARDS
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index de88b33c0974..e6146d0074f8 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -1127,11 +1127,11 @@ do { \
> * if the UNLOCK and LOCK are executed by the same CPU or if the
> * UNLOCK and LOCK operate on the same lock variable.
> */
> -#ifdef CONFIG_PPC
> +#ifdef CONFIG_ARCH_WEAK_RELEASE_ACQUIRE
> #define smp_mb__after_unlock_lock() smp_mb() /* Full ordering for lock. */
> -#else /* #ifdef CONFIG_PPC */
> +#else /* #ifdef CONFIG_ARCH_WEAK_RELEASE_ACQUIRE */
> #define smp_mb__after_unlock_lock() do { } while (0)
> -#endif /* #else #ifdef CONFIG_PPC */
> +#endif /* #else #ifdef CONFIG_ARCH_WEAK_RELEASE_ACQUIRE */
>
>
> #endif /* __LINUX_RCUPDATE_H */
> --
> 2.5.2
>
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 10/39] rcu: Eliminate flavor scan in rcu_momentary_dyntick_idle() |
| Message-ID | <txoz8-789-15@gated-at.bofh.it> |
| In reply to | #1624925 |
The rcu_momentary_dyntick_idle() function scans the RCU flavors, checking
that one of them still needs a quiescent state before doing an expensive
atomic operation on the ->dynticks counter. However, this check reduces
overhead only after a rare race condition, and increases complexity. This
commit therefore removes the scan and the mechanism enabling the scan.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
.../Design/Data-Structures/Data-Structures.html | 4 +-
kernel/rcu/tree.c | 62 +++++-----------------
kernel/rcu/tree.h | 3 +-
3 files changed, 15 insertions(+), 54 deletions(-)
diff --git a/Documentation/RCU/Design/Data-Structures/Data-Structures.html b/Documentation/RCU/Design/Data-Structures/Data-Structures.html
index 3d0311657533..e4bf20a68fa3 100644
--- a/Documentation/RCU/Design/Data-Structures/Data-Structures.html
+++ b/Documentation/RCU/Design/Data-Structures/Data-Structures.html
@@ -1104,7 +1104,7 @@ Its fields are as follows:
1 int dynticks_nesting;
2 int dynticks_nmi_nesting;
3 atomic_t dynticks;
- 4 int rcu_sched_qs_mask;
+ 4 bool rcu_need_heavy_qs;
5 unsigned long rcu_qs_ctr;
</pre>
@@ -1124,7 +1124,7 @@ CPU's transitions to and from dyntick-idle mode, so that this counter
has an even value when the CPU is in dyntick-idle mode and an odd
value otherwise.
-</p><p>The <tt>->rcu_sched_qs_mask</tt> field is used
+</p><p>The <tt>->rcu_need_heavy_qs</tt> field is used
to record the fact that the RCU core code would really like to
see a quiescent state from the corresponding CPU, so much so that
it is willing to call for heavy-weight dyntick-counter operations.
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index fbee1d729c4b..3c62ea06edb3 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -443,44 +443,14 @@ bool rcu_eqs_special_set(int cpu)
* memory barriers to let the RCU core know about it, regardless of what
* this CPU might (or might not) do in the near future.
*
- * We inform the RCU core by emulating a zero-duration dyntick-idle
- * period, which we in turn do by incrementing the ->dynticks counter
- * by two.
+ * We inform the RCU core by emulating a zero-duration dyntick-idle period.
*
* The caller must have disabled interrupts.
*/
static void rcu_momentary_dyntick_idle(void)
{
- struct rcu_data *rdp;
- int resched_mask;
- struct rcu_state *rsp;
-
- /*
- * Yes, we can lose flag-setting operations. This is OK, because
- * the flag will be set again after some delay.
- */
- resched_mask = raw_cpu_read(rcu_dynticks.rcu_sched_qs_mask);
- raw_cpu_write(rcu_dynticks.rcu_sched_qs_mask, 0);
-
- /* Find the flavor that needs a quiescent state. */
- for_each_rcu_flavor(rsp) {
- rdp = raw_cpu_ptr(rsp->rda);
- if (!(resched_mask & rsp->flavor_mask))
- continue;
- smp_mb(); /* rcu_sched_qs_mask before cond_resched_completed. */
- if (READ_ONCE(rdp->mynode->completed) !=
- READ_ONCE(rdp->cond_resched_completed))
- continue;
-
- /*
- * Pretend to be momentarily idle for the quiescent state.
- * This allows the grace-period kthread to record the
- * quiescent state, with no need for this CPU to do anything
- * further.
- */
- rcu_dynticks_momentary_idle();
- break;
- }
+ raw_cpu_write(rcu_dynticks.rcu_need_heavy_qs, false);
+ rcu_dynticks_momentary_idle();
}
/*
@@ -494,7 +464,7 @@ void rcu_note_context_switch(void)
trace_rcu_utilization(TPS("Start context switch"));
rcu_sched_qs();
rcu_preempt_note_context_switch();
- if (unlikely(raw_cpu_read(rcu_dynticks.rcu_sched_qs_mask)))
+ if (unlikely(raw_cpu_read(rcu_dynticks.rcu_need_heavy_qs)))
rcu_momentary_dyntick_idle();
trace_rcu_utilization(TPS("End context switch"));
barrier(); /* Avoid RCU read-side critical sections leaking up. */
@@ -519,7 +489,7 @@ void rcu_all_qs(void)
unsigned long flags;
barrier(); /* Avoid RCU read-side critical sections leaking down. */
- if (unlikely(raw_cpu_read(rcu_dynticks.rcu_sched_qs_mask))) {
+ if (unlikely(raw_cpu_read(rcu_dynticks.rcu_need_heavy_qs))) {
local_irq_save(flags);
rcu_momentary_dyntick_idle();
local_irq_restore(flags);
@@ -1275,7 +1245,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
bool *isidle, unsigned long *maxj)
{
unsigned long jtsq;
- int *rcrmp;
+ bool *rnhqp;
unsigned long rjtsc;
struct rcu_node *rnp;
@@ -1332,7 +1302,7 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
* in-kernel CPU-bound tasks cannot advance grace periods.
* So if the grace period is old enough, make the CPU pay attention.
* Note that the unsynchronized assignments to the per-CPU
- * rcu_sched_qs_mask variable are safe. Yes, setting of
+ * rcu_need_heavy_qs variable are safe. Yes, setting of
* bits can be lost, but they will be set again on the next
* force-quiescent-state pass. So lost bit sets do not result
* in incorrect behavior, merely in a grace period lasting
@@ -1346,16 +1316,11 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp,
* is set too high, we override with half of the RCU CPU stall
* warning delay.
*/
- rcrmp = &per_cpu(rcu_dynticks.rcu_sched_qs_mask, rdp->cpu);
- if (time_after(jiffies, rdp->rsp->gp_start + jtsq) ||
- time_after(jiffies, rdp->rsp->jiffies_resched)) {
- if (!(READ_ONCE(*rcrmp) & rdp->rsp->flavor_mask)) {
- WRITE_ONCE(rdp->cond_resched_completed,
- READ_ONCE(rdp->mynode->completed));
- smp_mb(); /* ->cond_resched_completed before *rcrmp. */
- WRITE_ONCE(*rcrmp,
- READ_ONCE(*rcrmp) + rdp->rsp->flavor_mask);
- }
+ rnhqp = &per_cpu(rcu_dynticks.rcu_need_heavy_qs, rdp->cpu);
+ if (!READ_ONCE(*rnhqp) &&
+ (time_after(jiffies, rdp->rsp->gp_start + jtsq) ||
+ time_after(jiffies, rdp->rsp->jiffies_resched))) {
+ WRITE_ONCE(*rnhqp, true);
rdp->rsp->jiffies_resched += 5; /* Re-enable beating. */
}
@@ -4169,7 +4134,6 @@ static void __init rcu_init_one(struct rcu_state *rsp)
static const char * const fqs[] = RCU_FQS_NAME_INIT;
static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS];
- static u8 fl_mask = 0x1;
int levelcnt[RCU_NUM_LVLS]; /* # nodes in each level. */
int levelspread[RCU_NUM_LVLS]; /* kids/node in each level. */
@@ -4191,8 +4155,6 @@ static void __init rcu_init_one(struct rcu_state *rsp)
for (i = 1; i < rcu_num_lvls; i++)
rsp->level[i] = rsp->level[i - 1] + levelcnt[i - 1];
rcu_init_levelspread(levelspread, levelcnt);
- rsp->flavor_mask = fl_mask;
- fl_mask <<= 1;
/* Initialize the elements themselves, starting from the leaves. */
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 76e4467bc765..b212cd0f22c7 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -113,7 +113,7 @@ struct rcu_dynticks {
/* Process level is worth LLONG_MAX/2. */
int dynticks_nmi_nesting; /* Track NMI nesting level. */
atomic_t dynticks; /* Even value for idle, else odd. */
- int rcu_sched_qs_mask; /* GP old, need heavy quiescent state. */
+ bool rcu_need_heavy_qs; /* GP old, need heavy quiescent state. */
unsigned long rcu_qs_ctr; /* Light universal quiescent state ctr. */
#ifdef CONFIG_NO_HZ_FULL_SYSIDLE
long long dynticks_idle_nesting;
@@ -484,7 +484,6 @@ struct rcu_state {
struct rcu_node *level[RCU_NUM_LVLS + 1];
/* Hierarchy levels (+1 to */
/* shut bogus gcc warning) */
- u8 flavor_mask; /* bit in flavor mask. */
struct rcu_data __percpu *rda; /* pointer of percu rcu_data. */
call_rcu_func_t call; /* call_rcu() flavor. */
int ncpus; /* # CPUs seen so far. */
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 24/39] srcu: Move combining-tree definitions for SRCU's benefit |
| Message-ID | <txoz8-789-27@gated-at.bofh.it> |
| In reply to | #1624925 |
This commit moves the C preprocessor code that defines the default shape
of the rcu_node combining tree to a new include/linux/rcu_node_tree.h
file as a first step towards enabling SRCU to create its own combining
tree, which in turn enables SRCU to implement per-CPU callback handling,
thus avoiding contention on the lock currently guarding the single list
of callbacks. Note that users of SRCU still need to know the size of
the srcu_struct structure, hence include/linux rather than kernel/rcu.
This commit is code-movement only.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcu_node_tree.h | 102 ++++++++++++++++++++++++++++++++++++++++++
kernel/rcu/tree.h | 71 +----------------------------
2 files changed, 103 insertions(+), 70 deletions(-)
create mode 100644 include/linux/rcu_node_tree.h
diff --git a/include/linux/rcu_node_tree.h b/include/linux/rcu_node_tree.h
new file mode 100644
index 000000000000..b7eb97096b1c
--- /dev/null
+++ b/include/linux/rcu_node_tree.h
@@ -0,0 +1,102 @@
+/*
+ * RCU node combining tree definitions. These are used to compute
+ * global attributes while avoiding common-case global contention. A key
+ * property that these computations rely on is a tournament-style approach
+ * where only one of the tasks contending a lower level in the tree need
+ * advance to the next higher level. If properly configured, this allows
+ * unlimited scalability while maintaining a constant level of contention
+ * on the root node.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you can access it online at
+ * http://www.gnu.org/licenses/gpl-2.0.html.
+ *
+ * Copyright IBM Corporation, 2017
+ *
+ * Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
+ */
+
+#ifndef __LINUX_RCU_NODE_TREE_H
+#define __LINUX_RCU_NODE_TREE_H
+
+/*
+ * Define shape of hierarchy based on NR_CPUS, CONFIG_RCU_FANOUT, and
+ * CONFIG_RCU_FANOUT_LEAF.
+ * In theory, it should be possible to add more levels straightforwardly.
+ * In practice, this did work well going from three levels to four.
+ * Of course, your mileage may vary.
+ */
+
+#ifdef CONFIG_RCU_FANOUT
+#define RCU_FANOUT CONFIG_RCU_FANOUT
+#else /* #ifdef CONFIG_RCU_FANOUT */
+# ifdef CONFIG_64BIT
+# define RCU_FANOUT 64
+# else
+# define RCU_FANOUT 32
+# endif
+#endif /* #else #ifdef CONFIG_RCU_FANOUT */
+
+#ifdef CONFIG_RCU_FANOUT_LEAF
+#define RCU_FANOUT_LEAF CONFIG_RCU_FANOUT_LEAF
+#else /* #ifdef CONFIG_RCU_FANOUT_LEAF */
+#define RCU_FANOUT_LEAF 16
+#endif /* #else #ifdef CONFIG_RCU_FANOUT_LEAF */
+
+#define RCU_FANOUT_1 (RCU_FANOUT_LEAF)
+#define RCU_FANOUT_2 (RCU_FANOUT_1 * RCU_FANOUT)
+#define RCU_FANOUT_3 (RCU_FANOUT_2 * RCU_FANOUT)
+#define RCU_FANOUT_4 (RCU_FANOUT_3 * RCU_FANOUT)
+
+#if NR_CPUS <= RCU_FANOUT_1
+# define RCU_NUM_LVLS 1
+# define NUM_RCU_LVL_0 1
+# define NUM_RCU_NODES NUM_RCU_LVL_0
+# define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0 }
+# define RCU_NODE_NAME_INIT { "rcu_node_0" }
+# define RCU_FQS_NAME_INIT { "rcu_node_fqs_0" }
+#elif NR_CPUS <= RCU_FANOUT_2
+# define RCU_NUM_LVLS 2
+# define NUM_RCU_LVL_0 1
+# define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
+# define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1)
+# define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1 }
+# define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1" }
+# define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1" }
+#elif NR_CPUS <= RCU_FANOUT_3
+# define RCU_NUM_LVLS 3
+# define NUM_RCU_LVL_0 1
+# define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_2)
+# define NUM_RCU_LVL_2 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
+# define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1 + NUM_RCU_LVL_2)
+# define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1, NUM_RCU_LVL_2 }
+# define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1", "rcu_node_2" }
+# define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1", "rcu_node_fqs_2" }
+#elif NR_CPUS <= RCU_FANOUT_4
+# define RCU_NUM_LVLS 4
+# define NUM_RCU_LVL_0 1
+# define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_3)
+# define NUM_RCU_LVL_2 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_2)
+# define NUM_RCU_LVL_3 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
+# define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1 + NUM_RCU_LVL_2 + NUM_RCU_LVL_3)
+# define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1, NUM_RCU_LVL_2, NUM_RCU_LVL_3 }
+# define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1", "rcu_node_2", "rcu_node_3" }
+# define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1", "rcu_node_fqs_2", "rcu_node_fqs_3" }
+#else
+# error "CONFIG_RCU_FANOUT insufficient for NR_CPUS"
+#endif /* #if (NR_CPUS) <= RCU_FANOUT_1 */
+
+extern int rcu_num_lvls;
+extern int rcu_num_nodes;
+
+#endif /* __LINUX_RCU_NODE_TREE_H */
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 4f62651588ea..1bec3958d44f 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -31,76 +31,7 @@
#include <linux/swait.h>
#include <linux/stop_machine.h>
#include <linux/rcu_segcblist.h>
-
-/*
- * Define shape of hierarchy based on NR_CPUS, CONFIG_RCU_FANOUT, and
- * CONFIG_RCU_FANOUT_LEAF.
- * In theory, it should be possible to add more levels straightforwardly.
- * In practice, this did work well going from three levels to four.
- * Of course, your mileage may vary.
- */
-
-#ifdef CONFIG_RCU_FANOUT
-#define RCU_FANOUT CONFIG_RCU_FANOUT
-#else /* #ifdef CONFIG_RCU_FANOUT */
-# ifdef CONFIG_64BIT
-# define RCU_FANOUT 64
-# else
-# define RCU_FANOUT 32
-# endif
-#endif /* #else #ifdef CONFIG_RCU_FANOUT */
-
-#ifdef CONFIG_RCU_FANOUT_LEAF
-#define RCU_FANOUT_LEAF CONFIG_RCU_FANOUT_LEAF
-#else /* #ifdef CONFIG_RCU_FANOUT_LEAF */
-#define RCU_FANOUT_LEAF 16
-#endif /* #else #ifdef CONFIG_RCU_FANOUT_LEAF */
-
-#define RCU_FANOUT_1 (RCU_FANOUT_LEAF)
-#define RCU_FANOUT_2 (RCU_FANOUT_1 * RCU_FANOUT)
-#define RCU_FANOUT_3 (RCU_FANOUT_2 * RCU_FANOUT)
-#define RCU_FANOUT_4 (RCU_FANOUT_3 * RCU_FANOUT)
-
-#if NR_CPUS <= RCU_FANOUT_1
-# define RCU_NUM_LVLS 1
-# define NUM_RCU_LVL_0 1
-# define NUM_RCU_NODES NUM_RCU_LVL_0
-# define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0 }
-# define RCU_NODE_NAME_INIT { "rcu_node_0" }
-# define RCU_FQS_NAME_INIT { "rcu_node_fqs_0" }
-#elif NR_CPUS <= RCU_FANOUT_2
-# define RCU_NUM_LVLS 2
-# define NUM_RCU_LVL_0 1
-# define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
-# define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1)
-# define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1 }
-# define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1" }
-# define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1" }
-#elif NR_CPUS <= RCU_FANOUT_3
-# define RCU_NUM_LVLS 3
-# define NUM_RCU_LVL_0 1
-# define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_2)
-# define NUM_RCU_LVL_2 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
-# define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1 + NUM_RCU_LVL_2)
-# define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1, NUM_RCU_LVL_2 }
-# define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1", "rcu_node_2" }
-# define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1", "rcu_node_fqs_2" }
-#elif NR_CPUS <= RCU_FANOUT_4
-# define RCU_NUM_LVLS 4
-# define NUM_RCU_LVL_0 1
-# define NUM_RCU_LVL_1 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_3)
-# define NUM_RCU_LVL_2 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_2)
-# define NUM_RCU_LVL_3 DIV_ROUND_UP(NR_CPUS, RCU_FANOUT_1)
-# define NUM_RCU_NODES (NUM_RCU_LVL_0 + NUM_RCU_LVL_1 + NUM_RCU_LVL_2 + NUM_RCU_LVL_3)
-# define NUM_RCU_LVL_INIT { NUM_RCU_LVL_0, NUM_RCU_LVL_1, NUM_RCU_LVL_2, NUM_RCU_LVL_3 }
-# define RCU_NODE_NAME_INIT { "rcu_node_0", "rcu_node_1", "rcu_node_2", "rcu_node_3" }
-# define RCU_FQS_NAME_INIT { "rcu_node_fqs_0", "rcu_node_fqs_1", "rcu_node_fqs_2", "rcu_node_fqs_3" }
-#else
-# error "CONFIG_RCU_FANOUT insufficient for NR_CPUS"
-#endif /* #if (NR_CPUS) <= RCU_FANOUT_1 */
-
-extern int rcu_num_lvls;
-extern int rcu_num_nodes;
+#include <linux/rcu_node_tree.h>
/*
* Dynticks per-CPU state.
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 31/39] srcu: Allow a second bit in rcu_seq for SRCU state |
| Message-ID | <txoz8-789-31@gated-at.bofh.it> |
| In reply to | #1624925 |
This commit increases the number of reserved bits at the bottom of an rcu_seq grace-period counter from one to two, as will be needed to accommodate SRCU's three-state grace periods. Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> --- kernel/rcu/rcu.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index c62df93bfc1b..87a0ac95b551 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -61,7 +61,7 @@ * Grace-period counter management. */ -#define RCU_SEQ_CTR_SHIFT 1 +#define RCU_SEQ_CTR_SHIFT 2 #define RCU_SEQ_STATE_MASK ((1 << RCU_SEQ_CTR_SHIFT) - 1) /* -- 2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 15/39] srcu: Allow early boot use of synchronize_srcu() |
| Message-ID | <txoz8-789-17@gated-at.bofh.it> |
| In reply to | #1624925 |
This commit checks for pre-scheduler state, and if that early in the boot process, synchronize_srcu() and friends are no-ops. Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> --- kernel/rcu/srcu.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c index 6beeba7b0b67..1026ce24922f 100644 --- a/kernel/rcu/srcu.c +++ b/kernel/rcu/srcu.c @@ -411,6 +411,8 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount) lock_is_held(&rcu_sched_lock_map), "Illegal synchronize_srcu() in same-type SRCU (or in RCU) read-side critical section"); + if (rcu_scheduler_active == RCU_SCHEDULER_INACTIVE) + return; might_sleep(); init_completion(&rcu.completion); -- 2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 14/39] srcu: Allow SRCU to access rcu_scheduler_active |
| Message-ID | <txoz8-789-19@gated-at.bofh.it> |
| In reply to | #1624925 |
This is primarily a code-movement commit in preparation for allowing
SRCU to handle early-boot SRCU grace periods.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcutiny.h | 6 +++---
kernel/rcu/tiny_plugin.h | 9 +++++----
kernel/rcu/tree.c | 2 +-
kernel/rcu/tree_exp.h | 12 -----------
kernel/rcu/update.c | 52 +++++++++++++++++++++++++++++++-----------------
5 files changed, 43 insertions(+), 38 deletions(-)
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index 6c9d941e3962..5219be250f00 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -217,14 +217,14 @@ static inline void exit_rcu(void)
{
}
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
+#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU)
extern int rcu_scheduler_active __read_mostly;
void rcu_scheduler_starting(void);
-#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+#else /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
static inline void rcu_scheduler_starting(void)
{
}
-#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+#endif /* #else #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE)
diff --git a/kernel/rcu/tiny_plugin.h b/kernel/rcu/tiny_plugin.h
index df3a60e19f07..371034e77f87 100644
--- a/kernel/rcu/tiny_plugin.h
+++ b/kernel/rcu/tiny_plugin.h
@@ -52,7 +52,7 @@ static struct rcu_ctrlblk rcu_bh_ctrlblk = {
RCU_TRACE(.name = "rcu_bh")
};
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
+#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU)
#include <linux/kernel_stat.h>
int rcu_scheduler_active __read_mostly;
@@ -65,15 +65,16 @@ EXPORT_SYMBOL_GPL(rcu_scheduler_active);
* to RCU_SCHEDULER_RUNNING, skipping the RCU_SCHEDULER_INIT stage.
* The reason for this is that Tiny RCU does not need kthreads, so does
* not have to care about the fact that the scheduler is half-initialized
- * at a certain phase of the boot process.
+ * at a certain phase of the boot process. Unless SRCU is in the mix.
*/
void __init rcu_scheduler_starting(void)
{
WARN_ON(nr_context_switches() > 0);
- rcu_scheduler_active = RCU_SCHEDULER_RUNNING;
+ rcu_scheduler_active = IS_ENABLED(CONFIG_SRCU)
+ ? RCU_SCHEDULER_INIT : RCU_SCHEDULER_RUNNING;
}
-#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
+#endif /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
#ifdef CONFIG_RCU_TRACE
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index ea14e6410cb8..03a1e3e09e82 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3974,7 +3974,7 @@ early_initcall(rcu_spawn_gp_kthread);
* task is booting the system, and such primitives are no-ops). After this
* function is called, any synchronous grace-period primitives are run as
* expedited, with the requesting task driving the grace period forward.
- * A later core_initcall() rcu_exp_runtime_mode() will switch to full
+ * A later core_initcall() rcu_set_runtime_mode() will switch to full
* runtime RCU functionality.
*/
void rcu_scheduler_starting(void)
diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index a1f52bbe9db6..51ca287828a2 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -737,15 +737,3 @@ void synchronize_rcu_expedited(void)
EXPORT_SYMBOL_GPL(synchronize_rcu_expedited);
#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
-
-/*
- * Switch to run-time mode once Tree RCU has fully initialized.
- */
-static int __init rcu_exp_runtime_mode(void)
-{
- rcu_test_sync_prims();
- rcu_scheduler_active = RCU_SCHEDULER_RUNNING;
- rcu_test_sync_prims();
- return 0;
-}
-core_initcall(rcu_exp_runtime_mode);
diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index 55c8530316c7..c5df0d756900 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -124,7 +124,7 @@ EXPORT_SYMBOL(rcu_read_lock_sched_held);
* non-expedited counterparts? Intended for use within RCU. Note
* that if the user specifies both rcu_expedited and rcu_normal, then
* rcu_normal wins. (Except during the time period during boot from
- * when the first task is spawned until the rcu_exp_runtime_mode()
+ * when the first task is spawned until the rcu_set_runtime_mode()
* core_initcall() is invoked, at which point everything is expedited.)
*/
bool rcu_gp_is_normal(void)
@@ -190,6 +190,39 @@ void rcu_end_inkernel_boot(void)
#endif /* #ifndef CONFIG_TINY_RCU */
+/*
+ * Test each non-SRCU synchronous grace-period wait API. This is
+ * useful just after a change in mode for these primitives, and
+ * during early boot.
+ */
+void rcu_test_sync_prims(void)
+{
+ if (!IS_ENABLED(CONFIG_PROVE_RCU))
+ return;
+ synchronize_rcu();
+ synchronize_rcu_bh();
+ synchronize_sched();
+ synchronize_rcu_expedited();
+ synchronize_rcu_bh_expedited();
+ synchronize_sched_expedited();
+}
+
+#if !defined(CONFIG_TINY_RCU) || defined(CONFIG_SRCU)
+
+/*
+ * Switch to run-time mode once RCU has fully initialized.
+ */
+static int __init rcu_set_runtime_mode(void)
+{
+ rcu_test_sync_prims();
+ rcu_scheduler_active = RCU_SCHEDULER_RUNNING;
+ rcu_test_sync_prims();
+ return 0;
+}
+core_initcall(rcu_set_runtime_mode);
+
+#endif /* #if !defined(CONFIG_TINY_RCU) || defined(CONFIG_SRCU) */
+
#ifdef CONFIG_PREEMPT_RCU
/*
@@ -817,23 +850,6 @@ static void rcu_spawn_tasks_kthread(void)
#endif /* #ifdef CONFIG_TASKS_RCU */
-/*
- * Test each non-SRCU synchronous grace-period wait API. This is
- * useful just after a change in mode for these primitives, and
- * during early boot.
- */
-void rcu_test_sync_prims(void)
-{
- if (!IS_ENABLED(CONFIG_PROVE_RCU))
- return;
- synchronize_rcu();
- synchronize_rcu_bh();
- synchronize_sched();
- synchronize_rcu_expedited();
- synchronize_rcu_bh_expedited();
- synchronize_sched_expedited();
-}
-
#ifdef CONFIG_PROVE_RCU
/*
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 05/39] rcu: Semicolon inside RCU_TRACE() for rcu.h |
| Message-ID | <txoz8-789-21@gated-at.bofh.it> |
| In reply to | #1624925 |
The current use of "RCU_TRACE(statement);" can cause odd bugs, especially
where "statement" is a local-variable declaration, as it can leave a
misplaced ";" in the source code. This commit therefore converts these
to "RCU_TRACE(statement;)", which avoids the misplaced ";".
Reported-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/rcu.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
index 0d6ff3e471be..8700a81daf56 100644
--- a/kernel/rcu/rcu.h
+++ b/kernel/rcu/rcu.h
@@ -109,12 +109,12 @@ static inline bool __rcu_reclaim(const char *rn, struct rcu_head *head)
rcu_lock_acquire(&rcu_callback_map);
if (__is_kfree_rcu_offset(offset)) {
- RCU_TRACE(trace_rcu_invoke_kfree_callback(rn, head, offset));
+ RCU_TRACE(trace_rcu_invoke_kfree_callback(rn, head, offset);)
kfree((void *)head - offset);
rcu_lock_release(&rcu_callback_map);
return true;
} else {
- RCU_TRACE(trace_rcu_invoke_callback(rn, head));
+ RCU_TRACE(trace_rcu_invoke_callback(rn, head);)
head->func(head);
rcu_lock_release(&rcu_callback_map);
return false;
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 33/39] srcu: Crude control of expedited grace periods |
| Message-ID | <txoz8-789-23@gated-at.bofh.it> |
| In reply to | #1624925 |
SRCU's implementation of expedited grace periods has always assumed
that the SRCU instance is idle when the expedited request arrives.
This commit improves this a bit by maintaining a count of the number
of outstanding expedited requests, thus allowing prior non-expedited
grace periods accommodate these requests by shifting to expedited mode.
However, any non-expedited wait already in progress will still wait for
the full duration.
Improved control of expedited grace periods is planned, but one step
at a time.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/srcu.h | 1 +
kernel/rcu/srcu.c | 84 ++++++++++++++++++++++++++++------------------------
2 files changed, 47 insertions(+), 38 deletions(-)
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index e7dbc01b61a1..73a1b6296224 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -42,6 +42,7 @@ struct srcu_array {
struct srcu_struct {
unsigned long completed;
unsigned long srcu_gp_seq;
+ atomic_t srcu_exp_cnt;
struct srcu_array __percpu *per_cpu_ref;
spinlock_t queue_lock; /* protect ->srcu_cblist */
struct rcu_segcblist srcu_cblist;
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index 97aec5d7b316..b62919be99e7 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -43,6 +43,7 @@ static int init_srcu_struct_fields(struct srcu_struct *sp)
{
sp->completed = 0;
sp->srcu_gp_seq = 0;
+ atomic_set(&sp->srcu_exp_cnt, 0);
spin_lock_init(&sp->queue_lock);
rcu_segcblist_init(&sp->srcu_cblist);
INIT_DELAYED_WORK(&sp->work, process_srcu);
@@ -179,7 +180,6 @@ static bool srcu_readers_active(struct srcu_struct *sp)
return sum;
}
-#define SRCU_CALLBACK_BATCH 10
#define SRCU_INTERVAL 1
/**
@@ -191,6 +191,7 @@ static bool srcu_readers_active(struct srcu_struct *sp)
*/
void cleanup_srcu_struct(struct srcu_struct *sp)
{
+ WARN_ON_ONCE(atomic_read(&sp->srcu_exp_cnt));
if (WARN_ON(srcu_readers_active(sp)))
return; /* Leakage unless caller handles error. */
if (WARN_ON(!rcu_segcblist_empty(&sp->srcu_cblist)))
@@ -238,13 +239,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock);
* We use an adaptive strategy for synchronize_srcu() and especially for
* synchronize_srcu_expedited(). We spin for a fixed time period
* (defined below) to allow SRCU readers to exit their read-side critical
- * sections. If there are still some readers after 10 microseconds,
- * we repeatedly block for 1-millisecond time periods. This approach
- * has done well in testing, so there is no need for a config parameter.
+ * sections. If there are still some readers after a few microseconds,
+ * we repeatedly block for 1-millisecond time periods.
*/
#define SRCU_RETRY_CHECK_DELAY 5
-#define SYNCHRONIZE_SRCU_TRYCOUNT 2
-#define SYNCHRONIZE_SRCU_EXP_TRYCOUNT 12
/*
* Start an SRCU grace period.
@@ -261,16 +259,16 @@ static void srcu_gp_start(struct srcu_struct *sp)
}
/*
- * Wait until all readers counted by array index idx complete, but loop
- * a maximum of trycount times. The caller must ensure that ->completed
- * is not changed while checking.
+ * Wait until all readers counted by array index idx complete, but
+ * loop an additional time if there is an expedited grace period pending.
+ * The caller must ensure that ->completed is not changed while checking.
*/
static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
{
for (;;) {
if (srcu_readers_active_idx_check(sp, idx))
return true;
- if (--trycount <= 0)
+ if (--trycount + !!atomic_read(&sp->srcu_exp_cnt) <= 0)
return false;
udelay(SRCU_RETRY_CHECK_DELAY);
}
@@ -358,7 +356,7 @@ static void srcu_reschedule(struct srcu_struct *sp, unsigned long delay);
/*
* Helper function for synchronize_srcu() and synchronize_srcu_expedited().
*/
-static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
+static void __synchronize_srcu(struct srcu_struct *sp)
{
struct rcu_synchronize rcu;
struct rcu_head *head = &rcu.head;
@@ -395,6 +393,32 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
}
/**
+ * synchronize_srcu_expedited - Brute-force SRCU grace period
+ * @sp: srcu_struct with which to synchronize.
+ *
+ * Wait for an SRCU grace period to elapse, but be more aggressive about
+ * spinning rather than blocking when waiting.
+ *
+ * Note that synchronize_srcu_expedited() has the same deadlock and
+ * memory-ordering properties as does synchronize_srcu().
+ */
+void synchronize_srcu_expedited(struct srcu_struct *sp)
+{
+ bool do_norm = rcu_gp_is_normal();
+
+ if (!do_norm) {
+ atomic_inc(&sp->srcu_exp_cnt);
+ smp_mb__after_atomic(); /* increment before GP. */
+ }
+ __synchronize_srcu(sp);
+ if (!do_norm) {
+ smp_mb__before_atomic(); /* GP before decrement. */
+ atomic_dec(&sp->srcu_exp_cnt);
+ }
+}
+EXPORT_SYMBOL_GPL(synchronize_srcu_expedited);
+
+/**
* synchronize_srcu - wait for prior SRCU read-side critical-section completion
* @sp: srcu_struct with which to synchronize.
*
@@ -435,29 +459,14 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)
*/
void synchronize_srcu(struct srcu_struct *sp)
{
- __synchronize_srcu(sp, (rcu_gp_is_expedited() && !rcu_gp_is_normal())
- ? SYNCHRONIZE_SRCU_EXP_TRYCOUNT
- : SYNCHRONIZE_SRCU_TRYCOUNT);
+ if (rcu_gp_is_expedited())
+ synchronize_srcu_expedited(sp);
+ else
+ __synchronize_srcu(sp);
}
EXPORT_SYMBOL_GPL(synchronize_srcu);
/**
- * synchronize_srcu_expedited - Brute-force SRCU grace period
- * @sp: srcu_struct with which to synchronize.
- *
- * Wait for an SRCU grace period to elapse, but be more aggressive about
- * spinning rather than blocking when waiting.
- *
- * Note that synchronize_srcu_expedited() has the same deadlock and
- * memory-ordering properties as does synchronize_srcu().
- */
-void synchronize_srcu_expedited(struct srcu_struct *sp)
-{
- __synchronize_srcu(sp, SYNCHRONIZE_SRCU_EXP_TRYCOUNT);
-}
-EXPORT_SYMBOL_GPL(synchronize_srcu_expedited);
-
-/**
* srcu_barrier - Wait until all in-flight call_srcu() callbacks complete.
* @sp: srcu_struct on which to wait for in-flight callbacks.
*/
@@ -484,7 +493,7 @@ EXPORT_SYMBOL_GPL(srcu_batches_completed);
* Core SRCU state machine. Advance callbacks from ->batch_check0 to
* ->batch_check1 and then to ->batch_done as readers drain.
*/
-static void srcu_advance_batches(struct srcu_struct *sp, int trycount)
+static void srcu_advance_batches(struct srcu_struct *sp)
{
int idx;
@@ -515,8 +524,8 @@ static void srcu_advance_batches(struct srcu_struct *sp, int trycount)
if (rcu_seq_state(READ_ONCE(sp->srcu_gp_seq)) == SRCU_STATE_SCAN1) {
idx = 1 ^ (sp->completed & 1);
- if (!try_check_zero(sp, idx, trycount))
- return; /* readers present, retry after SRCU_INTERVAL */
+ if (!try_check_zero(sp, idx, 1))
+ return; /* readers present, retry later. */
srcu_flip(sp);
rcu_seq_set_state(&sp->srcu_gp_seq, SRCU_STATE_SCAN2);
}
@@ -528,9 +537,8 @@ static void srcu_advance_batches(struct srcu_struct *sp, int trycount)
* so check at least twice in quick succession after a flip.
*/
idx = 1 ^ (sp->completed & 1);
- trycount = trycount < 2 ? 2 : trycount;
- if (!try_check_zero(sp, idx, trycount))
- return; /* readers present, retry after SRCU_INTERVAL */
+ if (!try_check_zero(sp, idx, 2))
+ return; /* readers present, retry after later. */
srcu_gp_end(sp);
}
}
@@ -596,8 +604,8 @@ void process_srcu(struct work_struct *work)
sp = container_of(work, struct srcu_struct, work.work);
- srcu_advance_batches(sp, 1);
+ srcu_advance_batches(sp);
srcu_invoke_callbacks(sp);
- srcu_reschedule(sp, SRCU_INTERVAL);
+ srcu_reschedule(sp, atomic_read(&sp->srcu_exp_cnt) ? 0 : SRCU_INTERVAL);
}
EXPORT_SYMBOL_GPL(process_srcu);
--
2.5.2
[toc] | [prev] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2017-04-18 01:50 +0200 |
| Subject | [PATCH v2 tip/core/rcu 29/39] srcu: Fix bogus try_check_zero() comment |
| Message-ID | <txoz8-789-25@gated-at.bofh.it> |
| In reply to | #1624925 |
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/srcu.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/kernel/rcu/srcu.c b/kernel/rcu/srcu.c
index d51ab050f777..5aeeaecfb673 100644
--- a/kernel/rcu/srcu.c
+++ b/kernel/rcu/srcu.c
@@ -254,10 +254,9 @@ static void srcu_gp_start(struct srcu_struct *sp)
}
/*
- * @@@ Wait until all pre-existing readers complete. Such readers
- * will have used the index specified by "idx".
- * the caller should ensures the ->completed is not changed while checking
- * and idx = (->completed & 1) ^ 1
+ * Wait until all readers counted by array index idx complete, but loop
+ * a maximum of trycount times. The caller must ensure that ->completed
+ * is not changed while checking.
*/
static bool try_check_zero(struct srcu_struct *sp, int idx, int trycount)
{
--
2.5.2
[toc] | [prev] | [next] | [standalone]
Page 1 of 5 [1] 2 3 4 5 Next page →
Back to top | Article view | linux.kernel
csiph-web