Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1376932

[PATCH tip/core/rcu 03/16] rcu: Make cond_resched_rcu_qs() supply RCU-sched expedited QS

Show all headers | View raw

Although cond_resched_rcu_qs() supplies quiescent states to all flavors
of normal RCU grace periods, it does nothing for expedited RCU-sched
grace periods.  This commit therefore adds a check for a need for a
quiescent state from the current CPU by an expedited RCU-sched grace
period, and invokes rcu_sched_qs() to supply that quiescent state if so.

Note that the check is racy in that we might be migrated to some other
CPU just after checking the per-CPU variable.  This is OK because the
act of migration will do a context switch, which will supply the needed
quiescent state.  The only downside is that we might do an unnecessary
call to rcu_sched_qs(), but the probability is low and the overhead
is small.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcu/tree.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 687d8a5f35c7..178575c01d09 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -370,6 +370,21 @@ void rcu_all_qs(void)
 		rcu_momentary_dyntick_idle();
 		local_irq_restore(flags);
 	}
+	if (unlikely(raw_cpu_read(rcu_sched_data.cpu_no_qs.b.exp))) {
+		/*
+		 * Yes, we just checked a per-CPU variable with preemption
+		 * enabled, so we might be migrated to some other CPU at
+		 * this point.  That is OK because in that case, the
+		 * migration will supply the needed quiescent state.
+		 * We might end up needlessly disabling preemption and
+		 * invoking rcu_sched_qs() on the destination CPU, but
+		 * the probability and cost are both quite low, so this
+		 * should not be a problem in practice.
+		 */
+		preempt_disable();
+		rcu_sched_qs();
+		preempt_enable();
+	}
 	this_cpu_inc(rcu_qs_ctr);
 	barrier(); /* Avoid RCU read-side critical sections leaking up. */
 }
-- 
2.5.2

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH tip/core/rcu 0/16] Expedited grace-period changes for 4.7 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:30 +0200
  [PATCH tip/core/rcu 07/16] rcu: Add expedited-grace-period event tracing "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:30 +0200
  [PATCH tip/core/rcu 02/16] rcu: Make expedited RCU-preempt stall warnings count accurately "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:30 +0200
  [PATCH tip/core/rcu 10/16] rcu: Shorten expedited_workdone* to exp_workdone* "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:30 +0200
  [PATCH tip/core/rcu 09/16] rcu: Force boolean subscript for expedited stall warnings "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:30 +0200
  [PATCH tip/core/rcu 08/16] rcu: Remove expedited GP funnel-lock bypass "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 05/16] rcu: Add event tracing definitions for expedited grace periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 03/16] rcu: Make cond_resched_rcu_qs() supply RCU-sched expedited QS "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 06/16] rcu: Add funnel-locking tracing for expedited grace periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 14/16] rcu: Consolidate expedited GP tracing into rcu_exp_gp_seq_snap() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 01/16] rcu: Make expedited RCU-sched grace period immediately detect idle "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 15/16] rcu: Consolidate expedited GP code into exp_funnel_lock() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 16/16] rcu: Overlap wakeups with next expedited grace period "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 12/16] rcu: Add exp_funnel_lock() fastpath "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 11/16] rcu: Enforce expedited-GP fairness via funnel wait queue "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 04/16] rcu: Fix synchronize_rcu_expedited() header comment "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200
  [PATCH tip/core/rcu 13/16] rcu: Consolidate expedited GP code into rcu_exp_wait_wake() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-04-12 17:40 +0200

csiph-web