Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1241646
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period |
| Date | 2015-10-07 18:20 +0200 |
| Message-ID | <qgZBD-1xt-5@gated-at.bofh.it> (permalink) |
| References | <qgDhN-2Zi-27@gated-at.bofh.it> <qgDhN-2Zi-29@gated-at.bofh.it> <qgDru-3aC-61@gated-at.bofh.it> <qgXgt-6rH-15@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
On Wed, Oct 07, 2015 at 03:43:11PM +0200, Peter Zijlstra wrote:
> On Tue, Oct 06, 2015 at 09:29:23AM -0700, Paul E. McKenney wrote:
> > @@ -167,42 +307,18 @@ static void rcu_preempt_note_context_switch(void)
>
> > - raw_spin_unlock_irqrestore(&rnp->lock, flags);
>
> This again reminds me that we should move rcu_note_context_switch()
> under the IRQ disable section the scheduler already has or remove the
> IRQ disable from rcu_note_context_switch().
Like this? I verified that all callers of rcu_virt_note_context_switch()
currently have interrupts disabled.
Thanx, Paul
------------------------------------------------------------------------
commit 945c702687f760872adf7ce6e030c81ba427bf34
Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Date: Wed Oct 7 09:10:48 2015 -0700
rcu: Stop disabling interrupts in scheduler fastpaths
We need the scheduler's fastpaths to be, well, fast, and unnecessarily
disabling and re-enabling interrupts is not necessarily consistent with
this goal. Especially given that there are regions of the scheduler that
already have interrupts disabled.
This commit therefore moves the call to rcu_note_context_switch()
to one of the interrupts-disabled regions of the scheduler, and
removes the now-redundant disabling and re-enabling of interrupts from
rcu_note_context_switch() and the functions it calls.
Reported-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
index 60d15a080d7c..9d3eda39bcd2 100644
--- a/include/linux/rcutree.h
+++ b/include/linux/rcutree.h
@@ -37,7 +37,7 @@ void rcu_cpu_stall_reset(void);
/*
* Note a virtualization-based context switch. This is simply a
* wrapper around rcu_note_context_switch(), which allows TINY_RCU
- * to save a few bytes.
+ * to save a few bytes. The caller must have disabled interrupts.
*/
static inline void rcu_virt_note_context_switch(int cpu)
{
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 3dd63ebf279a..1f81a960678a 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -295,17 +295,16 @@ EXPORT_PER_CPU_SYMBOL_GPL(rcu_qs_ctr);
* We inform the RCU core by emulating a zero-duration dyntick-idle
* period, which we in turn do by incrementing the ->dynticks counter
* by two.
+ *
+ * The caller must have disabled interrupts.
*/
static void rcu_momentary_dyntick_idle(void)
{
- unsigned long flags;
struct rcu_data *rdp;
struct rcu_dynticks *rdtp;
int resched_mask;
struct rcu_state *rsp;
- local_irq_save(flags);
-
/*
* Yes, we can lose flag-setting operations. This is OK, because
* the flag will be set again after some delay.
@@ -335,13 +334,12 @@ static void rcu_momentary_dyntick_idle(void)
smp_mb__after_atomic(); /* Later stuff after QS. */
break;
}
- local_irq_restore(flags);
}
/*
* Note a context switch. This is a quiescent state for RCU-sched,
* and requires special handling for preemptible RCU.
- * The caller must have disabled preemption.
+ * The caller must have disabled interrupts.
*/
void rcu_note_context_switch(void)
{
@@ -371,9 +369,14 @@ EXPORT_SYMBOL_GPL(rcu_note_context_switch);
*/
void rcu_all_qs(void)
{
+ unsigned long flags;
+
barrier(); /* Avoid RCU read-side critical sections leaking down. */
- if (unlikely(raw_cpu_read(rcu_sched_qs_mask)))
+ if (unlikely(raw_cpu_read(rcu_sched_qs_mask))) {
+ local_irq_save(flags);
rcu_momentary_dyntick_idle();
+ local_irq_restore(flags);
+ }
this_cpu_inc(rcu_qs_ctr);
barrier(); /* Avoid RCU read-side critical sections leaking up. */
}
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 97dfa7d57f79..7087fb047e2d 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -146,8 +146,8 @@ static void __init rcu_bootup_announce(void)
* the corresponding expedited grace period will also be the end of the
* normal grace period.
*/
-static void rcu_preempt_ctxt_queue(struct rcu_node *rnp, struct rcu_data *rdp,
- unsigned long flags) __releases(rnp->lock)
+static void rcu_preempt_ctxt_queue(struct rcu_node *rnp, struct rcu_data *rdp)
+ __releases(rnp->lock) /* But leaves rrupts disabled. */
{
int blkd_state = (rnp->gp_tasks ? RCU_GP_TASKS : 0) +
(rnp->exp_tasks ? RCU_EXP_TASKS : 0) +
@@ -235,7 +235,7 @@ static void rcu_preempt_ctxt_queue(struct rcu_node *rnp, struct rcu_data *rdp,
rnp->gp_tasks = &t->rcu_node_entry;
if (!rnp->exp_tasks && (blkd_state & RCU_EXP_BLKD))
rnp->exp_tasks = &t->rcu_node_entry;
- raw_spin_unlock(&rnp->lock);
+ raw_spin_unlock(&rnp->lock); /* rrupts remain disabled. */
/*
* Report the quiescent state for the expedited GP. This expedited
@@ -250,7 +250,6 @@ static void rcu_preempt_ctxt_queue(struct rcu_node *rnp, struct rcu_data *rdp,
} else {
WARN_ON_ONCE(t->rcu_read_unlock_special.b.exp_need_qs);
}
- local_irq_restore(flags);
}
/*
@@ -285,12 +284,11 @@ static void rcu_preempt_qs(void)
* predating the current grace period drain, in other words, until
* rnp->gp_tasks becomes NULL.
*
- * Caller must disable preemption.
+ * Caller must disable interrupts.
*/
static void rcu_preempt_note_context_switch(void)
{
struct task_struct *t = current;
- unsigned long flags;
struct rcu_data *rdp;
struct rcu_node *rnp;
@@ -300,7 +298,7 @@ static void rcu_preempt_note_context_switch(void)
/* Possibly blocking in an RCU read-side critical section. */
rdp = this_cpu_ptr(rcu_state_p->rda);
rnp = rdp->mynode;
- raw_spin_lock_irqsave(&rnp->lock, flags);
+ raw_spin_lock(&rnp->lock); /* rrupts already disabled. */
smp_mb__after_unlock_lock();
t->rcu_read_unlock_special.b.blocked = true;
t->rcu_blocked_node = rnp;
@@ -317,7 +315,7 @@ static void rcu_preempt_note_context_switch(void)
(rnp->qsmask & rdp->grpmask)
? rnp->gpnum
: rnp->gpnum + 1);
- rcu_preempt_ctxt_queue(rnp, rdp, flags);
+ rcu_preempt_ctxt_queue(rnp, rdp);
} else if (t->rcu_read_lock_nesting < 0 &&
t->rcu_read_unlock_special.s) {
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index c4e607873d6f..ef374ad506f0 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3056,7 +3056,6 @@ static void __sched __schedule(void)
cpu = smp_processor_id();
rq = cpu_rq(cpu);
- rcu_note_context_switch();
prev = rq->curr;
schedule_debug(prev);
@@ -3072,6 +3071,7 @@ static void __sched __schedule(void)
smp_mb__before_spinlock();
raw_spin_lock_irq(&rq->lock);
lockdep_pin_lock(&rq->lock);
+ rcu_note_context_switch();
rq->clock_skip_update <<= 1; /* promote REQ to ACT */
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCH tip/core/rcu 0/18] Expedited grace-period improvements for 4.4 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:30 +0200
[PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() Peter Zijlstra <peterz@infradead.org> - 2015-10-07 16:30 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 18:30 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() Peter Zijlstra <peterz@infradead.org> - 2015-10-08 11:10 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-08 17:10 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() Peter Zijlstra <peterz@infradead.org> - 2015-10-08 17:20 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-08 17:20 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() Josh Triplett <josh@joshtriplett.org> - 2015-10-08 20:10 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-09 02:20 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() Josh Triplett <josh@joshtriplett.org> - 2015-10-09 02:50 +0200
Re: [PATCH tip/core/rcu 18/18] rcu: Better hotplug handling for synchronize_sched_expedited() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-09 06:00 +0200
[PATCH tip/core/rcu 10/18] rcu: Stop silencing lockdep false positive for expedited grace periods "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 08/18] rcu: Make ->cpu_no_qs be a union for aggregate OR "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-06 22:30 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 23:00 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-07 10:00 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Mathieu Desnoyers <mathieu.desnoyers@efficios.com> - 2015-10-07 10:50 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-07 13:10 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-07 14:00 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-07 14:10 +0200
Re: Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation kbuild test robot <lkp@intel.com> - 2015-10-07 14:10 +0200
Re: Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation kbuild test robot <lkp@intel.com> - 2015-10-07 14:10 +0200
Re: Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-07 14:20 +0200
Re: [kbuild-all] [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Fengguang Wu <lkp@intel.com> - 2015-10-07 15:50 +0200
Re: [kbuild-all] [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-07 16:00 +0200
Re: [kbuild-all] [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-07 16:30 +0200
Re: [kbuild-all] [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Fengguang Wu <lkp@intel.com> - 2015-10-07 16:30 +0200
Re: Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation kbuild test robot <lkp@intel.com> - 2015-10-07 14:20 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 17:20 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-08 12:30 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 17:20 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 16:40 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-07 16:50 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 18:50 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-08 11:50 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-08 17:40 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-08 19:20 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-08 19:50 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-09 02:20 +0200
Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Peter Zijlstra <peterz@infradead.org> - 2015-10-09 10:50 +0200
[PATCH tip/core/rcu 05/18] rcu: Move synchronize_sched_expedited() to combining tree "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 16/18] rcu: Add tasks to expedited stall-warning messages "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 12/18] cpu: Remove try_get_online_cpus() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 03/18] rcu: Consolidate tree setup for synchronize_rcu_expedited() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 13/18] rcu: Prepare for consolidating expedited CPU selection "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 07/18] rcu: Invert passed_quiesce and rename to cpu_no_qs "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period Peter Zijlstra <peterz@infradead.org> - 2015-10-07 15:30 +0200
Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 20:20 +0200
Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period Peter Zijlstra <peterz@infradead.org> - 2015-10-07 15:40 +0200
Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 17:50 +0200
Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period Peter Zijlstra <peterz@infradead.org> - 2015-10-07 15:50 +0200
Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 18:20 +0200
Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period Peter Zijlstra <peterz@infradead.org> - 2015-10-08 11:10 +0200
Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period Peter Zijlstra <peterz@infradead.org> - 2015-10-07 15:50 +0200
Re: [PATCH tip/core/rcu 04/18] rcu: Use single-stage IPI algorithm for RCU expedited grace period "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 18:20 +0200
[PATCH tip/core/rcu 09/18] rcu: Switch synchronize_sched_expedited() to IPI "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
Re: [PATCH tip/core/rcu 09/18] rcu: Switch synchronize_sched_expedited() to IPI Peter Zijlstra <peterz@infradead.org> - 2015-10-07 16:20 +0200
Re: [PATCH tip/core/rcu 09/18] rcu: Switch synchronize_sched_expedited() to IPI "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-07 18:30 +0200
[PATCH tip/core/rcu 15/18] rcu: Add online/offline info to expedited stall warning message "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 14/18] rcu: Consolidate expedited CPU selection "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 01/18] rcu: Use rsp->expedited_wq instead of sync_rcu_preempt_exp_wq "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 06/18] rcu: Rename qs_pending to core_needs_qs "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
[PATCH tip/core/rcu 17/18] rcu: Enable stall warnings for synchronize_rcu_expedited() "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-06 18:40 +0200
csiph-web