Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1196125 > unrolled thread

Re: [PATCH tip/core/rcu 02/12] rcu: Panic if RCU tree can not accommodate all CPUs

Started by"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
First post2015-07-30 17:30 +0200
Last post2015-07-31 18:00 +0200
Articles 5 — 2 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH tip/core/rcu 02/12] rcu: Panic if RCU tree can not  accommodate all CPUs "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-07-30 17:30 +0200
    Re: [PATCH tip/core/rcu 02/12] rcu: Panic if RCU tree can not  accommodate all CPUs Peter Zijlstra <peterz@infradead.org> - 2015-07-30 17:40 +0200
      Re: [PATCH tip/core/rcu 02/12] rcu: Panic if RCU tree can not  accommodate all CPUs Peter Zijlstra <peterz@infradead.org> - 2015-07-30 17:40 +0200
      Re: [PATCH tip/core/rcu 02/12] rcu: Panic if RCU tree can not  accommodate all CPUs "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-07-30 18:00 +0200
        Re: [PATCH tip/core/rcu 02/12] rcu: Panic if RCU tree can not  accommodate all CPUs "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-07-31 18:00 +0200

#1196125 — Re: [PATCH tip/core/rcu 02/12] rcu: Panic if RCU tree can not accommodate all CPUs

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2015-07-30 17:30 +0200
SubjectRe: [PATCH tip/core/rcu 02/12] rcu: Panic if RCU tree can not accommodate all CPUs
Message-ID<pRXWq-2jR-3@gated-at.bofh.it>
On Thu, Jul 30, 2015 at 02:28:35PM +0200, Peter Zijlstra wrote:
> On Fri, Jul 17, 2015 at 03:30:53PM -0700, Paul E. McKenney wrote:
> > +	if (n > rcu_capacity[MAX_RCU_LVLS])
> > +		panic("rcu_init_geometry: rcu_capacity[] is too small");
> 
> How can this ever happen? We _know_ NR_CPUS at compile time, there's no
> way we can get more CPUs than that -- even if the hardware has more,
> we'll stop enumerating.

You can make this happen by building with CONFIG_RCU_FANOUT=2 and
CONFIG_RCU_FANOUT_LEAF=2, then running on a system with more than 16 CPUs.
The kernel boot parameter rcutree.rcu_fanout_leaf=2 can be substituted for
CONFIG_RCU_FANOUT_LEAF=2, hence the need for a runtime test.  I do this
sort of thing for my rcutorture testing in order to test a four-level
rcu_node tree with only 16 CPUs.

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1196135

FromPeter Zijlstra <peterz@infradead.org>
Date2015-07-30 17:40 +0200
Message-ID<pRY65-2v3-7@gated-at.bofh.it>
In reply to#1196125
On Thu, Jul 30, 2015 at 08:25:17AM -0700, Paul E. McKenney wrote:
> On Thu, Jul 30, 2015 at 02:28:35PM +0200, Peter Zijlstra wrote:
> > On Fri, Jul 17, 2015 at 03:30:53PM -0700, Paul E. McKenney wrote:
> > > +	if (n > rcu_capacity[MAX_RCU_LVLS])
> > > +		panic("rcu_init_geometry: rcu_capacity[] is too small");
> > 
> > How can this ever happen? We _know_ NR_CPUS at compile time, there's no
> > way we can get more CPUs than that -- even if the hardware has more,
> > we'll stop enumerating.
> 
> You can make this happen by building with CONFIG_RCU_FANOUT=2 and
> CONFIG_RCU_FANOUT_LEAF=2, then running on a system with more than 16 CPUs.
> The kernel boot parameter rcutree.rcu_fanout_leaf=2 can be substituted for
> CONFIG_RCU_FANOUT_LEAF=2, hence the need for a runtime test.  I do this
> sort of thing for my rcutorture testing in order to test a four-level
> rcu_node tree with only 16 CPUs.

How about we make the build fail if NR_CPUS exceeds that maximum fanout?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1196137

FromPeter Zijlstra <peterz@infradead.org>
Date2015-07-30 17:40 +0200
Message-ID<pRY66-2v3-17@gated-at.bofh.it>
In reply to#1196135
On Thu, Jul 30, 2015 at 05:32:51PM +0200, Peter Zijlstra wrote:
> On Thu, Jul 30, 2015 at 08:25:17AM -0700, Paul E. McKenney wrote:
> > On Thu, Jul 30, 2015 at 02:28:35PM +0200, Peter Zijlstra wrote:
> > > On Fri, Jul 17, 2015 at 03:30:53PM -0700, Paul E. McKenney wrote:
> > > > +	if (n > rcu_capacity[MAX_RCU_LVLS])
> > > > +		panic("rcu_init_geometry: rcu_capacity[] is too small");
> > > 
> > > How can this ever happen? We _know_ NR_CPUS at compile time, there's no
> > > way we can get more CPUs than that -- even if the hardware has more,
> > > we'll stop enumerating.
> > 
> > You can make this happen by building with CONFIG_RCU_FANOUT=2 and
> > CONFIG_RCU_FANOUT_LEAF=2, then running on a system with more than 16 CPUs.
> > The kernel boot parameter rcutree.rcu_fanout_leaf=2 can be substituted for
> > CONFIG_RCU_FANOUT_LEAF=2, hence the need for a runtime test.  I do this
> > sort of thing for my rcutorture testing in order to test a four-level
> > rcu_node tree with only 16 CPUs.
> 
> How about we make the build fail if NR_CPUS exceeds that maximum fanout?

Or better yet, auto-select these numbers based on NR_CPUS and remove the
configs.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1196153

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2015-07-30 18:00 +0200
Message-ID<pRYps-2RD-15@gated-at.bofh.it>
In reply to#1196135
On Thu, Jul 30, 2015 at 05:32:51PM +0200, Peter Zijlstra wrote:
> On Thu, Jul 30, 2015 at 08:25:17AM -0700, Paul E. McKenney wrote:
> > On Thu, Jul 30, 2015 at 02:28:35PM +0200, Peter Zijlstra wrote:
> > > On Fri, Jul 17, 2015 at 03:30:53PM -0700, Paul E. McKenney wrote:
> > > > +	if (n > rcu_capacity[MAX_RCU_LVLS])
> > > > +		panic("rcu_init_geometry: rcu_capacity[] is too small");
> > > 
> > > How can this ever happen? We _know_ NR_CPUS at compile time, there's no
> > > way we can get more CPUs than that -- even if the hardware has more,
> > > we'll stop enumerating.
> > 
> > You can make this happen by building with CONFIG_RCU_FANOUT=2 and
> > CONFIG_RCU_FANOUT_LEAF=2, then running on a system with more than 16 CPUs.
> > The kernel boot parameter rcutree.rcu_fanout_leaf=2 can be substituted for
> > CONFIG_RCU_FANOUT_LEAF=2, hence the need for a runtime test.  I do this
> > sort of thing for my rcutorture testing in order to test a four-level
> > rcu_node tree with only 16 CPUs.
> 
> How about we make the build fail if NR_CPUS exceeds that maximum fanout?

Good point, and it already does, and I clearly was confused, apologies.

So the real way to make this happen is (for example) to build
with CONFIG_RCU_FANOUT=2 and CONFIG_RCU_FANOUT_LEAF=16 (the
default), which could accommodate up to 128 CPUs.  Then boot with
rcutree.rcu_fanout_leaf=2 on a system with more than 16 CPUs, with
rcutree.rcu_fanout_leaf=3 on a system with more than 24 CPUs, and so on.

Of course, the truly macho way to get this error message is to build
with CONFIG_RCU_FANOUT=64 and CONFIG_RCU_FANOUT_LEAF=64, then boot with
rcutree.rcu_fanout_leaf=63 on a system with more than 16,515,072 CPUs.
Of course, you get serious style points if the system manages to stay
up for more than 24 hours without a hardware failure.  ;-)

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1197153

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2015-07-31 18:00 +0200
Message-ID<pSkT1-1TL-11@gated-at.bofh.it>
In reply to#1196153
On Thu, Jul 30, 2015 at 06:22:02PM +0200, Peter Zijlstra wrote:
> On Thu, Jul 30, 2015 at 08:54:54AM -0700, Paul E. McKenney wrote:
> 
> > Good point, and it already does, and I clearly was confused, apologies.
> > 
> > So the real way to make this happen is (for example) to build
> > with CONFIG_RCU_FANOUT=2 and CONFIG_RCU_FANOUT_LEAF=16 (the
> > default), which could accommodate up to 128 CPUs.  Then boot with
> > rcutree.rcu_fanout_leaf=2 on a system with more than 16 CPUs, with
> > rcutree.rcu_fanout_leaf=3 on a system with more than 24 CPUs, and so on.
> 
> Ah, runtime overrides and operator error, but then we can WARN(), reset
> the arguments and try again, right? No need to panic the machine and
> fail to boot.

Good point, like the patch below?  Which also legitimizes my
example after the fact, as it previously simply prohibited having
rcutree.rcu_fanout_leaf less than CONFIG_RCU_FANOUT_LEAF.  :-/

> > Of course, the truly macho way to get this error message is to build
> > with CONFIG_RCU_FANOUT=64 and CONFIG_RCU_FANOUT_LEAF=64, then boot with
> > rcutree.rcu_fanout_leaf=63 on a system with more than 16,515,072 CPUs.
> > Of course, you get serious style points if the system manages to stay
> > up for more than 24 hours without a hardware failure.  ;-)
> 
> Yes, I'll go power up the nuclear reactor in the basement first :-)

Only one?  ;-)

							Thanx, Paul

------------------------------------------------------------------------

    rcu: Eliminate panic when silly boot-time fanout specified
    
    This commit loosens rcutree.rcu_fanout_leaf range checks
    and replaces a panic() with a fallback to compile-time values.
    This fallback is accompanied by a WARN_ON(), and both occur when the
    rcutree.rcu_fanout_leaf value is too small to accommodate the number of
    CPUs.  For example, given the current four-level limit for the rcu_node
    tree, a system with more than 16 CPUs built with CONFIG_FANOUT=2 must
    have rcutree.rcu_fanout_leaf larger than 2.
    
    Reported-by: Peter Zijlstra <peterz@infradead.org>
    Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 01b5b68a237a..2a5d4696bdb9 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -3059,9 +3059,12 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			cache-to-cache transfer latencies.
 
 	rcutree.rcu_fanout_leaf= [KNL]
-			Increase the number of CPUs assigned to each
-			leaf rcu_node structure.  Useful for very large
-			systems.
+			Change the number of CPUs assigned to each
+			leaf rcu_node structure.  Useful for very
+			large systems, which will choose the value 64,
+			and for NUMA systems with large remote-access
+			latencies, which will choose a value aligned
+			with the appropriate hardware boundaries.
 
 	rcutree.jiffies_till_sched_qs= [KNL]
 			Set required age in jiffies for a
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index ce43fac5ff91..9f8040396d3e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4216,13 +4216,12 @@ static void __init rcu_init_geometry(void)
 		rcu_fanout_leaf, nr_cpu_ids);
 
 	/*
-	 * The boot-time rcu_fanout_leaf parameter is only permitted
-	 * to increase the leaf-level fanout, not decrease it.  Of course,
-	 * the leaf-level fanout cannot exceed the number of bits in
-	 * the rcu_node masks.  Complain and fall back to the compile-
-	 * time values if these limits are exceeded.
+	 * The boot-time rcu_fanout_leaf parameter must be at least two
+	 * and cannot exceed the number of bits in the rcu_node masks.
+	 * Complain and fall back to the compile-time values if this
+	 * limit is exceeded.
 	 */
-	if (rcu_fanout_leaf < RCU_FANOUT_LEAF ||
+	if (rcu_fanout_leaf < 2 ||
 	    rcu_fanout_leaf > sizeof(unsigned long) * 8) {
 		rcu_fanout_leaf = RCU_FANOUT_LEAF;
 		WARN_ON(1);
@@ -4239,10 +4238,13 @@ static void __init rcu_init_geometry(void)
 
 	/*
 	 * The tree must be able to accommodate the configured number of CPUs.
-	 * If this limit is exceeded than we have a serious problem elsewhere.
+	 * If this limit is exceeded, fall back to the compile-time values.
 	 */
-	if (nr_cpu_ids > rcu_capacity[RCU_NUM_LVLS - 1])
-		panic("rcu_init_geometry: rcu_capacity[] is too small");
+	if (nr_cpu_ids > rcu_capacity[RCU_NUM_LVLS - 1]) {
+		rcu_fanout_leaf = RCU_FANOUT_LEAF;
+		WARN_ON(1);
+		return;
+	}
 
 	/* Calculate the number of levels in the tree. */
 	for (i = 0; nr_cpu_ids > rcu_capacity[i]; i++) {

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web