Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1360334 > unrolled thread

Re: 4.5.0+ panic when setup loop device

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: 4.5.0+ panic when setup loop device Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-03-18 05:20 +0100
    Re: 4.5.0+ panic when setup loop device Peter Zijlstra <peterz@infradead.org> - 2016-03-18 09:00 +0100
    Re: 4.5.0+ panic when setup loop device Peter Zijlstra <peterz@infradead.org> - 2016-03-18 11:20 +0100
      Re: 4.5.0+ panic when setup loop device Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-03-18 13:40 +0100
        Re: 4.5.0+ panic when setup loop device Peter Zijlstra <peterz@infradead.org> - 2016-03-18 14:40 +0100
          Re: 4.5.0+ panic when setup loop device Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-03-18 15:10 +0100
    Re: 4.5.0+ panic when setup loop device Thomas Gleixner <tglx@linutronix.de> - 2016-03-18 13:00 +0100
      Re: 4.5.0+ panic when setup loop device Mike Galbraith <umgwanakikbuti@gmail.com> - 2016-03-18 13:50 +0100
      [tip:x86/urgent] x86/topology: Use total_cpus not nr_cpu_ids for  logical packages tip-bot for Thomas Gleixner <tipbot@zytor.com> - 2016-03-19 10:40 +0100

#1360334 — Re: 4.5.0+ panic when setup loop device

On Thu, 2016-03-17 at 10:52 +0100, Peter Zijlstra wrote:

> Andreas; Borislav said to Cc you since you wrote all this.
> The issue is that Linux assumes:
> 
> 	> nr_logical_cpus = nr_cores * nr_siblings

It also seems to now assume that if SMT is possible, it's enabled.

Below is my 8 socket DL980 G7, which has SMT turned off for RT testing,
booting NOPREEMPT master tuned for maximum bloat ala distro and getting
confused by me telling it (as always) nr_cpus=64.  Bad juju ensues.

[    0.216180] max_cores: 8, cpu_ids: 64, num_siblings: 2, coreid_bits: 5
[    0.226593] smpboot: Max logical packages: 4 <== not
[    0.233742] smpboot: APIC(0) Converting physical 0 to logical package 0
[    0.244233] smpboot: APIC(20) Converting physical 1 to logical package 1
[    0.253765] smpboot: APIC(40) Converting physical 2 to logical package 2
[    0.264081] smpboot: APIC(60) Converting physical 3 to logical package 3
[    0.274827] smpboot: APIC(80) Package 4 exceeds logical package map
[    0.284705] smpboot: CPU 32 APICId 80 disabled
[    0.292277] smpboot: APIC(a0) Package 5 exceeds logical package map
[    0.302141] smpboot: CPU 40 APICId a0 disabled
[    0.308607] smpboot: APIC(c0) Package 6 exceeds logical package map
[    0.321682] smpboot: CPU 48 APICId c0 disabled
[    0.328179] smpboot: APIC(e0) Package 7 exceeds logical package map
[    0.337902] smpboot: CPU 56 APICId e0 disabled
[    0.345695] DMAR: Host address width 40
[    0.351511] DMAR: DRHD base: 0x000000b0100000 flags: 0x0
[    0.360018] DMAR: dmar0: reg_base_addr b0100000 ver 1:0 cap c90780106f0462 ecap f0207e
[    0.373342] DMAR: DRHD base: 0x000000a8000000 flags: 0x1
[    0.383164] DMAR: dmar1: reg_base_addr a8000000 ver 1:0 cap c90780106f0462 ecap f0207e
[    0.396475] DMAR: RMRR base: 0x0000007f7ee000 end: 0x0000007f7effff
[    0.407255] DMAR: RMRR base: 0x0000007f7e7000 end: 0x0000007f7ecfff
[    0.418136] DMAR: RMRR base: 0x0000007f62e000 end: 0x0000007f62ffff
[    0.429787] DMAR: ATSR flags: 0x0
[    0.434778] DMAR: ATSR flags: 0x0
[    0.441624] DMAR-IR: IOAPIC id 10 under DRHD base  0xb0100000 IOMMU 0
[    0.452716] DMAR-IR: IOAPIC id 8 under DRHD base  0xa8000000 IOMMU 1
[    0.465782] DMAR-IR: IOAPIC id 0 under DRHD base  0xa8000000 IOMMU 1
[    0.477123] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.492918] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.502549] x2apic enabled
[    0.506678] Switched APIC routing to cluster x2apic.
[    0.519955] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.642858] smpboot: CPU0: Intel(R) Xeon(R) CPU           X7560  @ 2.27GHz (family: 0x6, model: 0x2e, stepping: 0x6)
[    0.668111] Performance Events: PEBS fmt1+, 16-deep LBR, Nehalem events, Broken BIOS detected, complain to your hardware vendor.
[    0.694907] [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 38d is 330)
[    0.713186] Intel PMU driver.
[    0.719091] core: CPU erratum AAJ80 worked around
[    0.731647] core: CPUID marked event: 'bus cycles' unavailable
[    0.741499] ... version:                3
[    0.747982] ... bit width:              48
[    0.754109] ... generic registers:      4
[    0.760980] ... value mask:             0000ffffffffffff
[    0.769336] ... max period:             000000007fffffff
[    0.776913] ... fixed-purpose events:   3
[    0.783861] ... event mask:             000000070000000f
[    0.793737] x86: Booting SMP configuration:
[    0.800069] .... node  #0, CPUs:        #1  #2  #3  #4  #5  #6  #7  #8  #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #33 #34 #35 #36 #37 #38 #39 #41 #42 #43 #44 #45 #46 #47 #49 #50 #51 #5>
[    4.717309] x86: Booted up 1 node, 60 CPUs
[    4.724551] smpboot: Total of 60 processors activated (271280.00 BogoMIPS)
[    5.007438] node 0 initialised, 1013474 pages in 36ms

[toc] | [next] | [standalone]

#1360402

On Fri, Mar 18, 2016 at 05:11:54AM +0100, Mike Galbraith wrote:
> On Thu, 2016-03-17 at 10:52 +0100, Peter Zijlstra wrote:
> 
> > Andreas; Borislav said to Cc you since you wrote all this.
> > The issue is that Linux assumes:
> > 
> > 	> nr_logical_cpus = nr_cores * nr_siblings
> 
> It also seems to now assume that if SMT is possible, it's enabled.

Urgh..

What I think, with my pre wakeup brain, happens is that the CPUID
topology muck still reports 2 siblings (it tends to do that).

But the BIOS only reports APIC-IDs for all your cores, so our
nr_cpu_ids is reduced, while the nr_siblings count it not.

And them *boom*.

This is the same old problem that is nearly impossible to tell if HT is
enabled or not -- complete and utter trainwreck :/

I need to go make wake-up-juice and ponder wth to do about this.

[toc] | [prev] | [next] | [standalone]

#1360493

On Fri, Mar 18, 2016 at 05:11:54AM +0100, Mike Galbraith wrote:
> On Thu, 2016-03-17 at 10:52 +0100, Peter Zijlstra wrote:
> 
> > Andreas; Borislav said to Cc you since you wrote all this.
> > The issue is that Linux assumes:
> > 
> > 	> nr_logical_cpus = nr_cores * nr_siblings
> 
> It also seems to now assume that if SMT is possible, it's enabled.
> 
> Below is my 8 socket DL980 G7, which has SMT turned off for RT testing,
> booting NOPREEMPT master tuned for maximum bloat ala distro and getting
> confused by me telling it (as always) nr_cpus=64.  Bad juju ensues.

Ah, did you actually disable HT in the BIOS, or just skip the HT
enumeration by saying nr_cpus=64 (knowing that all the siblings are
last)?

In any case, Thomas has a clue and I'm going to test, but 4 socket
machine takes forever to boot, so might be a few minutes :/

[toc] | [prev] | [next] | [standalone]

#1360571

On Fri, 2016-03-18 at 11:15 +0100, Peter Zijlstra wrote:
> On Fri, Mar 18, 2016 at 05:11:54AM +0100, Mike Galbraith wrote:
> > On Thu, 2016-03-17 at 10:52 +0100, Peter Zijlstra wrote:
> > 
> > > Andreas; Borislav said to Cc you since you wrote all this.
> > > The issue is that Linux assumes:
> > > 
> > > 	> nr_logical_cpus = nr_cores * nr_siblings
> > 
> > It also seems to now assume that if SMT is possible, it's enabled.
> > 
> > Below is my 8 socket DL980 G7, which has SMT turned off for RT
> > testing,
> > booting NOPREEMPT master tuned for maximum bloat ala distro and
> > getting
> > confused by me telling it (as always) nr_cpus=64.  Bad juju ensues.
> 
> Ah, did you actually disable HT in the BIOS, or just skip the HT
> enumeration by saying nr_cpus=64 (knowing that all the siblings are
> last)?

It's disabled in BIOS.

> In any case, Thomas has a clue and I'm going to test, but 4 socket
> machine takes forever to boot, so might be a few minutes :/

His one-liner made my DL980 all better.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1360633

On Fri, Mar 18, 2016 at 01:39:16PM +0100, Mike Galbraith wrote:
> On Fri, 2016-03-18 at 11:15 +0100, Peter Zijlstra wrote:

> > Ah, did you actually disable HT in the BIOS, or just skip the HT
> > enumeration by saying nr_cpus=64 (knowing that all the siblings are
> > last)?
> 
> It's disabled in BIOS.

OK, so I disabled HT in the BIOS too, on my IVB-EX

> > In any case, Thomas has a clue and I'm going to test, but 4 socket
> > machine takes forever to boot, so might be a few minutes :/
> 
> His one-liner made my DL980 all better.

My machine is profoundly unhappy though; and since I have no nr_cpus=
nr_cpu_ids == total_cpus and his patch wouldn't do anything anyway.

[    0.286838] max_cores: 15, cpu_ids: 60, num_siblings: 2, coreid_bits: 5
[    0.293463] smpboot: Max logical packages: 2
[    0.297733] smpboot: APIC(0) Converting physical 0 to logical package 0
[    0.304346] smpboot: APIC(20) Converting physical 1 to logical package 1
[    0.311047] smpboot: APIC(40) Package 2 exceeds logical package map
[    0.317309] smpboot: CPU 30 APICId 40 disabled
[    0.321757] smpboot: APIC(60) Package 3 exceeds logical package map
[    0.328022] smpboot: CPU 45 APICId 60 disabled

This machine does exactly what I suspected yours did.

[toc] | [prev] | [next] | [standalone]

#1360654

On Fri, 2016-03-18 at 14:32 +0100, Peter Zijlstra wrote:
> On Fri, Mar 18, 2016 at 01:39:16PM +0100, Mike Galbraith wrote:
> > On Fri, 2016-03-18 at 11:15 +0100, Peter Zijlstra wrote:
> 
> > > Ah, did you actually disable HT in the BIOS, or just skip the HT
> > > enumeration by saying nr_cpus=64 (knowing that all the siblings are
> > > last)?
> > 
> > It's disabled in BIOS.
> 
> OK, so I disabled HT in the BIOS too, on my IVB-EX
> 
> > > In any case, Thomas has a clue and I'm going to test, but 4 socket
> > > machine takes forever to boot, so might be a few minutes :/
> > 
> > His one-liner made my DL980 all better.
> 
> My machine is profoundly unhappy though; and since I have no nr_cpus=
> nr_cpu_ids == total_cpus and his patch wouldn't do anything anyway.
> 
> [    0.286838] max_cores: 15, cpu_ids: 60, num_siblings: 2, coreid_bits: 5
> [    0.293463] smpboot: Max logical packages: 2
> [    0.297733] smpboot: APIC(0) Converting physical 0 to logical package 0
> [    0.304346] smpboot: APIC(20) Converting physical 1 to logical package 1
> [    0.311047] smpboot: APIC(40) Package 2 exceeds logical package map
> [    0.317309] smpboot: CPU 30 APICId 40 disabled
> [    0.321757] smpboot: APIC(60) Package 3 exceeds logical package map
> [    0.328022] smpboot: CPU 45 APICId 60 disabled
> 
> This machine does exactly what I suspected yours did.

Yup, that looks very familiar.  I just booted without nr_cpus=64 to
make sure it's a happy camper both w/wo, and it says all is peachy.

	-Mike  

[toc] | [prev] | [next] | [standalone]

#1360554

On Fri, 18 Mar 2016, Mike Galbraith wrote:
> On Thu, 2016-03-17 at 10:52 +0100, Peter Zijlstra wrote:
> 
> > Andreas; Borislav said to Cc you since you wrote all this.
> > The issue is that Linux assumes:
> > 
> > 	> nr_logical_cpus = nr_cores * nr_siblings
> 
> It also seems to now assume that if SMT is possible, it's enabled.
> 
> Below is my 8 socket DL980 G7, which has SMT turned off for RT testing,
> booting NOPREEMPT master tuned for maximum bloat ala distro and getting
> confused by me telling it (as always) nr_cpus=64.  Bad juju ensues.

:)

Does the patch below fix the wreckage?

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 643dbdccf4bc..c5ac71276076 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -319,7 +319,7 @@ static void __init smp_init_package_map(void)
 	 * might change in the future....
 	 */
 	ncpus = boot_cpu_data.x86_max_cores * smp_num_siblings;
-	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
 
 	/*
 	 * Possibly larger than what we need as the number of apic ids per

[toc] | [prev] | [next] | [standalone]

#1360588

On Fri, 2016-03-18 at 12:55 +0100, Thomas Gleixner wrote:

> Does the patch below fix the wreckage?

Yup, all better.

> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index 643dbdccf4bc..c5ac71276076 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -319,7 +319,7 @@ static void __init smp_init_package_map(void)
>          * might change in the future....
>          */
>         ncpus = boot_cpu_data.x86_max_cores * smp_num_siblings;
> -       __max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
> +       __max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
>  
>         /*
>          * Possibly larger than what we need as the number of apic
> ids per

[toc] | [prev] | [next] | [standalone]

#1361067 — [tip:x86/urgent] x86/topology: Use total_cpus not nr_cpu_ids for logical packages

Commit-ID:  3e8db2246b434c6b18a6a9f09904038bddcf76c7
Gitweb:     http://git.kernel.org/tip/3e8db2246b434c6b18a6a9f09904038bddcf76c7
Author:     Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Fri, 18 Mar 2016 17:20:30 +0100
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Sat, 19 Mar 2016 10:26:40 +0100

x86/topology: Use total_cpus not nr_cpu_ids for logical packages

nr_cpu_ids can be limited on the command line via nr_cpus=. That can break the
logical package management because it results in a smaller number of packages,
but the cpus to online are occupying the full package space as the hyper
threads are enumerated after the physical cores typically.

total_cpus is the real possible cpu space not limited by nr_cpus command line
and gives us the proper number of packages.

Reported-by: Mike Galbraith <umgwanakikbuti@gmail.com>
Fixes: 1f12e32f4cd5 ("x86/topology: Create logical package id")
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Xiong Zhou <jencce.kernel@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Andreas Herrmann <aherrmann@suse.com>
Link: http://lkml.kernel.org/r/alpine.DEB.2.11.1603181254330.3978@nanos
---
 arch/x86/kernel/smpboot.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 81e6a43..b2c99f8 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -325,9 +325,14 @@ static void __init smp_init_package_map(void)
 	 * By not including this we'll sometimes over-estimate the number of
 	 * logical packages by the amount of !present siblings, but this is
 	 * still better than MAX_LOCAL_APIC.
+	 *
+	 * We use total_cpus not nr_cpu_ids because nr_cpu_ids can be limited
+	 * on the command line leading to a similar issue as the HT disable
+	 * problem because the hyperthreads are usually enumerated after the
+	 * primary cores.
 	 */
 	ncpus = boot_cpu_data.x86_max_cores;
-	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
 
 	/*
 	 * Possibly larger than what we need as the number of apic ids per

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web