Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1284333 > unrolled thread

[Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

Started byAshok Raj <ashok.raj@intel.com>
First post2015-12-05 00:30 +0100
Last post2015-12-07 21:10 +0100
Articles 13 — 4 participants

Back to article view | Back to linux.kernel

Contents

  [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce  rendezvous process. Ashok Raj <ashok.raj@intel.com> - 2015-12-05 00:30 +0100
    RE: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. "Luck, Tony" <tony.luck@intel.com> - 2015-12-07 21:10 +0100
      Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. Borislav Petkov <bp@alien8.de> - 2015-12-07 21:30 +0100
        RE: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. "Luck, Tony" <tony.luck@intel.com> - 2015-12-07 23:10 +0100
          Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. Borislav Petkov <bp@alien8.de> - 2015-12-07 23:40 +0100
            Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. "Raj, Ashok" <ashok.raj@intel.com> - 2015-12-07 23:50 +0100
              Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. Borislav Petkov <bp@alien8.de> - 2015-12-08 00:30 +0100
                Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. "Raj, Ashok" <ashok.raj@intel.com> - 2015-12-08 01:50 +0100
                  Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. Borislav Petkov <bp@alien8.de> - 2015-12-08 10:20 +0100
                    RE: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. "Luck, Tony" <tony.luck@intel.com> - 2015-12-08 17:10 +0100
                      Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. Borislav Petkov <bp@alien8.de> - 2015-12-08 20:00 +0100
            Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. "Luck, Tony" <tony.luck@intel.com> - 2015-12-08 00:30 +0100
    Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in  mce rendezvous process. Borislav Petkov <bp@alien8.de> - 2015-12-07 21:10 +0100

#1284333 — [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

FromAshok Raj <ashok.raj@intel.com>
Date2015-12-05 00:30 +0100
Subject[Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qC7Xz-Qn-5@gated-at.bofh.it>
Linux has logical cpu offline capability. That can be triggered by:

# echo 0 > /sys/devices/system/cpu/cpuX/online

In Intel Architecture, MCE's are broadcasted to all CPUs in the system.

This includes the CPUs marked offline by Linux. Unless the CPU's were removed
via an ACPI notification, in which case the cpu's are removed from the
cpu_present_map.

This patch ensures offline CPU's don't participate in MCE rendezvous, but
simply perform clearing some status bits to ensure a second MCE wont cause
automatic shutdown.

Without the patch, mce_start will increment mce_callin, but mce_start would
only wait for all online_cpus. So offline cpu's should avoid participating
in the rendezvous process.

Reviewed-by: Tony Luck <tony.luck@intel.com>
Cc: stable@vger.kernel.org
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
---
 arch/x86/kernel/cpu/mcheck/mce.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index c5b0d56..23ecb1d 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -998,7 +998,20 @@ void do_machine_check(struct pt_regs *regs, long error_code)
 	u64 recover_paddr = ~0ull;
 	int flags = MF_ACTION_REQUIRED;
 	int lmce = 0;
+	unsigned int cpu = smp_processor_id();
+
+	/*
+	 * if this cpu is offline, just bail out.
+	 */
+	if (cpu_is_offline(cpu)) {
+		u64 mcgstatus;
 
+		mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
+		if (mcgstatus & MCG_STATUS_RIPV) {
+			mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
+			return;
+		}
+	}
 	ist_enter(regs);
 
 	this_cpu_inc(mce_exception_count);
-- 
2.4.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1285882 — RE: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

From"Luck, Tony" <tony.luck@intel.com>
Date2015-12-07 21:10 +0100
SubjectRE: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDagG-Lz-5@gated-at.bofh.it>
In reply to#1284333
PiAgS2VybmVsIHBhbmljIC0gbm90IHN5bmNpbmc6IFRpbWVvdXQ6IE5vdCBhbGwgQ1BVcyBlbnRl
cmVkIGJyb2FkY2FzdCBleGNlcHRpb24gaGFuZGxlcg0KDQpJcyB0aGF0IHdoYXQgd2UgcHJpbnRl
ZCBpbiB0aGlzIGNhc2U/IC4uLiBib3kgaXMgdGhhdCBhIG1pc2xlYWRpbmcgbWVzc2FnZSAuLi4g
d2UgZ290ICpleHRyYSoNCmNwdXMgKHRoZSBvZmZsaW5lIG9uZXMpLCBub3QgIk5vdCBhbGwiLg0K
DQpHb29kIGpvYiB3ZSBoYXZlIGEgZml4IDotKQ0KDQotVG9ueQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1285901 — Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

FromBorislav Petkov <bp@alien8.de>
Date2015-12-07 21:30 +0100
SubjectRe: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDaA1-T7-7@gated-at.bofh.it>
In reply to#1285882
On Mon, Dec 07, 2015 at 08:04:30PM +0000, Luck, Tony wrote:
> >  Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
> 
> Is that what we printed in this case? ... boy is that a misleading message ... we got *extra*
> cpus (the offline ones), not "Not all".
> 
> Good job we have a fix :-)

Well, we still have that printk string in there.

And that is incorrect too, because the MCE (at least the one I'm
injecting) gets broadcasted to the CPUs on the *node* and not to the
whole system.

If we had to be precise, text should say "Not all CPUs which the MCE was
broadcasted to entered the exception handler..."

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1286001 — RE: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

From"Luck, Tony" <tony.luck@intel.com>
Date2015-12-07 23:10 +0100
SubjectRE: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDc8O-26T-29@gated-at.bofh.it>
In reply to#1285901
PiBBbmQgdGhhdCBpcyBpbmNvcnJlY3QgdG9vLCBiZWNhdXNlIHRoZSBNQ0UgKGF0IGxlYXN0IHRo
ZSBvbmUgSSdtDQo+IGluamVjdGluZykgZ2V0cyBicm9hZGNhc3RlZCB0byB0aGUgQ1BVcyBvbiB0
aGUgKm5vZGUqIGFuZCBub3QgdG8gdGhlDQo+IHdob2xlIHN5c3RlbS4NCg0KV2hpY2ggc3lzdGVt
PyAgV2hhdCBraW5kIG9mIG1hY2hpbmUgY2hlY2s/ICBPbiBJbnRlbCB3ZSBleHBlY3QgbWFjaGlu
ZSBjaGVja3MNCnRvIGJlIGJyb2FkY2FzdCB0byBhbGwgbG9naWNhbCBjcHVzIG9uIGFsbCBub2Rl
cyAodW5sZXNzIGxvY2FsIG1hY2hpbmUgY2hlY2sgaXMgZW5hYmxlZCwNCmluIHdoaWNoIGNhc2Ug
U1JBUiBzdHlsZSBtYWNoaW5lIGNoZWNrcyBnbyBvbmx5IHRvIHRoZSBsb2dpY2FsIGNwdSB0aGF0
IGhpdCB0aGUgZXJyb3IpLg0KDQpUaGUgY29kZSBpcyB3cml0dGVuIHRvIHRoYXQgZXhwZWN0YXRp
b24gLi4uIGFuZCB3ZSBkb24ndCByZXBvcnQgdGhpbmdzIGFzIHdlbGwgaWYNCnNvbWV0aGluZyBl
bHNlIGhhcHBlbnMgKGxpa2UgdG9vIG1hbnkgb3IgdG9vIGZldyBjcHVzIHNob3dpbmcgdXApLg0K
DQotVG9ueQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1286058 — Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

FromBorislav Petkov <bp@alien8.de>
Date2015-12-07 23:40 +0100
SubjectRe: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDcBR-2jb-31@gated-at.bofh.it>
In reply to#1286001
On Mon, Dec 07, 2015 at 10:07:59PM +0000, Luck, Tony wrote:
> > And that is incorrect too, because the MCE (at least the one I'm
> > injecting) gets broadcasted to the CPUs on the *node* and not to the
> > whole system.
> 
> Which system?  What kind of machine check?  On Intel we expect machine checks
> to be broadcast to all logical cpus on all nodes (unless local machine check is enabled,
> in which case SRAR style machine checks go only to the logical cpu that hit the error).
> 
> The code is written to that expectation ... and we don't report things as well if
> something else happens (like too many or too few cpus showing up).

Box logs below.

BIOS is doing funny cores enumeration:

node #0, CPUs 0-7
node #1, CPUs 8-15
node #2, CPUs 16-23
node #3, CPUs 24-31

and then starts from node 0 again:

 .... node  #0, CPUs:    #32  #33  #34  #35  #36  #37  #38  #39
 .... node  #1, CPUs:    #40  #41  #42  #43  #44  #45  #46  #47
 .... node  #2, CPUs:    #48  #49  #50  #51  #52  #53  #54  #55
 .... node  #3, CPUs:    #56  #57  #58  #59  #60  #61  #62  #63

So I went and offlined cores 5 and 34 which are on node 0.

Why node 0? Well, when I inject error type 0x10 which is

0x00000010      Memory Uncorrectable non-fatal

it generates an MCE only on the node 0 cores. For that log see the end
of this mail. The gist of it is that the CPUs on which #MC gets raised
are the cores on node 0, i.e., 0-7 and 32-39.

Cores 5 and 34 are gone, of course.

I mean, even if the #MC gets raised only on the node, the fix still
works.

$ grep -Ei "hardware.*CPU" /tmp/mce | sed 's/^.*CPU//' | sort -n
 0: Machine Check Exception: 5 Bank 5: be00000000010090
 1: Machine Check Exception: 5 Bank 5: be00000000010090
 2: Machine Check Exception: 5 Bank 5: be00000000010090
 3: Machine Check Exception: 5 Bank 5: be00000000010090
 4: Machine Check Exception: 5 Bank 5: be00000000010090
 6: Machine Check Exception: 5 Bank 5: be00000000010090
 7: Machine Check Exception: 5 Bank 5: be00000000010090
 32: Machine Check Exception: 5 Bank 5: be00000000010090
 33: Machine Check Exception: 5 Bank 5: be00000000010090
 35: Machine Check Exception: 5 Bank 5: be00000000010090
 36: Machine Check Exception: 5 Bank 5: be00000000010090
 37: Machine Check Exception: 5 Bank 5: be00000000010090
 38: Machine Check Exception: 5 Bank 5: be00000000010090
 39: Machine Check Exception: 5 Bank 5: be00000000010090




[    0.859060] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-4650 0 @ 2.70GHz (family: 0x6, model: 0x2d, stepping: 0x7
...
[    0.981593] x86: Booting SMP configuration:
[    0.991092] .... node  #0, CPUs:          #1
[    1.013485] microcode: CPU1 microcode updated early to revision 0x710, date = 2013-06-17
[    1.034219]    #2
[    1.049577] microcode: CPU2 microcode updated early to revision 0x710, date = 2013-06-17
[    1.070309]    #3
[    1.085865] microcode: CPU3 microcode updated early to revision 0x710, date = 2013-06-17
[    1.106618]    #4
[    1.121978] microcode: CPU4 microcode updated early to revision 0x710, date = 2013-06-17
[    1.142720]    #5
[    1.158079] microcode: CPU5 microcode updated early to revision 0x710, date = 2013-06-17
[    1.178833]    #6
[    1.194191] microcode: CPU6 microcode updated early to revision 0x710, date = 2013-06-17
[    1.214914]    #7
[    1.230471] microcode: CPU7 microcode updated early to revision 0x710, date = 2013-06-17
[    1.251309] 
[    1.254854] .... node  #1, CPUs:     #8
[    1.275173] microcode: CPU8 microcode updated early to revision 0x710, date = 2013-06-17
[    1.390509]    #9
[    1.406859] microcode: CPU9 microcode updated early to revision 0x710, date = 2013-06-17
[    1.427735]   #10
[    1.444303] microcode: CPU10 microcode updated early to revision 0x710, date = 2013-06-17
[    1.465343]   #11
[    1.481718] microcode: CPU11 microcode updated early to revision 0x710, date = 2013-06-17
[    1.502779]   #12
[    1.519156] microcode: CPU12 microcode updated early to revision 0x710, date = 2013-06-17
[    1.540171]   #13
[    1.556536] microcode: CPU13 microcode updated early to revision 0x710, date = 2013-06-17
[    1.577587]   #14
[    1.594127] microcode: CPU14 microcode updated early to revision 0x710, date = 2013-06-17
[    1.615131]   #15
[    1.631471] microcode: CPU15 microcode updated early to revision 0x710, date = 2013-06-17
[    1.652590] 
[    1.656132] .... node  #2, CPUs:    #16
[    1.676518] microcode: CPU16 microcode updated early to revision 0x710, date = 2013-06-17
[    1.791812]   #17
[    1.808189] microcode: CPU17 microcode updated early to revision 0x710, date = 2013-06-17
[    1.829292]   #18
[    1.845868] microcode: CPU18 microcode updated early to revision 0x710, date = 2013-06-17
[    1.866925]   #19
[    1.883311] microcode: CPU19 microcode updated early to revision 0x710, date = 2013-06-17
[    1.904386]   #20
[    1.920765] microcode: CPU20 microcode updated early to revision 0x710, date = 2013-06-17
[    1.941810]   #21
[    1.958169] microcode: CPU21 microcode updated early to revision 0x710, date = 2013-06-17
[    1.979242]   #22
[    1.995787] microcode: CPU22 microcode updated early to revision 0x710, date = 2013-06-17
[    2.016842]   #23
[    2.033182] microcode: CPU23 microcode updated early to revision 0x710, date = 2013-06-17
[    2.054314] 
[    2.057854] .... node  #3, CPUs:    #24
[    2.078330] microcode: CPU24 microcode updated early to revision 0x710, date = 2013-06-17
[    2.193513]   #25
[    2.209874] microcode: CPU25 microcode updated early to revision 0x710, date = 2013-06-17
[    2.230996]   #26
[    2.247563] microcode: CPU26 microcode updated early to revision 0x710, date = 2013-06-17
[    2.268627]   #27
[    2.284998] microcode: CPU27 microcode updated early to revision 0x710, date = 2013-06-17
[    2.306061]   #28
[    2.322437] microcode: CPU28 microcode updated early to revision 0x710, date = 2013-06-17
[    2.343433]   #29
[    2.359780] microcode: CPU29 microcode updated early to revision 0x710, date = 2013-06-17
[    2.380855]   #30
[    2.397397] microcode: CPU30 microcode updated early to revision 0x710, date = 2013-06-17
[    2.418432]   #31
[    2.434759] microcode: CPU31 microcode updated early to revision 0x710, date = 2013-06-17
[    2.455792] 
[    2.459336] .... node  #0, CPUs:    #32  #33  #34  #35  #36  #37  #38  #39
[    2.583817] .... node  #1, CPUs:    #40  #41  #42  #43  #44  #45  #46  #47
[    2.710873] .... node  #2, CPUs:    #48  #49  #50  #51  #52  #53  #54  #55
[    2.838069] .... node  #3, CPUs:    #56  #57  #58  #59  #60  #61  #62  #63
[    2.964288] x86: Booted up 4 nodes, 64 CPUs
[    2.974471] smpboot: Total of 64 processors activated (344907.86 BogoMIPS)


[ 5290.635126] Broke affinity for irq 82
[ 5290.643222] Broke affinity for irq 111
[ 5290.651507] Broke affinity for irq 125
[ 5290.664107] smpboot: CPU 5 is now offline
[ 5298.371336] Broke affinity for irq 31
[ 5298.379528] Broke affinity for irq 82
[ 5298.387627] Broke affinity for irq 103
[ 5298.395908] Broke affinity for irq 110
[ 5298.404187] Broke affinity for irq 111
[ 5298.412450] Broke affinity for irq 112
[ 5298.420733] Broke affinity for irq 118
[ 5298.429017] Broke affinity for irq 124
[ 5298.437295] Broke affinity for irq 125
[ 5298.445584] Broke affinity for irq 127
[ 5298.453880] Broke affinity for irq 137
[ 5298.466543] smpboot: CPU 34 is now offline
[ 5302.187338] EINJ: Error INJection is initialized.
[ 5318.897170] Disabling lock debugging due to kernel taint
[ 5318.910775] mce: [Hardware Error]: CPU 37: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5318.931171] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5318.951567] mce: [Hardware Error]: TSC bab9f2d8a4e00 ADDR bb68ec00 MISC 20403ebe86 
[ 5318.969835] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC b microcode 710
[ 5318.990959] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.003825] EDAC sbridge MC0: CPU 37: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.023215] EDAC sbridge MC0: TSC bab9f2d8a4e00 
[ 5319.033036] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5319.050338] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC b
[ 5319.069542] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset
:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.122943] mce: [Hardware Error]: CPU 3: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.143355] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5319.163846] mce: [Hardware Error]: TSC bab9f2d8a51c1 ADDR bb68ec00 MISC 20403ebe86 
[ 5319.182249] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 6 microcode 710
[ 5319.203539] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.216586] EDAC sbridge MC0: CPU 3: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.235994] EDAC sbridge MC0: TSC bab9f2d8a51c1 
[ 5319.245814] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5319.263348] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 6
[ 5319.283041] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset
:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.337311] mce: [Hardware Error]: CPU 2: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.357960] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8159a4d0> {mutex_lock+0x10/0x27}
[ 5319.378519] mce: [Hardware Error]: TSC bab9f2d8a3feb ADDR bb68ec00 MISC 20403ebe86 
[ 5319.397151] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 4 microcode 710
[ 5319.418650] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.431902] EDAC sbridge MC0: CPU 2: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.451491] EDAC sbridge MC0: TSC bab9f2d8a3feb 
[ 5319.461311] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5319.479022] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 4
[ 5319.499014] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset
:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.553209] mce: [Hardware Error]: CPU 6: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.574029] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5319.594953] mce: [Hardware Error]: TSC bab9f2d8a87ea ADDR bb68ec00 MISC 20403ebe86 
[ 5319.613756] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC c microcode 710
[ 5319.635431] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.648873] EDAC sbridge MC0: CPU 6: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.668661] EDAC sbridge MC0: TSC bab9f2d8a87ea 
[ 5319.678483] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5319.696422] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC c
[ 5319.716789] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset
:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.771531] mce: [Hardware Error]: CPU 38: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.792743] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5319.813836] mce: [Hardware Error]: TSC bab9f2d8a87ce ADDR bb68ec00 MISC 20403ebe86 
[ 5319.832819] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC d microcode 710
[ 5319.854654] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.868243] EDAC sbridge MC0: CPU 38: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.888366] EDAC sbridge MC0: TSC bab9f2d8a87ce 
[ 5319.898186] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5319.916192] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC d
[ 5319.936752] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.991752] mce: [Hardware Error]: CPU 35: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.013034] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.034166] mce: [Hardware Error]: TSC bab9f2d8a59dd ADDR bb68ec00 MISC 20403ebe86 
[ 5320.053149] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 7 microcode 710
[ 5320.074972] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.088567] EDAC sbridge MC0: CPU 35: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.108688] EDAC sbridge MC0: TSC bab9f2d8a59dd 
[ 5320.118511] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5320.136527] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 7
[ 5320.157079] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5320.212025] mce: [Hardware Error]: CPU 39: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.233316] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.254462] mce: [Hardware Error]: TSC bab9f2d8a4f5c ADDR bb68ec00 MISC 20403ebe86 
[ 5320.273455] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC f microcode 710
[ 5320.295303] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.308905] EDAC sbridge MC0: CPU 39: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.329026] EDAC sbridge MC0: TSC bab9f2d8a4f5c 
[ 5320.338847] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5320.356858] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC f
[ 5320.377433] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5320.432474] mce: [Hardware Error]: CPU 7: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.453569] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.474703] mce: [Hardware Error]: TSC bab9f2d8a4d60 ADDR bb68ec00 MISC 20403ebe86 
[ 5320.493689] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC e microcode 710
[ 5320.515532] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.529139] EDAC sbridge MC0: CPU 7: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.549050] EDAC sbridge MC0: TSC bab9f2d8a4d60 
[ 5320.558870] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5320.576890] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC e
[ 5320.597478] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5320.652525] mce: [Hardware Error]: CPU 36: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.673804] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.694918] mce: [Hardware Error]: TSC bab9f2d8a5823 ADDR bb68ec00 MISC 20403ebe86 
[ 5320.713916] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 9 microcode 710
[ 5320.735759] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.749347] EDAC sbridge MC0: CPU 36: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.769452] EDAC sbridge MC0: TSC bab9f2d8a5823 
[ 5320.779273] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5320.797296] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 9
[ 5320.817877] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5320.872972] mce: [Hardware Error]: CPU 33: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.894249] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.915390] mce: [Hardware Error]: TSC bab9f2d8a5326 ADDR bb68ec00 MISC 20403ebe86 
[ 5320.934374] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 3 microcode 710
[ 5320.956222] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.969807] EDAC sbridge MC0: CPU 33: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.989913] EDAC sbridge MC0: TSC bab9f2d8a5326 
[ 5320.999734] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5321.017750] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 3
[ 5321.038284] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.093686] mce: [Hardware Error]: CPU 1: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.114770] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5321.135925] mce: [Hardware Error]: TSC bab9f2d8a5562 ADDR bb68ec00 MISC 20403ebe86 
[ 5321.154918] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 2 microcode 710
[ 5321.176765] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5321.190369] EDAC sbridge MC0: CPU 1: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.210303] EDAC sbridge MC0: TSC bab9f2d8a5562 
[ 5321.220123] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5321.238146] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 2
[ 5321.258723] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.303358] mce: [Hardware Error]: CPU 4: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.324279] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5321.345397] mce: [Hardware Error]: TSC bab9f2d8a572f ADDR bb68ec00 MISC 20403ebe86 
[ 5321.364380] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 8 microcode 710
[ 5321.386184] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5321.399729] EDAC sbridge MC0: CPU 4: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.419624] EDAC sbridge MC0: TSC bab9f2d8a572f 
[ 5321.429445] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5321.447454] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 8
[ 5321.467989] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.511475] mce: [Hardware Error]: CPU 32: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.532587] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5321.553689] mce: [Hardware Error]: TSC bab9f2d8a50f4 ADDR bb68ec00 MISC 20403ebe86 
[ 5321.572681] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 1 microcode 710
[ 5321.594500] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5321.608057] EDAC sbridge MC0: CPU 32: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.628161] EDAC sbridge MC0: TSC bab9f2d8a50f4 
[ 5321.637982] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5321.655998] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 1
[ 5321.676524] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.720020] mce: [Hardware Error]: CPU 0: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.740939] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5321.762058] mce: [Hardware Error]: TSC bab9f2d8a5034 ADDR bb68ec00 MISC 20403ebe86 
[ 5321.781022] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 0 microcode 710
[ 5321.802837] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5321.816395] EDAC sbridge MC0: CPU 0: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.836300] EDAC sbridge MC0: TSC bab9f2d8a5034 
[ 5321.846121] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86 
[ 5321.864127] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 0
[ 5321.884647] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 -  area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.928136] mce: [Hardware Error]: Machine check: Processor context corrupt
[ 5321.945589] Kernel panic - not syncing: Fatal machine check
[ 5321.985122] Kernel Offset: disabled
[ 5322.008492] Rebooting in 100 seconds..
[ 5421.226077] ACPI MEMORY or I/O RESET_REG.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1286069 — Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

From"Raj, Ashok" <ashok.raj@intel.com>
Date2015-12-07 23:50 +0100
SubjectRe: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDcLx-2nc-31@gated-at.bofh.it>
In reply to#1286058
On Mon, Dec 07, 2015 at 11:34:27PM +0100, Borislav Petkov wrote:
> 
> Box logs below.

Do you have the dmidecode strings to find which platform this is?

> 
> BIOS is doing funny cores enumeration:
> 
> node #0, CPUs 0-7
> node #1, CPUs 8-15
> node #2, CPUs 16-23
> node #3, CPUs 24-31
> 
> and then starts from node 0 again:
> 
>  .... node  #0, CPUs:    #32  #33  #34  #35  #36  #37  #38  #39
>  .... node  #1, CPUs:    #40  #41  #42  #43  #44  #45  #46  #47
>  .... node  #2, CPUs:    #48  #49  #50  #51  #52  #53  #54  #55
>  .... node  #3, CPUs:    #56  #57  #58  #59  #60  #61  #62  #63
> 
> So I went and offlined cores 5 and 34 which are on node 0.
> 
> Why node 0? Well, when I inject error type 0x10 which is
> 
> 0x00000010      Memory Uncorrectable non-fatal
> 
> it generates an MCE only on the node 0 cores. For that log see the end
> of this mail. The gist of it is that the CPUs on which #MC gets raised
> are the cores on node 0, i.e., 0-7 and 32-39.
> 
> Cores 5 and 34 are gone, of course.
> 
> I mean, even if the #MC gets raised only on the node, the fix still
> works.


Not sure how the fix works.. since we excluded only the ones offline. So 
unless all online cpu's check in, the code should give you the old behavior.

What does cat /proc/interrupts | grep MCE 

In a system broadcasting, all cpu counts should be the same. Since we didn't 
increment the offline stats, if you were to bring the cpu up, it should be one
less than other cpus... 


Cheers,
Ashok

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1286102 — Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

FromBorislav Petkov <bp@alien8.de>
Date2015-12-08 00:30 +0100
SubjectRe: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDdoe-2Si-33@gated-at.bofh.it>
In reply to#1286069
On Mon, Dec 07, 2015 at 06:46:40PM -0500, Raj, Ashok wrote:
> On Mon, Dec 07, 2015 at 11:34:27PM +0100, Borislav Petkov wrote:
> > 
> > Box logs below.
> 
> Do you have the dmidecode strings to find which platform this is?

Is this enough or you want complete dmidecode dump?

DMI: Intel Corporation LH Pass/S4600LH...., BIOS SE5C600.86B.99.99.2050.043020121425 04/30/2012

> Not sure how the fix works.. since we excluded only the ones offline.
> So unless all online cpu's check in, the code should give you the old
> behavior.

Did you miss my statement in my previous mail where I said that the MCE
is being raised only on the cores of node 0?

> What does cat /proc/interrupts | grep MCE 

Can't. Shell on the box is dead after the injection.

> In a system broadcasting, all cpu counts should be the same. Since we didn't 
> increment the offline stats, if you were to bring the cpu up, it should be one
> less than other cpus...

See the logs at the end of my previous email. #MC gets raised - or at
least output from mce_panic() comes out only - on the cores of node 0.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1286130 — Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

From"Raj, Ashok" <ashok.raj@intel.com>
Date2015-12-08 01:50 +0100
SubjectRe: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDeDD-3zV-7@gated-at.bofh.it>
In reply to#1286102
On Tue, Dec 08, 2015 at 12:25:24AM +0100, Borislav Petkov wrote:
> 
> Did you miss my statement in my previous mail where I said that the MCE
> is being raised only on the cores of node 0?
> 

That's right.. but i think if MCE is only given to node0, then the system 
would panic eveytime with or without the patch. which is why i got confused.

I somehow misunderstood that with this patch the system didn't panic. 

Cheers,
Ashok

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1286341 — Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

FromBorislav Petkov <bp@alien8.de>
Date2015-12-08 10:20 +0100
SubjectRe: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDmBb-nT-5@gated-at.bofh.it>
In reply to#1286130
On Mon, Dec 07, 2015 at 08:41:43PM -0500, Raj, Ashok wrote:
> On Tue, Dec 08, 2015 at 12:25:24AM +0100, Borislav Petkov wrote:
> > 
> > Did you miss my statement in my previous mail where I said that the MCE
> > is being raised only on the cores of node 0?
> > 
> 
> That's right.. but i think if MCE is only given to node0, then the system 
> would panic eveytime with or without the patch. which is why i got confused.
> 
> I somehow misunderstood that with this patch the system didn't panic.

No, the system did panic in both times. The "strange" observation is
that the MCE gets reported only on the cores on node 0. Or at least only
the printks from mce_panic() on the cores on node0 reach the serial
console.

If we really broadcast only on node0, then that would be a problem if
the corrupted data leaves the node and manages to corrupt storage when
written out on some of the other nodes. I'm not sure if the kernel
panicking the whole system is on time and there's not a small window
between the detection and the panicking, in which the corruption might
happen.

If so, this'd defeat the purpose of MCE broadcasting but I'm just
hypothesizing here.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1286633 — RE: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

From"Luck, Tony" <tony.luck@intel.com>
Date2015-12-08 17:10 +0100
SubjectRE: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDsZY-4yl-25@gated-at.bofh.it>
In reply to#1286341
PiBObywgdGhlIHN5c3RlbSBkaWQgcGFuaWMgaW4gYm90aCB0aW1lcy4gVGhlICJzdHJhbmdlIiBv
YnNlcnZhdGlvbiBpcw0KPiB0aGF0IHRoZSBNQ0UgZ2V0cyByZXBvcnRlZCBvbmx5IG9uIHRoZSBj
b3JlcyBvbiBub2RlIDAuIE9yIGF0IGxlYXN0IG9ubHkNCj4gdGhlIHByaW50a3MgZnJvbSBtY2Vf
cGFuaWMoKSBvbiB0aGUgY29yZXMgb24gbm9kZTAgcmVhY2ggdGhlIHNlcmlhbA0KPiBjb25zb2xl
Lg0KDQpZb3Ugb25seSBzZWUgbWVzc2FnZXMgYW5kIGxvZ3MgZnJvbSBub2RlMCwgYmVjYXVzZSB0
aGUgY3B1cyB0aGVyZSBhcmUNCnRoZSBvbmx5IG9uZXMgdGhhdCBzZWUgYW55IGVycm9ycyBsb2dn
ZWQgaW4gdGhlaXIgYmFua3MuDQoNClRoZSBjcHVzIG9uIG5vZGUgMSwgMiwgMyBzY2FuIGFsbCBi
YW5rcyBhbmQgZmluZCBub3RoaW5nLCBzbyBzYXkgbm90aGluZy4NClRoZXJlIGFyZSBubyBzeXN0
ZW0td2lkZSBiYW5rcyAuLi4ganVzdCBjb3JlLXdpZGUgKGluIHJlY2VudCBnZW5lcmF0aW9ucw0K
YmFua3MgMC0zKSBhbmQgc29ja2V0LXdpZGUgKGJhbmtzID49NCkuICBCdXQgZG9uJ3QgY29kZSB0
aG9zZSBudW1iZXJzDQppbnRvIGFueSBnZW5lcmljIGNvZGUgLi4uIHdlIHdpbGwgY2hhbmdlIHRo
ZW0gc29vbmVyIG9yIGxhdGVyLg0KDQotVG9ueQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1286772 — Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

FromBorislav Petkov <bp@alien8.de>
Date2015-12-08 20:00 +0100
SubjectRe: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDvEt-60F-1@gated-at.bofh.it>
In reply to#1286633
On Tue, Dec 08, 2015 at 03:59:58PM +0000, Luck, Tony wrote:
> > No, the system did panic in both times. The "strange" observation is
> > that the MCE gets reported only on the cores on node 0. Or at least only
> > the printks from mce_panic() on the cores on node0 reach the serial
> > console.
> 
> You only see messages and logs from node0, because the cpus there are
> the only ones that see any errors logged in their banks.
> 
> The cpus on node 1, 2, 3 scan all banks and find nothing, so say nothing.

Right, sure, of course. Doh!

Confirmation:

[  183.840517] mce: do_machine_check: CPU: 30
[  183.840531] mce: do_machine_check: CPU: 27
[  183.840536] mce: do_machine_check: CPU: 29
[  183.840541] mce: do_machine_check: CPU: 56
[  183.840546] mce: do_machine_check: CPU: 28
[  183.840548] mce: do_machine_check: CPU: 60
[  183.840550] mce: do_machine_check: CPU: 24
[  183.840557] mce: do_machine_check: CPU: 12
[  183.840561] mce: do_machine_check: CPU: 45
[  183.840565] mce: do_machine_check: CPU: 59
[  183.840569] mce: do_machine_check: CPU: 57
[  183.840572] mce: do_machine_check: CPU: 61
[  183.840584] mce: do_machine_check: CPU: 0
[  183.840587] mce: do_machine_check: CPU: 32
[  183.840593] mce: do_machine_check: CPU: 63
[  183.840596] mce: do_machine_check: CPU: 31
[  183.840602] mce: do_machine_check: CPU: 42
[  183.840606] mce: do_machine_check: CPU: 11
[  183.840611] mce: do_machine_check: CPU: 41
[  183.840613] mce: do_machine_check: CPU: 9
[  183.840617] mce: do_machine_check: CPU: 62
[  183.840619] mce: do_machine_check: CPU: 25
[  183.840624] mce: do_machine_check: CPU: 58
[  183.840627] mce: do_machine_check: CPU: 26
[  183.840633] mce: do_machine_check: CPU: 5
[  183.840638] mce: do_machine_check: CPU: 1
[  183.840642] mce: do_machine_check: CPU: 37
[  183.840648] mce: do_machine_check: CPU: 15
[  183.840650] mce: do_machine_check: CPU: 47
[  183.840653] mce: do_machine_check: CPU: 44
[  183.840657] mce: do_machine_check: CPU: 14
[  183.840659] mce: do_machine_check: CPU: 46
[  183.840666] mce: do_machine_check: CPU: 52
[  183.840670] mce: do_machine_check: CPU: 50
[  183.840675] mce: do_machine_check: CPU: 48
[  183.840677] mce: do_machine_check: CPU: 16
[  183.840682] mce: do_machine_check: CPU: 54
[  183.840686] mce: do_machine_check: CPU: 18
[  183.840692] mce: do_machine_check: CPU: 40
[  183.840695] mce: do_machine_check: CPU: 8
[  183.840701] mce: do_machine_check: CPU: 2
[  183.840705] mce: do_machine_check: CPU: 20
[  183.840710] mce: do_machine_check: CPU: 13
[  183.840712] mce: do_machine_check: CPU: 43
[  183.840716] mce: do_machine_check: CPU: 10
[  183.840722] mce: do_machine_check: CPU: 3
[  183.840724] mce: do_machine_check: CPU: 35
[  183.840727] mce: do_machine_check: CPU: 33
[  183.840730] mce: do_machine_check: CPU: 34
[  183.840734] mce: do_machine_check: CPU: 6
[  183.840738] mce: do_machine_check: CPU: 38
[  183.840743] mce: do_machine_check: CPU: 53
[  183.840745] mce: do_machine_check: CPU: 21
[  183.840750] mce: do_machine_check: CPU: 23
[  183.840752] mce: do_machine_check: CPU: 55
[  183.840755] mce: do_machine_check: CPU: 22
[  183.840759] mce: do_machine_check: CPU: 49
[  183.840761] mce: do_machine_check: CPU: 17
[  183.840767] mce: do_machine_check: CPU: 19
[  183.840770] mce: do_machine_check: CPU: 51
[  183.840776] mce: do_machine_check: CPU: 39
[  183.840778] mce: do_machine_check: CPU: 7
[  183.840784] mce: do_machine_check: CPU: 36
[  183.840786] mce: do_machine_check: CPU: 4
[  184.485104] Disabling lock debugging due to kernel taint
[  184.498006] mce: [Hardware Error]: CPU 32: Machine Check Exception: 5 Bank 5: be00000000010090
[  184.498023] mce: [Hardware Error]: Machine check events logged
[  184.531428] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[  184.551126] mce: [Hardware Error]: TSC c760ad064ccce ADDR bb68ec00 MISC 421c8c86 
[  184.568358] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449600598 SOCKET 0 APIC 1 microcode 710
[  184.588862] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
...

mce: [Hardware Error]: CPU 0: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 1: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 2: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 32: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 33: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 34: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 35: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 36: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 37: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 38: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 39: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 3: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 4: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 5: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 6: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 7: Machine Check Exception: 5 Bank 5: be00000000010090

CPUs:

[    1.103200] x86: Booting SMP configuration:
[    1.112441] .... node  #0, CPUs:          #1   #2   #3   #4   #5   #6   #7
[    1.227835] .... node  #1, CPUs:     #8   #9  #10  #11  #12  #13  #14  #15
[    1.451861] .... node  #2, CPUs:    #16  #17  #18  #19  #20  #21  #22  #23
[    1.674819] .... node  #3, CPUs:    #24  #25  #26  #27  #28  #29  #30  #31
[    1.899011] .... node  #0, CPUs:    #32  #33  #34  #35  #36  #37  #38  #39
[    2.026616] .... node  #1, CPUs:    #40  #41  #42  #43  #44  #45  #46  #47
[    2.152645] .... node  #2, CPUs:    #48  #49  #50  #51  #52  #53  #54  #55
[    2.276782] .... node  #3, CPUs:    #56  #57  #58  #59  #60  #61  #62  #63
[    2.402263] x86: Booted up 4 nodes, 64 CPUs

Ok, all clear.

Thanks!

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1286097 — Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

From"Luck, Tony" <tony.luck@intel.com>
Date2015-12-08 00:30 +0100
SubjectRe: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDdoe-2Si-19@gated-at.bofh.it>
In reply to#1286058
On Mon, Dec 07, 2015 at 11:34:27PM +0100, Borislav Petkov wrote:
> BIOS is doing funny cores enumeration:
> 
> node #0, CPUs 0-7
> node #1, CPUs 8-15
> node #2, CPUs 16-23
> node #3, CPUs 24-31
> 
> and then starts from node 0 again:
> 
>  .... node  #0, CPUs:    #32  #33  #34  #35  #36  #37  #38  #39
>  .... node  #1, CPUs:    #40  #41  #42  #43  #44  #45  #46  #47
>  .... node  #2, CPUs:    #48  #49  #50  #51  #52  #53  #54  #55
>  .... node  #3, CPUs:    #56  #57  #58  #59  #60  #61  #62  #63

That's normal. BIOS writers are encouraged to list all the
hyperthread 0 cpus from each core, and then add the hyperthread
1 cpus later in the table. That way an OS that boots less than
all the cpus will get the maximum number of real cores into
play.

> 0x00000010      Memory Uncorrectable non-fatal
> 
> it generates an MCE only on the node 0 cores. For that log see the end
> of this mail. The gist of it is that the CPUs on which #MC gets raised
> are the cores on node 0, i.e., 0-7 and 32-39.

I think all the threads on all the sockets must have shown up
in the machine check handler ... but only the ones on socket0
printed anything (they can all see the error in bank5 which is
shared across the socket ... but cpus 8-15 etc. will see no errors
in any banks ... so will be silent.)

-Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1285886 — Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

FromBorislav Petkov <bp@alien8.de>
Date2015-12-07 21:10 +0100
SubjectRe: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Message-ID<qDagG-Lz-7@gated-at.bofh.it>
In reply to#1284333
On Fri, Dec 04, 2015 at 07:29:36PM -0500, Ashok Raj wrote:
> Linux has logical cpu offline capability. That can be triggered by:
> 
> # echo 0 > /sys/devices/system/cpu/cpuX/online
> 
> In Intel Architecture, MCE's are broadcasted to all CPUs in the system.
> 
> This includes the CPUs marked offline by Linux. Unless the CPU's were removed
> via an ACPI notification, in which case the cpu's are removed from the
> cpu_present_map.
> 
> This patch ensures offline CPU's don't participate in MCE rendezvous, but
> simply perform clearing some status bits to ensure a second MCE wont cause
> automatic shutdown.
> 
> Without the patch, mce_start will increment mce_callin, but mce_start would
> only wait for all online_cpus. So offline cpu's should avoid participating
> in the rendezvous process.
> 
> Reviewed-by: Tony Luck <tony.luck@intel.com>
> Cc: stable@vger.kernel.org
> Signed-off-by: Ashok Raj <ashok.raj@intel.com>
> ---
>  arch/x86/kernel/cpu/mcheck/mce.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)

Tested on a box here, massaged commit message and queued for 4.4,
thanks.

---
From: Ashok Raj <ashok.raj@intel.com>
Date: Thu, 3 Dec 2015 19:16:10 -0500
Subject: [PATCH] x86/mce: Ensure offline CPUs don't participate in rendezvous
 process

Intel's MCA implementation broadcasts MCEs to all CPUs on the node.
This poses a problem for offlined CPUs which cannot participate in the
rendezvous process:

  Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
  Kernel Offset: disabled
  Rebooting in 100 seconds..

More specifically, Linux does a soft offline of a CPU when writing a 0
to /sys/devices/system/cpu/cpuX/online, which doesn't prevent the #MC
exception from being broadcasted to that CPU.

Ensure that offline CPUs don't participate in the MCE rendezvous and
clear the RIP valid status bit so that a second MCE won't cause a
shutdown.

Without the patch, mce_start() will increment mce_callin and wait for
all CPUs. Offlined CPUs should avoid participating in the rendezvous
process altogether.

Reviewed-by: Tony Luck <tony.luck@intel.com>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: linux-edac <linux-edac@vger.kernel.org>
Cc: stable@vger.kernel.org
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: x86-ml <x86@kernel.org>
Link: http://lkml.kernel.org/r/1449188170-3909-1-git-send-email-ashok.raj@intel.com
[ Massage commit message. ]
Signed-off-by: Borislav Petkov <bp@suse.de>
---
 arch/x86/kernel/cpu/mcheck/mce.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 3865e95cc5ec..a006f4cd792b 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -1002,6 +1002,17 @@ void do_machine_check(struct pt_regs *regs, long error_code)
 	int flags = MF_ACTION_REQUIRED;
 	int lmce = 0;
 
+	/* If this CPU is offline, just bail out. */
+	if (cpu_is_offline(smp_processor_id())) {
+		u64 mcgstatus;
+
+		mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
+		if (mcgstatus & MCG_STATUS_RIPV) {
+			mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
+			return;
+		}
+	}
+
 	ist_enter(regs);
 
 	this_cpu_inc(mce_exception_count);
-- 
2.3.5

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web