Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1273264 > unrolled thread
| Started by | Borislav Petkov <bp@alien8.de> |
|---|---|
| First post | 2015-11-19 17:20 +0100 |
| Last post | 2015-11-23 19:00 +0100 |
| Articles | 4 — 2 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: [UNTESTED PATCH] x86, mce: Avoid double entry of deferred errors into the genpool. Borislav Petkov <bp@alien8.de> - 2015-11-19 17:20 +0100
RE: [UNTESTED PATCH] x86, mce: Avoid double entry of deferred errors into the genpool. "Luck, Tony" <tony.luck@intel.com> - 2015-11-19 20:40 +0100
Re: [UNTESTED PATCH] x86, mce: Avoid double entry of deferred errors into the genpool. Borislav Petkov <bp@alien8.de> - 2015-11-19 21:40 +0100
Re: [UNTESTED PATCH] x86, mce: Avoid double entry of deferred errors into the genpool. Borislav Petkov <bp@alien8.de> - 2015-11-23 19:00 +0100
| From | Borislav Petkov <bp@alien8.de> |
|---|---|
| Date | 2015-11-19 17:20 +0100 |
| Subject | Re: [UNTESTED PATCH] x86, mce: Avoid double entry of deferred errors into the genpool. |
| Message-ID | <qwA6e-3ds-17@gated-at.bofh.it> |
On Wed, Nov 11, 2015 at 02:01:51PM -0800, Tony Luck wrote:
> We used to have a special ring buffer for deferred errors that
> was used to mark problem pages. We replaced that with a genpool.
> Then later converted mce_log() to also use the same genpool. As
> a result we end up adding all deferred errors to the genpool twice.
>
> Rearrange this code. Make sure to set the m.severity and m.usable_addr
> fields for deferred errors. Then if flags and mca_cfg.dont_log_ce mean
> we call mce_log() we are done, because that will add this entry to the
> genpool.
>
> If we skipped mce_log(), then we still want to take action for the
> deferred error, so add to the genpool.
>
> Changed the name of the boolean "error_logged" to "error_seen", we
> should set it whether of not we logged an error because the return
> value from machine_check_poll() is used to decide whether storms
> have subsided or not.
>
> Reported-by: Chen, Gong <gong.chen.linux.intel.com>
> Signed-off-by: Tony Luck <tony.luck@intel.com>
> ---
> arch/x86/kernel/cpu/mcheck/mce.c | 24 +++++++++++++-----------
> 1 file changed, 13 insertions(+), 11 deletions(-)
Applied, thanks.
Btw, looking at that mce.usable_addr, it doesn't make a whole lotta
sense to me and we can use mce_usable_address() directly instead and use
the byte in struct mce for something more important. So how about I kill
it (diff ontop of yours):
---
diff --git a/arch/x86/include/uapi/asm/mce.h b/arch/x86/include/uapi/asm/mce.h
index 03429da2fa80..2184943341bf 100644
--- a/arch/x86/include/uapi/asm/mce.h
+++ b/arch/x86/include/uapi/asm/mce.h
@@ -16,7 +16,7 @@ struct mce {
__u8 cpuvendor; /* cpu vendor as encoded in system.h */
__u8 inject_flags; /* software inject flags */
__u8 severity;
- __u8 usable_addr;
+ __u8 pad;
__u32 cpuid; /* CPUID 1 EAX */
__u8 cs; /* code segment */
__u8 bank; /* machine check bank */
diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 6531cb46803c..fb8b1db7b150 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -484,7 +484,7 @@ static int srao_decode_notifier(struct notifier_block *nb, unsigned long val,
if (!mce)
return NOTIFY_DONE;
- if (mce->usable_addr && (mce->severity == MCE_AO_SEVERITY)) {
+ if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) {
pfn = mce->addr >> PAGE_SHIFT;
memory_failure(pfn, MCE_VECTOR, 0);
}
@@ -610,12 +610,9 @@ bool machine_check_poll(enum mcp_flags flags, mce_banks_t *b)
severity = mce_severity(&m, mca_cfg.tolerant, NULL, false);
- if (severity == MCE_DEFERRED_SEVERITY && memory_error(&m)) {
- if (m.status & MCI_STATUS_ADDRV) {
+ if (severity == MCE_DEFERRED_SEVERITY && memory_error(&m))
+ if (m.status & MCI_STATUS_ADDRV)
m.severity = severity;
- m.usable_addr = mce_usable_address(&m);
- }
- }
/*
* Don't get the IP here because it's unlikely to
@@ -623,7 +620,7 @@ bool machine_check_poll(enum mcp_flags flags, mce_banks_t *b)
*/
if (!(flags & MCP_DONTLOG) && !mca_cfg.dont_log_ce)
mce_log(&m);
- else if (m.usable_addr) {
+ else if (mce_usable_address(&m)) {
/*
* Although we skipped logging this, we still want
* to take action. Add to the pool so the registered
@@ -1091,7 +1088,6 @@ void do_machine_check(struct pt_regs *regs, long error_code)
/* assuming valid severity level != 0 */
m.severity = severity;
- m.usable_addr = mce_usable_address(&m);
mce_log(&m);
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | "Luck, Tony" <tony.luck@intel.com> |
|---|---|
| Date | 2015-11-19 20:40 +0100 |
| Subject | RE: [UNTESTED PATCH] x86, mce: Avoid double entry of deferred errors into the genpool. |
| Message-ID | <qwDdL-5dZ-9@gated-at.bofh.it> |
| In reply to | #1273264 |
PiBBcHBsaWVkLCB0aGFua3MuDQoNCkRpZCB5b3UgdGVzdCBpdCAobm90ZSB0aGUgIlVOVEVTVEVE IiBpbiB0aGUgc3ViamVjdCEpLiAgTXkgdXN1YWwgc3lzdGVtIGZvciB0aGlzIGlzIGdldHRpbmcg dXBncmFkZXMgYW5kIGJlaW5nDQpmbGFreSBhdCB0aGUgbW9tZW50Lg0KDQo+IEJ0dywgbG9va2lu ZyBhdCB0aGF0IG1jZS51c2FibGVfYWRkciwgaXQgZG9lc24ndCBtYWtlIGEgd2hvbGUgbG90dGEN Cj4gc2Vuc2UgdG8gbWUgYW5kIHdlIGNhbiB1c2UgbWNlX3VzYWJsZV9hZGRyZXNzKCkgZGlyZWN0 bHkgaW5zdGVhZCBhbmQgdXNlDQo+IHRoZSBieXRlIGluIHN0cnVjdCBtY2UgZm9yIHNvbWV0aGlu ZyBtb3JlIGltcG9ydGFudC4gU28gaG93IGFib3V0IEkga2lsbA0KPiBpdCAoZGlmZiBvbnRvcCBv ZiB5b3Vycyk6DQoNClN1cmUuICAic3RydWN0IG1jZSIgaXMgdmlzaWJsZSB0byB1c2VyIHNwYWNl IHZpYSAvZGV2L21jZWxvZy4gQnV0IHRoZSBvbmx5IHVzZXIgaXMgdGhlDQptY2Vsb2coOCkgZGFl bW9uIC4uLiBhbmQgaXQgd2FzIG5ldmVyIHVwZGF0ZWQgdG8gbG9vayBhdCB0aGUgdXNhYmxlX2Fk ZHIgZmllbGQuIFNvDQpyZXR1cm5pbmcgaXQgdG8gInBhZCIgc3RhdHVzIGlzIGZpbmUuDQoNCi1U b255DQo= -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Borislav Petkov <bp@alien8.de> |
|---|---|
| Date | 2015-11-19 21:40 +0100 |
| Message-ID | <qwE9Q-5PC-29@gated-at.bofh.it> |
| In reply to | #1273437 |
On Thu, Nov 19, 2015 at 07:33:58PM +0000, Luck, Tony wrote:
> > Applied, thanks.
>
> Did you test it (note the "UNTESTED" in the subject!). My usual system for this is getting upgrades and being
> flaky at the moment.
Bah, it builds, should be enough. Ship it. :-)
Lemme get a box...
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Borislav Petkov <bp@alien8.de> |
|---|---|
| Date | 2015-11-23 19:00 +0100 |
| Message-ID | <qy3zd-55c-21@gated-at.bofh.it> |
| In reply to | #1273481 |
On Thu, Nov 19, 2015 at 09:39:20PM +0100, Borislav Petkov wrote:
> On Thu, Nov 19, 2015 at 07:33:58PM +0000, Luck, Tony wrote:
> > > Applied, thanks.
> >
> > Did you test it (note the "UNTESTED" in the subject!). My usual system for this is getting upgrades and being
> > flaky at the moment.
>
> Bah, it builds, should be enough. Ship it. :-)
>
> Lemme get a box...
Here some results:
# grep . /sys/kernel/debug/apei/einj/*
/sys/kernel/debug/apei/einj/available_error_type:0x00000002 Processor Uncorrectable non-fatal
/sys/kernel/debug/apei/einj/available_error_type:0x00000008 Memory Correctable
/sys/kernel/debug/apei/einj/available_error_type:0x00000010 Memory Uncorrectable non-fatal
grep: /sys/kernel/debug/apei/einj/error_inject: Permission denied
/sys/kernel/debug/apei/einj/error_type:0x0
Looks like some old EINJ without all the features. Oh well, let's see
what'll happen anyway:
# echo 0x8 > error_type
# echo 1 > error_inject
[ 840.461666] mce: [Hardware Error]: Machine check events logged
[ 840.476221] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 840.489214] EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 5: 8c00004000010090
[ 840.507685] EDAC sbridge MC0: TSC 0
[ 840.515223] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 840.532477] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1448299322 SOCKET 0 APIC 0
[ 840.551279] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 840.563872] EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 8: 8800004100800090
[ 840.581970] EDAC sbridge MC0: TSC 0
[ 840.589513] EDAC sbridge MC0: ADDR 0 EDAC sbridge MC0: MISC 4908400040004200
[ 840.606267] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1448299322 SOCKET 0 APIC 0
[ 841.499090] EDAC MC0: 1 CE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
So yeah, mce_notify_irq() is visible there, i.e. we did mce_log() here
which sets mce_need_notify.
# echo 0x2 > error_type
# echo 1 > error_inject
bash: echo: write error: Invalid argument
[ 885.272000] [Firmware Warn]: APEI: Invalid action table, unknown instruction type: 5
ACPI_EINJ_FLUSH_CACHELINE??
Yeah, we're missing some functionality.
# echo 0x10 > error_type
# echo 1 > error_inject
That went BOOM:
[ 1296.233435] Disabling lock debugging due to kernel taint
[ 1296.248010] mce: [Hardware Error]: CPU 6: Machine Check Exception: 5 Bank 5: be00000000010090
[ 1296.269245] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8136260f> {intel_idle+0xbf/0x130}
[ 1296.290735] mce: [Hardware Error]: TSC 37c1fb53beb ADDR bb68f400 MISC 20401a9a86
[ 1296.309772] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1448299778 SOCKET 0 APIC c microcode 710
[ 1296.332058] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 1296.346094] EDAC sbridge MC0: CPU 6: Machine Check Exception: 5 Bank 5: be00000000010090
[ 1296.366517] EDAC sbridge MC0: TSC 37c1fb53beb
[ 1296.375974] EDAC sbridge MC0: ADDR bb68f400 EDAC sbridge MC0: MISC 20401a9a86
[ 1296.394493] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1448299778 SOCKET 0 APIC c
[ 1296.416153] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68f offset:
0x400 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
...
judging by the CPU numbers, looks like node 0 got that error in the shared bank:
.... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7
.... node #0, CPUs: #32 #33 #34 #35 #36 #37 #38 #39
finishing with
[ 1299.907994] mce: [Hardware Error]: Machine check: Processor context corrupt
[ 1299.926783] Kernel panic - not syncing: Fatal machine check
[ 1299.959632] Kernel Offset: disabled
[ 1299.984254] Rebooting in 100 seconds..
dont_log_ce:
$ for i in $(seq 0 63); do echo 1 > /sys/devices/system/machinecheck/machinecheck$i/dont_log_ce; cat /sys/devices/system/machinecheck/machinecheck$i/dont_log_ce; done | uniq
1
# echo 0x8 > error_type
# echo 1 > error_inject
[ 318.263797] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 318.277029] EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 5: 8c00004000010090
[ 318.295631] EDAC sbridge MC0: TSC 0
[ 318.303143] EDAC sbridge MC0: ADDR bb68f000 EDAC sbridge MC0: MISC 2040262686
[ 318.320473] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1448300397 SOCKET 0 APIC 0
[ 318.809112] EDAC MC0: 1 CE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68f offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
This looks ok, we're missing the mce_notify_irq() line "mce: [Hardware
Error]: Machine check events logged" which is as expected but the EDAC
lines are there because we sent the error on the notify chain.
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web