Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1167487 > unrolled thread

RE: MCE Bug?

Started by"Luck, Tony" <tony.luck@intel.com>
First post2015-06-18 02:00 +0200
Last post2015-06-19 14:30 +0200
Articles 3 — 2 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  RE: MCE Bug? "Luck, Tony" <tony.luck@intel.com> - 2015-06-18 02:00 +0200
    [PATCH] x86/mce: Initialize workqueues only once Borislav Petkov <bp@suse.de> - 2015-06-19 11:30 +0200
      Re: [PATCH] x86/mce: Initialize workqueues only once Borislav Petkov <bp@suse.de> - 2015-06-19 14:30 +0200

#1167487 — RE: MCE Bug?

From"Luck, Tony" <tony.luck@intel.com>
Date2015-06-18 02:00 +0200
SubjectRE: MCE Bug?
Message-ID<pCvpn-8ib-5@gated-at.bofh.it>
PiBpZiB5b3Ugd2FudCB0byBnaXZlIHRob3NlIGNoYW5nZXMgYSBydW4sIEkndmUgdXBsb2FkZWQg
dGhlbSBoZXJlOg0KPg0KPiBnaXQ6Ly9naXQua2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5l
bC9naXQvcmFzL3Jhcy5naXQjdGlwLXJhcw0KDQpMYXRlc3QgZXhwZXJpbWVudHMgc2hvdyB0aGF0
IHNvbWV0aW1lcyBjaGVja2luZyBrdmVudGRfdXAoKSBiZWZvcmUgY2FsbGluZyBzY2hlZHVsZV93
b3JrKCkNCmhlbHBzIC4uLiBidXQgbW9zdGx5IG9ubHkgd2hlbiBJIGZha2Ugc29tZSBlYXJseSBs
b2dzIGZyb20gbG93IG51bWJlcmVkICBjcHVzLiAgSSBhZGRlZCBzb21lDQp0cmFjZXMgdG8gdGhl
IHJlYWwgY2FzZSBvZiBhIGxlZnQtb3ZlciBmYXRhbCBlcnJvciBhbmQgZ290IHRoaXMgc3BsYXQ6
DQoNClsgICAgMC4zMzE1NTFdIHNtcGJvb3Q6IENQVTA6IEludGVsKFIpIFhlb24oUikgQ1BVIEU3
LTg4OTAgdjMgQCAyLjUwR0h6IChmYW06IDA2LCBtb2RlbDogM2YsIHN0ZXBwaW5nOiAwNCkNClsg
ICAgMC4zNDIxMTddIFBlcmZvcm1hbmNlIEV2ZW50czogUEVCUyBmbXQyKywgMTYtZGVlcCBMQlIs
IEhhc3dlbGwgZXZlbnRzLCBmdWxsLXdpZHRoIGNvdW50ZXJzLCBJbnRlbCBQTVUgZHJpdmVyLg0K
WyAgICAwLjM1MzQ3MV0gLi4uIHZlcnNpb246ICAgICAgICAgICAgICAgIDMNClsgICAgMC4zNTc5
NDhdIC4uLiBiaXQgd2lkdGg6ICAgICAgICAgICAgICA0OA0KWyAgICAwLjM2MjUyM10gLi4uIGdl
bmVyaWMgcmVnaXN0ZXJzOiAgICAgIDQNClsgICAgMC4zNjcwMDBdIC4uLiB2YWx1ZSBtYXNrOiAg
ICAgICAgICAgICAwMDAwZmZmZmZmZmZmZmZmDQpbICAgIDAuMzcyOTM1XSAuLi4gbWF4IHBlcmlv
ZDogICAgICAgICAgICAgMDAwMGZmZmZmZmZmZmZmZg0KWyAgICAwLjM3ODg3MF0gLi4uIGZpeGVk
LXB1cnBvc2UgZXZlbnRzOiAgIDMNClsgICAgMC4zODMzNDddIC4uLiBldmVudCBtYXNrOiAgICAg
ICAgICAgICAwMDAwMDAwNzAwMDAwMDBmDQpbICAgIDAuMzkyMzU3XSB4ODY6IEJvb3RpbmcgU01Q
IGNvbmZpZ3VyYXRpb246DQpbICAgIDAuMzk3MDMxXSAuLi4uIG5vZGUgICMwLCBDUFVzOiAgICAg
ICAgICAjMQ0KWyAgICAwLjQyMzM3M10gTk1JIHdhdGNoZG9nOiBlbmFibGVkIG9uIGFsbCBDUFVz
LCBwZXJtYW5lbnRseSBjb25zdW1lcyBvbmUgaHctUE1VIGNvdW50ZXIuDQpbICAgIDAuNDMyNzA1
XSAgICAjMiAgICMzICAgIzQgICAjNSAgICM2ICAgIzcgICAjOCAgICM5ICAjMTAgICMxMSAgIzEy
ICAjMTMgICMxNCAgIzE1ICAjMTYgICMxNw0KWyAgICAwLjcwNjg3OF0gLi4uLiBub2RlICAjMSwg
Q1BVczogICAgIzE4ICAjMTkgICMyMCAgIzIxICAjMjIgICMyMyAgIzI0ICAjMjUgICMyNiAgIzI3
ICAjMjggICMyOSAgIzMwICAjMzEgICMzMiAgIzMzICAjMzQgICMzNQ0KWyAgICAxLjA5NDYyNV0g
Li4uLiBub2RlICAjMiwgQ1BVczogICAgIzM2DQpbICAgIDEuMTEyOTU4XSBtY2Vsb2c6IGNwdSAz
NiBiYW5rIDggc3RhdHVzIGJlMDAwMDAwMDAwMTAwOTANClsgICAgMS4xMTkyMDFdIG1jZWxvZygp
IHN0YXNoZWQgYXQgZW50cnk9MA0KWyAgICAxLjIwMzYwMl0gbWNlOiBbSGFyZHdhcmUgRXJyb3Jd
OiBNYWNoaW5lIGNoZWNrIGV2ZW50cyBsb2dnZWQNClsgICAgMS4yMjAzMTNdICAgIzM3DQpbICAg
IDEuMjIwNDEyXSBCVUc6IHVuYWJsZSB0byBoYW5kbGUga2VybmVsIA0KWyAgICAxLjIyNjk1NF0g
ICAjMzgNClsgICAgMS4yMjkxMDddIE5VTEwgcG9pbnRlciBkZXJlZmVyZW5jZSBhdCAwMDAwMDAw
MDAwMDAwMDA4DQpbICAgIDEuMjM1MDUyXSBJUDogWzxmZmZmZmZmZjgxMDk4MGExPl0gcHJvY2Vz
c19vbmVfd29yaysweDMxLzB4NDIwDQpbICAgIDEuMjM2ODI5XSAgICMzOVBHRCAwIA0KWyAgICAx
LjI0NDU1OF0gT29wczogMDAwMCBbIzFdIFNNUCANClsgICAgMS4yNDgxODldIE1vZHVsZXMgbGlu
a2VkIGluOg0KWyAgICAxLjI1MTYxN10gQ1BVOiAzNiBQSUQ6IDI2MyBDb21tOiBrd29ya2VyLzM2
OjAgTm90IHRhaW50ZWQgNC4xLjAtcmM4ICM5DQpbICAgIDEuMjU5MTAwXSAgICM0MA0KWyAgICAx
LjI1OTEwMF0gSGFyZHdhcmUgbmFtZTogSW50ZWwgQ29ycG9yYXRpb24gQlJJQ0tMQU5EL0JSSUNL
TEFORCwgQklPUyBCUkhTWFNEMS44NkIuMDA2NS5SMDEuMTUwNTAxMTY0MCAwNS8wMS8yMDE1DQpb
ICAgIDEuMjcyODMyXSAgICM0MQ0KWyAgICAxLjI3MjgzM10gdGFzazogZmZmZjg4MTgxYzFmNDQ3
MCB0aTogZmZmZjg4MTgxYmQyNDAwMCB0YXNrLnRpOiBmZmZmODgxODFiZDI0MDAwDQpbICAgIDEu
MjgzMzUwXSBSSVA6IDAwMTA6WzxmZmZmZmZmZjgxMDk4MGExPl0gWyAgICAxLjI4NjQzM10gICAj
NDINCiBbPGZmZmZmZmZmODEwOTgwYTE+XSBwcm9jZXNzX29uZV93b3JrKzB4MzEvMHg0MjANClsg
ICAgMS4yOTQ5NzZdIFJTUDogMDAwMDpmZmZmODgxODFiZDI3ZTA4ICBFRkxBR1M6IDAwMDEwMDQ2
DQoNCkkuZS4gd2UgZGllIG9uIHRoZSBmaXJzdCBhdHRlbXB0IHRvIGxvZyAuLi4gYnV0IHRoYXQg
YXR0ZW1wdCBpcyBhIGxvbmcgd2F5IGludG8gYnJpbmdpbmcgdXAgYWxsIHRoZSBjcHVzLiANCkNQ
VSMzNiBpcyB0aGUgZmlyc3Qgb25lIGZyb20gc29ja2V0MiAoY291bnRpbmcgMCwgMSwgMiwgMyku
DQoNCi1Ub255DQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1168654 — [PATCH] x86/mce: Initialize workqueues only once

FromBorislav Petkov <bp@suse.de>
Date2015-06-19 11:30 +0200
Subject[PATCH] x86/mce: Initialize workqueues only once
Message-ID<pD0My-3fQ-17@gated-at.bofh.it>
In reply to#1167487
On Wed, Jun 17, 2015 at 11:53:53PM +0000, Luck, Tony wrote:
> > if you want to give those changes a run, I've uploaded them here:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras.git#tip-ras
> 
> Latest experiments show that sometimes checking kventd_up() before calling schedule_work()
> helps ... but mostly only when I fake some early logs from low numbered  cpus.  I added some
> traces to the real case of a left-over fatal error and got this splat:

Here's the fix:

--
From: Borislav Petkov <bp@suse.de>
Subject: [PATCH] x86/mce: Initialize workqueues only once

96d98bfd0366 ("x86/mce: Don't use percpu workqueues") dropped the
per-CPU workqueues in the MCE code but left the initialization per-CPU.
This lead to early boot time splats (below) in the workqueues code
because we were overwriting the workqueue during INIT_WORK() on each new
CPU which would appear.

And since mcheck_cpu_init() happens very early, using an initcall to do
this one-time initialization doesn't fly. Using work->func as a check
whether the workqueue has been initialized already might break in the
future if someone changes ->func, so doesn't work either.

So let's have a simple static boolean flag to do that one-time work.

  mce: [Hardware Error]: Machine check events logged
  BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
  IP: [<ffffffff810980a1>] process_one_work+0x31/0x420
   PGD 0
  Oops: 0000 [#1] SMP
  Modules linked in:
  CPU: 36 PID: 263 Comm: kworker/36:0 Not tainted 4.1.0-rc8 #1
  Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0065.R01.1505011640 05/01/2015
  task: ffff88181c284470 ti: ffff88181bd94000 task.ti: ffff88181bd94000
  RIP: 0010:[<ffffffff810980a1>] process_one_work+0x31/0x420
  RSP: 0000:ffff88181bd97e08  EFLAGS: 00010046
  RAX: 0000000fffffffe0 RBX: ffffffff81d0fa20 RCX: 0000000000000000
  RDX: 0000000fffffff00 RSI: ffffffff81d0fa20 RDI: ffff88181c2660c0
  RBP: ffff88181bd97e48 R08: ffff88181f416ec0 R09: ffff88181c284470
  R10: 0000000000000002 R11: ffffffff8109e5ac R12: ffff88181c2660c0
  R13: ffff88181f416ec0 R14: 0000000000000000 R15: ffff88181c2660f0
  			     ^^^^^^^^^^^^^^^^^

  27:   4c 0f 45 f2             cmovne %rdx,%r14
  2b:*  49 8b 46 08             mov    0x8(%r14),%rax           <-- trapping instruction
  2f:   44 8b b8 00 01 00 00    mov    0x100(%rax),%r15d

  ...

  Call Trace:
   worker_thread
   ? rescuer_thread
   kthread
   ? kthread_create_on_node
   ret_from_fork
   ? kthread_create_on_node
  Code: 48 89 e5 41 57 41 56 45 31 f6 41 55 41 54 49 89 fc 53 48 89 f3 48 83 ec 18 48 8b 06 4c 8b 6f 48 48 89 c2 30 d2 a8 04 4c 0f 45 f2 <49> 8b 46 08 44 8b b8 00 01 00 00 41 c1 ef 05 44 89 f8 83 e0 01
  RIP  [<ffffffff810980a1>] process_one_work
   RSP <ffff88181bd97e08>
  CR2: 0000000000000008
  ---[ end trace 8229a011b97532a0 ]---
  Kernel panic - not syncing: Fatal exception
  ---[ end Kernel panic - not syncing: Fatal exception

Signed-off-by: Borislav Petkov <bp@suse.de>
---
 arch/x86/kernel/cpu/mcheck/mce.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 478f81a6d824..216d44d074df 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -1645,6 +1645,8 @@ void (*machine_check_vector)(struct pt_regs *, long error_code) =
  */
 void mcheck_cpu_init(struct cpuinfo_x86 *c)
 {
+	static bool __mce_init_once;
+
 	if (mca_cfg.disabled)
 		return;
 
@@ -1665,8 +1667,11 @@ void mcheck_cpu_init(struct cpuinfo_x86 *c)
 		return;
 	}
 
-	INIT_WORK(&mce_work, mce_process_work);
-	init_irq_work(&mce_irq_work, mce_irq_work_cb);
+	if (!__mce_init_once) {
+		INIT_WORK(&mce_work, mce_process_work);
+		init_irq_work(&mce_irq_work, mce_irq_work_cb);
+		__mce_init_once = 1;
+	}
 
 	machine_check_vector = do_machine_check;
 
-- 
2.3.5

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1168741 — Re: [PATCH] x86/mce: Initialize workqueues only once

FromBorislav Petkov <bp@suse.de>
Date2015-06-19 14:30 +0200
SubjectRe: [PATCH] x86/mce: Initialize workqueues only once
Message-ID<pD3AK-7hF-25@gated-at.bofh.it>
In reply to#1168654
On Fri, Jun 19, 2015 at 11:27:18AM +0200, Borislav Petkov wrote:
> Here's the fix:

Injected error:

[  201.237241] EINJ: Error INJection is initialized.
[  246.146290] [Firmware Bug]: APEI: Invalid bit width + offset in GAR [0x1234/64/0/3/0]
[  257.868432] Disabling lock debugging due to kernel taint
[  259.808068] mce: [Hardware Error]: CPU 61: Machine Check Exception: 5 Bank 7: be00000000010090
[  257.868432] mce: [Hardware Error]: Machine check events logged
[  259.808068] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff81344bbf> {intel_idle+0xbf/0x130}
[  259.808068] mce: [Hardware Error]: TSC 98aa47539f28 ADDR 1200 MISC 142303c86 
[  259.808068] mce: [Hardware Error]: PROCESSOR 0:306e7 TIME 1434716502 SOCKET 0 APIC 3 microcode 700
[  259.808068] EDAC sbridge MC3: HANDLING MCE MEMORY ERROR
[  259.808068] EDAC sbridge MC3: CPU 61: Machine Check Exception: 5 Bank 7: be00000000010090
[  259.808068] EDAC sbridge MC3: TSC 98aa47539f28 
[  259.808068] EDAC sbridge MC3: ADDR 1200 EDAC sbridge MC3: MISC 142303c86 
[  259.808068] EDAC sbridge MC3: PROCESSOR 0:306e7 TIME 1434716502 SOCKET 0 APIC 3
[  259.808068] EDAC MC3: 0 UE memory read error on CPU_SrcID#0_Channel#0_DIMM#0 (channel:0 slot:0 page:0x1 offset:0x200 grain:32 -  recoverable area:DRAM err_code:0001:0090 socket:0 channel_mask:4 rank:0)

Let's see whether something's in the MCA MSRs when this monster reboots
(it takes forever...).

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web