Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1582322 > unrolled thread

Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after system panic

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after  system panic Xunlei Pang <xpang@redhat.com> - 2017-02-16 06:40 +0100
    Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after  system panic Borislav Petkov <bp@alien8.de> - 2017-02-16 11:20 +0100
      Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after  system panic Xunlei Pang <xpang@redhat.com> - 2017-02-16 13:00 +0100
        Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after  system panic Borislav Petkov <bp@alien8.de> - 2017-02-16 13:30 +0100
          Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after  system panic Xunlei Pang <xpang@redhat.com> - 2017-02-17 03:00 +0100
            Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after  system panic Borislav Petkov <bp@alien8.de> - 2017-02-17 10:10 +0100
              Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after  system panic Xunlei Pang <xpang@redhat.com> - 2017-02-17 17:20 +0100
                RE: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after  system panic "Luck, Tony" <tony.luck@intel.com> - 2017-02-21 19:50 +0100
                  Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after  system panic Xunlei Pang <xpang@redhat.com> - 2017-02-22 06:50 +0100

#1582322 — Re: [PATCH] x86/mce: Keep quiet in case of broadcasted mce after system panic

On 01/26/2017 at 02:44 PM, Borislav Petkov wrote:
> On Thu, Jan 26, 2017 at 02:30:02PM +0800, Xunlei Pang wrote:
>> The hardware machine check is hard to reproduce, but the mce code of
>> RHEL7 is quite the same as that of tip/master, anyway we are able to
>> inject software mce to reproduce it.
> Please give me your exact steps so that I can try to reproduce it here
> too.
>

Hi Borislav,

I tried to use qemu to inject SRAO("mce -b 0 0 0xb100000000000000 0x5 0x0 0x0"),
it works well in 1st kernel, but it doesn't work for 1st kernel after kdump boots(seems
the cpus remain in 1st kernel don't respond to the simulated broadcasting mce).

But in theory, we know cpus belong to kdump kernel can't respond to the
old mce handler, so a single SRAO injection in 1st kernel should be similar.
For example, I used "... -smp 2 -cpu Haswell" to launch a simulation with broadcast
mce supported, and inject SRAO to cpu0 only through qemu monitor
"mce 0 0 0xb100000000000000 0x5 0x0 0x0", cpu0 will timeout/panic and reboot
the machine as follows(running on linux-4.9):
  Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
  Kernel Offset: disabled
  Rebooting in 30 seconds..

Regards,
Xunlei

[toc] | [next] | [standalone]

#1582437

On Thu, Feb 16, 2017 at 01:36:37PM +0800, Xunlei Pang wrote:
> I tried to use qemu to inject SRAO("mce -b 0 0 0xb100000000000000 0x5 0x0 0x0"),
> it works well in 1st kernel, but it doesn't work for 1st kernel after kdump boots(seems
> the cpus remain in 1st kernel don't respond to the simulated broadcasting mce).
> 
> But in theory, we know cpus belong to kdump kernel can't respond to the
> old mce handler, so a single SRAO injection in 1st kernel should be similar.
> For example, I used "... -smp 2 -cpu Haswell" to launch a simulation with broadcast
> mce supported, and inject SRAO to cpu0 only through qemu monitor
> "mce 0 0 0xb100000000000000 0x5 0x0 0x0", cpu0 will timeout/panic and reboot
> the machine as follows(running on linux-4.9):
>   Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler

Sounds to me like you're trying hard to prove some point of yours which
doesn't make much sense to me. And when you say "in theory", that makes
it even less believable. So I remember asking you for exact steps. That
above doesn't read like steps but like some babbling and I've actually
tried to make sense of it for a couple of minutes but failed.

So lemme spell it out for ya. I'd like for you to give me this:

1. Build kernel with this config
2. Boot it in kvm with this settings
3. Do this in the guest
4. Do that in the guest
5. ...
6. ...


And all should be exact commands so that I can do them here on my machine.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1582514

On 02/16/2017 at 06:18 PM, Borislav Petkov wrote:
> On Thu, Feb 16, 2017 at 01:36:37PM +0800, Xunlei Pang wrote:
>> I tried to use qemu to inject SRAO("mce -b 0 0 0xb100000000000000 0x5 0x0 0x0"),
>> it works well in 1st kernel, but it doesn't work for 1st kernel after kdump boots(seems
>> the cpus remain in 1st kernel don't respond to the simulated broadcasting mce).
>>
>> But in theory, we know cpus belong to kdump kernel can't respond to the
>> old mce handler, so a single SRAO injection in 1st kernel should be similar.
>> For example, I used "... -smp 2 -cpu Haswell" to launch a simulation with broadcast
>> mce supported, and inject SRAO to cpu0 only through qemu monitor
>> "mce 0 0 0xb100000000000000 0x5 0x0 0x0", cpu0 will timeout/panic and reboot
>> the machine as follows(running on linux-4.9):
>>   Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
> Sounds to me like you're trying hard to prove some point of yours which
> doesn't make much sense to me. And when you say "in theory", that makes
> it even less believable. So I remember asking you for exact steps. That
> above doesn't read like steps but like some babbling and I've actually
> tried to make sense of it for a couple of minutes but failed.
>
> So lemme spell it out for ya. I'd like for you to give me this:
>
> 1. Build kernel with this config
> 2. Boot it in kvm with this settings
> 3. Do this in the guest
> 4. Do that in the guest
> 5. ...
> 6. ...
>
>
> And all should be exact commands so that I can do them here on my machine.
>

Sorry, missed your point.

The steps should be as follows:
1. Prepare a multi-core intel machine with broadcasted mce support.
    Enable kdump(crashkernel=256M) and configure kdump kernel to boot with "nr_cpus=1".
2. Activate kdump, and crash the first kernel on some cpu, say cpu1
    (taskset -c 1 echo 0 > /proc/sysrq-trigger), then kdump will boot on cpu1.
3. After kdump boots up(let it enter shell), trigger a SRAO on cpu1
   (QEMU monitor cmd: mce -b 1 0 0xb100000000000000 0x5 0x0 0x0),
    then mce will be broadcast to the other cpus which are still running
    in the first kernel(i.e. looping in crash_nmi_callback).
    If you own some hardware to inject mce, it would be great, as QEMU does not work correctly for me.
4. Then something like below is expected to happen:

[    1.468556] tsc: Refined TSC clocksource calibration: 2933.437 MHz
         Starting Kdump Vmcore Save Service...
kdump: saving to /sysroot//var/crash/127.0.0.1-2015-09-01-05:07:03/
kdump: saving vmcore-dmesg.txt
[   39.000010] mce: [Hardware Error]: CPU 0: Machine Check Exception: 0 Bank 2: bd0000000000017a
[   39.000010] mce: [Hardware Error]: TSC 0 ADDR 61600000 MISC 8c 
[   39.000010] mce: [Hardware Error]: PROCESSOR 0:106a3 TIME 1441083980 SOCKET 0 APIC 0 microcode 1
[   39.000010] mce: [Hardware Error]: Run the above through 'mcelog --ascii'
[   39.000010] Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
[   39.000010] Shutting down cpus with NMI
[    1.758463] Uhhuh. NMI received for unknown reason 20 on CPU 0.
[    1.758463] Do you have a strange power saving mode enabled?
[    1.758463] Dazed and confused, but trying to continue
[   39.000010] Rebooting in 30 seconds..

Regards,
Xunlei

[toc] | [prev] | [next] | [standalone]

#1582525

On Thu, Feb 16, 2017 at 07:52:09PM +0800, Xunlei Pang wrote:
>     then mce will be broadcast to the other cpus which are still running
>     in the first kernel(i.e. looping in crash_nmi_callback).

Simple: the crash code should really mark CPUs as not being online:

void do_machine_check(struct pt_regs *regs, long error_code)

	...

        /* If this CPU is offline, just bail out. */
        if (cpu_is_offline(smp_processor_id())) {
                u64 mcgstatus;

                mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
                if (mcgstatus & MCG_STATUS_RIPV) {
                        mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
                        return;
                }
        }

because looping in crash_nmi_callback() does not really denote them as
CPUs being online.

And just so that you don't disturb the machine too much during crashing,
you could simply clear them from the online masks, i.e., perhaps call
remove_cpu_from_maps() with the proper locking around it instead of
doing a full cpu_down().

The machine will be killed anyway after kdump is done writing out
memory.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1583029

On 02/16/2017 at 08:22 PM, Borislav Petkov wrote:
> On Thu, Feb 16, 2017 at 07:52:09PM +0800, Xunlei Pang wrote:
>>     then mce will be broadcast to the other cpus which are still running
>>     in the first kernel(i.e. looping in crash_nmi_callback).
> Simple: the crash code should really mark CPUs as not being online:
>
> void do_machine_check(struct pt_regs *regs, long error_code)
>
> 	...
>
>         /* If this CPU is offline, just bail out. */
>         if (cpu_is_offline(smp_processor_id())) {
>                 u64 mcgstatus;
>
>                 mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
>                 if (mcgstatus & MCG_STATUS_RIPV) {
>                         mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
>                         return;
>                 }
>         }
>
> because looping in crash_nmi_callback() does not really denote them as
> CPUs being online.
>
> And just so that you don't disturb the machine too much during crashing,
> you could simply clear them from the online masks, i.e., perhaps call
> remove_cpu_from_maps() with the proper locking around it instead of
> doing a full cpu_down().

It changes the value of cpu_online_mask/etc which will cause confusion to vmcore analysis.
Moreover, for the code(see comment inlined)

        if (cpu_is_offline(smp_processor_id())) {
                u64 mcgstatus;

                mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
                if (mcgstatus & MCG_STATUS_RIPV) { // This condition may be not true, the mce triggered on kdump cpu 
                                                                     // doesn't need to have this bit set for the other cpus remain in 1st kernel. 
                        mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
                        return;
                }
        }


Regards,
Xunlei

>
> The machine will be killed anyway after kdump is done writing out
> memory.
>

[toc] | [prev] | [next] | [standalone]

#1583225

On Fri, Feb 17, 2017 at 09:53:21AM +0800, Xunlei Pang wrote:
> It changes the value of cpu_online_mask/etc which will cause confusion to vmcore analysis.

Then export the crashing_cpu variable, initialize it to something
invalid in the first kernel, -1 for example, and test it in the #MC
handlier like this:

	int cpu;

	...

	cpu = smp_processor_id();

	if (cpu_is_offline(cpu) ||
	    ((crashing_cpu != -1) && (crashing_cpu != cpu)) {
                u64 mcgstatus;

                mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
                if (mcgstatus & MCG_STATUS_RIPV) {
                        mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
			return;
		}
	}

> Moreover, for the code(see comment inlined)
> 
>         if (cpu_is_offline(smp_processor_id())) {
>                 u64 mcgstatus;
> 
>                 mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
>                 if (mcgstatus & MCG_STATUS_RIPV) { // This condition may be not true, the mce triggered on kdump cpu 
>                                                                      // doesn't need to have this bit set for the other cpus remain in 1st kernel. 

Is this on kvm or on a real hardware? Because for kvm I don't care. And
don't say "theoretically".

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1583565

On 02/17/2017 at 05:07 PM, Borislav Petkov wrote:
> On Fri, Feb 17, 2017 at 09:53:21AM +0800, Xunlei Pang wrote:
>> It changes the value of cpu_online_mask/etc which will cause confusion to vmcore analysis.
> Then export the crashing_cpu variable, initialize it to something
> invalid in the first kernel, -1 for example, and test it in the #MC
> handlier like this:
>
> 	int cpu;
>
> 	...
>
> 	cpu = smp_processor_id();
>
> 	if (cpu_is_offline(cpu) ||
> 	    ((crashing_cpu != -1) && (crashing_cpu != cpu)) {
>                 u64 mcgstatus;
>
>                 mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
>                 if (mcgstatus & MCG_STATUS_RIPV) {
>                         mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
> 			return;
> 		}
> 	}

Yes, it is doable, I will do some tests later.

>> Moreover, for the code(see comment inlined)
>>
>>         if (cpu_is_offline(smp_processor_id())) {
>>                 u64 mcgstatus;
>>
>>                 mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
>>                 if (mcgstatus & MCG_STATUS_RIPV) { // This condition may be not true, the mce triggered on kdump cpu 
>>                                                                      // doesn't need to have this bit set for the other cpus remain in 1st kernel. 
> Is this on kvm or on a real hardware? Because for kvm I don't care. And
> don't say "theoretically".
>

It's from my understanding, I didn't get the explicit description from the intel SDM on this point.
If a broadcast SRAO comes on real hardware, will MSR_IA32_MCG_STATUS of each cpu have MCG_STATUS_RIPV bit set?

Regards,
Xunlei

[toc] | [prev] | [next] | [standalone]

#1585616

> It's from my understanding, I didn't get the explicit description from the intel SDM on this point.
> If a broadcast SRAO comes on real hardware, will MSR_IA32_MCG_STATUS of each cpu have MCG_STATUS_RIPV bit set?

MCG_STATUS is a per-thread MSR and will contain the status appropriate for that thread when #MC is delivered.
So the RIPV bit will be set if, and only if, the thread saved a valid return address for this exception. The net result
is that it is almost always set for "innocent bystander" CPUs that were dragged into the exception handler because
of a broadcast #MC. We make the test because if it isn't set, then the do_machine_check() had better not return
because we have no idea where it will return to - since there is not a valid return IP.

-Tony

[toc] | [prev] | [next] | [standalone]

#1585938

On 02/22/2017 at 02:20 AM, Luck, Tony wrote:
>> It's from my understanding, I didn't get the explicit description from the intel SDM on this point.
>> If a broadcast SRAO comes on real hardware, will MSR_IA32_MCG_STATUS of each cpu have MCG_STATUS_RIPV bit set?
> MCG_STATUS is a per-thread MSR and will contain the status appropriate for that thread when #MC is delivered.
> So the RIPV bit will be set if, and only if, the thread saved a valid return address for this exception. The net result
> is that it is almost always set for "innocent bystander" CPUs that were dragged into the exception handler because
> of a broadcast #MC. We make the test because if it isn't set, then the do_machine_check() had better not return
> because we have no idea where it will return to - since there is not a valid return IP.
>

Got it, thanks for the details.

Regards,
Xunlei

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web