Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1742450 > unrolled thread

random insta-reboots on AMD Phenom II

Started byAdam Borowski <kilobyte@angband.pl>
First post2017-09-30 04:10 +0200
Last post2017-10-01 15:10 +0200
Articles 12 — 6 participants

Back to article view | Back to linux.kernel

Contents

  random insta-reboots on AMD Phenom II Adam Borowski <kilobyte@angband.pl> - 2017-09-30 04:10 +0200
    Re: random insta-reboots on AMD Phenom II Borislav Petkov <bp@alien8.de> - 2017-09-30 13:20 +0200
      Re: random insta-reboots on AMD Phenom II Adam Borowski <kilobyte@angband.pl> - 2017-09-30 13:30 +0200
        Re: random insta-reboots on AMD Phenom II Borislav Petkov <bp@alien8.de> - 2017-09-30 14:00 +0200
          Re: random insta-reboots on AMD Phenom II Markus Trippelsdorf <markus@trippelsdorf.de> - 2017-09-30 14:50 +0200
            Re: random insta-reboots on AMD Phenom II Brian Gerst <brgerst@gmail.com> - 2017-09-30 16:30 +0200
              Re: random insta-reboots on AMD Phenom II Markus Trippelsdorf <markus@trippelsdorf.de> - 2017-09-30 17:30 +0200
            Re: random insta-reboots on AMD Phenom II Borislav Petkov <bp@alien8.de> - 2017-09-30 18:00 +0200
              Re: random insta-reboots on AMD Phenom II Andy Lutomirski <luto@kernel.org> - 2017-09-30 18:10 +0200
          Re: random insta-reboots on AMD Phenom II Andy Lutomirski <luto@amacapital.net> - 2017-09-30 17:20 +0200
            Re: random insta-reboots on AMD Phenom II Borislav Petkov <bp@alien8.de> - 2017-09-30 17:50 +0200
          Re: random insta-reboots on AMD Phenom II Adam Borowski <kilobyte@angband.pl> - 2017-10-01 15:10 +0200

#1742450 — random insta-reboots on AMD Phenom II

FromAdam Borowski <kilobyte@angband.pl>
Date2017-09-30 04:10 +0200
Subjectrandom insta-reboots on AMD Phenom II
Message-ID<uvfo5-69K-3@gated-at.bofh.it>
Hi!
I'm afraid I see random instant reboots on current -rc, approximately
once per day, only under CPU load.  There's nothing on serial/etc -- just
an immediate reboot.  4.13 works perfectly; last kernel I've tried is
v4.14-rc2-165-g770b782f555d.  gcc 7.2.0-7 (Debian).

CPU is AMD Phenom II X6 1055T (family 10h).

Sometimes it dies within a few minutes of load, sometimes all is fine for a
couple of days.  This randomness makes bisecting not really an option.

Any hints how to debug this?


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ We domesticated dogs 36000 years ago; together we chased
⣾⠁⢰⠒⠀⣿⡁ animals, hung out and licked or scratched our private parts.
⢿⡄⠘⠷⠚⠋⠀ Cats domesticated us 9500 years ago, and immediately we got
⠈⠳⣄⠀⠀⠀⠀ agriculture, towns then cities.     -- whitroth on /.

[toc] | [next] | [standalone]

#1742601

FromBorislav Petkov <bp@alien8.de>
Date2017-09-30 13:20 +0200
Message-ID<uvnYl-3oZ-7@gated-at.bofh.it>
In reply to#1742450
On Sat, Sep 30, 2017 at 04:05:16AM +0200, Adam Borowski wrote:
> Any hints how to debug this?

Do

rdmsr -a 0xc0010015

as root and paste it here.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1742603

FromAdam Borowski <kilobyte@angband.pl>
Date2017-09-30 13:30 +0200
Message-ID<uvo81-3sE-9@gated-at.bofh.it>
In reply to#1742601
On Sat, Sep 30, 2017 at 01:11:37PM +0200, Borislav Petkov wrote:
> On Sat, Sep 30, 2017 at 04:05:16AM +0200, Adam Borowski wrote:
> > Any hints how to debug this?
> 
> Do
> rdmsr -a 0xc0010015
> as root and paste it here.

1000010
1000010
1000010
1000010
1000010
1000010

on both 4.13.4 and 4.14-rc2+.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ We domesticated dogs 36000 years ago; together we chased
⣾⠁⢰⠒⠀⣿⡁ animals, hung out and licked or scratched our private parts.
⢿⡄⠘⠷⠚⠋⠀ Cats domesticated us 9500 years ago, and immediately we got
⠈⠳⣄⠀⠀⠀⠀ agriculture, towns then cities.     -- whitroth on /.

[toc] | [prev] | [next] | [standalone]

#1742624

FromBorislav Petkov <bp@alien8.de>
Date2017-09-30 14:00 +0200
Message-ID<uvoB5-3DX-7@gated-at.bofh.it>
In reply to#1742603
On Sat, Sep 30, 2017 at 01:29:03PM +0200, Adam Borowski wrote:
> On Sat, Sep 30, 2017 at 01:11:37PM +0200, Borislav Petkov wrote:
> > On Sat, Sep 30, 2017 at 04:05:16AM +0200, Adam Borowski wrote:
> > > Any hints how to debug this?
> > 
> > Do
> > rdmsr -a 0xc0010015
> > as root and paste it here.
> 
> 1000010
> 1000010
> 1000010
> 1000010
> 1000010
> 1000010
> 
> on both 4.13.4 and 4.14-rc2+.

Boot into -rc2+ and do as root:

# wrmsr -a 0xc0010015 0x1000018

If the issue gets fixed then Mr. Luto better revert the new lazy TLB
flushing fun'n'games for 4.14 before it is too late and that kernel
releases b0rked.

Thx.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1742628

FromMarkus Trippelsdorf <markus@trippelsdorf.de>
Date2017-09-30 14:50 +0200
Message-ID<uvpnr-4bL-5@gated-at.bofh.it>
In reply to#1742624
On 2017.09.30 at 13:53 +0200, Borislav Petkov wrote:
> On Sat, Sep 30, 2017 at 01:29:03PM +0200, Adam Borowski wrote:
> > On Sat, Sep 30, 2017 at 01:11:37PM +0200, Borislav Petkov wrote:
> > > On Sat, Sep 30, 2017 at 04:05:16AM +0200, Adam Borowski wrote:
> > > > Any hints how to debug this?
> > > 
> > > Do
> > > rdmsr -a 0xc0010015
> > > as root and paste it here.
> > 
> > 1000010
> > 1000010
> > 1000010
> > 1000010
> > 1000010
> > 1000010
> > 
> > on both 4.13.4 and 4.14-rc2+.
> 
> Boot into -rc2+ and do as root:
> 
> # wrmsr -a 0xc0010015 0x1000018
> 
> If the issue gets fixed then Mr. Luto better revert the new lazy TLB
> flushing fun'n'games for 4.14 before it is too late and that kernel
> releases b0rked.

The issue does get fixed by setting TlbCacheDis to 1. I have been
running it for the last few weeks without any problems. 
Performance is not affected at all. So it might by easier to just set
the bit for older AMD processors as a boot quirk.
Changing the TLB code so late might not be a good idea...

-- 
Markus

[toc] | [prev] | [next] | [standalone]

#1742634

FromBrian Gerst <brgerst@gmail.com>
Date2017-09-30 16:30 +0200
Message-ID<uvqWd-5id-3@gated-at.bofh.it>
In reply to#1742628
On Sat, Sep 30, 2017 at 8:47 AM, Markus Trippelsdorf
<markus@trippelsdorf.de> wrote:
> On 2017.09.30 at 13:53 +0200, Borislav Petkov wrote:
>> On Sat, Sep 30, 2017 at 01:29:03PM +0200, Adam Borowski wrote:
>> > On Sat, Sep 30, 2017 at 01:11:37PM +0200, Borislav Petkov wrote:
>> > > On Sat, Sep 30, 2017 at 04:05:16AM +0200, Adam Borowski wrote:
>> > > > Any hints how to debug this?
>> > >
>> > > Do
>> > > rdmsr -a 0xc0010015
>> > > as root and paste it here.
>> >
>> > 1000010
>> > 1000010
>> > 1000010
>> > 1000010
>> > 1000010
>> > 1000010
>> >
>> > on both 4.13.4 and 4.14-rc2+.
>>
>> Boot into -rc2+ and do as root:
>>
>> # wrmsr -a 0xc0010015 0x1000018
>>
>> If the issue gets fixed then Mr. Luto better revert the new lazy TLB
>> flushing fun'n'games for 4.14 before it is too late and that kernel
>> releases b0rked.
>
> The issue does get fixed by setting TlbCacheDis to 1. I have been
> running it for the last few weeks without any problems.
> Performance is not affected at all. So it might by easier to just set
> the bit for older AMD processors as a boot quirk.
> Changing the TLB code so late might not be a good idea...

Looking at the AMD K10 revision guide
(http://support.amd.com/TechDocs/41322_10h_Rev_Gd.pdf), errata #298
that this fixes should only apply to revisions DR-BA and DR-B2, which
include the original Phenom, but not Phenom II.  The Phenom II X6 is
revision PH-E0, which does not have this errata.

--
Brian Gerst

[toc] | [prev] | [next] | [standalone]

#1742640

FromMarkus Trippelsdorf <markus@trippelsdorf.de>
Date2017-09-30 17:30 +0200
Message-ID<uvrSh-5UR-1@gated-at.bofh.it>
In reply to#1742634
On 2017.09.30 at 10:20 -0400, Brian Gerst wrote:
> On Sat, Sep 30, 2017 at 8:47 AM, Markus Trippelsdorf
> <markus@trippelsdorf.de> wrote:
> > On 2017.09.30 at 13:53 +0200, Borislav Petkov wrote:
> >> On Sat, Sep 30, 2017 at 01:29:03PM +0200, Adam Borowski wrote:
> >> > On Sat, Sep 30, 2017 at 01:11:37PM +0200, Borislav Petkov wrote:
> >> > > On Sat, Sep 30, 2017 at 04:05:16AM +0200, Adam Borowski wrote:
> >> > > > Any hints how to debug this?
> >> > >
> >> > > Do
> >> > > rdmsr -a 0xc0010015
> >> > > as root and paste it here.
> >> >
> >> > 1000010
> >> > 1000010
> >> > 1000010
> >> > 1000010
> >> > 1000010
> >> > 1000010
> >> >
> >> > on both 4.13.4 and 4.14-rc2+.
> >>
> >> Boot into -rc2+ and do as root:
> >>
> >> # wrmsr -a 0xc0010015 0x1000018
> >>
> >> If the issue gets fixed then Mr. Luto better revert the new lazy TLB
> >> flushing fun'n'games for 4.14 before it is too late and that kernel
> >> releases b0rked.
> >
> > The issue does get fixed by setting TlbCacheDis to 1. I have been
> > running it for the last few weeks without any problems.
> > Performance is not affected at all. So it might by easier to just set
> > the bit for older AMD processors as a boot quirk.
> > Changing the TLB code so late might not be a good idea...
> 
> Looking at the AMD K10 revision guide
> (http://support.amd.com/TechDocs/41322_10h_Rev_Gd.pdf), errata #298
> that this fixes should only apply to revisions DR-BA and DR-B2, which
> include the original Phenom, but not Phenom II.  The Phenom II X6 is
> revision PH-E0, which does not have this errata.

It has nothing to do with errata #298. The new lazy TLB code causes
MCEs, because the page tables may now contain garbage.
See the long "Current mainline git (24e700e291d52bd2) hangs when
building e.g. perf" LKML thread.
-- 
Markus

[toc] | [prev] | [next] | [standalone]

#1742647

FromBorislav Petkov <bp@alien8.de>
Date2017-09-30 18:00 +0200
Message-ID<uvslj-66i-9@gated-at.bofh.it>
In reply to#1742628
On Sat, Sep 30, 2017 at 02:47:11PM +0200, Markus Trippelsdorf wrote:
> Changing the TLB code so late might not be a good idea...

The new lazy code is too risky to keep as we don't know what else will
break. The conservative and thus safe thing to do is to revert to the
old behavior for old machines.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1742649

FromAndy Lutomirski <luto@kernel.org>
Date2017-09-30 18:10 +0200
Message-ID<uvsv0-6pn-1@gated-at.bofh.it>
In reply to#1742647
On Sat, Sep 30, 2017 at 8:50 AM, Borislav Petkov <bp@alien8.de> wrote:
> On Sat, Sep 30, 2017 at 02:47:11PM +0200, Markus Trippelsdorf wrote:
>> Changing the TLB code so late might not be a good idea...
>
> The new lazy code is too risky to keep as we don't know what else will
> break. The conservative and thus safe thing to do is to revert to the
> old behavior for old machines.

Agreed.

The only problem is that the code has changed so much on top of the
problematic commit that just reverting it won't work.

>
> --
> Regards/Gruss,
>     Boris.
>
> Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1742637

FromAndy Lutomirski <luto@amacapital.net>
Date2017-09-30 17:20 +0200
Message-ID<uvrIC-5R9-15@gated-at.bofh.it>
In reply to#1742624

> On Sep 30, 2017, at 4:53 AM, Borislav Petkov <bp@alien8.de> wrote:
> 
>> On Sat, Sep 30, 2017 at 01:29:03PM +0200, Adam Borowski wrote:
>>> On Sat, Sep 30, 2017 at 01:11:37PM +0200, Borislav Petkov wrote:
>>>> On Sat, Sep 30, 2017 at 04:05:16AM +0200, Adam Borowski wrote:
>>>> Any hints how to debug this?
>>> 
>>> Do
>>> rdmsr -a 0xc0010015
>>> as root and paste it here.
>> 
>> 1000010
>> 1000010
>> 1000010
>> 1000010
>> 1000010
>> 1000010
>> 
>> on both 4.13.4 and 4.14-rc2+.
> 
> Boot into -rc2+ and do as root:
> 
> # wrmsr -a 0xc0010015 0x1000018
> 
> If the issue gets fixed then Mr. Luto better revert the new lazy TLB
> flushing fun'n'games for 4.14 before it is too late and that kernel
> releases b0rked.

Yeah, working on it.  It's not a straightforward revert.

> 
> Thx.
> 
> -- 
> Regards/Gruss,
>    Boris.
> 
> Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1742644

FromBorislav Petkov <bp@alien8.de>
Date2017-09-30 17:50 +0200
Message-ID<uvsbE-62p-5@gated-at.bofh.it>
In reply to#1742637
On Sat, Sep 30, 2017 at 08:11:51AM -0700, Andy Lutomirski wrote:
> Yeah, working on it.  It's not a straightforward revert.

Thanks. At least you have testers :-)

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

[toc] | [prev] | [next] | [standalone]

#1742790

FromAdam Borowski <kilobyte@angband.pl>
Date2017-10-01 15:10 +0200
Message-ID<uvMam-2AV-7@gated-at.bofh.it>
In reply to#1742624
On Sat, Sep 30, 2017 at 01:53:02PM +0200, Borislav Petkov wrote:
> On Sat, Sep 30, 2017 at 01:29:03PM +0200, Adam Borowski wrote:
> > On Sat, Sep 30, 2017 at 01:11:37PM +0200, Borislav Petkov wrote:
> > > On Sat, Sep 30, 2017 at 04:05:16AM +0200, Adam Borowski wrote:

> Boot into -rc2+ and do as root:
> 
> # wrmsr -a 0xc0010015 0x1000018

Seems to help, thus it's indeed this issue.  I failed to mention that "once
per day" meant a day of regular use, of which heavy loads were only a tiny
fraction.  I've applied this register setting, then kept the machine busy
(mostly with randconfig kernel builds) for a day, no explosions yet -- so
there's a good chance the problem would have triggered.

> If the issue gets fixed then Mr. Luto better revert the new lazy TLB
> flushing fun'n'games for 4.14 before it is too late and that kernel
> releases b0rked.

I have no clue about these matters so I'll leave it to you guys.  But, as
the other report I see gives different effects (frequent segfaults vs rare
insta-reboots), you do want to include my machine in testing.

Thanks for the workaround!

Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ We domesticated dogs 36000 years ago; together we chased
⣾⠁⢰⠒⠀⣿⡁ animals, hung out and licked or scratched our private parts.
⢿⡄⠘⠷⠚⠋⠀ Cats domesticated us 9500 years ago, and immediately we got
⠈⠳⣄⠀⠀⠀⠀ agriculture, towns then cities.     -- whitroth on /.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web