Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1344523 > unrolled thread

Re: BUG: unable to handle kernel paging request from pty_write [was: Linux 4.4.2]

Back to article view | Back to linux.kernel

Contents

  Re: BUG: unable to handle kernel paging request from pty_write [was:  Linux 4.4.2] Linus Torvalds <torvalds@linux-foundation.org> - 2016-02-26 19:10 +0100
    Re: BUG: unable to handle kernel paging request from pty_write [was:  Linux 4.4.2] Peter Hurley <peter@hurleysoftware.com> - 2016-02-26 19:20 +0100
    Re: BUG: unable to handle kernel paging request from pty_write [was:  Linux 4.4.2] Borislav Petkov <bp@alien8.de> - 2016-02-26 19:20 +0100
    Re: BUG: unable to handle kernel paging request from pty_write [was:  Linux 4.4.2] Linus Torvalds <torvalds@linux-foundation.org> - 2016-02-26 20:50 +0100
      Re: BUG: unable to handle kernel paging request from pty_write [was:  Linux 4.4.2] Robert Święcki <robert@swiecki.net> - 2016-02-26 21:10 +0100
        Re: BUG: unable to handle kernel paging request from pty_write [was:  Linux 4.4.2] Jiri Slaby <jslaby@suse.cz> - 2016-02-29 08:40 +0100
          Re: BUG: unable to handle kernel paging request from pty_write [was:  Linux 4.4.2] Henrique de Moraes Holschuh <hmh@hmh.eng.br> - 2016-02-29 13:50 +0100

#1344523 — Re: BUG: unable to handle kernel paging request from pty_write [was: Linux 4.4.2]

On Fri, Feb 26, 2016 at 9:52 AM, Peter Hurley <peter@hurleysoftware.com> wrote:
>
> So more analysis would seem to confirm that RSP has been bumped +8
> while in ttwu_stat() so when the epilog executed, register restore
> was off by 1 qword. However, there's nothing in ttwu_stat() that
> results in stack pointer offset by +1 qword from prolog.

I agree.

That's why I'm actually starting to suspect that it's an AMD microcode
bug that we know very little about. There's apparently register
corruption (the guess being from NMI handling, but virtualization was
also involved) under some circumstances.

Of course, if Jiri isn't actually running this on an AMD CPU, that
theory flies right out the window. But we do have a reported oops on
the security list that looks totally different in the big picture, but
shares the exact same "corrupted stack pointer register state
resulting in crazy instruction pointer, resulting in NX fault"
behavior in the end.

In the other case, microcode patchlevel 0x0600081c was fine, and
0x06000832 is the one exhibiting the corruption problem.

I've contacted Robert Święcki (who found the microcode problem) in
case he wants to weigh in in this thread.. He was talking to some AMD
people, but I don't know the exactly who.

                  Linus

[toc] | [next] | [standalone]

#1344527

On 02/26/2016 10:05 AM, Linus Torvalds wrote:
> On Fri, Feb 26, 2016 at 9:52 AM, Peter Hurley <peter@hurleysoftware.com> wrote:
>>
>> So more analysis would seem to confirm that RSP has been bumped +8
>> while in ttwu_stat() so when the epilog executed, register restore
>> was off by 1 qword. However, there's nothing in ttwu_stat() that
>> results in stack pointer offset by +1 qword from prolog.
> 
> I agree.
> 
> That's why I'm actually starting to suspect that it's an AMD microcode
> bug that we know very little about. There's apparently register
> corruption (the guess being from NMI handling, but virtualization was
> also involved) under some circumstances.

Yep, that could explain it.

> Of course, if Jiri isn't actually running this on an AMD CPU, that
> theory flies right out the window.

I'll wait for Jiri to confirm before sinking more time here.


> But we do have a reported oops on
> the security list that looks totally different in the big picture, but
> shares the exact same "corrupted stack pointer register state
> resulting in crazy instruction pointer, resulting in NX fault"
> behavior in the end.
> 
> In the other case, microcode patchlevel 0x0600081c was fine, and
> 0x06000832 is the one exhibiting the corruption problem.
> 
> I've contacted Robert Święcki (who found the microcode problem) in
> case he wants to weigh in in this thread.. He was talking to some AMD
> people, but I don't know the exactly who.

Ok, thanks for the info.

[toc] | [prev] | [next] | [standalone]

#1344528

On Fri, Feb 26, 2016 at 10:05:26AM -0800, Linus Torvalds wrote:
> Of course, if Jiri isn't actually running this on an AMD CPU, that
> theory flies right out the window. But we do have a reported oops on
> the security list that looks totally different in the big picture, but
> shares the exact same "corrupted stack pointer register state
> resulting in crazy instruction pointer, resulting in NX fault"
> behavior in the end.
> 
> In the other case, microcode patchlevel 0x0600081c was fine, and
> 0x06000832 is the one exhibiting the corruption problem.
> 
> I've contacted Robert Święcki (who found the microcode problem) in
> case he wants to weigh in in this thread.. He was talking to some AMD
> people, but I don't know the exactly who.

It most likely is that problem. If Jiri is using the IBS machines -
and from quick look at http://labs.suse.cz/jslaby/bug-968218/gdb_log,
it looks like he is, then they are exactly those boxes with a b0rked
microcode patch. AMD is working on a fix.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1344626

On Fri, Feb 26, 2016 at 10:05 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> I've contacted Robert Święcki (who found the microcode problem) in
> case he wants to weigh in in this thread.. He was talking to some AMD
> people, but I don't know the exactly who.

And since it's looking increasingly likely that it's the same issue,
I'm adding Robert here explicitly to the cc so that he sees the
thread...

                Linus

[toc] | [prev] | [next] | [standalone]

#1344643

2016-02-26 20:44 GMT+01:00 Linus Torvalds <torvalds@linux-foundation.org>:

>> I've contacted Robert Święcki (who found the microcode problem) in
>> case he wants to weigh in in this thread.. He was talking to some AMD
>> people, but I don't know the exactly who.
>
> And since it's looking increasingly likely that it's the same issue,
> I'm adding Robert here explicitly to the cc so that he sees the
> thread...

Thx,

Some data I was able to gather:

It happens only with 0x6000832 ucode, and Piledriver-based CPUs: i.e.
newer AMD FX, and Opteron 300 series (4300, 6300 etc.).

The visible effects are in ~80% of cases incorrect RSP leading to bad
'rets' into kernel data/bss or stack-protector faults. But there are
also more elusive ones, like registers being cleared before use in
indirect memory fetches or so.

I can trigger it from within qemu guest (non-root), causing bad RIP in
the host kernel. When testing, a couple of times (maybe 2) out of
maybe 30 seen oopses, I was able to set it to user-space addresses
mapped in the guest. It greatly depends on timing, but I think with
some more effort and populating kernel stack with guest addresses it'd
be possible to create a more reliable qemu-guest to host ring0 escape.

I CC'd some AMD engineers from this list, and on of them replied with
"We are working on the final testing of a new microcode patch to
replace 0x06000832."
but without specifying any errata no, or ETA for the new ucode.

I can only now suggest not using 0x06000832 is possible (i.e. if it's
not embedded in BIOS), I tested a few from
http://www.amd64.org/microcode.html and only this version seemed
vulnerable.

PS. There's a bug on vmware pages -
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2061211
- which looks very similar to this problem (affects Opteron 6300 which
is Piledriver-based), and it was "somehow" patched by vmware in their
kernel. It points to AMD errata #815 -
http://support.amd.com/TechDocs/48063_15h_Mod_00h-0Fh_Rev_Guide.pdf -
but I cannot tell whether it's really the same problem, or whether it
can be somehow by-passed on the kernel side.

-- 
Robert Święcki

[toc] | [prev] | [next] | [standalone]

#1345525

On 02/26/2016, 08:59 PM, Robert Święcki wrote:
> It happens only with 0x6000832 ucode, and Piledriver-based CPUs: i.e.
> newer AMD FX, and Opteron 300 series (4300, 6300 etc.).

Ok, I can confirm this is:
AMD Opteron(tm) Processor 6348

And:
microcode: CPU0: patch_level=0x06000836

Thank all the interested parties!

-- 
js
suse labs

[toc] | [prev] | [next] | [standalone]

#1345757

On Mon, 29 Feb 2016, Jiri Slaby wrote:
> On 02/26/2016, 08:59 PM, Robert Święcki wrote:
> > It happens only with 0x6000832 ucode, and Piledriver-based CPUs: i.e.
> > newer AMD FX, and Opteron 300 series (4300, 6300 etc.).
> 
> Ok, I can confirm this is:
> AMD Opteron(tm) Processor 6348
> 
> And:
> microcode: CPU0: patch_level=0x06000836
> 
> Thank all the interested parties!

Jiri, does microcode 0x6000836 *fix* the erratum in microcode 0x6000832?

Or did you mean both 0x6000832 and 0x6000836 have the same erratum discussed
in this thread?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web