Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > linux.kernel > #1441308 > unrolled thread

Re: 4.7-rc6: NMI in intel_idle on HP Proliant G6

Started byMeelis Roos <mroos@linux.ee>
First post2016-07-12 14:30 +0200
Last post2016-07-13 16:00 +0200
Articles 16 — 5 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.


Contents

  Re: 4.7-rc6: NMI in intel_idle on HP Proliant G6 Meelis Roos <mroos@linux.ee> - 2016-07-12 14:30 +0200
    Re: 4.7-rc6: NMI in intel_idle on HP Proliant G6 "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-07-12 14:40 +0200
      Re: 4.7-rc6: NMI in intel_idle on HP Proliant G6 Meelis Roos <mroos@linux.ee> - 2016-07-13 09:20 +0200
      Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP  Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 10:20 +0200
        Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on  HP Proliant G6) Meelis Roos <mroos@linux.ee> - 2016-07-13 10:30 +0200
          Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP  Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 11:10 +0200
            Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on  HP Proliant G6) Meelis Roos <mroos@linux.ee> - 2016-07-13 11:20 +0200
              Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP  Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 11:50 +0200
            Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on  HP Proliant G6) Meelis Roos <mroos@linux.ee> - 2016-07-13 11:40 +0200
              Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP  Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 11:50 +0200
                Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP  Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 12:20 +0200
                  Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on  HP Proliant G6) Alex Williamson <alex.williamson@redhat.com> - 2016-07-13 17:00 +0200
                    Re: IOMMU+DMAR causing NMIs-s Meelis Roos <mroos@linux.ee> - 2016-07-14 00:00 +0200
                      Re: IOMMU+DMAR causing NMIs-s Alex Williamson <alex.williamson@redhat.com> - 2016-07-14 01:10 +0200
                        Re: IOMMU+DMAR causing NMIs-s Joerg Roedel <joro@8bytes.org> - 2016-07-14 10:40 +0200
            Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP  Proliant G6) Henrique de Moraes Holschuh <hmh@hmh.eng.br> - 2016-07-13 16:00 +0200

#1441308 — Re: 4.7-rc6: NMI in intel_idle on HP Proliant G6

FromMeelis Roos <mroos@linux.ee>
Date2016-07-12 14:30 +0200
SubjectRe: 4.7-rc6: NMI in intel_idle on HP Proliant G6
Message-ID<rU4Z8-4Ol-15@gated-at.bofh.it>
> On HP Proliant DL360 G6, Debian unstable 4.6 kernel runs fine but 
> selfcompiled 4.7-rc6 and 4.7-rc7 sometimes crash with NMI from 
> intel_idle. Sometimes it boots fine. With intel_idle disabled, it has 
> booted successful so far in 2 tries, one with rc6 and one with rc7.

More testing shows it can NMI on acpi_idle too, not just intel_idle.

> Screenshot with some backtrace: 
> http://kodu.ut.ee/~mroos/intel-idle-NMI.png

http://kodu.ut.ee/~mroos/acpi-idle-NMI.png

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [next] | [standalone]


#1441311

From"Rafael J. Wysocki" <rjw@rjwysocki.net>
Date2016-07-12 14:40 +0200
Message-ID<rU58K-4Sa-23@gated-at.bofh.it>
In reply to#1441308
On Tuesday, July 12, 2016 03:27:19 PM Meelis Roos wrote:
> > On HP Proliant DL360 G6, Debian unstable 4.6 kernel runs fine but 
> > selfcompiled 4.7-rc6 and 4.7-rc7 sometimes crash with NMI from 
> > intel_idle. Sometimes it boots fine. With intel_idle disabled, it has 
> > booted successful so far in 2 tries, one with rc6 and one with rc7.
> 
> More testing shows it can NMI on acpi_idle too, not just intel_idle.
> 
> > Screenshot with some backtrace: 
> > http://kodu.ut.ee/~mroos/intel-idle-NMI.png
> 
> http://kodu.ut.ee/~mroos/acpi-idle-NMI.png

There were almost no changes in the ACPI idle driver between 4.6 and
4.7-rc, so the reason is somewhere else.

Have you tried any earlier 4.7-rc?

The only change between 4.6 and now in the ACPI processor stuff at all is
this one

86314751c794 ACPI / processor: Avoid reserving IO regions too early

so does it help if that's reverted?

Otherwise, any chances to bisect?

Thanks,
Rafael

[toc] | [prev] | [next] | [standalone]


#1442070

FromMeelis Roos <mroos@linux.ee>
Date2016-07-13 09:20 +0200
Message-ID<rUmCC-8ei-25@gated-at.bofh.it>
In reply to#1441311
> > > > On HP Proliant DL360 G6, Debian unstable 4.6 kernel runs fine but 
> > > > selfcompiled 4.7-rc6 and 4.7-rc7 sometimes crash with NMI from 
> > > > intel_idle. Sometimes it boots fine. With intel_idle disabled, it has 
> > > > booted successful so far in 2 tries, one with rc6 and one with rc7.
> > > 
> > > More testing shows it can NMI on acpi_idle too, not just intel_idle.
> > > 
> > > > Screenshot with some backtrace: 
> > > > http://kodu.ut.ee/~mroos/intel-idle-NMI.png
> > > 
> > > http://kodu.ut.ee/~mroos/acpi-idle-NMI.png
> > 
> > There were almost no changes in the ACPI idle driver between 4.6 and
> > 4.7-rc, so the reason is somewhere else.
> > 
> > Have you tried any earlier 4.7-rc?
> 
> I tried selfcompiled 4.6.0 now with the same conf that 4.7-rc's have and 
> after multiple tries I got the same NMI out of 4.6 too, from kernfs 
> lookup (that seems to just a random victim).
> 
> So this is not a 4.7 regression and probably not idle-releated.

Confirmed this is about IOMMU+DMAR, I'm continuing in another reply 
thread with other CC-s to not bother linux-pm people with it.

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [next] | [standalone]


#1442158 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromJoerg Roedel <joro@8bytes.org>
Date2016-07-13 10:20 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUnyG-oj-15@gated-at.bofh.it>
In reply to#1441311
On Wed, Jul 13, 2016 at 10:17:59AM +0300, Meelis Roos wrote:
> Bisecting kernel configs shows that it's DMAR+IOMMU. When it is 
> activated, there is high probability of NMI-s in random places.

Hmm, strange. But nothing could really surprise when you have an HP
BIOS.

Can you probably use the faulty config and bisect this down to a
specific commit? In v4.7-rc1 some changes to the iova-allocation code
got merged, but I have no idea how those could cause NMIs.


Thanks,

	Joerg

[toc] | [prev] | [next] | [standalone]


#1442165 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromMeelis Roos <mroos@linux.ee>
Date2016-07-13 10:30 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUnIl-rS-9@gated-at.bofh.it>
In reply to#1442158
> > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is 
> > activated, there is high probability of NMI-s in random places.
> 
> Hmm, strange. But nothing could really surprise when you have an HP
> BIOS.

BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but 
the release notes only describe updated CPU microcode for security 
reasons.
 
> Can you probably use the faulty config and bisect this down to a
> specific commit? In v4.7-rc1 some changes to the iova-allocation code
> got merged, but I have no idea how those could cause NMIs.

Will try but I do not know a working base yet - this was broken in both 
4.6 and 4.7-rc.

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [next] | [standalone]


#1442241 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromJoerg Roedel <joro@8bytes.org>
Date2016-07-13 11:10 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUol3-VT-1@gated-at.bofh.it>
In reply to#1442165
On Wed, Jul 13, 2016 at 11:31:02AM +0300, Meelis Roos wrote:
> > > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is 
> > > activated, there is high probability of NMI-s in random places.
> > 
> > Hmm, strange. But nothing could really surprise when you have an HP
> > BIOS.
> 
> BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but 
> the release notes only describe updated CPU microcode for security 
> reasons.

It is probably something HP is selling as a "feature" and not a BIOS
bug.

> > Can you probably use the faulty config and bisect this down to a
> > specific commit? In v4.7-rc1 some changes to the iova-allocation code
> > got merged, but I have no idea how those could cause NMIs.
> 
> Will try but I do not know a working base yet - this was broken in both 
> 4.6 and 4.7-rc.

Oh, in that case it is not related to the recent iova changes. Does the
box have any hardware error log which you can access and send to us
(right after some NMIs happened)?


Thanks,

	Joerg

[toc] | [prev] | [next] | [standalone]


#1442249 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromMeelis Roos <mroos@linux.ee>
Date2016-07-13 11:20 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUouK-ZN-7@gated-at.bofh.it>
In reply to#1442241
> > > > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is 
> > > > activated, there is high probability of NMI-s in random places.
> > > 
> > > Hmm, strange. But nothing could really surprise when you have an HP
> > > BIOS.
> > 
> > BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but 
> > the release notes only describe updated CPU microcode for security 
> > reasons.
> 
> It is probably something HP is selling as a "feature" and not a BIOS
> bug.

ROM setup settings that might be of interest:

Advanced memory protection: advanced ecc support
No-Execute memory protection: enabled
Intel virtualization technology: enabled
Intel hyperthreading options: enabled
Processor core disable: all cored enabled
Intel turbo boost technology: enabled
Intel VT-d: enabled
HP power profile: custom
HP power regulator: hp dynamic power savings mode (not OS control)
Intel qpi link power management: enabled
Minimum processor idle power core state: C6
Minimum processor idle power package state: C6
Dynamic power saving mode response: fast
Collaborative power control: enabled
MPS table: full table apic
NMI debug button: enabled
PCI bus padding options: enabled
HW prefetcher: enabled
Adjacent sector prefetch: enabled
Node interleaving: disabled


> > > Can you probably use the faulty config and bisect this down to a
> > > specific commit? In v4.7-rc1 some changes to the iova-allocation code
> > > got merged, but I have no idea how those could cause NMIs.
> > 
> > Will try but I do not know a working base yet - this was broken in both 
> > 4.6 and 4.7-rc.
> 
> Oh, in that case it is not related to the recent iova changes. Does the
> box have any hardware error log which you can access and send to us
> (right after some NMIs happened)?

Nothing in ILO log or integrated management log (IML).

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [next] | [standalone]


#1442268 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromJoerg Roedel <joro@8bytes.org>
Date2016-07-13 11:50 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUoXL-1bB-5@gated-at.bofh.it>
In reply to#1442249
On Wed, Jul 13, 2016 at 12:16:46PM +0300, Meelis Roos wrote:
> ROM setup settings that might be of interest:
> 
> Advanced memory protection: advanced ecc support
> No-Execute memory protection: enabled
> Intel virtualization technology: enabled
> Intel hyperthreading options: enabled
> Processor core disable: all cored enabled
> Intel turbo boost technology: enabled
> Intel VT-d: enabled
> HP power profile: custom
> HP power regulator: hp dynamic power savings mode (not OS control)
> Intel qpi link power management: enabled
> Minimum processor idle power core state: C6
> Minimum processor idle power package state: C6
> Dynamic power saving mode response: fast
> Collaborative power control: enabled
> MPS table: full table apic
> NMI debug button: enabled
> PCI bus padding options: enabled
> HW prefetcher: enabled
> Adjacent sector prefetch: enabled
> Node interleaving: disabled

AFAICT these settings look normal.

> Nothing in ILO log or integrated management log (IML).

Then I think we need a bisect to make progress here.


	Joerg

[toc] | [prev] | [next] | [standalone]


#1442261 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromMeelis Roos <mroos@linux.ee>
Date2016-07-13 11:40 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUoO5-17A-13@gated-at.bofh.it>
In reply to#1442241
> > > Can you probably use the faulty config and bisect this down to a
> > > specific commit? In v4.7-rc1 some changes to the iova-allocation code
> > > got merged, but I have no idea how those could cause NMIs.
> > 
> > Will try but I do not know a working base yet - this was broken in both 
> > 4.6 and 4.7-rc.
> 
> Oh, in that case it is not related to the recent iova changes. Does the
> box have any hardware error log which you can access and send to us
> (right after some NMIs happened)?

Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with 
BIOS settings (disabling NUMA). It is the first time I see at least some 
info in NMI decode.

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [next] | [standalone]


#1442270 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromJoerg Roedel <joro@8bytes.org>
Date2016-07-13 11:50 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUoXM-1bB-15@gated-at.bofh.it>
In reply to#1442261
On Wed, Jul 13, 2016 at 12:40:39PM +0300, Meelis Roos wrote:
> Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with 
> BIOS settings (disabling NUMA). It is the first time I see at least some 
> info in NMI decode.

This looks interesting. Can you please post output of 'lspci -vvv' and
'lspci -t'?


Thanks,

	Joerg

[toc] | [prev] | [next] | [standalone]


#1442298 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromJoerg Roedel <joro@8bytes.org>
Date2016-07-13 12:20 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUpqO-1JD-17@gated-at.bofh.it>
In reply to#1442270
On Wed, Jul 13, 2016 at 12:58:24PM +0300, Meelis Roos wrote:
> > > Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with 
> > > BIOS settings (disabling NUMA). It is the first time I see at least some 
> > > info in NMI decode.
> > 
> > This looks interesting. Can you please post output of 'lspci -vvv' and
> > 'lspci -t'?
> 
> Here.

Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit
PCI-devices behind it. One of these devices tries to read address
0xb000, which is blocked by the IOMMU and causes the fault seen in the
screen-shot. The fault also causes a PCI-error which is then reported
through the NMI, causing your kernel panic.

So the 32bit PCI devices behind the bridge are:

01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller])
01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)
01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out  Processor (rev 03)
01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (prog-if 00 [UHCI])
01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface (prog-if 01)

Can you try to disable this 'Lights Out' processor? Maybe it is causing
the issues. On the other side, the radeon driver for the ATI card is
also know for causing faults from time to time. Can you capture the
kernel messages right before a crash too?


Thanks,

	Joerg

[toc] | [prev] | [next] | [standalone]


#1442521 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromAlex Williamson <alex.williamson@redhat.com>
Date2016-07-13 17:00 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUtNL-4qs-11@gated-at.bofh.it>
In reply to#1442298
On Wed, 13 Jul 2016 12:18:59 +0200
Joerg Roedel <joro@8bytes.org> wrote:

> On Wed, Jul 13, 2016 at 12:58:24PM +0300, Meelis Roos wrote:
> > > > Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with 
> > > > BIOS settings (disabling NUMA). It is the first time I see at least some 
> > > > info in NMI decode.  
> > > 
> > > This looks interesting. Can you please post output of 'lspci -vvv' and
> > > 'lspci -t'?  
> > 
> > Here.  
> 
> Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit
> PCI-devices behind it. One of these devices tries to read address
> 0xb000, which is blocked by the IOMMU and causes the fault seen in the
> screen-shot. The fault also causes a PCI-error which is then reported
> through the NMI, causing your kernel panic.
> 
> So the 32bit PCI devices behind the bridge are:
> 
> 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller])
> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)
> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out  Processor (rev 03)
> 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (prog-if 00 [UHCI])
> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface (prog-if 01)
> 
> Can you try to disable this 'Lights Out' processor? Maybe it is causing
> the issues. On the other side, the radeon driver for the ATI card is
> also know for causing faults from time to time. Can you capture the
> kernel messages right before a crash too?

IIRC, blacklisting the hpwdt module can defuse those NMIs and might
help us see more of the actual DMAR faults.  Blacklist in modprobe.d
and rebuild initrd. Thanks,

Alex

PS - never assume BIOS release notes are actually complete

[toc] | [prev] | [next] | [standalone]


#1442898 — Re: IOMMU+DMAR causing NMIs-s

FromMeelis Roos <mroos@linux.ee>
Date2016-07-14 00:00 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s
Message-ID<rUAmj-nH-21@gated-at.bofh.it>
In reply to#1442521
> >> Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit
> >> PCI-devices behind it. One of these devices tries to read address
> >> 0xb000, which is blocked by the IOMMU and causes the fault seen in the
> >> screen-shot. The fault also causes a PCI-error which is then reported
> >> through the NMI, causing your kernel panic.
> >>
> >> So the 32bit PCI devices behind the bridge are:
> >>
> >> 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller])
> >> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)
> >> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out  Processor (rev 03)
> >> 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (prog-if 00 [UHCI])
> >> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface (prog-if 01)
> >>
> >> Can you try to disable this 'Lights Out' processor? Maybe it is causing
> >> the issues. On the other side, the radeon driver for the ATI card is
> >> also know for causing faults from time to time. Can you capture the
> >> kernel messages right before a crash too?
> > 
> > IIRC, blacklisting the hpwdt module can defuse those NMIs and might
> > help us see more of the actual DMAR faults.  Blacklist in modprobe.d
> > and rebuild initrd. Thanks,
> > 
> > Alex
> > 
> > PS - never assume BIOS release notes are actually complete
> 
> I agree. I'd do the BIOS update and also make sure the iLO FW is current.

OK, updates to the latest BIOS, CPU microcode revison is now 0x1b 
instead of 0x19. ILO2 fw is already the latest.

The NMI-s still happen.

Disabled hpwdt. Now most boots are fine, only one hung with radeon 
loading:

http://kodu.ut.ee/~mroos/boot-radeon-1.png 

Vusal patterns also remind me that all the previous hangs were before 
radeon mode change - once radeon changes the mode, it already works.

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [next] | [standalone]


#1442932 — Re: IOMMU+DMAR causing NMIs-s

FromAlex Williamson <alex.williamson@redhat.com>
Date2016-07-14 01:10 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s
Message-ID<rUBrX-1jq-3@gated-at.bofh.it>
In reply to#1442898
On Thu, 14 Jul 2016 00:52:02 +0300 (EEST)
Meelis Roos <mroos@linux.ee> wrote:

> > >> Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit
> > >> PCI-devices behind it. One of these devices tries to read address
> > >> 0xb000, which is blocked by the IOMMU and causes the fault seen in the
> > >> screen-shot. The fault also causes a PCI-error which is then reported
> > >> through the NMI, causing your kernel panic.
> > >>
> > >> So the 32bit PCI devices behind the bridge are:
> > >>
> > >> 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller])
> > >> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)
> > >> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out  Processor (rev 03)
> > >> 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (prog-if 00 [UHCI])
> > >> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface (prog-if 01)
> > >>
> > >> Can you try to disable this 'Lights Out' processor? Maybe it is causing
> > >> the issues. On the other side, the radeon driver for the ATI card is
> > >> also know for causing faults from time to time. Can you capture the
> > >> kernel messages right before a crash too?  
> > > 
> > > IIRC, blacklisting the hpwdt module can defuse those NMIs and might
> > > help us see more of the actual DMAR faults.  Blacklist in modprobe.d
> > > and rebuild initrd. Thanks,
> > > 
> > > Alex
> > > 
> > > PS - never assume BIOS release notes are actually complete  
> > 
> > I agree. I'd do the BIOS update and also make sure the iLO FW is current.  
> 
> OK, updates to the latest BIOS, CPU microcode revison is now 0x1b 
> instead of 0x19. ILO2 fw is already the latest.
> 
> The NMI-s still happen.
> 
> Disabled hpwdt. Now most boots are fine, only one hung with radeon 
> loading:
> 
> http://kodu.ut.ee/~mroos/boot-radeon-1.png 
> 
> Vusal patterns also remind me that all the previous hangs were before 
> radeon mode change - once radeon changes the mode, it already works.

There might be multiple things going on here, I tend to associate the
NMIs with the DMAR faults on HP systems and I expect that disabling
hpwdt will remove that DMAR fault to NMI escalation, but it won't fix
the fact that DMAR faults are occurring.  I'm not sure how any of that
is related to radeon though other than the fact that the radeon device
is behind the bridge that generated the DMAR fault and since we can't
tell which device behind the bridge generated that transaction, it's
possible that it was the radeon card (though we're all more likely to
throw stones at the iLO first).

With hpwdt disabled, can you reboot a few times and look for DMAR
faults in the dmesg to see if they're all consistent, ie. device 1e.0
doing a read from 0xb000?  Is there any correlation to radeon hanging
and one of those DMAR faults appearing in the log (you can make use of
the iLO for a serial console so you can record more of the boot than
what can be captured on the VGA console).  Also, this should be an
X58-based system, which reminds me of Joerg's recent commit
a4c34ff1c029 that's intended to fix a hang on such systems.  That was
included starting in v4.7-rc4, so if this is a new issue, testing
before and after that commit might be relevant.  Thanks,

Alex

[toc] | [prev] | [next] | [standalone]


#1443238 — Re: IOMMU+DMAR causing NMIs-s

FromJoerg Roedel <joro@8bytes.org>
Date2016-07-14 10:40 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s
Message-ID<rUKlA-7jX-41@gated-at.bofh.it>
In reply to#1442932
On Wed, Jul 13, 2016 at 05:04:08PM -0600, Alex Williamson wrote:
> With hpwdt disabled, can you reboot a few times and look for DMAR
> faults in the dmesg to see if they're all consistent, ie. device 1e.0
> doing a read from 0xb000?  Is there any correlation to radeon hanging
> and one of those DMAR faults appearing in the log (you can make use of
> the iLO for a serial console so you can record more of the boot than
> what can be captured on the VGA console).  Also, this should be an
> X58-based system, which reminds me of Joerg's recent commit
> a4c34ff1c029 that's intended to fix a hang on such systems.  That was
> included starting in v4.7-rc4, so if this is a new issue, testing
> before and after that commit might be relevant.  Thanks,

That commit fixed a hang while the IOMMU is being initialized. This hang
is different, as the IOMMU already runs. So I doubt that it fixes this
problem, but it is worth a test, of course.


	Joerg

[toc] | [prev] | [next] | [standalone]


#1442477 — Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)

FromHenrique de Moraes Holschuh <hmh@hmh.eng.br>
Date2016-07-13 16:00 +0200
SubjectRe: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6)
Message-ID<rUsRH-3OH-3@gated-at.bofh.it>
In reply to#1442241
On Wed, 13 Jul 2016, Joerg Roedel wrote:
> On Wed, Jul 13, 2016 at 11:31:02AM +0300, Meelis Roos wrote:
> > > > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is 
> > > > activated, there is high probability of NMI-s in random places.
> > > 
> > > Hmm, strange. But nothing could really surprise when you have an HP
> > > BIOS.
> > 
> > BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but 
> > the release notes only describe updated CPU microcode for security 
> > reasons.
> 
> It is probably something HP is selling as a "feature" and not a BIOS
> bug.

Well, based on the date, it should be the microcode-level fix for this:
https://www.blackhat.com/docs/us-15/materials/us-15-Domas-The-Memory-Sinkhole-Unleashing-An-x86-Design-Flaw-Allowing-Universal-Privilege-Escalation-wp.pdf

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

[toc] | [prev] | [standalone]


Back to top | Article view | linux.kernel


csiph-web