Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1441308 > unrolled thread
| Started by | Meelis Roos <mroos@linux.ee> |
|---|---|
| First post | 2016-07-12 14:30 +0200 |
| Last post | 2016-07-13 16:00 +0200 |
| Articles | 16 — 5 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: 4.7-rc6: NMI in intel_idle on HP Proliant G6 Meelis Roos <mroos@linux.ee> - 2016-07-12 14:30 +0200
Re: 4.7-rc6: NMI in intel_idle on HP Proliant G6 "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-07-12 14:40 +0200
Re: 4.7-rc6: NMI in intel_idle on HP Proliant G6 Meelis Roos <mroos@linux.ee> - 2016-07-13 09:20 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 10:20 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Meelis Roos <mroos@linux.ee> - 2016-07-13 10:30 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 11:10 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Meelis Roos <mroos@linux.ee> - 2016-07-13 11:20 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 11:50 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Meelis Roos <mroos@linux.ee> - 2016-07-13 11:40 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 11:50 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Joerg Roedel <joro@8bytes.org> - 2016-07-13 12:20 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Alex Williamson <alex.williamson@redhat.com> - 2016-07-13 17:00 +0200
Re: IOMMU+DMAR causing NMIs-s Meelis Roos <mroos@linux.ee> - 2016-07-14 00:00 +0200
Re: IOMMU+DMAR causing NMIs-s Alex Williamson <alex.williamson@redhat.com> - 2016-07-14 01:10 +0200
Re: IOMMU+DMAR causing NMIs-s Joerg Roedel <joro@8bytes.org> - 2016-07-14 10:40 +0200
Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) Henrique de Moraes Holschuh <hmh@hmh.eng.br> - 2016-07-13 16:00 +0200
| From | Meelis Roos <mroos@linux.ee> |
|---|---|
| Date | 2016-07-12 14:30 +0200 |
| Subject | Re: 4.7-rc6: NMI in intel_idle on HP Proliant G6 |
| Message-ID | <rU4Z8-4Ol-15@gated-at.bofh.it> |
> On HP Proliant DL360 G6, Debian unstable 4.6 kernel runs fine but > selfcompiled 4.7-rc6 and 4.7-rc7 sometimes crash with NMI from > intel_idle. Sometimes it boots fine. With intel_idle disabled, it has > booted successful so far in 2 tries, one with rc6 and one with rc7. More testing shows it can NMI on acpi_idle too, not just intel_idle. > Screenshot with some backtrace: > http://kodu.ut.ee/~mroos/intel-idle-NMI.png http://kodu.ut.ee/~mroos/acpi-idle-NMI.png -- Meelis Roos (mroos@linux.ee)
[toc] | [next] | [standalone]
| From | "Rafael J. Wysocki" <rjw@rjwysocki.net> |
|---|---|
| Date | 2016-07-12 14:40 +0200 |
| Message-ID | <rU58K-4Sa-23@gated-at.bofh.it> |
| In reply to | #1441308 |
On Tuesday, July 12, 2016 03:27:19 PM Meelis Roos wrote: > > On HP Proliant DL360 G6, Debian unstable 4.6 kernel runs fine but > > selfcompiled 4.7-rc6 and 4.7-rc7 sometimes crash with NMI from > > intel_idle. Sometimes it boots fine. With intel_idle disabled, it has > > booted successful so far in 2 tries, one with rc6 and one with rc7. > > More testing shows it can NMI on acpi_idle too, not just intel_idle. > > > Screenshot with some backtrace: > > http://kodu.ut.ee/~mroos/intel-idle-NMI.png > > http://kodu.ut.ee/~mroos/acpi-idle-NMI.png There were almost no changes in the ACPI idle driver between 4.6 and 4.7-rc, so the reason is somewhere else. Have you tried any earlier 4.7-rc? The only change between 4.6 and now in the ACPI processor stuff at all is this one 86314751c794 ACPI / processor: Avoid reserving IO regions too early so does it help if that's reverted? Otherwise, any chances to bisect? Thanks, Rafael
[toc] | [prev] | [next] | [standalone]
| From | Meelis Roos <mroos@linux.ee> |
|---|---|
| Date | 2016-07-13 09:20 +0200 |
| Message-ID | <rUmCC-8ei-25@gated-at.bofh.it> |
| In reply to | #1441311 |
> > > > On HP Proliant DL360 G6, Debian unstable 4.6 kernel runs fine but > > > > selfcompiled 4.7-rc6 and 4.7-rc7 sometimes crash with NMI from > > > > intel_idle. Sometimes it boots fine. With intel_idle disabled, it has > > > > booted successful so far in 2 tries, one with rc6 and one with rc7. > > > > > > More testing shows it can NMI on acpi_idle too, not just intel_idle. > > > > > > > Screenshot with some backtrace: > > > > http://kodu.ut.ee/~mroos/intel-idle-NMI.png > > > > > > http://kodu.ut.ee/~mroos/acpi-idle-NMI.png > > > > There were almost no changes in the ACPI idle driver between 4.6 and > > 4.7-rc, so the reason is somewhere else. > > > > Have you tried any earlier 4.7-rc? > > I tried selfcompiled 4.6.0 now with the same conf that 4.7-rc's have and > after multiple tries I got the same NMI out of 4.6 too, from kernfs > lookup (that seems to just a random victim). > > So this is not a 4.7 regression and probably not idle-releated. Confirmed this is about IOMMU+DMAR, I'm continuing in another reply thread with other CC-s to not bother linux-pm people with it. -- Meelis Roos (mroos@linux.ee)
[toc] | [prev] | [next] | [standalone]
| From | Joerg Roedel <joro@8bytes.org> |
|---|---|
| Date | 2016-07-13 10:20 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUnyG-oj-15@gated-at.bofh.it> |
| In reply to | #1441311 |
On Wed, Jul 13, 2016 at 10:17:59AM +0300, Meelis Roos wrote: > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > activated, there is high probability of NMI-s in random places. Hmm, strange. But nothing could really surprise when you have an HP BIOS. Can you probably use the faulty config and bisect this down to a specific commit? In v4.7-rc1 some changes to the iova-allocation code got merged, but I have no idea how those could cause NMIs. Thanks, Joerg
[toc] | [prev] | [next] | [standalone]
| From | Meelis Roos <mroos@linux.ee> |
|---|---|
| Date | 2016-07-13 10:30 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUnIl-rS-9@gated-at.bofh.it> |
| In reply to | #1442158 |
> > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > > activated, there is high probability of NMI-s in random places. > > Hmm, strange. But nothing could really surprise when you have an HP > BIOS. BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but the release notes only describe updated CPU microcode for security reasons. > Can you probably use the faulty config and bisect this down to a > specific commit? In v4.7-rc1 some changes to the iova-allocation code > got merged, but I have no idea how those could cause NMIs. Will try but I do not know a working base yet - this was broken in both 4.6 and 4.7-rc. -- Meelis Roos (mroos@linux.ee)
[toc] | [prev] | [next] | [standalone]
| From | Joerg Roedel <joro@8bytes.org> |
|---|---|
| Date | 2016-07-13 11:10 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUol3-VT-1@gated-at.bofh.it> |
| In reply to | #1442165 |
On Wed, Jul 13, 2016 at 11:31:02AM +0300, Meelis Roos wrote: > > > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > > > activated, there is high probability of NMI-s in random places. > > > > Hmm, strange. But nothing could really surprise when you have an HP > > BIOS. > > BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but > the release notes only describe updated CPU microcode for security > reasons. It is probably something HP is selling as a "feature" and not a BIOS bug. > > Can you probably use the faulty config and bisect this down to a > > specific commit? In v4.7-rc1 some changes to the iova-allocation code > > got merged, but I have no idea how those could cause NMIs. > > Will try but I do not know a working base yet - this was broken in both > 4.6 and 4.7-rc. Oh, in that case it is not related to the recent iova changes. Does the box have any hardware error log which you can access and send to us (right after some NMIs happened)? Thanks, Joerg
[toc] | [prev] | [next] | [standalone]
| From | Meelis Roos <mroos@linux.ee> |
|---|---|
| Date | 2016-07-13 11:20 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUouK-ZN-7@gated-at.bofh.it> |
| In reply to | #1442241 |
> > > > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > > > > activated, there is high probability of NMI-s in random places. > > > > > > Hmm, strange. But nothing could really surprise when you have an HP > > > BIOS. > > > > BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but > > the release notes only describe updated CPU microcode for security > > reasons. > > It is probably something HP is selling as a "feature" and not a BIOS > bug. ROM setup settings that might be of interest: Advanced memory protection: advanced ecc support No-Execute memory protection: enabled Intel virtualization technology: enabled Intel hyperthreading options: enabled Processor core disable: all cored enabled Intel turbo boost technology: enabled Intel VT-d: enabled HP power profile: custom HP power regulator: hp dynamic power savings mode (not OS control) Intel qpi link power management: enabled Minimum processor idle power core state: C6 Minimum processor idle power package state: C6 Dynamic power saving mode response: fast Collaborative power control: enabled MPS table: full table apic NMI debug button: enabled PCI bus padding options: enabled HW prefetcher: enabled Adjacent sector prefetch: enabled Node interleaving: disabled > > > Can you probably use the faulty config and bisect this down to a > > > specific commit? In v4.7-rc1 some changes to the iova-allocation code > > > got merged, but I have no idea how those could cause NMIs. > > > > Will try but I do not know a working base yet - this was broken in both > > 4.6 and 4.7-rc. > > Oh, in that case it is not related to the recent iova changes. Does the > box have any hardware error log which you can access and send to us > (right after some NMIs happened)? Nothing in ILO log or integrated management log (IML). -- Meelis Roos (mroos@linux.ee)
[toc] | [prev] | [next] | [standalone]
| From | Joerg Roedel <joro@8bytes.org> |
|---|---|
| Date | 2016-07-13 11:50 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUoXL-1bB-5@gated-at.bofh.it> |
| In reply to | #1442249 |
On Wed, Jul 13, 2016 at 12:16:46PM +0300, Meelis Roos wrote: > ROM setup settings that might be of interest: > > Advanced memory protection: advanced ecc support > No-Execute memory protection: enabled > Intel virtualization technology: enabled > Intel hyperthreading options: enabled > Processor core disable: all cored enabled > Intel turbo boost technology: enabled > Intel VT-d: enabled > HP power profile: custom > HP power regulator: hp dynamic power savings mode (not OS control) > Intel qpi link power management: enabled > Minimum processor idle power core state: C6 > Minimum processor idle power package state: C6 > Dynamic power saving mode response: fast > Collaborative power control: enabled > MPS table: full table apic > NMI debug button: enabled > PCI bus padding options: enabled > HW prefetcher: enabled > Adjacent sector prefetch: enabled > Node interleaving: disabled AFAICT these settings look normal. > Nothing in ILO log or integrated management log (IML). Then I think we need a bisect to make progress here. Joerg
[toc] | [prev] | [next] | [standalone]
| From | Meelis Roos <mroos@linux.ee> |
|---|---|
| Date | 2016-07-13 11:40 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUoO5-17A-13@gated-at.bofh.it> |
| In reply to | #1442241 |
> > > Can you probably use the faulty config and bisect this down to a > > > specific commit? In v4.7-rc1 some changes to the iova-allocation code > > > got merged, but I have no idea how those could cause NMIs. > > > > Will try but I do not know a working base yet - this was broken in both > > 4.6 and 4.7-rc. > > Oh, in that case it is not related to the recent iova changes. Does the > box have any hardware error log which you can access and send to us > (right after some NMIs happened)? Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with BIOS settings (disabling NUMA). It is the first time I see at least some info in NMI decode. -- Meelis Roos (mroos@linux.ee)
[toc] | [prev] | [next] | [standalone]
| From | Joerg Roedel <joro@8bytes.org> |
|---|---|
| Date | 2016-07-13 11:50 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUoXM-1bB-15@gated-at.bofh.it> |
| In reply to | #1442261 |
On Wed, Jul 13, 2016 at 12:40:39PM +0300, Meelis Roos wrote: > Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with > BIOS settings (disabling NUMA). It is the first time I see at least some > info in NMI decode. This looks interesting. Can you please post output of 'lspci -vvv' and 'lspci -t'? Thanks, Joerg
[toc] | [prev] | [next] | [standalone]
| From | Joerg Roedel <joro@8bytes.org> |
|---|---|
| Date | 2016-07-13 12:20 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUpqO-1JD-17@gated-at.bofh.it> |
| In reply to | #1442270 |
On Wed, Jul 13, 2016 at 12:58:24PM +0300, Meelis Roos wrote: > > > Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with > > > BIOS settings (disabling NUMA). It is the first time I see at least some > > > info in NMI decode. > > > > This looks interesting. Can you please post output of 'lspci -vvv' and > > 'lspci -t'? > > Here. Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit PCI-devices behind it. One of these devices tries to read address 0xb000, which is blocked by the IOMMU and causes the fault seen in the screen-shot. The fault also causes a PCI-error which is then reported through the NMI, causing your kernel panic. So the 32bit PCI devices behind the bridge are: 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller]) 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03) 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03) 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (prog-if 00 [UHCI]) 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface (prog-if 01) Can you try to disable this 'Lights Out' processor? Maybe it is causing the issues. On the other side, the radeon driver for the ATI card is also know for causing faults from time to time. Can you capture the kernel messages right before a crash too? Thanks, Joerg
[toc] | [prev] | [next] | [standalone]
| From | Alex Williamson <alex.williamson@redhat.com> |
|---|---|
| Date | 2016-07-13 17:00 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUtNL-4qs-11@gated-at.bofh.it> |
| In reply to | #1442298 |
On Wed, 13 Jul 2016 12:18:59 +0200 Joerg Roedel <joro@8bytes.org> wrote: > On Wed, Jul 13, 2016 at 12:58:24PM +0300, Meelis Roos wrote: > > > > Just got http://kodu.ut.ee/~mroos/4.6-dmar-fault2.png when playing with > > > > BIOS settings (disabling NUMA). It is the first time I see at least some > > > > info in NMI decode. > > > > > > This looks interesting. Can you please post output of 'lspci -vvv' and > > > 'lspci -t'? > > > > Here. > > Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit > PCI-devices behind it. One of these devices tries to read address > 0xb000, which is blocked by the IOMMU and causes the fault seen in the > screen-shot. The fault also causes a PCI-error which is then reported > through the NMI, causing your kernel panic. > > So the 32bit PCI devices behind the bridge are: > > 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller]) > 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03) > 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03) > 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (prog-if 00 [UHCI]) > 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface (prog-if 01) > > Can you try to disable this 'Lights Out' processor? Maybe it is causing > the issues. On the other side, the radeon driver for the ATI card is > also know for causing faults from time to time. Can you capture the > kernel messages right before a crash too? IIRC, blacklisting the hpwdt module can defuse those NMIs and might help us see more of the actual DMAR faults. Blacklist in modprobe.d and rebuild initrd. Thanks, Alex PS - never assume BIOS release notes are actually complete
[toc] | [prev] | [next] | [standalone]
| From | Meelis Roos <mroos@linux.ee> |
|---|---|
| Date | 2016-07-14 00:00 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s |
| Message-ID | <rUAmj-nH-21@gated-at.bofh.it> |
| In reply to | #1442521 |
> >> Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit > >> PCI-devices behind it. One of these devices tries to read address > >> 0xb000, which is blocked by the IOMMU and causes the fault seen in the > >> screen-shot. The fault also causes a PCI-error which is then reported > >> through the NMI, causing your kernel panic. > >> > >> So the 32bit PCI devices behind the bridge are: > >> > >> 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller]) > >> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03) > >> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03) > >> 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (prog-if 00 [UHCI]) > >> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface (prog-if 01) > >> > >> Can you try to disable this 'Lights Out' processor? Maybe it is causing > >> the issues. On the other side, the radeon driver for the ATI card is > >> also know for causing faults from time to time. Can you capture the > >> kernel messages right before a crash too? > > > > IIRC, blacklisting the hpwdt module can defuse those NMIs and might > > help us see more of the actual DMAR faults. Blacklist in modprobe.d > > and rebuild initrd. Thanks, > > > > Alex > > > > PS - never assume BIOS release notes are actually complete > > I agree. I'd do the BIOS update and also make sure the iLO FW is current. OK, updates to the latest BIOS, CPU microcode revison is now 0x1b instead of 0x19. ILO2 fw is already the latest. The NMI-s still happen. Disabled hpwdt. Now most boots are fine, only one hung with radeon loading: http://kodu.ut.ee/~mroos/boot-radeon-1.png Vusal patterns also remind me that all the previous hangs were before radeon mode change - once radeon changes the mode, it already works. -- Meelis Roos (mroos@linux.ee)
[toc] | [prev] | [next] | [standalone]
| From | Alex Williamson <alex.williamson@redhat.com> |
|---|---|
| Date | 2016-07-14 01:10 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s |
| Message-ID | <rUBrX-1jq-3@gated-at.bofh.it> |
| In reply to | #1442898 |
On Thu, 14 Jul 2016 00:52:02 +0300 (EEST) Meelis Roos <mroos@linux.ee> wrote: > > >> Thanks. So device 00:1e.0 is a PCI-bridge which has some 32-bit > > >> PCI-devices behind it. One of these devices tries to read address > > >> 0xb000, which is blocked by the IOMMU and causes the fault seen in the > > >> screen-shot. The fault also causes a PCI-error which is then reported > > >> through the NMI, causing your kernel panic. > > >> > > >> So the 32bit PCI devices behind the bridge are: > > >> > > >> 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) (prog-if 00 [VGA controller]) > > >> 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03) > > >> 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03) > > >> 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (prog-if 00 [UHCI]) > > >> 01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface (prog-if 01) > > >> > > >> Can you try to disable this 'Lights Out' processor? Maybe it is causing > > >> the issues. On the other side, the radeon driver for the ATI card is > > >> also know for causing faults from time to time. Can you capture the > > >> kernel messages right before a crash too? > > > > > > IIRC, blacklisting the hpwdt module can defuse those NMIs and might > > > help us see more of the actual DMAR faults. Blacklist in modprobe.d > > > and rebuild initrd. Thanks, > > > > > > Alex > > > > > > PS - never assume BIOS release notes are actually complete > > > > I agree. I'd do the BIOS update and also make sure the iLO FW is current. > > OK, updates to the latest BIOS, CPU microcode revison is now 0x1b > instead of 0x19. ILO2 fw is already the latest. > > The NMI-s still happen. > > Disabled hpwdt. Now most boots are fine, only one hung with radeon > loading: > > http://kodu.ut.ee/~mroos/boot-radeon-1.png > > Vusal patterns also remind me that all the previous hangs were before > radeon mode change - once radeon changes the mode, it already works. There might be multiple things going on here, I tend to associate the NMIs with the DMAR faults on HP systems and I expect that disabling hpwdt will remove that DMAR fault to NMI escalation, but it won't fix the fact that DMAR faults are occurring. I'm not sure how any of that is related to radeon though other than the fact that the radeon device is behind the bridge that generated the DMAR fault and since we can't tell which device behind the bridge generated that transaction, it's possible that it was the radeon card (though we're all more likely to throw stones at the iLO first). With hpwdt disabled, can you reboot a few times and look for DMAR faults in the dmesg to see if they're all consistent, ie. device 1e.0 doing a read from 0xb000? Is there any correlation to radeon hanging and one of those DMAR faults appearing in the log (you can make use of the iLO for a serial console so you can record more of the boot than what can be captured on the VGA console). Also, this should be an X58-based system, which reminds me of Joerg's recent commit a4c34ff1c029 that's intended to fix a hang on such systems. That was included starting in v4.7-rc4, so if this is a new issue, testing before and after that commit might be relevant. Thanks, Alex
[toc] | [prev] | [next] | [standalone]
| From | Joerg Roedel <joro@8bytes.org> |
|---|---|
| Date | 2016-07-14 10:40 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s |
| Message-ID | <rUKlA-7jX-41@gated-at.bofh.it> |
| In reply to | #1442932 |
On Wed, Jul 13, 2016 at 05:04:08PM -0600, Alex Williamson wrote: > With hpwdt disabled, can you reboot a few times and look for DMAR > faults in the dmesg to see if they're all consistent, ie. device 1e.0 > doing a read from 0xb000? Is there any correlation to radeon hanging > and one of those DMAR faults appearing in the log (you can make use of > the iLO for a serial console so you can record more of the boot than > what can be captured on the VGA console). Also, this should be an > X58-based system, which reminds me of Joerg's recent commit > a4c34ff1c029 that's intended to fix a hang on such systems. That was > included starting in v4.7-rc4, so if this is a new issue, testing > before and after that commit might be relevant. Thanks, That commit fixed a hang while the IOMMU is being initialized. This hang is different, as the IOMMU already runs. So I doubt that it fixes this problem, but it is worth a test, of course. Joerg
[toc] | [prev] | [next] | [standalone]
| From | Henrique de Moraes Holschuh <hmh@hmh.eng.br> |
|---|---|
| Date | 2016-07-13 16:00 +0200 |
| Subject | Re: IOMMU+DMAR causing NMIs-s (was: 4.7-rc6: NMI in intel_idle on HP Proliant G6) |
| Message-ID | <rUsRH-3OH-3@gated-at.bofh.it> |
| In reply to | #1442241 |
On Wed, 13 Jul 2016, Joerg Roedel wrote: > On Wed, Jul 13, 2016 at 11:31:02AM +0300, Meelis Roos wrote: > > > > Bisecting kernel configs shows that it's DMAR+IOMMU. When it is > > > > activated, there is high probability of NMI-s in random places. > > > > > > Hmm, strange. But nothing could really surprise when you have an HP > > > BIOS. > > > > BIOS P64 01/22/2015. There seems to be a newer 2015.08.16 BIOS out but > > the release notes only describe updated CPU microcode for security > > reasons. > > It is probably something HP is selling as a "feature" and not a BIOS > bug. Well, based on the date, it should be the microcode-level fix for this: https://www.blackhat.com/docs/us-15/materials/us-15-Domas-The-Memory-Sinkhole-Unleashing-An-x86-Design-Flaw-Allowing-Universal-Privilege-Escalation-wp.pdf -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web