Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1204410 > unrolled thread

Re: X-Gene: Unhandled fault: synchronous external abort in pci_generic_config_read32

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: X-Gene: Unhandled fault: synchronous external abort in pci_generic_config_read32 Bjorn Helgaas <bhelgaas@google.com> - 2015-08-10 18:20 +0200
    Re: X-Gene: Unhandled fault: synchronous external abort in  pci_generic_config_read32 Catalin Marinas <catalin.marinas@arm.com> - 2015-08-10 19:40 +0200
    Re: X-Gene: Unhandled fault: synchronous external abort in pci_generic_config_read32 Bjorn Helgaas <bhelgaas@google.com> - 2015-08-10 19:50 +0200
      Re: X-Gene: Unhandled fault: synchronous external abort in pci_generic_config_read32 Duc Dang <dhdang@apm.com> - 2015-08-10 21:10 +0200
        Re: X-Gene: Unhandled fault: synchronous external abort in pci_generic_config_read32 Bjorn Helgaas <bhelgaas@google.com> - 2015-08-11 21:30 +0200

#1204410 — Re: X-Gene: Unhandled fault: synchronous external abort in pci_generic_config_read32

On Fri, Jul 31, 2015 at 12:00 PM, Duc Dang <dhdang@apm.com> wrote:
> On Wed, Jul 29, 2015 at 8:55 AM, Bjorn Helgaas <bhelgaas@google.com> wrote:
>> On Tue, Jul 28, 2015 at 08:22:55PM -0500, Bjorn Helgaas wrote:
>>> On Tue, Jul 28, 2015 at 02:50:39PM -0700, Duc Dang wrote:
>>
>>> > Do you have another PCIe card to try on the same reboot test on this board?
>>>
>>> I've seen this on at least two Mellanox cards.  I'm running similar tests
>>> on a different type of card now.
>>
>> FWIW, reboot tests on two machines with Mellanox cards failed, while the
>> same test on a machine with a different proprietary card succeeded.
>
> Thanks, Bjorn.
>
> I don't have the same Mellanox card as yours, but I will also run
> similar reboot test to see if I hit the same issue with my card.

Any more hints on this?  Nothing has changed on my end, so of course
I'm still seeing this, always on machines with Mellanox, and never on
other machines.  Could this be a hardware issue like a signal
integrity or margin issue?  I don't know where to go from here because
I'm not a hardware person, and I don't know anything to do in
software.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1204451 — Re: X-Gene: Unhandled fault: synchronous external abort in pci_generic_config_read32

On Mon, Aug 10, 2015 at 11:18:23AM -0500, Bjorn Helgaas wrote:
> On Fri, Jul 31, 2015 at 12:00 PM, Duc Dang <dhdang@apm.com> wrote:
> > On Wed, Jul 29, 2015 at 8:55 AM, Bjorn Helgaas <bhelgaas@google.com> wrote:
> >> On Tue, Jul 28, 2015 at 08:22:55PM -0500, Bjorn Helgaas wrote:
> >>> On Tue, Jul 28, 2015 at 02:50:39PM -0700, Duc Dang wrote:
> >>
> >>> > Do you have another PCIe card to try on the same reboot test on this board?
> >>>
> >>> I've seen this on at least two Mellanox cards.  I'm running similar tests
> >>> on a different type of card now.
> >>
> >> FWIW, reboot tests on two machines with Mellanox cards failed, while the
> >> same test on a machine with a different proprietary card succeeded.
> >
> > Thanks, Bjorn.
> >
> > I don't have the same Mellanox card as yours, but I will also run
> > similar reboot test to see if I hit the same issue with my card.
> 
> Any more hints on this?  Nothing has changed on my end, so of course
> I'm still seeing this, always on machines with Mellanox, and never on
> other machines.  Could this be a hardware issue like a signal
> integrity or margin issue?  I don't know where to go from here because
> I'm not a hardware person, and I don't know anything to do in
> software.

Silly hack below, not actually a solution (and it may not even work):

diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 94d98cd1aad8..e895e96b3d13 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -369,6 +369,14 @@ static int do_bad(unsigned long addr, unsigned int esr, struct pt_regs *regs)
 	return 1;
 }
 
+/*
+ * Retry the faulty access.
+ */
+static int do_good(unsigned long addr, unsigned int esr, struct pt_regs *regs)
+{
+	return 0;
+}
+
 static struct fault_info {
 	int	(*fn)(unsigned long addr, unsigned int esr, struct pt_regs *regs);
 	int	sig;
@@ -391,7 +399,7 @@ static struct fault_info {
 	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 1 permission fault"	},
 	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 2 permission fault"	},
 	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 3 permission fault"	},
-	{ do_bad,		SIGBUS,  0,		"synchronous external abort"	},
+	{ do_good,		SIGBUS,  0,		"synchronous external abort"	},
 	{ do_bad,		SIGBUS,  0,		"asynchronous external abort"	},
 	{ do_bad,		SIGBUS,  0,		"unknown 18"			},
 	{ do_bad,		SIGBUS,  0,		"unknown 19"			},

-- 
Catalin
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1204455

On Mon, Aug 10, 2015 at 12:16 PM, Duc Dang <dhdang@apm.com> wrote:
> On Monday, August 10, 2015, Bjorn Helgaas <bhelgaas@google.com> wrote:
>>
>> On Fri, Jul 31, 2015 at 12:00 PM, Duc Dang <dhdang@apm.com> wrote:
>> > On Wed, Jul 29, 2015 at 8:55 AM, Bjorn Helgaas <bhelgaas@google.com>
>> > wrote:
>> >> On Tue, Jul 28, 2015 at 08:22:55PM -0500, Bjorn Helgaas wrote:
>> >>> On Tue, Jul 28, 2015 at 02:50:39PM -0700, Duc Dang wrote:
>> >>
>> >>> > Do you have another PCIe card to try on the same reboot test on this
>> >>> > board?
>> >>>
>> >>> I've seen this on at least two Mellanox cards.  I'm running similar
>> >>> tests
>> >>> on a different type of card now.
>> >>
>> >> FWIW, reboot tests on two machines with Mellanox cards failed, while
>> >> the
>> >> same test on a machine with a different proprietary card succeeded.
>> >
>> > Thanks, Bjorn.
>> >
>> > I don't have the same Mellanox card as yours, but I will also run
>> > similar reboot test to see if I hit the same issue with my card.
>>
>> Any more hints on this?  Nothing has changed on my end, so of course
>> I'm still seeing this, always on machines with Mellanox, and never on
>> other machines.  Could this be a hardware issue like a signal
>> integrity or margin issue?  I don't know where to go from here because
>> I'm not a hardware person, and I don't know anything to do in
>> software.
>
>
> Hi Bjorn,
>
> I tried to run similar reboot tests on 2 different Mellanox cards (Connect-X
> family, one card has 2 10G interfaces, the other one has 1 port that
> supports InfiniBand) with U-Boot 1.15.12 and linux 4.2-rc5 and I did not see
> the crash that you encounterred.
>
> Did you check if your Mellanox cards have latest firmware? I did see some
> link issues on my Mellanox cards with its old firmware before.

Good idea; I'll check that, too.  Also, I just learned that these
cards on installed with an extender card because of some space issues,
so we're going to test again without the extender.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1204489

On Mon, Aug 10, 2015 at 10:42 AM, Bjorn Helgaas <bhelgaas@google.com> wrote:
> On Mon, Aug 10, 2015 at 12:16 PM, Duc Dang <dhdang@apm.com> wrote:
>> On Monday, August 10, 2015, Bjorn Helgaas <bhelgaas@google.com> wrote:
>>>
>>> On Fri, Jul 31, 2015 at 12:00 PM, Duc Dang <dhdang@apm.com> wrote:
>>> > On Wed, Jul 29, 2015 at 8:55 AM, Bjorn Helgaas <bhelgaas@google.com>
>>> > wrote:
>>> >> On Tue, Jul 28, 2015 at 08:22:55PM -0500, Bjorn Helgaas wrote:
>>> >>> On Tue, Jul 28, 2015 at 02:50:39PM -0700, Duc Dang wrote:
>>> >>
>>> >>> > Do you have another PCIe card to try on the same reboot test on this
>>> >>> > board?
>>> >>>
>>> >>> I've seen this on at least two Mellanox cards.  I'm running similar
>>> >>> tests
>>> >>> on a different type of card now.
>>> >>
>>> >> FWIW, reboot tests on two machines with Mellanox cards failed, while
>>> >> the
>>> >> same test on a machine with a different proprietary card succeeded.
>>> >
>>> > Thanks, Bjorn.
>>> >
>>> > I don't have the same Mellanox card as yours, but I will also run
>>> > similar reboot test to see if I hit the same issue with my card.
>>>
>>> Any more hints on this?  Nothing has changed on my end, so of course
>>> I'm still seeing this, always on machines with Mellanox, and never on
>>> other machines.  Could this be a hardware issue like a signal
>>> integrity or margin issue?  I don't know where to go from here because
>>> I'm not a hardware person, and I don't know anything to do in
>>> software.
>>
>>
>> Hi Bjorn,
>>
>> I tried to run similar reboot tests on 2 different Mellanox cards (Connect-X
>> family, one card has 2 10G interfaces, the other one has 1 port that
>> supports InfiniBand) with U-Boot 1.15.12 and linux 4.2-rc5 and I did not see
>> the crash that you encounterred.
>>
>> Did you check if your Mellanox cards have latest firmware? I did see some
>> link issues on my Mellanox cards with its old firmware before.
>
> Good idea; I'll check that, too.  Also, I just learned that these
> cards on installed with an extender card because of some space issues,
> so we're going to test again without the extender.

Hi Bjorn,

Are other cards that passed your test installed directly to the
on-board PCIe slot?
If yes, then this is a good data point and it will be useful to test
the case where
your Mellanox cards are directly installed into the on-board PCIe slot.

-- 
Regards,
Duc Dang.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1205394

On Mon, Aug 10, 2015 at 2:07 PM, Duc Dang <dhdang@apm.com> wrote:
> On Mon, Aug 10, 2015 at 10:42 AM, Bjorn Helgaas <bhelgaas@google.com> wrote:
>> On Mon, Aug 10, 2015 at 12:16 PM, Duc Dang <dhdang@apm.com> wrote:
>>> On Monday, August 10, 2015, Bjorn Helgaas <bhelgaas@google.com> wrote:
>>>>
>>>> On Fri, Jul 31, 2015 at 12:00 PM, Duc Dang <dhdang@apm.com> wrote:
>>>> > On Wed, Jul 29, 2015 at 8:55 AM, Bjorn Helgaas <bhelgaas@google.com>
>>>> > wrote:
>>>> >> On Tue, Jul 28, 2015 at 08:22:55PM -0500, Bjorn Helgaas wrote:
>>>> >>> On Tue, Jul 28, 2015 at 02:50:39PM -0700, Duc Dang wrote:
>>>> >>
>>>> >>> > Do you have another PCIe card to try on the same reboot test on this
>>>> >>> > board?
>>>> >>>
>>>> >>> I've seen this on at least two Mellanox cards.  I'm running similar
>>>> >>> tests
>>>> >>> on a different type of card now.
>>>> >>
>>>> >> FWIW, reboot tests on two machines with Mellanox cards failed, while
>>>> >> the
>>>> >> same test on a machine with a different proprietary card succeeded.
>>>> >
>>>> > Thanks, Bjorn.
>>>> >
>>>> > I don't have the same Mellanox card as yours, but I will also run
>>>> > similar reboot test to see if I hit the same issue with my card.
>>>>
>>>> Any more hints on this?  Nothing has changed on my end, so of course
>>>> I'm still seeing this, always on machines with Mellanox, and never on
>>>> other machines.  Could this be a hardware issue like a signal
>>>> integrity or margin issue?  I don't know where to go from here because
>>>> I'm not a hardware person, and I don't know anything to do in
>>>> software.
>>>
>>>
>>> Hi Bjorn,
>>>
>>> I tried to run similar reboot tests on 2 different Mellanox cards (Connect-X
>>> family, one card has 2 10G interfaces, the other one has 1 port that
>>> supports InfiniBand) with U-Boot 1.15.12 and linux 4.2-rc5 and I did not see
>>> the crash that you encounterred.
>>>
>>> Did you check if your Mellanox cards have latest firmware? I did see some
>>> link issues on my Mellanox cards with its old firmware before.
>>
>> Good idea; I'll check that, too.  Also, I just learned that these
>> cards on installed with an extender card because of some space issues,
>> so we're going to test again without the extender.
>
> Hi Bjorn,
>
> Are other cards that passed your test installed directly to the
> on-board PCIe slot?
> If yes, then this is a good data point and it will be useful to test
> the case where
> your Mellanox cards are directly installed into the on-board PCIe slot.

The cards that passed the test were installed directly, with  no
extender.  We removed the extender from one of the machines with the
Mellanox card and have not seen this issue since then.  I think it's
very likely that the problem is related to using the extender.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web