Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1282963 > unrolled thread

Re: [PATCH] irqchip: omap-intc: fix spurious irq handling

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] irqchip: omap-intc: fix spurious irq handling Sekhar Nori <nsekhar@ti.com> - 2015-12-03 12:30 +0100
    Re: [PATCH] irqchip: omap-intc: fix spurious irq handling Tony Lindgren <tony@atomide.com> - 2015-12-03 16:10 +0100
      Re: [PATCH] irqchip: omap-intc: fix spurious irq handling Sekhar Nori <nsekhar@ti.com> - 2015-12-03 16:30 +0100
        Re: [PATCH] irqchip: omap-intc: fix spurious irq handling Tony Lindgren <tony@atomide.com> - 2015-12-03 16:40 +0100

#1282963 — Re: [PATCH] irqchip: omap-intc: fix spurious irq handling

Hi Tony,

On Tuesday 20 October 2015 08:22 PM, Tony Lindgren wrote:
> * John Ogness <john.ogness@linutronix.de> [151020 00:33]:
>> On 2015-10-20, Sekhar Nori <nsekhar@ti.com> wrote:
>>>> Do you know what really is causing the spurious interrupts in your
>>>> case?
>>>
>>> No, not yet.
>>
>> According to the TRM this is normal behavior if conditions that might
>> affect priority are changed during priority sorting.
>>
>>     6.2.5 ARM A8 INTC Spurious Interrupt Handling
>>
>>     The spurious flag indicates whether the result of the sorting (a
>>     window of 10 INTC functional clock cycles after the interrupt
>>     assertion) is invalid. The sorting is invalid if:
>>
>>     - The interrupt that triggered the sorting is no longer active
>>       during the sorting.
>>
>>     - A change in the mask has affected the result during the sorting
>>       time.
>>
>>>> In all the cases I've seen, the spurious interrupts were caused by a
>>>> missing flush of posted write acking the IRQ at the device driver.
>>>> for the _previously triggered_ INTC interrupt.
>>>>
>>>> If you have a reproducable case, I suggest you test that by printing
>>>> out the previous interrupt to check if that makes sense. And then see
>>>> if adding the missing read back to that interrupt handler fixes the
>>>> issue.
>>>
>>> Okay, thats good to know. Thanks for the hints and history of your debug
>>> on OMAP3. The issue is not easily reproducible in my case. But if I try
>>> hard enough, I can get hit it though. So I can surely try your hints.
>>
>> I can reproduce the situation very easily. After running a test for a
>> few minutes and printing out the previous interrupt, I have the
>> following list. These are the irq numbers seen by the handler before the
>> spurious interrupt triggered.
>>
>>     INT12 - EDMACOMPINT - TPCC (EDMA)
>>     INT41 - 3PGSWRXINT0 - CPSW (Ethernet)
>>     INT42 - 3PGSWTXINT0 - CPSW (Ethernet)
>>     INT68 - TINT2       - DMTIMER2
>>     INT72 - UART0INT    - UART0
>>
>> From this I do not think we can put the blame on any single driver. I
>> trigger this situation very easily by putting a load of 7,000+
>> interrupts per second on the system. This means we have 70,000 INTC
>> clock cycles per second where a change in the interrupt priority
>> conditions would cause the priority sorting to become invalid and thus
>> cause the spurious interrupt.
>>
>> I'm not sure if we can/should do anything more than Sekhar's patch of
>> acknowledging the spurious interrupt so the priority sorting algorithm
>> can run again.
> 
> OK thanks for testing. My guess from the above list would be EDMA
> or CPSW missing a flush of posted write. Maybe try adding a readback
> of the related device revision register after acking the interrupt into
> TPCC interrupt handler and CPSW interrupt handler(s)?

I could get back to debugging this only now. I have converted
__raw_writel to writel() and also added readback from the same register
in both EDMA and CPSW drivers. But I am still able to reproduce the
spurious irq reports.

> The timer2 and uart0 seem to be false positives here naturally.

I also added readback in 8250 driver. I haven't touched the timer
driver, but I guess if that driver had an issue, it should have come out
much earlier.

I also saw that sometimes previous irq was the TI LCDC interrupt. Added
readback there too. Did not help.

> I would not yet rule out the "previous interrupt" theory until you have
> tried that. We really want to know the root cause of the issue, just
> printing out spurious interrupt does not fix the problem :)

While we cannot rule out a software issue completely, the description in
TRM around spurious interrupts suggests it can happen even with no role
of software.

May I suggest we go ahead and add this patch to the kernel after
addressing Thomas's comment? At least it will prevent kernel from
locking up with flood of prints when a spurious irq happens and allows
easier debug by others too.

Thanks,
Sekhar

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1283112

* Sekhar Nori <nsekhar@ti.com> [151203 03:29]:
> On Tuesday 20 October 2015 08:22 PM, Tony Lindgren wrote:
> > 
> > OK thanks for testing. My guess from the above list would be EDMA
> > or CPSW missing a flush of posted write. Maybe try adding a readback
> > of the related device revision register after acking the interrupt into
> > TPCC interrupt handler and CPSW interrupt handler(s)?
> 
> I could get back to debugging this only now. I have converted
> __raw_writel to writel() and also added readback from the same register
> in both EDMA and CPSW drivers. But I am still able to reproduce the
> spurious irq reports.
> 
> > The timer2 and uart0 seem to be false positives here naturally.
> 
> I also added readback in 8250 driver. I haven't touched the timer
> driver, but I guess if that driver had an issue, it should have come out
> much earlier.
> 
> I also saw that sometimes previous irq was the TI LCDC interrupt. Added
> readback there too. Did not help.

OK strange, so far all the ones we've seen have been fixable that way.

> > I would not yet rule out the "previous interrupt" theory until you have
> > tried that. We really want to know the root cause of the issue, just
> > printing out spurious interrupt does not fix the problem :)
> 
> While we cannot rule out a software issue completely, the description in
> TRM around spurious interrupts suggests it can happen even with no role
> of software.

Yes maybe we more than one reason for them.

> May I suggest we go ahead and add this patch to the kernel after
> addressing Thomas's comment? At least it will prevent kernel from
> locking up with flood of prints when a spurious irq happens and allows
> easier debug by others too.

Yes we should naturally fix up the kernel locking.

Please also add something like "enable debug for more information"
to the warning. And then print out the current and previous interrupt
if DEBUG is enabled. And in the comments mention that often the spurious
interrupts has been fixed by adding a flush of the posted write to the
previous interrupt handler in the device driver.

Also, do you have a reproducable test case with mainline kernel I
could add to my collection of shell scripts?

Regards,

Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1283126

On Thursday 03 December 2015 08:32 PM, Tony Lindgren wrote:
> * Sekhar Nori <nsekhar@ti.com> [151203 03:29]:
>> On Tuesday 20 October 2015 08:22 PM, Tony Lindgren wrote:
>>>
>>> OK thanks for testing. My guess from the above list would be EDMA
>>> or CPSW missing a flush of posted write. Maybe try adding a readback
>>> of the related device revision register after acking the interrupt into
>>> TPCC interrupt handler and CPSW interrupt handler(s)?
>>
>> I could get back to debugging this only now. I have converted
>> __raw_writel to writel() and also added readback from the same register
>> in both EDMA and CPSW drivers. But I am still able to reproduce the
>> spurious irq reports.
>>
>>> The timer2 and uart0 seem to be false positives here naturally.
>>
>> I also added readback in 8250 driver. I haven't touched the timer
>> driver, but I guess if that driver had an issue, it should have come out
>> much earlier.
>>
>> I also saw that sometimes previous irq was the TI LCDC interrupt. Added
>> readback there too. Did not help.
> 
> OK strange, so far all the ones we've seen have been fixable that way.
> 
>>> I would not yet rule out the "previous interrupt" theory until you have
>>> tried that. We really want to know the root cause of the issue, just
>>> printing out spurious interrupt does not fix the problem :)
>>
>> While we cannot rule out a software issue completely, the description in
>> TRM around spurious interrupts suggests it can happen even with no role
>> of software.
> 
> Yes maybe we more than one reason for them.
> 
>> May I suggest we go ahead and add this patch to the kernel after
>> addressing Thomas's comment? At least it will prevent kernel from
>> locking up with flood of prints when a spurious irq happens and allows
>> easier debug by others too.
> 
> Yes we should naturally fix up the kernel locking.

Alright. Thanks!

> 
> Please also add something like "enable debug for more information"
> to the warning. And then print out the current and previous interrupt

So I am unconvinced (based on the debug above) that the previous
interrupt information is actually giving any more useful information
than what can be gleaned from observing /proc/interrupts. It seems
previous interrupt noted can be any interrupt you would expect to occur
during the test case anyway.

> if DEBUG is enabled. And in the comments mention that often the spurious
> interrupts has been fixed by adding a flush of the posted write to the
> previous interrupt handler in the device driver.

I can add the comment, no problem.

> Also, do you have a reproducable test case with mainline kernel I
> could add to my collection of shell scripts?

The way I reproduce this is to run the serial port at 3Mbaud in internal
loopback mode with DMA enabled. The test program I use[1] compares the
data sent and received byte-for-byte. With current mainline, that can
mismatch pretty soon. The test will likely end before you see any
spurious irq. There are some patches John Ogness is working on
(currently included in TI's v4.1 kernel) which helps sustain the test
for long and then actually expose the spurious irq issue.

Thanks,
Sekhar

[1] https://git.breakpoint.cc/cgit/bigeasy/serialcheck.git
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1283134

* Sekhar Nori <nsekhar@ti.com> [151203 07:25]:
> On Thursday 03 December 2015 08:32 PM, Tony Lindgren wrote:
> > 
> > Yes we should naturally fix up the kernel locking.
> 
> Alright. Thanks!
> 
> > 
> > Please also add something like "enable debug for more information"
> > to the warning. And then print out the current and previous interrupt
> 
> So I am unconvinced (based on the debug above) that the previous
> interrupt information is actually giving any more useful information
> than what can be gleaned from observing /proc/interrupts. It seems
> previous interrupt noted can be any interrupt you would expect to occur
> during the test case anyway.

OK and the fact that I've fixed up 4-5 of these and all of them were
really caused by missing flush of posted write makes me still suspicious :)

> > if DEBUG is enabled. And in the comments mention that often the spurious
> > interrupts has been fixed by adding a flush of the posted write to the
> > previous interrupt handler in the device driver.
> 
> I can add the comment, no problem.

OK thanks. We can add more debug once you figure out what is the root
cause.

> > Also, do you have a reproducable test case with mainline kernel I
> > could add to my collection of shell scripts?
> 
> The way I reproduce this is to run the serial port at 3Mbaud in internal
> loopback mode with DMA enabled. The test program I use[1] compares the
> data sent and received byte-for-byte. With current mainline, that can
> mismatch pretty soon. The test will likely end before you see any
> spurious irq. There are some patches John Ogness is working on
> (currently included in TI's v4.1 kernel) which helps sustain the test
> for long and then actually expose the spurious irq issue.

OK. One thing you have to consider here though is that the EDMA driver
may still wrongly consider several interconnect targets as a single entity.
This can lead to issues where flushing a posted write really only flushes
one of the interconnect targets and that may not be the right one.

Peter has been patching the EDMA driver to solve this problem, but I don't
know if all of them are merged yet, I've added him to Cc.

My bets are on a lack of flush of posted write in the EDMA driver somewhere
and I suggest you investigate that a bit more considering the multiple
interconnect targets :)

Regards,

Tony

> [1] https://git.breakpoint.cc/cgit/bigeasy/serialcheck.git
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web