Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1411800 > unrolled thread

Re: [PATCH 2/2] aer: add support aer interrupt with none MSI/MSI-X/INTx mode

Started byBjorn Helgaas <helgaas@kernel.org>
First post2016-06-02 05:50 +0200
Last post2016-06-08 07:20 +0200
Articles 13 — 3 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Bjorn Helgaas <helgaas@kernel.org> - 2016-06-02 05:50 +0200
    RE: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Po Liu <po.liu@nxp.com> - 2016-06-02 07:10 +0200
      Re: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Bjorn Helgaas <helgaas@kernel.org> - 2016-06-02 16:00 +0200
        Re: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Murali Karicheri <m-karicheri2@ti.com> - 2016-06-02 17:40 +0200
          Re: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Bjorn Helgaas <helgaas@kernel.org> - 2016-06-03 06:20 +0200
            Re: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Murali Karicheri <m-karicheri2@ti.com> - 2016-06-03 19:40 +0200
              Re: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Bjorn Helgaas <helgaas@kernel.org> - 2016-06-04 05:50 +0200
                RE: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Po Liu <po.liu@nxp.com> - 2016-06-06 09:50 +0200
                  Re: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Murali Karicheri <m-karicheri2@ti.com> - 2016-06-06 16:10 +0200
                    Re: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Bjorn Helgaas <helgaas@kernel.org> - 2016-06-06 20:20 +0200
                      RE: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Po Liu <po.liu@nxp.com> - 2016-06-07 12:30 +0200
                        Re: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Bjorn Helgaas <helgaas@kernel.org> - 2016-06-08 00:50 +0200
                          RE: [PATCH 2/2] aer: add support aer interrupt with none  MSI/MSI-X/INTx mode Po Liu <po.liu@nxp.com> - 2016-06-08 07:20 +0200

#1411800 — Re: [PATCH 2/2] aer: add support aer interrupt with none MSI/MSI-X/INTx mode

FromBjorn Helgaas <helgaas@kernel.org>
Date2016-06-02 05:50 +0200
SubjectRe: [PATCH 2/2] aer: add support aer interrupt with none MSI/MSI-X/INTx mode
Message-ID<rFrNT-GD-1@gated-at.bofh.it>
[+cc Rob]

Hi Po,

On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
> When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
> maybe there is interrupt line for aer pme etc. Search the interrupt
> number in the fdt file.

My understanding is that AER interrupt signaling can be done via INTx,
MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
either?  How is the interrupt you're requesting here different from
INTx?

My guess is that your device *does* support INTx, and we should use
that.  ACPI has the generic _PRT that describes INTx routing.  There
must be something similar for DT, but I don't know what it is.  It
seems like this should be described using that DT mechanism (whatever
it is), and we shouldn't need special-case code in the portdrv code.

> Signed-off-by: Po Liu <po.liu@nxp.com>
> ---
>  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
>  1 file changed, 28 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
> index 32d4d0a..64833e5 100644
> --- a/drivers/pci/pcie/portdrv_core.c
> +++ b/drivers/pci/pcie/portdrv_core.c
> @@ -15,6 +15,7 @@
>  #include <linux/slab.h>
>  #include <linux/pcieport_if.h>
>  #include <linux/aer.h>
> +#include <linux/of_irq.h>
>  
>  #include "../pci.h"
>  #include "portdrv.h"
> @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev *dev, int *vectors, int mask)
>  static int init_service_irqs(struct pci_dev *dev, int *irqs, int mask)
>  {
>  	int i, irq = -1;
> +	int ret;
> +	struct device_node *np = NULL;
> +
> +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> +		irqs[i] = 0;
> +
> +	if (dev->bus->dev.of_node)
> +		np = dev->bus->dev.of_node;
> +
> +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
> +	 * request irq for aer
> +	 */
> +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
> +			(mask & PCIE_PORT_SERVICE_PME)) {
> +		ret = of_irq_get_byname(np, "aer");
> +		if (ret > 0) {
> +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
> +			if (dev->irq)
> +				irq = dev->irq;
> +			goto no_msi;
> +		}
> +	}
>  
>  	/*
>  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
> @@ -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev, int *irqs, int mask)
>  		irq = dev->irq;
>  
>   no_msi:
> -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> -		irqs[i] = irq;
> +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
> +		if (!irqs[i])
> +			irqs[i] = irq;
> +	}
>  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
>  
> -	if (irq < 0)
> +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
>  		return -ENODEV;
>  	return 0;
>  }
> -- 
> 2.1.0.27.g96db324
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[toc] | [next] | [standalone]

#1411839

FromPo Liu <po.liu@nxp.com>
Date2016-06-02 07:10 +0200
Message-ID<rFt3k-1Gn-21@gated-at.bofh.it>
In reply to#1411800
Hi Bjorn,


>  -----Original Message-----
>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>  Sent: Thursday, June 02, 2016 11:48 AM
>  To: Po Liu
>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
>  MSI/MSI-X/INTx mode
>  
>  [+cc Rob]
>  
>  Hi Po,
>  
>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
>  > maybe there is interrupt line for aer pme etc. Search the interrupt
>  > number in the fdt file.
>  
>  My understanding is that AER interrupt signaling can be done via INTx,
>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
>  either?  How is the interrupt you're requesting here different from INTx?

Layerscape use none of MSI or MSI-X or INTx to indicate the devices or root error
in RC mode. But use an independent SPI interrupt(arm interrupt controller) line.
And at same time, AER capability list in PCIe register descriptors. And also,
The Root Error Command/status register was proper to enable/disable the AER interrupt.

This hardware implementation make sense in some platforms, and this was described
in the function init_service_irqs() in the drivers/pci/pcie/portdrv_core.c in current
kernel as:

241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
244      */                                                                             

So I think this was the proper place to update the irq number before aer service
driver was loaded.

>  
>  My guess is that your device *does* support INTx, and we should use that.
>  ACPI has the generic _PRT that describes INTx routing.  There must be
>  something similar for DT, but I don't know what it is.  It seems like
>  this should be described using that DT mechanism (whatever it is), and
>  we shouldn't need special-case code in the portdrv code.
>  
>  > Signed-off-by: Po Liu <po.liu@nxp.com>
>  > ---
>  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
>  >  1 file changed, 28 insertions(+), 3 deletions(-)
>  >
>  > diff --git a/drivers/pci/pcie/portdrv_core.c
>  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644
>  > --- a/drivers/pci/pcie/portdrv_core.c
>  > +++ b/drivers/pci/pcie/portdrv_core.c
>  > @@ -15,6 +15,7 @@
>  >  #include <linux/slab.h>
>  >  #include <linux/pcieport_if.h>
>  >  #include <linux/aer.h>
>  > +#include <linux/of_irq.h>
>  >
>  >  #include "../pci.h"
>  >  #include "portdrv.h"
>  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev
>  > *dev, int *vectors, int mask)  static int init_service_irqs(struct
>  > pci_dev *dev, int *irqs, int mask)  {
>  >  	int i, irq = -1;
>  > +	int ret;
>  > +	struct device_node *np = NULL;
>  > +
>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>  > +		irqs[i] = 0;
>  > +
>  > +	if (dev->bus->dev.of_node)
>  > +		np = dev->bus->dev.of_node;
>  > +
>  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
>  > +	 * request irq for aer
>  > +	 */
>  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
>  > +			(mask & PCIE_PORT_SERVICE_PME)) {
>  > +		ret = of_irq_get_byname(np, "aer");
>  > +		if (ret > 0) {
>  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
>  > +			if (dev->irq)
>  > +				irq = dev->irq;
>  > +			goto no_msi;
>  > +		}
>  > +	}
>  >
>  >  	/*
>  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
>  @@
>  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,
>  int *irqs, int mask)
>  >  		irq = dev->irq;
>  >
>  >   no_msi:
>  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>  > -		irqs[i] = irq;
>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
>  > +		if (!irqs[i])
>  > +			irqs[i] = irq;
>  > +	}
>  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
>  >
>  > -	if (irq < 0)
>  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
>  >  		return -ENODEV;
>  >  	return 0;
>  >  }
>  > --
>  > 2.1.0.27.g96db324
>  >
>  >
>  > _______________________________________________
>  > linux-arm-kernel mailing list
>  > linux-arm-kernel@lists.infradead.org
>  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[toc] | [prev] | [next] | [standalone]

#1412243

FromBjorn Helgaas <helgaas@kernel.org>
Date2016-06-02 16:00 +0200
Message-ID<rFBkd-6AY-17@gated-at.bofh.it>
In reply to#1411839
On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >  -----Original Message-----
> >  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >  Sent: Thursday, June 02, 2016 11:48 AM
> >  To: Po Liu
> >  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> >  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
> >  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
> >  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
> >  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >  MSI/MSI-X/INTx mode
> >  
> >  [+cc Rob]
> >  
> >  Hi Po,
> >  
> >  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> >  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
> >  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
> >  > maybe there is interrupt line for aer pme etc. Search the interrupt
> >  > number in the fdt file.
> >  
> >  My understanding is that AER interrupt signaling can be done via INTx,
> >  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
> >  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
> >  either?  How is the interrupt you're requesting here different from INTx?
> 
> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
> or root error in RC mode. But use an independent SPI interrupt(arm
> interrupt controller) line.  

The Root Port is a PCI device and should follow the normal PCI rules
for interrupts.  As far as I understand, that means it should use MSI,
MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
That's all from the PCI point of view, of course.

What's on the other end of those interrupts is outside the scope of
PCI.  An INTx interrupt (either a conventional PCI wire or a PCIe
virtual INTx wire) might be connected to an IOAPIC, an ARM SPI, or
something else.  Drivers should not care about how it is connected,
and that's why I don't think this code really fits in portdrv.

Maybe your Root Port is non-compliant in some way and maybe we need a
quirk or something to work around that hardware defect.  But I'm not
convinced yet that we have that sort of problem.  It seems like we
just don't have the correct DT description.

> And at same time, AER capability list
> in PCIe register descriptors. And also, The Root Error
> Command/status register was proper to enable/disable the AER
> interrupt.

I'm not sure what you're saying here.  Here's what I think you said;
please correct me if I'm wrong:

  - Your Root Port has an AER capability.

  - Your Root Port does not support MSI or MSI-X.  (This would
    actually be a spec violation because the PCIe spec r3.0, sec 7.7
    says "All PCI Express device Functions that are capable of
    generating interrupts must implement MSI or MSI-X or both.")
  
  - The three enable bits in the Root Error Command Register enable or
    disable generation of interrupts.  These enable bits control all
    interrupts, whether MSI, MSI-X, or INTx.

  - The Root Error Status Register contains an "Advanced Error
    Interrupt Message Number" field, but that's only applicable if MSI
    or MSI-X is enabled, and if your device doesn't support MSI or
    MSI-X, this field doesn't apply.

> This hardware implementation make sense in some platforms, and this
> was described in the function init_service_irqs() in the
> drivers/pci/pcie/portdrv_core.c in current kernel as:

> 
> 241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
> 242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
> 243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
> 244      */                                                                             
> 
> So I think this was the proper place to update the irq number before aer service
> driver was loaded.
> 
> >  
> >  My guess is that your device *does* support INTx, and we should use that.
> >  ACPI has the generic _PRT that describes INTx routing.  There must be
> >  something similar for DT, but I don't know what it is.  It seems like
> >  this should be described using that DT mechanism (whatever it is), and
> >  we shouldn't need special-case code in the portdrv code.
> >  
> >  > Signed-off-by: Po Liu <po.liu@nxp.com>
> >  > ---
> >  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
> >  >  1 file changed, 28 insertions(+), 3 deletions(-)
> >  >
> >  > diff --git a/drivers/pci/pcie/portdrv_core.c
> >  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644
> >  > --- a/drivers/pci/pcie/portdrv_core.c
> >  > +++ b/drivers/pci/pcie/portdrv_core.c
> >  > @@ -15,6 +15,7 @@
> >  >  #include <linux/slab.h>
> >  >  #include <linux/pcieport_if.h>
> >  >  #include <linux/aer.h>
> >  > +#include <linux/of_irq.h>
> >  >
> >  >  #include "../pci.h"
> >  >  #include "portdrv.h"
> >  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev
> >  > *dev, int *vectors, int mask)  static int init_service_irqs(struct
> >  > pci_dev *dev, int *irqs, int mask)  {
> >  >  	int i, irq = -1;
> >  > +	int ret;
> >  > +	struct device_node *np = NULL;
> >  > +
> >  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> >  > +		irqs[i] = 0;
> >  > +
> >  > +	if (dev->bus->dev.of_node)
> >  > +		np = dev->bus->dev.of_node;
> >  > +
> >  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
> >  > +	 * request irq for aer
> >  > +	 */
> >  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
> >  > +			(mask & PCIE_PORT_SERVICE_PME)) {
> >  > +		ret = of_irq_get_byname(np, "aer");
> >  > +		if (ret > 0) {
> >  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
> >  > +			if (dev->irq)
> >  > +				irq = dev->irq;
> >  > +			goto no_msi;
> >  > +		}
> >  > +	}
> >  >
> >  >  	/*
> >  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
> >  @@
> >  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,
> >  int *irqs, int mask)
> >  >  		irq = dev->irq;
> >  >
> >  >   no_msi:
> >  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> >  > -		irqs[i] = irq;
> >  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
> >  > +		if (!irqs[i])
> >  > +			irqs[i] = irq;
> >  > +	}
> >  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
> >  >
> >  > -	if (irq < 0)
> >  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
> >  >  		return -ENODEV;
> >  >  	return 0;
> >  >  }
> >  > --
> >  > 2.1.0.27.g96db324
> >  >
> >  >
> >  > _______________________________________________
> >  > linux-arm-kernel mailing list
> >  > linux-arm-kernel@lists.infradead.org
> >  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[toc] | [prev] | [next] | [standalone]

#1412364

FromMurali Karicheri <m-karicheri2@ti.com>
Date2016-06-02 17:40 +0200
Message-ID<rFCSZ-7Dw-9@gated-at.bofh.it>
In reply to#1412243
On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
>>>  -----Original Message-----
>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>>>  Sent: Thursday, June 02, 2016 11:48 AM
>>>  To: Po Liu
>>>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
>>>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
>>>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
>>>  MSI/MSI-X/INTx mode
>>>  
>>>  [+cc Rob]
>>>  
>>>  Hi Po,
>>>  
>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
>>>  > maybe there is interrupt line for aer pme etc. Search the interrupt
>>>  > number in the fdt file.
>>>  
>>>  My understanding is that AER interrupt signaling can be done via INTx,
>>>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
>>>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
>>>  either?  How is the interrupt you're requesting here different from INTx?
>>
>> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
>> or root error in RC mode. But use an independent SPI interrupt(arm
>> interrupt controller) line.  
> 
> The Root Port is a PCI device and should follow the normal PCI rules
> for interrupts.  As far as I understand, that means it should use MSI,
> MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
> use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
> INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
> That's all from the PCI point of view, of course.
> 
Bjorn.

I am faced with the same issue on Keystone PCI hardware and it has been
on my TODO list  for quite some time. Keystone PCI hardware also doesn't
use MSI or MSI-X or INTx for reporting errors received at the root port,
but use a platform interrupt instead (not complaint to PCI standard as
per PCI base spec). So I would need similar change to have the error
interrupt passed to the aer driver. So there are hardware out there
like Keystone which requires to support this through platform IRQ.

> What's on the other end of those interrupts is outside the scope of
> PCI.  An INTx interrupt (either a conventional PCI wire or a PCIe
> virtual INTx wire) might be connected to an IOAPIC, an ARM SPI, or
> something else.  Drivers should not care about how it is connected,
> and that's why I don't think this code really fits in portdrv.
> 
> Maybe your Root Port is non-compliant in some way and maybe we need a
> quirk or something to work around that hardware defect.  But I'm not
> convinced yet that we have that sort of problem.  It seems like we
> just don't have the correct DT description.

Is quirk is what is suggested here to support this?

Murali
> 
>> And at same time, AER capability list
>> in PCIe register descriptors. And also, The Root Error
>> Command/status register was proper to enable/disable the AER
>> interrupt.
> 
> I'm not sure what you're saying here.  Here's what I think you said;
> please correct me if I'm wrong:
> 
>   - Your Root Port has an AER capability.
> 
>   - Your Root Port does not support MSI or MSI-X.  (This would
>     actually be a spec violation because the PCIe spec r3.0, sec 7.7
>     says "All PCI Express device Functions that are capable of
>     generating interrupts must implement MSI or MSI-X or both.")
>   
>   - The three enable bits in the Root Error Command Register enable or
>     disable generation of interrupts.  These enable bits control all
>     interrupts, whether MSI, MSI-X, or INTx.
> 
>   - The Root Error Status Register contains an "Advanced Error
>     Interrupt Message Number" field, but that's only applicable if MSI
>     or MSI-X is enabled, and if your device doesn't support MSI or
>     MSI-X, this field doesn't apply.
> 
>> This hardware implementation make sense in some platforms, and this
>> was described in the function init_service_irqs() in the
>> drivers/pci/pcie/portdrv_core.c in current kernel as:
> 
>>
>> 241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
>> 242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
>> 243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
>> 244      */                                                                             
>>
>> So I think this was the proper place to update the irq number before aer service
>> driver was loaded.
>>
>>>  
>>>  My guess is that your device *does* support INTx, and we should use that.
>>>  ACPI has the generic _PRT that describes INTx routing.  There must be
>>>  something similar for DT, but I don't know what it is.  It seems like
>>>  this should be described using that DT mechanism (whatever it is), and
>>>  we shouldn't need special-case code in the portdrv code.
>>>  
>>>  > Signed-off-by: Po Liu <po.liu@nxp.com>
>>>  > ---
>>>  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
>>>  >  1 file changed, 28 insertions(+), 3 deletions(-)
>>>  >
>>>  > diff --git a/drivers/pci/pcie/portdrv_core.c
>>>  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644
>>>  > --- a/drivers/pci/pcie/portdrv_core.c
>>>  > +++ b/drivers/pci/pcie/portdrv_core.c
>>>  > @@ -15,6 +15,7 @@
>>>  >  #include <linux/slab.h>
>>>  >  #include <linux/pcieport_if.h>
>>>  >  #include <linux/aer.h>
>>>  > +#include <linux/of_irq.h>
>>>  >
>>>  >  #include "../pci.h"
>>>  >  #include "portdrv.h"
>>>  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev
>>>  > *dev, int *vectors, int mask)  static int init_service_irqs(struct
>>>  > pci_dev *dev, int *irqs, int mask)  {
>>>  >  	int i, irq = -1;
>>>  > +	int ret;
>>>  > +	struct device_node *np = NULL;
>>>  > +
>>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>>>  > +		irqs[i] = 0;
>>>  > +
>>>  > +	if (dev->bus->dev.of_node)
>>>  > +		np = dev->bus->dev.of_node;
>>>  > +
>>>  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
>>>  > +	 * request irq for aer
>>>  > +	 */
>>>  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
>>>  > +			(mask & PCIE_PORT_SERVICE_PME)) {
>>>  > +		ret = of_irq_get_byname(np, "aer");
>>>  > +		if (ret > 0) {
>>>  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
>>>  > +			if (dev->irq)
>>>  > +				irq = dev->irq;
>>>  > +			goto no_msi;
>>>  > +		}
>>>  > +	}
>>>  >
>>>  >  	/*
>>>  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
>>>  @@
>>>  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,
>>>  int *irqs, int mask)
>>>  >  		irq = dev->irq;
>>>  >
>>>  >   no_msi:
>>>  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>>>  > -		irqs[i] = irq;
>>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
>>>  > +		if (!irqs[i])
>>>  > +			irqs[i] = irq;
>>>  > +	}
>>>  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
>>>  >
>>>  > -	if (irq < 0)
>>>  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
>>>  >  		return -ENODEV;
>>>  >  	return 0;
>>>  >  }
>>>  > --
>>>  > 2.1.0.27.g96db324
>>>  >
>>>  >
>>>  > _______________________________________________
>>>  > linux-arm-kernel mailing list
>>>  > linux-arm-kernel@lists.infradead.org
>>>  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Murali Karicheri
Linux Kernel, Keystone

[toc] | [prev] | [next] | [standalone]

#1412778

FromBjorn Helgaas <helgaas@kernel.org>
Date2016-06-03 06:20 +0200
Message-ID<rFOKu-6Pk-5@gated-at.bofh.it>
In reply to#1412364
On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> > On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >>>  -----Original Message-----
> >>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >>>  Sent: Thursday, June 02, 2016 11:48 AM
> >>>  To: Po Liu
> >>>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> >>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
> >>>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
> >>>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
> >>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >>>  MSI/MSI-X/INTx mode
> >>>  
> >>>  [+cc Rob]
> >>>  
> >>>  Hi Po,
> >>>  
> >>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> >>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
> >>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
> >>>  > maybe there is interrupt line for aer pme etc. Search the interrupt
> >>>  > number in the fdt file.
> >>>  
> >>>  My understanding is that AER interrupt signaling can be done via INTx,
> >>>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
> >>>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
> >>>  either?  How is the interrupt you're requesting here different from INTx?
> >>
> >> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
> >> or root error in RC mode. But use an independent SPI interrupt(arm
> >> interrupt controller) line.  
> > 
> > The Root Port is a PCI device and should follow the normal PCI rules
> > for interrupts.  As far as I understand, that means it should use MSI,
> > MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
> > use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
> > INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
> > That's all from the PCI point of view, of course.
> 
> I am faced with the same issue on Keystone PCI hardware and it has been
> on my TODO list  for quite some time. Keystone PCI hardware also doesn't
> use MSI or MSI-X or INTx for reporting errors received at the root port,
> but use a platform interrupt instead (not complaint to PCI standard as
> per PCI base spec). So I would need similar change to have the error
> interrupt passed to the aer driver. So there are hardware out there
> like Keystone which requires to support this through platform IRQ.

This is not a new area of the spec, and it's hard for me to believe
that these two new PCIe controllers are both broken the same way
(although I guess both are DesignWare-based, so maybe this is the same
underlying problem in both cases?).  I think it's more likely that we
just haven't figured out the right way to describe this in the DT.

I assume you have a Root Port with an AER capability, no MSI
capability, and no MSI-X capability, right?  What does its Interrupt
Pin register contain?  If it's zero, it doesn't use INTx either, so
according to the spec it should generate no interrupts.

But if Interrupt Pin is non-zero, that means the Root Port should be
able to generate virtual INTx interrupts.  Presumably the Root Complex
connects those interrupts to something; maybe to your platform
interrupt?

PCI doesn't say anything about an interrupt after it leaves the Root
Complex, so the fact that it's connected to a "platform interrupt" or
"SPI interrupt" or "IOAPIC interrupt" doesn't make it non-compliant.
Shouldn't we be able to use the interrupt-map and related DT
properties to express the fact that Root Port virtual INTx is routed
to platform interrupt Y, even without any special-case code in
portdrv?

> > What's on the other end of those interrupts is outside the scope of
> > PCI.  An INTx interrupt (either a conventional PCI wire or a PCIe
> > virtual INTx wire) might be connected to an IOAPIC, an ARM SPI, or
> > something else.  Drivers should not care about how it is connected,
> > and that's why I don't think this code really fits in portdrv.
> > 
> > Maybe your Root Port is non-compliant in some way and maybe we need a
> > quirk or something to work around that hardware defect.  But I'm not
> > convinced yet that we have that sort of problem.  It seems like we
> > just don't have the correct DT description.
> 
> Is quirk is what is suggested here to support this?

No, I don't understand the problem yet, so I'm not ready to suggest a
solution.

> >> And at same time, AER capability list
> >> in PCIe register descriptors. And also, The Root Error
> >> Command/status register was proper to enable/disable the AER
> >> interrupt.
> > 
> > I'm not sure what you're saying here.  Here's what I think you said;
> > please correct me if I'm wrong:
> > 
> >   - Your Root Port has an AER capability.
> > 
> >   - Your Root Port does not support MSI or MSI-X.  (This would
> >     actually be a spec violation because the PCIe spec r3.0, sec 7.7
> >     says "All PCI Express device Functions that are capable of
> >     generating interrupts must implement MSI or MSI-X or both.")
> >   
> >   - The three enable bits in the Root Error Command Register enable or
> >     disable generation of interrupts.  These enable bits control all
> >     interrupts, whether MSI, MSI-X, or INTx.
> > 
> >   - The Root Error Status Register contains an "Advanced Error
> >     Interrupt Message Number" field, but that's only applicable if MSI
> >     or MSI-X is enabled, and if your device doesn't support MSI or
> >     MSI-X, this field doesn't apply.
> > 
> >> This hardware implementation make sense in some platforms, and this
> >> was described in the function init_service_irqs() in the
> >> drivers/pci/pcie/portdrv_core.c in current kernel as:
> > 
> >>
> >> 241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
> >> 242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
> >> 243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
> >> 244      */                                                                             
> >>
> >> So I think this was the proper place to update the irq number before aer service
> >> driver was loaded.
> >>
> >>>  
> >>>  My guess is that your device *does* support INTx, and we should use that.
> >>>  ACPI has the generic _PRT that describes INTx routing.  There must be
> >>>  something similar for DT, but I don't know what it is.  It seems like
> >>>  this should be described using that DT mechanism (whatever it is), and
> >>>  we shouldn't need special-case code in the portdrv code.
> >>>  
> >>>  > Signed-off-by: Po Liu <po.liu@nxp.com>
> >>>  > ---
> >>>  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
> >>>  >  1 file changed, 28 insertions(+), 3 deletions(-)
> >>>  >
> >>>  > diff --git a/drivers/pci/pcie/portdrv_core.c
> >>>  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644
> >>>  > --- a/drivers/pci/pcie/portdrv_core.c
> >>>  > +++ b/drivers/pci/pcie/portdrv_core.c
> >>>  > @@ -15,6 +15,7 @@
> >>>  >  #include <linux/slab.h>
> >>>  >  #include <linux/pcieport_if.h>
> >>>  >  #include <linux/aer.h>
> >>>  > +#include <linux/of_irq.h>
> >>>  >
> >>>  >  #include "../pci.h"
> >>>  >  #include "portdrv.h"
> >>>  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev
> >>>  > *dev, int *vectors, int mask)  static int init_service_irqs(struct
> >>>  > pci_dev *dev, int *irqs, int mask)  {
> >>>  >  	int i, irq = -1;
> >>>  > +	int ret;
> >>>  > +	struct device_node *np = NULL;
> >>>  > +
> >>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> >>>  > +		irqs[i] = 0;
> >>>  > +
> >>>  > +	if (dev->bus->dev.of_node)
> >>>  > +		np = dev->bus->dev.of_node;
> >>>  > +
> >>>  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
> >>>  > +	 * request irq for aer
> >>>  > +	 */
> >>>  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
> >>>  > +			(mask & PCIE_PORT_SERVICE_PME)) {
> >>>  > +		ret = of_irq_get_byname(np, "aer");
> >>>  > +		if (ret > 0) {
> >>>  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
> >>>  > +			if (dev->irq)
> >>>  > +				irq = dev->irq;
> >>>  > +			goto no_msi;
> >>>  > +		}
> >>>  > +	}
> >>>  >
> >>>  >  	/*
> >>>  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
> >>>  @@
> >>>  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,
> >>>  int *irqs, int mask)
> >>>  >  		irq = dev->irq;
> >>>  >
> >>>  >   no_msi:
> >>>  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
> >>>  > -		irqs[i] = irq;
> >>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
> >>>  > +		if (!irqs[i])
> >>>  > +			irqs[i] = irq;
> >>>  > +	}
> >>>  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
> >>>  >
> >>>  > -	if (irq < 0)
> >>>  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
> >>>  >  		return -ENODEV;
> >>>  >  	return 0;
> >>>  >  }
> >>>  > --
> >>>  > 2.1.0.27.g96db324
> >>>  >
> >>>  >
> >>>  > _______________________________________________
> >>>  > linux-arm-kernel mailing list
> >>>  > linux-arm-kernel@lists.infradead.org
> >>>  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> 
> -- 
> Murali Karicheri
> Linux Kernel, Keystone

[toc] | [prev] | [next] | [standalone]

#1413374

FromMurali Karicheri <m-karicheri2@ti.com>
Date2016-06-03 19:40 +0200
Message-ID<rG1eF-6eI-1@gated-at.bofh.it>
In reply to#1412778
Po,

Sorry to hijack your discussion, but the problem seems to be same for
Keystone PCI controller which is also designware (old version) based.

On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
> On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
>> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
>>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
>>>>>  -----Original Message-----
>>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>>>>>  Sent: Thursday, June 02, 2016 11:48 AM
>>>>>  To: Po Liu
>>>>>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
>>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
>>>>>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
>>>>>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
>>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
>>>>>  MSI/MSI-X/INTx mode
>>>>>  
>>>>>  [+cc Rob]
>>>>>  
>>>>>  Hi Po,
>>>>>  
>>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
>>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
>>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
>>>>>  > maybe there is interrupt line for aer pme etc. Search the interrupt
>>>>>  > number in the fdt file.
>>>>>  
>>>>>  My understanding is that AER interrupt signaling can be done via INTx,
>>>>>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
>>>>>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
>>>>>  either?  How is the interrupt you're requesting here different from INTx?
>>>>
>>>> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
>>>> or root error in RC mode. But use an independent SPI interrupt(arm
>>>> interrupt controller) line.  
>>>
>>> The Root Port is a PCI device and should follow the normal PCI rules
>>> for interrupts.  As far as I understand, that means it should use MSI,
>>> MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
>>> use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
>>> INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
>>> That's all from the PCI point of view, of course.
>>
>> I am faced with the same issue on Keystone PCI hardware and it has been
>> on my TODO list  for quite some time. Keystone PCI hardware also doesn't
>> use MSI or MSI-X or INTx for reporting errors received at the root port,
>> but use a platform interrupt instead (not complaint to PCI standard as
>> per PCI base spec). So I would need similar change to have the error
>> interrupt passed to the aer driver. So there are hardware out there
>> like Keystone which requires to support this through platform IRQ.
> 
> This is not a new area of the spec, and it's hard for me to believe
> that these two new PCIe controllers are both broken the same way
> (although I guess both are DesignWare-based, so maybe this is the same
> underlying problem in both cases?).  I think it's more likely that we
> just haven't figured out the right way to describe this in the DT.

Keystone is using an older version of the designware IP and it implements
all of the interrupts in the application register space unlike other
newer version of the hardware. So I assume, the version used on Layerscape
is also an older version and the both have same issue in terms of 
non standard platform interrupt used for error reporting.

> 
> I assume you have a Root Port with an AER capability, no MSI
> capability, and no MSI-X capability, right? 

Has AER capability and both MSI and INTx (legacy) capability

> What does its Interrupt
> Pin register contain?  If it's zero, it doesn't use INTx either, so
> according to the spec it should generate no interrupts.
> 
At address offset 0x3C by default has a value of 1, but it is writable
by software. So default is INTx A.

> But if Interrupt Pin is non-zero, that means the Root Port should be
> able to generate virtual INTx interrupts.  Presumably the Root Complex
> connects those interrupts to something; maybe to your platform
> interrupt?

Probably that is what is happening. Both Power management and Error
interrupts are raised on platform interrupt lines.

12 Error Interrupts

[0] System error (OR of fatal, nonfatal, correctable errors) (RC mode only)
[1] PCIe fatal error (RC mode only)
[2] PCIe non-fatal error (RC mode only)
[3] PCIe correctable error (RC mode only)
[4] AXI Error due to fatal condition in AXI bridge (EP/RC modes)
[5] PCIe advanced error (RC mode only)

13 Power management and reset event interrupts

[0] Power management turn-off message interrupt (EP mode only)
[1] Power management ack message interrupt (RC mode only)
[2] Power management event interrupt (RC mode only)
[3] Link request reset interrupt (hot reset or link down) (RC mode only)

> 
> PCI doesn't say anything about an interrupt after it leaves the Root
> Complex, so the fact that it's connected to a "platform interrupt" or
> "SPI interrupt" or "IOAPIC interrupt" doesn't make it non-compliant.
> Shouldn't we be able to use the interrupt-map and related DT
> properties to express the fact that Root Port virtual INTx is routed
> to platform interrupt Y, even without any special-case code in
> portdrv?

My understanding is if RC also raise interrupt on INTx, then below 
map make sense, where either EP or RC can raise interrupt and the
line will be muxed for interrupt from EP or RC port.

Here is the DT entry in PCIE keystone for Legacy interrupt mapping
to host interrupt. 
                        #interrupt-cells = <1>;
                        interrupt-map-mask = <0 0 0 7>;
                        interrupt-map = <0 0 0 1 &pcie_intc0 0>, /* INT A */
                                        <0 0 0 2 &pcie_intc0 1>, /* INT B */
                                        <0 0 0 3 &pcie_intc0 2>, /* INT C */
                                        <0 0 0 4 &pcie_intc0 3>; /* INT D */


And  then

                        pcie_intc0: legacy-interrupt-controller {
                                interrupt-controller;
                                #interrupt-cells = <1>;
                                interrupt-parent = <&gic>;
                                interrupts = <GIC_SPI 48 IRQ_TYPE_EDGE_RISING>,
                                        <GIC_SPI 49 IRQ_TYPE_EDGE_RISING>,
                                        <GIC_SPI 50 IRQ_TYPE_EDGE_RISING>,
                                        <GIC_SPI 51 IRQ_TYPE_EDGE_RISING>;
                        };

So if RC interrupt for Power management and Error interrupt is a separate
line, how can software describe that using the above interrupt map?

Murali
> 
>>> What's on the other end of those interrupts is outside the scope of
>>> PCI.  An INTx interrupt (either a conventional PCI wire or a PCIe
>>> virtual INTx wire) might be connected to an IOAPIC, an ARM SPI, or
>>> something else.  Drivers should not care about how it is connected,
>>> and that's why I don't think this code really fits in portdrv.
>>>
>>> Maybe your Root Port is non-compliant in some way and maybe we need a
>>> quirk or something to work around that hardware defect.  But I'm not
>>> convinced yet that we have that sort of problem.  It seems like we
>>> just don't have the correct DT description.
>>
>> Is quirk is what is suggested here to support this?
> 
> No, I don't understand the problem yet, so I'm not ready to suggest a
> solution.
> 
>>>> And at same time, AER capability list
>>>> in PCIe register descriptors. And also, The Root Error
>>>> Command/status register was proper to enable/disable the AER
>>>> interrupt.
>>>
>>> I'm not sure what you're saying here.  Here's what I think you said;
>>> please correct me if I'm wrong:
>>>
>>>   - Your Root Port has an AER capability.
>>>
>>>   - Your Root Port does not support MSI or MSI-X.  (This would
>>>     actually be a spec violation because the PCIe spec r3.0, sec 7.7
>>>     says "All PCI Express device Functions that are capable of
>>>     generating interrupts must implement MSI or MSI-X or both.")
>>>   
>>>   - The three enable bits in the Root Error Command Register enable or
>>>     disable generation of interrupts.  These enable bits control all
>>>     interrupts, whether MSI, MSI-X, or INTx.
>>>
>>>   - The Root Error Status Register contains an "Advanced Error
>>>     Interrupt Message Number" field, but that's only applicable if MSI
>>>     or MSI-X is enabled, and if your device doesn't support MSI or
>>>     MSI-X, this field doesn't apply.
>>>
>>>> This hardware implementation make sense in some platforms, and this
>>>> was described in the function init_service_irqs() in the
>>>> drivers/pci/pcie/portdrv_core.c in current kernel as:
>>>
>>>>
>>>> 241      * We're not going to use MSI-X, so try MSI and fall back to INTx.     
>>>> 242      * If neither MSI/MSI-X nor INTx available, try other interrupt.  On  
>>>> 243      * some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode
>>>> 244      */                                                                             
>>>>
>>>> So I think this was the proper place to update the irq number before aer service
>>>> driver was loaded.
>>>>
>>>>>  
>>>>>  My guess is that your device *does* support INTx, and we should use that.
>>>>>  ACPI has the generic _PRT that describes INTx routing.  There must be
>>>>>  something similar for DT, but I don't know what it is.  It seems like
>>>>>  this should be described using that DT mechanism (whatever it is), and
>>>>>  we shouldn't need special-case code in the portdrv code.
>>>>>  
>>>>>  > Signed-off-by: Po Liu <po.liu@nxp.com>
>>>>>  > ---
>>>>>  >  drivers/pci/pcie/portdrv_core.c | 31 ++++++++++++++++++++++++++++---
>>>>>  >  1 file changed, 28 insertions(+), 3 deletions(-)
>>>>>  >
>>>>>  > diff --git a/drivers/pci/pcie/portdrv_core.c
>>>>>  > b/drivers/pci/pcie/portdrv_core.c index 32d4d0a..64833e5 100644
>>>>>  > --- a/drivers/pci/pcie/portdrv_core.c
>>>>>  > +++ b/drivers/pci/pcie/portdrv_core.c
>>>>>  > @@ -15,6 +15,7 @@
>>>>>  >  #include <linux/slab.h>
>>>>>  >  #include <linux/pcieport_if.h>
>>>>>  >  #include <linux/aer.h>
>>>>>  > +#include <linux/of_irq.h>
>>>>>  >
>>>>>  >  #include "../pci.h"
>>>>>  >  #include "portdrv.h"
>>>>>  > @@ -199,6 +200,28 @@ static int pcie_port_enable_msix(struct pci_dev
>>>>>  > *dev, int *vectors, int mask)  static int init_service_irqs(struct
>>>>>  > pci_dev *dev, int *irqs, int mask)  {
>>>>>  >  	int i, irq = -1;
>>>>>  > +	int ret;
>>>>>  > +	struct device_node *np = NULL;
>>>>>  > +
>>>>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>>>>>  > +		irqs[i] = 0;
>>>>>  > +
>>>>>  > +	if (dev->bus->dev.of_node)
>>>>>  > +		np = dev->bus->dev.of_node;
>>>>>  > +
>>>>>  > +	/* If root port doesn't support MSI/MSI-X/INTx in RC mode,
>>>>>  > +	 * request irq for aer
>>>>>  > +	 */
>>>>>  > +	if (IS_ENABLED(CONFIG_OF_IRQ) && np &&
>>>>>  > +			(mask & PCIE_PORT_SERVICE_PME)) {
>>>>>  > +		ret = of_irq_get_byname(np, "aer");
>>>>>  > +		if (ret > 0) {
>>>>>  > +			irqs[PCIE_PORT_SERVICE_AER_SHIFT] = ret;
>>>>>  > +			if (dev->irq)
>>>>>  > +				irq = dev->irq;
>>>>>  > +			goto no_msi;
>>>>>  > +		}
>>>>>  > +	}
>>>>>  >
>>>>>  >  	/*
>>>>>  >  	 * If MSI cannot be used for PCIe PME or hotplug, we have to use
>>>>>  @@
>>>>>  > -224,11 +247,13 @@ static int init_service_irqs(struct pci_dev *dev,
>>>>>  int *irqs, int mask)
>>>>>  >  		irq = dev->irq;
>>>>>  >
>>>>>  >   no_msi:
>>>>>  > -	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
>>>>>  > -		irqs[i] = irq;
>>>>>  > +	for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++) {
>>>>>  > +		if (!irqs[i])
>>>>>  > +			irqs[i] = irq;
>>>>>  > +	}
>>>>>  >  	irqs[PCIE_PORT_SERVICE_VC_SHIFT] = -1;
>>>>>  >
>>>>>  > -	if (irq < 0)
>>>>>  > +	if (irq < 0 && irqs[PCIE_PORT_SERVICE_AER_SHIFT] < 0)
>>>>>  >  		return -ENODEV;
>>>>>  >  	return 0;
>>>>>  >  }
>>>>>  > --
>>>>>  > 2.1.0.27.g96db324
>>>>>  >
>>>>>  >
>>>>>  > _______________________________________________
>>>>>  > linux-arm-kernel mailing list
>>>>>  > linux-arm-kernel@lists.infradead.org
>>>>>  > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>> -- 
>> Murali Karicheri
>> Linux Kernel, Keystone


-- 
Murali Karicheri
Linux Kernel, Keystone

[toc] | [prev] | [next] | [standalone]

#1413632

FromBjorn Helgaas <helgaas@kernel.org>
Date2016-06-04 05:50 +0200
Message-ID<rGaKZ-3BY-3@gated-at.bofh.it>
In reply to#1413374
On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
> Po,
> 
> Sorry to hijack your discussion, but the problem seems to be same for
> Keystone PCI controller which is also designware (old version) based.
> 
> On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
> > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
> >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >>>>>  -----Original Message-----
> >>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >>>>>  Sent: Thursday, June 02, 2016 11:48 AM
> >>>>>  To: Po Liu
> >>>>>  Cc: linux-pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org;
> >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd Bergmann;
> >>>>>  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn
> >>>>>  Helgaas; Shawn Guo; Mingkai Hu; Rob Herring
> >>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >>>>>  MSI/MSI-X/INTx mode
> >>>>>  
> >>>>>  [+cc Rob]
> >>>>>  
> >>>>>  Hi Po,
> >>>>>  
> >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> >>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx in RC mode.
> >>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx mode,
> >>>>>  > maybe there is interrupt line for aer pme etc. Search the interrupt
> >>>>>  > number in the fdt file.
> >>>>>  
> >>>>>  My understanding is that AER interrupt signaling can be done via INTx,
> >>>>>  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).  Apparently your device
> >>>>>  doesn't support MSI or MSI-X.  Are you saying it doesn't support INTx
> >>>>>  either?  How is the interrupt you're requesting here different from INTx?
> >>>>
> >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the devices
> >>>> or root error in RC mode. But use an independent SPI interrupt(arm
> >>>> interrupt controller) line.  
> >>>
> >>> The Root Port is a PCI device and should follow the normal PCI rules
> >>> for interrupts.  As far as I understand, that means it should use MSI,
> >>> MSI-X, or INTx.  If your Root Port doesn't use MSI or MSI-X, it should
> >>> use INTx, the PCI_INTERRUPT_PIN register should tell us which (INTA/
> >>> INTB/etc.), and PCI_COMMAND_INTX_DISABLE should work to disable it.
> >>> That's all from the PCI point of view, of course.
> >>
> >> I am faced with the same issue on Keystone PCI hardware and it has been
> >> on my TODO list  for quite some time. Keystone PCI hardware also doesn't
> >> use MSI or MSI-X or INTx for reporting errors received at the root port,
> >> but use a platform interrupt instead (not complaint to PCI standard as
> >> per PCI base spec). So I would need similar change to have the error
> >> interrupt passed to the aer driver. So there are hardware out there
> >> like Keystone which requires to support this through platform IRQ.
> > 
> > This is not a new area of the spec, and it's hard for me to believe
> > that these two new PCIe controllers are both broken the same way
> > (although I guess both are DesignWare-based, so maybe this is the same
> > underlying problem in both cases?).  I think it's more likely that we
> > just haven't figured out the right way to describe this in the DT.
> 
> Keystone is using an older version of the designware IP and it implements
> all of the interrupts in the application register space unlike other
> newer version of the hardware. So I assume, the version used on Layerscape
> is also an older version and the both have same issue in terms of 
> non standard platform interrupt used for error reporting.
> 
> > I assume you have a Root Port with an AER capability, no MSI
> > capability, and no MSI-X capability, right? 
> 
> Has AER capability and both MSI and INTx (legacy) capability
> 
> > What does its Interrupt
> > Pin register contain?  If it's zero, it doesn't use INTx either, so
> > according to the spec it should generate no interrupts.
> > 
> At address offset 0x3C by default has a value of 1, but it is writable
> by software. So default is INTx A.

0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
*Pin* is at 0x3d and should be read-only.

Does your Keystone driver support MSI?  If so, since your Root Port
supports MSI, I would think we would use that by default, and the INTx
stuff wouldn't even matter.

> > But if Interrupt Pin is non-zero, that means the Root Port should be
> > able to generate virtual INTx interrupts.  Presumably the Root Complex
> > connects those interrupts to something; maybe to your platform
> > interrupt?
> 
> Probably that is what is happening. Both Power management and Error
> interrupts are raised on platform interrupt lines.
> 
> 12 Error Interrupts
> 
> [0] System error (OR of fatal, nonfatal, correctable errors) (RC mode only)
> [1] PCIe fatal error (RC mode only)
> [2] PCIe non-fatal error (RC mode only)
> [3] PCIe correctable error (RC mode only)
> [4] AXI Error due to fatal condition in AXI bridge (EP/RC modes)
> [5] PCIe advanced error (RC mode only)
> 
> 13 Power management and reset event interrupts
> 
> [0] Power management turn-off message interrupt (EP mode only)
> [1] Power management ack message interrupt (RC mode only)
> [2] Power management event interrupt (RC mode only)
> [3] Link request reset interrupt (hot reset or link down) (RC mode only)
>
> > PCI doesn't say anything about an interrupt after it leaves the Root
> > Complex, so the fact that it's connected to a "platform interrupt" or
> > "SPI interrupt" or "IOAPIC interrupt" doesn't make it non-compliant.
> > Shouldn't we be able to use the interrupt-map and related DT
> > properties to express the fact that Root Port virtual INTx is routed
> > to platform interrupt Y, even without any special-case code in
> > portdrv?
> 
> My understanding is if RC also raise interrupt on INTx, then below 
> map make sense, where either EP or RC can raise interrupt and the
> line will be muxed for interrupt from EP or RC port.

I'm sorry, I didn't quite catch your meaning here, so let me try to
clarify some terminology.  Maybe we'll eventually blunder into a
common understanding :)

INTx is a PCI concept and only means something in the PCI hierarchy.
The RC would *receive* virtual INTx interrupts from the PCI hierarchy
and turn them into some platform-specific interrupt (not INTx) on the
upstream side.

So strictly speaking, the RC might raise platform-specific interrupts
when it receives INTx interrupts, but it would not raise any INTx
interrupts itself.

> Here is the DT entry in PCIE keystone for Legacy interrupt mapping
> to host interrupt. 
>                         #interrupt-cells = <1>;
>                         interrupt-map-mask = <0 0 0 7>;
>                         interrupt-map = <0 0 0 1 &pcie_intc0 0>, /* INT A */
>                                         <0 0 0 2 &pcie_intc0 1>, /* INT B */
>                                         <0 0 0 3 &pcie_intc0 2>, /* INT C */
>                                         <0 0 0 4 &pcie_intc0 3>; /* INT D */

If I understand correctly, this is the mapping from the PCI world
(INTA, INTB, etc.) to the platform-specific world (pcie_intc0 0, etc.)

If a Root Port raises a virtual INTA, the RC should turn it into the
corresponding platform interrupt, i.e., GIC_SPI 48 from the
description below.

> And  then
> 
>                         pcie_intc0: legacy-interrupt-controller {
>                                 interrupt-controller;
>                                 #interrupt-cells = <1>;
>                                 interrupt-parent = <&gic>;
>                                 interrupts = <GIC_SPI 48 IRQ_TYPE_EDGE_RISING>,
>                                         <GIC_SPI 49 IRQ_TYPE_EDGE_RISING>,
>                                         <GIC_SPI 50 IRQ_TYPE_EDGE_RISING>,
>                                         <GIC_SPI 51 IRQ_TYPE_EDGE_RISING>;
>                         };
> 
> So if RC interrupt for Power management and Error interrupt is a separate
> line, how can software describe that using the above interrupt map?

I don't know anything about interrupts the RC might generate on its
own behalf.  A lot of the RC behavior is not specified by the PCIe
spec because the RC is on the border between the upstream
platform-specific stuff and the downstream PCIe stuff.  Is there
something in the PCIe spec about the power management and error
interrupts you're talking about?  Of maybe you can point me to a
section of the Keystone spec that talks about interrupts generated by
the RC?

In any event, the AER interrupts we're looking for in portdrv are from
the Root Port, not from the RC.  For INTx, my understanding is that
the RC *transforms* virtual INTx messages from the Root Port in the
PCIe domain into GIC transactions in the platform domain.

Bjorn

[toc] | [prev] | [next] | [standalone]

#1414709

FromPo Liu <po.liu@nxp.com>
Date2016-06-06 09:50 +0200
Message-ID<rGXsl-1T2-3@gated-at.bofh.it>
In reply to#1413632
Hi Bjorn,
I confirm we met same problem with KeyStone base on DesignWare design.


Best regards,
Liu Po

>  -----Original Message-----
>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>  Sent: Saturday, June 04, 2016 11:49 AM
>  To: Murali Karicheri
>  Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-
>  kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
>  devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;
>  Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn Guo;
>  Mingkai Hu; Rob Herring
>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
>  MSI/MSI-X/INTx mode
>  
>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
>  > Po,
>  >
>  > Sorry to hijack your discussion, but the problem seems to be same for
>  > Keystone PCI controller which is also designware (old version) based.
>  >
>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
>  > >>>>>  -----Original Message-----
>  > >>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>  > >>>>>  Sent: Thursday, June 02, 2016 11:48 AM
>  > >>>>>  To: Po Liu
>  > >>>>>  Cc: linux-pci@vger.kernel.org;
>  > >>>>> linux-arm-kernel@lists.infradead.org;
>  > >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd
>  > >>>>> Bergmann;  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li;
>  > >>>>> Minghuan Lian; Bjorn  Helgaas; Shawn Guo; Mingkai Hu; Rob
>  > >>>>> Herring
>  > >>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with
>  > >>>>> none  MSI/MSI-X/INTx mode
>  > >>>>>
>  > >>>>>  [+cc Rob]
>  > >>>>>
>  > >>>>>  Hi Po,
>  > >>>>>
>  > >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
>  > >>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx
>  in RC mode.
>  > >>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx
>  > >>>>> mode,  > maybe there is interrupt line for aer pme etc. Search
>  > >>>>> the interrupt  > number in the fdt file.
>  > >>>>>
>  > >>>>>  My understanding is that AER interrupt signaling can be done
>  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).
>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.  Are you
>  > >>>>> saying it doesn't support INTx  either?  How is the interrupt
>  you're requesting here different from INTx?
>  > >>>>
>  > >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the
>  > >>>> devices or root error in RC mode. But use an independent SPI
>  > >>>> interrupt(arm interrupt controller) line.
>  > >>>
>  > >>> The Root Port is a PCI device and should follow the normal PCI
>  > >>> rules for interrupts.  As far as I understand, that means it
>  > >>> should use MSI, MSI-X, or INTx.  If your Root Port doesn't use MSI
>  > >>> or MSI-X, it should use INTx, the PCI_INTERRUPT_PIN register
>  > >>> should tell us which (INTA/ INTB/etc.), and
>  PCI_COMMAND_INTX_DISABLE should work to disable it.
>  > >>> That's all from the PCI point of view, of course.
>  > >>
>  > >> I am faced with the same issue on Keystone PCI hardware and it has
>  > >> been on my TODO list  for quite some time. Keystone PCI hardware
>  > >> also doesn't use MSI or MSI-X or INTx for reporting errors received
>  > >> at the root port, but use a platform interrupt instead (not
>  > >> complaint to PCI standard as per PCI base spec). So I would need
>  > >> similar change to have the error interrupt passed to the aer
>  > >> driver. So there are hardware out there like Keystone which
>  requires to support this through platform IRQ.
>  > >
>  > > This is not a new area of the spec, and it's hard for me to believe
>  > > that these two new PCIe controllers are both broken the same way
>  > > (although I guess both are DesignWare-based, so maybe this is the
>  > > same underlying problem in both cases?).  I think it's more likely
>  > > that we just haven't figured out the right way to describe this in
>  the DT.
>  >
>  > Keystone is using an older version of the designware IP and it
>  > implements all of the interrupts in the application register space
>  > unlike other newer version of the hardware. So I assume, the version
>  > used on Layerscape is also an older version and the both have same
>  > issue in terms of non standard platform interrupt used for error
>  reporting.
>  >
>  > > I assume you have a Root Port with an AER capability, no MSI
>  > > capability, and no MSI-X capability, right?
>  >
>  > Has AER capability and both MSI and INTx (legacy) capability
>  >
>  > > What does its Interrupt
>  > > Pin register contain?  If it's zero, it doesn't use INTx either, so
>  > > according to the spec it should generate no interrupts.
>  > >
>  > At address offset 0x3C by default has a value of 1, but it is writable
>  > by software. So default is INTx A.
>  
>  0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
>  *Pin* is at 0x3d and should be read-only.
>  
>  Does your Keystone driver support MSI?  If so, since your Root Port
>  supports MSI, I would think we would use that by default, and the INTx
>  stuff wouldn't even matter.

Layerscape is also shows "Both message signaled interrupts (MSI) and legacy INTx are supported."
But both of them not work for AER interrupt when devices or root port report aer error.
But another GIC interrupt line do.

>  
>  > > But if Interrupt Pin is non-zero, that means the Root Port should be
>  > > able to generate virtual INTx interrupts.  Presumably the Root
>  > > Complex connects those interrupts to something; maybe to your
>  > > platform interrupt?
>  >
>  > Probably that is what is happening. Both Power management and Error
>  > interrupts are raised on platform interrupt lines.
>  >
>  > 12 Error Interrupts
>  >
>  > [0] System error (OR of fatal, nonfatal, correctable errors) (RC mode
>  > only) [1] PCIe fatal error (RC mode only) [2] PCIe non-fatal error (RC
>  > mode only) [3] PCIe correctable error (RC mode only) [4] AXI Error due
>  > to fatal condition in AXI bridge (EP/RC modes) [5] PCIe advanced error
>  > (RC mode only)
>  >
>  > 13 Power management and reset event interrupts
>  >
>  > [0] Power management turn-off message interrupt (EP mode only) [1]
>  > Power management ack message interrupt (RC mode only) [2] Power
>  > management event interrupt (RC mode only) [3] Link request reset
>  > interrupt (hot reset or link down) (RC mode only)
>  >
>  > > PCI doesn't say anything about an interrupt after it leaves the Root
>  > > Complex, so the fact that it's connected to a "platform interrupt"
>  > > or "SPI interrupt" or "IOAPIC interrupt" doesn't make it non-
>  compliant.
>  > > Shouldn't we be able to use the interrupt-map and related DT
>  > > properties to express the fact that Root Port virtual INTx is routed
>  > > to platform interrupt Y, even without any special-case code in
>  > > portdrv?
>  >
>  > My understanding is if RC also raise interrupt on INTx, then below map
>  > make sense, where either EP or RC can raise interrupt and the line
>  > will be muxed for interrupt from EP or RC port.
>  
>  I'm sorry, I didn't quite catch your meaning here, so let me try to
>  clarify some terminology.  Maybe we'll eventually blunder into a common
>  understanding :)
>  
>  INTx is a PCI concept and only means something in the PCI hierarchy.
>  The RC would *receive* virtual INTx interrupts from the PCI hierarchy
>  and turn them into some platform-specific interrupt (not INTx) on the
>  upstream side.
>  
>  So strictly speaking, the RC might raise platform-specific interrupts
>  when it receives INTx interrupts, but it would not raise any INTx
>  interrupts itself.
>  
>  > Here is the DT entry in PCIE keystone for Legacy interrupt mapping to
>  > host interrupt.
>  >                         #interrupt-cells = <1>;
>  >                         interrupt-map-mask = <0 0 0 7>;
>  >                         interrupt-map = <0 0 0 1 &pcie_intc0 0>, /*
>  INT A */
>  >                                         <0 0 0 2 &pcie_intc0 1>, /*
>  INT B */
>  >                                         <0 0 0 3 &pcie_intc0 2>, /*
>  INT C */
>  >                                         <0 0 0 4 &pcie_intc0 3>; /*
>  > INT D */
>  
>  If I understand correctly, this is the mapping from the PCI world (INTA,
>  INTB, etc.) to the platform-specific world (pcie_intc0 0, etc.)
>  
>  If a Root Port raises a virtual INTA, the RC should turn it into the
>  corresponding platform interrupt, i.e., GIC_SPI 48 from the description
>  below.
>  
>  > And  then
>  >
>  >                         pcie_intc0: legacy-interrupt-controller {
>  >                                 interrupt-controller;
>  >                                 #interrupt-cells = <1>;
>  >                                 interrupt-parent = <&gic>;
>  >                                 interrupts = <GIC_SPI 48
>  IRQ_TYPE_EDGE_RISING>,
>  >                                         <GIC_SPI 49
>  IRQ_TYPE_EDGE_RISING>,
>  >                                         <GIC_SPI 50
>  IRQ_TYPE_EDGE_RISING>,
>  >                                         <GIC_SPI 51
>  IRQ_TYPE_EDGE_RISING>;
>  >                         };
>  >
>  > So if RC interrupt for Power management and Error interrupt is a
>  > separate line, how can software describe that using the above
>  interrupt map?
>  
>  I don't know anything about interrupts the RC might generate on its own
>  behalf.  A lot of the RC behavior is not specified by the PCIe spec
>  because the RC is on the border between the upstream platform-specific
>  stuff and the downstream PCIe stuff.  Is there something in the PCIe
>  spec about the power management and error interrupts you're talking
>  about?  Of maybe you can point me to a section of the Keystone spec that
>  talks about interrupts generated by the RC?

Below is one of the PCIE controller interrupts list: 

142 PEX1 INTA 
143 PEX1 INTB 
144 PEX1 INTC 
145 PEX1 INTD 
146-147 Reserved
148 PEX1 MSI 
149 PEX1 PME 
150 PEX1 CFG err interrupt

Only the "150 PEX1 CFG err interrupt" routing to the aer interrupt. 


>  
>  In any event, the AER interrupts we're looking for in portdrv are from
>  the Root Port, not from the RC.  For INTx, my understanding is that the
>  RC *transforms* virtual INTx messages from the Root Port in the PCIe
>  domain into GIC transactions in the platform domain.
>  
>  Bjorn

[toc] | [prev] | [next] | [standalone]

#1415067

FromMurali Karicheri <m-karicheri2@ti.com>
Date2016-06-06 16:10 +0200
Message-ID<rH3o7-5Yv-49@gated-at.bofh.it>
In reply to#1414709
On 06/06/2016 03:32 AM, Po Liu wrote:
> Hi Bjorn,
> I confirm we met same problem with KeyStone base on DesignWare design.
> 
> 
> Best regards,
> Liu Po
> 
>>  -----Original Message-----
>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>>  Sent: Saturday, June 04, 2016 11:49 AM
>>  To: Murali Karicheri
>>  Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-
>>  kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
>>  devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;
>>  Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn Guo;
>>  Mingkai Hu; Rob Herring
>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
>>  MSI/MSI-X/INTx mode
>>  
>>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
>>  > Po,
>>  >
>>  > Sorry to hijack your discussion, but the problem seems to be same for
>>  > Keystone PCI controller which is also designware (old version) based.
>>  >
>>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
>>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
>>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
>>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
>>  > >>>>>  -----Original Message-----
>>  > >>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>>  > >>>>>  Sent: Thursday, June 02, 2016 11:48 AM
>>  > >>>>>  To: Po Liu
>>  > >>>>>  Cc: linux-pci@vger.kernel.org;
>>  > >>>>> linux-arm-kernel@lists.infradead.org;
>>  > >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd
>>  > >>>>> Bergmann;  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li;
>>  > >>>>> Minghuan Lian; Bjorn  Helgaas; Shawn Guo; Mingkai Hu; Rob
>>  > >>>>> Herring
>>  > >>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with
>>  > >>>>> none  MSI/MSI-X/INTx mode
>>  > >>>>>
>>  > >>>>>  [+cc Rob]
>>  > >>>>>
>>  > >>>>>  Hi Po,
>>  > >>>>>
>>  > >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
>>  > >>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx
>>  in RC mode.
>>  > >>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx
>>  > >>>>> mode,  > maybe there is interrupt line for aer pme etc. Search
>>  > >>>>> the interrupt  > number in the fdt file.
>>  > >>>>>
>>  > >>>>>  My understanding is that AER interrupt signaling can be done
>>  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).
>>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.  Are you
>>  > >>>>> saying it doesn't support INTx  either?  How is the interrupt
>>  you're requesting here different from INTx?
>>  > >>>>
>>  > >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the
>>  > >>>> devices or root error in RC mode. But use an independent SPI
>>  > >>>> interrupt(arm interrupt controller) line.
>>  > >>>
>>  > >>> The Root Port is a PCI device and should follow the normal PCI
>>  > >>> rules for interrupts.  As far as I understand, that means it
>>  > >>> should use MSI, MSI-X, or INTx.  If your Root Port doesn't use MSI
>>  > >>> or MSI-X, it should use INTx, the PCI_INTERRUPT_PIN register
>>  > >>> should tell us which (INTA/ INTB/etc.), and
>>  PCI_COMMAND_INTX_DISABLE should work to disable it.
>>  > >>> That's all from the PCI point of view, of course.
>>  > >>
>>  > >> I am faced with the same issue on Keystone PCI hardware and it has
>>  > >> been on my TODO list  for quite some time. Keystone PCI hardware
>>  > >> also doesn't use MSI or MSI-X or INTx for reporting errors received
>>  > >> at the root port, but use a platform interrupt instead (not
>>  > >> complaint to PCI standard as per PCI base spec). So I would need
>>  > >> similar change to have the error interrupt passed to the aer
>>  > >> driver. So there are hardware out there like Keystone which
>>  requires to support this through platform IRQ.
>>  > >
>>  > > This is not a new area of the spec, and it's hard for me to believe
>>  > > that these two new PCIe controllers are both broken the same way
>>  > > (although I guess both are DesignWare-based, so maybe this is the
>>  > > same underlying problem in both cases?).  I think it's more likely
>>  > > that we just haven't figured out the right way to describe this in
>>  the DT.
>>  >
>>  > Keystone is using an older version of the designware IP and it
>>  > implements all of the interrupts in the application register space
>>  > unlike other newer version of the hardware. So I assume, the version
>>  > used on Layerscape is also an older version and the both have same
>>  > issue in terms of non standard platform interrupt used for error
>>  reporting.
>>  >
>>  > > I assume you have a Root Port with an AER capability, no MSI
>>  > > capability, and no MSI-X capability, right?
>>  >
>>  > Has AER capability and both MSI and INTx (legacy) capability
>>  >
>>  > > What does its Interrupt
>>  > > Pin register contain?  If it's zero, it doesn't use INTx either, so
>>  > > according to the spec it should generate no interrupts.
>>  > >
>>  > At address offset 0x3C by default has a value of 1, but it is writable
>>  > by software. So default is INTx A.
>>  
>>  0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
>>  *Pin* is at 0x3d and should be read-only.
>>  

You are right. But default is 1 at this address.

>>  Does your Keystone driver support MSI?  If so, since your Root Port
>>  supports MSI, I would think we would use that by default, and the INTx
>>  stuff wouldn't even matter.
> 
> Layerscape is also shows "Both message signaled interrupts (MSI) and legacy INTx are supported."
> But both of them not work for AER interrupt when devices or root port report aer error.
> But another GIC interrupt line do.

Same with Keystone. Even though both MSI and INTx are supported
error interrupt at root port is reported on a different interrupt
line than MSI/INTx. So for Power Management event interrupt is also
different line.

Murali
> 
>>  
>>  > > But if Interrupt Pin is non-zero, that means the Root Port should be
>>  > > able to generate virtual INTx interrupts.  Presumably the Root
>>  > > Complex connects those interrupts to something; maybe to your
>>  > > platform interrupt?
>>  >
>>  > Probably that is what is happening. Both Power management and Error
>>  > interrupts are raised on platform interrupt lines.
>>  >
>>  > 12 Error Interrupts
>>  >
>>  > [0] System error (OR of fatal, nonfatal, correctable errors) (RC mode
>>  > only) [1] PCIe fatal error (RC mode only) [2] PCIe non-fatal error (RC
>>  > mode only) [3] PCIe correctable error (RC mode only) [4] AXI Error due
>>  > to fatal condition in AXI bridge (EP/RC modes) [5] PCIe advanced error
>>  > (RC mode only)
>>  >
>>  > 13 Power management and reset event interrupts
>>  >
>>  > [0] Power management turn-off message interrupt (EP mode only) [1]
>>  > Power management ack message interrupt (RC mode only) [2] Power
>>  > management event interrupt (RC mode only) [3] Link request reset
>>  > interrupt (hot reset or link down) (RC mode only)
>>  >
>>  > > PCI doesn't say anything about an interrupt after it leaves the Root
>>  > > Complex, so the fact that it's connected to a "platform interrupt"
>>  > > or "SPI interrupt" or "IOAPIC interrupt" doesn't make it non-
>>  compliant.
>>  > > Shouldn't we be able to use the interrupt-map and related DT
>>  > > properties to express the fact that Root Port virtual INTx is routed
>>  > > to platform interrupt Y, even without any special-case code in
>>  > > portdrv?
>>  >
>>  > My understanding is if RC also raise interrupt on INTx, then below map
>>  > make sense, where either EP or RC can raise interrupt and the line
>>  > will be muxed for interrupt from EP or RC port.
>>  
>>  I'm sorry, I didn't quite catch your meaning here, so let me try to
>>  clarify some terminology.  Maybe we'll eventually blunder into a common
>>  understanding :)
>>  
>>  INTx is a PCI concept and only means something in the PCI hierarchy.
>>  The RC would *receive* virtual INTx interrupts from the PCI hierarchy
>>  and turn them into some platform-specific interrupt (not INTx) on the
>>  upstream side.
>>  
>>  So strictly speaking, the RC might raise platform-specific interrupts
>>  when it receives INTx interrupts, but it would not raise any INTx
>>  interrupts itself.
>>  
>>  > Here is the DT entry in PCIE keystone for Legacy interrupt mapping to
>>  > host interrupt.
>>  >                         #interrupt-cells = <1>;
>>  >                         interrupt-map-mask = <0 0 0 7>;
>>  >                         interrupt-map = <0 0 0 1 &pcie_intc0 0>, /*
>>  INT A */
>>  >                                         <0 0 0 2 &pcie_intc0 1>, /*
>>  INT B */
>>  >                                         <0 0 0 3 &pcie_intc0 2>, /*
>>  INT C */
>>  >                                         <0 0 0 4 &pcie_intc0 3>; /*
>>  > INT D */
>>  
>>  If I understand correctly, this is the mapping from the PCI world (INTA,
>>  INTB, etc.) to the platform-specific world (pcie_intc0 0, etc.)
>>  
>>  If a Root Port raises a virtual INTA, the RC should turn it into the
>>  corresponding platform interrupt, i.e., GIC_SPI 48 from the description
>>  below.
>>  
>>  > And  then
>>  >
>>  >                         pcie_intc0: legacy-interrupt-controller {
>>  >                                 interrupt-controller;
>>  >                                 #interrupt-cells = <1>;
>>  >                                 interrupt-parent = <&gic>;
>>  >                                 interrupts = <GIC_SPI 48
>>  IRQ_TYPE_EDGE_RISING>,
>>  >                                         <GIC_SPI 49
>>  IRQ_TYPE_EDGE_RISING>,
>>  >                                         <GIC_SPI 50
>>  IRQ_TYPE_EDGE_RISING>,
>>  >                                         <GIC_SPI 51
>>  IRQ_TYPE_EDGE_RISING>;
>>  >                         };
>>  >
>>  > So if RC interrupt for Power management and Error interrupt is a
>>  > separate line, how can software describe that using the above
>>  interrupt map?
>>  
>>  I don't know anything about interrupts the RC might generate on its own
>>  behalf.  A lot of the RC behavior is not specified by the PCIe spec
>>  because the RC is on the border between the upstream platform-specific
>>  stuff and the downstream PCIe stuff.  Is there something in the PCIe
>>  spec about the power management and error interrupts you're talking
>>  about?  Of maybe you can point me to a section of the Keystone spec that
>>  talks about interrupts generated by the RC?
> 
> Below is one of the PCIE controller interrupts list: 
> 
> 142 PEX1 INTA 
> 143 PEX1 INTB 
> 144 PEX1 INTC 
> 145 PEX1 INTD 
> 146-147 Reserved
> 148 PEX1 MSI 
> 149 PEX1 PME 
> 150 PEX1 CFG err interrupt
> 
> Only the "150 PEX1 CFG err interrupt" routing to the aer interrupt. 
> 
> 
>>  
>>  In any event, the AER interrupts we're looking for in portdrv are from
>>  the Root Port, not from the RC.  For INTx, my understanding is that the
>>  RC *transforms* virtual INTx messages from the Root Port in the PCIe
>>  domain into GIC transactions in the platform domain.
>>  
>>  Bjorn


-- 
Murali Karicheri
Linux Kernel, Keystone

[toc] | [prev] | [next] | [standalone]

#1415350

FromBjorn Helgaas <helgaas@kernel.org>
Date2016-06-06 20:20 +0200
Message-ID<rH7i1-8rx-15@gated-at.bofh.it>
In reply to#1415067
On Mon, Jun 06, 2016 at 10:01:44AM -0400, Murali Karicheri wrote:
> On 06/06/2016 03:32 AM, Po Liu wrote:
> > Hi Bjorn,
> > I confirm we met same problem with KeyStone base on DesignWare design.
> > 
> > 
> > Best regards,
> > Liu Po
> > 
> >>  -----Original Message-----
> >>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >>  Sent: Saturday, June 04, 2016 11:49 AM
> >>  To: Murali Karicheri
> >>  Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-
> >>  kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
> >>  devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;
> >>  Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn Guo;
> >>  Mingkai Hu; Rob Herring
> >>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >>  MSI/MSI-X/INTx mode
> >>  
> >>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
> >>  > Po,
> >>  >
> >>  > Sorry to hijack your discussion, but the problem seems to be same for
> >>  > Keystone PCI controller which is also designware (old version) based.
> >>  >
> >>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
> >>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri wrote:
> >>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> >>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >>  > >>>>>  -----Original Message-----
> >>  > >>>>>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >>  > >>>>>  Sent: Thursday, June 02, 2016 11:48 AM
> >>  > >>>>>  To: Po Liu
> >>  > >>>>>  Cc: linux-pci@vger.kernel.org;
> >>  > >>>>> linux-arm-kernel@lists.infradead.org;
> >>  > >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org; Arnd
> >>  > >>>>> Bergmann;  Roy Zang; Marc Zyngier; Stuart Yoder; Yang-Leo Li;
> >>  > >>>>> Minghuan Lian; Bjorn  Helgaas; Shawn Guo; Mingkai Hu; Rob
> >>  > >>>>> Herring
> >>  > >>>>>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with
> >>  > >>>>> none  MSI/MSI-X/INTx mode
> >>  > >>>>>
> >>  > >>>>>  [+cc Rob]
> >>  > >>>>>
> >>  > >>>>>  Hi Po,
> >>  > >>>>>
> >>  > >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu wrote:
> >>  > >>>>>  > On some platforms, root port doesn't support MSI/MSI-X/INTx
> >>  in RC mode.
> >>  > >>>>>  > When chip support the aer interrupt with none MSI/MSI-X/INTx
> >>  > >>>>> mode,  > maybe there is interrupt line for aer pme etc. Search
> >>  > >>>>> the interrupt  > number in the fdt file.
> >>  > >>>>>
> >>  > >>>>>  My understanding is that AER interrupt signaling can be done
> >>  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec 6.2.4.1.2).
> >>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.  Are you
> >>  > >>>>> saying it doesn't support INTx  either?  How is the interrupt
> >>  you're requesting here different from INTx?
> >>  > >>>>
> >>  > >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the
> >>  > >>>> devices or root error in RC mode. But use an independent SPI
> >>  > >>>> interrupt(arm interrupt controller) line.
> >>  > >>>
> >>  > >>> The Root Port is a PCI device and should follow the normal PCI
> >>  > >>> rules for interrupts.  As far as I understand, that means it
> >>  > >>> should use MSI, MSI-X, or INTx.  If your Root Port doesn't use MSI
> >>  > >>> or MSI-X, it should use INTx, the PCI_INTERRUPT_PIN register
> >>  > >>> should tell us which (INTA/ INTB/etc.), and
> >>  PCI_COMMAND_INTX_DISABLE should work to disable it.
> >>  > >>> That's all from the PCI point of view, of course.
> >>  > >>
> >>  > >> I am faced with the same issue on Keystone PCI hardware and it has
> >>  > >> been on my TODO list  for quite some time. Keystone PCI hardware
> >>  > >> also doesn't use MSI or MSI-X or INTx for reporting errors received
> >>  > >> at the root port, but use a platform interrupt instead (not
> >>  > >> complaint to PCI standard as per PCI base spec). So I would need
> >>  > >> similar change to have the error interrupt passed to the aer
> >>  > >> driver. So there are hardware out there like Keystone which
> >>  requires to support this through platform IRQ.
> >>  > >
> >>  > > This is not a new area of the spec, and it's hard for me to believe
> >>  > > that these two new PCIe controllers are both broken the same way
> >>  > > (although I guess both are DesignWare-based, so maybe this is the
> >>  > > same underlying problem in both cases?).  I think it's more likely
> >>  > > that we just haven't figured out the right way to describe this in
> >>  the DT.
> >>  >
> >>  > Keystone is using an older version of the designware IP and it
> >>  > implements all of the interrupts in the application register space
> >>  > unlike other newer version of the hardware. So I assume, the version
> >>  > used on Layerscape is also an older version and the both have same
> >>  > issue in terms of non standard platform interrupt used for error
> >>  reporting.
> >>  >
> >>  > > I assume you have a Root Port with an AER capability, no MSI
> >>  > > capability, and no MSI-X capability, right?
> >>  >
> >>  > Has AER capability and both MSI and INTx (legacy) capability
> >>  >
> >>  > > What does its Interrupt
> >>  > > Pin register contain?  If it's zero, it doesn't use INTx either, so
> >>  > > according to the spec it should generate no interrupts.
> >>  > >
> >>  > At address offset 0x3C by default has a value of 1, but it is writable
> >>  > by software. So default is INTx A.
> >>  
> >>  0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
> >>  *Pin* is at 0x3d and should be read-only.
> >>  
> 
> You are right. But default is 1 at this address.
> 
> >>  Does your Keystone driver support MSI?  If so, since your Root Port
> >>  supports MSI, I would think we would use that by default, and the INTx
> >>  stuff wouldn't even matter.
> > 
> > Layerscape is also shows "Both message signaled interrupts (MSI) and legacy INTx are supported."
> > But both of them not work for AER interrupt when devices or root port report aer error.
> > But another GIC interrupt line do.
> 
> Same with Keystone. Even though both MSI and INTx are supported
> error interrupt at root port is reported on a different interrupt
> line than MSI/INTx. So for Power Management event interrupt is also
> different line.

I'm looking at the "Error Message Controls" diagram in the PCIe spec
r3.0, sec 6.2.6.  Does this hardware fit into the platform-specific
"System Error" case there?  Do the Root Control enable bits (in the
PCIe Capability) control this interrupt?  If so, maybe this makes more
sense than I thought.

We have to assume both notification paths work (the normal
INTx/MSI/MSI-X AER interrupts as well as the platform-specific System
Errors), so if we add support for System Errors, it should be
structured so we prefer INTx/MSI/MSI-X, and only fall back to System
Errors if they don't work.  AER would have to know which path it's
using so aer_enable_rootport() can disable the other one (currently it
always disables System Errors).

Then you'd have to add quirks to mark MSI/MSI-X/INTx as being broken
on your devices to force the fallback to System Errors.

How much would this screw up other PCIe services (PME, hotplug, VC,
etc)?  Does MSI work for them?

Bjorn

[toc] | [prev] | [next] | [standalone]

#1415970

FromPo Liu <po.liu@nxp.com>
Date2016-06-07 12:30 +0200
Message-ID<rHmqK-1so-31@gated-at.bofh.it>
In reply to#1415350
Hi Bjorn,

>  -----Original Message-----
>  
>  On Mon, Jun 06, 2016 at 10:01:44AM -0400, Murali Karicheri wrote:
>  > On 06/06/2016 03:32 AM, Po Liu wrote:
>  > > Hi Bjorn,
>  > > I confirm we met same problem with KeyStone base on DesignWare
>  design.
>  > >
>  > >
>  > > Best regards,
>  > > Liu Po
>  > >
>  > >>  -----Original Message-----
>  > >>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>  > >>  Sent: Saturday, June 04, 2016 11:49 AM
>  > >>  To: Murali Karicheri
>  > >>  Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-
>  > >> kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
>  > >> devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;
>  > >> Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn Guo;
>  > >> Mingkai Hu; Rob Herring
>  > >>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
>  > >> MSI/MSI-X/INTx mode
>  > >>
>  > >>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
>  > >>  > Po,
>  > >>  >
>  > >>  > Sorry to hijack your discussion, but the problem seems to be
>  > >> same for  > Keystone PCI controller which is also designware (old
>  version) based.
>  > >>  >
>  > >>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
>  > >>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri
>  wrote:
>  > >>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
>  > >>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
>  > >>  > >>>>>  -----Original Message-----  > >>>>>  From: Bjorn Helgaas
>  > >> [mailto:helgaas@kernel.org]  > >>>>>  Sent: Thursday, June 02, 2016
>  > >> 11:48 AM  > >>>>>  To: Po Liu  > >>>>>  Cc:
>  > >> linux-pci@vger.kernel.org;  > >>>>>
>  > >> linux-arm-kernel@lists.infradead.org;
>  > >>  > >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org;
>  > >> Arnd  > >>>>> Bergmann;  Roy Zang; Marc Zyngier; Stuart Yoder;
>  > >> Yang-Leo Li;  > >>>>> Minghuan Lian; Bjorn  Helgaas; Shawn Guo;
>  > >> Mingkai Hu; Rob  > >>>>> Herring  > >>>>>  Subject: Re: [PATCH 2/2]
>  > >> aer: add support aer interrupt with  > >>>>> none  MSI/MSI-X/INTx
>  > >> mode  > >>>>>  > >>>>>  [+cc Rob]  > >>>>>  > >>>>>  Hi Po,  >
>  > >> >>>>>  > >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu
>  > >> wrote:
>  > >>  > >>>>>  > On some platforms, root port doesn't support
>  > >> MSI/MSI-X/INTx  in RC mode.
>  > >>  > >>>>>  > When chip support the aer interrupt with none
>  > >> MSI/MSI-X/INTx  > >>>>> mode,  > maybe there is interrupt line for
>  > >> aer pme etc. Search  > >>>>> the interrupt  > number in the fdt
>  file.
>  > >>  > >>>>>
>  > >>  > >>>>>  My understanding is that AER interrupt signaling can be
>  > >> done  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec
>  6.2.4.1.2).
>  > >>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.  Are
>  > >> you  > >>>>> saying it doesn't support INTx  either?  How is the
>  > >> interrupt  you're requesting here different from INTx?
>  > >>  > >>>>
>  > >>  > >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the
>  > >> > >>>> devices or root error in RC mode. But use an independent SPI
>  > >> > >>>> interrupt(arm interrupt controller) line.
>  > >>  > >>>
>  > >>  > >>> The Root Port is a PCI device and should follow the normal
>  > >> PCI  > >>> rules for interrupts.  As far as I understand, that
>  > >> means it  > >>> should use MSI, MSI-X, or INTx.  If your Root Port
>  > >> doesn't use MSI  > >>> or MSI-X, it should use INTx, the
>  > >> PCI_INTERRUPT_PIN register  > >>> should tell us which (INTA/
>  > >> INTB/etc.), and  PCI_COMMAND_INTX_DISABLE should work to disable it.
>  > >>  > >>> That's all from the PCI point of view, of course.
>  > >>  > >>
>  > >>  > >> I am faced with the same issue on Keystone PCI hardware and
>  > >> it has  > >> been on my TODO list  for quite some time. Keystone
>  > >> PCI hardware  > >> also doesn't use MSI or MSI-X or INTx for
>  > >> reporting errors received  > >> at the root port, but use a
>  > >> platform interrupt instead (not  > >> complaint to PCI standard as
>  > >> per PCI base spec). So I would need  > >> similar change to have
>  > >> the error interrupt passed to the aer  > >> driver. So there are
>  > >> hardware out there like Keystone which  requires to support this
>  through platform IRQ.
>  > >>  > >
>  > >>  > > This is not a new area of the spec, and it's hard for me to
>  > >> believe  > > that these two new PCIe controllers are both broken
>  > >> the same way  > > (although I guess both are DesignWare-based, so
>  > >> maybe this is the  > > same underlying problem in both cases?).  I
>  > >> think it's more likely  > > that we just haven't figured out the
>  > >> right way to describe this in  the DT.
>  > >>  >
>  > >>  > Keystone is using an older version of the designware IP and it
>  > >> > implements all of the interrupts in the application register
>  > >> space  > unlike other newer version of the hardware. So I assume,
>  > >> the version  > used on Layerscape is also an older version and the
>  > >> both have same  > issue in terms of non standard platform interrupt
>  > >> used for error  reporting.
>  > >>  >
>  > >>  > > I assume you have a Root Port with an AER capability, no MSI
>  > >> > > capability, and no MSI-X capability, right?
>  > >>  >
>  > >>  > Has AER capability and both MSI and INTx (legacy) capability  >
>  > >> > > What does its Interrupt  > > Pin register contain?  If it's
>  > >> zero, it doesn't use INTx either, so  > > according to the spec it
>  > >> should generate no interrupts.
>  > >>  > >
>  > >>  > At address offset 0x3C by default has a value of 1, but it is
>  > >> writable  > by software. So default is INTx A.
>  > >>
>  > >>  0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
>  > >>  *Pin* is at 0x3d and should be read-only.
>  > >>
>  >
>  > You are right. But default is 1 at this address.
>  >
>  > >>  Does your Keystone driver support MSI?  If so, since your Root
>  > >> Port  supports MSI, I would think we would use that by default, and
>  > >> the INTx  stuff wouldn't even matter.
>  > >
>  > > Layerscape is also shows "Both message signaled interrupts (MSI) and
>  legacy INTx are supported."
>  > > But both of them not work for AER interrupt when devices or root
>  port report aer error.
>  > > But another GIC interrupt line do.
>  >
>  > Same with Keystone. Even though both MSI and INTx are supported error
>  > interrupt at root port is reported on a different interrupt line than
>  > MSI/INTx. So for Power Management event interrupt is also different
>  > line.
>  
>  I'm looking at the "Error Message Controls" diagram in the PCIe spec
>  r3.0, sec 6.2.6.  Does this hardware fit into the platform-specific
>  "System Error" case there?  Do the Root Control enable bits (in the PCIe
>  Capability) control this interrupt?  If so, maybe this makes more sense
>  than I thought.

It supposedly not the "System Error" case. But "the Error Interrupt" case.
Which means " Root Error Command register " could control the interrupt
 line we have now. (refer PCIe spec r3.0, sec 6.2.6)

May this kind of hardware design route broken the spec? 

>  
>  We have to assume both notification paths work (the normal INTx/MSI/MSI-
>  X AER interrupts as well as the platform-specific System Errors), so if
>  we add support for System Errors, it should be structured so we prefer
>  INTx/MSI/MSI-X, and only fall back to System Errors if they don't work.
>  AER would have to know which path it's using so aer_enable_rootport()
>  can disable the other one (currently it always disables System Errors).
>  
>  Then you'd have to add quirks to mark MSI/MSI-X/INTx as being broken on
>  your devices to force the fallback to System Errors.
>  
>  How much would this screw up other PCIe services (PME, hotplug, VC, etc)?
>  Does MSI work for them?

PME also like the AER. Hotplug is not supported. Others not known.
Po Liu

>  
>  Bjorn

[toc] | [prev] | [next] | [standalone]

#1416692

FromBjorn Helgaas <helgaas@kernel.org>
Date2016-06-08 00:50 +0200
Message-ID<rHxYR-aA-9@gated-at.bofh.it>
In reply to#1415970
On Tue, Jun 07, 2016 at 10:07:40AM +0000, Po Liu wrote:
> Hi Bjorn,
> 
> >  -----Original Message-----
> >  
> >  On Mon, Jun 06, 2016 at 10:01:44AM -0400, Murali Karicheri wrote:
> >  > On 06/06/2016 03:32 AM, Po Liu wrote:
> >  > > Hi Bjorn,
> >  > > I confirm we met same problem with KeyStone base on DesignWare
> >  design.
> >  > >
> >  > >
> >  > > Best regards,
> >  > > Liu Po
> >  > >
> >  > >>  -----Original Message-----
> >  > >>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> >  > >>  Sent: Saturday, June 04, 2016 11:49 AM
> >  > >>  To: Murali Karicheri
> >  > >>  Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-
> >  > >> kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
> >  > >> devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;
> >  > >> Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn Guo;
> >  > >> Mingkai Hu; Rob Herring
> >  > >>  Subject: Re: [PATCH 2/2] aer: add support aer interrupt with none
> >  > >> MSI/MSI-X/INTx mode
> >  > >>
> >  > >>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali Karicheri wrote:
> >  > >>  > Po,
> >  > >>  >
> >  > >>  > Sorry to hijack your discussion, but the problem seems to be
> >  > >> same for  > Keystone PCI controller which is also designware (old
> >  version) based.
> >  > >>  >
> >  > >>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
> >  > >>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali Karicheri
> >  wrote:
> >  > >>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
> >  > >>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
> >  > >>  > >>>>>  -----Original Message-----  > >>>>>  From: Bjorn Helgaas
> >  > >> [mailto:helgaas@kernel.org]  > >>>>>  Sent: Thursday, June 02, 2016
> >  > >> 11:48 AM  > >>>>>  To: Po Liu  > >>>>>  Cc:
> >  > >> linux-pci@vger.kernel.org;  > >>>>>
> >  > >> linux-arm-kernel@lists.infradead.org;
> >  > >>  > >>>>>  linux-kernel@vger.kernel.org; devicetree@vger.kernel.org;
> >  > >> Arnd  > >>>>> Bergmann;  Roy Zang; Marc Zyngier; Stuart Yoder;
> >  > >> Yang-Leo Li;  > >>>>> Minghuan Lian; Bjorn  Helgaas; Shawn Guo;
> >  > >> Mingkai Hu; Rob  > >>>>> Herring  > >>>>>  Subject: Re: [PATCH 2/2]
> >  > >> aer: add support aer interrupt with  > >>>>> none  MSI/MSI-X/INTx
> >  > >> mode  > >>>>>  > >>>>>  [+cc Rob]  > >>>>>  > >>>>>  Hi Po,  >
> >  > >> >>>>>  > >>>>>  On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu
> >  > >> wrote:
> >  > >>  > >>>>>  > On some platforms, root port doesn't support
> >  > >> MSI/MSI-X/INTx  in RC mode.
> >  > >>  > >>>>>  > When chip support the aer interrupt with none
> >  > >> MSI/MSI-X/INTx  > >>>>> mode,  > maybe there is interrupt line for
> >  > >> aer pme etc. Search  > >>>>> the interrupt  > number in the fdt
> >  file.
> >  > >>  > >>>>>
> >  > >>  > >>>>>  My understanding is that AER interrupt signaling can be
> >  > >> done  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec
> >  6.2.4.1.2).
> >  > >>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.  Are
> >  > >> you  > >>>>> saying it doesn't support INTx  either?  How is the
> >  > >> interrupt  you're requesting here different from INTx?
> >  > >>  > >>>>
> >  > >>  > >>>> Layerscape use none of MSI or MSI-X or INTx to indicate the
> >  > >> > >>>> devices or root error in RC mode. But use an independent SPI
> >  > >> > >>>> interrupt(arm interrupt controller) line.
> >  > >>  > >>>
> >  > >>  > >>> The Root Port is a PCI device and should follow the normal
> >  > >> PCI  > >>> rules for interrupts.  As far as I understand, that
> >  > >> means it  > >>> should use MSI, MSI-X, or INTx.  If your Root Port
> >  > >> doesn't use MSI  > >>> or MSI-X, it should use INTx, the
> >  > >> PCI_INTERRUPT_PIN register  > >>> should tell us which (INTA/
> >  > >> INTB/etc.), and  PCI_COMMAND_INTX_DISABLE should work to disable it.
> >  > >>  > >>> That's all from the PCI point of view, of course.
> >  > >>  > >>
> >  > >>  > >> I am faced with the same issue on Keystone PCI hardware and
> >  > >> it has  > >> been on my TODO list  for quite some time. Keystone
> >  > >> PCI hardware  > >> also doesn't use MSI or MSI-X or INTx for
> >  > >> reporting errors received  > >> at the root port, but use a
> >  > >> platform interrupt instead (not  > >> complaint to PCI standard as
> >  > >> per PCI base spec). So I would need  > >> similar change to have
> >  > >> the error interrupt passed to the aer  > >> driver. So there are
> >  > >> hardware out there like Keystone which  requires to support this
> >  through platform IRQ.
> >  > >>  > >
> >  > >>  > > This is not a new area of the spec, and it's hard for me to
> >  > >> believe  > > that these two new PCIe controllers are both broken
> >  > >> the same way  > > (although I guess both are DesignWare-based, so
> >  > >> maybe this is the  > > same underlying problem in both cases?).  I
> >  > >> think it's more likely  > > that we just haven't figured out the
> >  > >> right way to describe this in  the DT.
> >  > >>  >
> >  > >>  > Keystone is using an older version of the designware IP and it
> >  > >> > implements all of the interrupts in the application register
> >  > >> space  > unlike other newer version of the hardware. So I assume,
> >  > >> the version  > used on Layerscape is also an older version and the
> >  > >> both have same  > issue in terms of non standard platform interrupt
> >  > >> used for error  reporting.
> >  > >>  >
> >  > >>  > > I assume you have a Root Port with an AER capability, no MSI
> >  > >> > > capability, and no MSI-X capability, right?
> >  > >>  >
> >  > >>  > Has AER capability and both MSI and INTx (legacy) capability  >
> >  > >> > > What does its Interrupt  > > Pin register contain?  If it's
> >  > >> zero, it doesn't use INTx either, so  > > according to the spec it
> >  > >> should generate no interrupts.
> >  > >>  > >
> >  > >>  > At address offset 0x3C by default has a value of 1, but it is
> >  > >> writable  > by software. So default is INTx A.
> >  > >>
> >  > >>  0x3c is the Interrupt *Line*, which is read/write.  The Interrupt
> >  > >>  *Pin* is at 0x3d and should be read-only.
> >  > >>
> >  >
> >  > You are right. But default is 1 at this address.
> >  >
> >  > >>  Does your Keystone driver support MSI?  If so, since your Root
> >  > >> Port  supports MSI, I would think we would use that by default, and
> >  > >> the INTx  stuff wouldn't even matter.
> >  > >
> >  > > Layerscape is also shows "Both message signaled interrupts (MSI) and
> >  legacy INTx are supported."
> >  > > But both of them not work for AER interrupt when devices or root
> >  port report aer error.
> >  > > But another GIC interrupt line do.
> >  >
> >  > Same with Keystone. Even though both MSI and INTx are supported error
> >  > interrupt at root port is reported on a different interrupt line than
> >  > MSI/INTx. So for Power Management event interrupt is also different
> >  > line.
> >  
> >  I'm looking at the "Error Message Controls" diagram in the PCIe spec
> >  r3.0, sec 6.2.6.  Does this hardware fit into the platform-specific
> >  "System Error" case there?  Do the Root Control enable bits (in the PCIe
> >  Capability) control this interrupt?  If so, maybe this makes more sense
> >  than I thought.
> 
> It supposedly not the "System Error" case. But "the Error Interrupt" case.
> Which means " Root Error Command register " could control the interrupt
>  line we have now. (refer PCIe spec r3.0, sec 6.2.6)

Did you actually try this out and verify that the PCIe Root Control
enable bits have no effect and the AER Root Error Command bits do
control it?  The names are very similar, so there's lots of room for
misunderstanding here :)

If the AER Root Error Command does control this interrupt, I think the
PCI_COMMAND_INTX_DISABLE bit in the PCI Command register should also
control it (per sec 6.2.4.1.2).

> May this kind of hardware design route broken the spec? 

If the Reporting Enable bits in the Root Port's AER Root Error Command
register control the interrupt, but the interrupt is not delivered via
the Root Port's INTx or MSI/MSI-X, I think the design is not following
the spec.

All the information needed by the AER driver should be communicated
via the config space mechanisms described in the spec (AER capability,
MSI/MSI-X capabilities, Interrupt Pin, etc.)  That way the driver
works without change on future spec-compliant hardware.

> PME also like the AER. Hotplug is not supported. Others not known.
> Po Liu

Per sec 6.1.6, I think PME *should* be signaled by the Root Port's
INTx or MSI/MSI-X.

In particular, it says "Note that all other interrupt sources within
the same Function will assert the same virtual INTx wire when
requesting service."  To me, that means that if we're using INTx, it
will be the same INTx for AER, PME, hotplug, etc., and it should be
the one indicated by the Interrupt Pin register.

But I think on your Root Port:

  - There is an MSI capability, but MSI doesn't actually work at all
  - Interrupt Pin contains 1, indicating INTA, which is routed to IRQ X
  - AER interrupts are routed to some different IRQ Y
  - PME interrupts are routed to a third IRQ Z

So how should we work around this?  I think you should be able to get
partway there with a quirk that sets:

  dev->no_msi = 1;
  dev->irq = Y;

for this device.  That should make AER work, but of course PME would
not work.

Is there a way to set up your interrupt controller so these three
interrupts (X, Y, Z above) all map to the same Linux IRQ?  If you can
do that, you could set up INTA, the AER interrupt, and the PME
interrupt to all be on the same IRQ and everything should work.

Bjorn

[toc] | [prev] | [next] | [standalone]

#1416873

FromPo Liu <po.liu@nxp.com>
Date2016-06-08 07:20 +0200
Message-ID<rHE4h-4dP-9@gated-at.bofh.it>
In reply to#1416692
Hi Bjorn,

Thanks for the kindly reply. All these are helpful.

>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]
>  On Wed, June 08, 2016 6:47 AM
>  
>  On Tue, Jun 07, 2016 at 10:07:40AM +0000, Po Liu wrote:
>  > Hi Bjorn,
>  >
>  > >  -----Original Message-----
>  > >
>  > >  On Mon, Jun 06, 2016 at 10:01:44AM -0400, Murali Karicheri wrote:
>  > >  > On 06/06/2016 03:32 AM, Po Liu wrote:
>  > >  > > Hi Bjorn,
>  > >  > > I confirm we met same problem with KeyStone base on DesignWare
>  > > design.
>  > >  > >
>  > >  > >
>  > >  > > Best regards,
>  > >  > > Liu Po
>  > >  > >
>  > >  > >>  -----Original Message-----
>  > >  > >>  From: Bjorn Helgaas [mailto:helgaas@kernel.org]  > >>  Sent:
>  > > Saturday, June 04, 2016 11:49 AM  > >>  To: Murali Karicheri  > >>
>  > > Cc: Po Liu; linux-pci@vger.kernel.org; linux-arm-  > >>
>  > > kernel@lists.infradead.org; linux-kernel@vger.kernel.org;  > >>
>  > > devicetree@vger.kernel.org; Arnd Bergmann; Roy Zang; Marc Zyngier;
>  > > > >> Stuart Yoder; Yang-Leo Li; Minghuan Lian; Bjorn Helgaas; Shawn
>  > > Guo;  > >> Mingkai Hu; Rob Herring  > >>  Subject: Re: [PATCH 2/2]
>  > > aer: add support aer interrupt with none  > >> MSI/MSI-X/INTx mode
>  > > > >>  > >>  On Fri, Jun 03, 2016 at 01:31:11PM -0400, Murali
>  > > Karicheri wrote:
>  > >  > >>  > Po,
>  > >  > >>  >
>  > >  > >>  > Sorry to hijack your discussion, but the problem seems to
>  > > be  > >> same for  > Keystone PCI controller which is also
>  > > designware (old
>  > >  version) based.
>  > >  > >>  >
>  > >  > >>  > On 06/03/2016 12:09 AM, Bjorn Helgaas wrote:
>  > >  > >>  > > On Thu, Jun 02, 2016 at 11:37:28AM -0400, Murali
>  > > Karicheri
>  > >  wrote:
>  > >  > >>  > >> On 06/02/2016 09:55 AM, Bjorn Helgaas wrote:
>  > >  > >>  > >>> On Thu, Jun 02, 2016 at 05:01:19AM +0000, Po Liu wrote:
>  > >  > >>  > >>>>>  -----Original Message-----  > >>>>>  From: Bjorn
>  > > Helgaas  > >> [mailto:helgaas@kernel.org]  > >>>>>  Sent: Thursday,
>  > > June 02, 2016  > >> 11:48 AM  > >>>>>  To: Po Liu  > >>>>>  Cc:
>  > >  > >> linux-pci@vger.kernel.org;  > >>>>>  > >>
>  > > linux-arm-kernel@lists.infradead.org;
>  > >  > >>  > >>>>>  linux-kernel@vger.kernel.org;
>  > > devicetree@vger.kernel.org;  > >> Arnd  > >>>>> Bergmann;  Roy Zang;
>  > > Marc Zyngier; Stuart Yoder;  > >> Yang-Leo Li;  > >>>>> Minghuan
>  > > Lian; Bjorn  Helgaas; Shawn Guo;  > >> Mingkai Hu; Rob  > >>>>>
>  > > Herring  > >>>>>  Subject: Re: [PATCH 2/2]  > >> aer: add support
>  > > aer interrupt with  > >>>>> none  MSI/MSI-X/INTx  > >> mode  > >>>>>
>  > > > >>>>>  [+cc Rob]  > >>>>>  > >>>>>  Hi Po,  >  > >> >>>>>  > >>>>>
>  > > On Thu, May 26, 2016 at 02:00:06PM +0800, Po Liu  > >> wrote:
>  > >  > >>  > >>>>>  > On some platforms, root port doesn't support  > >>
>  > > MSI/MSI-X/INTx  in RC mode.
>  > >  > >>  > >>>>>  > When chip support the aer interrupt with none  >
>  > > >> MSI/MSI-X/INTx  > >>>>> mode,  > maybe there is interrupt line
>  > > for  > >> aer pme etc. Search  > >>>>> the interrupt  > number in
>  > > the fdt  file.
>  > >  > >>  > >>>>>
>  > >  > >>  > >>>>>  My understanding is that AER interrupt signaling can
>  > > be  > >> done  > >>>>> via INTx,  MSI, or MSI-X (PCIe spec r3.0, sec
>  > > 6.2.4.1.2).
>  > >  > >>  > >>>>> Apparently your device  doesn't support MSI or MSI-X.
>  > > Are  > >> you  > >>>>> saying it doesn't support INTx  either?  How
>  > > is the  > >> interrupt  you're requesting here different from INTx?
>  > >  > >>  > >>>>
>  > >  > >>  > >>>> Layerscape use none of MSI or MSI-X or INTx to
>  > > indicate the  > >> > >>>> devices or root error in RC mode. But use
>  > > an independent SPI  > >> > >>>> interrupt(arm interrupt controller)
>  line.
>  > >  > >>  > >>>
>  > >  > >>  > >>> The Root Port is a PCI device and should follow the
>  > > normal  > >> PCI  > >>> rules for interrupts.  As far as I
>  > > understand, that  > >> means it  > >>> should use MSI, MSI-X, or
>  > > INTx.  If your Root Port  > >> doesn't use MSI  > >>> or MSI-X, it
>  > > should use INTx, the  > >> PCI_INTERRUPT_PIN register  > >>> should
>  > > tell us which (INTA/  > >> INTB/etc.), and  PCI_COMMAND_INTX_DISABLE
>  should work to disable it.
>  > >  > >>  > >>> That's all from the PCI point of view, of course.
>  > >  > >>  > >>
>  > >  > >>  > >> I am faced with the same issue on Keystone PCI hardware
>  > > and  > >> it has  > >> been on my TODO list  for quite some time.
>  > > Keystone  > >> PCI hardware  > >> also doesn't use MSI or MSI-X or
>  > > INTx for  > >> reporting errors received  > >> at the root port, but
>  > > use a  > >> platform interrupt instead (not  > >> complaint to PCI
>  > > standard as  > >> per PCI base spec). So I would need  > >> similar
>  > > change to have  > >> the error interrupt passed to the aer  > >>
>  > > driver. So there are  > >> hardware out there like Keystone which
>  > > requires to support this  through platform IRQ.
>  > >  > >>  > >
>  > >  > >>  > > This is not a new area of the spec, and it's hard for me
>  > > to  > >> believe  > > that these two new PCIe controllers are both
>  > > broken  > >> the same way  > > (although I guess both are
>  > > DesignWare-based, so  > >> maybe this is the  > > same underlying
>  > > problem in both cases?).  I  > >> think it's more likely  > > that
>  > > we just haven't figured out the  > >> right way to describe this in
>  the DT.
>  > >  > >>  >
>  > >  > >>  > Keystone is using an older version of the designware IP and
>  > > it  > >> > implements all of the interrupts in the application
>  > > register  > >> space  > unlike other newer version of the hardware.
>  > > So I assume,  > >> the version  > used on Layerscape is also an
>  > > older version and the  > >> both have same  > issue in terms of non
>  > > standard platform interrupt  > >> used for error  reporting.
>  > >  > >>  >
>  > >  > >>  > > I assume you have a Root Port with an AER capability, no
>  > > MSI  > >> > > capability, and no MSI-X capability, right?
>  > >  > >>  >
>  > >  > >>  > Has AER capability and both MSI and INTx (legacy)
>  > > capability  >  > >> > > What does its Interrupt  > > Pin register
>  > > contain?  If it's  > >> zero, it doesn't use INTx either, so  > >
>  > > according to the spec it  > >> should generate no interrupts.
>  > >  > >>  > >
>  > >  > >>  > At address offset 0x3C by default has a value of 1, but it
>  > > is  > >> writable  > by software. So default is INTx A.
>  > >  > >>
>  > >  > >>  0x3c is the Interrupt *Line*, which is read/write.  The
>  > > Interrupt  > >>  *Pin* is at 0x3d and should be read-only.
>  > >  > >>
>  > >  >
>  > >  > You are right. But default is 1 at this address.
>  > >  >
>  > >  > >>  Does your Keystone driver support MSI?  If so, since your
>  > > Root  > >> Port  supports MSI, I would think we would use that by
>  > > default, and  > >> the INTx  stuff wouldn't even matter.
>  > >  > >
>  > >  > > Layerscape is also shows "Both message signaled interrupts
>  > > (MSI) and  legacy INTx are supported."
>  > >  > > But both of them not work for AER interrupt when devices or
>  > > root  port report aer error.
>  > >  > > But another GIC interrupt line do.
>  > >  >
>  > >  > Same with Keystone. Even though both MSI and INTx are supported
>  > > error  > interrupt at root port is reported on a different interrupt
>  > > line than  > MSI/INTx. So for Power Management event interrupt is
>  > > also different  > line.
>  > >
>  > >  I'm looking at the "Error Message Controls" diagram in the PCIe
>  > > spec  r3.0, sec 6.2.6.  Does this hardware fit into the
>  > > platform-specific  "System Error" case there?  Do the Root Control
>  > > enable bits (in the PCIe
>  > >  Capability) control this interrupt?  If so, maybe this makes more
>  > > sense  than I thought.
>  >
>  > It supposedly not the "System Error" case. But "the Error Interrupt"
>  case.
>  > Which means " Root Error Command register " could control the
>  > interrupt  line we have now. (refer PCIe spec r3.0, sec 6.2.6)
>  
>  Did you actually try this out and verify that the PCIe Root Control
>  enable bits have no effect and the AER Root Error Command bits do
>  control it?  The names are very similar, so there's lots of room for
>  misunderstanding here :)

Yes, all these result were tested before reply.

>  
>  If the AER Root Error Command does control this interrupt, I think the
>  PCI_COMMAND_INTX_DISABLE bit in the PCI Command register should also
>  control it (per sec 6.2.4.1.2).

Yes, I am sure the PCI_COMMAND_INTX_DISABLE bit can also control this interrupt.

>  
>  > May this kind of hardware design route broken the spec?
>  
>  If the Reporting Enable bits in the Root Port's AER Root Error Command
>  register control the interrupt, but the interrupt is not delivered via
>  the Root Port's INTx or MSI/MSI-X, I think the design is not following
>  the spec.
>  
>  All the information needed by the AER driver should be communicated via
>  the config space mechanisms described in the spec (AER capability,
>  MSI/MSI-X capabilities, Interrupt Pin, etc.)  That way the driver works
>  without change on future spec-compliant hardware.
>  
>  > PME also like the AER. Hotplug is not supported. Others not known.
>  > Po Liu
>  
>  Per sec 6.1.6, I think PME *should* be signaled by the Root Port's INTx
>  or MSI/MSI-X.
>  
>  In particular, it says "Note that all other interrupt sources within the
>  same Function will assert the same virtual INTx wire when requesting
>  service."  To me, that means that if we're using INTx, it will be the
>  same INTx for AER, PME, hotplug, etc., and it should be the one
>  indicated by the Interrupt Pin register.
>  
>  But I think on your Root Port:
>  
>    - There is an MSI capability, but MSI doesn't actually work at all
>    - Interrupt Pin contains 1, indicating INTA, which is routed to IRQ X
>    - AER interrupts are routed to some different IRQ Y
>    - PME interrupts are routed to a third IRQ Z
>  

The descriptions are all right.

>  So how should we work around this?  I think you should be able to get
>  partway there with a quirk that sets:
>  
>    dev->no_msi = 1;
>    dev->irq = Y;
>  
>  for this device.  That should make AER work, but of course PME would not
>  work.
>  
>  Is there a way to set up your interrupt controller so these three
>  interrupts (X, Y, Z above) all map to the same Linux IRQ?  If you can do
>  that, you could set up INTA, the AER interrupt, and the PME interrupt to
>  all be on the same IRQ and everything should work.
>  
>  Bjorn

We'll think about all the ways. It is really helpful, thanks!

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web