Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1715465 > unrolled thread

[PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs()

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs() Bjorn Helgaas <bhelgaas@google.com> - 2017-08-18 23:40 +0200
    Re: [PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs() Sinan Kaya <okaya@codeaurora.org> - 2017-08-21 16:00 +0200
      Re: [PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs() Bjorn Helgaas <helgaas@kernel.org> - 2017-08-21 21:20 +0200
        Re: [PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs() Sinan Kaya <okaya@codeaurora.org> - 2017-08-21 21:40 +0200
          Re: [PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs() Bjorn Helgaas <helgaas@kernel.org> - 2017-08-21 22:30 +0200
            Re: [PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs() Sinan Kaya <okaya@codeaurora.org> - 2017-08-21 22:40 +0200
              Re: [PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs() Bjorn Helgaas <helgaas@kernel.org> - 2017-08-21 23:10 +0200
          Re: [PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs() Sinan Kaya <okaya@codeaurora.org> - 2017-08-23 06:50 +0200

#1715465 — [PATCH v11 2/4] PCI: Factor out pci_bus_wait_crs()

From: Sinan Kaya <okaya@codeaurora.org>

Configuration Request Retry Status (CRS) was previously hidden inside
pci_bus_read_dev_vendor_id().  We want to add support for CRS in other
situations, such as waiting for a device to become ready after a Function
Level Reset.

Move CRS handling into pci_bus_wait_crs() so it can be called from other
places.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
[bhelgaas: move CRS reserved Vendor ID test into pci_bus_wait_crs(), remove
EXPORT_SYMBOL]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
---
 drivers/pci/pci.h   |    1 +
 drivers/pci/probe.c |   39 +++++++++++++++++++++++----------------
 2 files changed, 24 insertions(+), 16 deletions(-)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 22e061738c6f..b0857052c04a 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -235,6 +235,7 @@ enum pci_bar_type {
 	pci_bar_mem64,		/* A 64-bit memory BAR */
 };
 
+bool pci_bus_wait_crs(struct pci_bus *bus, int devfn, u32 *l, int timeout);
 bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *pl,
 				int crs_timeout);
 int pci_setup_device(struct pci_dev *dev);
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 08ea844ac4ba..342a86640c6b 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1824,33 +1824,26 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus)
 }
 EXPORT_SYMBOL(pci_alloc_dev);
 
-bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *l,
-				int crs_timeout)
+bool pci_bus_wait_crs(struct pci_bus *bus, int devfn, u32 *l, int timeout)
 {
 	int delay = 1;
 
-	if (pci_bus_read_config_dword(bus, devfn, PCI_VENDOR_ID, l))
-		return false;
+	if ((*l & 0xffff) != 0x0001)
+		return true;	/* not a CRS completion */
 
-	/* some broken boards return 0 or ~0 if a slot is empty: */
-	if (*l == 0xffffffff || *l == 0x00000000 ||
-	    *l == 0x0000ffff || *l == 0xffff0000)
-		return false;
+	if (!timeout)
+		return false;	/* CRS, but caller doesn't want to wait */
 
 	/*
-	 * Configuration Request Retry Status.  Some root ports return the
-	 * actual device ID instead of the synthetic ID (0xFFFF) required
-	 * by the PCIe spec.  Ignore the device ID and only check for
-	 * (vendor id == 1).
+	 * We got the reserved Vendor ID that indicates a completion with
+	 * Configuration Request Retry Status (CRS).  Retry until we get a
+	 * valid Vendor ID or we time out.
 	 */
 	while ((*l & 0xffff) == 0x0001) {
-		if (!crs_timeout)
-			return false;
-
 		msleep(delay);
 		delay *= 2;
 
-		if (delay > crs_timeout) {
+		if (delay > timeout) {
 			printk(KERN_WARNING "pci %04x:%02x:%02x.%d: not responding\n",
 			       pci_domain_nr(bus), bus->number, PCI_SLOT(devfn),
 			       PCI_FUNC(devfn));
@@ -1863,6 +1856,20 @@ bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *l,
 
 	return true;
 }
+
+bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *l,
+				int timeout)
+{
+	if (pci_bus_read_config_dword(bus, devfn, PCI_VENDOR_ID, l))
+		return false;
+
+	/* some broken boards return 0 or ~0 if a slot is empty: */
+	if (*l == 0xffffffff || *l == 0x00000000 ||
+	    *l == 0x0000ffff || *l == 0xffff0000)
+		return false;
+
+	return pci_bus_wait_crs(bus, devfn, l, timeout);
+}
 EXPORT_SYMBOL(pci_bus_read_dev_vendor_id);
 
 /*

[toc] | [next] | [standalone]

#1716497

On 8/18/2017 5:32 PM, Bjorn Helgaas wrote:
> +	if ((*l & 0xffff) != 0x0001)
> +		return true;	/* not a CRS completion */
>  

This version certainly looks cleaner. However, it breaks pci_flr_wait().

If some root port doesn't support CRS and returns 0xFFFFFFFF, pci_bus_wait_crs()
function returns true. pci_flr_wait() prematurely bails out from here.


pci_flr_wait()
{

+	ret = pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000);
+	if (ret)
+		return;

}

We can change the return code to false above but then we break pci_bus_read_dev_vendor_id()
function. 

That's why, I was interested in creating a pci_bus_crs_visibility_supported() helper
function that would check for the magic 0x0001 value and return true. Otherwise, false. 

pci_bus_read_dev_vendor_id() would do this

pci_bus_read_dev_vendor_id()
{
	...
	if (pci_bus_crs_visibility_supported())
		return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000);

	return true
}

Similar pattern for pci_flr_wait().


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

[toc] | [prev] | [next] | [standalone]

#1716789

On Mon, Aug 21, 2017 at 09:53:56AM -0400, Sinan Kaya wrote:
> On 8/18/2017 5:32 PM, Bjorn Helgaas wrote:
> > +	if ((*l & 0xffff) != 0x0001)
> > +		return true;	/* not a CRS completion */
> >  
> 
> This version certainly looks cleaner. However, it breaks pci_flr_wait().
> 
> If some root port doesn't support CRS and returns 0xFFFFFFFF, pci_bus_wait_crs()
> function returns true. pci_flr_wait() prematurely bails out from here.
> 
> 
> pci_flr_wait()
> {
> 
> +	ret = pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000);
> +	if (ret)
> +		return;
> 
> }
> 
> We can change the return code to false above but then we break pci_bus_read_dev_vendor_id()
> function. 
> 
> That's why, I was interested in creating a pci_bus_crs_visibility_supported() helper
> function that would check for the magic 0x0001 value and return true. Otherwise, false. 
> 
> pci_bus_read_dev_vendor_id() would do this
> 
> pci_bus_read_dev_vendor_id()
> {
> 	...
> 	if (pci_bus_crs_visibility_supported())
> 		return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000);
> 
> 	return true
> }
> 
> Similar pattern for pci_flr_wait().

I think that makes sense.  We'd want to check for CRS SV being
enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in
pci_enable_crs() and cache it somewhere.  Maybe a crs_sv_enabled bit
in the root port's pci_dev, and check it with something like what
pcie_root_rcb_set() does?

[toc] | [prev] | [next] | [standalone]

#1716810

On 8/21/2017 3:18 PM, Bjorn Helgaas wrote:
>> pci_bus_read_dev_vendor_id()
>> {
>> 	...
>> 	if (pci_bus_crs_visibility_supported())
>> 		return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000);
>>
>> 	return true
>> }
>>
>> Similar pattern for pci_flr_wait().

Sorry for the poor choice of function name. 

I was thinking of something like this.

bool pci_bus_crs_pending(u32 l)
{
	return (l & 0xFFFF) == 0x0001
}

if (pci_bus_crs_pending(id))
	return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000);

> I think that makes sense.  We'd want to check for CRS SV being
> enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in
> pci_enable_crs() and cache it somewhere.  Maybe a crs_sv_enabled bit
> in the root port's pci_dev, and check it with something like what
> pcie_root_rcb_set() does?
> 

You can observe CRS under the following conditions

1. root port <-> endpoint 
2. bridge <-> endpoint 
3. root port<->bridge

I was relying on the fact that we are reading 0x001 as an indication that
this device detected CRS. Maybe, this is too indirect.

If we also want to capture the capability, I think the right thing is to
check the parent capability.

bool pci_bus_crs_vis_supported(struct pci_dev *bridge)
{
	if (device type(bridge) == root port)
		return read(root_crs_register_reg);

	if (device type(bridge) == switch)
		return read(switch_crs_register);

	return false;
}

bool pci_bus_crs_pending(struct pci_dev *dev, u32 l)
{
	if !pci_bus_crs_vis_supported(dev->parent)
		return false;

	return (l & 0xFFFF) == 0x0001;
}

I'll prototype this. Let me know if you have concerns.

-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

[toc] | [prev] | [next] | [standalone]

#1716842

On Mon, Aug 21, 2017 at 03:37:06PM -0400, Sinan Kaya wrote:
> On 8/21/2017 3:18 PM, Bjorn Helgaas wrote:
> ...
> if (pci_bus_crs_pending(id))
> 	return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000);
> 
> > I think that makes sense.  We'd want to check for CRS SV being
> > enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in
> > pci_enable_crs() and cache it somewhere.  Maybe a crs_sv_enabled bit
> > in the root port's pci_dev, and check it with something like what
> > pcie_root_rcb_set() does?
> > 
> 
> You can observe CRS under the following conditions
> 
> 1. root port <-> endpoint 
> 2. bridge <-> endpoint 
> 3. root port<->bridge
> 
> I was relying on the fact that we are reading 0x001 as an indication that
> this device detected CRS. Maybe, this is too indirect.
> 
> If we also want to capture the capability, I think the right thing is to
> check the parent capability.
> 
> bool pci_bus_crs_vis_supported(struct pci_dev *bridge)
> {
> 	if (device type(bridge) == root port)
> 		return read(root_crs_register_reg);
> 
> 	if (device type(bridge) == switch)
> 		return read(switch_crs_register);

I don't understand this part.  AFAIK, CRS SV is only a feature of root
ports.  The capability and enable bits are in the Root Capabilities
and Root Control registers.

It's certainly true that a device below a switch can respond with a
CRS completion, but the switch is not the requester, and my
understanding is that it would not take any action on the completion
other than passing it upstream.

[toc] | [prev] | [next] | [standalone]

#1716862

On 8/21/2017 4:23 PM, Bjorn Helgaas wrote:
> On Mon, Aug 21, 2017 at 03:37:06PM -0400, Sinan Kaya wrote:
>> On 8/21/2017 3:18 PM, Bjorn Helgaas wrote:
>> ...
>> if (pci_bus_crs_pending(id))
>> 	return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000);
>>
>>> I think that makes sense.  We'd want to check for CRS SV being
>>> enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in
>>> pci_enable_crs() and cache it somewhere.  Maybe a crs_sv_enabled bit
>>> in the root port's pci_dev, and check it with something like what
>>> pcie_root_rcb_set() does?
>>>
>>
>> You can observe CRS under the following conditions
>>
>> 1. root port <-> endpoint 
>> 2. bridge <-> endpoint 
>> 3. root port<->bridge
>>
>> I was relying on the fact that we are reading 0x001 as an indication that
>> this device detected CRS. Maybe, this is too indirect.
>>
>> If we also want to capture the capability, I think the right thing is to
>> check the parent capability.
>>
>> bool pci_bus_crs_vis_supported(struct pci_dev *bridge)
>> {
>> 	if (device type(bridge) == root port)
>> 		return read(root_crs_register_reg);
>>
>> 	if (device type(bridge) == switch)
>> 		return read(switch_crs_register);
> 
> I don't understand this part.  AFAIK, CRS SV is only a feature of root
> ports.  The capability and enable bits are in the Root Capabilities
> and Root Control registers.
> 

No question about it.

> It's certainly true that a device below a switch can respond with a
> CRS completion, but the switch is not the requester, and my
> understanding is that it would not take any action on the completion
> other than passing it upstream.
> 

I saw some bridge references in the spec for CRS. I was going to do
some research for it. You answered my question. I was curious how this
would impact the behavior.

"Bridge Configuration Retry Enable – When Set, this bit enables PCI Express
to PCI/PCI-X bridges to return Configuration Request Retry Status (CRS) in
response to Configuration Requests that target devices below the bridge. 
Refer to the PCI Express to PCI/PCI-X Bridge Specification, Revision 1.0 for
further details."

-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

[toc] | [prev] | [next] | [standalone]

#1716882

On Mon, Aug 21, 2017 at 04:32:26PM -0400, Sinan Kaya wrote:
> On 8/21/2017 4:23 PM, Bjorn Helgaas wrote:
> > On Mon, Aug 21, 2017 at 03:37:06PM -0400, Sinan Kaya wrote:
> >> On 8/21/2017 3:18 PM, Bjorn Helgaas wrote:
> >> ...
> >> if (pci_bus_crs_pending(id))
> >> 	return pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000);
> >>
> >>> I think that makes sense.  We'd want to check for CRS SV being
> >>> enabled, e.g., maybe read PCI_EXP_RTCTL_CRSSVE back in
> >>> pci_enable_crs() and cache it somewhere.  Maybe a crs_sv_enabled bit
> >>> in the root port's pci_dev, and check it with something like what
> >>> pcie_root_rcb_set() does?
> >>>
> >>
> >> You can observe CRS under the following conditions
> >>
> >> 1. root port <-> endpoint 
> >> 2. bridge <-> endpoint 
> >> 3. root port<->bridge
> >>
> >> I was relying on the fact that we are reading 0x001 as an indication that
> >> this device detected CRS. Maybe, this is too indirect.
> >>
> >> If we also want to capture the capability, I think the right thing is to
> >> check the parent capability.
> >>
> >> bool pci_bus_crs_vis_supported(struct pci_dev *bridge)
> >> {
> >> 	if (device type(bridge) == root port)
> >> 		return read(root_crs_register_reg);
> >>
> >> 	if (device type(bridge) == switch)
> >> 		return read(switch_crs_register);
> > 
> > I don't understand this part.  AFAIK, CRS SV is only a feature of root
> > ports.  The capability and enable bits are in the Root Capabilities
> > and Root Control registers.
> > 
> 
> No question about it.
> 
> > It's certainly true that a device below a switch can respond with a
> > CRS completion, but the switch is not the requester, and my
> > understanding is that it would not take any action on the completion
> > other than passing it upstream.
> > 
> 
> I saw some bridge references in the spec for CRS. I was going to do
> some research for it. You answered my question. I was curious how this
> would impact the behavior.
> 
> "Bridge Configuration Retry Enable – When Set, this bit enables PCI Express
> to PCI/PCI-X bridges to return Configuration Request Retry Status (CRS) in
> response to Configuration Requests that target devices below the bridge. 
> Refer to the PCI Express to PCI/PCI-X Bridge Specification, Revision 1.0 for
> further details."

(The above is from PCIe r3.1, sec 7.8.4, Device Control Register, and
also discussed in sec 4.3 of the PCIe-to-PCI/PCI-X bridge spec.)

In any event, the Bridge Configuration Retry Enable only determines
whether the bridge ever returns a CRS completion.  The bridge itself
never converts a CRS completion into the 0x0001 vendor ID.  

[toc] | [prev] | [next] | [standalone]

#1718029

On 8/21/2017 3:37 PM, Sinan Kaya wrote:
> bool pci_bus_crs_pending(struct pci_dev *dev, u32 l)
> {
> 	if !pci_bus_crs_vis_supported(dev->parent)
> 		return false;

Apparently, I can't do this. By the time, we come to here from vendor id read
function, the topology has not been set up yet. 

I'm getting an exception:

[    6.067392] [<ffff00000846c878>] pci_bus_crs_visibility_pending+0x4/0x7c
[    6.074085] [<ffff00000846cccc>] pci_scan_single_device+0x40/0xb4
[    6.080170] [<ffff00000846cd90>] pci_scan_slot+0x50/0xe8
[    6.085474] [<ffff00000846dc14>] pci_scan_child_bus+0x30/0x108
[    6.091300] [<ffff0000084bab94>] acpi_pci_root_create+0x184/0x1f0
[    6.097388] [<ffff000008091dc8>] pci_acpi_scan_root+0x188/0x1d4
[    6.103298] [<ffff0000084ba7a8>] acpi_pci_root_add+0x38c/0x44c
[    6.109125] [<ffff0000084b4d94>] acpi_bus_attach+0xe0/0x1ac
[    6.114689] [<ffff0000084b4e08>] acpi_bus_attach+0x154/0x1ac
[    6.120340] [<ffff0000084b4e08>] acpi_bus_attach+0x154/0x1ac
[    6.125991] [<ffff0000084b6608>] acpi_bus_scan+0x60/0x70
[    6.131297] [<ffff0000091c87f8>] acpi_scan_init+0xd8/0x228
[    6.136774] [<ffff0000091c84e0>] acpi_init+0x2d4/0x328
[    6.141905] [<ffff0000091a0c88>] do_one_initcall+0x80/0x108
[    6.147469] [<ffff0000091a0e98>] kernel_init_freeable+0x188/0x228
[    6.153556] [<ffff000008c9bcbc>] kernel_init+0x10/0xfc
[    6.158687] [<ffff000008082ec0>] ret_from_fork+0x10/0x50



> 
> 	return (l & 0xFFFF) == 0x0001;
> }


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web