Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1238670 > unrolled thread

[PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

Back to article view | Back to linux.kernel

Contents

  [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs" David Daney <ddaney.cavm@gmail.com> - 2015-10-03 00:40 +0200
    Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation  "BARs" "Sean O. Stalley" <sean.stalley@intel.com> - 2015-10-03 02:00 +0200
    Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs" Yinghai Lu <yinghai@kernel.org> - 2015-10-03 05:20 +0200
      Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation  "BARs" "Sean O. Stalley" <sean.stalley@intel.com> - 2015-10-06 01:10 +0200
        Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs" David Daney <ddaney.cavm@gmail.com> - 2015-10-06 03:20 +0200
          Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation  "BARs" "Sean O. Stalley" <sean.stalley@intel.com> - 2015-10-06 18:00 +0200

#1238670 — [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

From: David Daney <david.daney@cavium.com>

The original patches are from Sean O. Stalley. I made a few tweaks,
but feel that it is substancially Sean's work, so I am keeping the
patch set version numbering scheme going.

Tested on Cavium ThunderX system with 4 Root Complexes containing 50
devices/bridges provisioned with EA.

Here is Sean's description of the patches:

PCI Enhanced Allocation is a new method of allocating MMIO & IO
resources for PCI devices & bridges. It can be used instead
of the traditional PCI method of using BARs.

EA entries are hardware-initialized to a fixed address.
Unlike BARs, regions described by EA are cannot be moved.
Because of this, only devices which are permanently connected to
the PCI bus can use EA. A removable PCI card must not use EA.

This patchset adds support for using EA entries instead of BARs
on Root Complex Integrated Endpoints.

The Enhanced Allocation ECN is publicly available here:
https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf


Changes from V1:
	- Use generic PCI resource claim functions (instead of EA-specific functions)
	- Only add support for RCiEPs (instead of all devices).
	- Removed some debugging messages leftover from early testing.

Changes from V2 (By David Daney):
	- Add ea_cap to struct pci_device, to aid in finding the EA capability.
	- Factored EA entity decoding into a separate function.
	- Add functions to find EA entities by BEI or Property.
	- Add handling of EA provisioned bridges.
	- Add handling of EA SRIOV BARs.
	- Try to assign proper resource parent so that SRIOV device creation can occur.

Changes from V3 (By David Daney):
	- Discarded V3 changes and started over fresh based on Sean's V2.
	- Add more support/checking for Entry Properties.
	- Allow EA behind bridges.
	- Rewrite some error messages.
	- Add patch 3/5 to prevent resizing, and better handle
          assigning, of fixed EA resources.
	- Add patch 4/5 to handle EA provisioned SRIOV devices.
	- Add patch 5/5 to handle EA provisioned bridges.

David Daney (3):
  PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.
  PCI: Handle Enhanced Allocation (EA) capability for SRIOV devices.
  PCI: Handle Enhanced Allocation (EA) capability for bridges

Sean O. Stalley (2):
  PCI: Add Enhanced Allocation register entries
  PCI: Add support for Enhanced Allocation devices

 drivers/pci/bus.c             |   7 ++
 drivers/pci/iov.c             |  11 ++-
 drivers/pci/pci.c             | 202 ++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci.h             |   1 +
 drivers/pci/probe.c           |  34 ++++++-
 drivers/pci/setup-bus.c       |  63 ++++++++++++-
 include/linux/pci.h           |   1 +
 include/uapi/linux/pci_regs.h |  44 ++++++++-
 8 files changed, 355 insertions(+), 8 deletions(-)

-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1238703 — Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

Hi David,

I did a quick look through & overall I what you have done.
I will try to find some time to do a full review early next week.

Thanks Again,
Sean

On Fri, Oct 02, 2015 at 03:37:51PM -0700, David Daney wrote:
> From: David Daney <david.daney@cavium.com>
> 
> The original patches are from Sean O. Stalley. I made a few tweaks,
> but feel that it is substancially Sean's work, so I am keeping the
> patch set version numbering scheme going.
> 
> Tested on Cavium ThunderX system with 4 Root Complexes containing 50
> devices/bridges provisioned with EA.
> 
> Here is Sean's description of the patches:
> 
> PCI Enhanced Allocation is a new method of allocating MMIO & IO
> resources for PCI devices & bridges. It can be used instead
> of the traditional PCI method of using BARs.
> 
> EA entries are hardware-initialized to a fixed address.
> Unlike BARs, regions described by EA are cannot be moved.
> Because of this, only devices which are permanently connected to
> the PCI bus can use EA. A removable PCI card must not use EA.
> 
> This patchset adds support for using EA entries instead of BARs
> on Root Complex Integrated Endpoints.
> 
> The Enhanced Allocation ECN is publicly available here:
> https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf
> 
> 
> Changes from V1:
> 	- Use generic PCI resource claim functions (instead of EA-specific functions)
> 	- Only add support for RCiEPs (instead of all devices).
> 	- Removed some debugging messages leftover from early testing.
> 
> Changes from V2 (By David Daney):
> 	- Add ea_cap to struct pci_device, to aid in finding the EA capability.
> 	- Factored EA entity decoding into a separate function.
> 	- Add functions to find EA entities by BEI or Property.
> 	- Add handling of EA provisioned bridges.
> 	- Add handling of EA SRIOV BARs.
> 	- Try to assign proper resource parent so that SRIOV device creation can occur.
> 
> Changes from V3 (By David Daney):
> 	- Discarded V3 changes and started over fresh based on Sean's V2.
> 	- Add more support/checking for Entry Properties.
> 	- Allow EA behind bridges.
> 	- Rewrite some error messages.
> 	- Add patch 3/5 to prevent resizing, and better handle
>           assigning, of fixed EA resources.
> 	- Add patch 4/5 to handle EA provisioned SRIOV devices.
> 	- Add patch 5/5 to handle EA provisioned bridges.
> 
> David Daney (3):
>   PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.
>   PCI: Handle Enhanced Allocation (EA) capability for SRIOV devices.
>   PCI: Handle Enhanced Allocation (EA) capability for bridges
> 
> Sean O. Stalley (2):
>   PCI: Add Enhanced Allocation register entries
>   PCI: Add support for Enhanced Allocation devices
> 
>  drivers/pci/bus.c             |   7 ++
>  drivers/pci/iov.c             |  11 ++-
>  drivers/pci/pci.c             | 202 ++++++++++++++++++++++++++++++++++++++++++
>  drivers/pci/pci.h             |   1 +
>  drivers/pci/probe.c           |  34 ++++++-
>  drivers/pci/setup-bus.c       |  63 ++++++++++++-
>  include/linux/pci.h           |   1 +
>  include/uapi/linux/pci_regs.h |  44 ++++++++-
>  8 files changed, 355 insertions(+), 8 deletions(-)
> 
> -- 
> 1.9.1
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1238727

On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
> From: David Daney <david.daney@cavium.com>
>
> PCI Enhanced Allocation is a new method of allocating MMIO & IO
> resources for PCI devices & bridges. It can be used instead
> of the traditional PCI method of using BARs.
>
> EA entries are hardware-initialized to a fixed address.
> Unlike BARs, regions described by EA are cannot be moved.
> Because of this, only devices which are permanently connected to
> the PCI bus can use EA. A removable PCI card must not use EA.
>
> The Enhanced Allocation ECN is publicly available here:
> https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf

Looks like the EA will support more than just fixed address later.

"Enhanced Allocation is an optional Conventional PCI Capability that
may be implemented by
Functions to indicate fixed (non reprogrammable) I/O and memory ranges
assigned to the
Function, as well as supporting new resource “type” definitions and
future extensibility to also
support reprogrammable allocations."

so I would prefer to think more to make frame configurable to leave
space for that.

Bjorn,

I wonder if we need to revive the add-on resource support patchset
that i suggested couple years ago,
so we can extend it to support EA features.

URL: https://lkml.org/lkml/2012/3/19/86

Thanks

Yinghai
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1240021 — Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

On Fri, Oct 02, 2015 at 08:16:48PM -0700, Yinghai Lu wrote:
> On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
> > From: David Daney <david.daney@cavium.com>
> >
> > PCI Enhanced Allocation is a new method of allocating MMIO & IO
> > resources for PCI devices & bridges. It can be used instead
> > of the traditional PCI method of using BARs.
> >
> > EA entries are hardware-initialized to a fixed address.
> > Unlike BARs, regions described by EA are cannot be moved.
> > Because of this, only devices which are permanently connected to
> > the PCI bus can use EA. A removable PCI card must not use EA.
> >
> > The Enhanced Allocation ECN is publicly available here:
> > https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf
> 
> Looks like the EA will support more than just fixed address later.
> 
> "Enhanced Allocation is an optional Conventional PCI Capability that
> may be implemented by
> Functions to indicate fixed (non reprogrammable) I/O and memory ranges
> assigned to the
> Function, as well as supporting new resource “type” definitions and
> future extensibility to also
> support reprogrammable allocations."
> 
> so I would prefer to think more to make frame configurable to leave
> space for that.
> 
> Bjorn,
> 
> I wonder if we need to revive the add-on resource support patchset
> that i suggested couple years ago,
> so we can extend it to support EA features.
> 
> URL: https://lkml.org/lkml/2012/3/19/86
> 
> Thanks
> 
> Yinghai

This might be useful for fixed resources as well.

For some BEI values, EA allows for an arbitrary number of EA entries.
For PF & VF resource ranges, it allows 2 ranges.
(one below the 4GB boundry, and one above).
I don't think the current pci_dev struct can handle that many resources.

-Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1240095

On 10/05/2015 04:05 PM, Sean O. Stalley wrote:
> On Fri, Oct 02, 2015 at 08:16:48PM -0700, Yinghai Lu wrote:
>> On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
>>> From: David Daney <david.daney@cavium.com>
>>>
>>> PCI Enhanced Allocation is a new method of allocating MMIO & IO
>>> resources for PCI devices & bridges. It can be used instead
>>> of the traditional PCI method of using BARs.
>>>
>>> EA entries are hardware-initialized to a fixed address.
>>> Unlike BARs, regions described by EA are cannot be moved.
>>> Because of this, only devices which are permanently connected to
>>> the PCI bus can use EA. A removable PCI card must not use EA.
>>>
>>> The Enhanced Allocation ECN is publicly available here:
>>> https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf
>>
>> Looks like the EA will support more than just fixed address later.
>>
>> "Enhanced Allocation is an optional Conventional PCI Capability that
>> may be implemented by
>> Functions to indicate fixed (non reprogrammable) I/O and memory ranges
>> assigned to the
>> Function, as well as supporting new resource “type” definitions and
>> future extensibility to also
>> support reprogrammable allocations."
>>
>> so I would prefer to think more to make frame configurable to leave
>> space for that.
>>
>> Bjorn,
>>
>> I wonder if we need to revive the add-on resource support patchset
>> that i suggested couple years ago,
>> so we can extend it to support EA features.
>>
>> URL: https://lkml.org/lkml/2012/3/19/86
>>
>> Thanks
>>
>> Yinghai
>
> This might be useful for fixed resources as well.
>
> For some BEI values, EA allows for an arbitrary number of EA entries.

I think this is true only for BEI = 6 which is for type-1 config space 
only (i.e. for bridges)

I am thinking about splitting out the bridge part of the patch set, as 
my systems work fine without explicitly assigning bridge resources via 
EA.  That would allow us to more forward with the patches that are less 
controversial, and spend more time hashing out the proper approach to 
take with bridges.

> For PF & VF resource ranges, it allows 2 ranges.

I don't really understand what you are saying here.  My reading of the 
spec. is that BEI[0..5] are PF BARs and each may have any of the 
properties that are allowed for normal BARs (io, memory-nonprefetchable, 
memory-prefetchable).  BEI[9..14] are VF BARs, and likewise may have any 
of the properties that are alloed for normal VF bars 
(memory-nonprefetchable, memory-prefetchable)

I guess in theory EA allows you to allocate 6 64-bit BARs, where you 
would be limited to only 3 64-bit normal BARs


> (one below the 4GB boundry, and one above).
> I don't think the current pci_dev struct can handle that many resources.
>
> -Sean
>

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1240623 — Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

On Mon, Oct 05, 2015 at 06:17:20PM -0700, David Daney wrote:
> On 10/05/2015 04:05 PM, Sean O. Stalley wrote:
> >On Fri, Oct 02, 2015 at 08:16:48PM -0700, Yinghai Lu wrote:
> >>On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
> >>>From: David Daney <david.daney@cavium.com>
> >>>
> >>>PCI Enhanced Allocation is a new method of allocating MMIO & IO
> >>>resources for PCI devices & bridges. It can be used instead
> >>>of the traditional PCI method of using BARs.
> >>>
> >>>EA entries are hardware-initialized to a fixed address.
> >>>Unlike BARs, regions described by EA are cannot be moved.
> >>>Because of this, only devices which are permanently connected to
> >>>the PCI bus can use EA. A removable PCI card must not use EA.
> >>>
> >>>The Enhanced Allocation ECN is publicly available here:
> >>>https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf
> >>
> >>Looks like the EA will support more than just fixed address later.
> >>
> >>"Enhanced Allocation is an optional Conventional PCI Capability that
> >>may be implemented by
> >>Functions to indicate fixed (non reprogrammable) I/O and memory ranges
> >>assigned to the
> >>Function, as well as supporting new resource “type” definitions and
> >>future extensibility to also
> >>support reprogrammable allocations."
> >>
> >>so I would prefer to think more to make frame configurable to leave
> >>space for that.
> >>
> >>Bjorn,
> >>
> >>I wonder if we need to revive the add-on resource support patchset
> >>that i suggested couple years ago,
> >>so we can extend it to support EA features.
> >>
> >>URL: https://lkml.org/lkml/2012/3/19/86
> >>
> >>Thanks
> >>
> >>Yinghai
> >
> >This might be useful for fixed resources as well.
> >
> >For some BEI values, EA allows for an arbitrary number of EA entries.
> 
> I think this is true only for BEI = 6 which is for type-1 config
> space only (i.e. for bridges)

It's true for a BEI of 6 or 7.

From the ECN (specifically section 6.9.1.3):

Rules for use of BEI field:
	...
	It is permitted for an arbitrary number of entries to assign a BEI of 6 or 7.

> I am thinking about splitting out the bridge part of the patch set,
> as my systems work fine without explicitly assigning bridge
> resources via EA.  That would allow us to more forward with the
> patches that are less controversial, and spend more time hashing out
> the proper approach to take with bridges.

I like this idea. Adding support for Endpoints will be much simpler than
bridges.

> 
> >For PF & VF resource ranges, it allows 2 ranges.
> 
> I don't really understand what you are saying here.  My reading of
> the spec. is that BEI[0..5] are PF BARs and each may have any of the
> properties that are allowed for normal BARs (io,
> memory-nonprefetchable, memory-prefetchable).  BEI[9..14] are VF
> BARs, and likewise may have any of the properties that are alloed
> for normal VF bars (memory-nonprefetchable, memory-prefetchable)


From the ECN (specifically section 6.9.1.3):

Rules for use of BEI field:
...

For Memory or I/O BARs where the Primary or Secondary Property is 00h, 01h or
02h, it is permitted to assign the same BEI in the range of 0-5 once for a range where
Base + MaxOffset is below 4GB, and again for a range where Base + MaxOffset is
greater than 4GB; It is not otherwise permitted to assign the same BEI in the range 0-
5 for more than one entry.

For Virtual Function BARs where the Primary or Secondary Property is 03h or 04h it
is permitted to assign the same BEI in the range of 0-5 once for a range where Base +
MaxOffset is below 4GB, and again for a range where Base + MaxOffset is greater
than 4GB; It is not otherwise permitted to assign the same BEI in the range 0-5 for
more than one VF entry.

(Theres a typo in the VF rule. I think the '0-5' range should be '9-14' for VFs)

> I guess in theory EA allows you to allocate 6 64-bit BARs, where you
> would be limited to only 3 64-bit normal BARs

I guess so, I never thought about that.

> >(one below the 4GB boundry, and one above).
> >I don't think the current pci_dev struct can handle that many resources.
> >
> >-Sean
> >

Let me know if you want any help with the patchset. I'll do what I can.
I'm in today, but I'm out Wed-Fri this week.

-Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web