Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1370430 > unrolled thread

[PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes

Back to article view | Back to linux.kernel

Contents

  [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes Eric Auger <eric.auger@linaro.org> - 2016-04-04 10:20 +0200
    Re: [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel  part 1/3: iommu changes Alex Williamson <alex.williamson@redhat.com> - 2016-04-07 01:20 +0200
      Re: [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part  1/3: iommu changes Eric Auger <eric.auger@linaro.org> - 2016-04-07 14:40 +0200
        Re: [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel  part 1/3: iommu changes Alex Williamson <alex.williamson@redhat.com> - 2016-04-07 20:00 +0200
          Re: [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part  1/3: iommu changes Eric Auger <eric.auger@linaro.org> - 2016-04-08 15:40 +0200

#1370430 — [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes

This series introduces the dma-reserved-iommu api used to:
- create/destroy an iova domain dedicated to reserved iova bindings
- map/unmap physical addresses onto reserved IOVAs.
- unmap and destroy all IOVA reserved bindings

Currently reserved IOVAs are meant to map MSI physical doorbells. A single
reserved domain does exit per domain.

Also a new domain attribute is introduced to signal whether the MSI
addresses must be mapped in the IOMMU

VFIO subsystem is supposed to create/destroy the iommu reserved domain.

When the MSI sub-system is about to handle an MSI physical address
that needs to be bound, it uses the dma-reserved_iommu API to map/unmap
the address. Since several drivers are likely to use the same doorbell,
a reference counting must exist on the bindings. An RB-tree indexed by PA
is used.

More details & context can be found at:
http://www.linaro.org/blog/core-dump/kvm-pciemsi-passthrough-armarm64/

Best Regards

Eric

Git: complete series available at
https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.6-rc1-pcie-passthrough-v6

History:

RFC v5 -> patch v6:
- split to ease the review process
- in dma-reserved-api use a spin lock instead of a mutex (reported by
  Jean-Philippe)
- revisit iommu_get_reserved_iova API to pass a size parameter upon
  Marc's request
- Consistently use the page order passed when creating the iova domain.
- init reserved_binding_list (reported by Julien)

RFC v4 -> RFC v5:
- take into account Thomas' comments on MSI related patches
  - split "msi: IOMMU map the doorbell address when needed"
  - increase readability and add comments
  - fix style issues
 - split "iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute"
 - platform ITS now advertises IOMMU_CAP_INTR_REMAP
 - fix compilation issue with CONFIG_IOMMU API unset
 - arm-smmu-v3 now advertises DOMAIN_ATTR_MSI_MAPPING

RFC v3 -> v4:
- Move doorbell mapping/unmapping in msi.c
- fix ref count issue on set_affinity: in case of a change in the address
  the previous address is decremented
- doorbell map/unmap now is done on msi composition. Should allow the use
  case for platform MSI controllers
- create dma-reserved-iommu.h/c exposing/implementing a new API dedicated
  to reserved IOVA management (looking like dma-iommu glue)
- series reordering to ease the review:
  - first part is related to IOMMU
  - second related to MSI sub-system
  - third related to VFIO (except arm-smmu IOMMU_CAP_INTR_REMAP removal)
- expose the number of requested IOVA pages through VFIO_IOMMU_GET_INFO
  [this partially addresses Marc's comments on iommu_get/put_single_reserved
   size/alignment problematic - which I did not ignore - but I don't know
   how much I can do at the moment]

RFC v2 -> RFC v3:
- should fix wrong handling of some CONFIG combinations:
  CONFIG_IOVA, CONFIG_IOMMU_API, CONFIG_PCI_MSI_IRQ_DOMAIN
- fix MSI_FLAG_IRQ_REMAPPING setting in GICv3 ITS (although not tested)

PATCH v1 -> RFC v2:
- reverted to RFC since it looks more reasonable ;-) the code is split
  between VFIO, IOMMU, MSI controller and I am not sure I did the right
  choices. Also API need to be further discussed.
- iova API usage in arm-smmu.c.
- MSI controller natively programs the MSI addr with either the PA or IOVA.
  This is not done anymore in vfio-pci driver as suggested by Alex.
- check irq remapping capability of the group

RFC v1 [2] -> PATCH v1:
- use the existing dma map/unmap ioctl interface with a flag to register a
  reserved IOVA range. Use the legacy Rb to store this special vfio_dma.
- a single reserved IOVA contiguous region now is allowed
- use of an RB tree indexed by PA to store allocated reserved slots
- use of a vfio_domain iova_domain to manage iova allocation within the
  window provided by the userspace
- vfio alloc_map/unmap_free take a vfio_group handle
- vfio_group handle is cached in vfio_pci_device
- add ref counting to bindings
- user modality enabled at the end of the series


Eric Auger (7):
  iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute
  iommu/arm-smmu: advertise DOMAIN_ATTR_MSI_MAPPING attribute
  iommu: introduce a reserved iova cookie
  dma-reserved-iommu: alloc/free_reserved_iova_domain
  dma-reserved-iommu: reserved binding rb-tree and helpers
  dma-reserved-iommu: iommu_get/put_single_reserved
  dma-reserved-iommu: iommu_unmap_reserved

 drivers/iommu/Kconfig              |   8 +
 drivers/iommu/Makefile             |   1 +
 drivers/iommu/arm-smmu-v3.c        |   2 +
 drivers/iommu/arm-smmu.c           |   2 +
 drivers/iommu/dma-reserved-iommu.c | 321 +++++++++++++++++++++++++++++++++++++
 drivers/iommu/iommu.c              |   2 +
 include/linux/dma-reserved-iommu.h |  80 +++++++++
 include/linux/iommu.h              |   7 +
 8 files changed, 423 insertions(+)
 create mode 100644 drivers/iommu/dma-reserved-iommu.c
 create mode 100644 include/linux/dma-reserved-iommu.h

-- 
1.9.1

[toc] | [next] | [standalone]

#1372940 — Re: [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes

On Mon,  4 Apr 2016 08:06:55 +0000
Eric Auger <eric.auger@linaro.org> wrote:

> This series introduces the dma-reserved-iommu api used to:
> - create/destroy an iova domain dedicated to reserved iova bindings
> - map/unmap physical addresses onto reserved IOVAs.
> - unmap and destroy all IOVA reserved bindings

Why are we making the decision to have an unbalanced map vs unmap, we
can create individual mappings, but only unmap the whole thing and
start over?  That's a strange interface.  Thanks,

Alex
 
> Currently reserved IOVAs are meant to map MSI physical doorbells. A single
> reserved domain does exit per domain.
> 
> Also a new domain attribute is introduced to signal whether the MSI
> addresses must be mapped in the IOMMU
> 
> VFIO subsystem is supposed to create/destroy the iommu reserved domain.
> 
> When the MSI sub-system is about to handle an MSI physical address
> that needs to be bound, it uses the dma-reserved_iommu API to map/unmap
> the address. Since several drivers are likely to use the same doorbell,
> a reference counting must exist on the bindings. An RB-tree indexed by PA
> is used.
> 
> More details & context can be found at:
> http://www.linaro.org/blog/core-dump/kvm-pciemsi-passthrough-armarm64/
> 
> Best Regards
> 
> Eric
> 
> Git: complete series available at
> https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.6-rc1-pcie-passthrough-v6
> 
> History:
> 
> RFC v5 -> patch v6:
> - split to ease the review process
> - in dma-reserved-api use a spin lock instead of a mutex (reported by
>   Jean-Philippe)
> - revisit iommu_get_reserved_iova API to pass a size parameter upon
>   Marc's request
> - Consistently use the page order passed when creating the iova domain.
> - init reserved_binding_list (reported by Julien)
> 
> RFC v4 -> RFC v5:
> - take into account Thomas' comments on MSI related patches
>   - split "msi: IOMMU map the doorbell address when needed"
>   - increase readability and add comments
>   - fix style issues
>  - split "iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute"
>  - platform ITS now advertises IOMMU_CAP_INTR_REMAP
>  - fix compilation issue with CONFIG_IOMMU API unset
>  - arm-smmu-v3 now advertises DOMAIN_ATTR_MSI_MAPPING
> 
> RFC v3 -> v4:
> - Move doorbell mapping/unmapping in msi.c
> - fix ref count issue on set_affinity: in case of a change in the address
>   the previous address is decremented
> - doorbell map/unmap now is done on msi composition. Should allow the use
>   case for platform MSI controllers
> - create dma-reserved-iommu.h/c exposing/implementing a new API dedicated
>   to reserved IOVA management (looking like dma-iommu glue)
> - series reordering to ease the review:
>   - first part is related to IOMMU
>   - second related to MSI sub-system
>   - third related to VFIO (except arm-smmu IOMMU_CAP_INTR_REMAP removal)
> - expose the number of requested IOVA pages through VFIO_IOMMU_GET_INFO
>   [this partially addresses Marc's comments on iommu_get/put_single_reserved
>    size/alignment problematic - which I did not ignore - but I don't know
>    how much I can do at the moment]
> 
> RFC v2 -> RFC v3:
> - should fix wrong handling of some CONFIG combinations:
>   CONFIG_IOVA, CONFIG_IOMMU_API, CONFIG_PCI_MSI_IRQ_DOMAIN
> - fix MSI_FLAG_IRQ_REMAPPING setting in GICv3 ITS (although not tested)
> 
> PATCH v1 -> RFC v2:
> - reverted to RFC since it looks more reasonable ;-) the code is split
>   between VFIO, IOMMU, MSI controller and I am not sure I did the right
>   choices. Also API need to be further discussed.
> - iova API usage in arm-smmu.c.
> - MSI controller natively programs the MSI addr with either the PA or IOVA.
>   This is not done anymore in vfio-pci driver as suggested by Alex.
> - check irq remapping capability of the group
> 
> RFC v1 [2] -> PATCH v1:
> - use the existing dma map/unmap ioctl interface with a flag to register a
>   reserved IOVA range. Use the legacy Rb to store this special vfio_dma.
> - a single reserved IOVA contiguous region now is allowed
> - use of an RB tree indexed by PA to store allocated reserved slots
> - use of a vfio_domain iova_domain to manage iova allocation within the
>   window provided by the userspace
> - vfio alloc_map/unmap_free take a vfio_group handle
> - vfio_group handle is cached in vfio_pci_device
> - add ref counting to bindings
> - user modality enabled at the end of the series
> 
> 
> Eric Auger (7):
>   iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute
>   iommu/arm-smmu: advertise DOMAIN_ATTR_MSI_MAPPING attribute
>   iommu: introduce a reserved iova cookie
>   dma-reserved-iommu: alloc/free_reserved_iova_domain
>   dma-reserved-iommu: reserved binding rb-tree and helpers
>   dma-reserved-iommu: iommu_get/put_single_reserved
>   dma-reserved-iommu: iommu_unmap_reserved
> 
>  drivers/iommu/Kconfig              |   8 +
>  drivers/iommu/Makefile             |   1 +
>  drivers/iommu/arm-smmu-v3.c        |   2 +
>  drivers/iommu/arm-smmu.c           |   2 +
>  drivers/iommu/dma-reserved-iommu.c | 321 +++++++++++++++++++++++++++++++++++++
>  drivers/iommu/iommu.c              |   2 +
>  include/linux/dma-reserved-iommu.h |  80 +++++++++
>  include/linux/iommu.h              |   7 +
>  8 files changed, 423 insertions(+)
>  create mode 100644 drivers/iommu/dma-reserved-iommu.c
>  create mode 100644 include/linux/dma-reserved-iommu.h
> 

[toc] | [prev] | [next] | [standalone]

#1373359 — Re: [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes

Hi Alex,
On 04/07/2016 01:15 AM, Alex Williamson wrote:
> On Mon,  4 Apr 2016 08:06:55 +0000
> Eric Auger <eric.auger@linaro.org> wrote:
> 
>> This series introduces the dma-reserved-iommu api used to:
>> - create/destroy an iova domain dedicated to reserved iova bindings
>> - map/unmap physical addresses onto reserved IOVAs.
>> - unmap and destroy all IOVA reserved bindings
> 
> Why are we making the decision to have an unbalanced map vs unmap, we
> can create individual mappings, but only unmap the whole thing and
> start over?  That's a strange interface.  Thanks,
The "individual" balanced unmap also exists (iommu_put_reserved_iova)
and this is the "normal" path. This happens on msi_domain_deactivate
(and possibly on msi_domain_set_affinity).

I added iommu_unmap_reserved to handle the case where the userspace
registers a reserved iova domain and fails to unregister it. In that
case one need to handle the cleanup on kernel-side and I chose to
implement this on vfio_iommu_type1 release. All the reserved IOMMU
bindings get destroyed on that event.

Any advice to handle this situation?

Best Regards

Eric

> 
> Alex
>  
>> Currently reserved IOVAs are meant to map MSI physical doorbells. A single
>> reserved domain does exit per domain.
>>
>> Also a new domain attribute is introduced to signal whether the MSI
>> addresses must be mapped in the IOMMU
>>
>> VFIO subsystem is supposed to create/destroy the iommu reserved domain.
>>
>> When the MSI sub-system is about to handle an MSI physical address
>> that needs to be bound, it uses the dma-reserved_iommu API to map/unmap
>> the address. Since several drivers are likely to use the same doorbell,
>> a reference counting must exist on the bindings. An RB-tree indexed by PA
>> is used.
>>
>> More details & context can be found at:
>> http://www.linaro.org/blog/core-dump/kvm-pciemsi-passthrough-armarm64/
>>
>> Best Regards
>>
>> Eric
>>
>> Git: complete series available at
>> https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.6-rc1-pcie-passthrough-v6
>>
>> History:
>>
>> RFC v5 -> patch v6:
>> - split to ease the review process
>> - in dma-reserved-api use a spin lock instead of a mutex (reported by
>>   Jean-Philippe)
>> - revisit iommu_get_reserved_iova API to pass a size parameter upon
>>   Marc's request
>> - Consistently use the page order passed when creating the iova domain.
>> - init reserved_binding_list (reported by Julien)
>>
>> RFC v4 -> RFC v5:
>> - take into account Thomas' comments on MSI related patches
>>   - split "msi: IOMMU map the doorbell address when needed"
>>   - increase readability and add comments
>>   - fix style issues
>>  - split "iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute"
>>  - platform ITS now advertises IOMMU_CAP_INTR_REMAP
>>  - fix compilation issue with CONFIG_IOMMU API unset
>>  - arm-smmu-v3 now advertises DOMAIN_ATTR_MSI_MAPPING
>>
>> RFC v3 -> v4:
>> - Move doorbell mapping/unmapping in msi.c
>> - fix ref count issue on set_affinity: in case of a change in the address
>>   the previous address is decremented
>> - doorbell map/unmap now is done on msi composition. Should allow the use
>>   case for platform MSI controllers
>> - create dma-reserved-iommu.h/c exposing/implementing a new API dedicated
>>   to reserved IOVA management (looking like dma-iommu glue)
>> - series reordering to ease the review:
>>   - first part is related to IOMMU
>>   - second related to MSI sub-system
>>   - third related to VFIO (except arm-smmu IOMMU_CAP_INTR_REMAP removal)
>> - expose the number of requested IOVA pages through VFIO_IOMMU_GET_INFO
>>   [this partially addresses Marc's comments on iommu_get/put_single_reserved
>>    size/alignment problematic - which I did not ignore - but I don't know
>>    how much I can do at the moment]
>>
>> RFC v2 -> RFC v3:
>> - should fix wrong handling of some CONFIG combinations:
>>   CONFIG_IOVA, CONFIG_IOMMU_API, CONFIG_PCI_MSI_IRQ_DOMAIN
>> - fix MSI_FLAG_IRQ_REMAPPING setting in GICv3 ITS (although not tested)
>>
>> PATCH v1 -> RFC v2:
>> - reverted to RFC since it looks more reasonable ;-) the code is split
>>   between VFIO, IOMMU, MSI controller and I am not sure I did the right
>>   choices. Also API need to be further discussed.
>> - iova API usage in arm-smmu.c.
>> - MSI controller natively programs the MSI addr with either the PA or IOVA.
>>   This is not done anymore in vfio-pci driver as suggested by Alex.
>> - check irq remapping capability of the group
>>
>> RFC v1 [2] -> PATCH v1:
>> - use the existing dma map/unmap ioctl interface with a flag to register a
>>   reserved IOVA range. Use the legacy Rb to store this special vfio_dma.
>> - a single reserved IOVA contiguous region now is allowed
>> - use of an RB tree indexed by PA to store allocated reserved slots
>> - use of a vfio_domain iova_domain to manage iova allocation within the
>>   window provided by the userspace
>> - vfio alloc_map/unmap_free take a vfio_group handle
>> - vfio_group handle is cached in vfio_pci_device
>> - add ref counting to bindings
>> - user modality enabled at the end of the series
>>
>>
>> Eric Auger (7):
>>   iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute
>>   iommu/arm-smmu: advertise DOMAIN_ATTR_MSI_MAPPING attribute
>>   iommu: introduce a reserved iova cookie
>>   dma-reserved-iommu: alloc/free_reserved_iova_domain
>>   dma-reserved-iommu: reserved binding rb-tree and helpers
>>   dma-reserved-iommu: iommu_get/put_single_reserved
>>   dma-reserved-iommu: iommu_unmap_reserved
>>
>>  drivers/iommu/Kconfig              |   8 +
>>  drivers/iommu/Makefile             |   1 +
>>  drivers/iommu/arm-smmu-v3.c        |   2 +
>>  drivers/iommu/arm-smmu.c           |   2 +
>>  drivers/iommu/dma-reserved-iommu.c | 321 +++++++++++++++++++++++++++++++++++++
>>  drivers/iommu/iommu.c              |   2 +
>>  include/linux/dma-reserved-iommu.h |  80 +++++++++
>>  include/linux/iommu.h              |   7 +
>>  8 files changed, 423 insertions(+)
>>  create mode 100644 drivers/iommu/dma-reserved-iommu.c
>>  create mode 100644 include/linux/dma-reserved-iommu.h
>>
> 

[toc] | [prev] | [next] | [standalone]

#1373610 — Re: [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes

On Thu, 7 Apr 2016 14:28:59 +0200
Eric Auger <eric.auger@linaro.org> wrote:

> Hi Alex,
> On 04/07/2016 01:15 AM, Alex Williamson wrote:
> > On Mon,  4 Apr 2016 08:06:55 +0000
> > Eric Auger <eric.auger@linaro.org> wrote:
> >   
> >> This series introduces the dma-reserved-iommu api used to:
> >> - create/destroy an iova domain dedicated to reserved iova bindings
> >> - map/unmap physical addresses onto reserved IOVAs.
> >> - unmap and destroy all IOVA reserved bindings  
> > 
> > Why are we making the decision to have an unbalanced map vs unmap, we
> > can create individual mappings, but only unmap the whole thing and
> > start over?  That's a strange interface.  Thanks,  
> The "individual" balanced unmap also exists (iommu_put_reserved_iova)
> and this is the "normal" path. This happens on msi_domain_deactivate
> (and possibly on msi_domain_set_affinity).
> 
> I added iommu_unmap_reserved to handle the case where the userspace
> registers a reserved iova domain and fails to unregister it. In that
> case one need to handle the cleanup on kernel-side and I chose to
> implement this on vfio_iommu_type1 release. All the reserved IOMMU
> bindings get destroyed on that event.
> 
> Any advice to handle this situation?

If we want to model it similar to regular iommu domains, then
iommu_free_reserved_iova_domain() should release all the mappings and
destroy the iova domain.  Additionally, since the reserved iova domain
is just a construct on top of an iommu domain, it should be sufficient
to call iommu_domain_free() to also remove the reserved iova domain if
one exists.  Thanks,

Alex

[toc] | [prev] | [next] | [standalone]

#1374238 — Re: [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes

Hi Alex,
On 04/07/2016 07:50 PM, Alex Williamson wrote:
> On Thu, 7 Apr 2016 14:28:59 +0200
> Eric Auger <eric.auger@linaro.org> wrote:
> 
>> Hi Alex,
>> On 04/07/2016 01:15 AM, Alex Williamson wrote:
>>> On Mon,  4 Apr 2016 08:06:55 +0000
>>> Eric Auger <eric.auger@linaro.org> wrote:
>>>   
>>>> This series introduces the dma-reserved-iommu api used to:
>>>> - create/destroy an iova domain dedicated to reserved iova bindings
>>>> - map/unmap physical addresses onto reserved IOVAs.
>>>> - unmap and destroy all IOVA reserved bindings  
>>>
>>> Why are we making the decision to have an unbalanced map vs unmap, we
>>> can create individual mappings, but only unmap the whole thing and
>>> start over?  That's a strange interface.  Thanks,  
>> The "individual" balanced unmap also exists (iommu_put_reserved_iova)
>> and this is the "normal" path. This happens on msi_domain_deactivate
>> (and possibly on msi_domain_set_affinity).
>>
>> I added iommu_unmap_reserved to handle the case where the userspace
>> registers a reserved iova domain and fails to unregister it. In that
>> case one need to handle the cleanup on kernel-side and I chose to
>> implement this on vfio_iommu_type1 release. All the reserved IOMMU
>> bindings get destroyed on that event.
>>
>> Any advice to handle this situation?
> 
> If we want to model it similar to regular iommu domains, then
> iommu_free_reserved_iova_domain() should release all the mappings and
> destroy the iova domain.
Yes this sounds obvious now.
  Additionally, since the reserved iova domain
> is just a construct on top of an iommu domain, it should be sufficient
> to call iommu_domain_free() to also remove the reserved iova domain if
> one exists.  Thanks,
Yes. For dma cookie (iommu_put_dma_cookie) I see this is done from the
iommu driver domain_free callback.

Thanks

Eric
> 
> Alex
> 

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web