Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1321089 > unrolled thread

Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 Alex Williamson <alex.williamson@redhat.com> - 2016-01-28 23:00 +0100
    Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 Eric Auger <eric.auger@linaro.org> - 2016-01-29 15:40 +0100
      Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 Alex Williamson <alex.williamson@redhat.com> - 2016-01-29 20:40 +0100
        Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 Eric Auger <eric.auger@linaro.org> - 2016-01-29 22:30 +0100
          Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 Will Deacon <will.deacon@arm.com> - 2016-02-01 15:10 +0100
            Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 Christoffer Dall <christoffer.dall@linaro.org> - 2016-02-03 14:00 +0100
              Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 Will Deacon <will.deacon@arm.com> - 2016-02-03 14:20 +0100
                Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 Christoffer Dall <christoffer.dall@linaro.org> - 2016-02-03 16:40 +0100
                  ARM PCI/MSI KVM passthrough with GICv2M Eric Auger <eric.auger@linaro.org> - 2016-02-05 18:40 +0100
                    Re: ARM PCI/MSI KVM passthrough with GICv2M Alex Williamson <alex.williamson@redhat.com> - 2016-02-05 19:20 +0100
                      Re: ARM PCI/MSI KVM passthrough with GICv2M Christoffer Dall <christoffer.dall@linaro.org> - 2016-02-08 10:50 +0100
                        Re: ARM PCI/MSI KVM passthrough with GICv2M Eric Auger <eric.auger@linaro.org> - 2016-02-08 14:30 +0100

#1321089 — Re: [PATCH 00/10] KVM PCIe/MSI passthrough on ARM/ARM64

On Tue, 2016-01-26 at 13:12 +0000, Eric Auger wrote:
> This series addresses KVM PCIe passthrough with MSI enabled on ARM/ARM64.
> It pursues the efforts done on [1], [2], [3]. It also aims at covering the
> same need on some PowerPC platforms.
> 
> On x86 all accesses to the 1MB PA region [FEE0_0000h - FEF0_000h] are directed
> as interrupt messages: accesses to this special PA window directly target the
> APIC configuration space and not DRAM, meaning the downstream IOMMU is bypassed.
> 
> This is not the case on above mentionned platforms where MSI messages emitted
> by devices are conveyed through the IOMMU. This means an IOVA/host PA mapping
> must exist for the MSI to reach the MSI controller. Normal way to create
> IOVA bindings consists in using VFIO DMA MAP API. However in this case
> the MSI IOVA is not mapped onto guest RAM but on host physical page (the MSI
> controller frame).
> 
> Following first comments, the spirit of [2] is kept: the guest registers
> an IOVA range reserved for MSI mapping. When the VFIO-PCIe driver allocates
> its MSI vectors, it overwrites the MSI controller physical address with an IOVA,
> allocated within the window provided by the userspace. This IOVA is mapped
> onto the MSI controller frame physical page.
> 
> The series does not address yet the problematic of telling the userspace how
> much IOVA he should provision.

I'm sort of on a think-different approach today, so bear with me; how is
it that x86 can make interrupt remapping so transparent to drivers like
vfio-pci while for ARM and ppc we seem to be stuck with doing these
fixups of the physical vector ourselves, implying ugly (no offense)
paths bouncing through vfio to connect the driver and iommu backends?

We know that x86 handles MSI vectors specially, so there is some
hardware that helps the situation.  It's not just that x86 has a fixed
range for MSI, it's how it manages that range when interrupt remapping
hardware is enabled.  A device table indexed by source-ID references a
per device table indexed by data from the MSI write itself.  So we get
much, much finer granularity, but there's still effectively an interrupt
domain per device that's being transparently managed under the covers
whenever we request an MSI vector for a device.

So why can't we do something more like that here?  There's no predefined
MSI vector range, so defining an interface for the user to specify that
is unavoidable.  But why shouldn't everything else be transparent?  We
could add an interface to the IOMMU API that allows us to register that
reserved range for the IOMMU domain.  IOMMU-core (or maybe interrupt
remapping) code might allocate an IOVA domain for this just as you've
done in the type1 code here.  But rather than having any interaction
with vfio-pci, why not do this at lower levels such that the platform
interrupt vector allocation code automatically uses one of those IOVA
ranges and returns the IOVA rather than the physical address for the PCI
code to program into the device?  I think we know what needs to be done,
but we're taking the approach of managing the space ourselves and doing
a fixup of the device after the core code has done its job when we
really ought to be letting the core code manage a space that we define
and programming the device so that it doesn't need a fixup in the
vfio-pci code.  Wouldn't it be nicer if pci_enable_msix_range() returned
with the device properly programmed or generate an error if there's not
enough reserved mapping space in IOMMU domain?  Can it be done?  Thanks,

Alex

[toc] | [next] | [standalone]

#1321794

Hi Alex,
On 01/28/2016 10:51 PM, Alex Williamson wrote:
> On Tue, 2016-01-26 at 13:12 +0000, Eric Auger wrote:
>> This series addresses KVM PCIe passthrough with MSI enabled on ARM/ARM64.
>> It pursues the efforts done on [1], [2], [3]. It also aims at covering the
>> same need on some PowerPC platforms.
>>  
>> On x86 all accesses to the 1MB PA region [FEE0_0000h - FEF0_000h] are directed
>> as interrupt messages: accesses to this special PA window directly target the
>> APIC configuration space and not DRAM, meaning the downstream IOMMU is bypassed.
>>  
>> This is not the case on above mentionned platforms where MSI messages emitted
>> by devices are conveyed through the IOMMU. This means an IOVA/host PA mapping
>> must exist for the MSI to reach the MSI controller. Normal way to create
>> IOVA bindings consists in using VFIO DMA MAP API. However in this case
>> the MSI IOVA is not mapped onto guest RAM but on host physical page (the MSI
>> controller frame).
>>  
>> Following first comments, the spirit of [2] is kept: the guest registers
>> an IOVA range reserved for MSI mapping. When the VFIO-PCIe driver allocates
>> its MSI vectors, it overwrites the MSI controller physical address with an IOVA,
>> allocated within the window provided by the userspace. This IOVA is mapped
>> onto the MSI controller frame physical page.
>>  
>> The series does not address yet the problematic of telling the userspace how
>> much IOVA he should provision.
> 
> I'm sort of on a think-different approach today, so bear with me; how is
> it that x86 can make interrupt remapping so transparent to drivers like
> vfio-pci while for ARM and ppc we seem to be stuck with doing these
> fixups of the physical vector ourselves, implying ugly (no offense)
> paths bouncing through vfio to connect the driver and iommu backends?
> 
> We know that x86 handles MSI vectors specially, so there is some
> hardware that helps the situation.  It's not just that x86 has a fixed
> range for MSI, it's how it manages that range when interrupt remapping
> hardware is enabled.  A device table indexed by source-ID references a
> per device table indexed by data from the MSI write itself.  So we get
> much, much finer granularity,
About the granularity, I think ARM GICv3 now provides a similar
capability with GICv3 ITS (interrupt translation service). Along with
the MSI MSG write transaction, the device outputs a DeviceID conveyed on
the bus. This DeviceID (~ your source-ID) enables to index a device
table. The entry in the device table points to a DeviceId interrupt
translation table indexed by the EventID found in the msi msg. So the
entry in the interrupt translation table eventually gives you the
eventual interrupt ID targeted by the MSI MSG.
This translation capability if not available in GICv2M though, ie. the
one I am currently using.

Those tables currently are built by the ITS irqchip (irq-gic-v3-its.c)

 but there's still effectively an interrupt
> domain per device that's being transparently managed under the covers
> whenever we request an MSI vector for a device.
> 
> So why can't we do something more like that here?  There's no predefined
> MSI vector range, so defining an interface for the user to specify that
> is unavoidable.
Do you confirm that VFIO user API still still is the good choice to
provide that IOVA range?
  But why shouldn't everything else be transparent?  We
> could add an interface to the IOMMU API that allows us to register that
> reserved range for the IOMMU domain.  IOMMU-core (or maybe interrupt
> remapping) code might allocate an IOVA domain for this just as you've
> done in the type1 code here.
I have no objection to move that iova allocation scheme somewhere else.
I just need to figure out how to deal with the fact iova.c is not
compiled everywhere as I noticed too late ;-)

  But rather than having any interaction
> with vfio-pci, why not do this at lower levels such that the platform
> interrupt vector allocation code automatically uses one of those IOVA
> ranges and returns the IOVA rather than the physical address for the PCI
> code to program into the device?  I think we know what needs to be done,
> but we're taking the approach of managing the space ourselves and doing
> a fixup of the device after the core code has done its job when we
> really ought to be letting the core code manage a space that we define
> and programming the device so that it doesn't need a fixup in the
> vfio-pci code.  Wouldn't it be nicer if pci_enable_msix_range() returned
> with the device properly programmed or generate an error if there's not
> enough reserved mapping space in IOMMU domain?  Can it be done?
I agree with you on the fact it would be cleaner to manage that natively
at MSI controller level instead of patching the address value in
vfio_pci_intrs.c. I will investigate in that direction but I need some
more time to understand the links between the MSI controller, the PCI
device and the IOMMU.

Best Regards

Eric

  Thanks,
> 
> Alex
> 

[toc] | [prev] | [next] | [standalone]

#1322037

On Fri, 2016-01-29 at 15:35 +0100, Eric Auger wrote:
> Hi Alex,
> On 01/28/2016 10:51 PM, Alex Williamson wrote:
> > On Tue, 2016-01-26 at 13:12 +0000, Eric Auger wrote:
> > > This series addresses KVM PCIe passthrough with MSI enabled on ARM/ARM64.
> > > It pursues the efforts done on [1], [2], [3]. It also aims at covering the
> > > same need on some PowerPC platforms.
> > >  
> > > On x86 all accesses to the 1MB PA region [FEE0_0000h - FEF0_000h] are directed
> > > as interrupt messages: accesses to this special PA window directly target the
> > > APIC configuration space and not DRAM, meaning the downstream IOMMU is bypassed.
> > >  
> > > This is not the case on above mentionned platforms where MSI messages emitted
> > > by devices are conveyed through the IOMMU. This means an IOVA/host PA mapping
> > > must exist for the MSI to reach the MSI controller. Normal way to create
> > > IOVA bindings consists in using VFIO DMA MAP API. However in this case
> > > the MSI IOVA is not mapped onto guest RAM but on host physical page (the MSI
> > > controller frame).
> > >  
> > > Following first comments, the spirit of [2] is kept: the guest registers
> > > an IOVA range reserved for MSI mapping. When the VFIO-PCIe driver allocates
> > > its MSI vectors, it overwrites the MSI controller physical address with an IOVA,
> > > allocated within the window provided by the userspace. This IOVA is mapped
> > > onto the MSI controller frame physical page.
> > >  
> > > The series does not address yet the problematic of telling the userspace how
> > > much IOVA he should provision.
> > 
> > I'm sort of on a think-different approach today, so bear with me; how is
> > it that x86 can make interrupt remapping so transparent to drivers like
> > vfio-pci while for ARM and ppc we seem to be stuck with doing these
> > fixups of the physical vector ourselves, implying ugly (no offense)
> > paths bouncing through vfio to connect the driver and iommu backends?
> > 
> > We know that x86 handles MSI vectors specially, so there is some
> > hardware that helps the situation.  It's not just that x86 has a fixed
> > range for MSI, it's how it manages that range when interrupt remapping
> > hardware is enabled.  A device table indexed by source-ID references a
> > per device table indexed by data from the MSI write itself.  So we get
> > much, much finer granularity,
> About the granularity, I think ARM GICv3 now provides a similar
> capability with GICv3 ITS (interrupt translation service). Along with
> the MSI MSG write transaction, the device outputs a DeviceID conveyed on
> the bus. This DeviceID (~ your source-ID) enables to index a device
> table. The entry in the device table points to a DeviceId interrupt
> translation table indexed by the EventID found in the msi msg. So the
> entry in the interrupt translation table eventually gives you the
> eventual interrupt ID targeted by the MSI MSG.
> This translation capability if not available in GICv2M though, ie. the
> one I am currently using.
> 
> Those tables currently are built by the ITS irqchip (irq-gic-v3-its.c)

So it sounds like the interrupt remapping plumbing needs to be
implemented for those chips.  How does ITS identify an MSI versus any
other DMA write?  Does it need to be within a preconfigured address
space like on x86 or does it know this implicitly by the transaction
(which doesn't seem possible on PCIe)?

Along with this discussion, we should probably be revisiting whether
existing ARM SMMUs should be exposing the IOMMU_CAP_INTR_REMAP
capability.  This capability is meant to indicate interrupt isolation,
but if an entire page of IOVA space is mapped through the IOMMU to a
range of interrupts and some of those interrupts are shared with host
devices or other VMs, then we really don't have that isolation and the
system is susceptible to one VM interfering with another or with the
host.  If that's the case, the SMMU should not be claiming
IOMMU_CAP_INTR_REMAP.

>  but there's still effectively an interrupt
> > domain per device that's being transparently managed under the covers
> > whenever we request an MSI vector for a device.
> > 
> > So why can't we do something more like that here?  There's no predefined
> > MSI vector range, so defining an interface for the user to specify that
> > is unavoidable.
> Do you confirm that VFIO user API still still is the good choice to
> provide that IOVA range?

I don't see that we have an option there unless ARM wants to
retroactively reserve a range of IOVA space in the spec, which is
certainly not going to happen.  The only other thing that comes to mind
would be if there was an existing address space which could never be
backed by RAM or other DMA capable targets.  But that seems far fetched
as well.

>   But why shouldn't everything else be transparent?  We
> > could add an interface to the IOMMU API that allows us to register that
> > reserved range for the IOMMU domain.  IOMMU-core (or maybe interrupt
> > remapping) code might allocate an IOVA domain for this just as you've
> > done in the type1 code here.
> I have no objection to move that iova allocation scheme somewhere else.
> I just need to figure out how to deal with the fact iova.c is not
> compiled everywhere as I noticed too late ;-)
>   But rather than having any interaction
> > with vfio-pci, why not do this at lower levels such that the platform
> > interrupt vector allocation code automatically uses one of those IOVA
> > ranges and returns the IOVA rather than the physical address for the PCI
> > code to program into the device?  I think we know what needs to be done,
> > but we're taking the approach of managing the space ourselves and doing
> > a fixup of the device after the core code has done its job when we
> > really ought to be letting the core code manage a space that we define
> > and programming the device so that it doesn't need a fixup in the
> > vfio-pci code.  Wouldn't it be nicer if pci_enable_msix_range() returned
> > with the device properly programmed or generate an error if there's not
> > enough reserved mapping space in IOMMU domain?  Can it be done?
> I agree with you on the fact it would be cleaner to manage that natively
> at MSI controller level instead of patching the address value in
> vfio_pci_intrs.c. I will investigate in that direction but I need some
> more time to understand the links between the MSI controller, the PCI
> device and the IOMMU.

Since the current interrupt remapping schemes seem to operate in a
different address space, I expect there will be work to do to fit the
interrupt remapping within a provided address space, but it seems like a
very reasonable constraint to add.  Thanks,

Alex

[toc] | [prev] | [next] | [standalone]

#1322088

Hi Alex,
On 01/29/2016 08:33 PM, Alex Williamson wrote:
> On Fri, 2016-01-29 at 15:35 +0100, Eric Auger wrote:
>> Hi Alex,
>> On 01/28/2016 10:51 PM, Alex Williamson wrote:
>>> On Tue, 2016-01-26 at 13:12 +0000, Eric Auger wrote:
>>>> This series addresses KVM PCIe passthrough with MSI enabled on ARM/ARM64.
>>>> It pursues the efforts done on [1], [2], [3]. It also aims at covering the
>>>> same need on some PowerPC platforms.
>>>>  
>>>> On x86 all accesses to the 1MB PA region [FEE0_0000h - FEF0_000h] are directed
>>>> as interrupt messages: accesses to this special PA window directly target the
>>>> APIC configuration space and not DRAM, meaning the downstream IOMMU is bypassed.
>>>>  
>>>> This is not the case on above mentionned platforms where MSI messages emitted
>>>> by devices are conveyed through the IOMMU. This means an IOVA/host PA mapping
>>>> must exist for the MSI to reach the MSI controller. Normal way to create
>>>> IOVA bindings consists in using VFIO DMA MAP API. However in this case
>>>> the MSI IOVA is not mapped onto guest RAM but on host physical page (the MSI
>>>> controller frame).
>>>>  
>>>> Following first comments, the spirit of [2] is kept: the guest registers
>>>> an IOVA range reserved for MSI mapping. When the VFIO-PCIe driver allocates
>>>> its MSI vectors, it overwrites the MSI controller physical address with an IOVA,
>>>> allocated within the window provided by the userspace. This IOVA is mapped
>>>> onto the MSI controller frame physical page.
>>>>  
>>>> The series does not address yet the problematic of telling the userspace how
>>>> much IOVA he should provision.
>>>  
>>> I'm sort of on a think-different approach today, so bear with me; how is
>>> it that x86 can make interrupt remapping so transparent to drivers like
>>> vfio-pci while for ARM and ppc we seem to be stuck with doing these
>>> fixups of the physical vector ourselves, implying ugly (no offense)
>>> paths bouncing through vfio to connect the driver and iommu backends?
>>>  
>>> We know that x86 handles MSI vectors specially, so there is some
>>> hardware that helps the situation.  It's not just that x86 has a fixed
>>> range for MSI, it's how it manages that range when interrupt remapping
>>> hardware is enabled.  A device table indexed by source-ID references a
>>> per device table indexed by data from the MSI write itself.  So we get
>>> much, much finer granularity,
>> About the granularity, I think ARM GICv3 now provides a similar
>> capability with GICv3 ITS (interrupt translation service). Along with
>> the MSI MSG write transaction, the device outputs a DeviceID conveyed on
>> the bus. This DeviceID (~ your source-ID) enables to index a device
>> table. The entry in the device table points to a DeviceId interrupt
>> translation table indexed by the EventID found in the msi msg. So the
>> entry in the interrupt translation table eventually gives you the
>> eventual interrupt ID targeted by the MSI MSG.
>> This translation capability if not available in GICv2M though, ie. the
>> one I am currently using.
>>  
>> Those tables currently are built by the ITS irqchip (irq-gic-v3-its.c)
> 
> So it sounds like the interrupt remapping plumbing needs to be
> implemented for those chips.  How does ITS identify an MSI versus any
> other DMA write?  Does it need to be within a preconfigured address
> space like on x86 or does it know this implicitly by the transaction
> (which doesn't seem possible on PCIe)?

It seems there is a kind of misunderstanding here. Assuming a "simple"
system with a single ITS, all devices likely to produce MSI must write
those messages in a single register, located in the ITS MSI 64kB frame
(this register is called GITS_TRANSLATER). Then the ITS discriminates
between senders using the DeviceID conveyed out-of-band on the bus (or
by other implementation defined means). For those DeviceId, a deviceId
interrupt translation table is supposed to exist, else it is going to
fault. If any "undeclared" device is writing into that register, its
deviceid will be unknown. It looks like on Intel the interrupt remapping
HW rather is abstracted on the IOMMU side; I did not take time yet to
carefully read the VT-d spec but maybe the Intel interrupt remapping HW
rather acts as an IOMMU that takes an input MSI address within the
famous window and apply a translation scheme based on the MSI address &
data? On ARM the input MSI address always is the GITS_TRANSLATER and
then the translation scheme is based on out-of-band info (deviceid) +
data content(eventid). I Hope this clarifies.
> 
> Along with this discussion, we should probably be revisiting whether
> existing ARM SMMUs should be exposing the IOMMU_CAP_INTR_REMAP
> capability.

so according to the above explanation not sure it is relevant. Will/Marc
might correct me if I told some wrong things.
  This capability is meant to indicate interrupt isolation,
> but if an entire page of IOVA space is mapped through the IOMMU to a
> range of interrupts and some of those interrupts are shared with host
> devices or other VMs, then we really don't have that isolation and the
> system is susceptible to one VM interfering with another or with the
> host.  If that's the case, the SMMU should not be claiming
> IOMMU_CAP_INTR_REMAP.
My understanding is a PCI device working for the host must have its own
deviceid translation table while another one assigned to a guest needs
to have another one. Each of those will then trigger different final
interrupt IDs in separate domains.

To be honest for the time being I was not addressing the ITS case but
just the simpler GICv2m case where we do not have interrupt translation.
In GICv2m with a single 4kB MSI frame you still have a single register
written by devices. The msg data content then induces a given interrupt ID.

My kernel series "just" aimed at allowing the device to reach the
physical address of the GICv2m MSI frame through the IOMMU.

But you're right here I think I should have a larger vision of what is
targeted with ITS. In GICv2m with a single MSI frame the discrimination
only works on the msi data (there is no deviceid). However it is also
possible to have several GICv2M MSI 4kB frames and in that case you can
give 1 MSI 4kB frame per VM but it is yet another use case. My AMD
system currently exposes a single MSI frame - in which case we have poor
isolation as you say -.
> 
>>  but there's still effectively an interrupt
>>> domain per device that's being transparently managed under the covers
>>> whenever we request an MSI vector for a device.
>>>  
>>> So why can't we do something more like that here?  There's no predefined
>>> MSI vector range, so defining an interface for the user to specify that
>>> is unavoidable.
>> Do you confirm that VFIO user API still still is the good choice to
>> provide that IOVA range?
> 
> I don't see that we have an option there unless ARM wants to
> retroactively reserve a range of IOVA space in the spec, which is
> certainly not going to happen.  The only other thing that comes to mind
> would be if there was an existing address space which could never be
> backed by RAM or other DMA capable targets.  But that seems far fetched
> as well.
I don't think there is a plan for such change and I am afraid we need to
integrate above configurations (GICv2M with a single frame, GICv2M with
several frames, ITS and there may be others not covered here that I am
not aware of).
> 
>>   But why shouldn't everything else be transparent?  We
>>> could add an interface to the IOMMU API that allows us to register that
>>> reserved range for the IOMMU domain.  IOMMU-core (or maybe interrupt
>>> remapping) code might allocate an IOVA domain for this just as you've
>>> done in the type1 code here.
>> I have no objection to move that iova allocation scheme somewhere else.
>> I just need to figure out how to deal with the fact iova.c is not
>> compiled everywhere as I noticed too late ;-)
>>   But rather than having any interaction
>>> with vfio-pci, why not do this at lower levels such that the platform
>>> interrupt vector allocation code automatically uses one of those IOVA
>>> ranges and returns the IOVA rather than the physical address for the PCI
>>> code to program into the device?  I think we know what needs to be done,
>>> but we're taking the approach of managing the space ourselves and doing
>>> a fixup of the device after the core code has done its job when we
>>> really ought to be letting the core code manage a space that we define
>>> and programming the device so that it doesn't need a fixup in the
>>> vfio-pci code.  Wouldn't it be nicer if pci_enable_msix_range() returned
>>> with the device properly programmed or generate an error if there's not
>>> enough reserved mapping space in IOMMU domain?  Can it be done?
>> I agree with you on the fact it would be cleaner to manage that natively
>> at MSI controller level instead of patching the address value in
>> vfio_pci_intrs.c. I will investigate in that direction but I need some
>> more time to understand the links between the MSI controller, the PCI
>> device and the IOMMU.
> 
> Since the current interrupt remapping schemes seem to operate in a
> different address space, I expect there will be work to do to fit the
> interrupt remapping within a provided address space, but it seems like a
> very reasonable constraint to add.  Thanks,

I hope this discussion will help. Please ARM guys, correct me if there
are some unclarities or wrong statements.

Thank you for reading up to here and have a nice WE!

Best Regards

Eric
> 
> Alex
> 

[toc] | [prev] | [next] | [standalone]

#1323152

On Fri, Jan 29, 2016 at 10:25:52PM +0100, Eric Auger wrote:
> On 01/29/2016 08:33 PM, Alex Williamson wrote:
> >>> We know that x86 handles MSI vectors specially, so there is some
> >>> hardware that helps the situation.  It's not just that x86 has a fixed
> >>> range for MSI, it's how it manages that range when interrupt remapping
> >>> hardware is enabled.  A device table indexed by source-ID references a
> >>> per device table indexed by data from the MSI write itself.  So we get
> >>> much, much finer granularity,
> >> About the granularity, I think ARM GICv3 now provides a similar
> >> capability with GICv3 ITS (interrupt translation service). Along with
> >> the MSI MSG write transaction, the device outputs a DeviceID conveyed on
> >> the bus. This DeviceID (~ your source-ID) enables to index a device
> >> table. The entry in the device table points to a DeviceId interrupt
> >> translation table indexed by the EventID found in the msi msg. So the
> >> entry in the interrupt translation table eventually gives you the
> >> eventual interrupt ID targeted by the MSI MSG.
> >> This translation capability if not available in GICv2M though, ie. the
> >> one I am currently using.
> >>  
> >> Those tables currently are built by the ITS irqchip (irq-gic-v3-its.c)

That's right. GICv3/ITS disambiguates the interrupt source using the
DeviceID, which for PCI is derived from the Requester ID of the endpoint.
GICv2m is less flexible and requires a separate physical frame per guest
to achieve isolation.

Will

[toc] | [prev] | [next] | [standalone]

#1325387

On Mon, Feb 01, 2016 at 02:03:51PM +0000, Will Deacon wrote:
> On Fri, Jan 29, 2016 at 10:25:52PM +0100, Eric Auger wrote:
> > On 01/29/2016 08:33 PM, Alex Williamson wrote:
> > >>> We know that x86 handles MSI vectors specially, so there is some
> > >>> hardware that helps the situation.  It's not just that x86 has a fixed
> > >>> range for MSI, it's how it manages that range when interrupt remapping
> > >>> hardware is enabled.  A device table indexed by source-ID references a
> > >>> per device table indexed by data from the MSI write itself.  So we get
> > >>> much, much finer granularity,
> > >> About the granularity, I think ARM GICv3 now provides a similar
> > >> capability with GICv3 ITS (interrupt translation service). Along with
> > >> the MSI MSG write transaction, the device outputs a DeviceID conveyed on
> > >> the bus. This DeviceID (~ your source-ID) enables to index a device
> > >> table. The entry in the device table points to a DeviceId interrupt
> > >> translation table indexed by the EventID found in the msi msg. So the
> > >> entry in the interrupt translation table eventually gives you the
> > >> eventual interrupt ID targeted by the MSI MSG.
> > >> This translation capability if not available in GICv2M though, ie. the
> > >> one I am currently using.
> > >>  
> > >> Those tables currently are built by the ITS irqchip (irq-gic-v3-its.c)
> 
> That's right. GICv3/ITS disambiguates the interrupt source using the
> DeviceID, which for PCI is derived from the Requester ID of the endpoint.
> GICv2m is less flexible and requires a separate physical frame per guest
> to achieve isolation.
> 
We should still support MSI passthrough with a single MSI frame host
system though, right?

(Users should just be aware that guests are not fully protected against
misbehaving hardware in that case).

-Christoffer

[toc] | [prev] | [next] | [standalone]

#1325399

On Wed, Feb 03, 2016 at 01:50:47PM +0100, Christoffer Dall wrote:
> On Mon, Feb 01, 2016 at 02:03:51PM +0000, Will Deacon wrote:
> > On Fri, Jan 29, 2016 at 10:25:52PM +0100, Eric Auger wrote:
> > > On 01/29/2016 08:33 PM, Alex Williamson wrote:
> > > >>> We know that x86 handles MSI vectors specially, so there is some
> > > >>> hardware that helps the situation.  It's not just that x86 has a fixed
> > > >>> range for MSI, it's how it manages that range when interrupt remapping
> > > >>> hardware is enabled.  A device table indexed by source-ID references a
> > > >>> per device table indexed by data from the MSI write itself.  So we get
> > > >>> much, much finer granularity,
> > > >> About the granularity, I think ARM GICv3 now provides a similar
> > > >> capability with GICv3 ITS (interrupt translation service). Along with
> > > >> the MSI MSG write transaction, the device outputs a DeviceID conveyed on
> > > >> the bus. This DeviceID (~ your source-ID) enables to index a device
> > > >> table. The entry in the device table points to a DeviceId interrupt
> > > >> translation table indexed by the EventID found in the msi msg. So the
> > > >> entry in the interrupt translation table eventually gives you the
> > > >> eventual interrupt ID targeted by the MSI MSG.
> > > >> This translation capability if not available in GICv2M though, ie. the
> > > >> one I am currently using.
> > > >>  
> > > >> Those tables currently are built by the ITS irqchip (irq-gic-v3-its.c)
> > 
> > That's right. GICv3/ITS disambiguates the interrupt source using the
> > DeviceID, which for PCI is derived from the Requester ID of the endpoint.
> > GICv2m is less flexible and requires a separate physical frame per guest
> > to achieve isolation.
> > 
> We should still support MSI passthrough with a single MSI frame host
> system though, right?

I think we should treat the frame as an exclusive resource and assign it
to a single VM.

> (Users should just be aware that guests are not fully protected against
> misbehaving hardware in that case).

Is it confined to misbehaving hardware? What if a malicious/buggy guest
configures its device to DMA all over the doorbell?

Will

[toc] | [prev] | [next] | [standalone]

#1325545

On Wed, Feb 03, 2016 at 01:10:58PM +0000, Will Deacon wrote:
> On Wed, Feb 03, 2016 at 01:50:47PM +0100, Christoffer Dall wrote:
> > On Mon, Feb 01, 2016 at 02:03:51PM +0000, Will Deacon wrote:
> > > On Fri, Jan 29, 2016 at 10:25:52PM +0100, Eric Auger wrote:
> > > > On 01/29/2016 08:33 PM, Alex Williamson wrote:
> > > > >>> We know that x86 handles MSI vectors specially, so there is some
> > > > >>> hardware that helps the situation.  It's not just that x86 has a fixed
> > > > >>> range for MSI, it's how it manages that range when interrupt remapping
> > > > >>> hardware is enabled.  A device table indexed by source-ID references a
> > > > >>> per device table indexed by data from the MSI write itself.  So we get
> > > > >>> much, much finer granularity,
> > > > >> About the granularity, I think ARM GICv3 now provides a similar
> > > > >> capability with GICv3 ITS (interrupt translation service). Along with
> > > > >> the MSI MSG write transaction, the device outputs a DeviceID conveyed on
> > > > >> the bus. This DeviceID (~ your source-ID) enables to index a device
> > > > >> table. The entry in the device table points to a DeviceId interrupt
> > > > >> translation table indexed by the EventID found in the msi msg. So the
> > > > >> entry in the interrupt translation table eventually gives you the
> > > > >> eventual interrupt ID targeted by the MSI MSG.
> > > > >> This translation capability if not available in GICv2M though, ie. the
> > > > >> one I am currently using.
> > > > >>  
> > > > >> Those tables currently are built by the ITS irqchip (irq-gic-v3-its.c)
> > > 
> > > That's right. GICv3/ITS disambiguates the interrupt source using the
> > > DeviceID, which for PCI is derived from the Requester ID of the endpoint.
> > > GICv2m is less flexible and requires a separate physical frame per guest
> > > to achieve isolation.
> > > 
> > We should still support MSI passthrough with a single MSI frame host
> > system though, right?
> 
> I think we should treat the frame as an exclusive resource and assign it
> to a single VM.

so on a single frame GICv2m system, either your host or a single VM gets
to do MSIs...

> 
> > (Users should just be aware that guests are not fully protected against
> > misbehaving hardware in that case).
> 
> Is it confined to misbehaving hardware? What if a malicious/buggy guest
> configures its device to DMA all over the doorbell?
> 
I guess not, I suppose we can't trap any configuration access and
mediate that for any device.  Bummer.

-Christoffer

[toc] | [prev] | [next] | [standalone]

#1327961 — ARM PCI/MSI KVM passthrough with GICv2M

Hi Alex,

I tried to sketch a proposal for guaranteeing the IRQ integrity when
doing ARM PCI/MSI passthrough with ARM GICv2M msi-controller. This is
based on extended VFIO group viability control, as detailed below.

As opposed to ARM GICv3 ITS, this MSI controller does *not* support IRQ
remapping. It can expose 1 or more 4kB MSI frame. Each frame contains a
single register where the msi data is written.

I would be grateful to you if you could tell me whether it makes any sense.

Thanks in advance

Best Regards

Eric


1) GICv2m with a single 4kB single frame
   all devices having this msi-controller as msi-parent share this
   single MSI frame. Those devices can work on behalf of the host
   or work on behalf of 1 or more guests (KVM assigned devices). We
   must make sure either the host only or 1 single VM can access to the
   single frame to guarantee interrupt integrity: a device assigned
   to 1 VM should not be able to trigger MSI targeted to the host
   or another VM.

   I would propose to extend the VFIO notion of group viability.
   Currently a VFIO group is viable if:
   all devices belonging to the same group are bound to a VFIO driver
   or unbound.

   Let's imagine we extend the viability check as follows:

   0) keep the current viable check: all the devices belonging to
      the group must be vfio bound or unbound.
   1) retrieve the MSI parent of the device and list all the
      other devices using that MSI controller as MSI-parent (does not
      look straightforward):
   2) they must be VFIO driver bound or unbound as well (meaning
      they are not used by the host). If not, reject device attachment
   - in case they are VFIO bound (a VFIO group is set):
     x if all VFIO containers are the same as the one of the device's
       we try to attach, that's OK. This means the other devices
       use different IOMMU mappings, eventually will target the
       MSI frame but they all work for the same user space client/VM.
     x 1 or more devices has a different container than the device
       under attachment:
       It works on behalf of a different user space client/VM,
       we can't attach the new device. I think there is a case however
       where severals containers can be opened by a single QEMU.

Of course the dynamic aspects, ie a new device showing up or an unbind
event bring significant complexity.

2) GICv2M with multiple 4kB frames
   Each msi-frame is enumerated as msi-controller. The device tree
   statically defines which device is attached to each msi frame.
   In case devices are assigned we cannot change this attachment
   anyway since there might be physical contraints behind.
   So devices likely to be assigned to guests should be linked to a
   different MSI frame than devices that are not.

   I think extended viability concept can be used as well.

   This model still is not ideal: in case we have a SR-IOV device
   plugged onto an host bridge attached to a single MSI parent you won't
   be able anyway to have 1 Virtual Function working for host and 1 VF
   working for a guest. Only Interrupt translation (ITS) will bring that
   feature.

3) GICv3 ITS
   This one supports interrupt translation service ~ Intel
   IRQ remapping.
   This means a single frame can be used by all devices. A deviceID is
   used exclusively by the host or a guest. I assume the ITS driver
   allocates/populates deviceid interrupt translation table featuring
   separate LPI spaces ie by construction different ITT cannot feature
   same LPIs. So no need to do the extended viability test.

   The MSI controller should have a property telling whether
   it supports interrupt translation. This kind of property currently
   exists on IOMMU side for INTEL remapping.

[toc] | [prev] | [next] | [standalone]

#1327983 — Re: ARM PCI/MSI KVM passthrough with GICv2M

On Fri, 5 Feb 2016 18:32:07 +0100
Eric Auger <eric.auger@linaro.org> wrote:

> Hi Alex,
> 
> I tried to sketch a proposal for guaranteeing the IRQ integrity when
> doing ARM PCI/MSI passthrough with ARM GICv2M msi-controller. This is
> based on extended VFIO group viability control, as detailed below.
> 
> As opposed to ARM GICv3 ITS, this MSI controller does *not* support IRQ
> remapping. It can expose 1 or more 4kB MSI frame. Each frame contains a
> single register where the msi data is written.
> 
> I would be grateful to you if you could tell me whether it makes any sense.
> 
> Thanks in advance
> 
> Best Regards
> 
> Eric
> 
> 
> 1) GICv2m with a single 4kB single frame
>    all devices having this msi-controller as msi-parent share this
>    single MSI frame. Those devices can work on behalf of the host
>    or work on behalf of 1 or more guests (KVM assigned devices). We
>    must make sure either the host only or 1 single VM can access to the
>    single frame to guarantee interrupt integrity: a device assigned
>    to 1 VM should not be able to trigger MSI targeted to the host
>    or another VM.
> 
>    I would propose to extend the VFIO notion of group viability.
>    Currently a VFIO group is viable if:
>    all devices belonging to the same group are bound to a VFIO driver
>    or unbound.
> 
>    Let's imagine we extend the viability check as follows:
> 
>    0) keep the current viable check: all the devices belonging to
>       the group must be vfio bound or unbound.
>    1) retrieve the MSI parent of the device and list all the
>       other devices using that MSI controller as MSI-parent (does not
>       look straightforward):
>    2) they must be VFIO driver bound or unbound as well (meaning
>       they are not used by the host). If not, reject device attachment
>    - in case they are VFIO bound (a VFIO group is set):
>      x if all VFIO containers are the same as the one of the device's
>        we try to attach, that's OK. This means the other devices
>        use different IOMMU mappings, eventually will target the
>        MSI frame but they all work for the same user space client/VM.
>      x 1 or more devices has a different container than the device
>        under attachment:
>        It works on behalf of a different user space client/VM,
>        we can't attach the new device. I think there is a case however
>        where severals containers can be opened by a single QEMU.
> 
> Of course the dynamic aspects, ie a new device showing up or an unbind
> event bring significant complexity.
> 
> 2) GICv2M with multiple 4kB frames
>    Each msi-frame is enumerated as msi-controller. The device tree
>    statically defines which device is attached to each msi frame.
>    In case devices are assigned we cannot change this attachment
>    anyway since there might be physical contraints behind.
>    So devices likely to be assigned to guests should be linked to a
>    different MSI frame than devices that are not.
> 
>    I think extended viability concept can be used as well.
> 
>    This model still is not ideal: in case we have a SR-IOV device
>    plugged onto an host bridge attached to a single MSI parent you won't
>    be able anyway to have 1 Virtual Function working for host and 1 VF
>    working for a guest. Only Interrupt translation (ITS) will bring that
>    feature.
> 
> 3) GICv3 ITS
>    This one supports interrupt translation service ~ Intel
>    IRQ remapping.
>    This means a single frame can be used by all devices. A deviceID is
>    used exclusively by the host or a guest. I assume the ITS driver
>    allocates/populates deviceid interrupt translation table featuring
>    separate LPI spaces ie by construction different ITT cannot feature
>    same LPIs. So no need to do the extended viability test.
> 
>    The MSI controller should have a property telling whether
>    it supports interrupt translation. This kind of property currently
>    exists on IOMMU side for INTEL remapping.
> 

Hi Eric,

Would anyone be terribly upset if we simply assume the worst case
scenario on GICv2m/M, have the IOMMU not claim IOMMU_CAP_INTR_REMAP, and
require the user to opt-in via the allow_unsafe_interrupts on the
vfio_iommu_type1 module?  That would make it very compatible with what
we already do on x86, where it really is all or nothing.  My assumption
is that GICv2 would be phased out in favor of GICv3, so there's always
a hardware upgrade path to having more complete isolation, but the
return on investment for figuring out whether a given device really has
this sort of isolation seems pretty low.  Often users already have some
degree of trust in the VMs they use for device assignment anyway.  An
especially prudent user can still look at the hardware specs for their
specific system to understand whether any devices are fully isolated
and only make use of those for device assignment.  Does that seem like
a reasonable alternative?  Thanks,

Alex

[toc] | [prev] | [next] | [standalone]

#1328864 — Re: ARM PCI/MSI KVM passthrough with GICv2M

On Fri, Feb 05, 2016 at 11:17:00AM -0700, Alex Williamson wrote:
> On Fri, 5 Feb 2016 18:32:07 +0100
> Eric Auger <eric.auger@linaro.org> wrote:
> 
> > Hi Alex,
> > 
> > I tried to sketch a proposal for guaranteeing the IRQ integrity when
> > doing ARM PCI/MSI passthrough with ARM GICv2M msi-controller. This is
> > based on extended VFIO group viability control, as detailed below.
> > 
> > As opposed to ARM GICv3 ITS, this MSI controller does *not* support IRQ
> > remapping. It can expose 1 or more 4kB MSI frame. Each frame contains a
> > single register where the msi data is written.
> > 
> > I would be grateful to you if you could tell me whether it makes any sense.
> > 
> > Thanks in advance
> > 
> > Best Regards
> > 
> > Eric
> > 
> > 
> > 1) GICv2m with a single 4kB single frame
> >    all devices having this msi-controller as msi-parent share this
> >    single MSI frame. Those devices can work on behalf of the host
> >    or work on behalf of 1 or more guests (KVM assigned devices). We
> >    must make sure either the host only or 1 single VM can access to the
> >    single frame to guarantee interrupt integrity: a device assigned
> >    to 1 VM should not be able to trigger MSI targeted to the host
> >    or another VM.
> > 
> >    I would propose to extend the VFIO notion of group viability.
> >    Currently a VFIO group is viable if:
> >    all devices belonging to the same group are bound to a VFIO driver
> >    or unbound.
> > 
> >    Let's imagine we extend the viability check as follows:
> > 
> >    0) keep the current viable check: all the devices belonging to
> >       the group must be vfio bound or unbound.
> >    1) retrieve the MSI parent of the device and list all the
> >       other devices using that MSI controller as MSI-parent (does not
> >       look straightforward):
> >    2) they must be VFIO driver bound or unbound as well (meaning
> >       they are not used by the host). If not, reject device attachment
> >    - in case they are VFIO bound (a VFIO group is set):
> >      x if all VFIO containers are the same as the one of the device's
> >        we try to attach, that's OK. This means the other devices
> >        use different IOMMU mappings, eventually will target the
> >        MSI frame but they all work for the same user space client/VM.
> >      x 1 or more devices has a different container than the device
> >        under attachment:
> >        It works on behalf of a different user space client/VM,
> >        we can't attach the new device. I think there is a case however
> >        where severals containers can be opened by a single QEMU.
> > 
> > Of course the dynamic aspects, ie a new device showing up or an unbind
> > event bring significant complexity.
> > 
> > 2) GICv2M with multiple 4kB frames
> >    Each msi-frame is enumerated as msi-controller. The device tree
> >    statically defines which device is attached to each msi frame.
> >    In case devices are assigned we cannot change this attachment
> >    anyway since there might be physical contraints behind.
> >    So devices likely to be assigned to guests should be linked to a
> >    different MSI frame than devices that are not.
> > 
> >    I think extended viability concept can be used as well.
> > 
> >    This model still is not ideal: in case we have a SR-IOV device
> >    plugged onto an host bridge attached to a single MSI parent you won't
> >    be able anyway to have 1 Virtual Function working for host and 1 VF
> >    working for a guest. Only Interrupt translation (ITS) will bring that
> >    feature.
> > 
> > 3) GICv3 ITS
> >    This one supports interrupt translation service ~ Intel
> >    IRQ remapping.
> >    This means a single frame can be used by all devices. A deviceID is
> >    used exclusively by the host or a guest. I assume the ITS driver
> >    allocates/populates deviceid interrupt translation table featuring
> >    separate LPI spaces ie by construction different ITT cannot feature
> >    same LPIs. So no need to do the extended viability test.
> > 
> >    The MSI controller should have a property telling whether
> >    it supports interrupt translation. This kind of property currently
> >    exists on IOMMU side for INTEL remapping.
> > 
> 
> Hi Eric,
> 
> Would anyone be terribly upset if we simply assume the worst case
> scenario on GICv2m/M, have the IOMMU not claim IOMMU_CAP_INTR_REMAP, and
> require the user to opt-in via the allow_unsafe_interrupts on the
> vfio_iommu_type1 module?  That would make it very compatible with what
> we already do on x86, where it really is all or nothing.  

meaning either you allow unsafe multiplexing with passthrough in every
flavor (unsafely) or you don't allow it at all?

I didn't know such on option existed, but it seems to me that this fits
the bill exactly.


> My assumption
> is that GICv2 would be phased out in favor of GICv3, so there's always
> a hardware upgrade path to having more complete isolation, but the
> return on investment for figuring out whether a given device really has
> this sort of isolation seems pretty low.  Often users already have some
> degree of trust in the VMs they use for device assignment anyway.  An
> especially prudent user can still look at the hardware specs for their
> specific system to understand whether any devices are fully isolated
> and only make use of those for device assignment.  Does that seem like
> a reasonable alternative?
> 

It sounds good to me, that would allow us to release a GICv2m-based
solution for MSI passthrough on currently available hardware like the
Seattle.

Thanks,
-Christoffer

[toc] | [prev] | [next] | [standalone]

#1329038 — Re: ARM PCI/MSI KVM passthrough with GICv2M

Hi Alex, Christoffer,
On 02/08/2016 10:48 AM, Christoffer Dall wrote:
> On Fri, Feb 05, 2016 at 11:17:00AM -0700, Alex Williamson wrote:
>> On Fri, 5 Feb 2016 18:32:07 +0100
>> Eric Auger <eric.auger@linaro.org> wrote:
>>
>>> Hi Alex,
>>>
>>> I tried to sketch a proposal for guaranteeing the IRQ integrity when
>>> doing ARM PCI/MSI passthrough with ARM GICv2M msi-controller. This is
>>> based on extended VFIO group viability control, as detailed below.
>>>
>>> As opposed to ARM GICv3 ITS, this MSI controller does *not* support IRQ
>>> remapping. It can expose 1 or more 4kB MSI frame. Each frame contains a
>>> single register where the msi data is written.
>>>
>>> I would be grateful to you if you could tell me whether it makes any sense.
>>>
>>> Thanks in advance
>>>
>>> Best Regards
>>>
>>> Eric
>>>
>>>
>>> 1) GICv2m with a single 4kB single frame
>>>    all devices having this msi-controller as msi-parent share this
>>>    single MSI frame. Those devices can work on behalf of the host
>>>    or work on behalf of 1 or more guests (KVM assigned devices). We
>>>    must make sure either the host only or 1 single VM can access to the
>>>    single frame to guarantee interrupt integrity: a device assigned
>>>    to 1 VM should not be able to trigger MSI targeted to the host
>>>    or another VM.
>>>
>>>    I would propose to extend the VFIO notion of group viability.
>>>    Currently a VFIO group is viable if:
>>>    all devices belonging to the same group are bound to a VFIO driver
>>>    or unbound.
>>>
>>>    Let's imagine we extend the viability check as follows:
>>>
>>>    0) keep the current viable check: all the devices belonging to
>>>       the group must be vfio bound or unbound.
>>>    1) retrieve the MSI parent of the device and list all the
>>>       other devices using that MSI controller as MSI-parent (does not
>>>       look straightforward):
>>>    2) they must be VFIO driver bound or unbound as well (meaning
>>>       they are not used by the host). If not, reject device attachment
>>>    - in case they are VFIO bound (a VFIO group is set):
>>>      x if all VFIO containers are the same as the one of the device's
>>>        we try to attach, that's OK. This means the other devices
>>>        use different IOMMU mappings, eventually will target the
>>>        MSI frame but they all work for the same user space client/VM.
>>>      x 1 or more devices has a different container than the device
>>>        under attachment:
>>>        It works on behalf of a different user space client/VM,
>>>        we can't attach the new device. I think there is a case however
>>>        where severals containers can be opened by a single QEMU.
>>>
>>> Of course the dynamic aspects, ie a new device showing up or an unbind
>>> event bring significant complexity.
>>>
>>> 2) GICv2M with multiple 4kB frames
>>>    Each msi-frame is enumerated as msi-controller. The device tree
>>>    statically defines which device is attached to each msi frame.
>>>    In case devices are assigned we cannot change this attachment
>>>    anyway since there might be physical contraints behind.
>>>    So devices likely to be assigned to guests should be linked to a
>>>    different MSI frame than devices that are not.
>>>
>>>    I think extended viability concept can be used as well.
>>>
>>>    This model still is not ideal: in case we have a SR-IOV device
>>>    plugged onto an host bridge attached to a single MSI parent you won't
>>>    be able anyway to have 1 Virtual Function working for host and 1 VF
>>>    working for a guest. Only Interrupt translation (ITS) will bring that
>>>    feature.
>>>
>>> 3) GICv3 ITS
>>>    This one supports interrupt translation service ~ Intel
>>>    IRQ remapping.
>>>    This means a single frame can be used by all devices. A deviceID is
>>>    used exclusively by the host or a guest. I assume the ITS driver
>>>    allocates/populates deviceid interrupt translation table featuring
>>>    separate LPI spaces ie by construction different ITT cannot feature
>>>    same LPIs. So no need to do the extended viability test.
>>>
>>>    The MSI controller should have a property telling whether
>>>    it supports interrupt translation. This kind of property currently
>>>    exists on IOMMU side for INTEL remapping.
>>>
>>
>> Hi Eric,
>>
>> Would anyone be terribly upset if we simply assume the worst case
>> scenario on GICv2m/M, have the IOMMU not claim IOMMU_CAP_INTR_REMAP, and
>> require the user to opt-in via the allow_unsafe_interrupts on the
>> vfio_iommu_type1 module?  That would make it very compatible with what
>> we already do on x86, where it really is all or nothing.  
> 
> meaning either you allow unsafe multiplexing with passthrough in every
> flavor (unsafely) or you don't allow it at all?
that's my understanding. if the iommu does not expose
IOMMU_CAP_INTR_REMAP, the end-user must explicitly turn
allow_unsafe_interrupts on. On ARM we will have the handle the fact the
interrupt translation is handled on interrupt controller side and not on
iommu side though;
>
> I didn't know such on option existed, but it seems to me that this fits
> the bill exactly.
well I think the support of multiple GICv2m MSI frames was devised to
allow safe interrupts but extending the VFIO viability notion as
described above effectively seems a huge work with small benefits since
we don't have much HW featuring multiple frames I am afraid. So I think
it is a good compromise to have a minimal integration with GICv2m and
full feature with best fitted HW, ie. GICv3 ITS.
> 
> 
>> My assumption
>> is that GICv2 would be phased out in favor of GICv3, so there's always
>> a hardware upgrade path to having more complete isolation, but the
>> return on investment for figuring out whether a given device really has
>> this sort of isolation seems pretty low.  Often users already have some
>> degree of trust in the VMs they use for device assignment anyway.  An
>> especially prudent user can still look at the hardware specs for their
>> specific system to understand whether any devices are fully isolated
>> and only make use of those for device assignment.  Does that seem like
>> a reasonable alternative?
>>
> 
> It sounds good to me, that would allow us to release a GICv2m-based
> solution for MSI passthrough on currently available hardware like the
> Seattle.

Sounds good to me too. I am going to respin the kernel series according
to this discussion and previous comments.

Thanks for your comments!

Best Regards

Eric
> 
> Thanks,
> -Christoffer
> 

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web