Re: Summary of LPC guest MSI discussion in Santa Fe (was: Re: [RFC 0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II))

From	Christoffer Dall <christoffer.dall@linaro.org>
Newsgroups	linux.kernel
Subject	Re: Summary of LPC guest MSI discussion in Santa Fe (was: Re: [RFC 0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II))
Date	2016-11-08 21:30 +0100
Message-ID	<sBlbP-5QB-19@gated-at.bofh.it> (permalink)
References	<szy3w-Yc-7@gated-at.bofh.it> <szDZf-50p-1@gated-at.bofh.it> <sB4E1-3lS-3@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

Hi Will,

On Tue, Nov 08, 2016 at 02:45:59AM +0000, Will Deacon wrote:
> Hi all,
> 
> I figured this was a reasonable post to piggy-back on for the LPC minutes
> relating to guest MSIs on arm64.
> 
> On Thu, Nov 03, 2016 at 10:02:05PM -0600, Alex Williamson wrote:
> > We can always have QEMU reject hot-adding the device if the reserved
> > region overlaps existing guest RAM, but I don't even really see how we
> > advise users to give them a reasonable chance of avoiding that
> > possibility.  Apparently there are also ARM platforms where MSI pages
> > cannot be remapped to support the previous programmable user/VM
> > address, is it even worthwhile to support those platforms?  Does that
> > decision influence whether user programmable MSI reserved regions are
> > really a second class citizen to fixed reserved regions?  I expect
> > we'll be talking about this tomorrow morning, but I certainly haven't
> > come up with any viable solutions to this.  Thanks,
> 
> At LPC last week, we discussed guest MSIs on arm64 as part of the PCI
> microconference. I presented some slides to illustrate some of the issues
> we're trying to solve:
> 
>   http://www.willdeacon.ukfsn.org/bitbucket/lpc-16/msi-in-guest-arm64.pdf
> 
> Punit took some notes (thanks!) on the etherpad here:
> 
>   https://etherpad.openstack.org/p/LPC2016_PCI
> 
> although the discussion was pretty lively and jumped about, so I've had
> to go from memory where the notes didn't capture everything that was
> said.
> 
> To summarise, arm64 platforms differ in their handling of MSIs when compared
> to x86:
> 
>   1. The physical memory map is not standardised (Jon pointed out that
>      this is something that was realised late on)
>   2. MSIs are usually treated the same as DMA writes, in that they must be
>      mapped by the SMMU page tables so that they target a physical MSI
>      doorbell
>   3. On some platforms, MSIs bypass the SMMU entirely (e.g. due to an MSI
>      doorbell built into the PCI RC)
>   4. Platforms typically have some set of addresses that abort before
>      reaching the SMMU (e.g. because the PCI identifies them as P2P).
> 
> All of this means that userspace (QEMU) needs to identify the memory
> regions corresponding to points (3) and (4) and ensure that they are
> not allocated in the guest physical (IPA) space. For platforms that can
> remap the MSI doorbell as in (2), then some space also needs to be
> allocated for that.
> 
> Rather than treat these as separate problems, a better interface is to
> tell userspace about a set of reserved regions, and have this include
> the MSI doorbell, irrespective of whether or not it can be remapped.

Is my understanding correct, that you need to tell userspace about the
location of the doorbell (in the IOVA space) in case (2), because even
though the configuration of the device is handled by the (host) kernel
through trapping of the BARs, we have to avoid the VFIO user programming
the device to create other DMA transactions to this particular address,
since that will obviously conflict and either not produce the desired
DMA transactions or result in unintended weird interrupts?

Thanks,
Christoffer

Thread

[RFC 0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II) Eric Auger <eric.auger@redhat.com> - 2016-11-03 22:50 +0100
  [RFC 2/8] iommu/iova: fix __alloc_and_insert_iova_range Eric Auger <eric.auger@redhat.com> - 2016-11-03 22:50 +0100
  [RFC 4/8] iommu: Add a list of iommu_reserved_region in iommu_domain Eric Auger <eric.auger@redhat.com> - 2016-11-03 22:50 +0100
  [RFC 5/8] vfio/type1: Introduce RESV_IOVA_RANGE capability Eric Auger <eric.auger@redhat.com> - 2016-11-03 22:50 +0100
  [RFC 3/8] iommu/dma: Allow MSI-only cookies Eric Auger <eric.auger@redhat.com> - 2016-11-03 22:50 +0100
  [RFC 7/8] iommu/vt-d: Implement add_reserved_regions callback Eric Auger <eric.auger@redhat.com> - 2016-11-03 22:50 +0100
  [RFC 1/8] vfio: fix vfio_info_cap_add/shift Eric Auger <eric.auger@redhat.com> - 2016-11-03 22:50 +0100
  [RFC 6/8] iommu: Handle the list of reserved regions Eric Auger <eric.auger@redhat.com> - 2016-11-03 22:50 +0100
  Re: [RFC 0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II) Alex Williamson <alex.williamson@redhat.com> - 2016-11-04 05:10 +0100
    Summary of LPC guest MSI discussion in Santa Fe (was: Re: [RFC 0/8]  KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II)) Will Deacon <will.deacon@arm.com> - 2016-11-08 03:50 +0100
      Re: Summary of LPC guest MSI discussion in Santa Fe Auger Eric <eric.auger@redhat.com> - 2016-11-08 15:30 +0100
        Re: Summary of LPC guest MSI discussion in Santa Fe Will Deacon <will.deacon@arm.com> - 2016-11-08 19:00 +0100
          Re: Summary of LPC guest MSI discussion in Santa Fe Don Dutile <ddutile@redhat.com> - 2016-11-08 20:10 +0100
            Re: Summary of LPC guest MSI discussion in Santa Fe Will Deacon <will.deacon@arm.com> - 2016-11-08 20:20 +0100
            Re: Summary of LPC guest MSI discussion in Santa Fe Auger Eric <eric.auger@redhat.com> - 2016-11-09 08:50 +0100
      Re: Summary of LPC guest MSI discussion in Santa Fe Don Dutile <ddutile@redhat.com> - 2016-11-08 17:10 +0100
      Re: Summary of LPC guest MSI discussion in Santa Fe (was: Re: [RFC  0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II)) Christoffer Dall <christoffer.dall@linaro.org> - 2016-11-08 21:30 +0100
        Re: Summary of LPC guest MSI discussion in Santa Fe (was: Re: [RFC  0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II)) Alex Williamson <alex.williamson@redhat.com> - 2016-11-09 00:40 +0100
          Re: Summary of LPC guest MSI discussion in Santa Fe Don Dutile <ddutile@redhat.com> - 2016-11-09 04:00 +0100
            Re: Summary of LPC guest MSI discussion in Santa Fe Will Deacon <will.deacon@arm.com> - 2016-11-09 18:10 +0100
              Re: Summary of LPC guest MSI discussion in Santa Fe Don Dutile <ddutile@redhat.com> - 2016-11-09 20:00 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Christoffer Dall <christoffer.dall@linaro.org> - 2016-11-09 20:30 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Alex Williamson <alex.williamson@redhat.com> - 2016-11-09 21:10 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Joerg Roedel <joro@8bytes.org> - 2016-11-10 15:50 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Alex Williamson <alex.williamson@redhat.com> - 2016-11-10 18:10 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Will Deacon <will.deacon@arm.com> - 2016-11-09 21:40 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Alex Williamson <alex.williamson@redhat.com> - 2016-11-09 23:20 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Will Deacon <will.deacon@arm.com> - 2016-11-09 23:30 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Alex Williamson <alex.williamson@redhat.com> - 2016-11-10 00:30 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Will Deacon <will.deacon@arm.com> - 2016-11-10 00:40 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Alex Williamson <alex.williamson@redhat.com> - 2016-11-10 01:10 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Auger Eric <eric.auger@redhat.com> - 2016-11-10 01:20 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Alex Williamson <alex.williamson@redhat.com> - 2016-11-10 02:00 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Will Deacon <will.deacon@arm.com> - 2016-11-10 03:10 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Auger Eric <eric.auger@redhat.com> - 2016-11-10 12:20 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Alex Williamson <alex.williamson@redhat.com> - 2016-11-10 18:50 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Joerg Roedel <joro@8bytes.org> - 2016-11-11 12:20 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Alex Williamson <alex.williamson@redhat.com> - 2016-11-11 17:00 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Alex Williamson <alex.williamson@redhat.com> - 2016-11-11 17:10 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Don Dutile <ddutile@redhat.com> - 2016-11-11 17:30 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Don Dutile <ddutile@redhat.com> - 2016-11-11 17:10 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Joerg Roedel <joro@8bytes.org> - 2016-11-10 16:00 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Robin Murphy <robin.murphy@arm.com> - 2016-11-09 21:20 +0100
                Re: Summary of LPC guest MSI discussion in Santa Fe Joerg Roedel <joro@8bytes.org> - 2016-11-10 16:30 +0100

csiph-web