Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1683763 > unrolled thread

[RFC 0/2] arm-smmu-v3 tlbi-on-map option

Back to article view | Back to linux.kernel

Contents

  [RFC 0/2] arm-smmu-v3 tlbi-on-map option Eric Auger <eric.auger@redhat.com> - 2017-07-09 17:20 +0200
    Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option Will Deacon <will.deacon@arm.com> - 2017-07-12 20:00 +0200
      Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option "Michael S. Tsirkin" <mst@redhat.com> - 2017-07-13 00:10 +0200
        Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option Jean-Philippe Brucker <jean-philippe.brucker@arm.com> - 2017-07-13 11:30 +0200
          Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option "Michael S. Tsirkin" <mst@redhat.com> - 2017-07-13 19:50 +0200
            Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option Jean-Philippe Brucker <jean-philippe.brucker@arm.com> - 2017-07-13 21:20 +0200
              Re: [RFC 0/2] arm-smmu-v3 tlbi-on-map option "Michael S. Tsirkin" <mst@redhat.com> - 2017-07-13 22:40 +0200

#1683763 — [RFC 0/2] arm-smmu-v3 tlbi-on-map option

This series adds a new tlbi-on-map option to the smmuv3 driver.
When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for 
LPAE tables and the smmuv3 driver sends TLB invalidations on map.

This mode is useful when running the driver on a guest as it allows
the virtualizer to trap any change to the translation structures.
This is similar to the Intel vtd caching mode (CM).

This is especially needed for VFIO integration integration where
guest mappings must be applied to the physical IOMMU.

At the moment the option only is available for DT probing.

Best Regards

Eric

Eric Auger (2):
  iommu/io-pgtable-arm: flush TLBs when IO_PGTABLE_QUIRK_TLBI_ON_MAP
  arm-smmu-v3: Add tlbi_on_map option

 Documentation/devicetree/bindings/iommu/arm,smmu-v3.txt |  4 ++++
 drivers/iommu/arm-smmu-v3.c                             |  5 +++++
 drivers/iommu/io-pgtable-arm.c                          | 13 +++++++++++--
 3 files changed, 20 insertions(+), 2 deletions(-)

-- 
2.5.5

[toc] | [next] | [standalone]

#1685941

Hi Eric,

On Sun, Jul 09, 2017 at 05:15:01PM +0200, Eric Auger wrote:
> This series adds a new tlbi-on-map option to the smmuv3 driver.
> When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for 
> LPAE tables and the smmuv3 driver sends TLB invalidations on map.
> 
> This mode is useful when running the driver on a guest as it allows
> the virtualizer to trap any change to the translation structures.
> This is similar to the Intel vtd caching mode (CM).
> 
> This is especially needed for VFIO integration integration where
> guest mappings must be applied to the physical IOMMU.
> 
> At the moment the option only is available for DT probing.

I'm really not a fan of this approach. If a virtual IOMMU implementation is
advertising itself as an SMMUv3, then it should adhere to the SMMUv3
architecture and not require non-standard behaviour from the driver. If
we're going to allow that, then we're better off going the extra mile and
using a PV approach. Given that the the SMMU3 architecture does *not*
require TLBI on map, then I don't think we should be quirking our behaviour
in this way. The fact that you only have this implemented for DT is the
canary in the coal mine imo.

Will

[toc] | [prev] | [next] | [standalone]

#1686069

On Wed, Jul 12, 2017 at 06:54:56PM +0100, Will Deacon wrote:
> Hi Eric,
> 
> On Sun, Jul 09, 2017 at 05:15:01PM +0200, Eric Auger wrote:
> > This series adds a new tlbi-on-map option to the smmuv3 driver.
> > When set, the IO_PGTABLE_QUIRK_TLBI_ON_MAP quirk is applied for 
> > LPAE tables and the smmuv3 driver sends TLB invalidations on map.
> > 
> > This mode is useful when running the driver on a guest as it allows
> > the virtualizer to trap any change to the translation structures.
> > This is similar to the Intel vtd caching mode (CM).
> > 
> > This is especially needed for VFIO integration integration where
> > guest mappings must be applied to the physical IOMMU.
> > 
> > At the moment the option only is available for DT probing.
> 
> I'm really not a fan of this approach. If a virtual IOMMU implementation is
> advertising itself as an SMMUv3, then it should adhere to the SMMUv3
> architecture and not require non-standard behaviour from the driver.

You can also just enable e.g. building existing intel VTD code on ARM,
that emulation is already there.

> If
> we're going to allow that, then we're better off going the extra mile and
> using a PV approach. Given that the the SMMU3 architecture does *not*
> require TLBI on map, then I don't think we should be quirking our behaviour
> in this way. 

I think using hardware support for nesting is the right final
solution. It will take some time though. Given this, what should
we do meanwhile?

Assuming that's the final destination, a simple quirk like this
seems to be preferable to virtio-iommu which we'll never be
able to offload to hardware.

>The fact that you only have this implemented for DT is the
> canary in the coal mine imo.
> 
> Will

As to the last point, I agree absolutely. Need to support ACPI
as well or nothing. It would be nicer IMHO to get it from some
kind of device register, assuming ARM can promise to reserve
some bits for hypervisors to play with.

-- 
MST

[toc] | [prev] | [next] | [standalone]

#1686403

On 12/07/17 23:07, Michael S. Tsirkin wrote:
[...]
> I think using hardware support for nesting is the right final
> solution. It will take some time though. Given this, what should
> we do meanwhile?
> 
> Assuming that's the final destination, a simple quirk like this
> seems to be preferable to virtio-iommu which we'll never be
> able to offload to hardware.

That's not entirely true. virtio-iommu will have an extension for hardware
nesting support. It was presented in my initial proposal, and I've made
significant progress since then.

Thanks,
Jean

[toc] | [prev] | [next] | [standalone]

#1686804

On Thu, Jul 13, 2017 at 10:29:42AM +0100, Jean-Philippe Brucker wrote:
> On 12/07/17 23:07, Michael S. Tsirkin wrote:
> [...]
> > I think using hardware support for nesting is the right final
> > solution. It will take some time though. Given this, what should
> > we do meanwhile?
> > 
> > Assuming that's the final destination, a simple quirk like this
> > seems to be preferable to virtio-iommu which we'll never be
> > able to offload to hardware.
> 
> That's not entirely true. virtio-iommu will have an extension for hardware
> nesting support. It was presented in my initial proposal, and I've made
> significant progress since then.
> 
> Thanks,
> Jean

I don't recall seeing this.

Hardware specific extensions to virtio would be interesting, the
difficulty is in finding the balance between enabling minor quirks and
each vendor going their own way.

Is this the proposal you refer to?
https://www.spinics.net/lists/kvm/msg147990.html

I couldn't find any mention of nesting, it seems to
say that map requests are relayed through host.

-- 
MST

[toc] | [prev] | [next] | [standalone]

#1686853

On 13/07/17 18:44, Michael S. Tsirkin wrote:
> On Thu, Jul 13, 2017 at 10:29:42AM +0100, Jean-Philippe Brucker wrote:
>> On 12/07/17 23:07, Michael S. Tsirkin wrote:
>> [...]
>>> I think using hardware support for nesting is the right final
>>> solution. It will take some time though. Given this, what should
>>> we do meanwhile?
>>>
>>> Assuming that's the final destination, a simple quirk like this
>>> seems to be preferable to virtio-iommu which we'll never be
>>> able to offload to hardware.
>>
>> That's not entirely true. virtio-iommu will have an extension for hardware
>> nesting support. It was presented in my initial proposal, and I've made
>> significant progress since then.
>>
>> Thanks,
>> Jean
> 
> I don't recall seeing this.
> 
> Hardware specific extensions to virtio would be interesting, the
> difficulty is in finding the balance between enabling minor quirks and
> each vendor going their own way.

Yes, in order to avoid too many quirks and vendor-specific formats, we'd
like the guest to only manage page tables, which have relatively stable
format, while the host kernel manages context tables (PASID tables) and
other structures, that are more volatile across implementations.

Unavoidably, a few architecture-specific details need to be described in
the API, but it seems manageable. The host tells the guest which page
table format is supported by hardware, and the guest sends back a page
directory pointer along with a few configuration parameters.

In the guest, page table operations may be abstracted in a module and used
by multiple IOMMU drivers (what ARM does in drivers/iommu/io-pgtable-arm.c
for the various vendor drivers). This abstraction is quite helpful to find
out which information needs to be exchanged between virtio-iommu device
and driver. For ARM-based IOMMUs it amounts to 6 configuration registers
and 4 quirk bits.

The other problem is forwarding TLB invalidations to the host kernel.
There is an ongoing discussion [1] about the best way to do it in VFIO,
and it seems like the format can stay mostly generic, with a few
optimization hints depending on the hardware IOMMU.

> Is this the proposal you refer to?
> https://www.spinics.net/lists/kvm/msg147990.html

Yes, I'm referring to "II. Page table sharing" in
https://www.spinics.net/lists/kvm/msg147993.html

But it is now mostly obsolete. Lots of things had to change while writing
a prototype and following public discussions. At the moment my focus is on
tidying up the base, but I will send another RFC afterwards.

Thanks,
Jean

[1] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg01117.html

[toc] | [prev] | [next] | [standalone]

#1686896

On Thu, Jul 13, 2017 at 08:17:49PM +0100, Jean-Philippe Brucker wrote:
> But it is now mostly obsolete. Lots of things had to change while writing
> a prototype and following public discussions. At the moment my focus is on
> tidying up the base, but I will send another RFC afterwards.

Thanks!  Pls remember to copy virtio-dev at oasis (you need to subscribe
to post) as that's a requirement for anything involving host/guest APIs.

-- 
MST

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web