Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1243605 > unrolled thread

[RFC PATCH 0/2] VFIO no-iommu

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH 0/2] VFIO no-iommu Alex Williamson <alex.williamson@redhat.com> - 2015-10-09 20:50 +0200
    Re: [RFC PATCH 1/2] vfio: Move vfio.c vfio_core.c Greg KH <gregkh@linuxfoundation.org> - 2015-10-09 21:30 +0200
    RE: [RFC PATCH 0/2] VFIO no-iommu Varun Sethi <Varun.Sethi@freescale.com> - 2015-10-11 20:10 +0200
      Re: [RFC PATCH 0/2] VFIO no-iommu Alex Williamson <alex.williamson@redhat.com> - 2015-10-11 20:30 +0200
    Re: [RFC PATCH 0/2] VFIO no-iommu "Michael S. Tsirkin" <mst@redhat.com> - 2015-10-11 20:30 +0200
      Re: [RFC PATCH 0/2] VFIO no-iommu Alex Williamson <alex.williamson@redhat.com> - 2015-10-11 21:30 +0200
    Re: [RFC PATCH 0/2] VFIO no-iommu "Michael S. Tsirkin" <mst@redhat.com> - 2015-10-11 20:30 +0200

#1243605 — [RFC PATCH 0/2] VFIO no-iommu

Recent patches for UIO have been attempting to add MSI/X support,
which unfortunately implies DMA support, which users have been
enabling anyway, but was never intended for UIO.  VFIO on the other
hand expects an IOMMU to provide isolation of devices, but provides
a much more complete device interface, which already supports full
MSI/X support.  There's really no way to support userspace drivers
with DMA capable devices without an IOMMU to protect the host, but
we can at least think about doing it in a way that properly taints
the kernel and avoids creating new code duplicating existing code,
that does have a supportable use case.

The diffstat is only so large because I moved vfio.c to vfio_core.c
so I could more easily keep the module named vfio.ko while keeping
the bulk of the no-iommu support in a separate file that can be
optionally compiled.  We're really looking at a couple hundred lines
of mostly stub code.  The VFIO_NOIOMMU_IOMMU could certainly be
expanded to do page pinning and virt_to_bus() translation, but I
didn't want to complicate anything yet.

I've only compiled this and tested loading the module with the new
no-iommu mode enabled, I haven't actually tried to port a DPDK
driver to it, though it ought to be a pretty obvious mix of the
existing UIO and VFIO versions (set the IOMMU, but avoid using it
for mapping, use however bus translations are done w/ UIO).  The core
vfio device file is still /dev/vfio/vfio, but all the groups become
/dev/vfio-noiommu/$GROUP.

It should be obvious, but I always feel obligated to state that this
does not and will not ever enable device assignment to virtual
machines on non-IOMMU capable platforms.

I'm curious what IOMMU folks think of this.  This hack is really
only possible because we don't use iommu_ops for regular DMA, so we
can hijack it fairly safely.  I believe that's intended to change
though, so this may not be practical long term.  Thanks,

Alex

---

Alex Williamson (2):
      vfio: Move vfio.c vfio_core.c
      vfio: Include no-iommu mode


 drivers/vfio/Kconfig        |   15 
 drivers/vfio/Makefile       |    4 
 drivers/vfio/vfio.c         | 1640 ------------------------------------------
 drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio_noiommu.c |  185 +++++
 drivers/vfio/vfio_private.h |   31 +
 include/uapi/linux/vfio.h   |    2 
 7 files changed, 1917 insertions(+), 1640 deletions(-)
 delete mode 100644 drivers/vfio/vfio.c
 create mode 100644 drivers/vfio/vfio_core.c
 create mode 100644 drivers/vfio/vfio_noiommu.c
 create mode 100644 drivers/vfio/vfio_private.h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1243620 — Re: [RFC PATCH 1/2] vfio: Move vfio.c vfio_core.c

On Fri, Oct 09, 2015 at 12:41:03PM -0600, Alex Williamson wrote:
> This allows us to more easily create a "vfio" module that includes
> multiple files.  No code change, rename and Makefile update only.
> 
> Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> ---
>  drivers/vfio/Makefile    |    1 
>  drivers/vfio/vfio.c      | 1640 ----------------------------------------------
>  drivers/vfio/vfio_core.c | 1640 ++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 1641 insertions(+), 1640 deletions(-)
>  delete mode 100644 drivers/vfio/vfio.c
>  create mode 100644 drivers/vfio/vfio_core.c

Use -S to git format-patch so we can see the rename, not a delete/add
patch please.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1244179

Hi Alex,
Thanks for the patch Alex. This would also require support in Qemu to expose the physical address to the VM. Are you looking at that part as well?

Regards
Varun

-----Original Message-----
From: iommu-bounces@lists.linux-foundation.org [mailto:iommu-bounces@lists.linux-foundation.org] On Behalf Of Alex Williamson
Sent: Saturday, October 10, 2015 12:11 AM
To: alex.williamson@redhat.com
Cc: avi@scylladb.com; avi@cloudius-systems.com; gleb@scylladb.com; mst@redhat.com; bruce.richardson@intel.com; corbet@lwn.net; linux-kernel@vger.kernel.org; alexander.duyck@gmail.com; gleb@cloudius-systems.com; stephen@networkplumber.org; vladz@cloudius-systems.com; iommu@lists.linux-foundation.org; hjk@hansjkoch.de; gregkh@linuxfoundation.org
Subject: [RFC PATCH 0/2] VFIO no-iommu

Recent patches for UIO have been attempting to add MSI/X support, which unfortunately implies DMA support, which users have been enabling anyway, but was never intended for UIO.  VFIO on the other hand expects an IOMMU to provide isolation of devices, but provides a much more complete device interface, which already supports full MSI/X support.  There's really no way to support userspace drivers with DMA capable devices without an IOMMU to protect the host, but we can at least think about doing it in a way that properly taints the kernel and avoids creating new code duplicating existing code, that does have a supportable use case.

The diffstat is only so large because I moved vfio.c to vfio_core.c so I could more easily keep the module named vfio.ko while keeping the bulk of the no-iommu support in a separate file that can be optionally compiled.  We're really looking at a couple hundred lines of mostly stub code.  The VFIO_NOIOMMU_IOMMU could certainly be expanded to do page pinning and virt_to_bus() translation, but I didn't want to complicate anything yet.

I've only compiled this and tested loading the module with the new no-iommu mode enabled, I haven't actually tried to port a DPDK driver to it, though it ought to be a pretty obvious mix of the existing UIO and VFIO versions (set the IOMMU, but avoid using it for mapping, use however bus translations are done w/ UIO).  The core vfio device file is still /dev/vfio/vfio, but all the groups become /dev/vfio-noiommu/$GROUP.

It should be obvious, but I always feel obligated to state that this does not and will not ever enable device assignment to virtual machines on non-IOMMU capable platforms.

I'm curious what IOMMU folks think of this.  This hack is really only possible because we don't use iommu_ops for regular DMA, so we can hijack it fairly safely.  I believe that's intended to change though, so this may not be practical long term.  Thanks,

Alex

---

Alex Williamson (2):
      vfio: Move vfio.c vfio_core.c
      vfio: Include no-iommu mode


 drivers/vfio/Kconfig        |   15 
 drivers/vfio/Makefile       |    4 
 drivers/vfio/vfio.c         | 1640 ------------------------------------------
 drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio_noiommu.c |  185 +++++
 drivers/vfio/vfio_private.h |   31 +
 include/uapi/linux/vfio.h   |    2 
 7 files changed, 1917 insertions(+), 1640 deletions(-)  delete mode 100644 drivers/vfio/vfio.c  create mode 100644 drivers/vfio/vfio_core.c  create mode 100644 drivers/vfio/vfio_noiommu.c  create mode 100644 drivers/vfio/vfio_private.h _______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1244181

On Sun, 2015-10-11 at 17:29 +0000, Varun Sethi wrote:
> Hi Alex,
> Thanks for the patch Alex. This would also require support in Qemu to expose the physical address to the VM. Are you looking at that part as well?

Quoting from below:

        It should be obvious, but I always feel obligated to state that
        this does not and will not ever enable device assignment to
        virtual machines on non-IOMMU capable platforms.


I have no intention whatsoever, nor would I encourage or support use of
this mode for assignment to a virtual machine.  VFIO has uses beyond
QEMU.  Thanks,

Alex

> -----Original Message-----
> From: iommu-bounces@lists.linux-foundation.org [mailto:iommu-bounces@lists.linux-foundation.org] On Behalf Of Alex Williamson
> Sent: Saturday, October 10, 2015 12:11 AM
> To: alex.williamson@redhat.com
> Cc: avi@scylladb.com; avi@cloudius-systems.com; gleb@scylladb.com; mst@redhat.com; bruce.richardson@intel.com; corbet@lwn.net; linux-kernel@vger.kernel.org; alexander.duyck@gmail.com; gleb@cloudius-systems.com; stephen@networkplumber.org; vladz@cloudius-systems.com; iommu@lists.linux-foundation.org; hjk@hansjkoch.de; gregkh@linuxfoundation.org
> Subject: [RFC PATCH 0/2] VFIO no-iommu
> 
> Recent patches for UIO have been attempting to add MSI/X support, which unfortunately implies DMA support, which users have been enabling anyway, but was never intended for UIO.  VFIO on the other hand expects an IOMMU to provide isolation of devices, but provides a much more complete device interface, which already supports full MSI/X support.  There's really no way to support userspace drivers with DMA capable devices without an IOMMU to protect the host, but we can at least think about doing it in a way that properly taints the kernel and avoids creating new code duplicating existing code, that does have a supportable use case.
> 
> The diffstat is only so large because I moved vfio.c to vfio_core.c so I could more easily keep the module named vfio.ko while keeping the bulk of the no-iommu support in a separate file that can be optionally compiled.  We're really looking at a couple hundred lines of mostly stub code.  The VFIO_NOIOMMU_IOMMU could certainly be expanded to do page pinning and virt_to_bus() translation, but I didn't want to complicate anything yet.
> 
> I've only compiled this and tested loading the module with the new no-iommu mode enabled, I haven't actually tried to port a DPDK driver to it, though it ought to be a pretty obvious mix of the existing UIO and VFIO versions (set the IOMMU, but avoid using it for mapping, use however bus translations are done w/ UIO).  The core vfio device file is still /dev/vfio/vfio, but all the groups become /dev/vfio-noiommu/$GROUP.
> 
> It should be obvious, but I always feel obligated to state that this does not and will not ever enable device assignment to virtual machines on non-IOMMU capable platforms.
> 
> I'm curious what IOMMU folks think of this.  This hack is really only possible because we don't use iommu_ops for regular DMA, so we can hijack it fairly safely.  I believe that's intended to change though, so this may not be practical long term.  Thanks,
> 
> Alex
> 
> ---
> 
> Alex Williamson (2):
>       vfio: Move vfio.c vfio_core.c
>       vfio: Include no-iommu mode
> 
> 
>  drivers/vfio/Kconfig        |   15 
>  drivers/vfio/Makefile       |    4 
>  drivers/vfio/vfio.c         | 1640 ------------------------------------------
>  drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
>  drivers/vfio/vfio_noiommu.c |  185 +++++
>  drivers/vfio/vfio_private.h |   31 +
>  include/uapi/linux/vfio.h   |    2 
>  7 files changed, 1917 insertions(+), 1640 deletions(-)  delete mode 100644 drivers/vfio/vfio.c  create mode 100644 drivers/vfio/vfio_core.c  create mode 100644 drivers/vfio/vfio_noiommu.c  create mode 100644 drivers/vfio/vfio_private.h _______________________________________________
> iommu mailing list
> iommu@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1244182

On Sun, Oct 11, 2015 at 09:28:09PM +0300, Michael S. Tsirkin wrote:
> On Fri, Oct 09, 2015 at 12:40:56PM -0600, Alex Williamson wrote:
> > Recent patches for UIO have been attempting to add MSI/X support,
> > which unfortunately implies DMA support, which users have been
> > enabling anyway, but was never intended for UIO.  VFIO on the other
> > hand expects an IOMMU to provide isolation of devices, but provides
> > a much more complete device interface, which already supports full
> > MSI/X support.  There's really no way to support userspace drivers
> > with DMA capable devices without an IOMMU to protect the host, but
> > we can at least think about doing it in a way that properly taints
> > the kernel and avoids creating new code duplicating existing code,
> > that does have a supportable use case.
> > 
> > The diffstat is only so large because I moved vfio.c to vfio_core.c
> > so I could more easily keep the module named vfio.ko while keeping
> > the bulk of the no-iommu support in a separate file that can be
> > optionally compiled.  We're really looking at a couple hundred lines
> > of mostly stub code.  The VFIO_NOIOMMU_IOMMU could certainly be
> > expanded to do page pinning and virt_to_bus() translation, but I
> > didn't want to complicate anything yet.
> 
> I think it's already useful like this, since all current users
> seem happy enough to just use hugetlbfs to do pinning, and
> ignore translation.
> 
> > I've only compiled this and tested loading the module with the new
> > no-iommu mode enabled, I haven't actually tried to port a DPDK
> > driver to it, though it ought to be a pretty obvious mix of the
> > existing UIO and VFIO versions (set the IOMMU, but avoid using it
> > for mapping, use however bus translations are done w/ UIO).  The core
> > vfio device file is still /dev/vfio/vfio, but all the groups become
> > /dev/vfio-noiommu/$GROUP.
> > 
> > It should be obvious, but I always feel obligated to state that this
> > does not and will not ever enable device assignment to virtual
> > machines on non-IOMMU capable platforms.
> 
> In theory, it's kind of possible using paravirtualization.
> 
> Within guest, you'd make map_page retrieve the io address from the host
> and return that as dma_addr_t.  The only question would be APIs that
> require more than one contigious page in IO space (e.g. I think alloc
> coherent is like this?).
> Not a problem if host is using hugetlbfs, but if not, I guess we could
> add a hypercall and some Linux API on the host to trigger compaction
> on the host aggressively. MADV_CONTIGIOUS?

Not that I see a good reason for that.
Just use an iommu.


> 
> > I'm curious what IOMMU folks think of this.  This hack is really
> > only possible because we don't use iommu_ops for regular DMA, so we
> > can hijack it fairly safely.  I believe that's intended to change
> > though, so this may not be practical long term.  Thanks,
> > 
> > Alex
> > 
> > ---
> > 
> > Alex Williamson (2):
> >       vfio: Move vfio.c vfio_core.c
> >       vfio: Include no-iommu mode
> > 
> > 
> >  drivers/vfio/Kconfig        |   15 
> >  drivers/vfio/Makefile       |    4 
> >  drivers/vfio/vfio.c         | 1640 ------------------------------------------
> >  drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
> >  drivers/vfio/vfio_noiommu.c |  185 +++++
> >  drivers/vfio/vfio_private.h |   31 +
> >  include/uapi/linux/vfio.h   |    2 
> >  7 files changed, 1917 insertions(+), 1640 deletions(-)
> >  delete mode 100644 drivers/vfio/vfio.c
> >  create mode 100644 drivers/vfio/vfio_core.c
> >  create mode 100644 drivers/vfio/vfio_noiommu.c
> >  create mode 100644 drivers/vfio/vfio_private.h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1244198

On Sun, 2015-10-11 at 21:29 +0300, Michael S. Tsirkin wrote:
> On Sun, Oct 11, 2015 at 09:28:09PM +0300, Michael S. Tsirkin wrote:
> > On Fri, Oct 09, 2015 at 12:40:56PM -0600, Alex Williamson wrote:
> > > Recent patches for UIO have been attempting to add MSI/X support,
> > > which unfortunately implies DMA support, which users have been
> > > enabling anyway, but was never intended for UIO.  VFIO on the other
> > > hand expects an IOMMU to provide isolation of devices, but provides
> > > a much more complete device interface, which already supports full
> > > MSI/X support.  There's really no way to support userspace drivers
> > > with DMA capable devices without an IOMMU to protect the host, but
> > > we can at least think about doing it in a way that properly taints
> > > the kernel and avoids creating new code duplicating existing code,
> > > that does have a supportable use case.
> > > 
> > > The diffstat is only so large because I moved vfio.c to vfio_core.c
> > > so I could more easily keep the module named vfio.ko while keeping
> > > the bulk of the no-iommu support in a separate file that can be
> > > optionally compiled.  We're really looking at a couple hundred lines
> > > of mostly stub code.  The VFIO_NOIOMMU_IOMMU could certainly be
> > > expanded to do page pinning and virt_to_bus() translation, but I
> > > didn't want to complicate anything yet.
> > 
> > I think it's already useful like this, since all current users
> > seem happy enough to just use hugetlbfs to do pinning, and
> > ignore translation.

That was sort of my thought too...
 
> > > I've only compiled this and tested loading the module with the new
> > > no-iommu mode enabled, I haven't actually tried to port a DPDK
> > > driver to it, though it ought to be a pretty obvious mix of the
> > > existing UIO and VFIO versions (set the IOMMU, but avoid using it
> > > for mapping, use however bus translations are done w/ UIO).  The core
> > > vfio device file is still /dev/vfio/vfio, but all the groups become
> > > /dev/vfio-noiommu/$GROUP.
> > > 
> > > It should be obvious, but I always feel obligated to state that this
> > > does not and will not ever enable device assignment to virtual
> > > machines on non-IOMMU capable platforms.
> > 
> > In theory, it's kind of possible using paravirtualization.
> > 
> > Within guest, you'd make map_page retrieve the io address from the host
> > and return that as dma_addr_t.  The only question would be APIs that
> > require more than one contigious page in IO space (e.g. I think alloc
> > coherent is like this?).
> > Not a problem if host is using hugetlbfs, but if not, I guess we could
> > add a hypercall and some Linux API on the host to trigger compaction
> > on the host aggressively. MADV_CONTIGIOUS?
> 
> Not that I see a good reason for that.
> Just use an iommu.

Right, I think it boils down to how much code are we willing to maintain
for an interface that we consider so dangerous and unsupportable that we
immediately taint the kernel.  This is partially why I stopped short of
expanding the no-iommu interface to do page pinning or virt-to-bus
translation.  A few hundred lines of boilerplate stubs to enable re-use
of code that is maintained for supportable interfaces is one thing.
Building onto that with paravirtual IOMMU interfaces to a VM for
something that ultimately cannot be supported is not something I want to
participate in.  Get an IOMMU.  Thanks,

Alex

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1244183

On Fri, Oct 09, 2015 at 12:40:56PM -0600, Alex Williamson wrote:
> Recent patches for UIO have been attempting to add MSI/X support,
> which unfortunately implies DMA support, which users have been
> enabling anyway, but was never intended for UIO.  VFIO on the other
> hand expects an IOMMU to provide isolation of devices, but provides
> a much more complete device interface, which already supports full
> MSI/X support.  There's really no way to support userspace drivers
> with DMA capable devices without an IOMMU to protect the host, but
> we can at least think about doing it in a way that properly taints
> the kernel and avoids creating new code duplicating existing code,
> that does have a supportable use case.
> 
> The diffstat is only so large because I moved vfio.c to vfio_core.c
> so I could more easily keep the module named vfio.ko while keeping
> the bulk of the no-iommu support in a separate file that can be
> optionally compiled.  We're really looking at a couple hundred lines
> of mostly stub code.  The VFIO_NOIOMMU_IOMMU could certainly be
> expanded to do page pinning and virt_to_bus() translation, but I
> didn't want to complicate anything yet.

I think it's already useful like this, since all current users
seem happy enough to just use hugetlbfs to do pinning, and
ignore translation.

> I've only compiled this and tested loading the module with the new
> no-iommu mode enabled, I haven't actually tried to port a DPDK
> driver to it, though it ought to be a pretty obvious mix of the
> existing UIO and VFIO versions (set the IOMMU, but avoid using it
> for mapping, use however bus translations are done w/ UIO).  The core
> vfio device file is still /dev/vfio/vfio, but all the groups become
> /dev/vfio-noiommu/$GROUP.
> 
> It should be obvious, but I always feel obligated to state that this
> does not and will not ever enable device assignment to virtual
> machines on non-IOMMU capable platforms.

In theory, it's kind of possible using paravirtualization.

Within guest, you'd make map_page retrieve the io address from the host
and return that as dma_addr_t.  The only question would be APIs that
require more than one contigious page in IO space (e.g. I think alloc
coherent is like this?).
Not a problem if host is using hugetlbfs, but if not, I guess we could
add a hypercall and some Linux API on the host to trigger compaction
on the host aggressively. MADV_CONTIGIOUS?


> I'm curious what IOMMU folks think of this.  This hack is really
> only possible because we don't use iommu_ops for regular DMA, so we
> can hijack it fairly safely.  I believe that's intended to change
> though, so this may not be practical long term.  Thanks,
> 
> Alex
> 
> ---
> 
> Alex Williamson (2):
>       vfio: Move vfio.c vfio_core.c
>       vfio: Include no-iommu mode
> 
> 
>  drivers/vfio/Kconfig        |   15 
>  drivers/vfio/Makefile       |    4 
>  drivers/vfio/vfio.c         | 1640 ------------------------------------------
>  drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
>  drivers/vfio/vfio_noiommu.c |  185 +++++
>  drivers/vfio/vfio_private.h |   31 +
>  include/uapi/linux/vfio.h   |    2 
>  7 files changed, 1917 insertions(+), 1640 deletions(-)
>  delete mode 100644 drivers/vfio/vfio.c
>  create mode 100644 drivers/vfio/vfio_core.c
>  create mode 100644 drivers/vfio/vfio_noiommu.c
>  create mode 100644 drivers/vfio/vfio_private.h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web