Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1393054 > unrolled thread

RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt remapping is supported

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported "Tian, Kevin" <kevin.tian@intel.com> - 2016-05-03 07:40 +0200
    Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Yongji Xie <xyjxie@linux.vnet.ibm.com> - 2016-05-03 08:10 +0200
      RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported "Tian, Kevin" <kevin.tian@intel.com> - 2016-05-03 08:30 +0200
        Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Yongji Xie <xyjxie@linux.vnet.ibm.com> - 2016-05-03 09:40 +0200
          RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported "Tian, Kevin" <kevin.tian@intel.com> - 2016-05-05 11:40 +0200
            RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported David Laight <David.Laight@ACULAB.COM> - 2016-05-05 12:00 +0200
              Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Yongji Xie <xyjxie@linux.vnet.ibm.com> - 2016-05-05 13:50 +0200
                RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported "Tian, Kevin" <kevin.tian@intel.com> - 2016-05-05 14:20 +0200
                  Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Yongji Xie <xyjxie@linux.vnet.ibm.com> - 2016-05-05 15:30 +0200
                  Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Alex Williamson <alex.williamson@redhat.com> - 2016-05-05 17:10 +0200
                    Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-05-06 08:40 +0200
                      Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Alex Williamson <alex.williamson@redhat.com> - 2016-05-06 19:00 +0200
                    RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported "Tian, Kevin" <kevin.tian@intel.com> - 2016-05-11 08:30 +0200
                      Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Alex Williamson <alex.williamson@redhat.com> - 2016-05-11 18:00 +0200
                        RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported "Tian, Kevin" <kevin.tian@intel.com> - 2016-05-12 03:30 +0200
                          Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Alex Williamson <alex.williamson@redhat.com> - 2016-05-12 04:30 +0200
                            RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported "Tian, Kevin" <kevin.tian@intel.com> - 2016-05-12 07:00 +0200
            Re: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt  remapping is supported Yongji Xie <xyjxie@linux.vnet.ibm.com> - 2016-05-05 13:50 +0200

#1393054 — RE: [PATCH 5/5] vfio-pci: Allow to mmap MSI-X table if interrupt remapping is supported

> From: Yongji Xie
> Sent: Wednesday, April 27, 2016 8:43 PM
> 
> This patch enables mmapping MSI-X tables if hardware supports
> interrupt remapping which can ensure that a given pci device
> can only shoot the MSIs assigned for it.
> 
> With MSI-X table mmapped, we also need to expose the
> read/write interface which will be used to access MSI-X table.
> 
> Signed-off-by: Yongji Xie <xyjxie@linux.vnet.ibm.com>

A curious question here. Does "allow to mmap MSI-X" essentially
mean that KVM guest can directly read/write physical MSI-X
structure then?

Thanks
Kevin

[toc] | [next] | [standalone]

#1393074

On 2016/5/3 13:34, Tian, Kevin wrote:

>> From: Yongji Xie
>> Sent: Wednesday, April 27, 2016 8:43 PM
>>
>> This patch enables mmapping MSI-X tables if hardware supports
>> interrupt remapping which can ensure that a given pci device
>> can only shoot the MSIs assigned for it.
>>
>> With MSI-X table mmapped, we also need to expose the
>> read/write interface which will be used to access MSI-X table.
>>
>> Signed-off-by: Yongji Xie <xyjxie@linux.vnet.ibm.com>
> A curious question here. Does "allow to mmap MSI-X" essentially
> mean that KVM guest can directly read/write physical MSI-X
> structure then?
>
> Thanks
> Kevin
>

Here we just allow to mmap MSI-X table in kernel. It doesn't
mean all KVM guest can directly read/write physical MSI-X
structure. This should be decided by QEMU. For PPC64
platform, we would allow to passthrough the MSI-X table
because we know guest kernel would not write physical
MSI-X structure when enabling MSI.

Thanks,
Yongji

[toc] | [prev] | [next] | [standalone]

#1393080

> From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> Sent: Tuesday, May 03, 2016 2:08 PM
> 
> On 2016/5/3 13:34, Tian, Kevin wrote:
> 
> >> From: Yongji Xie
> >> Sent: Wednesday, April 27, 2016 8:43 PM
> >>
> >> This patch enables mmapping MSI-X tables if hardware supports
> >> interrupt remapping which can ensure that a given pci device
> >> can only shoot the MSIs assigned for it.
> >>
> >> With MSI-X table mmapped, we also need to expose the
> >> read/write interface which will be used to access MSI-X table.
> >>
> >> Signed-off-by: Yongji Xie <xyjxie@linux.vnet.ibm.com>
> > A curious question here. Does "allow to mmap MSI-X" essentially
> > mean that KVM guest can directly read/write physical MSI-X
> > structure then?
> >
> > Thanks
> > Kevin
> >
> 
> Here we just allow to mmap MSI-X table in kernel. It doesn't
> mean all KVM guest can directly read/write physical MSI-X
> structure. This should be decided by QEMU. For PPC64
> platform, we would allow to passthrough the MSI-X table
> because we know guest kernel would not write physical
> MSI-X structure when enabling MSI.
> 

A bit confused here. If guest kernel doesn't need to write
physical MSI-X structure, what's the point of passing through
the table then?

I think the key whether MSI-X table can be passed through
is related to where hypervisor control is deployed. At least
for x86:

- When irq remapping is not enabled, host/hypervisor needs
to control physical interrupt message including vector/dest/etc.
directly in MSI-X structure, so we cannot allow a guest to 
access it;

- when irq remapping is enabled, host/hypervisor can control
interrupt routing in irq remapping table. However MSI-X
also needs to be configured as remappable format. In this
manner we also cannot allow direct access from guest.

The only sane case to pass through MSI-X structure, is a
mechanism similar to irq remapping but w/o need to change
original MSI-X format so direct access from guest side is
safe. Is it the case in PPC64?

Thanks
Kevin

[toc] | [prev] | [next] | [standalone]

#1393111

On 2016/5/3 14:22, Tian, Kevin wrote:

>> From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
>> Sent: Tuesday, May 03, 2016 2:08 PM
>>
>> On 2016/5/3 13:34, Tian, Kevin wrote:
>>
>>>> From: Yongji Xie
>>>> Sent: Wednesday, April 27, 2016 8:43 PM
>>>>
>>>> This patch enables mmapping MSI-X tables if hardware supports
>>>> interrupt remapping which can ensure that a given pci device
>>>> can only shoot the MSIs assigned for it.
>>>>
>>>> With MSI-X table mmapped, we also need to expose the
>>>> read/write interface which will be used to access MSI-X table.
>>>>
>>>> Signed-off-by: Yongji Xie <xyjxie@linux.vnet.ibm.com>
>>> A curious question here. Does "allow to mmap MSI-X" essentially
>>> mean that KVM guest can directly read/write physical MSI-X
>>> structure then?
>>>
>>> Thanks
>>> Kevin
>>>
>> Here we just allow to mmap MSI-X table in kernel. It doesn't
>> mean all KVM guest can directly read/write physical MSI-X
>> structure. This should be decided by QEMU. For PPC64
>> platform, we would allow to passthrough the MSI-X table
>> because we know guest kernel would not write physical
>> MSI-X structure when enabling MSI.
>>
> A bit confused here. If guest kernel doesn't need to write
> physical MSI-X structure, what's the point of passing through
> the table then?

We want to allow the MSI-X table because there may be
some critical registers in the same page as the MSI-X table.
We have to handle the mmio access to these register in QEMU
rather than in guest if mmapping MSI-X table is disallowed.

> I think the key whether MSI-X table can be passed through
> is related to where hypervisor control is deployed. At least
> for x86:
>
> - When irq remapping is not enabled, host/hypervisor needs
> to control physical interrupt message including vector/dest/etc.
> directly in MSI-X structure, so we cannot allow a guest to
> access it;
>
> - when irq remapping is enabled, host/hypervisor can control
> interrupt routing in irq remapping table. However MSI-X
> also needs to be configured as remappable format. In this
> manner we also cannot allow direct access from guest.
>
> The only sane case to pass through MSI-X structure, is a
> mechanism similar to irq remapping but w/o need to change
> original MSI-X format so direct access from guest side is
> safe. Is it the case in PPC64?
>
> Thanks
> Kevin

Acutually, we are not aimed at accessing MSI-X table from
guest. So I think it's safe to passthrough MSI-X table if we
can make sure guest kernel would not touch MSI-X table in
normal code path such as para-virtualized guest kernel on PPC64.

Thanks,
Yongji

[toc] | [prev] | [next] | [standalone]

#1394966

> From: Yongji Xie
> Sent: Tuesday, May 03, 2016 3:34 PM
> 
> On 2016/5/3 14:22, Tian, Kevin wrote:
> 
> >> From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> >> Sent: Tuesday, May 03, 2016 2:08 PM
> >>
> >> On 2016/5/3 13:34, Tian, Kevin wrote:
> >>
> >>>> From: Yongji Xie
> >>>> Sent: Wednesday, April 27, 2016 8:43 PM
> >>>>
> >>>> This patch enables mmapping MSI-X tables if hardware supports
> >>>> interrupt remapping which can ensure that a given pci device
> >>>> can only shoot the MSIs assigned for it.
> >>>>
> >>>> With MSI-X table mmapped, we also need to expose the
> >>>> read/write interface which will be used to access MSI-X table.
> >>>>
> >>>> Signed-off-by: Yongji Xie <xyjxie@linux.vnet.ibm.com>
> >>> A curious question here. Does "allow to mmap MSI-X" essentially
> >>> mean that KVM guest can directly read/write physical MSI-X
> >>> structure then?
> >>>
> >>> Thanks
> >>> Kevin
> >>>
> >> Here we just allow to mmap MSI-X table in kernel. It doesn't
> >> mean all KVM guest can directly read/write physical MSI-X
> >> structure. This should be decided by QEMU. For PPC64
> >> platform, we would allow to passthrough the MSI-X table
> >> because we know guest kernel would not write physical
> >> MSI-X structure when enabling MSI.
> >>
> > A bit confused here. If guest kernel doesn't need to write
> > physical MSI-X structure, what's the point of passing through
> > the table then?
> 
> We want to allow the MSI-X table because there may be
> some critical registers in the same page as the MSI-X table.
> We have to handle the mmio access to these register in QEMU
> rather than in guest if mmapping MSI-X table is disallowed.

So you mean critical registers in same MMIO BAR as MSI-X
table, instead of two MMIO BARs in same page (the latter I
suppose with your whole patchset it won't happen then)?

> 
> > I think the key whether MSI-X table can be passed through
> > is related to where hypervisor control is deployed. At least
> > for x86:
> >
> > - When irq remapping is not enabled, host/hypervisor needs
> > to control physical interrupt message including vector/dest/etc.
> > directly in MSI-X structure, so we cannot allow a guest to
> > access it;
> >
> > - when irq remapping is enabled, host/hypervisor can control
> > interrupt routing in irq remapping table. However MSI-X
> > also needs to be configured as remappable format. In this
> > manner we also cannot allow direct access from guest.
> >
> > The only sane case to pass through MSI-X structure, is a
> > mechanism similar to irq remapping but w/o need to change
> > original MSI-X format so direct access from guest side is
> > safe. Is it the case in PPC64?
> >
> > Thanks
> > Kevin
> 
> Acutually, we are not aimed at accessing MSI-X table from
> guest. So I think it's safe to passthrough MSI-X table if we
> can make sure guest kernel would not touch MSI-X table in
> normal code path such as para-virtualized guest kernel on PPC64.
> 

Then how do you prevent malicious guest kernel accessing it?

Thanks
Kevin

[toc] | [prev] | [next] | [standalone]

#1395009

From: Tian, Kevin
> Sent: 05 May 2016 10:37
...
> > Acutually, we are not aimed at accessing MSI-X table from
> > guest. So I think it's safe to passthrough MSI-X table if we
> > can make sure guest kernel would not touch MSI-X table in
> > normal code path such as para-virtualized guest kernel on PPC64.
> >
> 
> Then how do you prevent malicious guest kernel accessing it?

Or a malicious guest driver for an ethernet card setting up
the receive buffer ring to contain a single word entry that
contains the address associated with an MSI-X interrupt and
then using a loopback mode to cause a specific packet be
received that writes the required word through that address.

Remember the PCIe cycle for an interrupt is a normal memory write
cycle.

	David

[toc] | [prev] | [next] | [standalone]

#1395056

Hi David and Kevin,

On 2016/5/5 17:54, David Laight wrote:

> From: Tian, Kevin
>> Sent: 05 May 2016 10:37
> ...
>>> Acutually, we are not aimed at accessing MSI-X table from
>>> guest. So I think it's safe to passthrough MSI-X table if we
>>> can make sure guest kernel would not touch MSI-X table in
>>> normal code path such as para-virtualized guest kernel on PPC64.
>>>
>> Then how do you prevent malicious guest kernel accessing it?
> Or a malicious guest driver for an ethernet card setting up
> the receive buffer ring to contain a single word entry that
> contains the address associated with an MSI-X interrupt and
> then using a loopback mode to cause a specific packet be
> received that writes the required word through that address.
>
> Remember the PCIe cycle for an interrupt is a normal memory write
> cycle.
>
> 	David
>

If we have enough permission to load a malicious driver or
kernel, we can easily break the guest without exposed
MSI-X table.

I think it should be safe to expose MSI-X table if we can
make sure that malicious guest driver/kernel can't use
the MSI-X table to break other guest or host. The
capability of IRQ remapping could provide this
kind of protection.

Thanks,
Yongji

[toc] | [prev] | [next] | [standalone]

#1395065

> From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> Sent: Thursday, May 05, 2016 7:43 PM
> 
> Hi David and Kevin,
> 
> On 2016/5/5 17:54, David Laight wrote:
> 
> > From: Tian, Kevin
> >> Sent: 05 May 2016 10:37
> > ...
> >>> Acutually, we are not aimed at accessing MSI-X table from
> >>> guest. So I think it's safe to passthrough MSI-X table if we
> >>> can make sure guest kernel would not touch MSI-X table in
> >>> normal code path such as para-virtualized guest kernel on PPC64.
> >>>
> >> Then how do you prevent malicious guest kernel accessing it?
> > Or a malicious guest driver for an ethernet card setting up
> > the receive buffer ring to contain a single word entry that
> > contains the address associated with an MSI-X interrupt and
> > then using a loopback mode to cause a specific packet be
> > received that writes the required word through that address.
> >
> > Remember the PCIe cycle for an interrupt is a normal memory write
> > cycle.
> >
> > 	David
> >
> 
> If we have enough permission to load a malicious driver or
> kernel, we can easily break the guest without exposed
> MSI-X table.
> 
> I think it should be safe to expose MSI-X table if we can
> make sure that malicious guest driver/kernel can't use
> the MSI-X table to break other guest or host. The
> capability of IRQ remapping could provide this
> kind of protection.
> 

With IRQ remapping it doesn't mean you can pass through MSI-X
structure to guest. I know actual IRQ remapping might be platform
specific, but at least for Intel VT-d specification, MSI-X entry must
be configured with a remappable format by host kernel which
contains an index into IRQ remapping table. The index will find a
IRQ remapping entry which controls interrupt routing for a specific
device. If you allow a malicious program random index into MSI-X 
entry of assigned device, the hole is obvious...

Above might make sense only for a IRQ remapping implementation 
which doesn't rely on extended MSI-X format (e.g. simply based on 
BDF). If that's the case for PPC, then you should build MSI-X 
passthrough based on this fact instead of general IRQ remapping 
enabled or not.

Thanks
Kevin

[toc] | [prev] | [next] | [standalone]

#1395121

On 2016/5/5 20:15, Tian, Kevin wrote:

>> From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
>> Sent: Thursday, May 05, 2016 7:43 PM
>>
>> Hi David and Kevin,
>>
>> On 2016/5/5 17:54, David Laight wrote:
>>
>>> From: Tian, Kevin
>>>> Sent: 05 May 2016 10:37
>>> ...
>>>>> Acutually, we are not aimed at accessing MSI-X table from
>>>>> guest. So I think it's safe to passthrough MSI-X table if we
>>>>> can make sure guest kernel would not touch MSI-X table in
>>>>> normal code path such as para-virtualized guest kernel on PPC64.
>>>>>
>>>> Then how do you prevent malicious guest kernel accessing it?
>>> Or a malicious guest driver for an ethernet card setting up
>>> the receive buffer ring to contain a single word entry that
>>> contains the address associated with an MSI-X interrupt and
>>> then using a loopback mode to cause a specific packet be
>>> received that writes the required word through that address.
>>>
>>> Remember the PCIe cycle for an interrupt is a normal memory write
>>> cycle.
>>>
>>> 	David
>>>
>> If we have enough permission to load a malicious driver or
>> kernel, we can easily break the guest without exposed
>> MSI-X table.
>>
>> I think it should be safe to expose MSI-X table if we can
>> make sure that malicious guest driver/kernel can't use
>> the MSI-X table to break other guest or host. The
>> capability of IRQ remapping could provide this
>> kind of protection.
>>
> With IRQ remapping it doesn't mean you can pass through MSI-X
> structure to guest. I know actual IRQ remapping might be platform
> specific, but at least for Intel VT-d specification, MSI-X entry must
> be configured with a remappable format by host kernel which
> contains an index into IRQ remapping table. The index will find a
> IRQ remapping entry which controls interrupt routing for a specific
> device. If you allow a malicious program random index into MSI-X
> entry of assigned device, the hole is obvious...

Do you mean we can trigger MSIs that correspond to interrupt
IDs of other devices by writing to MSI-X table although IRQ
remapping is enabled?

On PPC64, there is a mapping between MSIs and PE num
which can be used to identify a PCI device on PHB. So the
hardware can ensure a given pci device can only shoot the
MSIs assigned for it.  Isn't there a similar mapping in IRQ
remapping table on Intel.

Thanks,
Yongji

> Above might make sense only for a IRQ remapping implementation
> which doesn't rely on extended MSI-X format (e.g. simply based on
> BDF). If that's the case for PPC, then you should build MSI-X
> passthrough based on this fact instead of general IRQ remapping
> enabled or not.
>
> Thanks
> Kevin

[toc] | [prev] | [next] | [standalone]

#1395170

On Thu, 5 May 2016 12:15:46 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> > Sent: Thursday, May 05, 2016 7:43 PM
> > 
> > Hi David and Kevin,
> > 
> > On 2016/5/5 17:54, David Laight wrote:
> >   
> > > From: Tian, Kevin  
> > >> Sent: 05 May 2016 10:37  
> > > ...  
> > >>> Acutually, we are not aimed at accessing MSI-X table from
> > >>> guest. So I think it's safe to passthrough MSI-X table if we
> > >>> can make sure guest kernel would not touch MSI-X table in
> > >>> normal code path such as para-virtualized guest kernel on PPC64.
> > >>>  
> > >> Then how do you prevent malicious guest kernel accessing it?  
> > > Or a malicious guest driver for an ethernet card setting up
> > > the receive buffer ring to contain a single word entry that
> > > contains the address associated with an MSI-X interrupt and
> > > then using a loopback mode to cause a specific packet be
> > > received that writes the required word through that address.
> > >
> > > Remember the PCIe cycle for an interrupt is a normal memory write
> > > cycle.
> > >
> > > 	David
> > >  
> > 
> > If we have enough permission to load a malicious driver or
> > kernel, we can easily break the guest without exposed
> > MSI-X table.
> > 
> > I think it should be safe to expose MSI-X table if we can
> > make sure that malicious guest driver/kernel can't use
> > the MSI-X table to break other guest or host. The
> > capability of IRQ remapping could provide this
> > kind of protection.
> >   
> 
> With IRQ remapping it doesn't mean you can pass through MSI-X
> structure to guest. I know actual IRQ remapping might be platform
> specific, but at least for Intel VT-d specification, MSI-X entry must
> be configured with a remappable format by host kernel which
> contains an index into IRQ remapping table. The index will find a
> IRQ remapping entry which controls interrupt routing for a specific
> device. If you allow a malicious program random index into MSI-X 
> entry of assigned device, the hole is obvious...
> 
> Above might make sense only for a IRQ remapping implementation 
> which doesn't rely on extended MSI-X format (e.g. simply based on 
> BDF). If that's the case for PPC, then you should build MSI-X 
> passthrough based on this fact instead of general IRQ remapping 
> enabled or not.

I don't think anyone is expecting that we can expose the MSI-X vector
table to the guest and the guest can make direct use of it.  The end
goal here is that the guest on a power system is already
paravirtualized to not program the device MSI-X by directly writing to
the MSI-X vector table.  They have hypercalls for this since they
always run virtualized.  Therefore a) they never intend to touch the
MSI-X vector table and b) they have sufficient isolation that a guest
can only hurt itself by doing so.

On x86 we don't have a), our method of programming the MSI-X vector
table is to directly write to it. Therefore we will always require QEMU
to place a MemoryRegion over the vector table to intercept those
accesses.  However with interrupt remapping, we do have b) on x86, which
means that we don't need to be so strict in disallowing user accesses
to the MSI-X vector table.  It's not useful for configuring MSI-X on
the device, but the user should only be able to hurt themselves by
writing it directly.  x86 doesn't really get anything out of this
change, but it helps this special case on power pretty significantly
aiui.  Thanks,

Alex

[toc] | [prev] | [next] | [standalone]

#1395625

On 05/06/2016 01:05 AM, Alex Williamson wrote:
> On Thu, 5 May 2016 12:15:46 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
>>> From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
>>> Sent: Thursday, May 05, 2016 7:43 PM
>>>
>>> Hi David and Kevin,
>>>
>>> On 2016/5/5 17:54, David Laight wrote:
>>>
>>>> From: Tian, Kevin
>>>>> Sent: 05 May 2016 10:37
>>>> ...
>>>>>> Acutually, we are not aimed at accessing MSI-X table from
>>>>>> guest. So I think it's safe to passthrough MSI-X table if we
>>>>>> can make sure guest kernel would not touch MSI-X table in
>>>>>> normal code path such as para-virtualized guest kernel on PPC64.
>>>>>>
>>>>> Then how do you prevent malicious guest kernel accessing it?
>>>> Or a malicious guest driver for an ethernet card setting up
>>>> the receive buffer ring to contain a single word entry that
>>>> contains the address associated with an MSI-X interrupt and
>>>> then using a loopback mode to cause a specific packet be
>>>> received that writes the required word through that address.
>>>>
>>>> Remember the PCIe cycle for an interrupt is a normal memory write
>>>> cycle.
>>>>
>>>> 	David
>>>>
>>>
>>> If we have enough permission to load a malicious driver or
>>> kernel, we can easily break the guest without exposed
>>> MSI-X table.
>>>
>>> I think it should be safe to expose MSI-X table if we can
>>> make sure that malicious guest driver/kernel can't use
>>> the MSI-X table to break other guest or host. The
>>> capability of IRQ remapping could provide this
>>> kind of protection.
>>>
>>
>> With IRQ remapping it doesn't mean you can pass through MSI-X
>> structure to guest. I know actual IRQ remapping might be platform
>> specific, but at least for Intel VT-d specification, MSI-X entry must
>> be configured with a remappable format by host kernel which
>> contains an index into IRQ remapping table. The index will find a
>> IRQ remapping entry which controls interrupt routing for a specific
>> device. If you allow a malicious program random index into MSI-X
>> entry of assigned device, the hole is obvious...
>>
>> Above might make sense only for a IRQ remapping implementation
>> which doesn't rely on extended MSI-X format (e.g. simply based on
>> BDF). If that's the case for PPC, then you should build MSI-X
>> passthrough based on this fact instead of general IRQ remapping
>> enabled or not.
>
> I don't think anyone is expecting that we can expose the MSI-X vector
> table to the guest and the guest can make direct use of it.  The end
> goal here is that the guest on a power system is already
> paravirtualized to not program the device MSI-X by directly writing to
> the MSI-X vector table.  They have hypercalls for this since they
> always run virtualized.  Therefore a) they never intend to touch the
> MSI-X vector table and b) they have sufficient isolation that a guest
> can only hurt itself by doing so.
>
> On x86 we don't have a), our method of programming the MSI-X vector
> table is to directly write to it. Therefore we will always require QEMU
> to place a MemoryRegion over the vector table to intercept those
> accesses.  However with interrupt remapping, we do have b) on x86, which
> means that we don't need to be so strict in disallowing user accesses
> to the MSI-X vector table.  It's not useful for configuring MSI-X on
> the device, but the user should only be able to hurt themselves by
> writing it directly.  x86 doesn't really get anything out of this
> change, but it helps this special case on power pretty significantly
> aiui.  Thanks,

Excellent short overview, saved :)

How do we proceed with these patches? Nobody seems objecting them but also 
nobody seems taking them either...




-- 
Alexey

[toc] | [prev] | [next] | [standalone]

#1395977

On Fri, 6 May 2016 16:35:38 +1000
Alexey Kardashevskiy <aik@ozlabs.ru> wrote:

> On 05/06/2016 01:05 AM, Alex Williamson wrote:
> > On Thu, 5 May 2016 12:15:46 +0000
> > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> >  
> >>> From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> >>> Sent: Thursday, May 05, 2016 7:43 PM
> >>>
> >>> Hi David and Kevin,
> >>>
> >>> On 2016/5/5 17:54, David Laight wrote:
> >>>  
> >>>> From: Tian, Kevin  
> >>>>> Sent: 05 May 2016 10:37  
> >>>> ...  
> >>>>>> Acutually, we are not aimed at accessing MSI-X table from
> >>>>>> guest. So I think it's safe to passthrough MSI-X table if we
> >>>>>> can make sure guest kernel would not touch MSI-X table in
> >>>>>> normal code path such as para-virtualized guest kernel on PPC64.
> >>>>>>  
> >>>>> Then how do you prevent malicious guest kernel accessing it?  
> >>>> Or a malicious guest driver for an ethernet card setting up
> >>>> the receive buffer ring to contain a single word entry that
> >>>> contains the address associated with an MSI-X interrupt and
> >>>> then using a loopback mode to cause a specific packet be
> >>>> received that writes the required word through that address.
> >>>>
> >>>> Remember the PCIe cycle for an interrupt is a normal memory write
> >>>> cycle.
> >>>>
> >>>> 	David
> >>>>  
> >>>
> >>> If we have enough permission to load a malicious driver or
> >>> kernel, we can easily break the guest without exposed
> >>> MSI-X table.
> >>>
> >>> I think it should be safe to expose MSI-X table if we can
> >>> make sure that malicious guest driver/kernel can't use
> >>> the MSI-X table to break other guest or host. The
> >>> capability of IRQ remapping could provide this
> >>> kind of protection.
> >>>  
> >>
> >> With IRQ remapping it doesn't mean you can pass through MSI-X
> >> structure to guest. I know actual IRQ remapping might be platform
> >> specific, but at least for Intel VT-d specification, MSI-X entry must
> >> be configured with a remappable format by host kernel which
> >> contains an index into IRQ remapping table. The index will find a
> >> IRQ remapping entry which controls interrupt routing for a specific
> >> device. If you allow a malicious program random index into MSI-X
> >> entry of assigned device, the hole is obvious...
> >>
> >> Above might make sense only for a IRQ remapping implementation
> >> which doesn't rely on extended MSI-X format (e.g. simply based on
> >> BDF). If that's the case for PPC, then you should build MSI-X
> >> passthrough based on this fact instead of general IRQ remapping
> >> enabled or not.  
> >
> > I don't think anyone is expecting that we can expose the MSI-X vector
> > table to the guest and the guest can make direct use of it.  The end
> > goal here is that the guest on a power system is already
> > paravirtualized to not program the device MSI-X by directly writing to
> > the MSI-X vector table.  They have hypercalls for this since they
> > always run virtualized.  Therefore a) they never intend to touch the
> > MSI-X vector table and b) they have sufficient isolation that a guest
> > can only hurt itself by doing so.
> >
> > On x86 we don't have a), our method of programming the MSI-X vector
> > table is to directly write to it. Therefore we will always require QEMU
> > to place a MemoryRegion over the vector table to intercept those
> > accesses.  However with interrupt remapping, we do have b) on x86, which
> > means that we don't need to be so strict in disallowing user accesses
> > to the MSI-X vector table.  It's not useful for configuring MSI-X on
> > the device, but the user should only be able to hurt themselves by
> > writing it directly.  x86 doesn't really get anything out of this
> > change, but it helps this special case on power pretty significantly
> > aiui.  Thanks,  
> 
> Excellent short overview, saved :)
> 
> How do we proceed with these patches? Nobody seems objecting them but also 
> nobody seems taking them either...

Well, this series is still based on some non-upstream patches, so...
Once that dependency is resolved this series should probably be split
into functional areas for acceptance by the appropriate subsystem
maintainers.

[toc] | [prev] | [next] | [standalone]

#1398710

> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Thursday, May 05, 2016 11:05 PM
> 
> On Thu, 5 May 2016 12:15:46 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > > From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> > > Sent: Thursday, May 05, 2016 7:43 PM
> > >
> > > Hi David and Kevin,
> > >
> > > On 2016/5/5 17:54, David Laight wrote:
> > >
> > > > From: Tian, Kevin
> > > >> Sent: 05 May 2016 10:37
> > > > ...
> > > >>> Acutually, we are not aimed at accessing MSI-X table from
> > > >>> guest. So I think it's safe to passthrough MSI-X table if we
> > > >>> can make sure guest kernel would not touch MSI-X table in
> > > >>> normal code path such as para-virtualized guest kernel on PPC64.
> > > >>>
> > > >> Then how do you prevent malicious guest kernel accessing it?
> > > > Or a malicious guest driver for an ethernet card setting up
> > > > the receive buffer ring to contain a single word entry that
> > > > contains the address associated with an MSI-X interrupt and
> > > > then using a loopback mode to cause a specific packet be
> > > > received that writes the required word through that address.
> > > >
> > > > Remember the PCIe cycle for an interrupt is a normal memory write
> > > > cycle.
> > > >
> > > > 	David
> > > >
> > >
> > > If we have enough permission to load a malicious driver or
> > > kernel, we can easily break the guest without exposed
> > > MSI-X table.
> > >
> > > I think it should be safe to expose MSI-X table if we can
> > > make sure that malicious guest driver/kernel can't use
> > > the MSI-X table to break other guest or host. The
> > > capability of IRQ remapping could provide this
> > > kind of protection.
> > >
> >
> > With IRQ remapping it doesn't mean you can pass through MSI-X
> > structure to guest. I know actual IRQ remapping might be platform
> > specific, but at least for Intel VT-d specification, MSI-X entry must
> > be configured with a remappable format by host kernel which
> > contains an index into IRQ remapping table. The index will find a
> > IRQ remapping entry which controls interrupt routing for a specific
> > device. If you allow a malicious program random index into MSI-X
> > entry of assigned device, the hole is obvious...
> >
> > Above might make sense only for a IRQ remapping implementation
> > which doesn't rely on extended MSI-X format (e.g. simply based on
> > BDF). If that's the case for PPC, then you should build MSI-X
> > passthrough based on this fact instead of general IRQ remapping
> > enabled or not.
> 
> I don't think anyone is expecting that we can expose the MSI-X vector
> table to the guest and the guest can make direct use of it.  The end
> goal here is that the guest on a power system is already
> paravirtualized to not program the device MSI-X by directly writing to
> the MSI-X vector table.  They have hypercalls for this since they
> always run virtualized.  Therefore a) they never intend to touch the
> MSI-X vector table and b) they have sufficient isolation that a guest
> can only hurt itself by doing so.
> 
> On x86 we don't have a), our method of programming the MSI-X vector
> table is to directly write to it. Therefore we will always require QEMU
> to place a MemoryRegion over the vector table to intercept those
> accesses.  However with interrupt remapping, we do have b) on x86, which
> means that we don't need to be so strict in disallowing user accesses
> to the MSI-X vector table.  It's not useful for configuring MSI-X on
> the device, but the user should only be able to hurt themselves by
> writing it directly.  x86 doesn't really get anything out of this
> change, but it helps this special case on power pretty significantly
> aiui.  Thanks,
> 

Allowing guest direct write to MSI-x table has system-wide impact.
As I explained earlier, hypervisor needs to control "interrupt_index"
programmed in MSI-X entry, which is used to associate a specific
IRQ remapping entry. Now if you expose whole MSI-x table to guest, 
it can program random index into MSI-X entry to associate with 
any IRQ remapping entry and then there won't be any isolation per se.

You can check "5.5.2 MSI and MSI-X Register Programming" in VT-d
spec.

Thanks
Kevin

[toc] | [prev] | [next] | [standalone]

#1399289

On Wed, 11 May 2016 06:29:06 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Thursday, May 05, 2016 11:05 PM
> > 
> > On Thu, 5 May 2016 12:15:46 +0000
> > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> >   
> > > > From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> > > > Sent: Thursday, May 05, 2016 7:43 PM
> > > >
> > > > Hi David and Kevin,
> > > >
> > > > On 2016/5/5 17:54, David Laight wrote:
> > > >  
> > > > > From: Tian, Kevin  
> > > > >> Sent: 05 May 2016 10:37  
> > > > > ...  
> > > > >>> Acutually, we are not aimed at accessing MSI-X table from
> > > > >>> guest. So I think it's safe to passthrough MSI-X table if we
> > > > >>> can make sure guest kernel would not touch MSI-X table in
> > > > >>> normal code path such as para-virtualized guest kernel on PPC64.
> > > > >>>  
> > > > >> Then how do you prevent malicious guest kernel accessing it?  
> > > > > Or a malicious guest driver for an ethernet card setting up
> > > > > the receive buffer ring to contain a single word entry that
> > > > > contains the address associated with an MSI-X interrupt and
> > > > > then using a loopback mode to cause a specific packet be
> > > > > received that writes the required word through that address.
> > > > >
> > > > > Remember the PCIe cycle for an interrupt is a normal memory write
> > > > > cycle.
> > > > >
> > > > > 	David
> > > > >  
> > > >
> > > > If we have enough permission to load a malicious driver or
> > > > kernel, we can easily break the guest without exposed
> > > > MSI-X table.
> > > >
> > > > I think it should be safe to expose MSI-X table if we can
> > > > make sure that malicious guest driver/kernel can't use
> > > > the MSI-X table to break other guest or host. The
> > > > capability of IRQ remapping could provide this
> > > > kind of protection.
> > > >  
> > >
> > > With IRQ remapping it doesn't mean you can pass through MSI-X
> > > structure to guest. I know actual IRQ remapping might be platform
> > > specific, but at least for Intel VT-d specification, MSI-X entry must
> > > be configured with a remappable format by host kernel which
> > > contains an index into IRQ remapping table. The index will find a
> > > IRQ remapping entry which controls interrupt routing for a specific
> > > device. If you allow a malicious program random index into MSI-X
> > > entry of assigned device, the hole is obvious...
> > >
> > > Above might make sense only for a IRQ remapping implementation
> > > which doesn't rely on extended MSI-X format (e.g. simply based on
> > > BDF). If that's the case for PPC, then you should build MSI-X
> > > passthrough based on this fact instead of general IRQ remapping
> > > enabled or not.  
> > 
> > I don't think anyone is expecting that we can expose the MSI-X vector
> > table to the guest and the guest can make direct use of it.  The end
> > goal here is that the guest on a power system is already
> > paravirtualized to not program the device MSI-X by directly writing to
> > the MSI-X vector table.  They have hypercalls for this since they
> > always run virtualized.  Therefore a) they never intend to touch the
> > MSI-X vector table and b) they have sufficient isolation that a guest
> > can only hurt itself by doing so.
> > 
> > On x86 we don't have a), our method of programming the MSI-X vector
> > table is to directly write to it. Therefore we will always require QEMU
> > to place a MemoryRegion over the vector table to intercept those
> > accesses.  However with interrupt remapping, we do have b) on x86, which
> > means that we don't need to be so strict in disallowing user accesses
> > to the MSI-X vector table.  It's not useful for configuring MSI-X on
> > the device, but the user should only be able to hurt themselves by
> > writing it directly.  x86 doesn't really get anything out of this
> > change, but it helps this special case on power pretty significantly
> > aiui.  Thanks,
> >   
> 
> Allowing guest direct write to MSI-x table has system-wide impact.
> As I explained earlier, hypervisor needs to control "interrupt_index"
> programmed in MSI-X entry, which is used to associate a specific
> IRQ remapping entry. Now if you expose whole MSI-x table to guest, 
> it can program random index into MSI-X entry to associate with 
> any IRQ remapping entry and then there won't be any isolation per se.
> 
> You can check "5.5.2 MSI and MSI-X Register Programming" in VT-d
> spec.

I think you're extrapolating beyond the vfio interface.  The change
here is to remove the vfio protection of the MSI-X vector table when
the system provides interrupt isolation.  The argument is that this is
safe to do because the hardware protects the host from erroneous and
malicious user programming, but it is not meant to provide a means to
program MSI-X directly through the vector table.  This is effectively
the same as general DMA programming, if the vfio programming model is
not followed the device generates iommu faults.  I do have a concern
that userspace driver writers are going to more often presume they can
use the vector table directly because of this change, but I don't know
that that is sufficient reason to prevent such a change.  They'll
quickly discover the device generates faults on interrupt rather than
working as expected.

The question of how this affects the hypervisor is completely
separate.  Vfio in the kernel is a userspace driver interface, not a
hypervisor.  QEMU is the hypervisor.  We have no plans to provide the VM
with direct access to the MSI-X vector table for x86 guests on QEMU.
There will still be a MemoryRegion emulating access to the vector table
in order to translate writes into vfio interrupt ioctls.  POWER would
drop the MemoryRegion so that the full page is mapped to the guest,
with the expectation that the guest never makes use of it since MSI-X
is always configured via hypercalls on POWER systems.  Likewise I
expect ARM will still make use of the MemoryRegion emulating the vector
table, which leaves them exposed to the performance issue POWER is
trying to solve here since ARM also has 64k page support and has no
paravirtualized MSI-X programming interface afaik.  x86 is not
impervious to this issue either, but a 4k page size falls within the
PCI spec recommendations for MSI-X structure alignment, so it's much
more rare to have issues. We have certainly seen hardware vendors that
ignore the PCI spec alignment recommendations, but so far only for
placing device registers within the same page as the PBA, which is an
easier problem to deal with since the PBA is relatively unused by
drivers.  This may be an area where we need to develop a paravirt
interface for MSI-X programming which disable the MemoryRegion
emulating the vector table when used.  Thanks,

Alex

[toc] | [prev] | [next] | [standalone]

#1399631

> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Wednesday, May 11, 2016 11:54 PM
> 
> On Wed, 11 May 2016 06:29:06 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > > Sent: Thursday, May 05, 2016 11:05 PM
> > >
> > > On Thu, 5 May 2016 12:15:46 +0000
> > > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> > >
> > > > > From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> > > > > Sent: Thursday, May 05, 2016 7:43 PM
> > > > >
> > > > > Hi David and Kevin,
> > > > >
> > > > > On 2016/5/5 17:54, David Laight wrote:
> > > > >
> > > > > > From: Tian, Kevin
> > > > > >> Sent: 05 May 2016 10:37
> > > > > > ...
> > > > > >>> Acutually, we are not aimed at accessing MSI-X table from
> > > > > >>> guest. So I think it's safe to passthrough MSI-X table if we
> > > > > >>> can make sure guest kernel would not touch MSI-X table in
> > > > > >>> normal code path such as para-virtualized guest kernel on PPC64.
> > > > > >>>
> > > > > >> Then how do you prevent malicious guest kernel accessing it?
> > > > > > Or a malicious guest driver for an ethernet card setting up
> > > > > > the receive buffer ring to contain a single word entry that
> > > > > > contains the address associated with an MSI-X interrupt and
> > > > > > then using a loopback mode to cause a specific packet be
> > > > > > received that writes the required word through that address.
> > > > > >
> > > > > > Remember the PCIe cycle for an interrupt is a normal memory write
> > > > > > cycle.
> > > > > >
> > > > > > 	David
> > > > > >
> > > > >
> > > > > If we have enough permission to load a malicious driver or
> > > > > kernel, we can easily break the guest without exposed
> > > > > MSI-X table.
> > > > >
> > > > > I think it should be safe to expose MSI-X table if we can
> > > > > make sure that malicious guest driver/kernel can't use
> > > > > the MSI-X table to break other guest or host. The
> > > > > capability of IRQ remapping could provide this
> > > > > kind of protection.
> > > > >
> > > >
> > > > With IRQ remapping it doesn't mean you can pass through MSI-X
> > > > structure to guest. I know actual IRQ remapping might be platform
> > > > specific, but at least for Intel VT-d specification, MSI-X entry must
> > > > be configured with a remappable format by host kernel which
> > > > contains an index into IRQ remapping table. The index will find a
> > > > IRQ remapping entry which controls interrupt routing for a specific
> > > > device. If you allow a malicious program random index into MSI-X
> > > > entry of assigned device, the hole is obvious...
> > > >
> > > > Above might make sense only for a IRQ remapping implementation
> > > > which doesn't rely on extended MSI-X format (e.g. simply based on
> > > > BDF). If that's the case for PPC, then you should build MSI-X
> > > > passthrough based on this fact instead of general IRQ remapping
> > > > enabled or not.
> > >
> > > I don't think anyone is expecting that we can expose the MSI-X vector
> > > table to the guest and the guest can make direct use of it.  The end
> > > goal here is that the guest on a power system is already
> > > paravirtualized to not program the device MSI-X by directly writing to
> > > the MSI-X vector table.  They have hypercalls for this since they
> > > always run virtualized.  Therefore a) they never intend to touch the
> > > MSI-X vector table and b) they have sufficient isolation that a guest
> > > can only hurt itself by doing so.
> > >
> > > On x86 we don't have a), our method of programming the MSI-X vector
> > > table is to directly write to it. Therefore we will always require QEMU
> > > to place a MemoryRegion over the vector table to intercept those
> > > accesses.  However with interrupt remapping, we do have b) on x86, which
> > > means that we don't need to be so strict in disallowing user accesses
> > > to the MSI-X vector table.  It's not useful for configuring MSI-X on
> > > the device, but the user should only be able to hurt themselves by
> > > writing it directly.  x86 doesn't really get anything out of this
> > > change, but it helps this special case on power pretty significantly
> > > aiui.  Thanks,
> > >
> >
> > Allowing guest direct write to MSI-x table has system-wide impact.
> > As I explained earlier, hypervisor needs to control "interrupt_index"
> > programmed in MSI-X entry, which is used to associate a specific
> > IRQ remapping entry. Now if you expose whole MSI-x table to guest,
> > it can program random index into MSI-X entry to associate with
> > any IRQ remapping entry and then there won't be any isolation per se.
> >
> > You can check "5.5.2 MSI and MSI-X Register Programming" in VT-d
> > spec.
> 
> I think you're extrapolating beyond the vfio interface.  The change
> here is to remove the vfio protection of the MSI-X vector table when
> the system provides interrupt isolation.  The argument is that this is
> safe to do because the hardware protects the host from erroneous and
> malicious user programming, but it is not meant to provide a means to
> program MSI-X directly through the vector table.  This is effectively

Sorry I didn't get this point. Once we allow userspace to mmap MSI-X
table, isn't it equivalent to allowing userspace directly program vector
table? Or is there other mechanism to prevent direct programming?

> the same as general DMA programming, if the vfio programming model is
> not followed the device generates iommu faults.  I do have a concern
> that userspace driver writers are going to more often presume they can
> use the vector table directly because of this change, but I don't know
> that that is sufficient reason to prevent such a change.  They'll
> quickly discover the device generates faults on interrupt rather than
> working as expected.

If userspace can actually program vector table directly, there is not
always fault triggered. As long as MSI-X table is fully under control
of userspace, any interrupt index can be used here which may link
to a IRQ remapping entry allocated for other devices.

> 
> The question of how this affects the hypervisor is completely
> separate.  Vfio in the kernel is a userspace driver interface, not a
> hypervisor.  QEMU is the hypervisor.  We have no plans to provide the VM
> with direct access to the MSI-X vector table for x86 guests on QEMU.
> There will still be a MemoryRegion emulating access to the vector table
> in order to translate writes into vfio interrupt ioctls.  POWER would
> drop the MemoryRegion so that the full page is mapped to the guest,
> with the expectation that the guest never makes use of it since MSI-X
> is always configured via hypercalls on POWER systems.  Likewise I
> expect ARM will still make use of the MemoryRegion emulating the vector
> table, which leaves them exposed to the performance issue POWER is
> trying to solve here since ARM also has 64k page support and has no
> paravirtualized MSI-X programming interface afaik.  x86 is not
> impervious to this issue either, but a 4k page size falls within the
> PCI spec recommendations for MSI-X structure alignment, so it's much
> more rare to have issues. We have certainly seen hardware vendors that
> ignore the PCI spec alignment recommendations, but so far only for
> placing device registers within the same page as the PBA, which is an
> easier problem to deal with since the PBA is relatively unused by
> drivers.  This may be an area where we need to develop a paravirt
> interface for MSI-X programming which disable the MemoryRegion
> emulating the vector table when used.  Thanks,
> 

I get this point. I incorrectly line allowing userspace mmap MSI-X
table to allowing guest direct access to MSI-X.

Thanks
Kevin

[toc] | [prev] | [next] | [standalone]

#1399644

On Thu, 12 May 2016 01:19:44 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Wednesday, May 11, 2016 11:54 PM
> > 
> > On Wed, 11 May 2016 06:29:06 +0000
> > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> >   
> > > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > > > Sent: Thursday, May 05, 2016 11:05 PM
> > > >
> > > > On Thu, 5 May 2016 12:15:46 +0000
> > > > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> > > >  
> > > > > > From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> > > > > > Sent: Thursday, May 05, 2016 7:43 PM
> > > > > >
> > > > > > Hi David and Kevin,
> > > > > >
> > > > > > On 2016/5/5 17:54, David Laight wrote:
> > > > > >  
> > > > > > > From: Tian, Kevin  
> > > > > > >> Sent: 05 May 2016 10:37  
> > > > > > > ...  
> > > > > > >>> Acutually, we are not aimed at accessing MSI-X table from
> > > > > > >>> guest. So I think it's safe to passthrough MSI-X table if we
> > > > > > >>> can make sure guest kernel would not touch MSI-X table in
> > > > > > >>> normal code path such as para-virtualized guest kernel on PPC64.
> > > > > > >>>  
> > > > > > >> Then how do you prevent malicious guest kernel accessing it?  
> > > > > > > Or a malicious guest driver for an ethernet card setting up
> > > > > > > the receive buffer ring to contain a single word entry that
> > > > > > > contains the address associated with an MSI-X interrupt and
> > > > > > > then using a loopback mode to cause a specific packet be
> > > > > > > received that writes the required word through that address.
> > > > > > >
> > > > > > > Remember the PCIe cycle for an interrupt is a normal memory write
> > > > > > > cycle.
> > > > > > >
> > > > > > > 	David
> > > > > > >  
> > > > > >
> > > > > > If we have enough permission to load a malicious driver or
> > > > > > kernel, we can easily break the guest without exposed
> > > > > > MSI-X table.
> > > > > >
> > > > > > I think it should be safe to expose MSI-X table if we can
> > > > > > make sure that malicious guest driver/kernel can't use
> > > > > > the MSI-X table to break other guest or host. The
> > > > > > capability of IRQ remapping could provide this
> > > > > > kind of protection.
> > > > > >  
> > > > >
> > > > > With IRQ remapping it doesn't mean you can pass through MSI-X
> > > > > structure to guest. I know actual IRQ remapping might be platform
> > > > > specific, but at least for Intel VT-d specification, MSI-X entry must
> > > > > be configured with a remappable format by host kernel which
> > > > > contains an index into IRQ remapping table. The index will find a
> > > > > IRQ remapping entry which controls interrupt routing for a specific
> > > > > device. If you allow a malicious program random index into MSI-X
> > > > > entry of assigned device, the hole is obvious...
> > > > >
> > > > > Above might make sense only for a IRQ remapping implementation
> > > > > which doesn't rely on extended MSI-X format (e.g. simply based on
> > > > > BDF). If that's the case for PPC, then you should build MSI-X
> > > > > passthrough based on this fact instead of general IRQ remapping
> > > > > enabled or not.  
> > > >
> > > > I don't think anyone is expecting that we can expose the MSI-X vector
> > > > table to the guest and the guest can make direct use of it.  The end
> > > > goal here is that the guest on a power system is already
> > > > paravirtualized to not program the device MSI-X by directly writing to
> > > > the MSI-X vector table.  They have hypercalls for this since they
> > > > always run virtualized.  Therefore a) they never intend to touch the
> > > > MSI-X vector table and b) they have sufficient isolation that a guest
> > > > can only hurt itself by doing so.
> > > >
> > > > On x86 we don't have a), our method of programming the MSI-X vector
> > > > table is to directly write to it. Therefore we will always require QEMU
> > > > to place a MemoryRegion over the vector table to intercept those
> > > > accesses.  However with interrupt remapping, we do have b) on x86, which
> > > > means that we don't need to be so strict in disallowing user accesses
> > > > to the MSI-X vector table.  It's not useful for configuring MSI-X on
> > > > the device, but the user should only be able to hurt themselves by
> > > > writing it directly.  x86 doesn't really get anything out of this
> > > > change, but it helps this special case on power pretty significantly
> > > > aiui.  Thanks,
> > > >  
> > >
> > > Allowing guest direct write to MSI-x table has system-wide impact.
> > > As I explained earlier, hypervisor needs to control "interrupt_index"
> > > programmed in MSI-X entry, which is used to associate a specific
> > > IRQ remapping entry. Now if you expose whole MSI-x table to guest,
> > > it can program random index into MSI-X entry to associate with
> > > any IRQ remapping entry and then there won't be any isolation per se.
> > >
> > > You can check "5.5.2 MSI and MSI-X Register Programming" in VT-d
> > > spec.  
> > 
> > I think you're extrapolating beyond the vfio interface.  The change
> > here is to remove the vfio protection of the MSI-X vector table when
> > the system provides interrupt isolation.  The argument is that this is
> > safe to do because the hardware protects the host from erroneous and
> > malicious user programming, but it is not meant to provide a means to
> > program MSI-X directly through the vector table.  This is effectively  
> 
> Sorry I didn't get this point. Once we allow userspace to mmap MSI-X
> table, isn't it equivalent to allowing userspace directly program vector
> table? Or is there other mechanism to prevent direct programming?

This would remove the mechanism that prevents direct programming.
Users can configure the device to perform DMA w/o setting up an IOMMU
mapping, which generates a fault.  Users can incorrectly manipulate the
MSI-X vector table instead of using the proper vfio programming
interface, which generates a fault.

> > the same as general DMA programming, if the vfio programming model is
> > not followed the device generates iommu faults.  I do have a concern
> > that userspace driver writers are going to more often presume they can
> > use the vector table directly because of this change, but I don't know
> > that that is sufficient reason to prevent such a change.  They'll
> > quickly discover the device generates faults on interrupt rather than
> > working as expected.  
> 
> If userspace can actually program vector table directly, there is not
> always fault triggered. As long as MSI-X table is fully under control
> of userspace, any interrupt index can be used here which may link
> to a IRQ remapping entry allocated for other devices.

Is not part of the vector lookup comparing the source ID of the DMA
write?  If not then VT-d interrupt remapping offers us no protection
from a malicious guest performing DMA to the same address.  Thanks,

Alex

[toc] | [prev] | [next] | [standalone]

#1399667

> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Thursday, May 12, 2016 10:21 AM
> 
> On Thu, 12 May 2016 01:19:44 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > > Sent: Wednesday, May 11, 2016 11:54 PM
> > >
> > > On Wed, 11 May 2016 06:29:06 +0000
> > > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> > >
> > > > > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > > > > Sent: Thursday, May 05, 2016 11:05 PM
> > > > >
> > > > > On Thu, 5 May 2016 12:15:46 +0000
> > > > > "Tian, Kevin" <kevin.tian@intel.com> wrote:
> > > > >
> > > > > > > From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
> > > > > > > Sent: Thursday, May 05, 2016 7:43 PM
> > > > > > >
> > > > > > > Hi David and Kevin,
> > > > > > >
> > > > > > > On 2016/5/5 17:54, David Laight wrote:
> > > > > > >
> > > > > > > > From: Tian, Kevin
> > > > > > > >> Sent: 05 May 2016 10:37
> > > > > > > > ...
> > > > > > > >>> Acutually, we are not aimed at accessing MSI-X table from
> > > > > > > >>> guest. So I think it's safe to passthrough MSI-X table if we
> > > > > > > >>> can make sure guest kernel would not touch MSI-X table in
> > > > > > > >>> normal code path such as para-virtualized guest kernel on PPC64.
> > > > > > > >>>
> > > > > > > >> Then how do you prevent malicious guest kernel accessing it?
> > > > > > > > Or a malicious guest driver for an ethernet card setting up
> > > > > > > > the receive buffer ring to contain a single word entry that
> > > > > > > > contains the address associated with an MSI-X interrupt and
> > > > > > > > then using a loopback mode to cause a specific packet be
> > > > > > > > received that writes the required word through that address.
> > > > > > > >
> > > > > > > > Remember the PCIe cycle for an interrupt is a normal memory write
> > > > > > > > cycle.
> > > > > > > >
> > > > > > > > 	David
> > > > > > > >
> > > > > > >
> > > > > > > If we have enough permission to load a malicious driver or
> > > > > > > kernel, we can easily break the guest without exposed
> > > > > > > MSI-X table.
> > > > > > >
> > > > > > > I think it should be safe to expose MSI-X table if we can
> > > > > > > make sure that malicious guest driver/kernel can't use
> > > > > > > the MSI-X table to break other guest or host. The
> > > > > > > capability of IRQ remapping could provide this
> > > > > > > kind of protection.
> > > > > > >
> > > > > >
> > > > > > With IRQ remapping it doesn't mean you can pass through MSI-X
> > > > > > structure to guest. I know actual IRQ remapping might be platform
> > > > > > specific, but at least for Intel VT-d specification, MSI-X entry must
> > > > > > be configured with a remappable format by host kernel which
> > > > > > contains an index into IRQ remapping table. The index will find a
> > > > > > IRQ remapping entry which controls interrupt routing for a specific
> > > > > > device. If you allow a malicious program random index into MSI-X
> > > > > > entry of assigned device, the hole is obvious...
> > > > > >
> > > > > > Above might make sense only for a IRQ remapping implementation
> > > > > > which doesn't rely on extended MSI-X format (e.g. simply based on
> > > > > > BDF). If that's the case for PPC, then you should build MSI-X
> > > > > > passthrough based on this fact instead of general IRQ remapping
> > > > > > enabled or not.
> > > > >
> > > > > I don't think anyone is expecting that we can expose the MSI-X vector
> > > > > table to the guest and the guest can make direct use of it.  The end
> > > > > goal here is that the guest on a power system is already
> > > > > paravirtualized to not program the device MSI-X by directly writing to
> > > > > the MSI-X vector table.  They have hypercalls for this since they
> > > > > always run virtualized.  Therefore a) they never intend to touch the
> > > > > MSI-X vector table and b) they have sufficient isolation that a guest
> > > > > can only hurt itself by doing so.
> > > > >
> > > > > On x86 we don't have a), our method of programming the MSI-X vector
> > > > > table is to directly write to it. Therefore we will always require QEMU
> > > > > to place a MemoryRegion over the vector table to intercept those
> > > > > accesses.  However with interrupt remapping, we do have b) on x86, which
> > > > > means that we don't need to be so strict in disallowing user accesses
> > > > > to the MSI-X vector table.  It's not useful for configuring MSI-X on
> > > > > the device, but the user should only be able to hurt themselves by
> > > > > writing it directly.  x86 doesn't really get anything out of this
> > > > > change, but it helps this special case on power pretty significantly
> > > > > aiui.  Thanks,
> > > > >
> > > >
> > > > Allowing guest direct write to MSI-x table has system-wide impact.
> > > > As I explained earlier, hypervisor needs to control "interrupt_index"
> > > > programmed in MSI-X entry, which is used to associate a specific
> > > > IRQ remapping entry. Now if you expose whole MSI-x table to guest,
> > > > it can program random index into MSI-X entry to associate with
> > > > any IRQ remapping entry and then there won't be any isolation per se.
> > > >
> > > > You can check "5.5.2 MSI and MSI-X Register Programming" in VT-d
> > > > spec.
> > >
> > > I think you're extrapolating beyond the vfio interface.  The change
> > > here is to remove the vfio protection of the MSI-X vector table when
> > > the system provides interrupt isolation.  The argument is that this is
> > > safe to do because the hardware protects the host from erroneous and
> > > malicious user programming, but it is not meant to provide a means to
> > > program MSI-X directly through the vector table.  This is effectively
> >
> > Sorry I didn't get this point. Once we allow userspace to mmap MSI-X
> > table, isn't it equivalent to allowing userspace directly program vector
> > table? Or is there other mechanism to prevent direct programming?
> 
> This would remove the mechanism that prevents direct programming.
> Users can configure the device to perform DMA w/o setting up an IOMMU
> mapping, which generates a fault.  Users can incorrectly manipulate the
> MSI-X vector table instead of using the proper vfio programming
> interface, which generates a fault.
> 
> > > the same as general DMA programming, if the vfio programming model is
> > > not followed the device generates iommu faults.  I do have a concern
> > > that userspace driver writers are going to more often presume they can
> > > use the vector table directly because of this change, but I don't know
> > > that that is sufficient reason to prevent such a change.  They'll
> > > quickly discover the device generates faults on interrupt rather than
> > > working as expected.
> >
> > If userspace can actually program vector table directly, there is not
> > always fault triggered. As long as MSI-X table is fully under control
> > of userspace, any interrupt index can be used here which may link
> > to a IRQ remapping entry allocated for other devices.
> 
> Is not part of the vector lookup comparing the source ID of the DMA
> write?  If not then VT-d interrupt remapping offers us no protection
> from a malicious guest performing DMA to the same address.  Thanks,
> 

For x86 IRQ remapping doesn't rely on existing DMAR remapping table for
DMA. Instead IRQ remapping table is a global table per VT-d engine, shared
by all devices behind this VT-d engine. Each IRQ remapping table entry 
(IRTE) can specify:

- whether to validate source-id of the interrupt requests;
- whether to verify the whole source id;
- whether to verify some bits of the source id;
...
(section 9.10 Interrupt Remapping Table Entry (IRTE) in VT-d spec)

IRTE index is programmed into corresponding MSI-X entry of a device.
When an interrupt message is triggered from that device, IRQ remapping
hardware will use the index to find the IRTE entry.

As long as host kernel has strict control of MSI-X table, it can choose to
not validate source-id as long as IRTE index is trusted.

That is why I concerned you cannot do this simply based on whether 
IRQ remapping is available on x86. If such usage is really required,
you'll need some more accurate indicator per iommu structure to tell
whether safe to allow mmap MSI-X to user space.

Thanks
Kevin

[toc] | [prev] | [next] | [standalone]

#1395054

On 2016/5/5 17:36, Tian, Kevin wrote:

>> From: Yongji Xie
>> Sent: Tuesday, May 03, 2016 3:34 PM
>>
>> On 2016/5/3 14:22, Tian, Kevin wrote:
>>
>>>> From: Yongji Xie [mailto:xyjxie@linux.vnet.ibm.com]
>>>> Sent: Tuesday, May 03, 2016 2:08 PM
>>>>
>>>> On 2016/5/3 13:34, Tian, Kevin wrote:
>>>>
>>>>>> From: Yongji Xie
>>>>>> Sent: Wednesday, April 27, 2016 8:43 PM
>>>>>>
>>>>>> This patch enables mmapping MSI-X tables if hardware supports
>>>>>> interrupt remapping which can ensure that a given pci device
>>>>>> can only shoot the MSIs assigned for it.
>>>>>>
>>>>>> With MSI-X table mmapped, we also need to expose the
>>>>>> read/write interface which will be used to access MSI-X table.
>>>>>>
>>>>>> Signed-off-by: Yongji Xie <xyjxie@linux.vnet.ibm.com>
>>>>> A curious question here. Does "allow to mmap MSI-X" essentially
>>>>> mean that KVM guest can directly read/write physical MSI-X
>>>>> structure then?
>>>>>
>>>>> Thanks
>>>>> Kevin
>>>>>
>>>> Here we just allow to mmap MSI-X table in kernel. It doesn't
>>>> mean all KVM guest can directly read/write physical MSI-X
>>>> structure. This should be decided by QEMU. For PPC64
>>>> platform, we would allow to passthrough the MSI-X table
>>>> because we know guest kernel would not write physical
>>>> MSI-X structure when enabling MSI.
>>>>
>>> A bit confused here. If guest kernel doesn't need to write
>>> physical MSI-X structure, what's the point of passing through
>>> the table then?
>> We want to allow the MSI-X table because there may be
>> some critical registers in the same page as the MSI-X table.
>> We have to handle the mmio access to these register in QEMU
>> rather than in guest if mmapping MSI-X table is disallowed.
> So you mean critical registers in same MMIO BAR as MSI-X
> table, instead of two MMIO BARs in same page (the latter I
> suppose with your whole patchset it won't happen then)?

Yes. That's what I mean!

Thanks,
Yongji

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web