Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1517538
| From | Kirti Wankhede <kwankhede@nvidia.com> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | Re: [PATCH v11 11/22] vfio iommu: Add blocking notifier to notify DMA_UNMAP |
| Date | 2016-11-08 21:10 +0100 |
| Message-ID | <sBkSt-5IO-7@gated-at.bofh.it> (permalink) |
| References | <szTUm-6Y7-25@gated-at.bofh.it> <szU42-71A-53@gated-at.bofh.it> <sB1PQ-1Bj-17@gated-at.bofh.it> <sBhrA-3gP-17@gated-at.bofh.it> <sBiGZ-3WR-21@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
On 11/8/2016 11:16 PM, Alex Williamson wrote:
> On Tue, 8 Nov 2016 21:56:29 +0530
> Kirti Wankhede <kwankhede@nvidia.com> wrote:
>
>> On 11/8/2016 5:15 AM, Alex Williamson wrote:
>>> On Sat, 5 Nov 2016 02:40:45 +0530
>>> Kirti Wankhede <kwankhede@nvidia.com> wrote:
>>>
>> ...
>>>>
>>>> +int vfio_register_notifier(struct device *dev, struct notifier_block *nb)
>>>
>>> Is the expectation here that this is a generic notifier for all
>>> vfio->mdev signaling? That should probably be made clear in the mdev
>>> API to avoid vendor drivers assuming their notifier callback only
>>> occurs for unmaps, even if that's currently the case.
>>>
>>
>> Ok. Adding comment about notifier callback in mdev_device which is part
>> of next patch.
>>
>> ...
>>
>>>> mutex_lock(&iommu->lock);
>>>>
>>>> - if (!iommu->external_domain) {
>>>> + /* Fail if notifier list is empty */
>>>> + if ((!iommu->external_domain) || (!iommu->notifier.head)) {
>>>> ret = -EINVAL;
>>>> goto pin_done;
>>>> }
>>>> @@ -867,6 +870,11 @@ unlock:
>>>> /* Report how much was unmapped */
>>>> unmap->size = unmapped;
>>>>
>>>> + if (unmapped && iommu->external_domain)
>>>> + blocking_notifier_call_chain(&iommu->notifier,
>>>> + VFIO_IOMMU_NOTIFY_DMA_UNMAP,
>>>> + unmap);
>>>
>>> This is after the fact, there's already a gap here where pages are
>>> unpinned and the mdev device is still running.
>>
>> Oh, there is a bug here, now unpin_pages() take user_pfn as argument and
>> find vfio_dma. If its not found, it doesn't unpin pages. We have to call
>> this notifier before vfio_remove_dma(). But if we call this before
>> vfio_remove_dma() there will be deadlock since iommu->lock is already
>> held here and vfio_iommu_type1_unpin_pages() will also try to hold
>> iommu->lock.
>> If we want to call blocking_notifier_call_chain() before
>> vfio_remove_dma(), sequence should be:
>>
>> unmapped += dma->size;
>> mutex_unlock(&iommu->lock);
>> if (iommu->external_domain)) {
>> struct vfio_iommu_type1_dma_unmap nb_unmap;
>>
>> nb_unmap.iova = dma->iova;
>> nb_unmap.size = dma->size;
>> blocking_notifier_call_chain(&iommu->notifier,
>> VFIO_IOMMU_NOTIFY_DMA_UNMAP,
>> &nb_unmap);
>> }
>> mutex_lock(&iommu->lock);
>> vfio_remove_dma(iommu, dma);
>
> It seems like it would be worthwhile to have the rb-tree rooted in the
> vfio-dma, then we only need to call the notifier if there are pages
> pinned within that vfio-dma (ie. the rb-tree is not empty). We can
> then release the lock call the notifier, re-acquire the lock, and
> BUG_ON if the rb-tree still is not empty. We might get duplicate pfns
> between separate vfio_dma structs, but as I mentioned in other replies,
> that seems like an exception that we don't need to optimize for.
>
If we don't optimize for the case where iova from different vfio_dma are
mapped to same pfn and we would not consider this case for page
accounting then:
- have rb tree of pinned iova, where key would be iova, in each vfio_dma
structure.
- iova tracking structure would have iova and ref_count only.
- page accounting would only count number of iova's in rb_tree, case
where different iova could map to same pfn would not be considered in
this implementation for now.
- vfio_unpin_pages() would have user_pfn and pfn as input, we would
validate that iova exist in rb tree and trust vendor driver that
corresponding pfn is correct, there is no validation of pfn. If want
validate pfn, call GUP, verify pfn and call put_pfn().
- In .release() or .detach_group() path, if there are entries in this rb
tree, call GUP again using that iova, get pfn and then call
put_pfn(pfn) for ref_count+1 times. This is because we are not keeping
pfn in our tracking logic.
Does this sound reasonable?
Thanks,
Kirti
>>> The notifier needs to
>>> happen prior to that and I suspect that we need to validate that we
>>> have no remaining external pfn references within this vfio_dma block.
>>> It seems like we need to root our pfn tracking in the vfio_dma so that
>>> we can see that it's empty after the notifier chain and BUG_ON if not.
>>
>> There is no way to find pfns from that iova range with current
>> implementation. We can have this validate if we go with linear array of
>> iova to track pfns.
>
> Right, I was still hoping to avoid storing the pfn even with the
> array/page-table approach though, ask the mm layer for the mapping
> again. Is that too much overhead? Maybe the page table could store
> the phys addr and we could use PAGE_MASK to store the reference count
> so that each entry is still only 8bytes(?)
>
>>> I would also add some enforcement that external pinning is only enabled
>>> when vfio_iommu_type1 is configured for v2 semantics (ie. we only
>>> support unmaps exactly matching previous maps).
>>>
>>
>> Ok I'll add that check.
>>
>> Thanks,
>> Kirti
>
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCH v11 00/22] Add Mediated device support Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:10 +0100
[PATCH v11 12/22] vfio: Add notifier callback to parent's ops structure of mdev Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
Re: [PATCH v11 12/22] vfio: Add notifier callback to parent's ops structure of mdev Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 01:00 +0100
[PATCH v11 06/22] vfio iommu type1: Update arguments of vfio_lock_acct Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 17/22] vfio_platform: Updated to use vfio_set_irqs_validate_and_prepare() Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
Re: [PATCH v11 17/22] vfio_platform: Updated to use vfio_set_irqs_validate_and_prepare() Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-08 11:00 +0100
Re: [PATCH v11 17/22] vfio_platform: Updated to use vfio_set_irqs_validate_and_prepare() Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 21:50 +0100
[PATCH v11 20/22] docs: Sysfs ABI for mediated device framework Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 16/22] vfio_pci: Updated to use vfio_set_irqs_validate_and_prepare() Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 04/22] vfio: Common function to increment container_users Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 19/22] docs: Add Documentation for Mediated devices Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 13/22] vfio: Introduce common function to add capabilities Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
Re: [PATCH v11 13/22] vfio: Introduce common function to add capabilities Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-08 08:30 +0100
Re: [PATCH v11 13/22] vfio: Introduce common function to add capabilities Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 21:50 +0100
Re: [PATCH v11 13/22] vfio: Introduce common function to add capabilities Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 22:50 +0100
Re: [PATCH v11 13/22] vfio: Introduce common function to add capabilities Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-09 03:30 +0100
[PATCH v11 03/22] vfio: Rearrange functions to get vfio_group from dev Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 14/22] vfio_pci: Update vfio_pci to use vfio_info_add_capability() Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 09/22] vfio iommu type1: Add task structure to vfio_dma Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
Re: [PATCH v11 09/22] vfio iommu type1: Add task structure to vfio_dma Alex Williamson <alex.williamson@redhat.com> - 2016-11-07 22:10 +0100
Re: [PATCH v11 09/22] vfio iommu type1: Add task structure to vfio_dma Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 15:20 +0100
Re: [PATCH v11 09/22] vfio iommu type1: Add task structure to vfio_dma Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 17:50 +0100
[PATCH v11 02/22] vfio: VFIO based driver for Mediated devices Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 18/22] vfio: Define device_api strings Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 07/22] vfio iommu type1: Update argument of vaddr_get_pfn() Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
Re: [PATCH v11 07/22] vfio iommu type1: Update argument of vaddr_get_pfn() Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-07 09:50 +0100
[PATCH v11 15/22] vfio: Introduce vfio_set_irqs_validate_and_prepare() Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
Re: [PATCH v11 15/22] vfio: Introduce vfio_set_irqs_validate_and_prepare() Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-08 09:50 +0100
Re: [PATCH v11 15/22] vfio: Introduce vfio_set_irqs_validate_and_prepare() Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 21:30 +0100
Re: [PATCH v11 15/22] vfio: Introduce vfio_set_irqs_validate_and_prepare() Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-09 04:10 +0100
Re: [PATCH v11 15/22] vfio: Introduce vfio_set_irqs_validate_and_prepare() Alex Williamson <alex.williamson@redhat.com> - 2016-11-09 04:40 +0100
[PATCH v11 10/22] vfio iommu type1: Add support for mediated devices Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 00:20 +0100
Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices Jike Song <jike.song@intel.com> - 2016-11-08 03:30 +0100
Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 17:20 +0100
Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 16:10 +0100
Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 18:10 +0100
Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-08 08:00 +0100
[PATCH v11 22/22] MAINTAINERS: Add entry VFIO based Mediated device drivers Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 08/22] vfio iommu type1: Add find_iommu_group() function Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
[PATCH v11 11/22] vfio iommu: Add blocking notifier to notify DMA_UNMAP Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
Re: [PATCH v11 11/22] vfio iommu: Add blocking notifier to notify DMA_UNMAP Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 00:50 +0100
Re: [PATCH v11 11/22] vfio iommu: Add blocking notifier to notify DMA_UNMAP Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 17:30 +0100
Re: [PATCH v11 11/22] vfio iommu: Add blocking notifier to notify DMA_UNMAP Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 18:50 +0100
Re: [PATCH v11 11/22] vfio iommu: Add blocking notifier to notify DMA_UNMAP Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 21:10 +0100
Re: [PATCH v11 11/22] vfio iommu: Add blocking notifier to notify DMA_UNMAP Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 22:30 +0100
[PATCH v11 01/22] vfio: Mediated device Core driver Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
RE: [PATCH v11 01/22] vfio: Mediated device Core driver "Tian, Kevin" <kevin.tian@intel.com> - 2016-11-07 07:50 +0100
Re: [PATCH v11 01/22] vfio: Mediated device Core driver Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 22:20 +0100
[PATCH v11 05/22] vfio iommu: Added pin and unpin callback functions to vfio_iommu_driver_ops Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-04 22:20 +0100
Re: [PATCH v11 05/22] vfio iommu: Added pin and unpin callback functions to vfio_iommu_driver_ops Alex Williamson <alex.williamson@redhat.com> - 2016-11-07 20:40 +0100
Re: [PATCH v11 05/22] vfio iommu: Added pin and unpin callback functions to vfio_iommu_driver_ops Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 15:00 +0100
Re: [PATCH v11 05/22] vfio iommu: Added pin and unpin callback functions to vfio_iommu_driver_ops Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 17:50 +0100
Re: [PATCH v11 05/22] vfio iommu: Added pin and unpin callback functions to vfio_iommu_driver_ops Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-08 20:00 +0100
Re: [PATCH v11 05/22] vfio iommu: Added pin and unpin callback functions to vfio_iommu_driver_ops Alex Williamson <alex.williamson@redhat.com> - 2016-11-08 20:20 +0100
Re: [PATCH v11 00/22] Add Mediated device support Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-07 04:50 +0100
Re: [PATCH v11 00/22] Add Mediated device support Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-07 05:00 +0100
Re: [PATCH v11 00/22] Add Mediated device support Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-07 06:10 +0100
Re: [PATCH v11 00/22] Add Mediated device support Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-07 07:20 +0100
Re: [PATCH v11 00/22] Add Mediated device support Kirti Wankhede <kwankhede@nvidia.com> - 2016-11-07 07:40 +0100
Re: [PATCH v11 00/22] Add Mediated device support Alexey Kardashevskiy <aik@ozlabs.ru> - 2016-11-07 07:50 +0100
csiph-web