Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1529546
| From | Jason Gunthorpe <jgunthorpe@obsidianresearch.com> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | Re: Enabling peer to peer device transactions for PCIe devices |
| Date | 2016-11-24 17:30 +0100 |
| Message-ID | <sH54m-5b9-39@gated-at.bofh.it> (permalink) |
| References | (5 earlier) <sGMuJ-1d5-9@gated-at.bofh.it> <sGMuJ-1d5-7@gated-at.bofh.it> <sGN7r-1Fi-5@gated-at.bofh.it> <sGOmR-2pg-21@gated-at.bofh.it> <sGQoF-3CM-9@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
On Thu, Nov 24, 2016 at 12:40:37AM +0000, Sagalovitch, Serguei wrote:
> On Wed, Nov 23, 2016 at 02:11:29PM -0700, Logan Gunthorpe wrote:
>
> > Perhaps I am not following what Serguei is asking for, but I
> > understood the desire was for a complex GPU allocator that could
> > migrate pages between GPU and CPU memory under control of the GPU
> > driver, among other things. The desire is for DMA to continue to work
> > even after these migrations happen.
>
> The main issue is to how to solve use cases when p2p is
> requested/initiated via CPU pointers where such pointers could
> point to non-system memory location e.g. VRAM.
Okay, but your list is conflating a whole bunch of problems..
1) How to go from a __user pointer to a p2p DMA address
a) How to validate, setup iommu and maybe worst case bounce buffer
these p2p DMAs
2) How to allow drivers (ie GPU allocator) dynamically
remap pages in a VMA to/from p2p DMA addresses
3) How to expose uncachable p2p DMA address to user space via mmap
> to allow "get_user_pages" to work transparently similar
> how it is/was done for "DAX Device" case. Unfortunately
> based on my understanding "DAX Device" implementation
> deal only with permanently "locked" memory (fixed location)
> unrelated to "get_user_pages"/"put_page" scope
> which doesn't satisfy requirements for "eviction" / "moving" of
> memory keeping CPU address intact.
Hurm, isn't that issue with DAX only to do with being coherent with
the page cache?
A GPU allocator would not use the page cache, it would have to
construct VMAs some other way.
> My understanding is that It will not solve RDMA MR issue where "lock"
> could be during the whole application life but (a) it will not make
> RDMA MR case worse (b) should be enough for all other cases for
> "get_user_pages"/"put_page" controlled by kernel.
Right. There is no solution to the RDMA MR issue on old hardware. Apps
that are using GPU+RDMA+Old hardware will have to use short lived MRs
and pay that performance cost, or give up on migration.
Jason
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
Enabling peer to peer device transactions for PCIe devices "Deucher, Alexander" <Alexander.Deucher@amd.com> - 2016-11-21 21:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-11-22 19:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-11-22 21:10 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-11-22 21:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Daniel Vetter <daniel@ffwll.ch> - 2016-11-22 21:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Daniel Vetter <daniel@ffwll.ch> - 2016-11-22 22:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-11-22 22:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices "Sagalovitch, Serguei" <Serguei.Sagalovitch@amd.com> - 2016-11-22 23:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Daniel Vetter <daniel@ffwll.ch> - 2016-11-23 08:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dave Hansen <dave.hansen@linux.intel.com> - 2016-11-23 18:10 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-11-23 18:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-11-23 19:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-23 20:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-23 20:10 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-23 20:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-23 21:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-11-23 22:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-23 23:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-11-23 23:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-24 00:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-24 17:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-11-24 19:00 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-11-25 17:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices "Sagalovitch, Serguei" <Serguei.Sagalovitch@amd.com> - 2016-11-24 01:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-24 17:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-11-24 02:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-24 17:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-11-24 19:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Christoph Hellwig <hch@infradead.org> - 2016-11-25 09:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-25 20:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-25 20:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-25 21:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Alex Deucher <alexdeucher@gmail.com> - 2016-11-26 00:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-25 20:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Christian König <deathsimple@vodafone.de> - 2016-11-25 21:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-25 22:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-28 18:00 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-11-28 19:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-11-28 22:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-28 23:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-28 20:10 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-11-30 17:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-11-30 19:10 +0100
Re: Enabling peer to peer device transactions for PCIe devices "Stephen Bates" <sbates@raithlin.com> - 2016-12-04 14:10 +0100
Re: Enabling peer to peer device transactions for PCIe devices "Stephen Bates" <sbates@raithlin.com> - 2016-12-04 14:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-12-05 18:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-12-05 18:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-12-05 19:10 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-12-05 19:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-12-05 19:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-12-05 20:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-12-05 20:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-12-05 20:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-12-05 21:00 +0100
Re: Enabling peer to peer device transactions for PCIe devices Christoph Hellwig <hch@infradead.org> - 2016-12-05 21:10 +0100
Re: Enabling peer to peer device transactions for PCIe devices "Stephen Bates" <sbates@raithlin.com> - 2016-12-06 09:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-12-06 17:40 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-12-06 18:00 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-12-06 18:30 +0100
Re: Enabling peer to peer device transactions for PCIe devices Logan Gunthorpe <logang@deltatee.com> - 2016-12-06 22:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Dan Williams <dan.j.williams@intel.com> - 2016-12-06 23:10 +0100
Re: Enabling peer to peer device transactions for PCIe devices Christoph Hellwig <hch@infradead.org> - 2016-12-06 18:20 +0100
Re: Enabling peer to peer device transactions for PCIe devices Jason Gunthorpe <jgunthorpe@obsidianresearch.com> - 2016-12-05 19:10 +0100
RE: Enabling peer to peer device transactions for PCIe devices "Deucher, Alexander" <Alexander.Deucher@amd.com> - 2016-11-30 18:50 +0100
Re: Enabling peer to peer device transactions for PCIe devices Haggai Eran <haggaie@mellanox.com> - 2016-11-28 21:00 +0100
Re: Enabling peer to peer device transactions for PCIe devices Haggai Eran <haggaie@mellanox.com> - 2016-11-28 19:40 +0100
csiph-web