Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1508380

Re: [RFC 0/8] Define coherent device memory node

Show all headers | View raw

On Tue, Oct 25, 2016 at 11:07:39PM +1100, Balbir Singh wrote:
> On 25/10/16 04:09, Jerome Glisse wrote:
> > On Mon, Oct 24, 2016 at 10:01:49AM +0530, Anshuman Khandual wrote:
> > 
> >> [...]
> > 
> >> 	Core kernel memory features like reclamation, evictions etc. might
> >> need to be restricted or modified on the coherent device memory node as
> >> they can be performance limiting. The RFC does not propose anything on this
> >> yet but it can be looked into later on. For now it just disables Auto NUMA
> >> for any VMA which has coherent device memory.
> >>
> >> 	Seamless integration of coherent device memory with system memory
> >> will enable various other features, some of which can be listed as follows.
> >>
> >> 	a. Seamless migrations between system RAM and the coherent memory
> >> 	b. Will have asynchronous and high throughput migrations
> >> 	c. Be able to allocate huge order pages from these memory regions
> >> 	d. Restrict allocations to a large extent to the tasks using the
> >> 	   device for workload acceleration
> >>
> >> 	Before concluding, will look into the reasons why the existing
> >> solutions don't work. There are two basic requirements which have to be
> >> satisfies before the coherent device memory can be integrated with core
> >> kernel seamlessly.
> >>
> >> 	a. PFN must have struct page
> >> 	b. Struct page must able to be inside standard LRU lists
> >>
> >> 	The above two basic requirements discard the existing method of
> >> device memory representation approaches like these which then requires the
> >> need of creating a new framework.
> > 
> > I do not believe the LRU list is a hard requirement, yes when faulting in
> > a page inside the page cache it assumes it needs to be added to lru list.
> > But i think this can easily be work around.
> > 
> > In HMM i am using ZONE_DEVICE and because memory is not accessible from CPU
> > (not everyone is bless with decent system bus like CAPI, CCIX, Gen-Z, ...)
> > so in my case a file back page must always be spawn first from a regular
> > page and once read from disk then i can migrate to GPU page.
> > 
> 
> I've not seen the HMM patchset, but read from disk will go to ZONE_DEVICE?
> Then get migrated?

Because in my case device memory is not accessible by anything except the device
(not entirely true but for sake of design it is) any page read from disk will be
first read into regular page (from regular system memory). It is only once it is
uptodate and in page cache that it can be migrated to a ZONE_DEVICE page.

So read from disk use an intermediary page. Write back is kind of the same i plan
on using a bounce page by leveraging existing bio bounce infrastructure.

Cheers,
Jérôme

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[RFC 0/8] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:40 +0200
  [RFC 3/8] mm: Isolate coherent device memory nodes from HugeTLB allocation paths Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:40 +0200
    Re: [RFC 3/8] mm: Isolate coherent device memory nodes from HugeTLB  allocation paths Dave Hansen <dave.hansen@intel.com> - 2016-10-24 19:20 +0200
      Re: [RFC 3/8] mm: Isolate coherent device memory nodes from HugeTLB allocation paths "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-10-25 06:20 +0200
        Re: [RFC 3/8] mm: Isolate coherent device memory nodes from HugeTLB  allocation paths Balbir Singh <bsingharora@gmail.com> - 2016-10-25 09:20 +0200
          Re: [RFC 3/8] mm: Isolate coherent device memory nodes from HugeTLB  allocation paths Balbir Singh <bsingharora@gmail.com> - 2016-10-25 09:30 +0200
  [RFC 7/8] mm: Add a new migration function migrate_virtual_range() Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:40 +0200
  [DEBUG 06/10] mm: Export definition of 'zone_names' array through mmzone.h Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
  [DEBUG 00/10] Test and debug patches for coherent device memory Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
    [DEBUG 02/10] powerpc/mm: Create numa nodes for hotplug memory Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
    [DEBUG 03/10] powerpc/mm: Allow memory hotplug into a memory less node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
    [DEBUG 08/10] powerpc: Enable CONFIG_MOVABLE_NODE for PPC64 platform Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
    [DEBUG 01/10] dt-bindings: Add doc for ibm,hotplug-aperture Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
    [DEBUG 05/10] powerpc/mm: Identify isolation seeking coherent memory nodes during boot Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
    [DEBUG 10/10] test: Add a script to perform random VMA migrations across nodes Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
    [DEBUG 07/10] mm: Add debugfs interface to dump each node's zonelist information Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
    [DEBUG 04/10] mm: Enable CONFIG_MOVABLE_NODE on powerpc Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
    [DEBUG 09/10] drivers: Add two drivers for coherent device memory tests Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:50 +0200
  Re: [RFC 0/8] Define coherent device memory node Jerome Glisse <j.glisse@gmail.com> - 2016-10-24 19:10 +0200
    Re: [RFC 0/8] Define coherent device memory node "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-10-25 06:30 +0200
      Re: [RFC 0/8] Define coherent device memory node Jerome Glisse <j.glisse@gmail.com> - 2016-10-25 17:20 +0200
        Re: [RFC 0/8] Define coherent device memory node "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-10-26 13:10 +0200
          Re: [RFC 0/8] Define coherent device memory node Jerome Glisse <j.glisse@gmail.com> - 2016-10-26 18:10 +0200
    Re: [RFC 0/8] Define coherent device memory node "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-10-25 07:10 +0200
      Re: [RFC 0/8] Define coherent device memory node Jerome Glisse <j.glisse@gmail.com> - 2016-10-25 17:40 +0200
        Re: [RFC 0/8] Define coherent device memory node "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-10-25 19:40 +0200
          Re: [RFC 0/8] Define coherent device memory node Jerome Glisse <j.glisse@gmail.com> - 2016-10-25 21:00 +0200
            Re: [RFC 0/8] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-26 13:20 +0200
              Re: [RFC 0/8] Define coherent device memory node Jerome Glisse <j.glisse@gmail.com> - 2016-10-26 18:10 +0200
            Re: [RFC 0/8] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-26 15:00 +0200
              Re: [RFC 0/8] Define coherent device memory node Jerome Glisse <j.glisse@gmail.com> - 2016-10-26 18:30 +0200
    Re: [RFC 0/8] Define coherent device memory node Balbir Singh <bsingharora@gmail.com> - 2016-10-25 14:10 +0200
      Re: [RFC 0/8] Define coherent device memory node Jerome Glisse <j.glisse@gmail.com> - 2016-10-25 17:30 +0200
  Re: [RFC 0/8] Define coherent device memory node Dave Hansen <dave.hansen@intel.com> - 2016-10-24 20:10 +0200
    Re: [RFC 0/8] Define coherent device memory node David Nellans <dnellans@nvidia.com> - 2016-10-24 20:40 +0200
      Re: [RFC 0/8] Define coherent device memory node Dave Hansen <dave.hansen@intel.com> - 2016-10-24 21:40 +0200

csiph-web