Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1587293

Re: [PATCH V3 0/4] Define coherent device memory node

Show all headers | View raw

On 02/23/2017 05:06 PM, Bob Liu wrote:
> On 2017/2/21 21:39, Anshuman Khandual wrote:
>> On 02/21/2017 04:41 PM, Michal Hocko wrote:
>>> On Fri 17-02-17 17:11:57, Anshuman Khandual wrote:
>>> [...]
>>>
>>> Could you also explain why the transparent view is really better than
>>> using a device specific mmap (aka CDM awareness)?
>>
>> Okay with a transparent view, we can achieve a control flow of application
>> like the following.
>>
>> (1) Allocate a buffer:		alloc_buffer(buf, size)
>> (2) CPU compute on buffer:	cpu_compute(buf, size)
>> (3) Device compute on buffer:	device_compute(buf, size)
>> (4) CPU compute on buffer:	cpu_compute(buf, size)
>> (5) Release the buffer:		release_buffer(buf, size)
>>
>> With assistance from a device specific driver, the actual page mapping of
>> the buffer can change between system RAM and device memory depending on
>> which side is accessing at a given point. This will be achieved through
>> driver initiated migrations.
>>
>
> Sorry, I'm a bit confused here.
> What's the difference with the Heterogeneous memory management?
> Which also "allows to use device memory transparently inside any process
> without any modifications to process program code."

OK, Jerome, let me answer this one. :)

Hi Bob,

Yes, from a userspace app's point of view, both HMM and the various NUMA-based proposals appear to 
provide the same thing: transparent, coherent access to both CPU and device memory. It's just the 
implementation that's different, and each implementation has a role:

HMM: for systems that do not provide direct access to device memory, we do need HMM. It provides a 
fault-based mechanism for transparently moving pages to the right place, and mapping them to the 
local process (CPU or device). You can think of HMM as something that provides coherent memory 
access, via software.

NUMA-based solutions: for systems that *can* provide directly addressable, coherent device memory, 
we let programs directly address the memory, and let the (probably enhanced) NUMA system handle page 
placement. There will be lots more NUMA enhancement discussions and patchsets coming, from what I 
can tell.

There are distinct advantages and disadvantages to each approach. For example, fault-based HMM can 
be slow, but it works even with hardware that doesn't directly provide coherent access--and it also 
has page fault information to guide it on page placement (thrashing detection). And NUMA systems, 
which do *not* fault nearly as much, require various artificial ways to detect when a page (or 
process) is on a suboptimal node. The NUMA approach is also, very arguably, conceptually simpler (it 
really depends on which area you look at).

So again: yes, both systems are providing a sort of coherent memory. HMM provides software based 
coherence, while NUMA assumes hardware-based memory coherence as a prerequisite.

I hope that helps, and doesn't just further muddy the waters?

--
John Hubbard
NVIDIA

>
> Thanks,
> -Bob

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-15 13:10 +0100
  [PATCH V3 1/4] mm: Define coherent device memory (CDM) node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-15 13:10 +0100
    Re: [PATCH V3 1/4] mm: Define coherent device memory (CDM) node Bob Liu <lliubbo@gmail.com> - 2017-02-17 15:10 +0100
      Re: [PATCH V3 1/4] mm: Define coherent device memory (CDM) node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-21 11:30 +0100
  [PATCH V3 3/4] mm: Add new parameter to get_page_from_freelist() function Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-15 13:10 +0100
  [PATCH V3 2/4] mm: Enable HugeTLB allocation isolation for CDM nodes Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-15 13:10 +0100
  Re: [PATCH V3 0/4] Define coherent device memory node Mel Gorman <mgorman@suse.de> - 2017-02-15 19:30 +0100
    Re: [PATCH V3 0/4] Define coherent device memory node Balbir Singh <bsingharora@gmail.com> - 2017-02-16 23:20 +0100
      Re: [PATCH V3 0/4] Define coherent device memory node Mel Gorman <mgorman@suse.de> - 2017-02-17 10:40 +0100
        Re: [PATCH V3 0/4] Define coherent device memory node Balbir Singh <bsingharora@gmail.com> - 2017-02-21 04:00 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Balbir Singh <bsingharora@gmail.com> - 2017-03-01 04:00 +0100
    Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-17 12:50 +0100
      Re: [PATCH V3 0/4] Define coherent device memory node Mel Gorman <mgorman@suse.de> - 2017-02-17 14:40 +0100
        Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-21 14:20 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Jerome Glisse <jglisse@redhat.com> - 2017-02-21 21:20 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-23 09:20 +0100
              Re: [PATCH V3 0/4] Define coherent device memory node Jerome Glisse <jglisse@redhat.com> - 2017-02-23 16:30 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Michal Hocko <mhocko@kernel.org> - 2017-02-22 10:30 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Jerome Glisse <jglisse@redhat.com> - 2017-02-22 16:00 +0100
              Re: [PATCH V3 0/4] Define coherent device memory node Michal Hocko <mhocko@kernel.org> - 2017-02-22 18:00 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-23 10:00 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Mel Gorman <mgorman@suse.de> - 2017-02-23 17:00 +0100
      Re: [PATCH V3 0/4] Define coherent device memory node Michal Hocko <mhocko@kernel.org> - 2017-02-21 12:20 +0100
        Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-21 14:50 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Michal Hocko <mhocko@kernel.org> - 2017-02-22 11:00 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-23 08:00 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Bob Liu <liubo95@huawei.com> - 2017-02-24 02:20 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node John Hubbard <jhubbard@nvidia.com> - 2017-02-24 05:40 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Jerome Glisse <jglisse@redhat.com> - 2017-02-24 06:20 +0100
              Re: [PATCH V3 0/4] Define coherent device memory node Bob Liu <liubo95@huawei.com> - 2017-02-27 03:00 +0100
                Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-27 06:50 +0100

csiph-web