Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1586029

Re: [PATCH V3 0/4] Define coherent device memory node

Show all headers | View raw

On Tue 21-02-17 19:09:18, Anshuman Khandual wrote:
> On 02/21/2017 04:41 PM, Michal Hocko wrote:
> > On Fri 17-02-17 17:11:57, Anshuman Khandual wrote:
> > [...]
> >> * User space using mbind() to get CDM memory is an additional benefit
> >>   we get by making the CDM plug in as a node and be part of the buddy
> >>   allocator. But the over all idea from the user space point of view
> >>   is that the application can allocate any generic buffer and try to
> >>   use the buffer either from the CPU side or from the device without
> >>   knowing about where the buffer is really mapped physically. That
> >>   gives a seamless and transparent view to the user space where CPU
> >>   compute and possible device based compute can work together. This
> >>   is not possible through a driver allocated buffer.
> > 
> > But how are you going to define any policy around that. Who is allowed
> 
> The user space VMA can define the policy with a mbind(MPOL_BIND) call
> with CDM/CDMs in the nodemask.
>
> > to allocate and how much of this "special memory". Is it possible that
> 
> Any user space application with mbind(MPOL_BIND) call with CDM/CDMs in
> the nodemask can allocate from the CDM memory. "How much" gets controlled
> by how we fault from CPU and the default behavior of the buddy allocator.

In other words the policy is implemented by the kernel. Why is this a
good thing?

> > we will eventually need some access control mechanism? If yes then mbind
> 
> No access control mechanism is needed. If an application wants to use
> CDM memory by specifying in the mbind() it can. Nothing prevents it
> from using the CDM memory.

What if we find out that an access control _is_ really needed? I can
easily imagine that some devices will come up with really fast and expensive
memory. You do not want some random user to steal it from you when you
want to use it for your workload.

> > is really not suitable interface to (ab)use. Also what should happen if
> > the mbind mentions only CDM memory and that is depleted?
> 
> IIUC *only CDM* cannot be requested from user space as there are no user
> visible interface which can translate to __GFP_THISNODE.

I do not understand what __GFP_THISNODE has to do with this. This is an
internal flag.

> MPOL_BIND with
> CDM in the nodemask will eventually pick a FALLBACK zonelist which will
> have zones of the system including CDM ones. If the resultant CDM zones
> run out of memory, we fail the allocation request as usual.

OK, so let's say you mbind to a single node which is CDM. You seem to be
saying that we will simply break the NUMA affinity in this special case?
Currently we invoke the OOM killer if nodes which the application binds
to are depleted and cannot be reclaimed.
 
> > Could you also explain why the transparent view is really better than
> > using a device specific mmap (aka CDM awareness)?
> 
> Okay with a transparent view, we can achieve a control flow of application
> like the following.
> 
> (1) Allocate a buffer:		alloc_buffer(buf, size)
> (2) CPU compute on buffer:	cpu_compute(buf, size)
> (3) Device compute on buffer:	device_compute(buf, size)
> (4) CPU compute on buffer:	cpu_compute(buf, size)
> (5) Release the buffer:		release_buffer(buf, size)
> 
> With assistance from a device specific driver, the actual page mapping of
> the buffer can change between system RAM and device memory depending on
> which side is accessing at a given point. This will be achieved through
> driver initiated migrations.

But then you do not need any NUMA affinity, right? The driver can do
all this automagically. How does the numa policy comes into the game in
your above example. Sorry for being dense, I might be really missing
something important here, but I really fail to see why the NUMA is the
proper interface here.
-- 
Michal Hocko
SUSE Labs

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-15 13:10 +0100
  [PATCH V3 1/4] mm: Define coherent device memory (CDM) node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-15 13:10 +0100
    Re: [PATCH V3 1/4] mm: Define coherent device memory (CDM) node Bob Liu <lliubbo@gmail.com> - 2017-02-17 15:10 +0100
      Re: [PATCH V3 1/4] mm: Define coherent device memory (CDM) node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-21 11:30 +0100
  [PATCH V3 3/4] mm: Add new parameter to get_page_from_freelist() function Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-15 13:10 +0100
  [PATCH V3 2/4] mm: Enable HugeTLB allocation isolation for CDM nodes Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-15 13:10 +0100
  Re: [PATCH V3 0/4] Define coherent device memory node Mel Gorman <mgorman@suse.de> - 2017-02-15 19:30 +0100
    Re: [PATCH V3 0/4] Define coherent device memory node Balbir Singh <bsingharora@gmail.com> - 2017-02-16 23:20 +0100
      Re: [PATCH V3 0/4] Define coherent device memory node Mel Gorman <mgorman@suse.de> - 2017-02-17 10:40 +0100
        Re: [PATCH V3 0/4] Define coherent device memory node Balbir Singh <bsingharora@gmail.com> - 2017-02-21 04:00 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Balbir Singh <bsingharora@gmail.com> - 2017-03-01 04:00 +0100
    Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-17 12:50 +0100
      Re: [PATCH V3 0/4] Define coherent device memory node Mel Gorman <mgorman@suse.de> - 2017-02-17 14:40 +0100
        Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-21 14:20 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Jerome Glisse <jglisse@redhat.com> - 2017-02-21 21:20 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-23 09:20 +0100
              Re: [PATCH V3 0/4] Define coherent device memory node Jerome Glisse <jglisse@redhat.com> - 2017-02-23 16:30 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Michal Hocko <mhocko@kernel.org> - 2017-02-22 10:30 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Jerome Glisse <jglisse@redhat.com> - 2017-02-22 16:00 +0100
              Re: [PATCH V3 0/4] Define coherent device memory node Michal Hocko <mhocko@kernel.org> - 2017-02-22 18:00 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-23 10:00 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Mel Gorman <mgorman@suse.de> - 2017-02-23 17:00 +0100
      Re: [PATCH V3 0/4] Define coherent device memory node Michal Hocko <mhocko@kernel.org> - 2017-02-21 12:20 +0100
        Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-21 14:50 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Michal Hocko <mhocko@kernel.org> - 2017-02-22 11:00 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-23 08:00 +0100
          Re: [PATCH V3 0/4] Define coherent device memory node Bob Liu <liubo95@huawei.com> - 2017-02-24 02:20 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node John Hubbard <jhubbard@nvidia.com> - 2017-02-24 05:40 +0100
            Re: [PATCH V3 0/4] Define coherent device memory node Jerome Glisse <jglisse@redhat.com> - 2017-02-24 06:20 +0100
              Re: [PATCH V3 0/4] Define coherent device memory node Bob Liu <liubo95@huawei.com> - 2017-02-27 03:00 +0100
                Re: [PATCH V3 0/4] Define coherent device memory node Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-27 06:50 +0100

csiph-web