Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1570313 > unrolled thread

Re: [RFC V2 12/12] mm: Tag VMA with VM_CDM flag explicitly during mbind(MPOL_BIND)

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC V2 12/12] mm: Tag VMA with VM_CDM flag explicitly during  mbind(MPOL_BIND) Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-01-31 05:40 +0100
    Re: [RFC V2 12/12] mm: Tag VMA with VM_CDM flag explicitly during  mbind(MPOL_BIND) Dave Hansen <dave.hansen@intel.com> - 2017-02-07 19:10 +0100
      Re: [RFC V2 12/12] mm: Tag VMA with VM_CDM flag explicitly during  mbind(MPOL_BIND) Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-02-08 17:50 +0100
      Re: [RFC V2 12/12] mm: Tag VMA with VM_CDM flag explicitly during  mbind(MPOL_BIND) Jerome Glisse <jglisse@redhat.com> - 2017-02-08 19:40 +0100

#1570313 — Re: [RFC V2 12/12] mm: Tag VMA with VM_CDM flag explicitly during mbind(MPOL_BIND)

On 01/30/2017 11:24 PM, Dave Hansen wrote:
> On 01/29/2017 07:35 PM, Anshuman Khandual wrote:
>> +		if ((new_pol->mode == MPOL_BIND)
>> +			&& nodemask_has_cdm(new_pol->v.nodes))
>> +			set_vm_cdm(vma);
> So, if you did:
> 
> 	mbind(addr, PAGE_SIZE, MPOL_BIND, all_nodes, ...);
> 	mbind(addr, PAGE_SIZE, MPOL_BIND, one_non_cdm_node, ...);
> 
> You end up with a VMA that can never have KSM done on it, etc...  Even
> though there's no good reason for it.  I guess /proc/$pid/smaps might be
> able to help us figure out what was going on here, but that still seems
> like an awful lot of damage.

Agreed, this VMA should not remain tagged after the second call. It does
not make sense. For this kind of scenarios we can re-evaluate the VMA
tag every time the nodemask change is attempted. But if we are looking for
some runtime re-evaluation then we need to steal some cycles are during
general VMA processing opportunity points like merging and split to do
the necessary re-evaluation. Should do we do these kind two kinds of
re-evaluation to be more optimal ?

[toc] | [next] | [standalone]

#1575932

On 01/30/2017 08:36 PM, Anshuman Khandual wrote:
> On 01/30/2017 11:24 PM, Dave Hansen wrote:
>> On 01/29/2017 07:35 PM, Anshuman Khandual wrote:
>>> +		if ((new_pol->mode == MPOL_BIND)
>>> +			&& nodemask_has_cdm(new_pol->v.nodes))
>>> +			set_vm_cdm(vma);
>> So, if you did:
>>
>> 	mbind(addr, PAGE_SIZE, MPOL_BIND, all_nodes, ...);
>> 	mbind(addr, PAGE_SIZE, MPOL_BIND, one_non_cdm_node, ...);
>>
>> You end up with a VMA that can never have KSM done on it, etc...  Even
>> though there's no good reason for it.  I guess /proc/$pid/smaps might be
>> able to help us figure out what was going on here, but that still seems
>> like an awful lot of damage.
> 
> Agreed, this VMA should not remain tagged after the second call. It does
> not make sense. For this kind of scenarios we can re-evaluate the VMA
> tag every time the nodemask change is attempted. But if we are looking for
> some runtime re-evaluation then we need to steal some cycles are during
> general VMA processing opportunity points like merging and split to do
> the necessary re-evaluation. Should do we do these kind two kinds of
> re-evaluation to be more optimal ?

I'm still unconvinced that you *need* detection like this.  Scanning big
VMAs is going to be really painful.

I thought I asked before but I can't find it in this thread.  But, we
have explicit interfaces for disabling KSM and khugepaged.  Why do we
need implicit ones like this in addition to those?

[toc] | [prev] | [next] | [standalone]

#1576702

On 02/07/2017 11:37 PM, Dave Hansen wrote:
>> On 01/30/2017 11:24 PM, Dave Hansen wrote:
>>> On 01/29/2017 07:35 PM, Anshuman Khandual wrote:
>>>> +		if ((new_pol->mode == MPOL_BIND)
>>>> +			&& nodemask_has_cdm(new_pol->v.nodes))
>>>> +			set_vm_cdm(vma);
>>> So, if you did:
>>>
>>> 	mbind(addr, PAGE_SIZE, MPOL_BIND, all_nodes, ...);
>>> 	mbind(addr, PAGE_SIZE, MPOL_BIND, one_non_cdm_node, ...);
>>>
>>> You end up with a VMA that can never have KSM done on it, etc...  Even
>>> though there's no good reason for it.  I guess /proc/$pid/smaps might be
>>> able to help us figure out what was going on here, but that still seems
>>> like an awful lot of damage.
>> Agreed, this VMA should not remain tagged after the second call. It does
>> not make sense. For this kind of scenarios we can re-evaluate the VMA
>> tag every time the nodemask change is attempted. But if we are looking for
>> some runtime re-evaluation then we need to steal some cycles are during
>> general VMA processing opportunity points like merging and split to do
>> the necessary re-evaluation. Should do we do these kind two kinds of
>> re-evaluation to be more optimal ?
> I'm still unconvinced that you *need* detection like this.  Scanning big
> VMAs is going to be really painful.
> 
> I thought I asked before but I can't find it in this thread.  But, we
> have explicit interfaces for disabling KSM and khugepaged.  Why do we
> need implicit ones like this in addition to those?

Missed the discussion we had on this last time around I think. My bad, sorry
about that. IIUC we can disable KSM through madvise() call, in fact I guess
its disabled by default and need to be enabled. We can just have a similar
interface to disable auto NUMA for a specific VMA or we can handle it page
by page basis with something like this.

diff --git a/mm/memory.c b/mm/memory.c
index 1099d35..101dfd9 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3518,6 +3518,9 @@ static int do_numa_page(struct vm_fault *vmf)
                goto out;
        }
 
+       if (is_cdm_node(page_to_nid(page)))
+               goto out;
+
        /* Migrate to the requested node */
        migrated = migrate_misplaced_page(page, vma, target_nid);
        if (migrated) {

I am still looking into these aspects. BTW have posted the minimum set of
CDM patches which defines and isolates CDM node.

[toc] | [prev] | [next] | [standalone]

#1576811

> On 01/30/2017 08:36 PM, Anshuman Khandual wrote:
> > On 01/30/2017 11:24 PM, Dave Hansen wrote:
> >> On 01/29/2017 07:35 PM, Anshuman Khandual wrote:
> >>> +		if ((new_pol->mode == MPOL_BIND)
> >>> +			&& nodemask_has_cdm(new_pol->v.nodes))
> >>> +			set_vm_cdm(vma);
> >> So, if you did:
> >>
> >> 	mbind(addr, PAGE_SIZE, MPOL_BIND, all_nodes, ...);
> >> 	mbind(addr, PAGE_SIZE, MPOL_BIND, one_non_cdm_node, ...);
> >>
> >> You end up with a VMA that can never have KSM done on it, etc...  Even
> >> though there's no good reason for it.  I guess /proc/$pid/smaps might be
> >> able to help us figure out what was going on here, but that still seems
> >> like an awful lot of damage.
> > 
> > Agreed, this VMA should not remain tagged after the second call. It does
> > not make sense. For this kind of scenarios we can re-evaluate the VMA
> > tag every time the nodemask change is attempted. But if we are looking for
> > some runtime re-evaluation then we need to steal some cycles are during
> > general VMA processing opportunity points like merging and split to do
> > the necessary re-evaluation. Should do we do these kind two kinds of
> > re-evaluation to be more optimal ?
> 
> I'm still unconvinced that you *need* detection like this.  Scanning big
> VMAs is going to be really painful.
> 
> I thought I asked before but I can't find it in this thread.  But, we
> have explicit interfaces for disabling KSM and khugepaged.  Why do we
> need implicit ones like this in addition to those?
> 

I said it in other part of the thread i think the vma flag is a no go. Because
it try to set something that is orthogonal to vma. That you want some vma to
use device memory on new allocation is a valid policy for a vma to have. But to
have a flag that say various kernel subsystem hey my memory is special skip me
is wrong.

The fact that you want to exclude device memory from KSM or autonuma is valid but
it should be done at struct page level ie KSM or autonuma should check the type
of page before doing anything. For CDM pages they would skip. It could be the flags
idea that was discussed.

The overhead of doing it at page level is far lower than trying to manage a vma
flags with all the issue related to vma merging, splitting and lifetime of such
flags. Moreover this flags is an all or nothing, it does not consider the case
where you have as much regular page as CDM page in a vma. It would block regular
page from under going the usual KSM/autonuma ...

I do strongly believe that this vma flag is a bad idea.

Cheers,
Jérôme

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web