Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1569358 > unrolled thread

[RFC V2 08/12] mm: Add new VMA flag VM_CDM

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [RFC V2 08/12] mm: Add new VMA flag VM_CDM Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-01-30 04:40 +0100
    Re: [RFC V2 08/12] mm: Add new VMA flag VM_CDM Jerome Glisse <jglisse@redhat.com> - 2017-01-30 20:00 +0100
      Re: [RFC V2 08/12] mm: Add new VMA flag VM_CDM Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2017-01-31 05:30 +0100
        Re: [RFC V2 08/12] mm: Add new VMA flag VM_CDM Jerome Glisse <jglisse@redhat.com> - 2017-01-31 07:10 +0100

#1569358 — [RFC V2 08/12] mm: Add new VMA flag VM_CDM

VMA which contains CDM memory pages should be marked with new VM_CDM flag.
These VMAs need to be identified in various core kernel paths for special
handling and this flag will help in their identification.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 include/linux/mm.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index b84615b..82482d3 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -182,6 +182,11 @@ extern unsigned int kobjsize(const void *objp);
 #define VM_ACCOUNT	0x00100000	/* Is a VM accounted object */
 #define VM_NORESERVE	0x00200000	/* should the VM suppress accounting */
 #define VM_HUGETLB	0x00400000	/* Huge TLB Page VM */
+
+#ifdef CONFIG_COHERENT_DEVICE
+#define VM_CDM		0x00800000	/* Contains coherent device memory */
+#endif
+
 #define VM_ARCH_1	0x01000000	/* Architecture-specific flag */
 #define VM_ARCH_2	0x02000000
 #define VM_DONTDUMP	0x04000000	/* Do not include in the core dump */
-- 
2.9.3

[toc] | [next] | [standalone]

#1570022

On Mon, Jan 30, 2017 at 09:05:49AM +0530, Anshuman Khandual wrote:
> VMA which contains CDM memory pages should be marked with new VM_CDM flag.
> These VMAs need to be identified in various core kernel paths for special
> handling and this flag will help in their identification.
> 
> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>


Why doing this on vma basis ? Why not special casing all those path on page
basis ?

After all you can have a big vma with some pages in it being cdm and other
being regular page. The CPU process might migrate to different CPU in a
different node and you might still want to have the regular page to migrate
to this new node and keep the cdm page while the device is still working
on them.

This is just an example, same can apply for ksm or any other kernel feature
you want to special case. Maybe we can store a set of flag in node that
tells what is allowed for page in node (ksm, hugetlb, migrate, numa, ...).

This would be more flexible and the policy choice can be left to each of
the device driver.

Cheers,
Jérôme

[toc] | [prev] | [next] | [standalone]

#1570311

On 01/31/2017 12:22 AM, Jerome Glisse wrote:
> On Mon, Jan 30, 2017 at 09:05:49AM +0530, Anshuman Khandual wrote:
>> VMA which contains CDM memory pages should be marked with new VM_CDM flag.
>> These VMAs need to be identified in various core kernel paths for special
>> handling and this flag will help in their identification.
>>
>> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> 
> 
> Why doing this on vma basis ? Why not special casing all those path on page
> basis ?

The primary motivation being the cost. Wont it be too expensive to account
for and act on individual pages rather than on the VMA as a whole ? For
example page_to_nid() seemed pretty expensive when tried to tag VMA on
individual page fault basis.

> 
> After all you can have a big vma with some pages in it being cdm and other
> being regular page. The CPU process might migrate to different CPU in a
> different node and you might still want to have the regular page to migrate
> to this new node and keep the cdm page while the device is still working
> on them.

Right, that is the ideal thing to do. But wont it be better to split the
big VMA into smaller chunks and tag them appropriately so that those VMAs
tagged would contain as much CDM pages as possible for them to be likely
restricted from auto NUMA, KSM etc.

> 
> This is just an example, same can apply for ksm or any other kernel feature
> you want to special case. Maybe we can store a set of flag in node that
> tells what is allowed for page in node (ksm, hugetlb, migrate, numa, ...).
> 
> This would be more flexible and the policy choice can be left to each of
> the device driver.

Hmm, thats another way of doing the special cases. The other way as Dave
had mentioned before is to classify coherent memory property into various
kinds and store them for each node and implement a predefined set of
restrictions for each kind of coherent memory which might include features
like auto NUMA, HugeTLB, KSM etc. Maintaining two different property sets
one for the kind of coherent memory and the other being for each special
cases) wont be too complicated ?

[toc] | [prev] | [next] | [standalone]

#1570424

On Tue, Jan 31, 2017 at 09:52:20AM +0530, Anshuman Khandual wrote:
> On 01/31/2017 12:22 AM, Jerome Glisse wrote:
> > On Mon, Jan 30, 2017 at 09:05:49AM +0530, Anshuman Khandual wrote:
> >> VMA which contains CDM memory pages should be marked with new VM_CDM flag.
> >> These VMAs need to be identified in various core kernel paths for special
> >> handling and this flag will help in their identification.
> >>
> >> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> > 
> > 
> > Why doing this on vma basis ? Why not special casing all those path on page
> > basis ?
> 
> The primary motivation being the cost. Wont it be too expensive to account
> for and act on individual pages rather than on the VMA as a whole ? For
> example page_to_nid() seemed pretty expensive when tried to tag VMA on
> individual page fault basis.

No i don't think it would be too expensive. What is confusing in this patchset
is that you are conflating 3 different problems. First one is how to create
struct page for coherent device memory and exclude those pages from regular
allocations.

Second one is how to allow userspace to set allocation policy that would direct
allocation for a given vma to use a specific device memory.

Finaly last one is how to block some kernel feature such as numa or ksm as you
expect (and i share that believe) that they will be hurtfull.

I do believe, that this last requirement, is better left to be done on a per page
basis as page_to_nid() is only a memory lookup and i would be stun if that memory
lookup register as more than a blip on any profiler radar.

The vma flag as all or nothing choice is bad in my view and its stickyness and how
to handle its lifetime and inheritance is troubling and hard. Checking through node
if a page should undergo ksm or numa is a better solution in my view.

> 
> > 
> > After all you can have a big vma with some pages in it being cdm and other
> > being regular page. The CPU process might migrate to different CPU in a
> > different node and you might still want to have the regular page to migrate
> > to this new node and keep the cdm page while the device is still working
> > on them.
> 
> Right, that is the ideal thing to do. But wont it be better to split the
> big VMA into smaller chunks and tag them appropriately so that those VMAs
> tagged would contain as much CDM pages as possible for them to be likely
> restricted from auto NUMA, KSM etc.

Think a vma in which every odd 4k address point to a device page is device and
even 4k address point to a regular page, would you want to create as many vma
for this ?

Setting policy for allocation make sense, but setting flag that enable/disable
kernel feature for a range, overridding other policy is bad in my view.

> 
> > 
> > This is just an example, same can apply for ksm or any other kernel feature
> > you want to special case. Maybe we can store a set of flag in node that
> > tells what is allowed for page in node (ksm, hugetlb, migrate, numa, ...).
> > 
> > This would be more flexible and the policy choice can be left to each of
> > the device driver.
> 
> Hmm, thats another way of doing the special cases. The other way as Dave
> had mentioned before is to classify coherent memory property into various
> kinds and store them for each node and implement a predefined set of
> restrictions for each kind of coherent memory which might include features
> like auto NUMA, HugeTLB, KSM etc. Maintaining two different property sets
> one for the kind of coherent memory and the other being for each special
> cases) wont be too complicated ?

I am not sure i follow, you have a single mask provided by the driver that
register the memory something like:

CDM_ALLOW_NUMA (1 << 0)
CDM_ALLOW_KSM  (1 << 1)
...

Then you have bool page_node_allow_numa(page), bool page_node_allow_ksm(page),
... that is it. Both numa and ksm perform heavy operations and having to go
check a mask inside node struct isn't gonna slow them down.

I am not talking about kind matching to sets of restriction. Just a simple
mask of thing that allowed on that memory. You can add thing like GUP or any
other mechanism that i can't think of right now.

I really think that the vma flag is a bad idea, my expectation is that we
will see more vma with a mix of device and regular memory. I don't think the
only workload will be some big vma device only (ie only access by device) or
CPU only. I believe we will see everything on the spectrum from highly
fragmented to completetly regular.

Cheers,
Jérôme

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web