Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1506869 > unrolled thread

[RFC 5/8] mm: Add new flag VM_CDM for coherent device memory

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [RFC 5/8] mm: Add new flag VM_CDM for coherent device memory Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-10-24 06:40 +0200
    Re: [RFC 5/8] mm: Add new flag VM_CDM for coherent device memory Dave Hansen <dave.hansen@intel.com> - 2016-10-24 19:40 +0200
      Re: [RFC 5/8] mm: Add new flag VM_CDM for coherent device memory Dave Hansen <dave.hansen@intel.com> - 2016-10-24 20:10 +0200
      Re: [RFC 5/8] mm: Add new flag VM_CDM for coherent device memory Balbir Singh <bsingharora@gmail.com> - 2016-10-25 14:40 +0200
      Re: [RFC 5/8] mm: Add new flag VM_CDM for coherent device memory "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-10-25 21:30 +0200
        Re: [RFC 5/8] mm: Add new flag VM_CDM for coherent device memory Dave Hansen <dave.hansen@intel.com> - 2016-10-25 22:10 +0200

#1506869 — [RFC 5/8] mm: Add new flag VM_CDM for coherent device memory

VMAs containing coherent device memory should be marked with VM_CDM. These
VMAs need to be identified in various core kernel paths and this new flag
will help in this regard.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 include/linux/mm.h |  5 +++++
 mm/mempolicy.c     | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 3a19185..acee4d1 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -182,6 +182,11 @@ extern unsigned int kobjsize(const void *objp);
 #define VM_ACCOUNT	0x00100000	/* Is a VM accounted object */
 #define VM_NORESERVE	0x00200000	/* should the VM suppress accounting */
 #define VM_HUGETLB	0x00400000	/* Huge TLB Page VM */
+
+#ifdef CONFIG_COHERENT_DEVICE
+#define VM_CDM		0x00800000	/* Contains coherent device memory */
+#endif
+
 #define VM_ARCH_1	0x01000000	/* Architecture-specific flag */
 #define VM_ARCH_2	0x02000000
 #define VM_DONTDUMP	0x04000000	/* Do not include in the core dump */
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index cb1ba01..b983cea 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -174,6 +174,47 @@ static void mpol_relative_nodemask(nodemask_t *ret, const nodemask_t *orig,
 	nodes_onto(*ret, tmp, *rel);
 }
 
+#ifdef CONFIG_COHERENT_DEVICE
+static bool nodemask_contains_cdm(nodemask_t *nodes)
+{
+	int weight, nid, i;
+	nodemask_t mask;
+
+
+	if (!nodes)
+		return false;
+
+	mask = *nodes;
+	weight = nodes_weight(mask);
+	nid = first_node(mask);
+	for (i = 0; i < weight; i++) {
+		if (isolated_cdm_node(nid))
+			return true;
+		nid = next_node(nid, mask);
+	}
+	return false;
+}
+
+static void update_coherent_vma_flag(nodemask_t *nmask,
+		struct page *page, struct vm_area_struct *vma)
+{
+	if (!page)
+		return;
+
+	if (nodemask_contains_cdm(nmask)) {
+		if (!(vma->vm_flags & VM_CDM)) {
+			if (isolated_cdm_node(page_to_nid(page)))
+				vma->vm_flags |= VM_CDM;
+		}
+	}
+}
+#else
+static void update_coherent_vma_flag(nodemask_t *nmask,
+		struct page *page, struct vm_area_struct *vma)
+{
+}
+#endif
+
 static int mpol_new_interleave(struct mempolicy *pol, const nodemask_t *nodes)
 {
 	if (nodes_empty(*nodes))
@@ -2045,6 +2086,8 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma,
 	zl = policy_zonelist(gfp, pol, node);
 	mpol_cond_put(pol);
 	page = __alloc_pages_nodemask(gfp, order, zl, nmask);
+	update_coherent_vma_flag(nmask, page, vma);
+
 out:
 	if (unlikely(!page && read_mems_allowed_retry(cpuset_mems_cookie)))
 		goto retry_cpuset;
-- 
2.1.0

[toc] | [next] | [standalone]

#1507485

On 10/23/2016 09:31 PM, Anshuman Khandual wrote:
> VMAs containing coherent device memory should be marked with VM_CDM. These
> VMAs need to be identified in various core kernel paths and this new flag
> will help in this regard.

... and it's sticky?  So if a VMA *ever* has one of these funky pages in
it, it's stuck being VM_CDM forever?  Never to be merged with other
VMAs?  Never to see the light of autonuma ever again?

What if a 100TB VMA has one page of fancy pants device memory, and the
rest normal vanilla memory?  Do we really want to consider the whole
thing fancy?

This whole patch set is looking really hackish.  If you want things to
be isolated from the VM, them it should probably *actually* be isolated
from the VM.  As Jerome mentioned, ZONE_DEVICE is probably a better
thing to use here than to try what you're attempting.

[toc] | [prev] | [next] | [standalone]

#1507537

On 10/24/2016 10:38 AM, Dave Hansen wrote:
> On 10/23/2016 09:31 PM, Anshuman Khandual wrote:
>> > VMAs containing coherent device memory should be marked with VM_CDM. These
>> > VMAs need to be identified in various core kernel paths and this new flag
>> > will help in this regard.
> ... and it's sticky?  So if a VMA *ever* has one of these funky pages in
> it, it's stuck being VM_CDM forever?  Never to be merged with other
> VMAs?  Never to see the light of autonuma ever again?

Urg, this is even worse than I suspected.

Does this handle shared pages (like the page cache mode you call out as
a requirement) where the "cdm" page is faulted into one process VMA, but
it was allocated against another?

Can't that give you a "cdm" page mapped into a non-VM_CDM VMA?  Or, a
VM_CDM VMA with no "cdm" pages in it?

[toc] | [prev] | [next] | [standalone]

#1508263

On 25/10/16 04:38, Dave Hansen wrote:
> On 10/23/2016 09:31 PM, Anshuman Khandual wrote:
>> VMAs containing coherent device memory should be marked with VM_CDM. These
>> VMAs need to be identified in various core kernel paths and this new flag
>> will help in this regard.
> 
> ... and it's sticky?  So if a VMA *ever* has one of these funky pages in
> it, it's stuck being VM_CDM forever?  Never to be merged with other
> VMAs?  Never to see the light of autonuma ever again?
> 
> What if a 100TB VMA has one page of fancy pants device memory, and the
> rest normal vanilla memory?  Do we really want to consider the whole
> thing fancy?
> 

Those are good review comments to improve the patchset.

> This whole patch set is looking really hackish.  If you want things to
> be isolated from the VM, them it should probably *actually* be isolated
> from the VM.  As Jerome mentioned, ZONE_DEVICE is probably a better
> thing to use here than to try what you're attempting.
> 

The RFC explains the motivation, this is not fancy pants, it is regular
memory from the systems perspective, with some changes as described

Thanks for the review!
Balbir Singh

[toc] | [prev] | [next] | [standalone]

#1508573

Dave Hansen <dave.hansen@intel.com> writes:

> On 10/23/2016 09:31 PM, Anshuman Khandual wrote:
>> VMAs containing coherent device memory should be marked with VM_CDM. These
>> VMAs need to be identified in various core kernel paths and this new flag
>> will help in this regard.
>
> ... and it's sticky?  So if a VMA *ever* has one of these funky pages in
> it, it's stuck being VM_CDM forever?  Never to be merged with other
> VMAs?  Never to see the light of autonuma ever again?
>
> What if a 100TB VMA has one page of fancy pants device memory, and the
> rest normal vanilla memory?  Do we really want to consider the whole
> thing fancy?

This definitely needs fine tuning. I guess we should look at this as
possibly stating that, coherent device would like to not participate in
auto numa balancing, because it is difficult to update the core kernel
about access patters within the coherent device. This can result in core
kernel always trying to migrate pages from coherent device to system ram
even though we have large number of access within coherent device.


One possible option is to use a software pte bit (may be steal
_PAGE_DEVMAP) and prevent a numa pte setup from change_prot_numa().
ie, if the pfn backing the pte is from coherent device we don't allow
that to be converted to a prot none pte for numa faults ?


-aneesh

[toc] | [prev] | [next] | [standalone]

#1508592

On 10/25/2016 12:20 PM, Aneesh Kumar K.V wrote:
> Dave Hansen <dave.hansen@intel.com> writes:
>> On 10/23/2016 09:31 PM, Anshuman Khandual wrote:
>>> VMAs containing coherent device memory should be marked with VM_CDM. These
>>> VMAs need to be identified in various core kernel paths and this new flag
>>> will help in this regard.
>>
>> ... and it's sticky?  So if a VMA *ever* has one of these funky pages in
>> it, it's stuck being VM_CDM forever?  Never to be merged with other
>> VMAs?  Never to see the light of autonuma ever again?
>>
>> What if a 100TB VMA has one page of fancy pants device memory, and the
>> rest normal vanilla memory?  Do we really want to consider the whole
>> thing fancy?
> 
> This definitely needs fine tuning. I guess we should look at this as
> possibly stating that, coherent device would like to not participate in
> auto numa balancing
...

Right, in this one, particular case you don't want NUMA balancing.  But,
if you have to take an _explicit_ action to even get access to this
coherent memory (setting a NUMA policy), why keeps that explicit action
from also explicitly disabling NUMA migration?

I really don't think we should tie together the isolation aspect with
anything else, including NUMA balancing.

For instance, on x86, we have the ability for devices to grok the CPU's
page tables, including doing faults.  There's very little to stop us
from doing things like autonuma.

> One possible option is to use a software pte bit (may be steal
> _PAGE_DEVMAP) and prevent a numa pte setup from change_prot_numa().
> ie, if the pfn backing the pte is from coherent device we don't allow
> that to be converted to a prot none pte for numa faults ?

Why would you need to tag individual pages, especially if the VMA has a
policy set on it that disallows migration?

But, even if you did need to identify individual pages from the PTE, you
can easily do:

	page_to_nid(pfn_to_page(pte_pfn(pte)))

and then tell if the node is a fancy-pants device node.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web