Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1491361 > unrolled thread

[PATCH v1 00/12] mm: THP migration support

Back to article view | Back to linux.kernel

Contents

  [PATCH v1 00/12] mm: THP migration support zi.yan@sent.com - 2016-09-26 17:30 +0200
    [PATCH v1 04/12] mm: thp: enable thp migration in generic path zi.yan@sent.com - 2016-09-26 17:30 +0200
    [PATCH v1 05/12] mm: thp: check pmd migration entry in common path zi.yan@sent.com - 2016-09-26 17:30 +0200
    [PATCH v1 01/12] mm: mempolicy: add queue_pages_node_check() zi.yan@sent.com - 2016-09-26 17:30 +0200
    [PATCH v1 09/12] mm: mempolicy: mbind and migrate_pages support thp migration zi.yan@sent.com - 2016-09-26 17:30 +0200
    Re: [PATCH v1 00/12] THP migration support "Zi Yan" <zi.yan@sent.com> - 2016-09-26 17:40 +0200
      Re: [PATCH v1 00/12] THP migration support Naoya Horiguchi <n-horiguchi@ah.jp.nec.com> - 2016-09-29 10:30 +0200
        Re: [PATCH v1 00/12] THP migration support "Zi Yan" <zi.yan@sent.com> - 2016-09-30 04:40 +0200

#1491361 — [PATCH v1 00/12] mm: THP migration support

From: Zi Yan <zi.yan@cs.rutgers.edu>

Hi all,

This patchset is based on Naoya Horiguchi's page migration enchancement 
for thp patchset with additional IBM ppc64 support. And I rebase it
on the latest upstream commit.

The motivation is that 4KB page migration is underutilizing the memory
bandwidth compared to 2MB THP migration.

As part of my internship work in NVIDIA, I compared the bandwidth
utilizations between 512 4KB pages and 1 2MB page in both x86_64 and ppc64.
And the results show that migrating 512 4KB pages takes only 3x and 1.15x of
the time, compared to migrating single 2MB THP, in x86_64 and ppc64 
respectively.

Here are the actual BW numbers (total_data_size/migration_time):
        | 512 4KB pages | 1 2MB THP  |  1 4KB page
x86_64  |  0.98GB/s     |  2.97GB/s  |   0.06GB/s
ppc64   |  6.14GB/s     |  7.10GB/s  |   1.24GB/s

Any comments or advices are welcome.

Here is the original message from Naoya:

This patchset enhances page migration functionality to handle thp migration
for various page migration's callers:
 - mbind(2)
 - move_pages(2)
 - migrate_pages(2)
 - cgroup/cpuset migration
 - memory hotremove
 - soft offline

The main benefit is that we can avoid unnecessary thp splits, which helps us
avoid performance decrease when your applications handles NUMA optimization on
their own.

The implementation is similar to that of normal page migration, the key point
is that we modify a pmd to a pmd migration entry in swap-entry like format.
pmd_present() is not simple and it's not enough by itself to determine whether
a given pmd is a pmd migration entry. See patch 3/11 and 5/11 for details.

Here're topics which might be helpful to start discussion:

- at this point, this functionality is limited to x86_64.

- there's alrealy an implementation of thp migration in autonuma code of which
  this patchset doesn't touch anything because it works fine as it is.

- fallback to thp split: current implementation just fails a migration trial if
  thp migration fails. It's possible to retry migration after splitting the thp,
  but that's not included in this version.



Thanks,
Zi Yan
---

Naoya Horiguchi (11):
  mm: mempolicy: add queue_pages_node_check()
  mm: thp: introduce CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION
  mm: thp: add helpers related to thp/pmd migration
  mm: thp: enable thp migration in generic path
  mm: thp: check pmd migration entry in common path
  mm: soft-dirty: keep soft-dirty bits over thp migration
  mm: hwpoison: fix race between unpoisoning and freeing migrate source
    page
  mm: hwpoison: soft offline supports thp migration
  mm: mempolicy: mbind and migrate_pages support thp migration
  mm: migrate: move_pages() supports thp migration
  mm: memory_hotplug: memory hotremove supports thp migration

Zi Yan (1):
  mm: ppc64: Add THP migration support for ppc64.

 arch/powerpc/Kconfig                         |   4 +
 arch/powerpc/include/asm/book3s/64/pgtable.h |  23 ++++
 arch/x86/Kconfig                             |   4 +
 arch/x86/include/asm/pgtable.h               |  28 ++++
 arch/x86/include/asm/pgtable_64.h            |   2 +
 arch/x86/include/asm/pgtable_types.h         |   8 +-
 arch/x86/mm/gup.c                            |   3 +
 fs/proc/task_mmu.c                           |  20 +--
 include/asm-generic/pgtable.h                |  34 ++++-
 include/linux/huge_mm.h                      |  13 ++
 include/linux/swapops.h                      |  64 ++++++++++
 mm/Kconfig                                   |   3 +
 mm/gup.c                                     |   8 ++
 mm/huge_memory.c                             | 184 +++++++++++++++++++++++++--
 mm/memcontrol.c                              |   2 +
 mm/memory-failure.c                          |  41 +++---
 mm/memory.c                                  |   5 +
 mm/memory_hotplug.c                          |   8 ++
 mm/mempolicy.c                               | 108 ++++++++++++----
 mm/migrate.c                                 |  49 ++++++-
 mm/page_isolation.c                          |   9 ++
 mm/rmap.c                                    |   5 +
 22 files changed, 549 insertions(+), 76 deletions(-)

-- 
2.9.3

[toc] | [next] | [standalone]

#1491362 — [PATCH v1 04/12] mm: thp: enable thp migration in generic path

From: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>

This patch makes it possible to support thp migration gradually. If you fail
to allocate a destination page as a thp, you just split the source thp as we
do now, and then enter the normal page migration. If you succeed to allocate
destination thp, you enter thp migration. Subsequent patches actually enable
thp migration for each caller of page migration by allowing its get_new_page()
callback to allocate thps.

Signed-off-by: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
---
 mm/migrate.c | 2 +-
 mm/rmap.c    | 5 +++++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 95613e7..dfca530 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1123,7 +1123,7 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page,
 		goto out;
 	}
 
-	if (unlikely(PageTransHuge(page))) {
+	if (unlikely(PageTransHuge(page) && !PageTransHuge(newpage))) {
 		lock_page(page);
 		rc = split_huge_page(page);
 		unlock_page(page);
diff --git a/mm/rmap.c b/mm/rmap.c
index 1ef3640..d53fff5 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1443,6 +1443,11 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 	struct rmap_private *rp = arg;
 	enum ttu_flags flags = rp->flags;
 
+	if (!PageHuge(page) && PageTransHuge(page)) {
+		VM_BUG_ON_PAGE(!(flags & TTU_MIGRATION), page);
+		return set_pmd_migration_entry(page, mm, address);
+	}
+
 	/* munlock has nothing to gain from examining un-locked vmas */
 	if ((flags & TTU_MUNLOCK) && !(vma->vm_flags & VM_LOCKED))
 		goto out;
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1491363 — [PATCH v1 05/12] mm: thp: check pmd migration entry in common path

From: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>

If one of callers of page migration starts to handle thp, memory management code
start to see pmd migration entry, so we need to prepare for it before enabling.
This patch changes various code point which checks the status of given pmds in
order to prevent race between thp migration and the pmd-related works.

Signed-off-by: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
---
 arch/x86/mm/gup.c  |  3 +++
 fs/proc/task_mmu.c | 20 +++++++-------
 mm/gup.c           |  8 ++++++
 mm/huge_memory.c   | 76 +++++++++++++++++++++++++++++++++++++++++++++++-------
 mm/memcontrol.c    |  2 ++
 mm/memory.c        |  5 ++++
 6 files changed, 95 insertions(+), 19 deletions(-)

diff --git a/arch/x86/mm/gup.c b/arch/x86/mm/gup.c
index b8b6a60..72d0bef 100644
--- a/arch/x86/mm/gup.c
+++ b/arch/x86/mm/gup.c
@@ -10,6 +10,7 @@
 #include <linux/highmem.h>
 #include <linux/swap.h>
 #include <linux/memremap.h>
+#include <linux/swapops.h>
 
 #include <asm/mmu_context.h>
 #include <asm/pgtable.h>
@@ -225,6 +226,8 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
 		if (pmd_none(pmd))
 			return 0;
 		if (unlikely(pmd_large(pmd) || !pmd_present(pmd))) {
+			if (unlikely(is_pmd_migration_entry(pmd)))
+				return 0;
 			/*
 			 * NUMA hinting faults need to be handled in the GUP
 			 * slowpath for accounting purposes and so that they
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index f6fa99e..60f6ce3 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -931,6 +931,9 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
 
 	ptl = pmd_trans_huge_lock(pmd, vma);
 	if (ptl) {
+		if (unlikely(is_pmd_migration_entry(*pmd)))
+			goto out;
+
 		if (cp->type == CLEAR_REFS_SOFT_DIRTY) {
 			clear_soft_dirty_pmd(vma, addr, pmd);
 			goto out;
@@ -1215,19 +1218,18 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,
 	if (ptl) {
 		u64 flags = 0, frame = 0;
 		pmd_t pmd = *pmdp;
+		struct page *page;
 
 		if ((vma->vm_flags & VM_SOFTDIRTY) || pmd_soft_dirty(pmd))
 			flags |= PM_SOFT_DIRTY;
 
-		/*
-		 * Currently pmd for thp is always present because thp
-		 * can not be swapped-out, migrated, or HWPOISONed
-		 * (split in such cases instead.)
-		 * This if-check is just to prepare for future implementation.
-		 */
-		if (pmd_present(pmd)) {
-			struct page *page = pmd_page(pmd);
-
+		if (is_pmd_migration_entry(pmd)) {
+			swp_entry_t entry = pmd_to_swp_entry(pmd);
+			frame = swp_type(entry) |
+				(swp_offset(entry) << MAX_SWAPFILES_SHIFT);
+			page = migration_entry_to_page(entry);
+		} else if (pmd_present(pmd)) {
+			page = pmd_page(pmd);
 			if (page_mapcount(page) == 1)
 				flags |= PM_MMAP_EXCLUSIVE;
 
diff --git a/mm/gup.c b/mm/gup.c
index 96b2b2f..ef56be2 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -272,6 +272,11 @@ struct page *follow_page_mask(struct vm_area_struct *vma,
 		spin_unlock(ptl);
 		return follow_page_pte(vma, address, pmd, flags);
 	}
+	if (is_pmd_migration_entry(*pmd)) {
+		spin_unlock(ptl);
+		return no_page_table(vma, flags);
+	}
+
 	if (flags & FOLL_SPLIT) {
 		int ret;
 		page = pmd_page(*pmd);
@@ -1362,6 +1367,9 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
 			return 0;
 
 		if (unlikely(pmd_trans_huge(pmd) || pmd_huge(pmd))) {
+			if (unlikely(is_pmd_migration_entry(pmd)))
+				return 0;
+
 			/*
 			 * NUMA hinting faults need to be handled in the GUP
 			 * slowpath for accounting purposes and so that they
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 0cd39ef..f4fcfc7 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -787,6 +787,19 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		goto out_unlock;
 	}
 
+	if (unlikely(is_pmd_migration_entry(pmd))) {
+		swp_entry_t entry = pmd_to_swp_entry(pmd);
+
+		if (is_write_migration_entry(entry)) {
+			make_migration_entry_read(&entry);
+			pmd = swp_entry_to_pmd(entry);
+			set_pmd_at(src_mm, addr, src_pmd, pmd);
+		}
+		set_pmd_at(dst_mm, addr, dst_pmd, pmd);
+		ret = 0;
+		goto out_unlock;
+	}
+
 	src_page = pmd_page(pmd);
 	VM_BUG_ON_PAGE(!PageHead(src_page), src_page);
 	get_page(src_page);
@@ -952,6 +965,9 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
 	if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
 		goto out_unlock;
 
+	if (unlikely(is_pmd_migration_entry(*fe->pmd)))
+		goto out_unlock;
+
 	page = pmd_page(orig_pmd);
 	VM_BUG_ON_PAGE(!PageCompound(page) || !PageHead(page), page);
 	/*
@@ -1077,7 +1093,15 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
 	if ((flags & FOLL_NUMA) && pmd_protnone(*pmd))
 		goto out;
 
-	page = pmd_page(*pmd);
+	if (is_pmd_migration_entry(*pmd)) {
+		swp_entry_t entry;
+		entry = pmd_to_swp_entry(*pmd);
+		if (!is_migration_entry(entry))
+			goto out;
+		page = pfn_to_page(swp_offset(entry));
+	} else
+		page = pmd_page(*pmd);
+
 	VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page);
 	if (flags & FOLL_TOUCH)
 		touch_pmd(vma, addr, pmd);
@@ -1273,6 +1297,9 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	if (is_huge_zero_pmd(orig_pmd))
 		goto out;
 
+	if (unlikely(is_pmd_migration_entry(orig_pmd)))
+		goto out;
+
 	page = pmd_page(orig_pmd);
 	/*
 	 * If other processes are mapping this page, we couldn't discard
@@ -1348,21 +1375,40 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		spin_unlock(ptl);
 		tlb_remove_page(tlb, pmd_page(orig_pmd));
 	} else {
-		struct page *page = pmd_page(orig_pmd);
-		page_remove_rmap(page, true);
-		VM_BUG_ON_PAGE(page_mapcount(page) < 0, page);
-		VM_BUG_ON_PAGE(!PageHead(page), page);
-		if (PageAnon(page)) {
+		struct page *page;
+		int migration = 0;
+
+		if (!is_pmd_migration_entry(orig_pmd)) {
+			page = pmd_page(orig_pmd);
+			page_remove_rmap(page, true);
+			VM_BUG_ON_PAGE(page_mapcount(page) < 0, page);
+			VM_BUG_ON_PAGE(!PageHead(page), page);
+			if (PageAnon(page)) {
+				pgtable_t pgtable;
+				pgtable = pgtable_trans_huge_withdraw(tlb->mm, pmd);
+				pte_free(tlb->mm, pgtable);
+				atomic_long_dec(&tlb->mm->nr_ptes);
+				add_mm_counter(tlb->mm, MM_ANONPAGES, -HPAGE_PMD_NR);
+			} else {
+				add_mm_counter(tlb->mm, MM_FILEPAGES, -HPAGE_PMD_NR);
+			}
+		} else {
+			swp_entry_t entry;
 			pgtable_t pgtable;
+
+			entry = pmd_to_swp_entry(orig_pmd);
+			free_swap_and_cache(entry); /* waring in failure? */
+
+			add_mm_counter(tlb->mm, MM_ANONPAGES, -HPAGE_PMD_NR);
 			pgtable = pgtable_trans_huge_withdraw(tlb->mm, pmd);
 			pte_free(tlb->mm, pgtable);
 			atomic_long_dec(&tlb->mm->nr_ptes);
-			add_mm_counter(tlb->mm, MM_ANONPAGES, -HPAGE_PMD_NR);
-		} else {
-			add_mm_counter(tlb->mm, MM_FILEPAGES, -HPAGE_PMD_NR);
+
+			migration = 1;
 		}
 		spin_unlock(ptl);
-		tlb_remove_page_size(tlb, page, HPAGE_PMD_SIZE);
+		if (!migration)
+			tlb_remove_page_size(tlb, page, HPAGE_PMD_SIZE);
 	}
 	return 1;
 }
@@ -1445,6 +1491,11 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 			return ret;
 		}
 
+		if (is_pmd_migration_entry(*pmd)) {
+			spin_unlock(ptl);
+			return ret;
+		}
+
 		if (!prot_numa || !pmd_protnone(*pmd)) {
 			entry = pmdp_huge_get_and_clear_notify(mm, addr, pmd);
 			entry = pmd_modify(entry, newprot);
@@ -1656,6 +1707,11 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
 
 	if (pmd_trans_huge(*pmd)) {
 		page = pmd_page(*pmd);
+
+		if (is_pmd_migration_entry(*pmd)) {
+			goto out;
+		}
+
 		if (PageMlocked(page))
 			clear_page_mlock(page);
 	} else if (!pmd_devmap(*pmd))
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 4be518d..421ac4ff 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4649,6 +4649,8 @@ static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma,
 	struct page *page = NULL;
 	enum mc_target_type ret = MC_TARGET_NONE;
 
+	if (unlikely(is_pmd_migration_entry(pmd)))
+		return ret;
 	page = pmd_page(pmd);
 	VM_BUG_ON_PAGE(!page || !PageHead(page), page);
 	if (!(mc.flags & MOVE_ANON))
diff --git a/mm/memory.c b/mm/memory.c
index 83be99d..3ad3bb2 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3590,6 +3590,11 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
 
 		barrier();
 		if (pmd_trans_huge(orig_pmd) || pmd_devmap(orig_pmd)) {
+			if (unlikely(is_pmd_migration_entry(orig_pmd))) {
+				pmd_migration_entry_wait(mm, fe.pmd);
+				return 0;
+			}
+
 			if (pmd_protnone(orig_pmd))
 				return do_huge_pmd_numa_page(&fe, orig_pmd);
 
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1491364 — [PATCH v1 01/12] mm: mempolicy: add queue_pages_node_check()

From: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>

Introduce a separate check routine related to MPOL_MF_INVERT flag. This patch
just does cleanup, no behavioral change.

Signed-off-by: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
---
 mm/mempolicy.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 2da72a5..dc8e913 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -475,6 +475,15 @@ struct queue_pages {
 	struct vm_area_struct *prev;
 };
 
+static inline bool queue_pages_node_check(struct page *page,
+					struct queue_pages *qp)
+{
+	int nid = page_to_nid(page);
+	unsigned long flags = qp->flags;
+
+	return node_isset(nid, *qp->nmask) == !!(flags & MPOL_MF_INVERT);
+}
+
 /*
  * Scan through pages checking if pages follow certain conditions,
  * and move them to the pagelist if they do.
@@ -528,8 +537,7 @@ retry:
 		 */
 		if (PageReserved(page))
 			continue;
-		nid = page_to_nid(page);
-		if (node_isset(nid, *qp->nmask) == !!(flags & MPOL_MF_INVERT))
+		if (queue_pages_node_check(page, qp))
 			continue;
 		if (PageTransCompound(page)) {
 			get_page(page);
@@ -561,7 +569,6 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask,
 #ifdef CONFIG_HUGETLB_PAGE
 	struct queue_pages *qp = walk->private;
 	unsigned long flags = qp->flags;
-	int nid;
 	struct page *page;
 	spinlock_t *ptl;
 	pte_t entry;
@@ -571,8 +578,7 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask,
 	if (!pte_present(entry))
 		goto unlock;
 	page = pte_page(entry);
-	nid = page_to_nid(page);
-	if (node_isset(nid, *qp->nmask) == !!(flags & MPOL_MF_INVERT))
+	if (queue_pages_node_check(page, qp))
 		goto unlock;
 	/* With MPOL_MF_MOVE, we migrate only unshared hugepage. */
 	if (flags & (MPOL_MF_MOVE_ALL) ||
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1491365 — [PATCH v1 09/12] mm: mempolicy: mbind and migrate_pages support thp migration

From: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>

This patch enables thp migration for mbind(2) and migrate_pages(2).

Signed-off-by: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
---
 mm/mempolicy.c | 92 ++++++++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 70 insertions(+), 22 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index dc8e913..c10f71b 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -94,6 +94,7 @@
 #include <linux/mm_inline.h>
 #include <linux/mmu_notifier.h>
 #include <linux/printk.h>
+#include <linux/swapops.h>
 
 #include <asm/tlbflush.h>
 #include <asm/uaccess.h>
@@ -484,6 +485,49 @@ static inline bool queue_pages_node_check(struct page *page,
 	return node_isset(nid, *qp->nmask) == !!(flags & MPOL_MF_INVERT);
 }
 
+static int queue_pages_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr,
+				unsigned long end, struct mm_walk *walk)
+{
+	int ret = 0;
+	struct page *page;
+	struct queue_pages *qp = walk->private;
+	unsigned long flags;
+
+	if (unlikely(is_pmd_migration_entry(*pmd))) {
+		ret = 1;
+		goto unlock;
+	}
+	page = pmd_page(*pmd);
+	if (is_huge_zero_page(page)) {
+		spin_unlock(ptl);
+		split_huge_pmd(walk->vma, pmd, addr);
+		goto out;
+	}
+	if ((end - addr != HPAGE_PMD_SIZE) || !thp_migration_supported()) {
+		get_page(page);
+		spin_unlock(ptl);
+		lock_page(page);
+		ret = split_huge_page(page);
+		unlock_page(page);
+		put_page(page);
+		goto out;
+	}
+	if (queue_pages_node_check(page, qp)) {
+		ret = 1;
+		goto unlock;
+	}
+
+	ret = 1;
+	flags = qp->flags;
+	/* go to thp migration */
+	if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
+		migrate_page_add(page, qp->pagelist, flags);
+unlock:
+	spin_unlock(ptl);
+out:
+	return ret;
+}
+
 /*
  * Scan through pages checking if pages follow certain conditions,
  * and move them to the pagelist if they do.
@@ -495,30 +539,15 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,
 	struct page *page;
 	struct queue_pages *qp = walk->private;
 	unsigned long flags = qp->flags;
-	int nid, ret;
+	int ret;
 	pte_t *pte;
 	spinlock_t *ptl;
 
-	if (pmd_trans_huge(*pmd)) {
-		ptl = pmd_lock(walk->mm, pmd);
-		if (pmd_trans_huge(*pmd)) {
-			page = pmd_page(*pmd);
-			if (is_huge_zero_page(page)) {
-				spin_unlock(ptl);
-				split_huge_pmd(vma, pmd, addr);
-			} else {
-				get_page(page);
-				spin_unlock(ptl);
-				lock_page(page);
-				ret = split_huge_page(page);
-				unlock_page(page);
-				put_page(page);
-				if (ret)
-					return 0;
-			}
-		} else {
-			spin_unlock(ptl);
-		}
+	ptl = pmd_trans_huge_lock(pmd, vma);
+	if (ptl) {
+		ret = queue_pages_pmd(pmd, ptl, addr, end, walk);
+		if (ret)
+			return 0;
 	}
 
 	if (pmd_trans_unstable(pmd))
@@ -979,7 +1008,17 @@ static struct page *new_node_page(struct page *page, unsigned long node, int **x
 	if (PageHuge(page))
 		return alloc_huge_page_node(page_hstate(compound_head(page)),
 					node);
-	else
+	else if (thp_migration_supported() && PageTransHuge(page)) {
+		struct page *thp;
+
+		thp = alloc_pages_node(node,
+			(GFP_TRANSHUGE | __GFP_THISNODE) & ~__GFP_RECLAIM,
+			HPAGE_PMD_ORDER);
+		if (!thp)
+			return NULL;
+		prep_transhuge_page(thp);
+		return thp;
+	} else
 		return __alloc_pages_node(node, GFP_HIGHUSER_MOVABLE |
 						    __GFP_THISNODE, 0);
 }
@@ -1145,6 +1184,15 @@ static struct page *new_page(struct page *page, unsigned long start, int **x)
 	if (PageHuge(page)) {
 		BUG_ON(!vma);
 		return alloc_huge_page_noerr(vma, address, 1);
+	} else if (thp_migration_supported() && PageTransHuge(page)) {
+		struct page *thp;
+
+		thp = alloc_hugepage_vma(GFP_TRANSHUGE, vma, address,
+					 HPAGE_PMD_ORDER);
+		if (!thp)
+			return NULL;
+		prep_transhuge_page(thp);
+		return thp;
 	}
 	/*
 	 * if !vma, alloc_page_vma() will use task or system default policy
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1491367 — Re: [PATCH v1 00/12] THP migration support

[Multipart message — attachments visible in raw view] — view raw

On 26 Sep 2016, at 11:22, zi.yan@sent.com wrote:

> From: Zi Yan <zi.yan@cs.rutgers.edu>
>
> Hi all,
>
> This patchset is based on Naoya Horiguchi's page migration enchancement
> for thp patchset with additional IBM ppc64 support. And I rebase it
> on the latest upstream commit.
>
> The motivation is that 4KB page migration is underutilizing the memory
> bandwidth compared to 2MB THP migration.

Sorry, in ppc64, 64KB page was used as the base page and 16MB THP
was used.

>
> As part of my internship work in NVIDIA, I compared the bandwidth
> utilizations between 512 4KB pages and 1 2MB page in both x86_64 and ppc64.
> And the results show that migrating 512 4KB pages takes only 3x and 1.15x of
> the time, compared to migrating single 2MB THP, in x86_64 and ppc64
> respectively.
>
> Here are the actual BW numbers (total_data_size/migration_time):
>         | 512 4KB pages | 1 2MB THP  |  1 4KB page
> x86_64  |  0.98GB/s     |  2.97GB/s  |   0.06GB/s
> ppc64   |  6.14GB/s     |  7.10GB/s  |   1.24GB/s

And the BW number should be:
         | 512 4KB pages | 1 2MB THP  |  1 4KB page
 x86_64  |  0.98GB/s     |  2.97GB/s  |   0.06GB/s

         | 512 64KB pages | 1 16MB THP  |  1 64KB page
 ppc64   |  6.14GB/s      |  7.10GB/s   |   1.24GB/s

>
> Any comments or advices are welcome.
>
> Here is the original message from Naoya:
>
> This patchset enhances page migration functionality to handle thp migration
> for various page migration's callers:
>  - mbind(2)
>  - move_pages(2)
>  - migrate_pages(2)
>  - cgroup/cpuset migration
>  - memory hotremove
>  - soft offline
>
> The main benefit is that we can avoid unnecessary thp splits, which helps us
> avoid performance decrease when your applications handles NUMA optimization on
> their own.
>
> The implementation is similar to that of normal page migration, the key point
> is that we modify a pmd to a pmd migration entry in swap-entry like format.
> pmd_present() is not simple and it's not enough by itself to determine whether
> a given pmd is a pmd migration entry. See patch 3/11 and 5/11 for details.
>
> Here're topics which might be helpful to start discussion:
>
> - at this point, this functionality is limited to x86_64.
>
> - there's alrealy an implementation of thp migration in autonuma code of which
>   this patchset doesn't touch anything because it works fine as it is.
>
> - fallback to thp split: current implementation just fails a migration trial if
>   thp migration fails. It's possible to retry migration after splitting the thp,
>   but that's not included in this version.
>
> Thanks,
> Zi Yan
> ---
>
> Naoya Horiguchi (11):
>   mm: mempolicy: add queue_pages_node_check()
>   mm: thp: introduce CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION
>   mm: thp: add helpers related to thp/pmd migration
>   mm: thp: enable thp migration in generic path
>   mm: thp: check pmd migration entry in common path
>   mm: soft-dirty: keep soft-dirty bits over thp migration
>   mm: hwpoison: fix race between unpoisoning and freeing migrate source
>     page
>   mm: hwpoison: soft offline supports thp migration
>   mm: mempolicy: mbind and migrate_pages support thp migration
>   mm: migrate: move_pages() supports thp migration
>   mm: memory_hotplug: memory hotremove supports thp migration
>
> Zi Yan (1):
>   mm: ppc64: Add THP migration support for ppc64.
>
>  arch/powerpc/Kconfig                         |   4 +
>  arch/powerpc/include/asm/book3s/64/pgtable.h |  23 ++++
>  arch/x86/Kconfig                             |   4 +
>  arch/x86/include/asm/pgtable.h               |  28 ++++
>  arch/x86/include/asm/pgtable_64.h            |   2 +
>  arch/x86/include/asm/pgtable_types.h         |   8 +-
>  arch/x86/mm/gup.c                            |   3 +
>  fs/proc/task_mmu.c                           |  20 +--
>  include/asm-generic/pgtable.h                |  34 ++++-
>  include/linux/huge_mm.h                      |  13 ++
>  include/linux/swapops.h                      |  64 ++++++++++
>  mm/Kconfig                                   |   3 +
>  mm/gup.c                                     |   8 ++
>  mm/huge_memory.c                             | 184 +++++++++++++++++++++++++--
>  mm/memcontrol.c                              |   2 +
>  mm/memory-failure.c                          |  41 +++---
>  mm/memory.c                                  |   5 +
>  mm/memory_hotplug.c                          |   8 ++
>  mm/mempolicy.c                               | 108 ++++++++++++----
>  mm/migrate.c                                 |  49 ++++++-
>  mm/page_isolation.c                          |   9 ++
>  mm/rmap.c                                    |   5 +
>  22 files changed, 549 insertions(+), 76 deletions(-)
>
> -- 
> 2.9.3
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>


--
Best Regards
Yan Zi

[toc] | [prev] | [next] | [standalone]

#1493196 — Re: [PATCH v1 00/12] THP migration support

Hi Yan,

On Mon, Sep 26, 2016 at 11:38:05AM -0400, Zi Yan wrote:
> On 26 Sep 2016, at 11:22, zi.yan@sent.com wrote:
> 
> > From: Zi Yan <zi.yan@cs.rutgers.edu>
> >
> > Hi all,
> >
> > This patchset is based on Naoya Horiguchi's page migration enchancement
> > for thp patchset with additional IBM ppc64 support. And I rebase it
> > on the latest upstream commit.

Thanks for helping,

I think that you seem to do some testing with these patches on powerpc,
which shows that thp migration can be enabled relatively easily for
non-x86_64. This is a good news to me.

And I apology for my slow development over this patchset.
My previous post was about 5 months ago, and I've not done ver.2 due to
many interruptions. Someone also privately asked me about the progress
of this work, so I promised ver.2 will be posted in a few weeks.
Your patch 12/12 will come with it.

Thanks,
Naoya Horiguchi

[toc] | [prev] | [next] | [standalone]

#1493848 — Re: [PATCH v1 00/12] THP migration support

[Multipart message — attachments visible in raw view] — view raw

>
> Thanks for helping,

:)

>
> I think that you seem to do some testing with these patches on powerpc,
> which shows that thp migration can be enabled relatively easily for
> non-x86_64. This is a good news to me.

Right. I did some THP migration tests on both x86_64 and IBM ppc64.

You can use the code here to test the THP migration,
and compare the migration time between 512 base pages and 1 THP.
https://github.com/x-y-z/thp-migration-bench

NUMA (or fake NUMA) setup and libnuma are needed. Since it simply tries to
migrate pages from node 0 to node 1.

make bench should give you the result like:

THP Migration
Total time: 676.870346 us
Test successful.
-------------------
Base Page Migration
Total time: 2340.078354 us
Test successful.

>
> And I apology for my slow development over this patchset.
> My previous post was about 5 months ago, and I've not done ver.2 due to
> many interruptions. Someone also privately asked me about the progress
> of this work, so I promised ver.2 will be posted in a few weeks.
> Your patch 12/12 will come with it.

Looking forward to it. :)

—
Best Regards,
Yan Zi

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web