Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1527672 > unrolled thread

[PATCH 0/5] Parallel hugepage migration optimization

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/5] Parallel hugepage migration optimization Zi Yan <zi.yan@sent.com> - 2016-11-22 17:30 +0100
    [PATCH 1/5] mm: migrate: Add mode parameter to support additional page copy routines. Zi Yan <zi.yan@sent.com> - 2016-11-22 17:30 +0100
      Re: [PATCH 1/5] mm: migrate: Add mode parameter to support  additional page copy routines. kbuild test robot <lkp@intel.com> - 2016-11-22 21:00 +0100
        Re: [PATCH 1/5] mm: migrate: Add mode parameter to support additional  page copy routines. Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-11-24 07:50 +0100
      Re: [PATCH 1/5] mm: migrate: Add mode parameter to support additional  page copy routines. Anshuman Khandual <khandual@linux.vnet.ibm.com> - 2016-11-24 08:00 +0100
      Re: [PATCH 1/5] mm: migrate: Add mode parameter to support additional  page copy routines. Balbir Singh <bsingharora@gmail.com> - 2016-11-25 01:00 +0100
    Re: [PATCH 0/5] Parallel hugepage migration optimization Balbir Singh <bsingharora@gmail.com> - 2016-11-25 01:00 +0100

#1527672 — [PATCH 0/5] Parallel hugepage migration optimization

From: Zi Yan <zi.yan@cs.rutgers.edu>

Hi all,

This patchset boosts the hugepage migration throughput and helps THP migration
which is added by Naoya's patches: https://lwn.net/Articles/705879/.

Motivation
===============================

In x86, 4KB page migrations are underutilizing the memory bandwidth compared
to 2MB THP migrations. I did some page migration benchmarking on a two-socket
Intel Xeon E5-2640v3 box, which has 23.4GB/s bandwidth, and discover
there are big throughput gap, ~3x, between 4KB and 2MB page migrations.

Here are the throughput numbers for different page sizes and page numbers:
        | 512 4KB pages | 1 2MB THP  |  1 4KB page
x86_64  |  0.98GB/s     |  2.97GB/s  |   0.06GB/s

As Linux currently use single-threaded page migration, the throughput is still
much lower than the hardware bandwidth, 2.97GB/s vs 23.4GB/s. So I parallelize
the copy_page() part of THP migration with workqueue and achieve 2.8x throughput.

Here are the throughput numbers of 2MB page migration:
           |  single-threaded   | 8-thread
x86_64 2MB |    2.97GB/s        | 8.58GB/s

Here is the benchmark you can use to compare page migration time:
https://github.com/x-y-z/thp-migration-bench

As this patchset requires Naoya's patch, this repo has both patchset applied:
https://github.com/x-y-z/linux-thp-migration/tree/page_migration_opt_upstream


Patchset desciption
===============================

This patchset adds a new migrate_mode MIGRATE_MT, which leads to parallelized
page migration routine. Only copy_huge_page() will be parallelized. This
MIGRATE_MT is enabled by a sysctl knob, vm.accel_page_copy, or an additional
flag, MPOL_MF_MOVE_MT, to move_pages() system call.

The parallelized copy page routine distributes a single huge page into 4 
workqueue threads and wait until they finish.

Discussion
===============================
1. For testing purpose, I choose to use sysctl to enable and disable the
parallel huge page migration. I need comments on how to enable and disable it,
or just enable it for all huge page migrations.

2. The hard-coded "4" workqueue threads is not adaptive, any suggestion?
Like boot time benchmark to find an appropriate number?

3. The parallel huge page migration works best with threads allocated at 
different physical cores, not all in the same hyper-threaded core. Is there
any way to find out the core topology easily?


Any comments are welcome. Thanks.

--
Best Regards,
Zi Yan


Zi Yan (5):
  mm: migrate: Add mode parameter to support additional page copy
    routines.
  mm: migrate: Change migrate_mode to support combination migration
    modes.
  migrate: Add copy_page_mt to use multi-threaded page migration.
  mm: migrate: Add copy_page_mt into migrate_pages.
  mm: migrate: Add vm.accel_page_copy in sysfs to control whether to use
    multi-threaded to accelerate page copy.

 fs/aio.c                       |  2 +-
 fs/hugetlbfs/inode.c           |  2 +-
 fs/ubifs/file.c                |  2 +-
 include/linux/highmem.h        |  2 +
 include/linux/migrate.h        |  6 ++-
 include/linux/migrate_mode.h   |  7 +--
 include/uapi/linux/mempolicy.h |  2 +
 kernel/sysctl.c                | 12 ++++++
 mm/Makefile                    |  2 +
 mm/compaction.c                | 20 ++++-----
 mm/copy_page.c                 | 96 ++++++++++++++++++++++++++++++++++++++++++
 mm/migrate.c                   | 61 ++++++++++++++++++---------
 12 files changed, 175 insertions(+), 39 deletions(-)
 create mode 100644 mm/copy_page.c

-- 
2.10.2

[toc] | [next] | [standalone]

#1527674 — [PATCH 1/5] mm: migrate: Add mode parameter to support additional page copy routines.

From: Zi Yan <zi.yan@cs.rutgers.edu>

From: Zi Yan <ziy@nvidia.com>

migrate_page_copy() and copy_huge_page() are affected.

Signed-off-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
---
 fs/aio.c                |  2 +-
 fs/hugetlbfs/inode.c    |  2 +-
 fs/ubifs/file.c         |  2 +-
 include/linux/migrate.h |  6 ++++--
 mm/migrate.c            | 14 ++++++++------
 5 files changed, 15 insertions(+), 11 deletions(-)

diff --git a/fs/aio.c b/fs/aio.c
index 428484f..a67c764 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -418,7 +418,7 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
 	 * events from being lost.
 	 */
 	spin_lock_irqsave(&ctx->completion_lock, flags);
-	migrate_page_copy(new, old);
+	migrate_page_copy(new, old, 0);
 	BUG_ON(ctx->ring_pages[idx] != old);
 	ctx->ring_pages[idx] = new;
 	spin_unlock_irqrestore(&ctx->completion_lock, flags);
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 4fb7b10..a17bfef 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -850,7 +850,7 @@ static int hugetlbfs_migrate_page(struct address_space *mapping,
 	rc = migrate_huge_page_move_mapping(mapping, newpage, page);
 	if (rc != MIGRATEPAGE_SUCCESS)
 		return rc;
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, 0);
 
 	return MIGRATEPAGE_SUCCESS;
 }
diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c
index b4fbeef..bf54e32 100644
--- a/fs/ubifs/file.c
+++ b/fs/ubifs/file.c
@@ -1468,7 +1468,7 @@ static int ubifs_migrate_page(struct address_space *mapping,
 		SetPagePrivate(newpage);
 	}
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, 0);
 	return MIGRATEPAGE_SUCCESS;
 }
 #endif
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index ae8d475..c78593d 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -42,7 +42,8 @@ extern void putback_movable_page(struct page *page);
 
 extern int migrate_prep(void);
 extern int migrate_prep_local(void);
-extern void migrate_page_copy(struct page *newpage, struct page *page);
+extern void migrate_page_copy(struct page *newpage, struct page *page,
+				  enum migrate_mode mode);
 extern int migrate_huge_page_move_mapping(struct address_space *mapping,
 				  struct page *newpage, struct page *page);
 extern int migrate_page_move_mapping(struct address_space *mapping,
@@ -61,7 +62,8 @@ static inline int migrate_prep(void) { return -ENOSYS; }
 static inline int migrate_prep_local(void) { return -ENOSYS; }
 
 static inline void migrate_page_copy(struct page *newpage,
-				     struct page *page) {}
+				     struct page *page,
+				     enum migrate_mode mode) {}
 
 static inline int migrate_huge_page_move_mapping(struct address_space *mapping,
 				  struct page *newpage, struct page *page)
diff --git a/mm/migrate.c b/mm/migrate.c
index 5bd202c..bc6c1c4 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -629,7 +629,8 @@ static void __copy_gigantic_page(struct page *dst, struct page *src,
 	}
 }
 
-static void copy_huge_page(struct page *dst, struct page *src)
+static void copy_huge_page(struct page *dst, struct page *src,
+				enum migrate_mode mode)
 {
 	int i;
 	int nr_pages;
@@ -658,12 +659,13 @@ static void copy_huge_page(struct page *dst, struct page *src)
 /*
  * Copy the page to its new location
  */
-void migrate_page_copy(struct page *newpage, struct page *page)
+void migrate_page_copy(struct page *newpage, struct page *page,
+					   enum migrate_mode mode)
 {
 	int cpupid;
 
 	if (PageHuge(page) || PageTransHuge(page))
-		copy_huge_page(newpage, page);
+		copy_huge_page(newpage, page, mode);
 	else
 		copy_highpage(newpage, page);
 
@@ -745,7 +747,7 @@ int migrate_page(struct address_space *mapping,
 	if (rc != MIGRATEPAGE_SUCCESS)
 		return rc;
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, mode);
 	return MIGRATEPAGE_SUCCESS;
 }
 EXPORT_SYMBOL(migrate_page);
@@ -795,7 +797,7 @@ int buffer_migrate_page(struct address_space *mapping,
 
 	SetPagePrivate(newpage);
 
-	migrate_page_copy(newpage, page);
+	migrate_page_copy(newpage, page, 0);
 
 	bh = head;
 	do {
@@ -2020,7 +2022,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
 	/* anon mapping, we can simply copy page->mapping to the new page: */
 	new_page->mapping = page->mapping;
 	new_page->index = page->index;
-	migrate_page_copy(new_page, page);
+	migrate_page_copy(new_page, page, 0);
 	WARN_ON(PageLRU(new_page));
 
 	/* Recheck the target PMD */
-- 
2.10.2

[toc] | [prev] | [next] | [standalone]

#1527882 — Re: [PATCH 1/5] mm: migrate: Add mode parameter to support additional page copy routines.

Hi Zi,

[auto build test WARNING on linus/master]
[also build test WARNING on v4.9-rc6 next-20161122]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Zi-Yan/Parallel-hugepage-migration-optimization/20161123-022913
reproduce:
        # apt-get install sparse
        make ARCH=x86_64 allmodconfig
        make C=1 CF=-D__CHECK_ENDIAN__


sparse warnings: (new ones prefixed by >>)

   include/linux/compiler.h:253:8: sparse: attribute 'no_sanitize_address': unknown attribute
>> fs/f2fs/data.c:1938:26: sparse: not enough arguments for function migrate_page_copy
   fs/f2fs/data.c: In function 'f2fs_migrate_page':
   fs/f2fs/data.c:1938:2: error: too few arguments to function 'migrate_page_copy'
     migrate_page_copy(newpage, page);
     ^~~~~~~~~~~~~~~~~
   In file included from fs/f2fs/data.c:1893:0:
   include/linux/migrate.h:45:13: note: declared here
    extern void migrate_page_copy(struct page *newpage, struct page *page,
                ^~~~~~~~~~~~~~~~~

vim +1938 fs/f2fs/data.c

5b7a487c Weichao Guo 2016-09-20  1922  	if (atomic_written) {
5b7a487c Weichao Guo 2016-09-20  1923  		struct inmem_pages *cur;
5b7a487c Weichao Guo 2016-09-20  1924  		list_for_each_entry(cur, &fi->inmem_pages, list)
5b7a487c Weichao Guo 2016-09-20  1925  			if (cur->page == page) {
5b7a487c Weichao Guo 2016-09-20  1926  				cur->page = newpage;
5b7a487c Weichao Guo 2016-09-20  1927  				break;
5b7a487c Weichao Guo 2016-09-20  1928  			}
5b7a487c Weichao Guo 2016-09-20  1929  		mutex_unlock(&fi->inmem_lock);
5b7a487c Weichao Guo 2016-09-20  1930  		put_page(page);
5b7a487c Weichao Guo 2016-09-20  1931  		get_page(newpage);
5b7a487c Weichao Guo 2016-09-20  1932  	}
5b7a487c Weichao Guo 2016-09-20  1933  
5b7a487c Weichao Guo 2016-09-20  1934  	if (PagePrivate(page))
5b7a487c Weichao Guo 2016-09-20  1935  		SetPagePrivate(newpage);
5b7a487c Weichao Guo 2016-09-20  1936  	set_page_private(newpage, page_private(page));
5b7a487c Weichao Guo 2016-09-20  1937  
5b7a487c Weichao Guo 2016-09-20 @1938  	migrate_page_copy(newpage, page);
5b7a487c Weichao Guo 2016-09-20  1939  
5b7a487c Weichao Guo 2016-09-20  1940  	return MIGRATEPAGE_SUCCESS;
5b7a487c Weichao Guo 2016-09-20  1941  }
5b7a487c Weichao Guo 2016-09-20  1942  #endif
5b7a487c Weichao Guo 2016-09-20  1943  
eb47b800 Jaegeuk Kim 2012-11-02  1944  const struct address_space_operations f2fs_dblock_aops = {
eb47b800 Jaegeuk Kim 2012-11-02  1945  	.readpage	= f2fs_read_data_page,
eb47b800 Jaegeuk Kim 2012-11-02  1946  	.readpages	= f2fs_read_data_pages,

:::::: The code at line 1938 was first introduced by commit
:::::: 5b7a487cf32d3a266fea83d590d3226b5ad817a7 f2fs: add customized migrate_page callback

:::::: TO: Weichao Guo <guoweichao@huawei.com>
:::::: CC: Jaegeuk Kim <jaegeuk@kernel.org>

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[toc] | [prev] | [next] | [standalone]

#1529029 — Re: [PATCH 1/5] mm: migrate: Add mode parameter to support additional page copy routines.

On 11/23/2016 01:26 AM, kbuild test robot wrote:
> Hi Zi,
> 
> [auto build test WARNING on linus/master]
> [also build test WARNING on v4.9-rc6 next-20161122]
> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
> 
> url:    https://github.com/0day-ci/linux/commits/Zi-Yan/Parallel-hugepage-migration-optimization/20161123-022913
> reproduce:
>         # apt-get install sparse
>         make ARCH=x86_64 allmodconfig
>         make C=1 CF=-D__CHECK_ENDIAN__
> 
> 
> sparse warnings: (new ones prefixed by >>)
> 
>    include/linux/compiler.h:253:8: sparse: attribute 'no_sanitize_address': unknown attribute
>>> >> fs/f2fs/data.c:1938:26: sparse: not enough arguments for function migrate_page_copy
>    fs/f2fs/data.c: In function 'f2fs_migrate_page':
>    fs/f2fs/data.c:1938:2: error: too few arguments to function 'migrate_page_copy'
>      migrate_page_copy(newpage, page);

Yeah, this got missed which needs to be fixed.

[toc] | [prev] | [next] | [standalone]

#1529031 — Re: [PATCH 1/5] mm: migrate: Add mode parameter to support additional page copy routines.

On 11/22/2016 09:55 PM, Zi Yan wrote:
> From: Zi Yan <zi.yan@cs.rutgers.edu>
> 
> From: Zi Yan <ziy@nvidia.com>

There are multiple "from" for this patch, should be fixed to reflect
just one of them.

> 
> migrate_page_copy() and copy_huge_page() are affected.

In this patch you are just expanding the arguments of both of these
functions to include "migration mode" which will then be implemented
later by subsequent patches or should these patches me merged ? I
guess its okay.

Please reword the commit message to include details.

> 
> Signed-off-by: Zi Yan <ziy@nvidia.com>
> Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>

Just a singled Signed-off-by.

[toc] | [prev] | [next] | [standalone]

#1529748 — Re: [PATCH 1/5] mm: migrate: Add mode parameter to support additional page copy routines.

On 23/11/16 03:25, Zi Yan wrote:
> From: Zi Yan <zi.yan@cs.rutgers.edu>
> 
> From: Zi Yan <ziy@nvidia.com>
> 
> migrate_page_copy() and copy_huge_page() are affected.
> 
> Signed-off-by: Zi Yan <ziy@nvidia.com>
> Signed-off-by: Zi Yan <zi.yan@cs.rutgers.edu>
> ---
>  fs/aio.c                |  2 +-
>  fs/hugetlbfs/inode.c    |  2 +-
>  fs/ubifs/file.c         |  2 +-
>  include/linux/migrate.h |  6 ++++--
>  mm/migrate.c            | 14 ++++++++------
>  5 files changed, 15 insertions(+), 11 deletions(-)
> 
> diff --git a/fs/aio.c b/fs/aio.c
> index 428484f..a67c764 100644
> --- a/fs/aio.c
> +++ b/fs/aio.c
> @@ -418,7 +418,7 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,
>  	 * events from being lost.
>  	 */
>  	spin_lock_irqsave(&ctx->completion_lock, flags);
> -	migrate_page_copy(new, old);
> +	migrate_page_copy(new, old, 0);

Can we have a useful enum instead of 0, its harder to read and understand

0

>  	BUG_ON(ctx->ring_pages[idx] != old);
>  	ctx->ring_pages[idx] = new;
>  	spin_unlock_irqrestore(&ctx->completion_lock, flags);
> diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
> index 4fb7b10..a17bfef 100644
> --- a/fs/hugetlbfs/inode.c
> +++ b/fs/hugetlbfs/inode.c
> @@ -850,7 +850,7 @@ static int hugetlbfs_migrate_page(struct address_space *mapping,
>  	rc = migrate_huge_page_move_mapping(mapping, newpage, page);
>  	if (rc != MIGRATEPAGE_SUCCESS)
>  		return rc;
> -	migrate_page_copy(newpage, page);
> +	migrate_page_copy(newpage, page, 0);

Ditto

>  
>  	return MIGRATEPAGE_SUCCESS;
>  }
> diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c
> index b4fbeef..bf54e32 100644
> --- a/fs/ubifs/file.c
> +++ b/fs/ubifs/file.c
> @@ -1468,7 +1468,7 @@ static int ubifs_migrate_page(struct address_space *mapping,
>  		SetPagePrivate(newpage);
>  	}
>  
> -	migrate_page_copy(newpage, page);
> +	migrate_page_copy(newpage, page, 0);

Here as well


Balbir Singh.

[toc] | [prev] | [next] | [standalone]

#1529747

On 23/11/16 03:25, Zi Yan wrote:
> From: Zi Yan <zi.yan@cs.rutgers.edu>
> 
> Hi all,
> 
> This patchset boosts the hugepage migration throughput and helps THP migration
> which is added by Naoya's patches: https://lwn.net/Articles/705879/.
> 
> Motivation
> ===============================
> 
> In x86, 4KB page migrations are underutilizing the memory bandwidth compared
> to 2MB THP migrations. I did some page migration benchmarking on a two-socket
> Intel Xeon E5-2640v3 box, which has 23.4GB/s bandwidth, and discover
> there are big throughput gap, ~3x, between 4KB and 2MB page migrations.
> 
> Here are the throughput numbers for different page sizes and page numbers:
>         | 512 4KB pages | 1 2MB THP  |  1 4KB page
> x86_64  |  0.98GB/s     |  2.97GB/s  |   0.06GB/s
> 
> As Linux currently use single-threaded page migration, the throughput is still
> much lower than the hardware bandwidth, 2.97GB/s vs 23.4GB/s. So I parallelize
> the copy_page() part of THP migration with workqueue and achieve 2.8x throughput.
> 
> Here are the throughput numbers of 2MB page migration:
>            |  single-threaded   | 8-thread
> x86_64 2MB |    2.97GB/s        | 8.58GB/s
> 

Whats the impact on CPU utilization? Is there a huge impact?

Balbir Singh.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web