Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1380432 > unrolled thread

[PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages

Started by"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
First post2016-04-16 02:40 +0200
Last post2016-04-20 10:40 +0200
Articles 14 — 7 participants

Back to article view | Back to linux.kernel

Contents

  [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-04-16 02:40 +0200
    [PATCHv7 07/29] thp, vmstats: add counters for huge file pages "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-04-16 02:40 +0200
    [PATCHv7 14/29] thp: file pages support for split_huge_page() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-04-16 02:40 +0200
    Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages "Shi, Yang" <yang.shi@linaro.org> - 2016-04-19 01:00 +0200
      Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages Jerome Marchand <jmarchan@redhat.com> - 2016-04-19 16:40 +0200
        Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages "Shi, Yang" <yang.shi@linaro.org> - 2016-04-19 18:20 +0200
      Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages Andrea Arcangeli <aarcange@redhat.com> - 2016-04-19 19:00 +0200
        Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages Andres Lagar-Cavilla <andreslc@google.com> - 2016-04-19 19:10 +0200
          Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages Wincy Van <fanwenyi0529@gmail.com> - 2016-04-24 07:50 +0200
            Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages Andres Lagar-Cavilla <andreslc@google.com> - 2016-04-25 15:40 +0200
              Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages Wincy Van <fanwenyi0529@gmail.com> - 2016-04-26 16:10 +0200
          Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages Andrea Arcangeli <aarcange@redhat.com> - 2016-04-27 17:50 +0200
        Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages "Shi, Yang" <yang.shi@linaro.org> - 2016-04-20 01:50 +0200
      Re: [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages Hugh Dickins <hughd@google.com> - 2016-04-20 10:40 +0200

#1380432 — [PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages

From"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Date2016-04-16 02:40 +0200
Subject[PATCHv7 00/29] THP-enabled tmpfs/shmem using compound pages
Message-ID<romhz-3cc-3@gated-at.bofh.it>
This is probably the last update before the mm summit. Main forcus is on
khugepaged stability.

khugepaged is in more reasonable shape now. I missed quite a few corner
cases on first try. I run this version via LTP, trinity and syzkaller
without crashes so far.

The patchset is on top of v4.6-rc3 plus Hugh's "easy preliminaries to
THPagecache" and Ebru's khugepaged swapin patches form -mm tree.

Git tree:

git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git hugetmpfs/v7

== Changelog ==

v7:
  - khugepaged updates:
    + fix page leak/page cache corruption on collapse fail;
    + filter out VMAs not suitable for huge pages due misaligned vm_pgoff;
    + fix build without CONFIG_SHMEM;
    + drop few over-protective checks;
  - fix bogus VM_BUG_ON() in __delete_from_page_cache();

v6:
  - experimental collapse support;
  - fix swapout mapped huge pages;
  - fix page leak in faularound code;
  - fix exessive huge page allocation with huge=within_size;
  - rename VM_NO_THP to VM_NO_KHUGEPAGED;
  - fix condition in hugepage_madvise();
  - accounting reworked again;

v5:
  - add FileHugeMapped to /proc/PID/smaps;
  - make FileHugeMapped in meminfo aligned with other fields;
  - Documentation/vm/transhuge.txt updated;

v4:
  - first four patch were applied to -mm tree;
  - drop pages beyond i_size on split_huge_pages;
  - few small random bugfixes;

v3:
  - huge= mountoption now can have values always, within_size, advice and
    never;
  - sysctl handle is replaced with sysfs knob;
  - MADV_HUGEPAGE/MADV_NOHUGEPAGE is now respected on page allocation via
    page fault;
  - mlock() handling had been fixed;
  - bunch of smaller bugfixes and cleanups.

== Design overview ==

Huge pages are allocated by shmem when it's allowed (by mount option) and
there's no entries for the range in radix-tree. Huge page is represented by
HPAGE_PMD_NR entries in radix-tree.

MM core maps a page with PMD if ->fault() returns huge page and the VMA is
suitable for huge pages (size, alignment). There's no need into two
requests to file system: filesystem returns huge page if it can,
graceful fallback to small pages otherwise.

As with DAX, split_huge_pmd() is implemented by unmapping the PMD: we can
re-fault the page with PTEs later.

Basic scheme for split_huge_page() is the same as for anon-THP.
Few differences:

  - File pages are on radix-tree, so we have head->_count offset by
    HPAGE_PMD_NR. The count got distributed to small pages during split.

  - mapping->tree_lock prevents non-lockless access to pages under split
    over radix-tree;

  - Lockless access is prevented by setting the head->_count to 0 during
    split, so get_page_unless_zero() would fail;

  - After split, some pages can be beyond i_size. We drop them from
    radix-tree.

  - We don't setup migration entries. Just unmap pages. It helps
    handling cases when i_size is in the middle of the page: no need
    handle unmap pages beyond i_size manually.

COW mapping handled on PTE-level. It's not clear how beneficial would be
allocation of huge pages on COW faults. And it would require some code to
make them work.

I think at some point we can consider teaching khugepaged to collapse
pages in COW mappings, but allocating huge on fault is probably overkill.

As with anon THP, we mlock file huge page only if it mapped with PMD.
PTE-mapped THPs are never mlocked. This way we can avoid all sorts of
scenarios when we can leak mlocked page.

As with anon THP, we split huge page on swap out.

Truncate and punch hole that only cover part of THP range is implemented
by zero out this part of THP.

This have visible effect on fallocate(FALLOC_FL_PUNCH_HOLE) behaviour.
As we don't really create hole in this case, lseek(SEEK_HOLE) may have
inconsistent results depending what pages happened to be allocated.
I don't think this will be a problem.

== Patchset overview ==

[01/29]
	Update documentation on THP vs. mlock. I've posted it separately
	before. It can go in.

[02-04/29]
        Rework fault path and rmap to handle file pmd. Unlike DAX with
        vm_ops->pmd_fault, we don't need to ask filesystem twice -- first
        for huge page and then for small. If ->fault happened to return
        huge page and VMA is suitable for mapping it as huge, we would
	do so.
[05/29]
	Add support for huge file pages in rmap;

[06-15/29]
        Various preparation of THP core for file pages.

[16-20/29]
        Various preparation of MM core for file pages.

[21-24/29]
        And finally, bring huge pages into tmpfs/shmem.

[25/29]
	Wire up madvise() existing hints for file THP.
	We can implement fadvise() later.

[26/29]
	Documentation update.

[27-29/29]
	Extend khugepaged to support shmem/tmpfs.
Hugh Dickins (1):
  shmem: get_unmapped_area align huge page

Kirill A. Shutemov (28):
  thp, mlock: update unevictable-lru.txt
  mm: do not pass mm_struct into handle_mm_fault
  mm: introduce fault_env
  mm: postpone page table allocation until we have page to map
  rmap: support file thp
  mm: introduce do_set_pmd()
  thp, vmstats: add counters for huge file pages
  thp: support file pages in zap_huge_pmd()
  thp: handle file pages in split_huge_pmd()
  thp: handle file COW faults
  thp: skip file huge pmd on copy_huge_pmd()
  thp: prepare change_huge_pmd() for file thp
  thp: run vma_adjust_trans_huge() outside i_mmap_rwsem
  thp: file pages support for split_huge_page()
  thp, mlock: do not mlock PTE-mapped file huge pages
  vmscan: split file huge pages before paging them out
  page-flags: relax policy for PG_mappedtodisk and PG_reclaim
  radix-tree: implement radix_tree_maybe_preload_order()
  filemap: prepare find and delete operations for huge pages
  truncate: handle file thp
  mm, rmap: account shmem thp pages
  shmem: prepare huge= mount option and sysfs knob
  shmem: add huge pages support
  shmem, thp: respect MADV_{NO,}HUGEPAGE for file mappings
  thp: update Documentation/vm/transhuge.txt
  thp: extract khugepaged from mm/huge_memory.c
  khugepaged: move up_read(mmap_sem) out of khugepaged_alloc_page()
  khugepaged: add support of collapse for tmpfs/shmem pages

 Documentation/filesystems/Locking    |   10 +-
 Documentation/vm/transhuge.txt       |  130 ++-
 Documentation/vm/unevictable-lru.txt |   21 +
 arch/alpha/mm/fault.c                |    2 +-
 arch/arc/mm/fault.c                  |    2 +-
 arch/arm/mm/fault.c                  |    2 +-
 arch/arm64/mm/fault.c                |    2 +-
 arch/avr32/mm/fault.c                |    2 +-
 arch/cris/mm/fault.c                 |    2 +-
 arch/frv/mm/fault.c                  |    2 +-
 arch/hexagon/mm/vm_fault.c           |    2 +-
 arch/ia64/mm/fault.c                 |    2 +-
 arch/m32r/mm/fault.c                 |    2 +-
 arch/m68k/mm/fault.c                 |    2 +-
 arch/metag/mm/fault.c                |    2 +-
 arch/microblaze/mm/fault.c           |    2 +-
 arch/mips/mm/fault.c                 |    2 +-
 arch/mn10300/mm/fault.c              |    2 +-
 arch/nios2/mm/fault.c                |    2 +-
 arch/openrisc/mm/fault.c             |    2 +-
 arch/parisc/mm/fault.c               |    2 +-
 arch/powerpc/mm/copro_fault.c        |    2 +-
 arch/powerpc/mm/fault.c              |    2 +-
 arch/s390/mm/fault.c                 |    2 +-
 arch/score/mm/fault.c                |    2 +-
 arch/sh/mm/fault.c                   |    2 +-
 arch/sparc/mm/fault_32.c             |    4 +-
 arch/sparc/mm/fault_64.c             |    2 +-
 arch/tile/mm/fault.c                 |    2 +-
 arch/um/kernel/trap.c                |    2 +-
 arch/unicore32/mm/fault.c            |    2 +-
 arch/x86/mm/fault.c                  |    2 +-
 arch/xtensa/mm/fault.c               |    2 +-
 drivers/base/node.c                  |   13 +-
 drivers/char/mem.c                   |   24 +
 drivers/iommu/amd_iommu_v2.c         |    3 +-
 drivers/iommu/intel-svm.c            |    2 +-
 fs/proc/meminfo.c                    |    7 +-
 fs/proc/task_mmu.c                   |   10 +-
 fs/userfaultfd.c                     |   22 +-
 include/linux/huge_mm.h              |   36 +-
 include/linux/khugepaged.h           |    6 +
 include/linux/mm.h                   |   51 +-
 include/linux/mmzone.h               |    4 +-
 include/linux/page-flags.h           |   19 +-
 include/linux/radix-tree.h           |    1 +
 include/linux/rmap.h                 |    2 +-
 include/linux/shmem_fs.h             |   29 +-
 include/linux/userfaultfd_k.h        |    8 +-
 include/linux/vm_event_item.h        |    7 +
 include/trace/events/huge_memory.h   |    3 +-
 ipc/shm.c                            |    6 +-
 lib/radix-tree.c                     |   68 +-
 mm/Makefile                          |    2 +-
 mm/filemap.c                         |  226 ++--
 mm/gup.c                             |    7 +-
 mm/huge_memory.c                     | 2028 ++++++----------------------------
 mm/internal.h                        |    4 +-
 mm/khugepaged.c                      | 1772 +++++++++++++++++++++++++++++
 mm/ksm.c                             |    5 +-
 mm/memory.c                          |  859 +++++++-------
 mm/mempolicy.c                       |    4 +-
 mm/migrate.c                         |    5 +-
 mm/mmap.c                            |   26 +-
 mm/nommu.c                           |    3 +-
 mm/page-writeback.c                  |    1 +
 mm/page_alloc.c                      |   21 +
 mm/rmap.c                            |   78 +-
 mm/shmem.c                           |  689 ++++++++++--
 mm/swap.c                            |    2 +
 mm/truncate.c                        |   22 +-
 mm/util.c                            |    6 +
 mm/vmscan.c                          |    6 +
 mm/vmstat.c                          |    4 +
 74 files changed, 3919 insertions(+), 2395 deletions(-)
 create mode 100644 mm/khugepaged.c

-- 
2.8.0.rc3

[toc] | [next] | [standalone]

#1380434 — [PATCHv7 07/29] thp, vmstats: add counters for huge file pages

From"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Date2016-04-16 02:40 +0200
Subject[PATCHv7 07/29] thp, vmstats: add counters for huge file pages
Message-ID<romrh-3hf-43@gated-at.bofh.it>
In reply to#1380432
THP_FILE_ALLOC: how many times huge page was allocated and put page
cache.

THP_FILE_MAPPED: how many times file huge page was mapped.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 include/linux/vm_event_item.h | 7 +++++++
 mm/memory.c                   | 1 +
 mm/vmstat.c                   | 2 ++
 3 files changed, 10 insertions(+)

diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index ec084321fe09..42604173f122 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -70,6 +70,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		THP_FAULT_FALLBACK,
 		THP_COLLAPSE_ALLOC,
 		THP_COLLAPSE_ALLOC_FAILED,
+		THP_FILE_ALLOC,
+		THP_FILE_MAPPED,
 		THP_SPLIT_PAGE,
 		THP_SPLIT_PAGE_FAILED,
 		THP_DEFERRED_SPLIT_PAGE,
@@ -100,4 +102,9 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		NR_VM_EVENT_ITEMS
 };
 
+#ifndef CONFIG_TRANSPARENT_HUGEPAGE
+#define THP_FILE_ALLOC ({ BUILD_BUG(); 0; })
+#define THP_FILE_MAPPED ({ BUILD_BUG(); 0; })
+#endif
+
 #endif		/* VM_EVENT_ITEM_H_INCLUDED */
diff --git a/mm/memory.c b/mm/memory.c
index ca45e9b19ad9..23de0567db18 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2907,6 +2907,7 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
 
 	/* fault is handled */
 	ret = 0;
+	count_vm_event(THP_FILE_MAPPED);
 out:
 	spin_unlock(fe->ptl);
 	return ret;
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 38ad49532b6f..904ac95fbf00 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -846,6 +846,8 @@ const char * const vmstat_text[] = {
 	"thp_fault_fallback",
 	"thp_collapse_alloc",
 	"thp_collapse_alloc_failed",
+	"thp_file_alloc",
+	"thp_file_mapped",
 	"thp_split_page",
 	"thp_split_page_failed",
 	"thp_deferred_split_page",
-- 
2.8.0.rc3

[toc] | [prev] | [next] | [standalone]

#1380435 — [PATCHv7 14/29] thp: file pages support for split_huge_page()

From"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Date2016-04-16 02:40 +0200
Subject[PATCHv7 14/29] thp: file pages support for split_huge_page()
Message-ID<romrh-3hf-35@gated-at.bofh.it>
In reply to#1380432
Basic scheme is the same as for anon THP.

Main differences:

  - File pages are on radix-tree, so we have head->_count offset by
    HPAGE_PMD_NR. The count got distributed to small pages during split.

  - mapping->tree_lock prevents non-lockless access to pages under split
    over radix-tree;

  - Lockless access is prevented by setting the head->_count to 0 during
    split;

  - After split, some pages can be beyond i_size. We drop them from
    radix-tree.

  - We don't setup migration entries. Just unmap pages. It helps
    handling cases when i_size is in the middle of the page: no need
    handle unmap pages beyond i_size manually.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/gup.c         |   2 +
 mm/huge_memory.c | 160 +++++++++++++++++++++++++++++++++++++++----------------
 mm/mempolicy.c   |   2 +
 3 files changed, 119 insertions(+), 45 deletions(-)

diff --git a/mm/gup.c b/mm/gup.c
index 39f751e1cb33..52f091e28f76 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -287,6 +287,8 @@ struct page *follow_page_mask(struct vm_area_struct *vma,
 			ret = split_huge_page(page);
 			unlock_page(page);
 			put_page(page);
+			if (pmd_none(*pmd))
+				return no_page_table(vma, flags);
 		}
 
 		return ret ? ERR_PTR(ret) :
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 5a71e0ffcf19..2e44741abeb6 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -31,6 +31,7 @@
 #include <linux/userfaultfd_k.h>
 #include <linux/page_idle.h>
 #include <linux/swapops.h>
+#include <linux/shmem_fs.h>
 
 #include <asm/tlb.h>
 #include <asm/pgalloc.h>
@@ -3143,12 +3144,15 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma,
 
 static void freeze_page(struct page *page)
 {
-	enum ttu_flags ttu_flags = TTU_MIGRATION | TTU_IGNORE_MLOCK |
-		TTU_IGNORE_ACCESS | TTU_RMAP_LOCKED;
+	enum ttu_flags ttu_flags = TTU_IGNORE_MLOCK | TTU_IGNORE_ACCESS |
+		TTU_RMAP_LOCKED;
 	int i, ret;
 
 	VM_BUG_ON_PAGE(!PageHead(page), page);
 
+	if (PageAnon(page))
+		ttu_flags |= TTU_MIGRATION;
+
 	/* We only need TTU_SPLIT_HUGE_PMD once */
 	ret = try_to_unmap(page, ttu_flags | TTU_SPLIT_HUGE_PMD);
 	for (i = 1; !ret && i < HPAGE_PMD_NR; i++) {
@@ -3158,7 +3162,7 @@ static void freeze_page(struct page *page)
 
 		ret = try_to_unmap(page + i, ttu_flags);
 	}
-	VM_BUG_ON(ret);
+	VM_BUG_ON_PAGE(ret, page + i - 1);
 }
 
 static void unfreeze_page(struct page *page)
@@ -3180,15 +3184,20 @@ static void __split_huge_page_tail(struct page *head, int tail,
 	/*
 	 * tail_page->_count is zero and not changing from under us. But
 	 * get_page_unless_zero() may be running from under us on the
-	 * tail_page. If we used atomic_set() below instead of atomic_inc(), we
-	 * would then run atomic_set() concurrently with
+	 * tail_page. If we used atomic_set() below instead of atomic_inc() or
+	 * atomic_add(), we would then run atomic_set() concurrently with
 	 * get_page_unless_zero(), and atomic_set() is implemented in C not
 	 * using locked ops. spin_unlock on x86 sometime uses locked ops
 	 * because of PPro errata 66, 92, so unless somebody can guarantee
 	 * atomic_set() here would be safe on all archs (and not only on x86),
-	 * it's safer to use atomic_inc().
+	 * it's safer to use atomic_inc()/atomic_add().
 	 */
-	page_ref_inc(page_tail);
+	if (PageAnon(head)) {
+		page_ref_inc(page_tail);
+	} else {
+		/* Additional pin to radix tree */
+		page_ref_add(page_tail, 2);
+	}
 
 	page_tail->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
 	page_tail->flags |= (head->flags &
@@ -3224,25 +3233,44 @@ static void __split_huge_page_tail(struct page *head, int tail,
 	lru_add_page_tail(head, page_tail, lruvec, list);
 }
 
-static void __split_huge_page(struct page *page, struct list_head *list)
+static void __split_huge_page(struct page *page, struct list_head *list,
+		unsigned long flags)
 {
 	struct page *head = compound_head(page);
 	struct zone *zone = page_zone(head);
 	struct lruvec *lruvec;
+	pgoff_t end = -1;
 	int i;
 
-	/* prevent PageLRU to go away from under us, and freeze lru stats */
-	spin_lock_irq(&zone->lru_lock);
 	lruvec = mem_cgroup_page_lruvec(head, zone);
 
 	/* complete memcg works before add pages to LRU */
 	mem_cgroup_split_huge_fixup(head);
 
-	for (i = HPAGE_PMD_NR - 1; i >= 1; i--)
+	if (!PageAnon(page))
+		end = DIV_ROUND_UP(i_size_read(head->mapping->host), PAGE_SIZE);
+
+	for (i = HPAGE_PMD_NR - 1; i >= 1; i--) {
 		__split_huge_page_tail(head, i, lruvec, list);
+		/* Some pages can be beyond i_size: drop them from page cache */
+		if (head[i].index >= end) {
+			__ClearPageDirty(head + i);
+			__delete_from_page_cache(head + i, NULL);
+			put_page(head + i);
+		}
+	}
 
 	ClearPageCompound(head);
-	spin_unlock_irq(&zone->lru_lock);
+	/* See comment in __split_huge_page_tail() */
+	if (PageAnon(head)) {
+		page_ref_inc(head);
+	} else {
+		/* Additional pin to radix tree */
+		page_ref_add(head, 2);
+		spin_unlock(&head->mapping->tree_lock);
+	}
+
+	spin_unlock_irqrestore(&page_zone(head)->lru_lock, flags);
 
 	unfreeze_page(head);
 
@@ -3309,36 +3337,54 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
 {
 	struct page *head = compound_head(page);
 	struct pglist_data *pgdata = NODE_DATA(page_to_nid(head));
-	struct anon_vma *anon_vma;
-	int count, mapcount, ret;
+	struct anon_vma *anon_vma = NULL;
+	struct address_space *mapping = NULL;
+	int count, mapcount, extra_pins, ret;
 	bool mlocked;
 	unsigned long flags;
 
 	VM_BUG_ON_PAGE(is_huge_zero_page(page), page);
-	VM_BUG_ON_PAGE(!PageAnon(page), page);
 	VM_BUG_ON_PAGE(!PageLocked(page), page);
 	VM_BUG_ON_PAGE(!PageSwapBacked(page), page);
 	VM_BUG_ON_PAGE(!PageCompound(page), page);
 
-	/*
-	 * The caller does not necessarily hold an mmap_sem that would prevent
-	 * the anon_vma disappearing so we first we take a reference to it
-	 * and then lock the anon_vma for write. This is similar to
-	 * page_lock_anon_vma_read except the write lock is taken to serialise
-	 * against parallel split or collapse operations.
-	 */
-	anon_vma = page_get_anon_vma(head);
-	if (!anon_vma) {
-		ret = -EBUSY;
-		goto out;
+	if (PageAnon(head)) {
+		/*
+		 * The caller does not necessarily hold an mmap_sem that would
+		 * prevent the anon_vma disappearing so we first we take a
+		 * reference to it and then lock the anon_vma for write. This
+		 * is similar to page_lock_anon_vma_read except the write lock
+		 * is taken to serialise against parallel split or collapse
+		 * operations.
+		 */
+		anon_vma = page_get_anon_vma(head);
+		if (!anon_vma) {
+			ret = -EBUSY;
+			goto out;
+		}
+		extra_pins = 0;
+		mapping = NULL;
+		anon_vma_lock_write(anon_vma);
+	} else {
+		mapping = head->mapping;
+
+		/* Truncated ? */
+		if (!mapping) {
+			ret = -EBUSY;
+			goto out;
+		}
+
+		/* Addidional pins from radix tree */
+		extra_pins = HPAGE_PMD_NR;
+		anon_vma = NULL;
+		i_mmap_lock_read(mapping);
 	}
-	anon_vma_lock_write(anon_vma);
 
 	/*
 	 * Racy check if we can split the page, before freeze_page() will
 	 * split PMDs
 	 */
-	if (total_mapcount(head) != page_count(head) - 1) {
+	if (total_mapcount(head) != page_count(head) - extra_pins - 1) {
 		ret = -EBUSY;
 		goto out_unlock;
 	}
@@ -3351,35 +3397,60 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
 	if (mlocked)
 		lru_add_drain();
 
+	/* prevent PageLRU to go away from under us, and freeze lru stats */
+	spin_lock_irqsave(&page_zone(head)->lru_lock, flags);
+
+	if (mapping) {
+		void **pslot;
+
+		spin_lock(&mapping->tree_lock);
+		pslot = radix_tree_lookup_slot(&mapping->page_tree,
+				page_index(head));
+		/*
+		 * Check if the head page is present in radix tree.
+		 * We assume all tail are present too, if head is there.
+		 */
+		if (radix_tree_deref_slot_protected(pslot,
+					&mapping->tree_lock) != head)
+			goto fail;
+	}
+
 	/* Prevent deferred_split_scan() touching ->_count */
-	spin_lock_irqsave(&pgdata->split_queue_lock, flags);
+	spin_lock(&pgdata->split_queue_lock);
 	count = page_count(head);
 	mapcount = total_mapcount(head);
-	if (!mapcount && count == 1) {
+	if (!mapcount && page_ref_freeze(head, 1 + extra_pins)) {
 		if (!list_empty(page_deferred_list(head))) {
 			pgdata->split_queue_len--;
 			list_del(page_deferred_list(head));
 		}
-		spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
-		__split_huge_page(page, list);
+		spin_unlock(&pgdata->split_queue_lock);
+		__split_huge_page(page, list, flags);
 		ret = 0;
-	} else if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) {
-		spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
-		pr_alert("total_mapcount: %u, page_count(): %u\n",
-				mapcount, count);
-		if (PageTail(page))
-			dump_page(head, NULL);
-		dump_page(page, "total_mapcount(head) > 0");
-		BUG();
 	} else {
-		spin_unlock_irqrestore(&pgdata->split_queue_lock, flags);
+		if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) {
+			pr_alert("total_mapcount: %u, page_count(): %u\n",
+					mapcount, count);
+			if (PageTail(page))
+				dump_page(head, NULL);
+			dump_page(page, "total_mapcount(head) > 0");
+			BUG();
+		}
+		spin_unlock(&pgdata->split_queue_lock);
+fail:		if (mapping)
+			spin_unlock(&mapping->tree_lock);
+		spin_unlock_irqrestore(&page_zone(head)->lru_lock, flags);
 		unfreeze_page(head);
 		ret = -EBUSY;
 	}
 
 out_unlock:
-	anon_vma_unlock_write(anon_vma);
-	put_anon_vma(anon_vma);
+	if (anon_vma) {
+		anon_vma_unlock_write(anon_vma);
+		put_anon_vma(anon_vma);
+	}
+	if (mapping)
+		i_mmap_unlock_read(mapping);
 out:
 	count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
 	return ret;
@@ -3502,8 +3573,7 @@ static int split_huge_pages_set(void *data, u64 val)
 			if (zone != page_zone(page))
 				goto next;
 
-			if (!PageHead(page) || !PageAnon(page) ||
-					PageHuge(page))
+			if (!PageHead(page) || PageHuge(page) || !PageLRU(page))
 				goto next;
 
 			total++;
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 36cc01bc950a..148143974c5b 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -515,6 +515,8 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,
 		}
 	}
 
+	if (pmd_trans_unstable(pmd))
+		return 0;
 retry:
 	pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
 	for (; addr != end; pte++, addr += PAGE_SIZE) {
-- 
2.8.0.rc3

[toc] | [prev] | [next] | [standalone]

#1382094

From"Shi, Yang" <yang.shi@linaro.org>
Date2016-04-19 01:00 +0200
Message-ID<rpqj8-52L-15@gated-at.bofh.it>
In reply to#1380432
Hi Kirill,

Finally, I got some time to look into and try yours and Hugh's patches, 
got two problems.

1. A quick boot up test on my ARM64 machine with your v7 tree shows some 
unexpected error:

systemd-journald[285]: Failed to save stream data 
/run/systemd/journal/streams/8:16863: No space left on device
systemd-journald[285]: Failed to save stream data 
/run/systemd/journal/streams/8:16865: No space left on device
          Starting DNS forwarder and DHCP server.systemd-journald[285]: 
Failed to save stream data /run/systemd/journal/streams/8:16867: No 
space left on device
..
systemd-journald[285]: Failed to save stream data 
/run/systemd/journal/streams/8:16869: No space left on device
          Starting Postfix Mail Transport Agent...
systemd-journald[285]: Failed to save stream data 
/run/systemd/journal/streams/8:16871: No space left on device
          Starting Berkeley Internet Name Domain (DNS)...
          Starting Wait for Network to be Configured...
systemd-journald[285]: Failed to save stream data 
/run/systemd/journal/streams/8:2422: No space left on device
[  OK  ] Started /etc/rc.local Compatibility.
[FAILED] Failed to start DNS forwarder and DHCP server.
See 'systemctl status dnsmasq.service' for details.
systemd-journald[285]: Failed to save stream data 
/run/systemd/journal/streams/8:2425: No space left on device
[  OK  ] Started Serial Getty on ttyS1.
[  OK  ] Started Serial Getty on ttyS0.
[  OK  ] Started Getty on tty1.
systemd-journald[285]: Failed to save stream data 
/run/systemd/journal/streams/8:2433: No space left on device
[FAILED] Failed to start Berkeley Internet Name Domain (DNS).
See 'systemctl status named.service' for details.


The /run dir is mounted as tmpfs.

x86 boot doesn't get such error. And, Hugh's patches don't have such 
problem.

2. I ran my THP test (generated a program with 4MB text section) on both 
x86-64 and ARM64 with yours and Hugh's patches (linux-next tree), I got 
the program execution time reduced by ~12% on x86-64, it looks very 
impressive.

But, on ARM64, there is just ~3% change, and sometimes huge tmpfs may 
show even worse data than non-hugepage.

Both yours and Hugh's patches has the same behavior.

Any idea?

Thanks,
Yang


On 4/15/2016 5:23 PM, Kirill A. Shutemov wrote:
> This is probably the last update before the mm summit. Main forcus is on
> khugepaged stability.
>
> khugepaged is in more reasonable shape now. I missed quite a few corner
> cases on first try. I run this version via LTP, trinity and syzkaller
> without crashes so far.
>
> The patchset is on top of v4.6-rc3 plus Hugh's "easy preliminaries to
> THPagecache" and Ebru's khugepaged swapin patches form -mm tree.
>
> Git tree:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git hugetmpfs/v7
>
> == Changelog ==
>
> v7:
>    - khugepaged updates:
>      + fix page leak/page cache corruption on collapse fail;
>      + filter out VMAs not suitable for huge pages due misaligned vm_pgoff;
>      + fix build without CONFIG_SHMEM;
>      + drop few over-protective checks;
>    - fix bogus VM_BUG_ON() in __delete_from_page_cache();
>
> v6:
>    - experimental collapse support;
>    - fix swapout mapped huge pages;
>    - fix page leak in faularound code;
>    - fix exessive huge page allocation with huge=within_size;
>    - rename VM_NO_THP to VM_NO_KHUGEPAGED;
>    - fix condition in hugepage_madvise();
>    - accounting reworked again;
>
> v5:
>    - add FileHugeMapped to /proc/PID/smaps;
>    - make FileHugeMapped in meminfo aligned with other fields;
>    - Documentation/vm/transhuge.txt updated;
>
> v4:
>    - first four patch were applied to -mm tree;
>    - drop pages beyond i_size on split_huge_pages;
>    - few small random bugfixes;
>
> v3:
>    - huge= mountoption now can have values always, within_size, advice and
>      never;
>    - sysctl handle is replaced with sysfs knob;
>    - MADV_HUGEPAGE/MADV_NOHUGEPAGE is now respected on page allocation via
>      page fault;
>    - mlock() handling had been fixed;
>    - bunch of smaller bugfixes and cleanups.
>
> == Design overview ==
>
> Huge pages are allocated by shmem when it's allowed (by mount option) and
> there's no entries for the range in radix-tree. Huge page is represented by
> HPAGE_PMD_NR entries in radix-tree.
>
> MM core maps a page with PMD if ->fault() returns huge page and the VMA is
> suitable for huge pages (size, alignment). There's no need into two
> requests to file system: filesystem returns huge page if it can,
> graceful fallback to small pages otherwise.
>
> As with DAX, split_huge_pmd() is implemented by unmapping the PMD: we can
> re-fault the page with PTEs later.
>
> Basic scheme for split_huge_page() is the same as for anon-THP.
> Few differences:
>
>    - File pages are on radix-tree, so we have head->_count offset by
>      HPAGE_PMD_NR. The count got distributed to small pages during split.
>
>    - mapping->tree_lock prevents non-lockless access to pages under split
>      over radix-tree;
>
>    - Lockless access is prevented by setting the head->_count to 0 during
>      split, so get_page_unless_zero() would fail;
>
>    - After split, some pages can be beyond i_size. We drop them from
>      radix-tree.
>
>    - We don't setup migration entries. Just unmap pages. It helps
>      handling cases when i_size is in the middle of the page: no need
>      handle unmap pages beyond i_size manually.
>
> COW mapping handled on PTE-level. It's not clear how beneficial would be
> allocation of huge pages on COW faults. And it would require some code to
> make them work.
>
> I think at some point we can consider teaching khugepaged to collapse
> pages in COW mappings, but allocating huge on fault is probably overkill.
>
> As with anon THP, we mlock file huge page only if it mapped with PMD.
> PTE-mapped THPs are never mlocked. This way we can avoid all sorts of
> scenarios when we can leak mlocked page.
>
> As with anon THP, we split huge page on swap out.
>
> Truncate and punch hole that only cover part of THP range is implemented
> by zero out this part of THP.
>
> This have visible effect on fallocate(FALLOC_FL_PUNCH_HOLE) behaviour.
> As we don't really create hole in this case, lseek(SEEK_HOLE) may have
> inconsistent results depending what pages happened to be allocated.
> I don't think this will be a problem.
>
> == Patchset overview ==
>
> [01/29]
> 	Update documentation on THP vs. mlock. I've posted it separately
> 	before. It can go in.
>
> [02-04/29]
>          Rework fault path and rmap to handle file pmd. Unlike DAX with
>          vm_ops->pmd_fault, we don't need to ask filesystem twice -- first
>          for huge page and then for small. If ->fault happened to return
>          huge page and VMA is suitable for mapping it as huge, we would
> 	do so.
> [05/29]
> 	Add support for huge file pages in rmap;
>
> [06-15/29]
>          Various preparation of THP core for file pages.
>
> [16-20/29]
>          Various preparation of MM core for file pages.
>
> [21-24/29]
>          And finally, bring huge pages into tmpfs/shmem.
>
> [25/29]
> 	Wire up madvise() existing hints for file THP.
> 	We can implement fadvise() later.
>
> [26/29]
> 	Documentation update.
>
> [27-29/29]
> 	Extend khugepaged to support shmem/tmpfs.
> Hugh Dickins (1):
>    shmem: get_unmapped_area align huge page
>
> Kirill A. Shutemov (28):
>    thp, mlock: update unevictable-lru.txt
>    mm: do not pass mm_struct into handle_mm_fault
>    mm: introduce fault_env
>    mm: postpone page table allocation until we have page to map
>    rmap: support file thp
>    mm: introduce do_set_pmd()
>    thp, vmstats: add counters for huge file pages
>    thp: support file pages in zap_huge_pmd()
>    thp: handle file pages in split_huge_pmd()
>    thp: handle file COW faults
>    thp: skip file huge pmd on copy_huge_pmd()
>    thp: prepare change_huge_pmd() for file thp
>    thp: run vma_adjust_trans_huge() outside i_mmap_rwsem
>    thp: file pages support for split_huge_page()
>    thp, mlock: do not mlock PTE-mapped file huge pages
>    vmscan: split file huge pages before paging them out
>    page-flags: relax policy for PG_mappedtodisk and PG_reclaim
>    radix-tree: implement radix_tree_maybe_preload_order()
>    filemap: prepare find and delete operations for huge pages
>    truncate: handle file thp
>    mm, rmap: account shmem thp pages
>    shmem: prepare huge= mount option and sysfs knob
>    shmem: add huge pages support
>    shmem, thp: respect MADV_{NO,}HUGEPAGE for file mappings
>    thp: update Documentation/vm/transhuge.txt
>    thp: extract khugepaged from mm/huge_memory.c
>    khugepaged: move up_read(mmap_sem) out of khugepaged_alloc_page()
>    khugepaged: add support of collapse for tmpfs/shmem pages
>
>   Documentation/filesystems/Locking    |   10 +-
>   Documentation/vm/transhuge.txt       |  130 ++-
>   Documentation/vm/unevictable-lru.txt |   21 +
>   arch/alpha/mm/fault.c                |    2 +-
>   arch/arc/mm/fault.c                  |    2 +-
>   arch/arm/mm/fault.c                  |    2 +-
>   arch/arm64/mm/fault.c                |    2 +-
>   arch/avr32/mm/fault.c                |    2 +-
>   arch/cris/mm/fault.c                 |    2 +-
>   arch/frv/mm/fault.c                  |    2 +-
>   arch/hexagon/mm/vm_fault.c           |    2 +-
>   arch/ia64/mm/fault.c                 |    2 +-
>   arch/m32r/mm/fault.c                 |    2 +-
>   arch/m68k/mm/fault.c                 |    2 +-
>   arch/metag/mm/fault.c                |    2 +-
>   arch/microblaze/mm/fault.c           |    2 +-
>   arch/mips/mm/fault.c                 |    2 +-
>   arch/mn10300/mm/fault.c              |    2 +-
>   arch/nios2/mm/fault.c                |    2 +-
>   arch/openrisc/mm/fault.c             |    2 +-
>   arch/parisc/mm/fault.c               |    2 +-
>   arch/powerpc/mm/copro_fault.c        |    2 +-
>   arch/powerpc/mm/fault.c              |    2 +-
>   arch/s390/mm/fault.c                 |    2 +-
>   arch/score/mm/fault.c                |    2 +-
>   arch/sh/mm/fault.c                   |    2 +-
>   arch/sparc/mm/fault_32.c             |    4 +-
>   arch/sparc/mm/fault_64.c             |    2 +-
>   arch/tile/mm/fault.c                 |    2 +-
>   arch/um/kernel/trap.c                |    2 +-
>   arch/unicore32/mm/fault.c            |    2 +-
>   arch/x86/mm/fault.c                  |    2 +-
>   arch/xtensa/mm/fault.c               |    2 +-
>   drivers/base/node.c                  |   13 +-
>   drivers/char/mem.c                   |   24 +
>   drivers/iommu/amd_iommu_v2.c         |    3 +-
>   drivers/iommu/intel-svm.c            |    2 +-
>   fs/proc/meminfo.c                    |    7 +-
>   fs/proc/task_mmu.c                   |   10 +-
>   fs/userfaultfd.c                     |   22 +-
>   include/linux/huge_mm.h              |   36 +-
>   include/linux/khugepaged.h           |    6 +
>   include/linux/mm.h                   |   51 +-
>   include/linux/mmzone.h               |    4 +-
>   include/linux/page-flags.h           |   19 +-
>   include/linux/radix-tree.h           |    1 +
>   include/linux/rmap.h                 |    2 +-
>   include/linux/shmem_fs.h             |   29 +-
>   include/linux/userfaultfd_k.h        |    8 +-
>   include/linux/vm_event_item.h        |    7 +
>   include/trace/events/huge_memory.h   |    3 +-
>   ipc/shm.c                            |    6 +-
>   lib/radix-tree.c                     |   68 +-
>   mm/Makefile                          |    2 +-
>   mm/filemap.c                         |  226 ++--
>   mm/gup.c                             |    7 +-
>   mm/huge_memory.c                     | 2028 ++++++----------------------------
>   mm/internal.h                        |    4 +-
>   mm/khugepaged.c                      | 1772 +++++++++++++++++++++++++++++
>   mm/ksm.c                             |    5 +-
>   mm/memory.c                          |  859 +++++++-------
>   mm/mempolicy.c                       |    4 +-
>   mm/migrate.c                         |    5 +-
>   mm/mmap.c                            |   26 +-
>   mm/nommu.c                           |    3 +-
>   mm/page-writeback.c                  |    1 +
>   mm/page_alloc.c                      |   21 +
>   mm/rmap.c                            |   78 +-
>   mm/shmem.c                           |  689 ++++++++++--
>   mm/swap.c                            |    2 +
>   mm/truncate.c                        |   22 +-
>   mm/util.c                            |    6 +
>   mm/vmscan.c                          |    6 +
>   mm/vmstat.c                          |    4 +
>   74 files changed, 3919 insertions(+), 2395 deletions(-)
>   create mode 100644 mm/khugepaged.c
>

[toc] | [prev] | [next] | [standalone]

#1382574

FromJerome Marchand <jmarchan@redhat.com>
Date2016-04-19 16:40 +0200
Message-ID<rpEYO-8k2-29@gated-at.bofh.it>
In reply to#1382094

[Multipart message — attachments visible in raw view] — view raw

On 04/19/2016 12:55 AM, Shi, Yang wrote:
> 2. I ran my THP test (generated a program with 4MB text section) on both
> x86-64 and ARM64 with yours and Hugh's patches (linux-next tree), I got
> the program execution time reduced by ~12% on x86-64, it looks very
> impressive.
> 
> But, on ARM64, there is just ~3% change, and sometimes huge tmpfs may
> show even worse data than non-hugepage.
> 
> Both yours and Hugh's patches has the same behavior.
> 
> Any idea?

Just a shot in the dark, but what page size do you use? If you use 4k
pages, then hugepage size should be the same as on x86 and a similar
behavior could be expected. Otherwise, hugepages would be too big to be
taken advantage of by your test program.

Jerome

[toc] | [prev] | [next] | [standalone]

#1382667

From"Shi, Yang" <yang.shi@linaro.org>
Date2016-04-19 18:20 +0200
Message-ID<rpGxz-1bm-1@gated-at.bofh.it>
In reply to#1382574
On 4/19/2016 7:33 AM, Jerome Marchand wrote:
> On 04/19/2016 12:55 AM, Shi, Yang wrote:
>> 2. I ran my THP test (generated a program with 4MB text section) on both
>> x86-64 and ARM64 with yours and Hugh's patches (linux-next tree), I got
>> the program execution time reduced by ~12% on x86-64, it looks very
>> impressive.
>>
>> But, on ARM64, there is just ~3% change, and sometimes huge tmpfs may
>> show even worse data than non-hugepage.
>>
>> Both yours and Hugh's patches has the same behavior.
>>
>> Any idea?
>
> Just a shot in the dark, but what page size do you use? If you use 4k
> pages, then hugepage size should be the same as on x86 and a similar

I do use 4K pages for both x86-64 and ARM64 in my testing.

Thanks,
Yang

> behavior could be expected. Otherwise, hugepages would be too big to be
> taken advantage of by your test program.
>
> Jerome
>

[toc] | [prev] | [next] | [standalone]

#1382706

FromAndrea Arcangeli <aarcange@redhat.com>
Date2016-04-19 19:00 +0200
Message-ID<rpHai-1sn-23@gated-at.bofh.it>
In reply to#1382094
Hello,

On Mon, Apr 18, 2016 at 03:55:44PM -0700, Shi, Yang wrote:
> Hi Kirill,
> 
> Finally, I got some time to look into and try yours and Hugh's patches, 
> got two problems.

One thing that come to mind to test is this: qemu with -machine
accel=kvm -mem-path=/dev/shm/,share=on .

The THP Compound approach in tmpfs may just happen to work already
with KVM (or at worst it'd require minor adjustments) because it uses
the exact same model KVM is already aware about from THP in anonymous
memory, example from arch/x86/kvm/mmu.c:

static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
					gfn_t *gfnp, kvm_pfn_t *pfnp,
					int *levelp)
{
	kvm_pfn_t pfn = *pfnp;
	gfn_t gfn = *gfnp;
	int level = *levelp;

	/*
	 * Check if it's a transparent hugepage. If this would be an
	 * hugetlbfs page, level wouldn't be set to
	 * PT_PAGE_TABLE_LEVEL and there would be no adjustment done
	 * here.
	 */
	if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) &&
	    level == PT_PAGE_TABLE_LEVEL &&
	    PageTransCompound(pfn_to_page(pfn)) &&
	    !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) {

Not using two different models between THP in tmpfs and THP in anon is
essential not just to significantly reduce the size of the kernel
code, but also because THP knowledge can't be self contained in the
mm/shmem.c file. Having to support two different models would
complicate things for secondary MMU drivers (i.e. mmu notifer users)
like KVM who also need to create huge mapping in the shadow pagetable
layer in arch/x86/kvm if the primary MMU allows for it.

> x86-64 and ARM64 with yours and Hugh's patches (linux-next tree), I got 
> the program execution time reduced by ~12% on x86-64, it looks very 
> impressive.

Agreed, both patchset are impressive works and achieving amazing
results!

My view is that in terms of long-lived computation from userland point
of view, both models are malleable enough and could achieve everything
we need in the end, but as far as the overall kernel efficiency is
concerned the compound model will always retain a slight advantage in
performance by leveraging a native THP compound refcounting that
requires just one atomic_inc/dec per THP mapcount instead of 512 of
them. Other advantages of the compound model is that it's half in code
size despite already including khugepaged (i.e. the same
split_huge_page works for both tmpfs and anon) and like said above it
won't introduce much complications for drivers like KVM as the model
didn't change.

Thanks,
Andrea

[toc] | [prev] | [next] | [standalone]

#1382716

FromAndres Lagar-Cavilla <andreslc@google.com>
Date2016-04-19 19:10 +0200
Message-ID<rpHjY-1Nv-17@gated-at.bofh.it>
In reply to#1382706
Andrea, we provide the, ahem, adjustments to
transparent_hugepage_adjust. Rest assured we aggressively use mmu
notifiers with no further changes required.

As in: zero changes have been required in the lifetime (years) of
kvm+huge tmpfs at Google, other than mod'ing
transparent_hugepage_adjust.

As noted by Paolo, the additions to transparent_hugepage_adjust could
be lifted outside of kvm (into shmem.c? maybe) for any consumer of
huge tmpfs with mmu notifiers.

Andres

On Tue, Apr 19, 2016 at 9:50 AM, Andrea Arcangeli <aarcange@redhat.com> wrote:
> Hello,
>
> On Mon, Apr 18, 2016 at 03:55:44PM -0700, Shi, Yang wrote:
>> Hi Kirill,
>>
>> Finally, I got some time to look into and try yours and Hugh's patches,
>> got two problems.
>
> One thing that come to mind to test is this: qemu with -machine
> accel=kvm -mem-path=/dev/shm/,share=on .
>
> The THP Compound approach in tmpfs may just happen to work already
> with KVM (or at worst it'd require minor adjustments) because it uses
> the exact same model KVM is already aware about from THP in anonymous
> memory, example from arch/x86/kvm/mmu.c:
>
> static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
>                                         gfn_t *gfnp, kvm_pfn_t *pfnp,
>                                         int *levelp)
> {
>         kvm_pfn_t pfn = *pfnp;
>         gfn_t gfn = *gfnp;
>         int level = *levelp;
>
>         /*
>          * Check if it's a transparent hugepage. If this would be an
>          * hugetlbfs page, level wouldn't be set to
>          * PT_PAGE_TABLE_LEVEL and there would be no adjustment done
>          * here.
>          */
>         if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) &&
>             level == PT_PAGE_TABLE_LEVEL &&
>             PageTransCompound(pfn_to_page(pfn)) &&
>             !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) {
>
> Not using two different models between THP in tmpfs and THP in anon is
> essential not just to significantly reduce the size of the kernel
> code, but also because THP knowledge can't be self contained in the
> mm/shmem.c file. Having to support two different models would
> complicate things for secondary MMU drivers (i.e. mmu notifer users)
> like KVM who also need to create huge mapping in the shadow pagetable
> layer in arch/x86/kvm if the primary MMU allows for it.
>
>> x86-64 and ARM64 with yours and Hugh's patches (linux-next tree), I got
>> the program execution time reduced by ~12% on x86-64, it looks very
>> impressive.
>
> Agreed, both patchset are impressive works and achieving amazing
> results!
>
> My view is that in terms of long-lived computation from userland point
> of view, both models are malleable enough and could achieve everything
> we need in the end, but as far as the overall kernel efficiency is
> concerned the compound model will always retain a slight advantage in
> performance by leveraging a native THP compound refcounting that
> requires just one atomic_inc/dec per THP mapcount instead of 512 of
> them. Other advantages of the compound model is that it's half in code
> size despite already including khugepaged (i.e. the same
> split_huge_page works for both tmpfs and anon) and like said above it
> won't introduce much complications for drivers like KVM as the model
> didn't change.
>
> Thanks,
> Andrea



-- 
Andres Lagar-Cavilla | Google Kernel Team | andreslc@google.com

[toc] | [prev] | [next] | [standalone]

#1385774

FromWincy Van <fanwenyi0529@gmail.com>
Date2016-04-24 07:50 +0200
Message-ID<rrl5D-7U4-3@gated-at.bofh.it>
In reply to#1382716
On Wed, Apr 20, 2016 at 1:07 AM, Andres Lagar-Cavilla
<andreslc@google.com> wrote:
> Andrea, we provide the, ahem, adjustments to
> transparent_hugepage_adjust. Rest assured we aggressively use mmu
> notifiers with no further changes required.
>
> As in: zero changes have been required in the lifetime (years) of
> kvm+huge tmpfs at Google, other than mod'ing
> transparent_hugepage_adjust.

We are using kvm + tmpfs to do qemu live upgrading, how does google
use this memory model ?
I think our pupose to use tmpfs may be the same.

And huge tmpfs is a really good improvement for that.

>
> As noted by Paolo, the additions to transparent_hugepage_adjust could
> be lifted outside of kvm (into shmem.c? maybe) for any consumer of
> huge tmpfs with mmu notifiers.
>

Thanks,
Wincy

[toc] | [prev] | [next] | [standalone]

#1386445

FromAndres Lagar-Cavilla <andreslc@google.com>
Date2016-04-25 15:40 +0200
Message-ID<rrOU2-6G6-25@gated-at.bofh.it>
In reply to#1385774
On Sat, Apr 23, 2016 at 10:46 PM, Wincy Van <fanwenyi0529@gmail.com> wrote:
> On Wed, Apr 20, 2016 at 1:07 AM, Andres Lagar-Cavilla
> <andreslc@google.com> wrote:
>> Andrea, we provide the, ahem, adjustments to
>> transparent_hugepage_adjust. Rest assured we aggressively use mmu
>> notifiers with no further changes required.
>>
>> As in: zero changes have been required in the lifetime (years) of
>> kvm+huge tmpfs at Google, other than mod'ing
>> transparent_hugepage_adjust.
>
> We are using kvm + tmpfs to do qemu live upgrading, how does google
> use this memory model ?
> I think our pupose to use tmpfs may be the same.

Nothing our of the ordinary. Guest memory is an mmap of a tmpfs fd.
Huge tmpfs gives us naturally a great guest performance boost.
MAP_SHARED, and having guest memory persist any one given process, are
what drives us to use tmpfs.

Andres
>
> And huge tmpfs is a really good improvement for that.
>
>>
>> As noted by Paolo, the additions to transparent_hugepage_adjust could
>> be lifted outside of kvm (into shmem.c? maybe) for any consumer of
>> huge tmpfs with mmu notifiers.
>>
>
> Thanks,
> Wincy



-- 
Andres Lagar-Cavilla | Google Kernel Team | andreslc@google.com

[toc] | [prev] | [next] | [standalone]

#1387513

FromWincy Van <fanwenyi0529@gmail.com>
Date2016-04-26 16:10 +0200
Message-ID<rsbQC-Gm-17@gated-at.bofh.it>
In reply to#1386445
On Mon, Apr 25, 2016 at 9:30 PM, Andres Lagar-Cavilla
<andreslc@google.com> wrote:
>>
>> We are using kvm + tmpfs to do qemu live upgrading, how does google
>> use this memory model ?
>> I think our pupose to use tmpfs may be the same.
>
> Nothing our of the ordinary. Guest memory is an mmap of a tmpfs fd.
> Huge tmpfs gives us naturally a great guest performance boost.
> MAP_SHARED, and having guest memory persist any one given process, are
> what drives us to use tmpfs.
>

OK. We are also using mmap.

Besides google's kvm userspace(as I know it is not qemu), google have another
userspace tool need to access guest memory, so that google use tmpfs?

If so, what function does that another userspace do?

Thanks,
Wincy

[toc] | [prev] | [next] | [standalone]

#1389276

FromAndrea Arcangeli <aarcange@redhat.com>
Date2016-04-27 17:50 +0200
Message-ID<rszSW-3yc-23@gated-at.bofh.it>
In reply to#1382716
Hello Andres,

On Tue, Apr 19, 2016 at 10:07:29AM -0700, Andres Lagar-Cavilla wrote:
> Andrea, we provide the, ahem, adjustments to
> transparent_hugepage_adjust. Rest assured we aggressively use mmu
> notifiers with no further changes required.

Did you notice I just fixed a THP related bug in the very function I
quoted that broke with the THP refcounting in v4.5? That very function
had a major bug after the THP refcounting that was corrupting memory
even with regular anonymous memory THP backing.

https://marc.info/?l=linux-mm&m=146175869123580&w=2

> As in: zero changes have been required in the lifetime (years) of
> kvm+huge tmpfs at Google, other than mod'ing
> transparent_hugepage_adjust.

Zero changes required until the THP model gets improved over time to
accomodate for huge-DAX, tmpfs, ext4 and everything else. THP
refcounting in v4.5 also didn't change this function and that thing
broke off silently.

When I found the bug I realized I just quoted the very buggy function
earlier in this thread, as an example of why we don't want more
complexity in the kernel... and that just reinforced my not wanting
more complexity and wanting just 1 single model for all THP in the
kernel, hence this email.

> As noted by Paolo, the additions to transparent_hugepage_adjust could
> be lifted outside of kvm (into shmem.c? maybe) for any consumer of
> huge tmpfs with mmu notifiers.

That function is not duplicated across the kernel, moving it to common
code can be helpful if others have the same needs to save some .text
but where such function goes and if it's duplicated or not, changes
nothing in terms of overall kernel complexity to maintain with two
completely different THP models in different memory management parts.

Dismissing the complexity of supporting and maintaining two completely
different models of transparent hugepages that provides different
kernel APIs to deal with them for secondary MMU drivers as a
triviality, is proven wrong by what just happened to this function in
v4.5 I think.

The model for THP should be just one, either PageTeam and disaband
works for all THP including ext4 and anonymous memory, or
PageTransCompoundMap and the same split_huge_pmd/split_huge_page
functions already works for tmpfs THP and anon THP exactly in the same
way.

We already have to support a slight different model for hugetlbfs, and
thankfully it's not really different, it's just a "subset" of the THP
model, and it's simpler, so it's not complicating anything (nor
get_user_pages, nor KVM, nor the transparent_hugepage_adjust
function).

In fact the PageTransCompoundMap already works for both hugetlbfs and
THP transparently, the page->_mapcount < 1 check is full bypass for
hugetlbfs exactly because the model is actually the same as THP but a
"subset". hugetlbfs uses compound pages too of course to be much
faster than it ever would with the Team page model.

We don't want another model that just increases the complexity of the
kernel for no good and this was agreed at the MM summit too.

In fact I'd go as far as saying Team Pages must work for hugetlbfs too
and not only anonymous memory, for them to be considered as an
attractive option for tmpfs.

You should drop compound pages from hugetlbfs too, if you intend to
pursue the Team pages direction for the upstream kernel.

Kirill great work for v4.5 in addition of simplifying
get_page/put_page (with a mico-performance improvement for
get_user_pages_fast for tail pages) was a dependency in order to allow
compound page to enter the tmpfs and ext4 land. Clearly it introduced
complexity elsewhere (i.e. split_huge_page now can fail) but the model
is now more generic and powerful in allowing both pmd_trans_huge and
ptes to map compound pages natively, which is needed for tmpfs.

Now that such work is done and upstream I don't see why we want to go
in a direction that isn't justified anymore. Team pages made sense to
reduce the time to market in not having to do the THP refcounting work
Kirill just did to achieve compound THP in tmpfs. They're a fine
not-upstream patchset and they would be suitable to ship in a
distribution kernel or in your proprietary
behind-the-firewall-source-not-released usage. For upstream we should
focus on the long term design not on short term production matters.

Furthermore even for production Team pages also still miss khugepaged,
so there's no point to keep going in that direction when the patchset
is double the size of compound THP pages in tmpfs, and the compound
THP patchset already inlcudes khugepaged in half the size.

I already mentioned why I think Team Pages can't work nearly as
efficiently as compound pages for Anonymous memory in the previous
email and the same issue applies to hugetlbfs too.

Furthermore even for small files the current team pages model of
allocating a contiguous hugepage and then mapping only 4k of the 2M
contiguous chunk of ram allocated is counter productive and is fine
for qemu production usage on tmpfs but not ok for generic production
usage. Team pages as currently implemented will trigger memory
pressure 512 times faster than Kirill's tmpfs version if dealing with
4k files only, running specfs or something. After memory pressure
triggers, the not mapped part of the team page is freed right away so
all work done at the allocation stage is just triggering memory
pressure 512 times faster and then the VM has to do even more useless
work to undo the initial contiguous allocation.

Kirill's compound THP in tmpfs by default is a full bypass for <2MB
i_size, so it'll perform exactly the same for small files and it won't
risk to trigger memory pressure nor require undoing the work done to
allocate the hugepages in the first place. This is fundamental for
XGD_RUNTIME_DIR even on the desktop, not just specfs. If the file
grows over time and the allocation is long lived, Kirill's khugepaged
will collapse THP compound pages asynchronously.

If you don't believe that allocating 2MB for small files and then
freeing the memory when eventually memory pressure trigger 512 times
faster, and missing khugepaged are a showstopper, just check the
discussion on linux-mm where they're proposing to disable direct
compaction and relay only on khugepaged and kcompactd for THP in
anonymous memory, because direct compaction is hurting short lived
allocations on large systems that may require lots of defrag to get
the hugepage (it's not THP itself the problem, THP native compound
faults are a speedup for short lived allocation too and they only get
allocated if the vma is large enough, and in Kirill's THP in tmpfs
version, when the i_size is large enough).

Again, if you only focus on qemu and long lived allocation, both
works great and are amazing work.

However for the long term design we need a single THP design, and
hugetlbfs has to be a subset of it. The design used for THP in
anonymous memory is the one that provides the lowest probability that
no matter the load (short lived, long lived, anything) the risk that
THP is a slowdown is the minimum possible and this shall not
change. Furthermore the compound design is a tremendous speedup also
for short allocations as we don't fault it 4k at time like team pages
would do if the i_size is truncated right away at >=2MB (and the vma
is large enough and properly file-offset hugepage aligned).

For long lived allocations and qemu usage both will work the same, and
you can't notice all the downsides of team pages if you only focus on 
THP craving workloads like KVM.

Thanks,
Andrea

[toc] | [prev] | [next] | [standalone]

#1382906

From"Shi, Yang" <yang.shi@linaro.org>
Date2016-04-20 01:50 +0200
Message-ID<rpNz3-6NM-1@gated-at.bofh.it>
In reply to#1382706
On 4/19/2016 9:50 AM, Andrea Arcangeli wrote:
> Hello,
>
> On Mon, Apr 18, 2016 at 03:55:44PM -0700, Shi, Yang wrote:
>> Hi Kirill,
>>
>> Finally, I got some time to look into and try yours and Hugh's patches,
>> got two problems.
>
> One thing that come to mind to test is this: qemu with -machine
> accel=kvm -mem-path=/dev/shm/,share=on .

Thanks for the suggestion, I will definitely have a try with KVM.

It would be better if Kirill and Hugh could share what benchmark they 
ran and how much they got improved since my test case is very simple and 
may just cover a small part of it.

Yang

>
> The THP Compound approach in tmpfs may just happen to work already
> with KVM (or at worst it'd require minor adjustments) because it uses
> the exact same model KVM is already aware about from THP in anonymous
> memory, example from arch/x86/kvm/mmu.c:
>
> static void transparent_hugepage_adjust(struct kvm_vcpu *vcpu,
> 					gfn_t *gfnp, kvm_pfn_t *pfnp,
> 					int *levelp)
> {
> 	kvm_pfn_t pfn = *pfnp;
> 	gfn_t gfn = *gfnp;
> 	int level = *levelp;
>
> 	/*
> 	 * Check if it's a transparent hugepage. If this would be an
> 	 * hugetlbfs page, level wouldn't be set to
> 	 * PT_PAGE_TABLE_LEVEL and there would be no adjustment done
> 	 * here.
> 	 */
> 	if (!is_error_noslot_pfn(pfn) && !kvm_is_reserved_pfn(pfn) &&
> 	    level == PT_PAGE_TABLE_LEVEL &&
> 	    PageTransCompound(pfn_to_page(pfn)) &&
> 	    !mmu_gfn_lpage_is_disallowed(vcpu, gfn, PT_DIRECTORY_LEVEL)) {
>
> Not using two different models between THP in tmpfs and THP in anon is
> essential not just to significantly reduce the size of the kernel
> code, but also because THP knowledge can't be self contained in the
> mm/shmem.c file. Having to support two different models would
> complicate things for secondary MMU drivers (i.e. mmu notifer users)
> like KVM who also need to create huge mapping in the shadow pagetable
> layer in arch/x86/kvm if the primary MMU allows for it.
>
>> x86-64 and ARM64 with yours and Hugh's patches (linux-next tree), I got
>> the program execution time reduced by ~12% on x86-64, it looks very
>> impressive.
>
> Agreed, both patchset are impressive works and achieving amazing
> results!
>
> My view is that in terms of long-lived computation from userland point
> of view, both models are malleable enough and could achieve everything
> we need in the end, but as far as the overall kernel efficiency is
> concerned the compound model will always retain a slight advantage in
> performance by leveraging a native THP compound refcounting that
> requires just one atomic_inc/dec per THP mapcount instead of 512 of
> them. Other advantages of the compound model is that it's half in code
> size despite already including khugepaged (i.e. the same
> split_huge_page works for both tmpfs and anon) and like said above it
> won't introduce much complications for drivers like KVM as the model
> didn't change.
>
> Thanks,
> Andrea
>

[toc] | [prev] | [next] | [standalone]

#1383157

FromHugh Dickins <hughd@google.com>
Date2016-04-20 10:40 +0200
Message-ID<rpVPX-4Ta-5@gated-at.bofh.it>
In reply to#1382094
On Mon, 18 Apr 2016, Shi, Yang wrote:

> Hi Kirill,
> 
> Finally, I got some time to look into and try yours and Hugh's patches, got

Thank you.

> two problems.
> 
> 1. A quick boot up test on my ARM64 machine with your v7 tree shows some
> unexpected error:
> 
> systemd-journald[285]: Failed to save stream data
> /run/systemd/journal/streams/8:16863: No space left on device
> systemd-journald[285]: Failed to save stream data
> /run/systemd/journal/streams/8:16865: No space left on device
>          Starting DNS forwarder and DHCP server.systemd-journald[285]: Failed
> to save stream data /run/systemd/journal/streams/8:16867: No space left on
> device
> ..
> systemd-journald[285]: Failed to save stream data
> /run/systemd/journal/streams/8:16869: No space left on device
>          Starting Postfix Mail Transport Agent...
> systemd-journald[285]: Failed to save stream data
> /run/systemd/journal/streams/8:16871: No space left on device
>          Starting Berkeley Internet Name Domain (DNS)...
>          Starting Wait for Network to be Configured...
> systemd-journald[285]: Failed to save stream data
> /run/systemd/journal/streams/8:2422: No space left on device
> [  OK  ] Started /etc/rc.local Compatibility.
> [FAILED] Failed to start DNS forwarder and DHCP server.
> See 'systemctl status dnsmasq.service' for details.
> systemd-journald[285]: Failed to save stream data
> /run/systemd/journal/streams/8:2425: No space left on device
> [  OK  ] Started Serial Getty on ttyS1.
> [  OK  ] Started Serial Getty on ttyS0.
> [  OK  ] Started Getty on tty1.
> systemd-journald[285]: Failed to save stream data
> /run/systemd/journal/streams/8:2433: No space left on device
> [FAILED] Failed to start Berkeley Internet Name Domain (DNS).
> See 'systemctl status named.service' for details.

Expected behaviour: that is a significant limitation of Kirill's current
implementation.  We have agreed at LSF/MM that he will fix that before
his patchset goes further.  (And different changes needed in my patchset.)

> 
> 
> The /run dir is mounted as tmpfs.
> 
> x86 boot doesn't get such error. And, Hugh's patches don't have such problem.
> 
> 2. I ran my THP test (generated a program with 4MB text section) on both
> x86-64 and ARM64 with yours and Hugh's patches (linux-next tree), I got the
> program execution time reduced by ~12% on x86-64, it looks very impressive.

12% sounds about right for x86.  Some loads have been seen to benefit 17%.

> 
> But, on ARM64, there is just ~3% change, and sometimes huge tmpfs may show
> even worse data than non-hugepage.
> 
> Both yours and Hugh's patches has the same behavior.
> 
> Any idea?

... and in a later posting..,

> 
> It would be better if Kirill and Hugh could share what benchmark they ran and
> how much they got improved since my test case is very simple and may just
> cover a small part of it.

Sorry, I've not run any benchmark myself (prefer to let others get more
objective results), nor run on arm64.  I have no idea what to expect on
arm64 - you need to ask the arm64 guys what hugepage advantage they see
with anon THP or hugetlbfs (and probably need to tell them what machine
you're running on): then expect a similar advantage from either Kirill's
or my huge tmpfs patchset.

Hugh

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web