Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1416035
| From | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCHv9-rebased 15/32] thp, mlock: do not mlock PTE-mapped file huge pages |
| Date | 2016-06-07 13:20 +0200 |
| Message-ID | <rHnd8-1YN-19@gated-at.bofh.it> (permalink) |
| References | <rH3o5-5Yv-3@gated-at.bofh.it> <rHn3r-1Vt-11@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
As with anon THP, we only mlock file huge pages if we can prove that the
page is not mapped with PTE. This way we can avoid mlock leak into
non-mlocked vma on split.
We rely on PageDoubleMap() under lock_page() to check if the the page
may be PTE mapped. PG_double_map is set by page_add_file_rmap() when the
page mapped with PTEs.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
include/linux/page-flags.h | 13 ++++++++++++-
mm/huge_memory.c | 27 ++++++++++++++++++++-------
mm/mmap.c | 6 ++++++
mm/page_alloc.c | 2 ++
mm/rmap.c | 16 ++++++++++++++--
5 files changed, 54 insertions(+), 10 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 7c8e82ac2eb7..8cf09639185a 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -581,6 +581,17 @@ static inline int PageDoubleMap(struct page *page)
return PageHead(page) && test_bit(PG_double_map, &page[1].flags);
}
+static inline void SetPageDoubleMap(struct page *page)
+{
+ VM_BUG_ON_PAGE(!PageHead(page), page);
+ set_bit(PG_double_map, &page[1].flags);
+}
+
+static inline void ClearPageDoubleMap(struct page *page)
+{
+ VM_BUG_ON_PAGE(!PageHead(page), page);
+ clear_bit(PG_double_map, &page[1].flags);
+}
static inline int TestSetPageDoubleMap(struct page *page)
{
VM_BUG_ON_PAGE(!PageHead(page), page);
@@ -598,7 +609,7 @@ TESTPAGEFLAG_FALSE(TransHuge)
TESTPAGEFLAG_FALSE(TransCompound)
TESTPAGEFLAG_FALSE(TransCompoundMap)
TESTPAGEFLAG_FALSE(TransTail)
-TESTPAGEFLAG_FALSE(DoubleMap)
+PAGEFLAG_FALSE(DoubleMap)
TESTSETFLAG_FALSE(DoubleMap)
TESTCLEARFLAG_FALSE(DoubleMap)
#endif
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index b3b2bf3da167..28e7d963cc1f 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1438,6 +1438,8 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
* We don't mlock() pte-mapped THPs. This way we can avoid
* leaking mlocked pages into non-VM_LOCKED VMAs.
*
+ * For anon THP:
+ *
* In most cases the pmd is the only mapping of the page as we
* break COW for the mlock() -- see gup_flags |= FOLL_WRITE for
* writable private mappings in populate_vma_page_range().
@@ -1445,15 +1447,26 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
* The only scenario when we have the page shared here is if we
* mlocking read-only mapping shared over fork(). We skip
* mlocking such pages.
+ *
+ * For file THP:
+ *
+ * We can expect PageDoubleMap() to be stable under page lock:
+ * for file pages we set it in page_add_file_rmap(), which
+ * requires page to be locked.
*/
- if (compound_mapcount(page) == 1 && !PageDoubleMap(page) &&
- page->mapping && trylock_page(page)) {
- lru_add_drain();
- if (page->mapping)
- mlock_vma_page(page);
- unlock_page(page);
- }
+
+ if (PageAnon(page) && compound_mapcount(page) != 1)
+ goto skip_mlock;
+ if (PageDoubleMap(page) || !page->mapping)
+ goto skip_mlock;
+ if (!trylock_page(page))
+ goto skip_mlock;
+ lru_add_drain();
+ if (page->mapping && !PageDoubleMap(page))
+ mlock_vma_page(page);
+ unlock_page(page);
}
+skip_mlock:
page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
VM_BUG_ON_PAGE(!PageCompound(page), page);
if (flags & FOLL_GET)
diff --git a/mm/mmap.c b/mm/mmap.c
index 02990e7dd70e..daabef097c78 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -2591,6 +2591,12 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,
/* drop PG_Mlocked flag for over-mapped range */
for (tmp = vma; tmp->vm_start >= start + size;
tmp = tmp->vm_next) {
+ /*
+ * Split pmd and munlock page on the border
+ * of the range.
+ */
+ vma_adjust_trans_huge(tmp, start, start + size, 0);
+
munlock_vma_pages_range(tmp,
max(tmp->vm_start, start),
min(tmp->vm_end, start + size));
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a46547389e53..e32ff3abe9da 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1005,6 +1005,8 @@ static __always_inline bool free_pages_prepare(struct page *page,
VM_BUG_ON_PAGE(compound && compound_order(page) != order, page);
+ if (compound)
+ ClearPageDoubleMap(page);
for (i = 1; i < (1 << order); i++) {
if (compound)
bad += free_tail_pages_check(page, page + i);
diff --git a/mm/rmap.c b/mm/rmap.c
index b78374519bac..26e3e784ad75 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1287,6 +1287,12 @@ void page_add_file_rmap(struct page *page, bool compound)
if (!atomic_inc_and_test(compound_mapcount_ptr(page)))
goto out;
} else {
+ if (PageTransCompound(page)) {
+ VM_BUG_ON_PAGE(!PageLocked(page), page);
+ SetPageDoubleMap(compound_head(page));
+ if (PageMlocked(page))
+ clear_page_mlock(compound_head(page));
+ }
if (!atomic_inc_and_test(&page->_mapcount))
goto out;
}
@@ -1460,8 +1466,14 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
*/
if (!(flags & TTU_IGNORE_MLOCK)) {
if (vma->vm_flags & VM_LOCKED) {
- /* Holding pte lock, we do *not* need mmap_sem here */
- mlock_vma_page(page);
+ /* PTE-mapped THP are never mlocked */
+ if (!PageTransCompound(page)) {
+ /*
+ * Holding pte lock, we do *not* need
+ * mmap_sem here
+ */
+ mlock_vma_page(page);
+ }
ret = SWAP_MLOCK;
goto out_unmap;
}
--
2.8.1
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCHv9 00/32] THP-enabled tmpfs/shmem using compound pages "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-06 16:20 +0200
[PATCHv9 11/32] thp: skip file huge pmd on copy_huge_pmd() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-06 16:20 +0200
[PATCHv9 09/32] thp: handle file pages in split_huge_pmd() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-06 16:30 +0200
[PATCHv9 06/32] mm: introduce do_set_pmd() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-06 16:30 +0200
[PATCHv9 13/32] thp: run vma_adjust_trans_huge() outside i_mmap_rwsem "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-06 16:30 +0200
[PATCHv9 04/32] mm: postpone page table allocation until we have page to map "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-06 16:30 +0200
[PATCHv9-rebased 24/32] shmem: add huge pages support "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 25/32] shmem, thp: respect MADV_{NO,}HUGEPAGE for file mappings "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 06/32] mm: introduce do_set_pmd() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 16/32] vmscan: split file huge pages before paging them out "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 07/32] thp, vmstats: add counters for huge file pages "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 23/32] shmem: get_unmapped_area align huge page "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 19/32] filemap: prepare find and delete operations for huge pages "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 30/32] thp: introduce CONFIG_TRANSPARENT_HUGE_PAGECACHE "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 01/32] thp, mlock: update unevictable-lru.txt "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 13/32] thp: run vma_adjust_trans_huge() outside i_mmap_rwsem "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 04/32] mm: postpone page table allocation until we have page to map "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 28/32] shmem: make shmem_inode_info::lock irq-safe "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 32/32] thp: update Documentation/{vm/transhuge,filesystems/proc}.txt "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 27/32] khugepaged: move up_read(mmap_sem) out of khugepaged_alloc_page() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 10/32] thp: handle file COW faults "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 29/32] khugepaged: add support of collapse for tmpfs/shmem pages "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 31/32] shmem: split huge pages beyond i_size under memory pressure "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:10 +0200
[PATCHv9-rebased 02/32] mm: do not pass mm_struct into handle_mm_fault "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 21/32] mm, rmap: account shmem thp pages "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 12/32] thp: prepare change_huge_pmd() for file thp "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 08/32] thp: support file pages in zap_huge_pmd() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 05/32] rmap: support file thp "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 11/32] thp: skip file huge pmd on copy_huge_pmd() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 00/32] THP-enabled tmpfs/shmem using compound pages "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 22/32] shmem: prepare huge= mount option and sysfs knob "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 15/32] thp, mlock: do not mlock PTE-mapped file huge pages "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 17/32] page-flags: relax policy for PG_mappedtodisk and PG_reclaim "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 20/32] truncate: handle file thp "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 18/32] radix-tree: implement radix_tree_maybe_preload_order() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 09/32] thp: handle file pages in split_huge_pmd() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
[PATCHv9-rebased 14/32] thp: file pages support for split_huge_page() "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2016-06-07 13:20 +0200
csiph-web