Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1371950 > unrolled thread
| Started by | Hugh Dickins <hughd@google.com> |
|---|---|
| First post | 2016-04-05 23:20 +0200 |
| Last post | 2016-04-06 00:10 +0200 |
| Articles | 20 on this page of 39 — 5 participants |
Back to article view | Back to linux.kernel
[PATCH 00/31] huge tmpfs: THPagecache implemented by teams Hugh Dickins <hughd@google.com> - 2016-04-05 23:20 +0200
[PATCH 03/31] huge tmpfs: huge=N mount option and /proc/sys/vm/shmem_huge Hugh Dickins <hughd@google.com> - 2016-04-05 23:20 +0200
Re: [PATCH 03/31] huge tmpfs: huge=N mount option and /proc/sys/vm/shmem_huge "Kirill A. Shutemov" <kirill@shutemov.name> - 2016-04-11 13:20 +0200
[PATCH 02/31] huge tmpfs: include shmem freeholes in available memory Hugh Dickins <hughd@google.com> - 2016-04-05 23:20 +0200
[PATCH 08/31] huge tmpfs: try_to_unmap_one use page_check_address_transhuge Hugh Dickins <hughd@google.com> - 2016-04-05 23:30 +0200
[PATCH 11/31] huge tmpfs: disband split huge pmds on race or memory failure Hugh Dickins <hughd@google.com> - 2016-04-05 23:30 +0200
[PATCH 06/31] huge tmpfs: shrinker to migrate and free underused holes Hugh Dickins <hughd@google.com> - 2016-04-05 23:30 +0200
[PATCH 10/31] huge tmpfs: map shmem by huge page pmd or by page team ptes Hugh Dickins <hughd@google.com> - 2016-04-05 23:30 +0200
[PATCH 07/31] huge tmpfs: get_unmapped_area align & fault supply huge page Hugh Dickins <hughd@google.com> - 2016-04-05 23:30 +0200
[PATCH 09/31] huge tmpfs: avoid premature exposure of new pagetable Hugh Dickins <hughd@google.com> - 2016-04-05 23:30 +0200
Re: [PATCH 09/31] huge tmpfs: avoid premature exposure of new pagetable "Kirill A. Shutemov" <kirill@shutemov.name> - 2016-04-11 14:00 +0200
[PATCH 13/31] huge tmpfs: use Unevictable lru with variable hpage_nr_pages Hugh Dickins <hughd@google.com> - 2016-04-05 23:40 +0200
[PATCH 12/31] huge tmpfs: extend get_user_pages_fast to shmem pmd Hugh Dickins <hughd@google.com> - 2016-04-05 23:40 +0200
Re: [PATCH 12/31] huge tmpfs: extend get_user_pages_fast to shmem pmd Ingo Molnar <mingo@kernel.org> - 2016-04-06 09:10 +0200
Re: [PATCH 12/31] huge tmpfs: extend get_user_pages_fast to shmem pmd Hugh Dickins <hughd@google.com> - 2016-04-07 05:00 +0200
Re: [PATCH 12/31] huge tmpfs: extend get_user_pages_fast to shmem pmd Ingo Molnar <mingo@kernel.org> - 2016-04-13 11:00 +0200
[PATCH 16/31] kvm: plumb return of hva when resolving page fault. Hugh Dickins <hughd@google.com> - 2016-04-05 23:40 +0200
[PATCH 14/31] huge tmpfs: fix Mlocked meminfo, track huge & unhuge mlocks Hugh Dickins <hughd@google.com> - 2016-04-05 23:40 +0200
[PATCH 15/31] huge tmpfs: fix Mapped meminfo, track huge & unhuge mappings Hugh Dickins <hughd@google.com> - 2016-04-05 23:40 +0200
[PATCH 18/31] huge tmpfs: mem_cgroup move charge on shmem huge pages Hugh Dickins <hughd@google.com> - 2016-04-05 23:50 +0200
[PATCH 17/31] kvm: teach kvm to map page teams as huge pages. Hugh Dickins <hughd@google.com> - 2016-04-05 23:50 +0200
Re: [PATCH 17/31] kvm: teach kvm to map page teams as huge pages. Paolo Bonzini <pbonzini@redhat.com> - 2016-04-06 01:40 +0200
Re: [PATCH 17/31] kvm: teach kvm to map page teams as huge pages. Hugh Dickins <hughd@google.com> - 2016-04-06 03:20 +0200
Re: [PATCH 17/31] kvm: teach kvm to map page teams as huge pages. Paolo Bonzini <pbonzini@redhat.com> - 2016-04-06 08:50 +0200
[PATCH 19/31] huge tmpfs: mem_cgroup shmem_pmdmapped accounting Hugh Dickins <hughd@google.com> - 2016-04-05 23:50 +0200
[PATCH 20/31] huge tmpfs: mem_cgroup shmem_hugepages accounting Hugh Dickins <hughd@google.com> - 2016-04-05 23:50 +0200
[PATCH 22/31] huge tmpfs: /proc/<pid>/smaps show ShmemHugePages Hugh Dickins <hughd@google.com> - 2016-04-06 00:00 +0200
[PATCH 24/31] huge tmpfs recovery: shmem_recovery_populate to fill huge page Hugh Dickins <hughd@google.com> - 2016-04-06 00:00 +0200
[PATCH 21/31] huge tmpfs: show page team flag in pageflags Hugh Dickins <hughd@google.com> - 2016-04-06 00:00 +0200
[PATCH 25/31] huge tmpfs recovery: shmem_recovery_remap & remap_team_by_pmd Hugh Dickins <hughd@google.com> - 2016-04-06 00:00 +0200
[PATCH 26/31] huge tmpfs recovery: shmem_recovery_swapin to read from swap Hugh Dickins <hughd@google.com> - 2016-04-06 00:00 +0200
[PATCH 23/31] huge tmpfs recovery: framework for reconstituting huge pages Hugh Dickins <hughd@google.com> - 2016-04-06 00:00 +0200
Re: [PATCH 23/31] huge tmpfs recovery: framework for reconstituting huge pages Mika Penttilä <mika.penttila@nextfour.com> - 2016-04-06 12:30 +0200
Re: [PATCH 23/31] huge tmpfs recovery: framework for reconstituting huge pages Hugh Dickins <hughd@google.com> - 2016-04-07 04:10 +0200
[PATCH 31/31] huge tmpfs: no kswapd by default on sync allocations Hugh Dickins <hughd@google.com> - 2016-04-06 00:10 +0200
[PATCH 27/31] huge tmpfs recovery: tweak shmem_getpage_gfp to fill team Hugh Dickins <hughd@google.com> - 2016-04-06 00:10 +0200
[PATCH 28/31] huge tmpfs recovery: debugfs stats to complete this phase Hugh Dickins <hughd@google.com> - 2016-04-06 00:10 +0200
[PATCH 30/31] huge tmpfs: shmem_huge_gfpmask and shmem_recovery_gfpmask Hugh Dickins <hughd@google.com> - 2016-04-06 00:10 +0200
[PATCH 29/31] huge tmpfs recovery: page migration call back into shmem Hugh Dickins <hughd@google.com> - 2016-04-06 00:10 +0200
Page 1 of 2 [1] 2 Next page →
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:20 +0200 |
| Subject | [PATCH 00/31] huge tmpfs: THPagecache implemented by teams |
| Message-ID | <rkGye-1F1-7@gated-at.bofh.it> |
Here is my "huge tmpfs" implementation of Transparent Huge Pagecache, rebased to v4.6-rc2 plus the "mm: easy preliminaries to THPagecache" series. The design is just the same as before, when I posted against v3.19: using a team of pagecache pages placed within a huge-order extent, instead of using a compound page (see 04/31 for more info on that). Patches 01-17 are much as before, but with whatever changes were needed for the rebase, and bugfixes folded back in. Patches 18-22 add memcg and smaps visibility. But the more important ones are patches 23-29, which add recovery: reassembling a hugepage after fragmentation or swapping. Patches 30-31 reflect gfpmask doubts: you might prefer that I fold 31 back in and keep 30 internal. It was lack of recovery which stopped me from proposing inclusion of the series a year ago: this series now is fully featured, and ready for v4.7 - but I expect we shall want to wait a release to give time to consider the alternatives. I currently believe that the same functionality (including the team implementation's support for small files, standard mlocking, and recovery) can be achieved with compound pages, but not easily: I think the huge tmpfs functionality should be made available soon, then converted at leisure to compound pages, if that works out (but it's not a job I want to do - what we have here is good enough). Huge tmpfs has been in use within Google for about a year: it's been a success, and gaining ever wider adoption. Several TODOs have not yet been toDONE, because they just haven't surfaced as real-life issues yet: that includes NUMA migration, which is at the top of my list, but so far we've done well enough without it. 01 huge tmpfs: prepare counts in meminfo, vmstat and SysRq-m 02 huge tmpfs: include shmem freeholes in available memory 03 huge tmpfs: huge=N mount option and /proc/sys/vm/shmem_huge 04 huge tmpfs: try to allocate huge pages, split into a team 05 huge tmpfs: avoid team pages in a few places 06 huge tmpfs: shrinker to migrate and free underused holes 07 huge tmpfs: get_unmapped_area align & fault supply huge page 08 huge tmpfs: try_to_unmap_one use page_check_address_transhuge 09 huge tmpfs: avoid premature exposure of new pagetable 10 huge tmpfs: map shmem by huge page pmd or by page team ptes 11 huge tmpfs: disband split huge pmds on race or memory failure 12 huge tmpfs: extend get_user_pages_fast to shmem pmd 13 huge tmpfs: use Unevictable lru with variable hpage_nr_pages 14 huge tmpfs: fix Mlocked meminfo, track huge & unhuge mlocks 15 huge tmpfs: fix Mapped meminfo, track huge & unhuge mappings 16 kvm: plumb return of hva when resolving page fault. 17 kvm: teach kvm to map page teams as huge pages. 18 huge tmpfs: mem_cgroup move charge on shmem huge pages 19 huge tmpfs: mem_cgroup shmem_pmdmapped accounting 20 huge tmpfs: mem_cgroup shmem_hugepages accounting 21 huge tmpfs: show page team flag in pageflags 22 huge tmpfs: /proc/<pid>/smaps show ShmemHugePages 23 huge tmpfs recovery: framework for reconstituting huge pages 24 huge tmpfs recovery: shmem_recovery_populate to fill huge page 25 huge tmpfs recovery: shmem_recovery_remap & remap_team_by_pmd 26 huge tmpfs recovery: shmem_recovery_swapin to read from swap 27 huge tmpfs recovery: tweak shmem_getpage_gfp to fill team 28 huge tmpfs recovery: debugfs stats to complete this phase 29 huge tmpfs recovery: page migration call back into shmem 30 huge tmpfs: shmem_huge_gfpmask and shmem_recovery_gfpmask 31 huge tmpfs: no kswapd by default on sync allocations Documentation/cgroup-v1/memory.txt | 2 Documentation/filesystems/proc.txt | 20 Documentation/filesystems/tmpfs.txt | 106 + Documentation/sysctl/vm.txt | 46 Documentation/vm/pagemap.txt | 2 Documentation/vm/transhuge.txt | 38 Documentation/vm/unevictable-lru.txt | 15 arch/mips/mm/gup.c | 15 arch/s390/mm/gup.c | 19 arch/sparc/mm/gup.c | 19 arch/x86/kvm/mmu.c | 150 + arch/x86/kvm/paging_tmpl.h | 6 arch/x86/mm/gup.c | 15 drivers/base/node.c | 20 drivers/char/mem.c | 23 fs/proc/meminfo.c | 11 fs/proc/page.c | 6 fs/proc/task_mmu.c | 28 include/linux/huge_mm.h | 14 include/linux/kvm_host.h | 2 include/linux/memcontrol.h | 17 include/linux/migrate.h | 2 include/linux/migrate_mode.h | 2 include/linux/mm.h | 3 include/linux/mm_types.h | 1 include/linux/mmzone.h | 5 include/linux/page-flags.h | 10 include/linux/shmem_fs.h | 29 include/trace/events/migrate.h | 7 include/trace/events/mmflags.h | 7 include/uapi/linux/kernel-page-flags.h | 3 ipc/shm.c | 6 kernel/sysctl.c | 33 mm/compaction.c | 5 mm/filemap.c | 10 mm/gup.c | 19 mm/huge_memory.c | 363 +++- mm/internal.h | 26 mm/memcontrol.c | 187 +- mm/memory-failure.c | 7 mm/memory.c | 225 +- mm/mempolicy.c | 13 mm/migrate.c | 37 mm/mlock.c | 183 +- mm/mmap.c | 16 mm/page-writeback.c | 2 mm/page_alloc.c | 55 mm/rmap.c | 129 - mm/shmem.c | 2066 ++++++++++++++++++++++- mm/swap.c | 5 mm/truncate.c | 2 mm/util.c | 1 mm/vmscan.c | 47 mm/vmstat.c | 3 tools/vm/page-types.c | 2 virt/kvm/kvm_main.c | 14 56 files changed, 3627 insertions(+), 472 deletions(-)
[toc] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:20 +0200 |
| Subject | [PATCH 03/31] huge tmpfs: huge=N mount option and /proc/sys/vm/shmem_huge |
| Message-ID | <rkGyf-1F1-31@gated-at.bofh.it> |
| In reply to | #1371950 |
Plumb in a new "huge=1" or "huge=0" mount option to tmpfs: I don't
want to get into a maze of boot options, madvises and fadvises at
this stage, nor extend the use of the existing THP tuning to tmpfs;
though either might be pursued later on. We just want a way to ask
a tmpfs filesystem to favor huge pages, and a way to turn that off
again when it doesn't work out so well. Default of course is off.
"mount -o remount,huge=N /mountpoint" works fine after mount:
remounting from huge=1 (on) to huge=0 (off) will not attempt to
break up huge pages at all, just stop more from being allocated.
It's possible that we shall allow more values for the option later,
to select different strategies (e.g. how hard to try when allocating
huge pages, or when to map hugely and when not, or how sparse a huge
page should be before it is split up), either for experiments, or well
baked in: so use an unsigned char in the superblock rather than a bool.
No new config option: put this under CONFIG_TRANSPARENT_HUGEPAGE,
which is the appropriate option to protect those who don't want
the new bloat, and with which we shall share some pmd code. Use a
"name=numeric_value" format like most other tmpfs options. Prohibit
the option when !CONFIG_TRANSPARENT_HUGEPAGE, just as mpol is invalid
without CONFIG_NUMA (was hidden in mpol_parse_str(): make it explicit).
Allow setting >0 only if the machine has_transparent_hugepage().
But what about Shmem with no user-visible mount? SysV SHM, memfds,
shared anonymous mmaps (of /dev/zero or MAP_ANONYMOUS), GPU drivers'
DRM objects, ashmem. Though unlikely to suit all usages, provide
sysctl /proc/sys/vm/shmem_huge to experiment with huge on those. We
may add a memfd_create flag and a per-file huge/non-huge fcntl later.
And allow shmem_huge two further values: -1 for use in emergencies,
to force the huge option off from all mounts; and (currently) 2,
to force the huge option on for all - very useful for testing.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
Documentation/filesystems/tmpfs.txt | 45 +++++++++++++++++
Documentation/sysctl/vm.txt | 16 ++++++
include/linux/shmem_fs.h | 16 ++++--
kernel/sysctl.c | 12 ++++
mm/shmem.c | 66 ++++++++++++++++++++++++++
5 files changed, 149 insertions(+), 6 deletions(-)
--- a/Documentation/filesystems/tmpfs.txt
+++ b/Documentation/filesystems/tmpfs.txt
@@ -140,9 +140,52 @@ will give you tmpfs instance on /mytmpfs
RAM/SWAP in 10240 inodes and it is only accessible by root.
+Huge tmpfs
+==========
+
+If CONFIG_TRANSPARENT_HUGEPAGE is enabled, tmpfs has a mount (or remount)
+option for transparent huge pagecache, giving the efficiency advantage of
+hugepages (from less TLB pressure and fewer pagetable levels), without
+the inflexibility of hugetlbfs. Huge tmpfs pages can be swapped out when
+memory pressure demands, just as ordinary tmpfs pages can be swapped out.
+
+huge=0 default, don't attempt to allocate hugepages.
+huge=1 allocate hugepages when available, and mmap on hugepage boundaries.
+
+So 'mount -t tmpfs -o huge=1 tmpfs /mytmpfs' will give you a huge tmpfs.
+
+Huge tmpfs pages can be slower to allocate than ordinary pages (since they
+may require compaction), and slower to set up initially than hugetlbfs pages
+(since a team of small pages is managed instead of a single compound page);
+but once set up and mapped, huge tmpfs performance should match hugetlbfs.
+
+/proc/sys/vm/shmem_huge (intended for experimentation only):
+
+Default 0; write 1 to set tmpfs mount option huge=1 on the kernel's
+internal shmem mount, to use huge pages transparently for SysV SHM,
+memfds, shared anonymous mmaps, GPU DRM objects, and ashmem.
+
+In addition to 0 and 1, it also accepts 2 to force the huge=1 option
+automatically on for all tmpfs mounts (intended for testing), or -1
+to force huge off for all (intended for safety if bugs appeared).
+
+/proc/meminfo, /sys/devices/system/node/nodeN/meminfo show:
+
+Shmem: 35016 kB total shmem/tmpfs memory (subset of Cached)
+ShmemHugePages: 26624 kB tmpfs hugepages completed (subset of Shmem)
+ShmemPmdMapped: 12288 kB tmpfs hugepages with huge mappings in userspace
+ShmemFreeHoles: 671444 kB reserved for team pages but available to shrinker
+
+/proc/vmstat, /proc/zoneinfo, /sys/devices/system/node/nodeN/vmstat show:
+
+nr_shmem 8754 total shmem/tmpfs pages (subset of nr_file_pages)
+nr_shmem_hugepages 13 tmpfs hugepages completed (each 512 in nr_shmem)
+nr_shmem_pmdmapped 6 tmpfs hugepages with huge mappings in userspace
+nr_shmem_freeholes 167861 pages reserved for team but available to shrinker
+
Author:
Christoph Rohland <cr@sap.com>, 1.12.01
Updated:
- Hugh Dickins, 4 June 2007
+ Hugh Dickins, 4 June 2007, 3 Oct 2015
Updated:
KOSAKI Motohiro, 16 Mar 2010
--- a/Documentation/sysctl/vm.txt
+++ b/Documentation/sysctl/vm.txt
@@ -56,6 +56,7 @@ Currently, these files are in /proc/sys/
- page-cluster
- panic_on_oom
- percpu_pagelist_fraction
+- shmem_huge
- stat_interval
- stat_refresh
- swappiness
@@ -748,6 +749,21 @@ sysctl, it will revert to this default b
==============================================================
+shmem_huge
+
+Default 0; write 1 to set tmpfs mount option huge=1 on the kernel's
+internal shmem mount, to use huge pages transparently for SysV SHM,
+memfds, shared anonymous mmaps, GPU DRM objects, and ashmem.
+
+In addition to 0 and 1, it also accepts 2 to force the huge=1 option
+automatically on for all tmpfs mounts (intended for testing), or -1
+to force huge off for all (intended for safety if bugs appeared).
+
+See Documentation/filesystems/tmpfs.txt for info on huge tmpfs.
+/proc/sys/vm/shmem_huge is intended for experimentation only.
+
+==============================================================
+
stat_interval
The time interval between which vm statistics are updated. The default
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -28,9 +28,10 @@ struct shmem_sb_info {
unsigned long max_inodes; /* How many inodes are allowed */
unsigned long free_inodes; /* How many are left for allocation */
spinlock_t stat_lock; /* Serialize shmem_sb_info changes */
+ umode_t mode; /* Mount mode for root directory */
+ unsigned char huge; /* Whether to try for hugepages */
kuid_t uid; /* Mount uid for root directory */
kgid_t gid; /* Mount gid for root directory */
- umode_t mode; /* Mount mode for root directory */
struct mempolicy *mpol; /* default memory policy for mappings */
};
@@ -69,18 +70,23 @@ static inline struct page *shmem_read_ma
}
#ifdef CONFIG_TMPFS
-
extern int shmem_add_seals(struct file *file, unsigned int seals);
extern int shmem_get_seals(struct file *file);
extern long shmem_fcntl(struct file *file, unsigned int cmd, unsigned long arg);
-
#else
-
static inline long shmem_fcntl(struct file *f, unsigned int c, unsigned long a)
{
return -EINVAL;
}
+#endif /* CONFIG_TMPFS */
-#endif
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_SHMEM)
+# ifdef CONFIG_SYSCTL
+struct ctl_table;
+extern int shmem_huge, shmem_huge_min, shmem_huge_max;
+extern int shmem_huge_sysctl(struct ctl_table *table, int write,
+ void __user *buffer, size_t *lenp, loff_t *ppos);
+# endif /* CONFIG_SYSCTL */
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE && CONFIG_SHMEM */
#endif
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -43,6 +43,7 @@
#include <linux/ratelimit.h>
#include <linux/compaction.h>
#include <linux/hugetlb.h>
+#include <linux/shmem_fs.h>
#include <linux/initrd.h>
#include <linux/key.h>
#include <linux/times.h>
@@ -1313,6 +1314,17 @@ static struct ctl_table vm_table[] = {
.extra1 = &zero,
.extra2 = &one_hundred,
},
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_SHMEM)
+ {
+ .procname = "shmem_huge",
+ .data = &shmem_huge,
+ .maxlen = sizeof(shmem_huge),
+ .mode = 0644,
+ .proc_handler = shmem_huge_sysctl,
+ .extra1 = &shmem_huge_min,
+ .extra2 = &shmem_huge_max,
+ },
+#endif
#ifdef CONFIG_HUGETLB_PAGE
{
.procname = "nr_hugepages",
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -58,6 +58,7 @@ static struct vfsmount *shm_mnt;
#include <linux/falloc.h>
#include <linux/splice.h>
#include <linux/security.h>
+#include <linux/sysctl.h>
#include <linux/swapops.h>
#include <linux/mempolicy.h>
#include <linux/namei.h>
@@ -289,6 +290,25 @@ static bool shmem_confirm_swap(struct ad
}
/*
+ * Definitions for "huge tmpfs": tmpfs mounted with the huge=1 option
+ */
+
+/* Special values for /proc/sys/vm/shmem_huge */
+#define SHMEM_HUGE_DENY (-1)
+#define SHMEM_HUGE_FORCE (2)
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+/* ifdef here to avoid bloating shmem.o when not necessary */
+
+int shmem_huge __read_mostly;
+
+#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
+
+#define shmem_huge SHMEM_HUGE_DENY
+
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+
+/*
* Like add_to_page_cache_locked, but error if expected item has gone.
*/
static int shmem_add_to_page_cache(struct page *page,
@@ -2857,11 +2877,21 @@ static int shmem_parse_options(char *opt
sbinfo->gid = make_kgid(current_user_ns(), gid);
if (!gid_valid(sbinfo->gid))
goto bad_val;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ } else if (!strcmp(this_char, "huge")) {
+ if (kstrtou8(value, 10, &sbinfo->huge) < 0 ||
+ sbinfo->huge >= SHMEM_HUGE_FORCE)
+ goto bad_val;
+ if (sbinfo->huge && !has_transparent_hugepage())
+ goto bad_val;
+#endif
+#ifdef CONFIG_NUMA
} else if (!strcmp(this_char,"mpol")) {
mpol_put(mpol);
mpol = NULL;
if (mpol_parse_str(value, &mpol))
goto bad_val;
+#endif
} else {
pr_err("tmpfs: Bad mount option %s\n", this_char);
goto error;
@@ -2907,6 +2937,7 @@ static int shmem_remount_fs(struct super
goto out;
error = 0;
+ sbinfo->huge = config.huge;
sbinfo->max_blocks = config.max_blocks;
sbinfo->max_inodes = config.max_inodes;
sbinfo->free_inodes = config.max_inodes - inodes;
@@ -2940,6 +2971,9 @@ static int shmem_show_options(struct seq
if (!gid_eq(sbinfo->gid, GLOBAL_ROOT_GID))
seq_printf(seq, ",gid=%u",
from_kgid_munged(&init_user_ns, sbinfo->gid));
+ /* Rightly or wrongly, show huge mount option unmasked by shmem_huge */
+ if (sbinfo->huge)
+ seq_printf(seq, ",huge=%u", sbinfo->huge);
shmem_show_mpol(seq, sbinfo->mpol);
return 0;
}
@@ -3278,6 +3312,13 @@ int __init shmem_init(void)
pr_err("Could not kern_mount tmpfs\n");
goto out1;
}
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ if (has_transparent_hugepage())
+ SHMEM_SB(shm_mnt->mnt_sb)->huge = (shmem_huge > 0);
+ else
+ shmem_huge = 0; /* just in case it was patched */
+#endif
return 0;
out1:
@@ -3289,6 +3330,31 @@ out3:
return error;
}
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_SYSCTL)
+int shmem_huge_min = SHMEM_HUGE_DENY;
+int shmem_huge_max = SHMEM_HUGE_FORCE;
+/*
+ * /proc/sys/vm/shmem_huge sysctl for internal shm_mnt, and mount override:
+ * -1 disables huge on shm_mnt and all mounts, for emergency use
+ * 0 disables huge on internal shm_mnt (which has no way to be remounted)
+ * 1 enables huge on internal shm_mnt (which has no way to be remounted)
+ * 2 enables huge on shm_mnt and all mounts, w/o needing option, for testing
+ * (but we may add more huge options, and push that 2 for testing upwards)
+ */
+int shmem_huge_sysctl(struct ctl_table *table, int write,
+ void __user *buffer, size_t *lenp, loff_t *ppos)
+{
+ int err;
+
+ if (!has_transparent_hugepage())
+ shmem_huge_max = 0;
+ err = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+ if (write && !err && !IS_ERR(shm_mnt))
+ SHMEM_SB(shm_mnt->mnt_sb)->huge = (shmem_huge > 0);
+ return err;
+}
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE && CONFIG_SYSCTL */
+
#else /* !CONFIG_SHMEM */
/*
[toc] | [prev] | [next] | [standalone]
| From | "Kirill A. Shutemov" <kirill@shutemov.name> |
|---|---|
| Date | 2016-04-11 13:20 +0200 |
| Subject | Re: [PATCH 03/31] huge tmpfs: huge=N mount option and /proc/sys/vm/shmem_huge |
| Message-ID | <rmI2T-4Vk-45@gated-at.bofh.it> |
| In reply to | #1371951 |
On Tue, Apr 05, 2016 at 02:15:05PM -0700, Hugh Dickins wrote: > Plumb in a new "huge=1" or "huge=0" mount option to tmpfs: I don't > want to get into a maze of boot options, madvises and fadvises at > this stage, nor extend the use of the existing THP tuning to tmpfs; > though either might be pursued later on. We just want a way to ask > a tmpfs filesystem to favor huge pages, and a way to turn that off > again when it doesn't work out so well. Default of course is off. > > "mount -o remount,huge=N /mountpoint" works fine after mount: > remounting from huge=1 (on) to huge=0 (off) will not attempt to > break up huge pages at all, just stop more from being allocated. > > It's possible that we shall allow more values for the option later, > to select different strategies (e.g. how hard to try when allocating > huge pages, or when to map hugely and when not, or how sparse a huge > page should be before it is split up), either for experiments, or well > baked in: so use an unsigned char in the superblock rather than a bool. Make the value a string from beginning would be better choice in my opinion. As more allocation policies would be implemented, number would not make much sense. For record, my implementation has four allocation policies: never, always, within_size and advise. > > No new config option: put this under CONFIG_TRANSPARENT_HUGEPAGE, > which is the appropriate option to protect those who don't want > the new bloat, and with which we shall share some pmd code. Use a > "name=numeric_value" format like most other tmpfs options. Prohibit > the option when !CONFIG_TRANSPARENT_HUGEPAGE, just as mpol is invalid > without CONFIG_NUMA (was hidden in mpol_parse_str(): make it explicit). > Allow setting >0 only if the machine has_transparent_hugepage(). > > But what about Shmem with no user-visible mount? SysV SHM, memfds, > shared anonymous mmaps (of /dev/zero or MAP_ANONYMOUS), GPU drivers' > DRM objects, ashmem. Though unlikely to suit all usages, provide > sysctl /proc/sys/vm/shmem_huge to experiment with huge on those. We > may add a memfd_create flag and a per-file huge/non-huge fcntl later. I use sysfs knob instead: /sys/kernel/mm/transparent_hugepage/shmem_enabled And string values there as well. It's better match current THP interface. > And allow shmem_huge two further values: -1 for use in emergencies, > to force the huge option off from all mounts; and (currently) 2, > to force the huge option on for all - very useful for testing. In my case, it's "deny" and "force". -- Kirill A. Shutemov
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:20 +0200 |
| Subject | [PATCH 02/31] huge tmpfs: include shmem freeholes in available memory |
| Message-ID | <rkGyf-1F1-35@gated-at.bofh.it> |
| In reply to | #1371950 |
ShmemFreeHoles will be freed under memory pressure, but are not included
in MemFree: they need to be added into MemAvailable, and wherever the
kernel calculates freeable pages, rather than actually free pages. They
must not be counted as free when considering whether to go to reclaim.
There is certainly room for debate about other places, but I think I've
got about the right list - though I'm unfamiliar with and undecided about
drivers/staging/android/lowmemorykiller.c and kernel/power/snapshot.c.
While NR_SHMEM_FREEHOLES should certainly not be counted in NR_FREE_PAGES,
there is a case for including ShmemFreeHoles in the user-visible MemFree
after all: I can see both sides of that argument, leaving it out so far.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
mm/page-writeback.c | 2 ++
mm/page_alloc.c | 6 ++++++
mm/util.c | 1 +
3 files changed, 9 insertions(+)
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -285,6 +285,7 @@ static unsigned long zone_dirtyable_memo
*/
nr_pages -= min(nr_pages, zone->totalreserve_pages);
+ nr_pages += zone_page_state(zone, NR_SHMEM_FREEHOLES);
nr_pages += zone_page_state(zone, NR_INACTIVE_FILE);
nr_pages += zone_page_state(zone, NR_ACTIVE_FILE);
@@ -344,6 +345,7 @@ static unsigned long global_dirtyable_me
*/
x -= min(x, totalreserve_pages);
+ x += global_page_state(NR_SHMEM_FREEHOLES);
x += global_page_state(NR_INACTIVE_FILE);
x += global_page_state(NR_ACTIVE_FILE);
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3760,6 +3760,12 @@ long si_mem_available(void)
available += pagecache;
/*
+ * Shmem freeholes help to keep huge pages intact, but contain
+ * no data, and can be shrunk whenever small pages are needed.
+ */
+ available += global_page_state(NR_SHMEM_FREEHOLES);
+
+ /*
* Part of the reclaimable slab consists of items that are in use,
* and cannot be freed. Cap this estimate at the low watermark.
*/
--- a/mm/util.c
+++ b/mm/util.c
@@ -496,6 +496,7 @@ int __vm_enough_memory(struct mm_struct
if (sysctl_overcommit_memory == OVERCOMMIT_GUESS) {
free = global_page_state(NR_FREE_PAGES);
+ free += global_page_state(NR_SHMEM_FREEHOLES);
free += global_page_state(NR_FILE_PAGES);
/*
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:30 +0200 |
| Subject | [PATCH 08/31] huge tmpfs: try_to_unmap_one use page_check_address_transhuge |
| Message-ID | <rkGHU-1Ji-7@gated-at.bofh.it> |
| In reply to | #1371950 |
Anon THP's huge pages are split for reclaim in add_to_swap(), before they
reach try_to_unmap(); migrate_misplaced_transhuge_page() does its own pmd
remapping, instead of needing try_to_unmap(); migratable hugetlbfs pages
masquerade as pte-mapped in page_check_address(). So try_to_unmap_one()
did not need to handle transparent pmd mappings as page_referenced_one()
does (beyond the TTU_SPLIT_HUGE_PMD case; though what about TTU_MUNLOCK?).
But tmpfs huge pages are split a little later in the reclaim sequence,
when pageout() calls shmem_writepage(): so try_to_unmap_one() now needs
to handle pmd-mapped pages by using page_check_address_transhuge(), and
a function unmap_team_by_pmd() that we shall place in huge_memory.c in
a later patch, but just use a stub for now.
Refine the lookup in page_check_address_transhuge() slightly, to match
what mm_find_pmd() does, and we've been using for a year: take a pmdval
snapshot of *pmd first, to avoid pmd_lock before the pmd_page check,
with a retry if it changes in between. Was the code wrong before?
I don't think it was, but I am more comfortable with how it is now.
Change its check on hpage_nr_pages() to use compound_order() instead,
two reasons for that: one being that there's now a case in anon THP
splitting where the new call to page_check_address_transhuge() may be on
a PageTail, which hits VM_BUG_ON in PageTransHuge in hpage_nr_pages();
the other being that hpage_nr_pages() on PageTeam gets more interesting
in a later patch, and would no longer be appropriate here.
Say "pmdval" as usual, instead of the "pmde" I made up for mm_find_pmd()
before. Update the comment in mm_find_pmd() to generalise it away from
just the anon_vma lock.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
include/linux/pageteam.h | 6 +++
mm/rmap.c | 65 +++++++++++++++++++++----------------
2 files changed, 43 insertions(+), 28 deletions(-)
--- a/include/linux/pageteam.h
+++ b/include/linux/pageteam.h
@@ -29,4 +29,10 @@ static inline struct page *team_head(str
return head;
}
+/* Temporary stub for mm/rmap.c until implemented in mm/huge_memory.c */
+static inline void unmap_team_by_pmd(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmd, struct page *page)
+{
+}
+
#endif /* _LINUX_PAGETEAM_H */
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -47,6 +47,7 @@
#include <linux/mm.h>
#include <linux/pagemap.h>
+#include <linux/pageteam.h>
#include <linux/swap.h>
#include <linux/swapops.h>
#include <linux/slab.h>
@@ -687,7 +688,7 @@ pmd_t *mm_find_pmd(struct mm_struct *mm,
pgd_t *pgd;
pud_t *pud;
pmd_t *pmd = NULL;
- pmd_t pmde;
+ pmd_t pmdval;
pgd = pgd_offset(mm, address);
if (!pgd_present(*pgd))
@@ -700,12 +701,12 @@ pmd_t *mm_find_pmd(struct mm_struct *mm,
pmd = pmd_offset(pud, address);
/*
* Some THP functions use the sequence pmdp_huge_clear_flush(), set_pmd_at()
- * without holding anon_vma lock for write. So when looking for a
- * genuine pmde (in which to find pte), test present and !THP together.
+ * without locking out concurrent rmap lookups. So when looking for a
+ * pmd entry, in which to find a pte, test present and !THP together.
*/
- pmde = *pmd;
+ pmdval = *pmd;
barrier();
- if (!pmd_present(pmde) || pmd_trans_huge(pmde))
+ if (!pmd_present(pmdval) || pmd_trans_huge(pmdval))
pmd = NULL;
out:
return pmd;
@@ -800,6 +801,7 @@ bool page_check_address_transhuge(struct
pgd_t *pgd;
pud_t *pud;
pmd_t *pmd;
+ pmd_t pmdval;
pte_t *pte;
spinlock_t *ptl;
@@ -821,32 +823,24 @@ bool page_check_address_transhuge(struct
if (!pud_present(*pud))
return false;
pmd = pmd_offset(pud, address);
+again:
+ pmdval = *pmd;
+ barrier();
+ if (!pmd_present(pmdval))
+ return false;
- if (pmd_trans_huge(*pmd)) {
+ if (pmd_trans_huge(pmdval)) {
+ if (pmd_page(pmdval) != page)
+ return false;
ptl = pmd_lock(mm, pmd);
- if (!pmd_present(*pmd))
- goto unlock_pmd;
- if (unlikely(!pmd_trans_huge(*pmd))) {
+ if (unlikely(!pmd_same(*pmd, pmdval))) {
spin_unlock(ptl);
- goto map_pte;
+ goto again;
}
-
- if (pmd_page(*pmd) != page)
- goto unlock_pmd;
-
pte = NULL;
goto found;
-unlock_pmd:
- spin_unlock(ptl);
- return false;
- } else {
- pmd_t pmde = *pmd;
-
- barrier();
- if (!pmd_present(pmde) || pmd_trans_huge(pmde))
- return false;
}
-map_pte:
+
pte = pte_offset_map(pmd, address);
if (!pte_present(*pte)) {
pte_unmap(pte);
@@ -863,7 +857,7 @@ check_pte:
}
/* THP can be referenced by any subpage */
- if (pte_pfn(*pte) - page_to_pfn(page) >= hpage_nr_pages(page)) {
+ if (pte_pfn(*pte) - page_to_pfn(page) >= (1 << compound_order(page))) {
pte_unmap_unlock(pte, ptl);
return false;
}
@@ -1404,6 +1398,7 @@ static int try_to_unmap_one(struct page
unsigned long address, void *arg)
{
struct mm_struct *mm = vma->vm_mm;
+ pmd_t *pmd;
pte_t *pte;
pte_t pteval;
spinlock_t *ptl;
@@ -1423,8 +1418,7 @@ static int try_to_unmap_one(struct page
goto out;
}
- pte = page_check_address(page, mm, address, &ptl, 0);
- if (!pte)
+ if (!page_check_address_transhuge(page, mm, address, &pmd, &pte, &ptl))
goto out;
/*
@@ -1442,6 +1436,19 @@ static int try_to_unmap_one(struct page
if (flags & TTU_MUNLOCK)
goto out_unmap;
}
+
+ if (!pte) {
+ if (!(flags & TTU_IGNORE_ACCESS) &&
+ IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
+ pmdp_clear_flush_young_notify(vma, address, pmd)) {
+ ret = SWAP_FAIL;
+ goto out_unmap;
+ }
+ spin_unlock(ptl);
+ unmap_team_by_pmd(vma, address, pmd, page);
+ goto out;
+ }
+
if (!(flags & TTU_IGNORE_ACCESS)) {
if (ptep_clear_flush_young_notify(vma, address, pte)) {
ret = SWAP_FAIL;
@@ -1542,7 +1549,9 @@ discard:
put_page(page);
out_unmap:
- pte_unmap_unlock(pte, ptl);
+ spin_unlock(ptl);
+ if (pte)
+ pte_unmap(pte);
if (ret != SWAP_FAIL && ret != SWAP_MLOCK && !(flags & TTU_MUNLOCK))
mmu_notifier_invalidate_page(mm, address);
out:
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:30 +0200 |
| Subject | [PATCH 11/31] huge tmpfs: disband split huge pmds on race or memory failure |
| Message-ID | <rkGHU-1Ji-9@gated-at.bofh.it> |
| In reply to | #1371950 |
Andres L-C has pointed out that the single-page unmap_mapping_range()
fallback in truncate_inode_page() cannot protect against the case when
a huge page was faulted in after the full-range unmap_mapping_range():
because page_mapped(page) checks tail page's mapcount, not the head's.
So, there's a danger that hole-punching (and maybe even truncation)
can free pages while they are mapped into userspace with a huge pmd.
And I don't believe that the CVE-2014-4171 protection in shmem_fault()
can fully protect from this, although it does make it much harder.
Fix that by adding a duplicate single-page unmap_mapping_range()
into shmem_disband_hugeteam() (called when punching or truncating
a PageTeam), at the point when we also hold the head's page lock
(without which there would still be races): which will then split
all huge pmd mappings covering the page into team pte mappings.
This is also just what's needed to handle memory_failure() correctly:
provide custom shmem_error_remove_page(), call shmem_disband_hugeteam()
from that before proceeding to generic_error_remove_page(), then this
additional unmap_mapping_range() will remap team by ptes as needed.
(There is an unlikely case that we're racing with another disbander,
or disband didn't get trylock on head page at first: memory_failure()
has almost finished with the page, so it's safe to unlock and relock
before retrying.)
But there is one further change needed in hwpoison_user_mappings():
it must recognize a hugely mapped team before concluding that the
page is not mapped. (And still no support for soft_offline(),
which will have to wait for page migration of teams.)
Signed-off-by: Hugh Dickins <hughd@google.com>
---
mm/memory-failure.c | 7 ++++++-
mm/shmem.c | 30 +++++++++++++++++++++++++++++-
2 files changed, 35 insertions(+), 2 deletions(-)
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -45,6 +45,7 @@
#include <linux/rmap.h>
#include <linux/export.h>
#include <linux/pagemap.h>
+#include <linux/pageteam.h>
#include <linux/swap.h>
#include <linux/backing-dev.h>
#include <linux/migrate.h>
@@ -902,6 +903,7 @@ static int hwpoison_user_mappings(struct
enum ttu_flags ttu = TTU_UNMAP | TTU_IGNORE_MLOCK | TTU_IGNORE_ACCESS;
struct address_space *mapping;
LIST_HEAD(tokill);
+ bool mapped;
int ret;
int kill = 1, forcekill;
struct page *hpage = *hpagep;
@@ -919,7 +921,10 @@ static int hwpoison_user_mappings(struct
* This check implies we don't kill processes if their pages
* are in the swap cache early. Those are always late kills.
*/
- if (!page_mapped(hpage))
+ mapped = page_mapped(hpage);
+ if (PageTeam(p) && team_pmd_mapped(team_head(p)))
+ mapped = true;
+ if (!mapped)
return SWAP_SUCCESS;
if (PageKsm(p)) {
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -605,6 +605,19 @@ static void shmem_disband_hugeteam(struc
}
/*
+ * truncate_inode_page() will unmap page if page_mapped(page),
+ * but there's a race by which the team could be hugely mapped,
+ * with page_mapped(page) saying false. So check here if the
+ * head is hugely mapped, and if so unmap page to remap team.
+ * Use a loop because there is no good locking against a
+ * concurrent remap_team_by_ptes().
+ */
+ while (team_pmd_mapped(head)) {
+ unmap_mapping_range(page->mapping,
+ (loff_t)page->index << PAGE_SHIFT, PAGE_SIZE, 0);
+ }
+
+ /*
* Disable preemption because truncation may end up spinning until a
* tail PageTeam has been cleared: we hold the lock as briefly as we
* can (splitting disband in two stages), but better not be preempted.
@@ -1305,6 +1318,21 @@ static int shmem_getattr(struct vfsmount
return 0;
}
+static int shmem_error_remove_page(struct address_space *mapping,
+ struct page *page)
+{
+ if (PageTeam(page)) {
+ shmem_disband_hugeteam(page);
+ while (unlikely(PageTeam(page))) {
+ unlock_page(page);
+ cond_resched();
+ lock_page(page);
+ shmem_disband_hugeteam(page);
+ }
+ }
+ return generic_error_remove_page(mapping, page);
+}
+
static int shmem_setattr(struct dentry *dentry, struct iattr *attr)
{
struct inode *inode = d_inode(dentry);
@@ -4088,7 +4116,7 @@ static const struct address_space_operat
#ifdef CONFIG_MIGRATION
.migratepage = migrate_page,
#endif
- .error_remove_page = generic_error_remove_page,
+ .error_remove_page = shmem_error_remove_page,
};
static const struct file_operations shmem_file_operations = {
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:30 +0200 |
| Subject | [PATCH 06/31] huge tmpfs: shrinker to migrate and free underused holes |
| Message-ID | <rkGHU-1Ji-21@gated-at.bofh.it> |
| In reply to | #1371950 |
Using 2MB for each small file is wasteful, and on average even a large
file is likely to waste 1MB at the end. We could say that a huge tmpfs
is only suitable for huge files, but I would much prefer not to limit
it in that way, and would not be very able to test such a filesystem.
In our model, the unused space in the team is not put on any LRU (nor
charged to any memcg), so not yet accessible to page reclaim: we need
a shrinker to disband the team, and free up the unused space, under
memory pressure. (Typically the freeable space is at the end, but
there's no assumption that it's at end of huge page or end of file.)
shmem_shrink_hugehole() is usually called from vmscan's shrink_slabs();
but I've found a direct call from shmem_alloc_page(), when it fails
to allocate a huge page (perhaps because too much memory is occupied
by shmem huge holes), is also helpful before a retry.
But each team holds a valuable resource: an extent of contiguous
memory that could be used for another team (or for an anonymous THP).
So try to proceed in such a way as to conserve that resource: rather
than just freeing the unused space and leaving yet another huge page
fragmented, also try to migrate the used space to another partially
occupied huge page.
The algorithm in shmem_choose_hugehole() (find least occupied huge page
in older half of shrinklist, and migrate its cachepages into the most
occupied huge page with enough space to fit, again chosen from older
half of shrinklist) is unlikely to be ideal; but easy to implement as
a demonstration of the pieces which can be used by any algorithm,
and good enough for now. A radix_tree tag helps to locate the
partially occupied huge pages more quickly: the tag available
since shmem does not participate in dirty/writeback accounting.
The "team_usage" field added to struct page (in union with "private")
is somewhat vaguely named: because while the huge page is sparsely
occupied, it counts the occupancy; but once the huge page is fully
occupied, it will come to be used differently in a later patch, as
the huge mapcount (offset by the HPAGE_PMD_NR occupancy) - it is
never possible to map a sparsely occupied huge page, because that
would expose stale data to the user.
With this patch, the ShmemHugePages and ShmemFreeHoles lines of
/proc/meminfo are shown correctly; but ShmemPmdMapped remains 0.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
Documentation/filesystems/tmpfs.txt | 9
include/linux/migrate.h | 1
include/linux/mm_types.h | 1
include/linux/shmem_fs.h | 3
include/trace/events/migrate.h | 3
mm/shmem.c | 440 +++++++++++++++++++++++++-
6 files changed, 443 insertions(+), 14 deletions(-)
--- a/Documentation/filesystems/tmpfs.txt
+++ b/Documentation/filesystems/tmpfs.txt
@@ -159,6 +159,15 @@ may require compaction), and slower to s
(since a team of small pages is managed instead of a single compound page);
but once set up and mapped, huge tmpfs performance should match hugetlbfs.
+When a file is created on a huge tmpfs (or copied there), a hugepage is
+allocated to it if possible. Initially only one small page of the hugepage
+will actually be used for the file: then the neighbouring free holes filled
+as more data is added, until the hugepage is completed. But if the hugepage
+is left incomplete, and memory needs to be reclaimed, then a shrinker can
+disband the team and free those holes; or page reclaim disband the team
+and swap out the tmpfs pagecache. Free holes are not charged to any
+memcg, and are counted in MemAvailable; but are not counted in MemFree.
+
/proc/sys/vm/shmem_huge (intended for experimentation only):
Default 0; write 1 to set tmpfs mount option huge=1 on the kernel's
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -24,6 +24,7 @@ enum migrate_reason {
MR_MEMPOLICY_MBIND,
MR_NUMA_MISPLACED,
MR_CMA,
+ MR_SHMEM_HUGEHOLE,
MR_TYPES
};
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -183,6 +183,7 @@ struct page {
#endif
#endif
struct kmem_cache *slab_cache; /* SL[AU]B: Pointer to slab */
+ atomic_long_t team_usage; /* In shmem's PageTeam page */
};
#ifdef CONFIG_MEMCG
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -16,8 +16,9 @@ struct shmem_inode_info {
unsigned long flags;
unsigned long alloced; /* data pages alloced to file */
unsigned long swapped; /* subtotal assigned to swap */
- struct shared_policy policy; /* NUMA memory alloc policy */
+ struct list_head shrinklist; /* shrinkable hpage inodes */
struct list_head swaplist; /* chain of maybes on swap */
+ struct shared_policy policy; /* NUMA memory alloc policy */
struct simple_xattrs xattrs; /* list of xattrs */
struct inode vfs_inode;
};
--- a/include/trace/events/migrate.h
+++ b/include/trace/events/migrate.h
@@ -19,7 +19,8 @@
EM( MR_SYSCALL, "syscall_or_cpuset") \
EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind") \
EM( MR_NUMA_MISPLACED, "numa_misplaced") \
- EMe(MR_CMA, "cma")
+ EM( MR_CMA, "cma") \
+ EMe(MR_SHMEM_HUGEHOLE, "shmem_hugehole")
/*
* First define the enums in the above macros to be exported to userspace
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -58,6 +58,7 @@ static struct vfsmount *shm_mnt;
#include <linux/falloc.h>
#include <linux/splice.h>
#include <linux/security.h>
+#include <linux/shrinker.h>
#include <linux/sysctl.h>
#include <linux/swapops.h>
#include <linux/pageteam.h>
@@ -304,6 +305,14 @@ static bool shmem_confirm_swap(struct ad
#define SHMEM_RETRY_HUGE_PAGE ((struct page *)3)
/* otherwise hugehint is the hugeteam page to be used */
+/* tag for shrinker to locate unfilled hugepages */
+#define SHMEM_TAG_HUGEHOLE PAGECACHE_TAG_DIRTY
+
+/* list of inodes with unfilled hugepages, from which shrinker may free */
+static LIST_HEAD(shmem_shrinklist);
+static unsigned long shmem_shrinklist_depth;
+static DEFINE_SPINLOCK(shmem_shrinklist_lock);
+
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
/* ifdef here to avoid bloating shmem.o when not necessary */
@@ -358,28 +367,106 @@ restart:
return page;
}
+static int shmem_freeholes(struct page *head)
+{
+ unsigned long nr = atomic_long_read(&head->team_usage);
+
+ return (nr >= HPAGE_PMD_NR) ? 0 : HPAGE_PMD_NR - nr;
+}
+
+static void shmem_clear_tag_hugehole(struct address_space *mapping,
+ pgoff_t index)
+{
+ struct page *page = NULL;
+
+ /*
+ * The tag was set on the first subpage to be inserted in cache.
+ * When written sequentially, or instantiated by a huge fault,
+ * it will be on the head page, but that's not always so. And
+ * radix_tree_tag_clear() succeeds when it finds a slot, whether
+ * tag was set on it or not. So first lookup and then clear.
+ */
+ radix_tree_gang_lookup_tag(&mapping->page_tree, (void **)&page,
+ index, 1, SHMEM_TAG_HUGEHOLE);
+ VM_BUG_ON(!page || page->index >= index + HPAGE_PMD_NR);
+ radix_tree_tag_clear(&mapping->page_tree, page->index,
+ SHMEM_TAG_HUGEHOLE);
+}
+
+static void shmem_added_to_hugeteam(struct page *page, struct zone *zone,
+ struct page *hugehint)
+{
+ struct address_space *mapping = page->mapping;
+ struct page *head = team_head(page);
+ int nr;
+
+ if (hugehint == SHMEM_ALLOC_HUGE_PAGE) {
+ atomic_long_set(&head->team_usage, 1);
+ radix_tree_tag_set(&mapping->page_tree, page->index,
+ SHMEM_TAG_HUGEHOLE);
+ __mod_zone_page_state(zone, NR_SHMEM_FREEHOLES, HPAGE_PMD_NR-1);
+ } else {
+ /* We do not need atomic ops until huge page gets mapped */
+ nr = atomic_long_read(&head->team_usage) + 1;
+ atomic_long_set(&head->team_usage, nr);
+ if (nr == HPAGE_PMD_NR) {
+ shmem_clear_tag_hugehole(mapping, head->index);
+ __inc_zone_state(zone, NR_SHMEM_HUGEPAGES);
+ }
+ __dec_zone_state(zone, NR_SHMEM_FREEHOLES);
+ }
+}
+
static int shmem_disband_hugehead(struct page *head)
{
struct address_space *mapping;
struct zone *zone;
int nr = -EALREADY; /* A racing task may have disbanded the team */
- mapping = head->mapping;
- zone = page_zone(head);
+ /*
+ * In most cases the head page is locked, or not yet exposed to others:
+ * only in the shrinker migration case might head have been truncated.
+ * But although head->mapping may then be zeroed at any moment, mapping
+ * stays safe because shmem_evict_inode must take the shrinklist_lock,
+ * and our caller shmem_choose_hugehole is already holding that lock.
+ */
+ mapping = READ_ONCE(head->mapping);
+ if (!mapping)
+ return nr;
+ zone = page_zone(head);
spin_lock_irq(&mapping->tree_lock);
+
if (PageTeam(head)) {
+ nr = atomic_long_read(&head->team_usage);
+ atomic_long_set(&head->team_usage, 0);
+ /*
+ * Disable additions to the team.
+ * Ensure head->private is written before PageTeam is
+ * cleared, so shmem_writepage() cannot write swap into
+ * head->private, then have it overwritten by that 0!
+ */
+ smp_mb__before_atomic();
ClearPageTeam(head);
if (!PageSwapBacked(head))
head->mapping = NULL;
- __dec_zone_state(zone, NR_SHMEM_HUGEPAGES);
- nr = 0;
+
+ if (nr >= HPAGE_PMD_NR) {
+ __dec_zone_state(zone, NR_SHMEM_HUGEPAGES);
+ VM_BUG_ON(nr != HPAGE_PMD_NR);
+ } else if (nr) {
+ shmem_clear_tag_hugehole(mapping, head->index);
+ __mod_zone_page_state(zone, NR_SHMEM_FREEHOLES,
+ nr - HPAGE_PMD_NR);
+ }
}
+
spin_unlock_irq(&mapping->tree_lock);
return nr;
}
-static void shmem_disband_hugetails(struct page *head)
+static void shmem_disband_hugetails(struct page *head,
+ struct list_head *list, int nr)
{
struct page *page;
struct page *endpage;
@@ -387,7 +474,7 @@ static void shmem_disband_hugetails(stru
page = head;
endpage = head + HPAGE_PMD_NR;
- /* Condition follows in next but one commit */ {
+ if (!nr) {
/*
* The usual case: disbanding team and freeing holes as cold
* (cold being more likely to preserve high-order extents).
@@ -402,7 +489,50 @@ static void shmem_disband_hugetails(stru
else if (put_page_testzero(page))
free_hot_cold_page(page, 1);
}
+ } else if (nr < 0) {
+ struct zone *zone = page_zone(page);
+ int orig_nr = nr;
+ /*
+ * Shrinker wants to migrate cache pages from this team.
+ */
+ if (!PageSwapBacked(page)) { /* head was not in cache */
+ if (put_page_testzero(page))
+ free_hot_cold_page(page, 1);
+ } else if (isolate_lru_page(page) == 0) {
+ list_add_tail(&page->lru, list);
+ nr++;
+ }
+ while (++page < endpage) {
+ if (PageTeam(page)) {
+ if (isolate_lru_page(page) == 0) {
+ list_add_tail(&page->lru, list);
+ nr++;
+ }
+ ClearPageTeam(page);
+ } else if (put_page_testzero(page))
+ free_hot_cold_page(page, 1);
+ }
+ /* Yes, shmem counts in NR_ISOLATED_ANON but NR_FILE_PAGES */
+ mod_zone_page_state(zone, NR_ISOLATED_ANON, nr - orig_nr);
+ } else {
+ /*
+ * Shrinker wants free pages from this team to migrate into.
+ */
+ if (!PageSwapBacked(page)) { /* head was not in cache */
+ list_add_tail(&page->lru, list);
+ nr--;
+ }
+ while (++page < endpage) {
+ if (PageTeam(page))
+ ClearPageTeam(page);
+ else if (nr) {
+ list_add_tail(&page->lru, list);
+ nr--;
+ } else if (put_page_testzero(page))
+ free_hot_cold_page(page, 1);
+ }
}
+ VM_BUG_ON(nr > 0); /* maybe a few were not isolated */
}
static void shmem_disband_hugeteam(struct page *page)
@@ -444,12 +574,254 @@ static void shmem_disband_hugeteam(struc
if (head != page)
unlock_page(head);
if (nr_used >= 0)
- shmem_disband_hugetails(head);
+ shmem_disband_hugetails(head, NULL, 0);
if (head != page)
put_page(head);
preempt_enable();
}
+static struct page *shmem_get_hugehole(struct address_space *mapping,
+ unsigned long *index)
+{
+ struct page *page;
+ struct page *head;
+
+ rcu_read_lock();
+ while (radix_tree_gang_lookup_tag(&mapping->page_tree, (void **)&page,
+ *index, 1, SHMEM_TAG_HUGEHOLE)) {
+ if (radix_tree_exception(page))
+ continue;
+ if (!page_cache_get_speculative(page))
+ continue;
+ if (!PageTeam(page) || page->mapping != mapping)
+ goto release;
+ head = team_head(page);
+ if (head != page) {
+ if (!page_cache_get_speculative(head))
+ goto release;
+ put_page(page);
+ page = head;
+ if (!PageTeam(page) || page->mapping != mapping)
+ goto release;
+ }
+ if (shmem_freeholes(head) > 0) {
+ rcu_read_unlock();
+ *index = head->index + HPAGE_PMD_NR;
+ return head;
+ }
+release:
+ put_page(page);
+ }
+ rcu_read_unlock();
+ return NULL;
+}
+
+static unsigned long shmem_choose_hugehole(struct list_head *fromlist,
+ struct list_head *tolist)
+{
+ unsigned long freed = 0;
+ unsigned long double_depth;
+ struct list_head *this, *next;
+ struct shmem_inode_info *info;
+ struct address_space *mapping;
+ struct page *frompage = NULL;
+ struct page *topage = NULL;
+ struct page *page;
+ pgoff_t index;
+ int fromused;
+ int toused;
+ int nid;
+
+ double_depth = 0;
+ spin_lock(&shmem_shrinklist_lock);
+ list_for_each_safe(this, next, &shmem_shrinklist) {
+ info = list_entry(this, struct shmem_inode_info, shrinklist);
+ mapping = info->vfs_inode.i_mapping;
+ if (!radix_tree_tagged(&mapping->page_tree,
+ SHMEM_TAG_HUGEHOLE)) {
+ list_del_init(&info->shrinklist);
+ shmem_shrinklist_depth--;
+ continue;
+ }
+ index = 0;
+ while ((page = shmem_get_hugehole(mapping, &index))) {
+ /* Choose to migrate from page with least in use */
+ if (!frompage ||
+ shmem_freeholes(page) > shmem_freeholes(frompage)) {
+ if (frompage)
+ put_page(frompage);
+ frompage = page;
+ if (shmem_freeholes(page) == HPAGE_PMD_NR-1) {
+ /* No point searching further */
+ double_depth = -3;
+ break;
+ }
+ } else
+ put_page(page);
+ }
+
+ /* Only reclaim from the older half of the shrinklist */
+ double_depth += 2;
+ if (double_depth >= min(shmem_shrinklist_depth, 2000UL))
+ break;
+ }
+
+ if (!frompage)
+ goto unlock;
+ preempt_disable();
+ fromused = shmem_disband_hugehead(frompage);
+ spin_unlock(&shmem_shrinklist_lock);
+ if (fromused > 0)
+ shmem_disband_hugetails(frompage, fromlist, -fromused);
+ preempt_enable();
+ nid = page_to_nid(frompage);
+ put_page(frompage);
+
+ if (fromused <= 0)
+ return 0;
+ freed = HPAGE_PMD_NR - fromused;
+ if (fromused > HPAGE_PMD_NR/2)
+ return freed;
+
+ double_depth = 0;
+ spin_lock(&shmem_shrinklist_lock);
+ list_for_each_safe(this, next, &shmem_shrinklist) {
+ info = list_entry(this, struct shmem_inode_info, shrinklist);
+ mapping = info->vfs_inode.i_mapping;
+ if (!radix_tree_tagged(&mapping->page_tree,
+ SHMEM_TAG_HUGEHOLE)) {
+ list_del_init(&info->shrinklist);
+ shmem_shrinklist_depth--;
+ continue;
+ }
+ index = 0;
+ while ((page = shmem_get_hugehole(mapping, &index))) {
+ /* Choose to migrate to page with just enough free */
+ if (shmem_freeholes(page) >= fromused &&
+ page_to_nid(page) == nid) {
+ if (!topage || shmem_freeholes(page) <
+ shmem_freeholes(topage)) {
+ if (topage)
+ put_page(topage);
+ topage = page;
+ if (shmem_freeholes(page) == fromused) {
+ /* No point searching further */
+ double_depth = -3;
+ break;
+ }
+ } else
+ put_page(page);
+ } else
+ put_page(page);
+ }
+
+ /* Only reclaim from the older half of the shrinklist */
+ double_depth += 2;
+ if (double_depth >= min(shmem_shrinklist_depth, 2000UL))
+ break;
+ }
+
+ if (!topage)
+ goto unlock;
+ preempt_disable();
+ toused = shmem_disband_hugehead(topage);
+ spin_unlock(&shmem_shrinklist_lock);
+ if (toused > 0) {
+ if (HPAGE_PMD_NR - toused >= fromused)
+ shmem_disband_hugetails(topage, tolist, fromused);
+ else
+ shmem_disband_hugetails(topage, NULL, 0);
+ freed += HPAGE_PMD_NR - toused;
+ }
+ preempt_enable();
+ put_page(topage);
+ return freed;
+unlock:
+ spin_unlock(&shmem_shrinklist_lock);
+ return freed;
+}
+
+static struct page *shmem_get_migrate_page(struct page *frompage,
+ unsigned long private, int **result)
+{
+ struct list_head *tolist = (struct list_head *)private;
+ struct page *topage;
+
+ VM_BUG_ON(list_empty(tolist));
+ topage = list_first_entry(tolist, struct page, lru);
+ list_del(&topage->lru);
+ return topage;
+}
+
+static void shmem_put_migrate_page(struct page *topage, unsigned long private)
+{
+ struct list_head *tolist = (struct list_head *)private;
+
+ list_add(&topage->lru, tolist);
+}
+
+static void shmem_putback_migrate_pages(struct list_head *tolist)
+{
+ struct page *topage;
+ struct page *next;
+
+ /*
+ * The tolist pages were not counted in NR_ISOLATED, so stats
+ * would go wrong if putback_movable_pages() were used on them.
+ * Indeed, even putback_lru_page() is wrong for these pages.
+ */
+ list_for_each_entry_safe(topage, next, tolist, lru) {
+ list_del(&topage->lru);
+ if (put_page_testzero(topage))
+ free_hot_cold_page(topage, 1);
+ }
+}
+
+static unsigned long shmem_shrink_hugehole(struct shrinker *shrink,
+ struct shrink_control *sc)
+{
+ unsigned long freed;
+ LIST_HEAD(fromlist);
+ LIST_HEAD(tolist);
+
+ if (list_empty(&shmem_shrinklist))
+ return SHRINK_STOP;
+ freed = shmem_choose_hugehole(&fromlist, &tolist);
+ if (list_empty(&fromlist))
+ return SHRINK_STOP;
+ if (!list_empty(&tolist)) {
+ migrate_pages(&fromlist, shmem_get_migrate_page,
+ shmem_put_migrate_page, (unsigned long)&tolist,
+ MIGRATE_SYNC, MR_SHMEM_HUGEHOLE);
+ preempt_disable();
+ drain_local_pages(NULL); /* try to preserve huge freed page */
+ preempt_enable();
+ shmem_putback_migrate_pages(&tolist);
+ }
+ putback_movable_pages(&fromlist); /* if any were left behind */
+ return freed;
+}
+
+static unsigned long shmem_count_hugehole(struct shrinker *shrink,
+ struct shrink_control *sc)
+{
+ /*
+ * Huge hole space is not charged to any memcg:
+ * only shrink it for global reclaim.
+ * But at present we're only called for global reclaim anyway.
+ */
+ if (list_empty(&shmem_shrinklist))
+ return 0;
+ return global_page_state(NR_SHMEM_FREEHOLES);
+}
+
+static struct shrinker shmem_hugehole_shrinker = {
+ .count_objects = shmem_count_hugehole,
+ .scan_objects = shmem_shrink_hugehole,
+ .seeks = DEFAULT_SEEKS, /* would another value work better? */
+ .batch = HPAGE_PMD_NR, /* would another value work better? */
+};
+
#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
#define shmem_huge SHMEM_HUGE_DENY
@@ -465,6 +837,17 @@ static inline void shmem_disband_hugetea
{
BUILD_BUG();
}
+
+static inline void shmem_added_to_hugeteam(struct page *page,
+ struct zone *zone, struct page *hugehint)
+{
+}
+
+static inline unsigned long shmem_shrink_hugehole(struct shrinker *shrink,
+ struct shrink_control *sc)
+{
+ return 0;
+}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
/*
@@ -504,10 +887,10 @@ shmem_add_to_page_cache(struct page *pag
if (unlikely(error))
goto errout;
- if (!PageTeam(page))
+ if (PageTeam(page))
+ shmem_added_to_hugeteam(page, zone, hugehint);
+ else
get_page(page);
- else if (hugehint == SHMEM_ALLOC_HUGE_PAGE)
- __inc_zone_state(zone, NR_SHMEM_HUGEPAGES);
mapping->nrpages++;
__inc_zone_state(zone, NR_FILE_PAGES);
@@ -932,6 +1315,14 @@ static void shmem_evict_inode(struct ino
shmem_unacct_size(info->flags, inode->i_size);
inode->i_size = 0;
shmem_truncate_range(inode, 0, (loff_t)-1);
+ if (!list_empty(&info->shrinklist)) {
+ spin_lock(&shmem_shrinklist_lock);
+ if (!list_empty(&info->shrinklist)) {
+ list_del_init(&info->shrinklist);
+ shmem_shrinklist_depth--;
+ }
+ spin_unlock(&shmem_shrinklist_lock);
+ }
if (!list_empty(&info->swaplist)) {
mutex_lock(&shmem_swaplist_mutex);
list_del_init(&info->swaplist);
@@ -1286,10 +1677,18 @@ static struct page *shmem_alloc_page(gfp
head = alloc_pages_vma(gfp|__GFP_NORETRY|__GFP_NOWARN,
HPAGE_PMD_ORDER, &pvma, 0, numa_node_id(),
true);
+ if (!head &&
+ shmem_shrink_hugehole(NULL, NULL) != SHRINK_STOP) {
+ head = alloc_pages_vma(
+ gfp|__GFP_NORETRY|__GFP_NOWARN,
+ HPAGE_PMD_ORDER, &pvma, 0,
+ numa_node_id(), true);
+ }
if (head) {
split_page(head, HPAGE_PMD_ORDER);
/* Prepare head page for add_to_page_cache */
+ atomic_long_set(&head->team_usage, 0);
__SetPageTeam(head);
head->mapping = mapping;
head->index = round_down(index, HPAGE_PMD_NR);
@@ -1613,6 +2012,21 @@ repeat:
alloced = true;
/*
+ * Might we see !list_empty a moment before the shrinker
+ * removes this inode from its list? Unlikely, since we
+ * already set a tag in the tree. Some barrier required?
+ */
+ if (alloced_huge && list_empty(&info->shrinklist)) {
+ spin_lock(&shmem_shrinklist_lock);
+ if (list_empty(&info->shrinklist)) {
+ list_add_tail(&info->shrinklist,
+ &shmem_shrinklist);
+ shmem_shrinklist_depth++;
+ }
+ spin_unlock(&shmem_shrinklist_lock);
+ }
+
+ /*
* Let SGP_FALLOC use the SGP_WRITE optimization on a new page.
*/
if (sgp == SGP_FALLOC)
@@ -1823,6 +2237,7 @@ static struct inode *shmem_get_inode(str
spin_lock_init(&info->lock);
info->seals = F_SEAL_SEAL;
info->flags = flags & VM_NORESERVE;
+ INIT_LIST_HEAD(&info->shrinklist);
INIT_LIST_HEAD(&info->swaplist);
simple_xattrs_init(&info->xattrs);
cache_no_acl(inode);
@@ -3613,9 +4028,10 @@ int __init shmem_init(void)
}
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- if (has_transparent_hugepage())
+ if (has_transparent_hugepage()) {
SHMEM_SB(shm_mnt->mnt_sb)->huge = (shmem_huge > 0);
- else
+ register_shrinker(&shmem_hugehole_shrinker);
+ } else
shmem_huge = 0; /* just in case it was patched */
#endif
return 0;
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:30 +0200 |
| Subject | [PATCH 10/31] huge tmpfs: map shmem by huge page pmd or by page team ptes |
| Message-ID | <rkGHU-1Ji-27@gated-at.bofh.it> |
| In reply to | #1371950 |
This is the commit which at last gets huge mappings of tmpfs working,
as can be seen from the ShmemPmdMapped line of /proc/meminfo.
The main thing here is the trio of functions map_team_by_pmd(),
unmap_team_by_pmd() and remap_team_by_ptes() added to huge_memory.c;
and of course the enablement of FAULT_FLAG_MAY_HUGE from memory.c
to shmem.c, with VM_FAULT_HUGE back from shmem.c to memory.c. But
one-line and few-line changes scattered throughout huge_memory.c.
Huge tmpfs is relying on the pmd_trans_huge() page table hooks which
the original Anonymous THP project placed throughout mm; but skips
almost all of its complications, going to its own simpler handling.
Kirill has a much better idea of what copy_huge_pmd() should do for
pagecache: nothing, just as we don't copy shared file ptes. I shall
adopt his idea in a future version, but for now show how to dup team.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
Documentation/vm/transhuge.txt | 38 ++++-
include/linux/pageteam.h | 48 ++++++
mm/huge_memory.c | 229 +++++++++++++++++++++++++++++--
mm/memory.c | 12 +
4 files changed, 307 insertions(+), 20 deletions(-)
--- a/Documentation/vm/transhuge.txt
+++ b/Documentation/vm/transhuge.txt
@@ -9,8 +9,8 @@ using huge pages for the backing of virt
that supports the automatic promotion and demotion of page sizes and
without the shortcomings of hugetlbfs.
-Currently it only works for anonymous memory mappings but in the
-future it can expand over the pagecache layer starting with tmpfs.
+Initially it only worked for anonymous memory mappings, but then was
+extended to the pagecache layer, starting with tmpfs.
The reason applications are running faster is because of two
factors. The first factor is almost completely irrelevant and it's not
@@ -57,9 +57,8 @@ miss is going to run faster.
feature that applies to all dynamic high order allocations in the
kernel)
-- this initial support only offers the feature in the anonymous memory
- regions but it'd be ideal to move it to tmpfs and the pagecache
- later
+- initial support only offered the feature in anonymous memory regions,
+ but then it was extended to huge tmpfs pagecache: see section below.
Transparent Hugepage Support maximizes the usefulness of free memory
if compared to the reservation approach of hugetlbfs by allowing all
@@ -458,3 +457,32 @@ exit(2) if an THP crosses VMA boundary.
Function deferred_split_huge_page() is used to queue page for splitting.
The splitting itself will happen when we get memory pressure via shrinker
interface.
+
+== Huge tmpfs ==
+
+Transparent hugepages were implemented much later in tmpfs.
+That implementation shares much of the "pmd" infrastructure
+devised for anonymous hugepages, and their reliance on compaction.
+
+But unlike hugetlbfs, which has always been free to impose its own
+restrictions, a transparent implementation of pagecache in tmpfs must
+be able to support files both large and small, with large extents
+mapped by hugepage pmds at the same time as small extents (of the
+very same pagecache) are mapped by ptes. For this reason, the
+compound pages used for hugetlbfs and anonymous hugepages were found
+unsuitable, and the opposite approach taken: the high-order backing
+page is split from the start, and managed as a team of partially
+independent small cache pages.
+
+Huge tmpfs is enabled simply by a "huge=1" mount option, and does not
+attend to the boot options, sysfs settings and madvice controlling
+anonymous hugepages. Huge tmpfs recovery (putting a hugepage back
+together after it was disbanded for reclaim, or after a period of
+fragmentation) is done by a workitem scheduled from fault, without
+involving khugepaged at all.
+
+For more info on huge tmpfs, see Documentation/filesystems/tmpfs.txt.
+It is an open question whether that implementation forms the basis for
+extending transparent hugepages to other filesystems' pagecache: in its
+present form, it makes use of struct page's private field, available on
+tmpfs, but already in use on most other filesystems.
--- a/include/linux/pageteam.h
+++ b/include/linux/pageteam.h
@@ -29,10 +29,56 @@ static inline struct page *team_head(str
return head;
}
-/* Temporary stub for mm/rmap.c until implemented in mm/huge_memory.c */
+/*
+ * Returns true if this team is mapped by pmd somewhere.
+ */
+static inline bool team_pmd_mapped(struct page *head)
+{
+ return atomic_long_read(&head->team_usage) > HPAGE_PMD_NR;
+}
+
+/*
+ * Returns true if this was the first mapping by pmd, whereupon mapped stats
+ * need to be updated.
+ */
+static inline bool inc_team_pmd_mapped(struct page *head)
+{
+ return atomic_long_inc_return(&head->team_usage) == HPAGE_PMD_NR+1;
+}
+
+/*
+ * Returns true if this was the last mapping by pmd, whereupon mapped stats
+ * need to be updated.
+ */
+static inline bool dec_team_pmd_mapped(struct page *head)
+{
+ return atomic_long_dec_return(&head->team_usage) == HPAGE_PMD_NR;
+}
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+int map_team_by_pmd(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmd, struct page *page);
+void unmap_team_by_pmd(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmd, struct page *page);
+void remap_team_by_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmd);
+#else
+static inline int map_team_by_pmd(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmd, struct page *page)
+{
+ VM_BUG_ON_PAGE(1, page);
+ return 0;
+}
static inline void unmap_team_by_pmd(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmd, struct page *page)
{
+ VM_BUG_ON_PAGE(1, page);
+}
+static inline void remap_team_by_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmd)
+{
+ VM_BUG_ON(1);
}
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
#endif /* _LINUX_PAGETEAM_H */
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -25,6 +25,7 @@
#include <linux/mman.h>
#include <linux/memremap.h>
#include <linux/pagemap.h>
+#include <linux/pageteam.h>
#include <linux/debugfs.h>
#include <linux/migrate.h>
#include <linux/hashtable.h>
@@ -63,6 +64,8 @@ enum scan_result {
#define CREATE_TRACE_POINTS
#include <trace/events/huge_memory.h>
+static void page_remove_team_rmap(struct page *);
+
/*
* By default transparent hugepage support is disabled in order that avoid
* to risk increase the memory footprint of applications without a guaranteed
@@ -1120,17 +1123,23 @@ int copy_huge_pmd(struct mm_struct *dst_
if (!vma_is_dax(vma)) {
/* thp accounting separate from pmd_devmap accounting */
src_page = pmd_page(pmd);
- VM_BUG_ON_PAGE(!PageHead(src_page), src_page);
get_page(src_page);
- page_dup_rmap(src_page, true);
- add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+ if (PageAnon(src_page)) {
+ VM_BUG_ON_PAGE(!PageHead(src_page), src_page);
+ page_dup_rmap(src_page, true);
+ pmdp_set_wrprotect(src_mm, addr, src_pmd);
+ pmd = pmd_wrprotect(pmd);
+ } else {
+ VM_BUG_ON_PAGE(!PageTeam(src_page), src_page);
+ page_dup_rmap(src_page, false);
+ inc_team_pmd_mapped(src_page);
+ }
+ add_mm_counter(dst_mm, mm_counter(src_page), HPAGE_PMD_NR);
atomic_long_inc(&dst_mm->nr_ptes);
pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
}
- pmdp_set_wrprotect(src_mm, addr, src_pmd);
- pmd = pmd_mkold(pmd_wrprotect(pmd));
- set_pmd_at(dst_mm, addr, dst_pmd, pmd);
+ set_pmd_at(dst_mm, addr, dst_pmd, pmd_mkold(pmd));
ret = 0;
out_unlock:
@@ -1429,7 +1438,7 @@ struct page *follow_trans_huge_pmd(struc
goto out;
page = pmd_page(*pmd);
- VM_BUG_ON_PAGE(!PageHead(page), page);
+ VM_BUG_ON_PAGE(!PageHead(page) && !PageTeam(page), page);
if (flags & FOLL_TOUCH)
touch_pmd(vma, addr, pmd);
if ((flags & FOLL_MLOCK) && (vma->vm_flags & VM_LOCKED)) {
@@ -1454,7 +1463,7 @@ struct page *follow_trans_huge_pmd(struc
}
}
page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
- VM_BUG_ON_PAGE(!PageCompound(page), page);
+ VM_BUG_ON_PAGE(!PageCompound(page) && !PageTeam(page), page);
if (flags & FOLL_GET)
get_page(page);
@@ -1692,10 +1701,12 @@ int zap_huge_pmd(struct mmu_gather *tlb,
put_huge_zero_page();
} else {
struct page *page = pmd_page(orig_pmd);
- page_remove_rmap(page, true);
+ if (PageTeam(page))
+ page_remove_team_rmap(page);
+ page_remove_rmap(page, PageHead(page));
VM_BUG_ON_PAGE(page_mapcount(page) < 0, page);
- add_mm_counter(tlb->mm, MM_ANONPAGES, -HPAGE_PMD_NR);
- VM_BUG_ON_PAGE(!PageHead(page), page);
+ VM_BUG_ON_PAGE(!PageHead(page) && !PageTeam(page), page);
+ add_mm_counter(tlb->mm, mm_counter(page), -HPAGE_PMD_NR);
pte_free(tlb->mm, pgtable_trans_huge_withdraw(tlb->mm, pmd));
atomic_long_dec(&tlb->mm->nr_ptes);
spin_unlock(ptl);
@@ -1739,7 +1750,7 @@ bool move_huge_pmd(struct vm_area_struct
VM_BUG_ON(!pmd_none(*new_pmd));
if (pmd_move_must_withdraw(new_ptl, old_ptl) &&
- vma_is_anonymous(vma)) {
+ !vma_is_dax(vma)) {
pgtable_t pgtable;
pgtable = pgtable_trans_huge_withdraw(mm, old_pmd);
pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
@@ -1789,7 +1800,6 @@ int change_huge_pmd(struct vm_area_struc
entry = pmd_mkwrite(entry);
ret = HPAGE_PMD_NR;
set_pmd_at(mm, addr, pmd, entry);
- BUG_ON(!preserve_write && pmd_write(entry));
}
spin_unlock(ptl);
}
@@ -2991,6 +3001,11 @@ void __split_huge_pmd(struct vm_area_str
struct mm_struct *mm = vma->vm_mm;
unsigned long haddr = address & HPAGE_PMD_MASK;
+ if (!vma_is_anonymous(vma) && !vma->vm_ops->pmd_fault) {
+ remap_team_by_ptes(vma, address, pmd);
+ return;
+ }
+
mmu_notifier_invalidate_range_start(mm, haddr, haddr + HPAGE_PMD_SIZE);
ptl = pmd_lock(mm, pmd);
if (pmd_trans_huge(*pmd)) {
@@ -3469,4 +3484,190 @@ static int __init split_huge_pages_debug
return 0;
}
late_initcall(split_huge_pages_debugfs);
-#endif
+#endif /* CONFIG_DEBUG_FS */
+
+/*
+ * huge pmd support for huge tmpfs
+ */
+
+static void page_add_team_rmap(struct page *page)
+{
+ VM_BUG_ON_PAGE(PageAnon(page), page);
+ VM_BUG_ON_PAGE(!PageTeam(page), page);
+ if (inc_team_pmd_mapped(page))
+ __inc_zone_page_state(page, NR_SHMEM_PMDMAPPED);
+}
+
+static void page_remove_team_rmap(struct page *page)
+{
+ VM_BUG_ON_PAGE(PageAnon(page), page);
+ VM_BUG_ON_PAGE(!PageTeam(page), page);
+ if (dec_team_pmd_mapped(page))
+ __dec_zone_page_state(page, NR_SHMEM_PMDMAPPED);
+}
+
+int map_team_by_pmd(struct vm_area_struct *vma, unsigned long addr,
+ pmd_t *pmd, struct page *page)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ pgtable_t pgtable;
+ spinlock_t *pml;
+ pmd_t pmdval;
+ int ret = VM_FAULT_NOPAGE;
+
+ /*
+ * Another task may have mapped it in just ahead of us; but we
+ * have the huge page locked, so others will wait on us now... or,
+ * is there perhaps some way another might still map in a single pte?
+ */
+ VM_BUG_ON_PAGE(!PageTeam(page), page);
+ VM_BUG_ON_PAGE(!PageLocked(page), page);
+ if (!pmd_none(*pmd))
+ goto raced2;
+
+ addr &= HPAGE_PMD_MASK;
+ pgtable = pte_alloc_one(mm, addr);
+ if (!pgtable) {
+ ret = VM_FAULT_OOM;
+ goto raced2;
+ }
+
+ pml = pmd_lock(mm, pmd);
+ if (!pmd_none(*pmd))
+ goto raced1;
+ pmdval = mk_pmd(page, vma->vm_page_prot);
+ pmdval = pmd_mkhuge(pmd_mkdirty(pmdval));
+ pgtable_trans_huge_deposit(mm, pmd, pgtable);
+ set_pmd_at(mm, addr, pmd, pmdval);
+ page_add_file_rmap(page);
+ page_add_team_rmap(page);
+ update_mmu_cache_pmd(vma, addr, pmd);
+ atomic_long_inc(&mm->nr_ptes);
+ spin_unlock(pml);
+
+ unlock_page(page);
+ add_mm_counter(mm, MM_SHMEMPAGES, HPAGE_PMD_NR);
+ return ret;
+raced1:
+ spin_unlock(pml);
+ pte_free(mm, pgtable);
+raced2:
+ unlock_page(page);
+ put_page(page);
+ return ret;
+}
+
+void unmap_team_by_pmd(struct vm_area_struct *vma, unsigned long addr,
+ pmd_t *pmd, struct page *page)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ pgtable_t pgtable = NULL;
+ unsigned long end;
+ spinlock_t *pml;
+
+ VM_BUG_ON_PAGE(!PageTeam(page), page);
+ VM_BUG_ON_PAGE(!PageLocked(page), page);
+ /*
+ * But even so there might be a racing zap_huge_pmd() or
+ * remap_team_by_ptes() while the page_table_lock is dropped.
+ */
+
+ addr &= HPAGE_PMD_MASK;
+ end = addr + HPAGE_PMD_SIZE;
+
+ mmu_notifier_invalidate_range_start(mm, addr, end);
+ pml = pmd_lock(mm, pmd);
+ if (pmd_trans_huge(*pmd) && pmd_page(*pmd) == page) {
+ pmdp_huge_clear_flush(vma, addr, pmd);
+ pgtable = pgtable_trans_huge_withdraw(mm, pmd);
+ page_remove_team_rmap(page);
+ page_remove_rmap(page, false);
+ atomic_long_dec(&mm->nr_ptes);
+ }
+ spin_unlock(pml);
+ mmu_notifier_invalidate_range_end(mm, addr, end);
+
+ if (!pgtable)
+ return;
+
+ pte_free(mm, pgtable);
+ update_hiwater_rss(mm);
+ add_mm_counter(mm, MM_SHMEMPAGES, -HPAGE_PMD_NR);
+ put_page(page);
+}
+
+void remap_team_by_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pmd_t *pmd)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ struct page *head;
+ struct page *page;
+ pgtable_t pgtable;
+ unsigned long end;
+ spinlock_t *pml;
+ spinlock_t *ptl;
+ pte_t *pte;
+ pmd_t _pmd;
+ pmd_t pmdval;
+ pte_t pteval;
+
+ addr &= HPAGE_PMD_MASK;
+ end = addr + HPAGE_PMD_SIZE;
+
+ mmu_notifier_invalidate_range_start(mm, addr, end);
+ pml = pmd_lock(mm, pmd);
+ if (!pmd_trans_huge(*pmd))
+ goto raced;
+
+ page = head = pmd_page(*pmd);
+ pmdval = pmdp_huge_clear_flush(vma, addr, pmd);
+ pgtable = pgtable_trans_huge_withdraw(mm, pmd);
+ pmd_populate(mm, &_pmd, pgtable);
+ ptl = pte_lockptr(mm, &_pmd);
+ if (ptl != pml)
+ spin_lock(ptl);
+ pmd_populate(mm, pmd, pgtable);
+ update_mmu_cache_pmd(vma, addr, pmd);
+
+ /*
+ * It would be nice to have prepared this page table in advance,
+ * so we could just switch from pmd to ptes under one lock.
+ * But a comment in zap_huge_pmd() warns that ppc64 needs
+ * to look at the deposited page table when clearing the pmd.
+ */
+ pte = pte_offset_map(pmd, addr);
+ do {
+ pteval = pte_mkdirty(mk_pte(page, vma->vm_page_prot));
+ if (!pmd_young(pmdval))
+ pteval = pte_mkold(pteval);
+ set_pte_at(mm, addr, pte, pteval);
+ VM_BUG_ON_PAGE(!PageTeam(page), page);
+ if (page != head) {
+ page_add_file_rmap(page);
+ get_page(page);
+ }
+ /*
+ * Move page flags from head to page,
+ * as __split_huge_page_tail() does for anon?
+ * Start off by assuming not, but reconsider later.
+ */
+ } while (pte++, page++, addr += PAGE_SIZE, addr != end);
+
+ /*
+ * remap_team_by_ptes() is called from various locking contexts.
+ * Don't dec_team_pmd_mapped() until after that page table has been
+ * completed (with atomic_long_sub_return supplying a barrier):
+ * otherwise shmem_disband_hugeteam() may disband it concurrently,
+ * and pages be freed while mapped.
+ */
+ page_remove_team_rmap(head);
+
+ pte -= HPAGE_PMD_NR;
+ addr -= HPAGE_PMD_NR;
+ if (ptl != pml)
+ spin_unlock(ptl);
+ pte_unmap(pte);
+raced:
+ spin_unlock(pml);
+ mmu_notifier_invalidate_range_end(mm, addr, end);
+}
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -45,6 +45,7 @@
#include <linux/swap.h>
#include <linux/highmem.h>
#include <linux/pagemap.h>
+#include <linux/pageteam.h>
#include <linux/ksm.h>
#include <linux/rmap.h>
#include <linux/export.h>
@@ -2809,11 +2810,21 @@ static int __do_fault(struct vm_area_str
vmf.gfp_mask = __get_fault_gfp_mask(vma);
vmf.cow_page = cow_page;
+ /*
+ * Give huge pmd a chance before allocating pte or trying fault around.
+ */
+ if (unlikely(pmd_none(*pmd)))
+ vmf.flags |= FAULT_FLAG_MAY_HUGE;
+
ret = vma->vm_ops->fault(vma, &vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
if (!vmf.page)
goto out;
+ if (unlikely(ret & VM_FAULT_HUGE)) {
+ ret |= map_team_by_pmd(vma, address, pmd, vmf.page);
+ return ret;
+ }
if (unlikely(!(ret & VM_FAULT_LOCKED)))
lock_page(vmf.page);
@@ -3304,6 +3315,7 @@ static int wp_huge_pmd(struct mm_struct
return do_huge_pmd_wp_page(mm, vma, address, pmd, orig_pmd);
if (vma->vm_ops->pmd_fault)
return vma->vm_ops->pmd_fault(vma, address, pmd, flags);
+ remap_team_by_ptes(vma, address, pmd);
return VM_FAULT_FALLBACK;
}
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:30 +0200 |
| Subject | [PATCH 07/31] huge tmpfs: get_unmapped_area align & fault supply huge page |
| Message-ID | <rkGHU-1Ji-23@gated-at.bofh.it> |
| In reply to | #1371950 |
Now make the shmem.c changes necessary for mapping its huge pages into
userspace with huge pmds: without actually doing so, since that needs
changes in huge_memory.c and across mm, better left to another patch.
Provide a shmem_get_unmapped_area method in file_operations, called
at mmap time to decide the mapping address. It could be conditional
on CONFIG_TRANSPARENT_HUGEPAGE, but save #ifdefs in other places by
making it unconditional.
shmem_get_unmapped_area() first calls the usual mm->get_unmapped_area
(which we treat as a black box, highly dependent on architecture and
config and executable layout). Lots of conditions, and in most cases
it just goes with the address that chose; but when our huge stars are
rightly aligned, yet that did not provide a suitable address, go back
to ask for a larger arena, within which to align the mapping suitably.
There have to be some direct calls to shmem_get_unmapped_area(),
not via the file_operations: because of the way shmem_zero_setup()
is called to create a shmem object late in the mmap sequence, when
MAP_SHARED is requested with MAP_ANONYMOUS or /dev/zero. Though
this only matters when /proc/sys/vm/shmem_huge has been set.
Then at fault time, shmem_fault() does its usual shmem_getpage_gfp(),
and if caller __do_fault() passed FAULT_FLAG_MAY_HUGE (in later patch),
checks if the 4kB page returned is PageTeam, and, subject to further
conditions, proceeds to populate the whole of the huge page (if it
was not already fully populated and uptodate: use PG_owner_priv_1
PageChecked to save repeating all this each time the object is mapped);
then returns it to __do_fault() with a VM_FAULT_HUGE flag to request
a huge pmd.
Two conditions you might expect, which are not enforced. Originally
I intended to support just MAP_SHARED at this stage, which should be
good enough for a first implementation; but support for MAP_PRIVATE
(on read fault) needs so little further change, that it was well worth
supporting too - it opens up the opportunity to copy your x86_64 ELF
executables to huge tmpfs, their text then automatically mapped huge.
The other missing condition: shmem_getpage_gfp() is checking that
the fault falls within (4kB-rounded-up) i_size, but shmem_fault() maps
hugely even when the tail of the 2MB falls outside the (4kB-rounded-up)
i_size. This is intentional, but may need reconsideration - especially
in the MAP_PRIVATE case (is it right for a private mapping to allocate
"hidden" pages to the object beyond its EOF?). The intent is that an
application can indicate its desire for huge pmds throughout, even of
the tail, by using a hugely-rounded-up mmap size; but we might end up
retracting this, asking for fallocate to be used explicitly for that.
(hugetlbfs behaves even less standardly: its mmap extends the i_size
of the object.)
Signed-off-by: Hugh Dickins <hughd@google.com>
---
Documentation/filesystems/tmpfs.txt | 8 +
drivers/char/mem.c | 23 ++
include/linux/mm.h | 3
include/linux/shmem_fs.h | 2
ipc/shm.c | 6
mm/mmap.c | 16 +-
mm/shmem.c | 204 +++++++++++++++++++++++++-
7 files changed, 253 insertions(+), 9 deletions(-)
--- a/Documentation/filesystems/tmpfs.txt
+++ b/Documentation/filesystems/tmpfs.txt
@@ -168,6 +168,14 @@ disband the team and free those holes; o
and swap out the tmpfs pagecache. Free holes are not charged to any
memcg, and are counted in MemAvailable; but are not counted in MemFree.
+If a hugepage is mapped into a well-aligned huge extent of userspace (and
+huge tmpfs defaults to suitable alignment for any mapping large enough), any
+remaining free holes are first filled with zeroes to complete the hugepage.
+So, if the mmap length extends to a hugepage boundary beyond end of file,
+user accesses between end of file and that hugepage boundary will normally
+not fail with SIGBUS, as they would on a huge=0 filesystem - but will fail
+with SIGBUS if the kernel could only allocate small pages to back it.
+
/proc/sys/vm/shmem_huge (intended for experimentation only):
Default 0; write 1 to set tmpfs mount option huge=1 on the kernel's
--- a/drivers/char/mem.c
+++ b/drivers/char/mem.c
@@ -22,6 +22,7 @@
#include <linux/device.h>
#include <linux/highmem.h>
#include <linux/backing-dev.h>
+#include <linux/shmem_fs.h>
#include <linux/splice.h>
#include <linux/pfn.h>
#include <linux/export.h>
@@ -661,6 +662,27 @@ static int mmap_zero(struct file *file,
return 0;
}
+static unsigned long get_unmapped_area_zero(struct file *file,
+ unsigned long addr, unsigned long len,
+ unsigned long pgoff, unsigned long flags)
+{
+#ifndef CONFIG_MMU
+ return -ENOSYS;
+#endif
+ if (flags & MAP_SHARED) {
+ /*
+ * mmap_zero() will call shmem_zero_setup() to create a file,
+ * so use shmem's get_unmapped_area in case it can be huge;
+ * and pass NULL for file as in mmap.c's get_unmapped_area(),
+ * so as not to confuse shmem with our handle on "/dev/zero".
+ */
+ return shmem_get_unmapped_area(NULL, addr, len, pgoff, flags);
+ }
+
+ /* Otherwise flags & MAP_PRIVATE: with no shmem object beneath it */
+ return current->mm->get_unmapped_area(file, addr, len, pgoff, flags);
+}
+
static ssize_t write_full(struct file *file, const char __user *buf,
size_t count, loff_t *ppos)
{
@@ -768,6 +790,7 @@ static const struct file_operations zero
.read_iter = read_iter_zero,
.write_iter = write_iter_zero,
.mmap = mmap_zero,
+ .get_unmapped_area = get_unmapped_area_zero,
#ifndef CONFIG_MMU
.mmap_capabilities = zero_mmap_capabilities,
#endif
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -276,6 +276,7 @@ extern pgprot_t protection_map[16];
#define FAULT_FLAG_USER 0x40 /* The fault originated in userspace */
#define FAULT_FLAG_REMOTE 0x80 /* faulting for non current tsk/mm */
#define FAULT_FLAG_INSTRUCTION 0x100 /* The fault was during an instruction fetch */
+#define FAULT_FLAG_MAY_HUGE 0x200 /* PT not alloced: could use huge pmd */
/*
* vm_fault is filled by the the pagefault handler and passed to the vma's
@@ -1079,7 +1080,7 @@ static inline void clear_page_pfmemalloc
#define VM_FAULT_HWPOISON 0x0010 /* Hit poisoned small page */
#define VM_FAULT_HWPOISON_LARGE 0x0020 /* Hit poisoned large page. Index encoded in upper bits */
#define VM_FAULT_SIGSEGV 0x0040
-
+#define VM_FAULT_HUGE 0x0080 /* ->fault needs page installed as huge pmd */
#define VM_FAULT_NOPAGE 0x0100 /* ->fault installed the pte, not return page */
#define VM_FAULT_LOCKED 0x0200 /* ->fault locked the returned page */
#define VM_FAULT_RETRY 0x0400 /* ->fault blocked, must retry */
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -51,6 +51,8 @@ extern struct file *shmem_file_setup(con
extern struct file *shmem_kernel_file_setup(const char *name, loff_t size,
unsigned long flags);
extern int shmem_zero_setup(struct vm_area_struct *);
+extern unsigned long shmem_get_unmapped_area(struct file *, unsigned long addr,
+ unsigned long len, unsigned long pgoff, unsigned long flags);
extern int shmem_lock(struct file *file, int lock, struct user_struct *user);
extern bool shmem_mapping(struct address_space *mapping);
extern void shmem_unlock_mapping(struct address_space *mapping);
--- a/ipc/shm.c
+++ b/ipc/shm.c
@@ -476,13 +476,15 @@ static const struct file_operations shm_
.mmap = shm_mmap,
.fsync = shm_fsync,
.release = shm_release,
-#ifndef CONFIG_MMU
.get_unmapped_area = shm_get_unmapped_area,
-#endif
.llseek = noop_llseek,
.fallocate = shm_fallocate,
};
+/*
+ * shm_file_operations_huge is now identical to shm_file_operations,
+ * but we keep it distinct for the sake of is_file_shm_hugepages().
+ */
static const struct file_operations shm_file_operations_huge = {
.mmap = shm_mmap,
.fsync = shm_fsync,
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -25,6 +25,7 @@
#include <linux/personality.h>
#include <linux/security.h>
#include <linux/hugetlb.h>
+#include <linux/shmem_fs.h>
#include <linux/profile.h>
#include <linux/export.h>
#include <linux/mount.h>
@@ -1900,8 +1901,19 @@ get_unmapped_area(struct file *file, uns
return -ENOMEM;
get_area = current->mm->get_unmapped_area;
- if (file && file->f_op->get_unmapped_area)
- get_area = file->f_op->get_unmapped_area;
+ if (file) {
+ if (file->f_op->get_unmapped_area)
+ get_area = file->f_op->get_unmapped_area;
+ } else if (flags & MAP_SHARED) {
+ /*
+ * mmap_region() will call shmem_zero_setup() to create a file,
+ * so use shmem's get_unmapped_area in case it can be huge.
+ * do_mmap_pgoff() will clear pgoff, so match alignment.
+ */
+ pgoff = 0;
+ get_area = shmem_get_unmapped_area;
+ }
+
addr = get_area(file, addr, len, pgoff, flags);
if (IS_ERR_VALUE(addr))
return addr;
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -104,6 +104,7 @@ struct shmem_falloc {
enum sgp_type {
SGP_READ, /* don't exceed i_size, don't allocate page */
SGP_CACHE, /* don't exceed i_size, may allocate page */
+ SGP_TEAM, /* may exceed i_size, may make team page Uptodate */
SGP_WRITE, /* may exceed i_size, may allocate !Uptodate page */
SGP_FALLOC, /* like SGP_WRITE, but make existing page Uptodate */
};
@@ -417,6 +418,44 @@ static void shmem_added_to_hugeteam(stru
}
}
+static int shmem_populate_hugeteam(struct inode *inode, struct page *head,
+ struct vm_area_struct *vma)
+{
+ gfp_t gfp = mapping_gfp_mask(inode->i_mapping);
+ struct page *page;
+ pgoff_t index;
+ int error;
+ int i;
+
+ /* We only have to do this once */
+ if (PageChecked(head))
+ return 0;
+
+ index = head->index;
+ for (i = 0; i < HPAGE_PMD_NR; i++, index++) {
+ if (!PageTeam(head))
+ return -EAGAIN;
+ if (PageChecked(head))
+ return 0;
+ /* Mark all pages dirty even when map is readonly, for now */
+ if (PageUptodate(head + i) && PageDirty(head + i))
+ continue;
+ error = shmem_getpage_gfp(inode, index, &page, SGP_TEAM,
+ gfp, vma->vm_mm, NULL);
+ if (error)
+ return error;
+ SetPageDirty(page);
+ unlock_page(page);
+ put_page(page);
+ if (page != head + i)
+ return -EAGAIN;
+ cond_resched();
+ }
+
+ /* Now safe from the shrinker, but not yet from truncate */
+ return 0;
+}
+
static int shmem_disband_hugehead(struct page *head)
{
struct address_space *mapping;
@@ -452,6 +491,7 @@ static int shmem_disband_hugehead(struct
head->mapping = NULL;
if (nr >= HPAGE_PMD_NR) {
+ ClearPageChecked(head);
__dec_zone_state(zone, NR_SHMEM_HUGEPAGES);
VM_BUG_ON(nr != HPAGE_PMD_NR);
} else if (nr) {
@@ -843,6 +883,12 @@ static inline void shmem_added_to_hugete
{
}
+static inline int shmem_populate_hugeteam(struct inode *inode,
+ struct page *head, struct vm_area_struct *vma)
+{
+ return -EAGAIN;
+}
+
static inline unsigned long shmem_shrink_hugehole(struct shrinker *shrink,
struct shrink_control *sc)
{
@@ -1817,8 +1863,8 @@ static int shmem_replace_page(struct pag
* vm. If we swap it in we mark it dirty since we also free the swap
* entry since a page cannot live in both the swap and page cache.
*
- * fault_mm and fault_type are only supplied by shmem_fault:
- * otherwise they are NULL.
+ * fault_mm and fault_type are only supplied by shmem_fault
+ * (or hugeteam population): otherwise they are NULL.
*/
static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
struct page **pagep, enum sgp_type sgp, gfp_t gfp,
@@ -2095,10 +2141,13 @@ unlock:
static int shmem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
+ unsigned long addr = (unsigned long)vmf->virtual_address;
struct inode *inode = file_inode(vma->vm_file);
gfp_t gfp = mapping_gfp_mask(inode->i_mapping);
+ struct page *head;
+ int ret = 0;
+ int once = 0;
int error;
- int ret = VM_FAULT_LOCKED;
/*
* Trinity finds that probing a hole which tmpfs is punching can
@@ -2158,11 +2207,150 @@ static int shmem_fault(struct vm_area_st
spin_unlock(&inode->i_lock);
}
+single:
+ vmf->page = NULL;
error = shmem_getpage_gfp(inode, vmf->pgoff, &vmf->page, SGP_CACHE,
gfp, vma->vm_mm, &ret);
if (error)
return ((error == -ENOMEM) ? VM_FAULT_OOM : VM_FAULT_SIGBUS);
- return ret;
+ ret |= VM_FAULT_LOCKED;
+
+ /*
+ * Shall we map a huge page hugely?
+ */
+ if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
+ return ret;
+ if (!(vmf->flags & FAULT_FLAG_MAY_HUGE))
+ return ret;
+ if (!PageTeam(vmf->page))
+ return ret;
+ if (once++)
+ return ret;
+ if (!(vma->vm_flags & VM_SHARED) && (vmf->flags & FAULT_FLAG_WRITE))
+ return ret;
+ if ((vma->vm_start-(vma->vm_pgoff<<PAGE_SHIFT)) & (HPAGE_PMD_SIZE-1))
+ return ret;
+ if (round_down(addr, HPAGE_PMD_SIZE) < vma->vm_start)
+ return ret;
+ if (round_up(addr + 1, HPAGE_PMD_SIZE) > vma->vm_end)
+ return ret;
+ /* But omit i_size check: allow up to huge page boundary */
+
+ head = team_head(vmf->page);
+ if (!get_page_unless_zero(head))
+ return ret;
+ if (!PageTeam(head)) {
+ put_page(head);
+ return ret;
+ }
+
+ ret &= ~VM_FAULT_LOCKED;
+ unlock_page(vmf->page);
+ put_page(vmf->page);
+ if (shmem_populate_hugeteam(inode, head, vma) < 0) {
+ put_page(head);
+ goto single;
+ }
+ lock_page(head);
+ if (!PageTeam(head)) {
+ unlock_page(head);
+ put_page(head);
+ goto single;
+ }
+ if (!PageChecked(head))
+ SetPageChecked(head);
+
+ /* Now safe from truncation */
+ vmf->page = head;
+ return ret | VM_FAULT_LOCKED | VM_FAULT_HUGE;
+}
+
+unsigned long shmem_get_unmapped_area(struct file *file,
+ unsigned long uaddr, unsigned long len,
+ unsigned long pgoff, unsigned long flags)
+{
+ unsigned long (*get_area)(struct file *,
+ unsigned long, unsigned long, unsigned long, unsigned long);
+ unsigned long addr;
+ unsigned long offset;
+ unsigned long inflated_len;
+ unsigned long inflated_addr;
+ unsigned long inflated_offset;
+
+ if (len > TASK_SIZE)
+ return -ENOMEM;
+
+ get_area = current->mm->get_unmapped_area;
+ addr = get_area(file, uaddr, len, pgoff, flags);
+
+ if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
+ return addr;
+ if (IS_ERR_VALUE(addr))
+ return addr;
+ if (addr & ~PAGE_MASK)
+ return addr;
+ if (addr > TASK_SIZE - len)
+ return addr;
+
+ if (shmem_huge == SHMEM_HUGE_DENY)
+ return addr;
+ if (len < HPAGE_PMD_SIZE)
+ return addr;
+ if (flags & MAP_FIXED)
+ return addr;
+ /*
+ * Our priority is to support MAP_SHARED mapped hugely;
+ * and support MAP_PRIVATE mapped hugely too, until it is COWed.
+ * But if caller specified an address hint, respect that as before.
+ */
+ if (uaddr)
+ return addr;
+
+ if (shmem_huge != SHMEM_HUGE_FORCE) {
+ struct super_block *sb;
+
+ if (file) {
+ VM_BUG_ON(file->f_op != &shmem_file_operations);
+ sb = file_inode(file)->i_sb;
+ } else {
+ /*
+ * Called directly from mm/mmap.c, or drivers/char/mem.c
+ * for "/dev/zero", to create a shared anonymous object.
+ */
+ if (IS_ERR(shm_mnt))
+ return addr;
+ sb = shm_mnt->mnt_sb;
+ }
+ if (!SHMEM_SB(sb)->huge)
+ return addr;
+ }
+
+ offset = (pgoff << PAGE_SHIFT) & (HPAGE_PMD_SIZE-1);
+ if (offset && offset + len < 2 * HPAGE_PMD_SIZE)
+ return addr;
+ if ((addr & (HPAGE_PMD_SIZE-1)) == offset)
+ return addr;
+
+ inflated_len = len + HPAGE_PMD_SIZE - PAGE_SIZE;
+ if (inflated_len > TASK_SIZE)
+ return addr;
+ if (inflated_len < len)
+ return addr;
+
+ inflated_addr = get_area(NULL, 0, inflated_len, 0, flags);
+ if (IS_ERR_VALUE(inflated_addr))
+ return addr;
+ if (inflated_addr & ~PAGE_MASK)
+ return addr;
+
+ inflated_offset = inflated_addr & (HPAGE_PMD_SIZE-1);
+ inflated_addr += offset - inflated_offset;
+ if (inflated_offset > offset)
+ inflated_addr += HPAGE_PMD_SIZE;
+
+ if (inflated_addr > TASK_SIZE - len)
+ return addr;
+ return inflated_addr;
}
#ifdef CONFIG_NUMA
@@ -3905,6 +4093,7 @@ static const struct address_space_operat
static const struct file_operations shmem_file_operations = {
.mmap = shmem_mmap,
+ .get_unmapped_area = shmem_get_unmapped_area,
#ifdef CONFIG_TMPFS
.llseek = shmem_file_llseek,
.read_iter = shmem_file_read_iter,
@@ -4112,6 +4301,13 @@ void shmem_unlock_mapping(struct address
{
}
+unsigned long shmem_get_unmapped_area(struct file *file,
+ unsigned long addr, unsigned long len,
+ unsigned long pgoff, unsigned long flags)
+{
+ return current->mm->get_unmapped_area(file, addr, len, pgoff, flags);
+}
+
void shmem_truncate_range(struct inode *inode, loff_t lstart, loff_t lend)
{
truncate_inode_pages_range(inode->i_mapping, lstart, lend);
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:30 +0200 |
| Subject | [PATCH 09/31] huge tmpfs: avoid premature exposure of new pagetable |
| Message-ID | <rkGHU-1Ji-33@gated-at.bofh.it> |
| In reply to | #1371950 |
In early development, a huge tmpfs fault simply replaced the pmd which
pointed to the empty pagetable just allocated in __handle_mm_fault():
but that is unsafe.
Andrea wrote a very interesting comment on THP in mm/memory.c,
just before the end of __handle_mm_fault():
* A regular pmd is established and it can't morph into a huge pmd
* from under us anymore at this point because we hold the mmap_sem
* read mode and khugepaged takes it in write mode. So now it's
* safe to run pte_offset_map().
This comment hints at several difficulties, which anon THP solved
for itself with mmap_sem and anon_vma lock, but which huge tmpfs
may need to solve differently.
The reference to pte_offset_map() above: I believe that's a hint
that on a 32-bit machine, the pagetables might need to come from
kernel-mapped memory, but a huge pmd pointing to user memory beyond
that limit could be racily substituted, causing undefined behavior
in the architecture-dependent pte_offset_map().
That itself is not a problem on x86_64, but there's plenty more:
how about those places which use pte_offset_map_lock() - if that
spinlock is in the struct page of a pagetable, which has been
deposited and might be withdrawn and freed at any moment (being
on a list unattached to the allocating pmd in the case of x86),
taking the spinlock might corrupt someone else's struct page.
Because THP has departed from the earlier rules (when pagetable
was only freed under exclusive mmap_sem, or at exit_mmap, after
removing all affected vmas from the rmap list): zap_huge_pmd()
does pte_free() even when serving MADV_DONTNEED under down_read
of mmap_sem.
And what of the "entry = *pte" at the start of handle_pte_fault(),
getting the entry used in pte_same(,orig_pte) tests to validate all
fault handling? If that entry can itself be junk picked out of some
freed and reused pagetable, it's hard to estimate the consequences.
We need to consider the safety of concurrent faults, and the
safety of rmap lookups, and the safety of miscellaneous operations
such as smaps_pte_range() for reading /proc/<pid>/smaps.
I set out to make safe the places which descend pgd,pud,pmd,pte,
using more careful access techniques like mm_find_pmd(); but with
pte_offset_map() being architecture-defined, found it too big a job
to tighten up all over.
Instead, approach from the opposite direction: just do not expose
a pagetable in an empty *pmd, until vm_ops->fault has had a chance
to ask for a huge pmd there. This is a much easier change to make,
and we are lucky that all the driver faults appear to be using
interfaces (like vm_insert_page() and remap_pfn_range()) which
automatically do the pte_alloc() if it was not already done.
But we must not get stuck refaulting: need FAULT_FLAG_MAY_HUGE for
__do_fault() to tell shmem_fault() to try for huge only when *pmd is
empty (could instead add pmd to vmf and let shmem work that out for
itself, but probably better to hide pmd from vm_ops->faults).
Without a pagetable to hold the pte_none() entry found in a newly
allocated pagetable, handle_pte_fault() would like to provide a static
none entry for later orig_pte checks. But architectures have never had
to provide that definition before; and although almost all use zeroes
for an empty pagetable, a few do not - nios2, s390, um, xtensa.
Never mind, forget about pte_same(,orig_pte), the three __do_fault()
callers can follow do_anonymous_page(), and just use a pte_none() check.
do_fault_around() presents one last problem: it wants pagetable to
have been allocated, but was being called by do_read_fault() before
__do_fault(). I see no disadvantage to moving it after, allowing huge
pmd to be chosen first; but Kirill reports additional radix-tree lookup
in hot pagecache case when he implemented faultaround: needs further
investigation.
Note: after months of use, we recently hit an OOM deadlock: this patch
moves the new pagetable allocation inside where page lock is held on a
pagecache page, and exit's munlock_vma_pages_all() takes page lock on
all mlocked pages. Both parties are behaving badly: we hope to change
munlock to use trylock_page() instead, but should certainly switch here
to preallocating the pagetable outside the page lock. But I've not yet
written and tested that change.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
mm/filemap.c | 10 +-
mm/memory.c | 215 ++++++++++++++++++++++++++-----------------------
2 files changed, 123 insertions(+), 102 deletions(-)
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2147,6 +2147,10 @@ void filemap_map_pages(struct vm_area_st
radix_tree_for_each_slot(slot, &mapping->page_tree, &iter, vmf->pgoff) {
if (iter.index > vmf->max_pgoff)
break;
+
+ pte = vmf->pte + iter.index - vmf->pgoff;
+ if (!pte_none(*pte))
+ goto next;
repeat:
page = radix_tree_deref_slot(slot);
if (unlikely(!page))
@@ -2168,6 +2172,8 @@ repeat:
goto repeat;
}
+ VM_BUG_ON_PAGE(page->index != iter.index, page);
+
if (!PageUptodate(page) ||
PageReadahead(page) ||
PageHWPoison(page))
@@ -2182,10 +2188,6 @@ repeat:
if (page->index >= size >> PAGE_SHIFT)
goto unlock;
- pte = vmf->pte + page->index - vmf->pgoff;
- if (!pte_none(*pte))
- goto unlock;
-
if (file->f_ra.mmap_miss > 0)
file->f_ra.mmap_miss--;
addr = address + (page->index - vmf->pgoff) * PAGE_SIZE;
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2678,20 +2678,17 @@ static inline int check_stack_guard_page
/*
* We enter with non-exclusive mmap_sem (to exclude vma changes,
- * but allow concurrent faults), and pte mapped but not yet locked.
- * We return with mmap_sem still held, but pte unmapped and unlocked.
+ * but allow concurrent faults). We return with mmap_sem still held.
*/
static int do_anonymous_page(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long address, pte_t *page_table, pmd_t *pmd,
- unsigned int flags)
+ unsigned long address, pmd_t *pmd, unsigned int flags)
{
struct mem_cgroup *memcg;
+ pte_t *page_table;
struct page *page;
spinlock_t *ptl;
pte_t entry;
- pte_unmap(page_table);
-
/* File mapping without ->vm_ops ? */
if (vma->vm_flags & VM_SHARED)
return VM_FAULT_SIGBUS;
@@ -2700,6 +2697,27 @@ static int do_anonymous_page(struct mm_s
if (check_stack_guard_page(vma, address) < 0)
return VM_FAULT_SIGSEGV;
+ /*
+ * Use pte_alloc instead of pte_alloc_map, because we can't
+ * run pte_offset_map on the pmd, if an huge pmd could
+ * materialize from under us from a different thread.
+ */
+ if (unlikely(pte_alloc(mm, pmd, address)))
+ return VM_FAULT_OOM;
+ /*
+ * If a huge pmd materialized under us just retry later. Use
+ * pmd_trans_unstable() instead of pmd_trans_huge() to ensure the pmd
+ * didn't become pmd_trans_huge under us and then back to pmd_none, as
+ * a result of MADV_DONTNEED running immediately after a huge pmd fault
+ * in a different thread of this mm, in turn leading to a misleading
+ * pmd_trans_huge() retval. All we have to ensure is that it is a
+ * regular pmd that we can walk with pte_offset_map() and we can do that
+ * through an atomic read in C, which is what pmd_trans_unstable()
+ * provides.
+ */
+ if (unlikely(pmd_trans_unstable(pmd) || pmd_devmap(*pmd)))
+ return 0;
+
/* Use the zero-page for reads */
if (!(flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(mm)) {
entry = pte_mkspecial(pfn_pte(my_zero_pfn(address),
@@ -2778,8 +2796,8 @@ oom:
* See filemap_fault() and __lock_page_retry().
*/
static int __do_fault(struct vm_area_struct *vma, unsigned long address,
- pgoff_t pgoff, unsigned int flags,
- struct page *cow_page, struct page **page)
+ pmd_t *pmd, pgoff_t pgoff, unsigned int flags,
+ struct page *cow_page, struct page **page)
{
struct vm_fault vmf;
int ret;
@@ -2797,21 +2815,40 @@ static int __do_fault(struct vm_area_str
if (!vmf.page)
goto out;
- if (unlikely(PageHWPoison(vmf.page))) {
- if (ret & VM_FAULT_LOCKED)
- unlock_page(vmf.page);
- put_page(vmf.page);
- return VM_FAULT_HWPOISON;
- }
-
if (unlikely(!(ret & VM_FAULT_LOCKED)))
lock_page(vmf.page);
else
VM_BUG_ON_PAGE(!PageLocked(vmf.page), vmf.page);
+ if (unlikely(PageHWPoison(vmf.page))) {
+ ret = VM_FAULT_HWPOISON;
+ goto err;
+ }
+
+ /*
+ * Use pte_alloc instead of pte_alloc_map, because we can't
+ * run pte_offset_map on the pmd, if an huge pmd could
+ * materialize from under us from a different thread.
+ */
+ if (unlikely(pte_alloc(vma->vm_mm, pmd, address))) {
+ ret = VM_FAULT_OOM;
+ goto err;
+ }
+ /*
+ * If a huge pmd materialized under us just retry later. Allow for
+ * a racing transition of huge pmd to none to huge pmd or pagetable.
+ */
+ if (unlikely(pmd_trans_unstable(pmd) || pmd_devmap(*pmd))) {
+ ret = VM_FAULT_NOPAGE;
+ goto err;
+ }
out:
*page = vmf.page;
return ret;
+err:
+ unlock_page(vmf.page);
+ put_page(vmf.page);
+ return ret;
}
/**
@@ -2961,32 +2998,19 @@ static void do_fault_around(struct vm_ar
static int do_read_fault(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long address, pmd_t *pmd,
- pgoff_t pgoff, unsigned int flags, pte_t orig_pte)
+ pgoff_t pgoff, unsigned int flags)
{
struct page *fault_page;
spinlock_t *ptl;
pte_t *pte;
- int ret = 0;
-
- /*
- * Let's call ->map_pages() first and use ->fault() as fallback
- * if page by the offset is not ready to be mapped (cold cache or
- * something).
- */
- if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
- pte = pte_offset_map_lock(mm, pmd, address, &ptl);
- do_fault_around(vma, address, pte, pgoff, flags);
- if (!pte_same(*pte, orig_pte))
- goto unlock_out;
- pte_unmap_unlock(pte, ptl);
- }
+ int ret;
- ret = __do_fault(vma, address, pgoff, flags, NULL, &fault_page);
+ ret = __do_fault(vma, address, pmd, pgoff, flags, NULL, &fault_page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
pte = pte_offset_map_lock(mm, pmd, address, &ptl);
- if (unlikely(!pte_same(*pte, orig_pte))) {
+ if (unlikely(!pte_none(*pte))) {
pte_unmap_unlock(pte, ptl);
unlock_page(fault_page);
put_page(fault_page);
@@ -2994,14 +3018,20 @@ static int do_read_fault(struct mm_struc
}
do_set_pte(vma, address, fault_page, pte, false, false);
unlock_page(fault_page);
-unlock_out:
+
+ /*
+ * Finally call ->map_pages() to fault around the pte we just set.
+ */
+ if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1)
+ do_fault_around(vma, address, pte, pgoff, flags);
+
pte_unmap_unlock(pte, ptl);
return ret;
}
static int do_cow_fault(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long address, pmd_t *pmd,
- pgoff_t pgoff, unsigned int flags, pte_t orig_pte)
+ pgoff_t pgoff, unsigned int flags)
{
struct page *fault_page, *new_page;
struct mem_cgroup *memcg;
@@ -3021,7 +3051,7 @@ static int do_cow_fault(struct mm_struct
return VM_FAULT_OOM;
}
- ret = __do_fault(vma, address, pgoff, flags, new_page, &fault_page);
+ ret = __do_fault(vma, address, pmd, pgoff, flags, new_page, &fault_page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
@@ -3030,7 +3060,7 @@ static int do_cow_fault(struct mm_struct
__SetPageUptodate(new_page);
pte = pte_offset_map_lock(mm, pmd, address, &ptl);
- if (unlikely(!pte_same(*pte, orig_pte))) {
+ if (unlikely(!pte_none(*pte))) {
pte_unmap_unlock(pte, ptl);
if (fault_page) {
unlock_page(fault_page);
@@ -3067,7 +3097,7 @@ uncharge_out:
static int do_shared_fault(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long address, pmd_t *pmd,
- pgoff_t pgoff, unsigned int flags, pte_t orig_pte)
+ pgoff_t pgoff, unsigned int flags)
{
struct page *fault_page;
struct address_space *mapping;
@@ -3076,7 +3106,7 @@ static int do_shared_fault(struct mm_str
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(vma, address, pgoff, flags, NULL, &fault_page);
+ ret = __do_fault(vma, address, pmd, pgoff, flags, NULL, &fault_page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3095,7 +3125,7 @@ static int do_shared_fault(struct mm_str
}
pte = pte_offset_map_lock(mm, pmd, address, &ptl);
- if (unlikely(!pte_same(*pte, orig_pte))) {
+ if (unlikely(!pte_none(*pte))) {
pte_unmap_unlock(pte, ptl);
unlock_page(fault_page);
put_page(fault_page);
@@ -3135,22 +3165,18 @@ static int do_shared_fault(struct mm_str
* return value. See filemap_fault() and __lock_page_or_retry().
*/
static int do_fault(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long address, pte_t *page_table, pmd_t *pmd,
- unsigned int flags, pte_t orig_pte)
+ unsigned long address, pmd_t *pmd, unsigned int flags)
{
pgoff_t pgoff = linear_page_index(vma, address);
- pte_unmap(page_table);
/* The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */
if (!vma->vm_ops->fault)
return VM_FAULT_SIGBUS;
if (!(flags & FAULT_FLAG_WRITE))
- return do_read_fault(mm, vma, address, pmd, pgoff, flags,
- orig_pte);
+ return do_read_fault(mm, vma, address, pmd, pgoff, flags);
if (!(vma->vm_flags & VM_SHARED))
- return do_cow_fault(mm, vma, address, pmd, pgoff, flags,
- orig_pte);
- return do_shared_fault(mm, vma, address, pmd, pgoff, flags, orig_pte);
+ return do_cow_fault(mm, vma, address, pmd, pgoff, flags);
+ return do_shared_fault(mm, vma, address, pmd, pgoff, flags);
}
static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
@@ -3290,20 +3316,49 @@ static int wp_huge_pmd(struct mm_struct
* with external mmu caches can use to update those (ie the Sparc or
* PowerPC hashed page tables that act as extended TLBs).
*
- * We enter with non-exclusive mmap_sem (to exclude vma changes,
- * but allow concurrent faults), and pte mapped but not yet locked.
- * We return with pte unmapped and unlocked.
- *
+ * We enter with non-exclusive mmap_sem
+ * (to exclude vma changes, but allow concurrent faults).
* The mmap_sem may have been released depending on flags and our
* return value. See filemap_fault() and __lock_page_or_retry().
*/
-static int handle_pte_fault(struct mm_struct *mm,
- struct vm_area_struct *vma, unsigned long address,
- pte_t *pte, pmd_t *pmd, unsigned int flags)
+static int handle_pte_fault(struct mm_struct *mm, struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmd, unsigned int flags)
{
+ pmd_t pmdval;
+ pte_t *pte;
pte_t entry;
spinlock_t *ptl;
+ /* If a huge pmd materialized under us just retry later */
+ pmdval = *pmd;
+ barrier();
+ if (unlikely(pmd_trans_huge(pmdval) || pmd_devmap(pmdval)))
+ return 0;
+
+ if (unlikely(pmd_none(pmdval))) {
+ /*
+ * Leave pte_alloc() until later: because huge tmpfs may
+ * want to map_team_by_pmd(), and if we expose page table
+ * for an instant, it will be difficult to retract from
+ * concurrent faults and from rmap lookups.
+ */
+ pte = NULL;
+ } else {
+ /*
+ * A regular pmd is established and it can't morph into a huge
+ * pmd from under us anymore at this point because we hold the
+ * mmap_sem read mode and khugepaged takes it in write mode.
+ * So now it's safe to run pte_offset_map().
+ */
+ pte = pte_offset_map(pmd, address);
+ entry = *pte;
+ barrier();
+ if (pte_none(entry)) {
+ pte_unmap(pte);
+ pte = NULL;
+ }
+ }
+
/*
* some architectures can have larger ptes than wordsize,
* e.g.ppc44x-defconfig has CONFIG_PTE_64BIT=y and CONFIG_32BIT=y,
@@ -3312,21 +3367,14 @@ static int handle_pte_fault(struct mm_st
* we later double check anyway with the ptl lock held. So here
* a barrier will do.
*/
- entry = *pte;
- barrier();
- if (!pte_present(entry)) {
- if (pte_none(entry)) {
- if (vma_is_anonymous(vma))
- return do_anonymous_page(mm, vma, address,
- pte, pmd, flags);
- else
- return do_fault(mm, vma, address, pte, pmd,
- flags, entry);
- }
- return do_swap_page(mm, vma, address,
- pte, pmd, flags, entry);
- }
+ if (!pte) {
+ if (!vma_is_anonymous(vma))
+ return do_fault(mm, vma, address, pmd, flags);
+ return do_anonymous_page(mm, vma, address, pmd, flags);
+ }
+ if (!pte_present(entry))
+ return do_swap_page(mm, vma, address, pte, pmd, flags, entry);
if (pte_protnone(entry))
return do_numa_page(mm, vma, address, entry, pte, pmd);
@@ -3370,7 +3418,6 @@ static int __handle_mm_fault(struct mm_s
pgd_t *pgd;
pud_t *pud;
pmd_t *pmd;
- pte_t *pte;
if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE,
flags & FAULT_FLAG_INSTRUCTION,
@@ -3416,35 +3463,7 @@ static int __handle_mm_fault(struct mm_s
}
}
- /*
- * Use pte_alloc() instead of pte_alloc_map, because we can't
- * run pte_offset_map on the pmd, if an huge pmd could
- * materialize from under us from a different thread.
- */
- if (unlikely(pte_alloc(mm, pmd, address)))
- return VM_FAULT_OOM;
- /*
- * If a huge pmd materialized under us just retry later. Use
- * pmd_trans_unstable() instead of pmd_trans_huge() to ensure the pmd
- * didn't become pmd_trans_huge under us and then back to pmd_none, as
- * a result of MADV_DONTNEED running immediately after a huge pmd fault
- * in a different thread of this mm, in turn leading to a misleading
- * pmd_trans_huge() retval. All we have to ensure is that it is a
- * regular pmd that we can walk with pte_offset_map() and we can do that
- * through an atomic read in C, which is what pmd_trans_unstable()
- * provides.
- */
- if (unlikely(pmd_trans_unstable(pmd) || pmd_devmap(*pmd)))
- return 0;
- /*
- * A regular pmd is established and it can't morph into a huge pmd
- * from under us anymore at this point because we hold the mmap_sem
- * read mode and khugepaged takes it in write mode. So now it's
- * safe to run pte_offset_map().
- */
- pte = pte_offset_map(pmd, address);
-
- return handle_pte_fault(mm, vma, address, pte, pmd, flags);
+ return handle_pte_fault(mm, vma, address, pmd, flags);
}
/*
[toc] | [prev] | [next] | [standalone]
| From | "Kirill A. Shutemov" <kirill@shutemov.name> |
|---|---|
| Date | 2016-04-11 14:00 +0200 |
| Subject | Re: [PATCH 09/31] huge tmpfs: avoid premature exposure of new pagetable |
| Message-ID | <rmIFA-5ju-11@gated-at.bofh.it> |
| In reply to | #1371963 |
On Tue, Apr 05, 2016 at 02:24:23PM -0700, Hugh Dickins wrote: > In early development, a huge tmpfs fault simply replaced the pmd which > pointed to the empty pagetable just allocated in __handle_mm_fault(): > but that is unsafe. > > Andrea wrote a very interesting comment on THP in mm/memory.c, > just before the end of __handle_mm_fault(): > > * A regular pmd is established and it can't morph into a huge pmd > * from under us anymore at this point because we hold the mmap_sem > * read mode and khugepaged takes it in write mode. So now it's > * safe to run pte_offset_map(). > > This comment hints at several difficulties, which anon THP solved > for itself with mmap_sem and anon_vma lock, but which huge tmpfs > may need to solve differently. > > The reference to pte_offset_map() above: I believe that's a hint > that on a 32-bit machine, the pagetables might need to come from > kernel-mapped memory, but a huge pmd pointing to user memory beyond > that limit could be racily substituted, causing undefined behavior > in the architecture-dependent pte_offset_map(). > > That itself is not a problem on x86_64, but there's plenty more: > how about those places which use pte_offset_map_lock() - if that > spinlock is in the struct page of a pagetable, which has been > deposited and might be withdrawn and freed at any moment (being > on a list unattached to the allocating pmd in the case of x86), > taking the spinlock might corrupt someone else's struct page. > > Because THP has departed from the earlier rules (when pagetable > was only freed under exclusive mmap_sem, or at exit_mmap, after > removing all affected vmas from the rmap list): zap_huge_pmd() > does pte_free() even when serving MADV_DONTNEED under down_read > of mmap_sem. Emm.. The pte table freed from zap_huge_pmd() is from deposit. It wasn't linked into process' page table tree. So I don't see how THP has departed from the rules. I don't think it changes anything to implementation, but this part of commit message, I believe, is inaccurate. > And what of the "entry = *pte" at the start of handle_pte_fault(), > getting the entry used in pte_same(,orig_pte) tests to validate all > fault handling? If that entry can itself be junk picked out of some > freed and reused pagetable, it's hard to estimate the consequences. > > We need to consider the safety of concurrent faults, and the > safety of rmap lookups, and the safety of miscellaneous operations > such as smaps_pte_range() for reading /proc/<pid>/smaps. > > I set out to make safe the places which descend pgd,pud,pmd,pte, > using more careful access techniques like mm_find_pmd(); but with > pte_offset_map() being architecture-defined, found it too big a job > to tighten up all over. > > Instead, approach from the opposite direction: just do not expose > a pagetable in an empty *pmd, until vm_ops->fault has had a chance > to ask for a huge pmd there. This is a much easier change to make, > and we are lucky that all the driver faults appear to be using > interfaces (like vm_insert_page() and remap_pfn_range()) which > automatically do the pte_alloc() if it was not already done. > > But we must not get stuck refaulting: need FAULT_FLAG_MAY_HUGE for > __do_fault() to tell shmem_fault() to try for huge only when *pmd is > empty (could instead add pmd to vmf and let shmem work that out for > itself, but probably better to hide pmd from vm_ops->faults). > > Without a pagetable to hold the pte_none() entry found in a newly > allocated pagetable, handle_pte_fault() would like to provide a static > none entry for later orig_pte checks. But architectures have never had > to provide that definition before; and although almost all use zeroes > for an empty pagetable, a few do not - nios2, s390, um, xtensa. > > Never mind, forget about pte_same(,orig_pte), the three __do_fault() > callers can follow do_anonymous_page(), and just use a pte_none() check. > > do_fault_around() presents one last problem: it wants pagetable to > have been allocated, but was being called by do_read_fault() before > __do_fault(). I see no disadvantage to moving it after, allowing huge > pmd to be chosen first; but Kirill reports additional radix-tree lookup > in hot pagecache case when he implemented faultaround: needs further > investigation. In my implementation faultaround can establish PMD mappings. So there's no disadvantage to call faultaround first. And if faultaround happened to solve the page fault we don't need to do usual ->fault lookup. > Note: after months of use, we recently hit an OOM deadlock: this patch > moves the new pagetable allocation inside where page lock is held on a > pagecache page, and exit's munlock_vma_pages_all() takes page lock on > all mlocked pages. Both parties are behaving badly: we hope to change > munlock to use trylock_page() instead, but should certainly switch here > to preallocating the pagetable outside the page lock. But I've not yet > written and tested that change. Hm. Okay, I need to fix this in my implementation too. It shouldn't be too hard as I have fe->pte_prealloc around already. -- Kirill A. Shutemov
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:40 +0200 |
| Subject | [PATCH 13/31] huge tmpfs: use Unevictable lru with variable hpage_nr_pages |
| Message-ID | <rkGRA-1Og-9@gated-at.bofh.it> |
| In reply to | #1371950 |
A big advantage of huge tmpfs over hugetlbfs is that its pages can
be swapped out; but too often it OOMs before swapping them out.
At first I tried changing page_evictable(), to treat all tail pages
of a hugely mapped team as unevictable: the anon LRUs were otherwise
swamped by pages that could not be freed before the head.
That worked quite well, some of the time, but has some drawbacks.
Most obviously, /proc/meminfo is liable to show 511/512ths of all
the ShmemPmdMapped as Unevictable; which is rather sad for a feature
intended to improve on hugetlbfs by letting the pages be swappable.
But more seriously, although it is helpful to have those tails out
of the way on the Unevictable list, page reclaim can very easily come
to a point where all the team heads to be freed are on the Active list,
but the Inactive is large enough that !inactive_anon_is_low(), so the
Active is never scanned to unmap those heads to release all the tails.
Eventually we OOM.
Perhaps that could be dealt with by hacking inactive_anon_is_low():
but it wouldn't help the Unevictable numbers, and has never been
necessary for anon THP. How does anon THP avoid this? It doesn't
put tails on the LRU at all, so doesn't then need to shift them to
Unevictable; but there would still be the danger of an Active list
full of heads, holding the unseen tails, but the ratio too high for
for Active scanning - except that hpage_nr_pages() weights each THP
head by the number of small pages the huge page holds, instead of the
usual 1, and that is what keeps the Active/Inactive balance working.
So in this patch we try to do the same for huge tmpfs pages. However,
a team is not one huge compound page, but a collection of independent
pages, and the fair and lazy way to accomplish this seems to be to
transfer each tail's weight to head at the time when shmem_writepage()
has been asked to evict the page, but refuses because the head has not
yet been evicted. So although the failed-to-be-evicted tails are moved
to the Unevictable LRU, each counts for 0kB in the Unevictable amount,
its 4kB going to the head in the Active(anon) or Inactive(anon) amount.
With a few exceptions, hpage_nr_pages() is now only called on a
maybe-PageTeam page while under lruvec lock: and we do need to hold
lruvec lock when transferring weight from one page to another.
Exceptions: mlock.c (next patch), subsequently self-correcting calls to
page_evictable(), and the "nr_rotated +=" line in shrink_active_list(),
which has no need to be precise.
(Aside: quite a few of our calls to hpage_nr_pages() are no more than
ways to side-step the THP-off BUILD_BUG_ON() buried in HPAGE_PMD_NR:
we might do better to kill that BUILD_BUG_ON() at last.)
Lru lock is a new overhead, which shmem_disband_hugehead() prefers
to avoid, if the head's weight is just the default 1. And it's not
clear how well this will all play out if different pages of a team
are charged to different memcgs: but the code allows for that, and
it should be fine while that's just an exceptional minority case.
A change I like in principle, but have not made, and do not intend
to make unless we see a workload that demands it: it would be natural
for mark_page_accessed() to retrieve such a 0-weight page from the
Unevictable LRU, assigning it weight again and giving it a new life
on the Active and Inactive LRUs. As it is, I'm hoping PageReferenced
gives a good enough hint as to whether a page should be retained, when
shmem_evictify_hugetails() brings it back from Unevictable to Inactive.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
Documentation/vm/unevictable-lru.txt | 15 ++
include/linux/huge_mm.h | 14 ++
include/linux/pageteam.h | 48 ++++++
mm/memcontrol.c | 10 +
mm/shmem.c | 173 +++++++++++++++++++++----
mm/swap.c | 5
mm/vmscan.c | 39 +++++
7 files changed, 274 insertions(+), 30 deletions(-)
--- a/Documentation/vm/unevictable-lru.txt
+++ b/Documentation/vm/unevictable-lru.txt
@@ -72,6 +72,8 @@ The unevictable list addresses the follo
(*) Those mapped into VM_LOCKED [mlock()ed] VMAs.
+ (*) Tails owned by huge tmpfs, unevictable until team head page is evicted.
+
The infrastructure may also be able to handle other conditions that make pages
unevictable, either by definition or by circumstance, in the future.
@@ -201,6 +203,15 @@ page_evictable() also checks for mlocked
flag, PG_mlocked (as wrapped by PageMlocked()), which is set when a page is
faulted into a VM_LOCKED vma, or found in a vma being VM_LOCKED.
+page_evictable() also uses hpage_nr_pages(), to check for a huge tmpfs team
+tail page which reached the bottom of the inactive list, but could not be
+evicted at that time because its team head had not yet been evicted. We
+must not evict any member of the team while the whole team is mapped; and
+at present we only disband the team for reclaim when its head is evicted.
+When an inactive tail is held back from eviction, putback_inactive_pages()
+shifts its "weight" of 1 page to the head, to increase pressure on the head,
+but leave the tail as unevictable, without adding to the Unevictable count.
+
VMSCAN'S HANDLING OF UNEVICTABLE PAGES
--------------------------------------
@@ -597,7 +608,9 @@ Some examples of these unevictable pages
unevictable list in mlock_vma_page().
shrink_inactive_list() also diverts any unevictable pages that it finds on the
-inactive lists to the appropriate zone's unevictable list.
+inactive lists to the appropriate zone's unevictable list, adding in those
+huge tmpfs team tails which were rejected by pageout() (shmem_writepage())
+because the team has not yet been disbanded by evicting the head.
shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd
after shrink_active_list() had moved them to the inactive list, or pages mapped
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -127,10 +127,24 @@ static inline spinlock_t *pmd_trans_huge
else
return NULL;
}
+
+/* Repeat definition from linux/pageteam.h to force error if different */
+#define TEAM_LRU_WEIGHT_MASK ((1L << (HPAGE_PMD_ORDER + 1)) - 1)
+
+/*
+ * hpage_nr_pages(page) returns the current LRU weight of the page.
+ * Beware of races when it is used: an Anon THPage might get split,
+ * so may need protection by compound lock or lruvec lock; a huge tmpfs
+ * team page might have weight 1 shifted from tail to head, or back to
+ * tail when disbanded, so may need protection by lruvec lock.
+ */
static inline int hpage_nr_pages(struct page *page)
{
if (unlikely(PageTransHuge(page)))
return HPAGE_PMD_NR;
+ if (PageTeam(page))
+ return atomic_long_read(&page->team_usage) &
+ TEAM_LRU_WEIGHT_MASK;
return 1;
}
--- a/include/linux/pageteam.h
+++ b/include/linux/pageteam.h
@@ -30,11 +30,32 @@ static inline struct page *team_head(str
}
/*
+ * Mask for lower bits of team_usage, giving the weight 0..HPAGE_PMD_NR of the
+ * page on its LRU: normal pages have weight 1, tails held unevictable until
+ * head is evicted have weight 0, and the head gathers weight 1..HPAGE_PMD_NR.
+ */
+#define TEAM_LRU_WEIGHT_ONE 1L
+#define TEAM_LRU_WEIGHT_MASK ((1L << (HPAGE_PMD_ORDER + 1)) - 1)
+
+#define TEAM_HIGH_COUNTER (1L << (HPAGE_PMD_ORDER + 1))
+/*
+ * Count how many pages of team are instantiated, as it is built up.
+ */
+#define TEAM_PAGE_COUNTER TEAM_HIGH_COUNTER
+#define TEAM_COMPLETE (TEAM_PAGE_COUNTER << HPAGE_PMD_ORDER)
+/*
+ * And when complete, count how many huge mappings (like page_mapcount): an
+ * incomplete team cannot be hugely mapped (would expose uninitialized holes).
+ */
+#define TEAM_MAPPING_COUNTER TEAM_HIGH_COUNTER
+#define TEAM_PMD_MAPPED (TEAM_COMPLETE + TEAM_MAPPING_COUNTER)
+
+/*
* Returns true if this team is mapped by pmd somewhere.
*/
static inline bool team_pmd_mapped(struct page *head)
{
- return atomic_long_read(&head->team_usage) > HPAGE_PMD_NR;
+ return atomic_long_read(&head->team_usage) >= TEAM_PMD_MAPPED;
}
/*
@@ -43,7 +64,8 @@ static inline bool team_pmd_mapped(struc
*/
static inline bool inc_team_pmd_mapped(struct page *head)
{
- return atomic_long_inc_return(&head->team_usage) == HPAGE_PMD_NR+1;
+ return atomic_long_add_return(TEAM_MAPPING_COUNTER, &head->team_usage)
+ < TEAM_PMD_MAPPED + TEAM_MAPPING_COUNTER;
}
/*
@@ -52,7 +74,27 @@ static inline bool inc_team_pmd_mapped(s
*/
static inline bool dec_team_pmd_mapped(struct page *head)
{
- return atomic_long_dec_return(&head->team_usage) == HPAGE_PMD_NR;
+ return atomic_long_sub_return(TEAM_MAPPING_COUNTER, &head->team_usage)
+ < TEAM_PMD_MAPPED;
+}
+
+static inline void inc_lru_weight(struct page *head)
+{
+ atomic_long_inc(&head->team_usage);
+ VM_BUG_ON_PAGE((atomic_long_read(&head->team_usage) &
+ TEAM_LRU_WEIGHT_MASK) > HPAGE_PMD_NR, head);
+}
+
+static inline void set_lru_weight(struct page *page)
+{
+ VM_BUG_ON_PAGE(atomic_long_read(&page->team_usage) != 0, page);
+ atomic_long_set(&page->team_usage, 1);
+}
+
+static inline void clear_lru_weight(struct page *page)
+{
+ VM_BUG_ON_PAGE(atomic_long_read(&page->team_usage) != 1, page);
+ atomic_long_set(&page->team_usage, 0);
}
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1047,6 +1047,16 @@ void mem_cgroup_update_lru_size(struct l
*lru_size += nr_pages;
size = *lru_size;
+ if (!size && !empty && lru == LRU_UNEVICTABLE) {
+ struct page *page;
+ /*
+ * The unevictable list might be full of team tail pages of 0
+ * weight: check the first, and skip the warning if that fits.
+ */
+ page = list_first_entry(lruvec->lists + lru, struct page, lru);
+ if (hpage_nr_pages(page) == 0)
+ empty = true;
+ }
if (WARN_ONCE(size < 0 || empty != !size,
"%s(%p, %d, %d): lru_size %ld but %sempty\n",
__func__, lruvec, lru, nr_pages, size, empty ? "" : "not ")) {
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -63,6 +63,7 @@ static struct vfsmount *shm_mnt;
#include <linux/swapops.h>
#include <linux/pageteam.h>
#include <linux/mempolicy.h>
+#include <linux/mm_inline.h>
#include <linux/namei.h>
#include <linux/ctype.h>
#include <linux/migrate.h>
@@ -372,7 +373,8 @@ static int shmem_freeholes(struct page *
{
unsigned long nr = atomic_long_read(&head->team_usage);
- return (nr >= HPAGE_PMD_NR) ? 0 : HPAGE_PMD_NR - nr;
+ return (nr >= TEAM_COMPLETE) ? 0 :
+ HPAGE_PMD_NR - (nr / TEAM_PAGE_COUNTER);
}
static void shmem_clear_tag_hugehole(struct address_space *mapping,
@@ -399,18 +401,16 @@ static void shmem_added_to_hugeteam(stru
{
struct address_space *mapping = page->mapping;
struct page *head = team_head(page);
- int nr;
if (hugehint == SHMEM_ALLOC_HUGE_PAGE) {
- atomic_long_set(&head->team_usage, 1);
+ atomic_long_set(&head->team_usage,
+ TEAM_PAGE_COUNTER + TEAM_LRU_WEIGHT_ONE);
radix_tree_tag_set(&mapping->page_tree, page->index,
SHMEM_TAG_HUGEHOLE);
__mod_zone_page_state(zone, NR_SHMEM_FREEHOLES, HPAGE_PMD_NR-1);
} else {
- /* We do not need atomic ops until huge page gets mapped */
- nr = atomic_long_read(&head->team_usage) + 1;
- atomic_long_set(&head->team_usage, nr);
- if (nr == HPAGE_PMD_NR) {
+ if (atomic_long_add_return(TEAM_PAGE_COUNTER,
+ &head->team_usage) >= TEAM_COMPLETE) {
shmem_clear_tag_hugehole(mapping, head->index);
__inc_zone_state(zone, NR_SHMEM_HUGEPAGES);
}
@@ -456,11 +456,14 @@ static int shmem_populate_hugeteam(struc
return 0;
}
-static int shmem_disband_hugehead(struct page *head)
+static int shmem_disband_hugehead(struct page *head, int *head_lru_weight)
{
struct address_space *mapping;
+ bool lru_locked = false;
+ unsigned long flags;
struct zone *zone;
- int nr = -EALREADY; /* A racing task may have disbanded the team */
+ long team_usage;
+ long nr = -EALREADY; /* A racing task may have disbanded the team */
/*
* In most cases the head page is locked, or not yet exposed to others:
@@ -469,27 +472,54 @@ static int shmem_disband_hugehead(struct
* stays safe because shmem_evict_inode must take the shrinklist_lock,
* and our caller shmem_choose_hugehole is already holding that lock.
*/
+ *head_lru_weight = 0;
mapping = READ_ONCE(head->mapping);
if (!mapping)
return nr;
zone = page_zone(head);
- spin_lock_irq(&mapping->tree_lock);
+ team_usage = atomic_long_read(&head->team_usage);
+again1:
+ if ((team_usage & TEAM_LRU_WEIGHT_MASK) != TEAM_LRU_WEIGHT_ONE) {
+ spin_lock_irq(&zone->lru_lock);
+ lru_locked = true;
+ }
+ spin_lock_irqsave(&mapping->tree_lock, flags);
if (PageTeam(head)) {
- nr = atomic_long_read(&head->team_usage);
- atomic_long_set(&head->team_usage, 0);
+again2:
+ nr = atomic_long_cmpxchg(&head->team_usage, team_usage,
+ TEAM_LRU_WEIGHT_ONE);
+ if (unlikely(nr != team_usage)) {
+ team_usage = nr;
+ if (lru_locked ||
+ (team_usage & TEAM_LRU_WEIGHT_MASK) ==
+ TEAM_LRU_WEIGHT_ONE)
+ goto again2;
+ spin_unlock_irqrestore(&mapping->tree_lock, flags);
+ goto again1;
+ }
+ *head_lru_weight = nr & TEAM_LRU_WEIGHT_MASK;
+ nr /= TEAM_PAGE_COUNTER;
+
/*
- * Disable additions to the team.
- * Ensure head->private is written before PageTeam is
- * cleared, so shmem_writepage() cannot write swap into
- * head->private, then have it overwritten by that 0!
+ * Disable additions to the team. The cmpxchg above
+ * ensures head->team_usage is read before PageTeam is cleared,
+ * when shmem_writepage() might write swap into head->private.
*/
- smp_mb__before_atomic();
ClearPageTeam(head);
+
+ /*
+ * If head has not yet been instantiated into the cache,
+ * reset its page->mapping now, while we have all the locks.
+ */
if (!PageSwapBacked(head))
head->mapping = NULL;
+ if (PageLRU(head) && *head_lru_weight > 1)
+ update_lru_size(mem_cgroup_page_lruvec(head, zone),
+ page_lru(head), 1 - *head_lru_weight);
+
if (nr >= HPAGE_PMD_NR) {
ClearPageChecked(head);
__dec_zone_state(zone, NR_SHMEM_HUGEPAGES);
@@ -501,10 +531,88 @@ static int shmem_disband_hugehead(struct
}
}
- spin_unlock_irq(&mapping->tree_lock);
+ spin_unlock_irqrestore(&mapping->tree_lock, flags);
+ if (lru_locked)
+ spin_unlock_irq(&zone->lru_lock);
return nr;
}
+static void shmem_evictify_hugetails(struct page *head, int head_lru_weight)
+{
+ struct page *page;
+ struct lruvec *lruvec = NULL;
+ struct zone *zone = page_zone(head);
+ bool lru_locked = false;
+
+ /*
+ * The head has been sheltering the rest of its team from reclaim:
+ * if any were moved to the unevictable list, now make them evictable.
+ */
+again:
+ for (page = head + HPAGE_PMD_NR - 1; page > head; page--) {
+ if (!PageTeam(page))
+ continue;
+ if (atomic_long_read(&page->team_usage) == TEAM_LRU_WEIGHT_ONE)
+ continue;
+
+ /*
+ * Delay getting lru lock until we reach a page that needs it.
+ */
+ if (!lru_locked) {
+ spin_lock_irq(&zone->lru_lock);
+ lru_locked = true;
+ }
+ lruvec = mem_cgroup_page_lruvec(page, zone);
+
+ if (unlikely(atomic_long_read(&page->team_usage) ==
+ TEAM_LRU_WEIGHT_ONE))
+ continue;
+
+ set_lru_weight(page);
+ head_lru_weight--;
+
+ /*
+ * Usually an Unevictable Team page just stays on its LRU;
+ * but isolation for migration might take it off briefly.
+ */
+ if (unlikely(!PageLRU(page)))
+ continue;
+
+ VM_BUG_ON_PAGE(!PageUnevictable(page), page);
+ VM_BUG_ON_PAGE(PageActive(page), page);
+
+ if (!page_evictable(page)) {
+ /*
+ * This is tiresome, but page_evictable() needs weight 1
+ * to make the right decision, whereas lru size update
+ * needs weight 0 to avoid a bogus "not empty" warning.
+ */
+ clear_lru_weight(page);
+ update_lru_size(lruvec, LRU_UNEVICTABLE, 1);
+ set_lru_weight(page);
+ continue;
+ }
+
+ ClearPageUnevictable(page);
+ update_lru_size(lruvec, LRU_INACTIVE_ANON, 1);
+
+ list_del(&page->lru);
+ list_add_tail(&page->lru, lruvec->lists + LRU_INACTIVE_ANON);
+ }
+
+ if (lru_locked) {
+ spin_unlock_irq(&zone->lru_lock);
+ lru_locked = false;
+ }
+
+ /*
+ * But how can we be sure that a racing putback_inactive_pages()
+ * did its clear_lru_weight() before we checked team_usage above?
+ */
+ if (unlikely(head_lru_weight != TEAM_LRU_WEIGHT_ONE))
+ goto again;
+}
+
static void shmem_disband_hugetails(struct page *head,
struct list_head *list, int nr)
{
@@ -578,6 +686,7 @@ static void shmem_disband_hugetails(stru
static void shmem_disband_hugeteam(struct page *page)
{
struct page *head = team_head(page);
+ int head_lru_weight;
int nr_used;
/*
@@ -623,9 +732,11 @@ static void shmem_disband_hugeteam(struc
* can (splitting disband in two stages), but better not be preempted.
*/
preempt_disable();
- nr_used = shmem_disband_hugehead(head);
+ nr_used = shmem_disband_hugehead(head, &head_lru_weight);
if (head != page)
unlock_page(head);
+ if (head_lru_weight > TEAM_LRU_WEIGHT_ONE)
+ shmem_evictify_hugetails(head, head_lru_weight);
if (nr_used >= 0)
shmem_disband_hugetails(head, NULL, 0);
if (head != page)
@@ -681,6 +792,7 @@ static unsigned long shmem_choose_hugeho
struct page *topage = NULL;
struct page *page;
pgoff_t index;
+ int head_lru_weight;
int fromused;
int toused;
int nid;
@@ -722,8 +834,10 @@ static unsigned long shmem_choose_hugeho
if (!frompage)
goto unlock;
preempt_disable();
- fromused = shmem_disband_hugehead(frompage);
+ fromused = shmem_disband_hugehead(frompage, &head_lru_weight);
spin_unlock(&shmem_shrinklist_lock);
+ if (head_lru_weight > TEAM_LRU_WEIGHT_ONE)
+ shmem_evictify_hugetails(frompage, head_lru_weight);
if (fromused > 0)
shmem_disband_hugetails(frompage, fromlist, -fromused);
preempt_enable();
@@ -777,8 +891,10 @@ static unsigned long shmem_choose_hugeho
if (!topage)
goto unlock;
preempt_disable();
- toused = shmem_disband_hugehead(topage);
+ toused = shmem_disband_hugehead(topage, &head_lru_weight);
spin_unlock(&shmem_shrinklist_lock);
+ if (head_lru_weight > TEAM_LRU_WEIGHT_ONE)
+ shmem_evictify_hugetails(topage, head_lru_weight);
if (toused > 0) {
if (HPAGE_PMD_NR - toused >= fromused)
shmem_disband_hugetails(topage, tolist, fromused);
@@ -930,7 +1046,11 @@ shmem_add_to_page_cache(struct page *pag
}
if (!PageSwapBacked(page)) { /* huge needs special care */
SetPageSwapBacked(page);
- SetPageTeam(page);
+ if (!PageTeam(page)) {
+ atomic_long_set(&page->team_usage,
+ TEAM_LRU_WEIGHT_ONE);
+ SetPageTeam(page);
+ }
}
}
@@ -1612,9 +1732,13 @@ static int shmem_writepage(struct page *
struct page *head = team_head(page);
/*
* Only proceed if this is head, or if head is unpopulated.
+ * Redirty any others, without setting PageActive, and then
+ * putback_inactive_pages() will shift them to unevictable.
*/
- if (page != head && PageSwapBacked(head))
+ if (page != head && PageSwapBacked(head)) {
+ wbc->for_reclaim = 0;
goto redirty;
+ }
}
swap = get_swap_page();
@@ -1762,7 +1886,8 @@ static struct page *shmem_alloc_page(gfp
split_page(head, HPAGE_PMD_ORDER);
/* Prepare head page for add_to_page_cache */
- atomic_long_set(&head->team_usage, 0);
+ atomic_long_set(&head->team_usage,
+ TEAM_LRU_WEIGHT_ONE);
__SetPageTeam(head);
head->mapping = mapping;
head->index = round_down(index, HPAGE_PMD_NR);
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -469,6 +469,11 @@ void lru_cache_add_active_or_unevictable
struct vm_area_struct *vma)
{
VM_BUG_ON_PAGE(PageLRU(page), page);
+ /*
+ * Using hpage_nr_pages() on a huge tmpfs team page might not give the
+ * 1 NR_MLOCK needs below; but this seems to be for anon pages only.
+ */
+ VM_BUG_ON_PAGE(!PageAnon(page), page);
if (likely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) != VM_LOCKED)) {
SetPageActive(page);
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -19,6 +19,7 @@
#include <linux/kernel_stat.h>
#include <linux/swap.h>
#include <linux/pagemap.h>
+#include <linux/pageteam.h>
#include <linux/init.h>
#include <linux/highmem.h>
#include <linux/vmpressure.h>
@@ -1514,6 +1515,39 @@ putback_inactive_pages(struct lruvec *lr
continue;
}
+ if (PageTeam(page) && !PageActive(page)) {
+ struct page *head = team_head(page);
+ struct address_space *mapping;
+ bool transferring_weight = false;
+ /*
+ * Team tail page was ready for eviction, but has
+ * been sent back from shmem_writepage(): transfer
+ * its weight to head, and move tail to unevictable.
+ */
+ mapping = READ_ONCE(page->mapping);
+ if (page != head && mapping) {
+ lruvec = mem_cgroup_page_lruvec(head, zone);
+ spin_lock(&mapping->tree_lock);
+ if (PageTeam(head)) {
+ VM_BUG_ON(head->mapping != mapping);
+ inc_lru_weight(head);
+ transferring_weight = true;
+ }
+ spin_unlock(&mapping->tree_lock);
+ }
+ if (transferring_weight) {
+ if (PageLRU(head))
+ update_lru_size(lruvec,
+ page_lru(head), 1);
+ /* Get this tail page out of the way for now */
+ SetPageUnevictable(page);
+ clear_lru_weight(page);
+ } else {
+ /* Traditional case of unswapped & redirtied */
+ SetPageActive(page);
+ }
+ }
+
lruvec = mem_cgroup_page_lruvec(page, zone);
SetPageLRU(page);
@@ -3791,11 +3825,12 @@ int zone_reclaim(struct zone *zone, gfp_
* Reasons page might not be evictable:
* (1) page's mapping marked unevictable
* (2) page is part of an mlocked VMA
- *
+ * (3) page is held in memory as part of a team
*/
int page_evictable(struct page *page)
{
- return !mapping_unevictable(page_mapping(page)) && !PageMlocked(page);
+ return !mapping_unevictable(page_mapping(page)) &&
+ !PageMlocked(page) && hpage_nr_pages(page);
}
#ifdef CONFIG_SHMEM
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:40 +0200 |
| Subject | [PATCH 12/31] huge tmpfs: extend get_user_pages_fast to shmem pmd |
| Message-ID | <rkGRA-1Og-19@gated-at.bofh.it> |
| In reply to | #1371950 |
The arch-specific get_user_pages_fast() has a gup_huge_pmd() designed to
optimize the refcounting on anonymous THP and hugetlbfs pages, with one
atomic addition to compound head's common refcount. That optimization
must be avoided on huge tmpfs team pages, which use normal separate page
refcounting. We could combine the PageTeam and PageCompound cases into
a single simple loop, but would lose the compound optimization that way.
One cannot go through these functions without wondering why some arches
(x86, mips) like to SetPageReferenced, while the rest do not: an x86
optimization that missed being propagated to the other architectures?
No, see commit 8ee53820edfd ("thp: mmu_notifier_test_young"): it's a
KVM GRU EPT thing, maybe not useful beyond x86. I've just followed
the established practice in each architecture.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
Cc'ed to arch maintainers as an FYI: this patch is not expected to
go into the tree in the next few weeks, and depends upon a PageTeam
definition not yet available outside this huge tmpfs patchset.
Please refer to linux-mm or linux-kernel for more context.
arch/mips/mm/gup.c | 15 ++++++++++++++-
arch/s390/mm/gup.c | 19 ++++++++++++++++++-
arch/sparc/mm/gup.c | 19 ++++++++++++++++++-
arch/x86/mm/gup.c | 15 ++++++++++++++-
mm/gup.c | 19 ++++++++++++++++++-
5 files changed, 82 insertions(+), 5 deletions(-)
--- a/arch/mips/mm/gup.c
+++ b/arch/mips/mm/gup.c
@@ -81,9 +81,22 @@ static int gup_huge_pmd(pmd_t pmd, unsig
VM_BUG_ON(pte_special(pte));
VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
- refs = 0;
head = pte_page(pte);
page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
+
+ if (PageTeam(head)) {
+ /* Handle a huge tmpfs team with normal refcounting. */
+ do {
+ get_page(page);
+ SetPageReferenced(page);
+ pages[*nr] = page;
+ (*nr)++;
+ page++;
+ } while (addr += PAGE_SIZE, addr != end);
+ return 1;
+ }
+
+ refs = 0;
do {
VM_BUG_ON(compound_head(page) != head);
pages[*nr] = page;
--- a/arch/s390/mm/gup.c
+++ b/arch/s390/mm/gup.c
@@ -66,9 +66,26 @@ static inline int gup_huge_pmd(pmd_t *pm
return 0;
VM_BUG_ON(!pfn_valid(pmd_val(pmd) >> PAGE_SHIFT));
- refs = 0;
head = pmd_page(pmd);
page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
+
+ if (PageTeam(head)) {
+ /* Handle a huge tmpfs team with normal refcounting. */
+ do {
+ if (!page_cache_get_speculative(page))
+ return 0;
+ if (unlikely(pmd_val(pmd) != pmd_val(*pmdp))) {
+ put_page(page);
+ return 0;
+ }
+ pages[*nr] = page;
+ (*nr)++;
+ page++;
+ } while (addr += PAGE_SIZE, addr != end);
+ return 1;
+ }
+
+ refs = 0;
do {
VM_BUG_ON(compound_head(page) != head);
pages[*nr] = page;
--- a/arch/sparc/mm/gup.c
+++ b/arch/sparc/mm/gup.c
@@ -77,9 +77,26 @@ static int gup_huge_pmd(pmd_t *pmdp, pmd
if (write && !pmd_write(pmd))
return 0;
- refs = 0;
head = pmd_page(pmd);
page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
+
+ if (PageTeam(head)) {
+ /* Handle a huge tmpfs team with normal refcounting. */
+ do {
+ if (!page_cache_get_speculative(page))
+ return 0;
+ if (unlikely(pmd_val(pmd) != pmd_val(*pmdp))) {
+ put_page(page);
+ return 0;
+ }
+ pages[*nr] = page;
+ (*nr)++;
+ page++;
+ } while (addr += PAGE_SIZE, addr != end);
+ return 1;
+ }
+
+ refs = 0;
do {
VM_BUG_ON(compound_head(page) != head);
pages[*nr] = page;
--- a/arch/x86/mm/gup.c
+++ b/arch/x86/mm/gup.c
@@ -196,9 +196,22 @@ static noinline int gup_huge_pmd(pmd_t p
/* hugepages are never "special" */
VM_BUG_ON(pmd_flags(pmd) & _PAGE_SPECIAL);
- refs = 0;
head = pmd_page(pmd);
page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
+
+ if (PageTeam(head)) {
+ /* Handle a huge tmpfs team with normal refcounting. */
+ do {
+ get_page(page);
+ SetPageReferenced(page);
+ pages[*nr] = page;
+ (*nr)++;
+ page++;
+ } while (addr += PAGE_SIZE, addr != end);
+ return 1;
+ }
+
+ refs = 0;
do {
VM_BUG_ON_PAGE(compound_head(page) != head, page);
pages[*nr] = page;
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1247,9 +1247,26 @@ static int gup_huge_pmd(pmd_t orig, pmd_
if (write && !pmd_write(orig))
return 0;
- refs = 0;
head = pmd_page(orig);
page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
+
+ if (PageTeam(head)) {
+ /* Handle a huge tmpfs team with normal refcounting. */
+ do {
+ if (!page_cache_get_speculative(page))
+ return 0;
+ if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {
+ put_page(page);
+ return 0;
+ }
+ pages[*nr] = page;
+ (*nr)++;
+ page++;
+ } while (addr += PAGE_SIZE, addr != end);
+ return 1;
+ }
+
+ refs = 0;
do {
VM_BUG_ON_PAGE(compound_head(page) != head, page);
pages[*nr] = page;
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-04-06 09:10 +0200 |
| Subject | Re: [PATCH 12/31] huge tmpfs: extend get_user_pages_fast to shmem pmd |
| Message-ID | <rkPLc-gu-9@gated-at.bofh.it> |
| In reply to | #1371971 |
* Hugh Dickins <hughd@google.com> wrote:
> The arch-specific get_user_pages_fast() has a gup_huge_pmd() designed to
> optimize the refcounting on anonymous THP and hugetlbfs pages, with one
> atomic addition to compound head's common refcount. That optimization
> must be avoided on huge tmpfs team pages, which use normal separate page
> refcounting. We could combine the PageTeam and PageCompound cases into
> a single simple loop, but would lose the compound optimization that way.
>
> One cannot go through these functions without wondering why some arches
> (x86, mips) like to SetPageReferenced, while the rest do not: an x86
> optimization that missed being propagated to the other architectures?
> No, see commit 8ee53820edfd ("thp: mmu_notifier_test_young"): it's a
> KVM GRU EPT thing, maybe not useful beyond x86. I've just followed
> the established practice in each architecture.
>
> Signed-off-by: Hugh Dickins <hughd@google.com>
> ---
> Cc'ed to arch maintainers as an FYI: this patch is not expected to
> go into the tree in the next few weeks, and depends upon a PageTeam
> definition not yet available outside this huge tmpfs patchset.
> Please refer to linux-mm or linux-kernel for more context.
>
> arch/mips/mm/gup.c | 15 ++++++++++++++-
> arch/s390/mm/gup.c | 19 ++++++++++++++++++-
> arch/sparc/mm/gup.c | 19 ++++++++++++++++++-
> arch/x86/mm/gup.c | 15 ++++++++++++++-
> mm/gup.c | 19 ++++++++++++++++++-
> 5 files changed, 82 insertions(+), 5 deletions(-)
>
> --- a/arch/mips/mm/gup.c
> +++ b/arch/mips/mm/gup.c
> @@ -81,9 +81,22 @@ static int gup_huge_pmd(pmd_t pmd, unsig
> VM_BUG_ON(pte_special(pte));
> VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
>
> - refs = 0;
> head = pte_page(pte);
> page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
> +
> + if (PageTeam(head)) {
> + /* Handle a huge tmpfs team with normal refcounting. */
> + do {
> + get_page(page);
> + SetPageReferenced(page);
> + pages[*nr] = page;
> + (*nr)++;
> + page++;
> + } while (addr += PAGE_SIZE, addr != end);
> + return 1;
> + }
> +
> + refs = 0;
> do {
> VM_BUG_ON(compound_head(page) != head);
> pages[*nr] = page;
> --- a/arch/s390/mm/gup.c
> +++ b/arch/s390/mm/gup.c
> @@ -66,9 +66,26 @@ static inline int gup_huge_pmd(pmd_t *pm
> return 0;
> VM_BUG_ON(!pfn_valid(pmd_val(pmd) >> PAGE_SHIFT));
>
> - refs = 0;
> head = pmd_page(pmd);
> page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
> +
> + if (PageTeam(head)) {
> + /* Handle a huge tmpfs team with normal refcounting. */
> + do {
> + if (!page_cache_get_speculative(page))
> + return 0;
> + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp))) {
> + put_page(page);
> + return 0;
> + }
> + pages[*nr] = page;
> + (*nr)++;
> + page++;
> + } while (addr += PAGE_SIZE, addr != end);
> + return 1;
> + }
> +
> + refs = 0;
> do {
> VM_BUG_ON(compound_head(page) != head);
> pages[*nr] = page;
> --- a/arch/sparc/mm/gup.c
> +++ b/arch/sparc/mm/gup.c
> @@ -77,9 +77,26 @@ static int gup_huge_pmd(pmd_t *pmdp, pmd
> if (write && !pmd_write(pmd))
> return 0;
>
> - refs = 0;
> head = pmd_page(pmd);
> page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
> +
> + if (PageTeam(head)) {
> + /* Handle a huge tmpfs team with normal refcounting. */
> + do {
> + if (!page_cache_get_speculative(page))
> + return 0;
> + if (unlikely(pmd_val(pmd) != pmd_val(*pmdp))) {
> + put_page(page);
> + return 0;
> + }
> + pages[*nr] = page;
> + (*nr)++;
> + page++;
> + } while (addr += PAGE_SIZE, addr != end);
> + return 1;
> + }
> +
> + refs = 0;
> do {
> VM_BUG_ON(compound_head(page) != head);
> pages[*nr] = page;
> --- a/arch/x86/mm/gup.c
> +++ b/arch/x86/mm/gup.c
> @@ -196,9 +196,22 @@ static noinline int gup_huge_pmd(pmd_t p
> /* hugepages are never "special" */
> VM_BUG_ON(pmd_flags(pmd) & _PAGE_SPECIAL);
>
> - refs = 0;
> head = pmd_page(pmd);
> page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
> +
> + if (PageTeam(head)) {
> + /* Handle a huge tmpfs team with normal refcounting. */
> + do {
> + get_page(page);
> + SetPageReferenced(page);
> + pages[*nr] = page;
> + (*nr)++;
> + page++;
> + } while (addr += PAGE_SIZE, addr != end);
> + return 1;
> + }
> +
> + refs = 0;
> do {
> VM_BUG_ON_PAGE(compound_head(page) != head, page);
> pages[*nr] = page;
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -1247,9 +1247,26 @@ static int gup_huge_pmd(pmd_t orig, pmd_
> if (write && !pmd_write(orig))
> return 0;
>
> - refs = 0;
> head = pmd_page(orig);
> page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
> +
> + if (PageTeam(head)) {
> + /* Handle a huge tmpfs team with normal refcounting. */
> + do {
> + if (!page_cache_get_speculative(page))
> + return 0;
> + if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {
> + put_page(page);
> + return 0;
> + }
> + pages[*nr] = page;
> + (*nr)++;
> + page++;
> + } while (addr += PAGE_SIZE, addr != end);
> + return 1;
> + }
> +
> + refs = 0;
> do {
> VM_BUG_ON_PAGE(compound_head(page) != head, page);
> pages[*nr] = page;
Ouch!
Looks like there are two main variants - so these kinds of repetitive patterns
very much call for some sort of factoring out of common code, right?
Then the fix could be applied to the common portion(s) only, which will cut down
this gigantic diffstat:
> 5 files changed, 82 insertions(+), 5 deletions(-)
Thanks,
Ingo
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-07 05:00 +0200 |
| Subject | Re: [PATCH 12/31] huge tmpfs: extend get_user_pages_fast to shmem pmd |
| Message-ID | <rl8kO-5xu-11@gated-at.bofh.it> |
| In reply to | #1372241 |
On Wed, 6 Apr 2016, Ingo Molnar wrote:
> * Hugh Dickins <hughd@google.com> wrote:
>
> > ---
> > Cc'ed to arch maintainers as an FYI: this patch is not expected to
> > go into the tree in the next few weeks, and depends upon a PageTeam
> > definition not yet available outside this huge tmpfs patchset.
> > Please refer to linux-mm or linux-kernel for more context.
Actually, Andrew took it and the rest into mmotm yesterday, to give them
better exposure through linux-next, so they should appear there soon.
> >
> > arch/mips/mm/gup.c | 15 ++++++++++++++-
> > arch/s390/mm/gup.c | 19 ++++++++++++++++++-
> > arch/sparc/mm/gup.c | 19 ++++++++++++++++++-
> > arch/x86/mm/gup.c | 15 ++++++++++++++-
> > mm/gup.c | 19 ++++++++++++++++++-
> > 5 files changed, 82 insertions(+), 5 deletions(-)
...
>
> Ouch!
Oh sorry, I didn't mean to hurt you ;)
>
> Looks like there are two main variants - so these kinds of repetitive patterns
> very much call for some sort of factoring out of common code, right?
Hmm. I'm still struggling between the two extremes, of
(a) agreeing completely with you, and saying, yeah, I'll take on the job
of refactoring every architecture's get_user_pages_as_fast_as_you_can(),
without much likelihood of testing more than one,
and
(b) running a mile, and pointing out that we have a tradition of using
arch/x86/mm/gup.c as a template for the others, and here I've just
added a few more lines to that template (which never gets built more
than once into any kernel).
Both are appealing in their different ways, but I think you can tell
which I'm leaning towards...
Honestly, I am still struggling between those two; but I think the patch
as it stands is one thing, and cleanup for commonality should be another
however weaselly that sounds ("I'll come back to it" - yeah, right).
Hugh
>
> Then the fix could be applied to the common portion(s) only, which will cut down
> this gigantic diffstat:
>
> > 5 files changed, 82 insertions(+), 5 deletions(-)
>
> Thanks,
>
> Ingo
[toc] | [prev] | [next] | [standalone]
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2016-04-13 11:00 +0200 |
| Subject | Re: [PATCH 12/31] huge tmpfs: extend get_user_pages_fast to shmem pmd |
| Message-ID | <rnoOu-714-17@gated-at.bofh.it> |
| In reply to | #1373042 |
* Hugh Dickins <hughd@google.com> wrote:
> > > arch/mips/mm/gup.c | 15 ++++++++++++++-
> > > arch/s390/mm/gup.c | 19 ++++++++++++++++++-
> > > arch/sparc/mm/gup.c | 19 ++++++++++++++++++-
> > > arch/x86/mm/gup.c | 15 ++++++++++++++-
> > > mm/gup.c | 19 ++++++++++++++++++-
> > > 5 files changed, 82 insertions(+), 5 deletions(-)
> ...
> > Looks like there are two main variants - so these kinds of repetitive patterns
> > very much call for some sort of factoring out of common code, right?
>
> Hmm. I'm still struggling between the two extremes, of
>
> (a) agreeing completely with you, and saying, yeah, I'll take on the job
> of refactoring every architecture's get_user_pages_as_fast_as_you_can(),
> without much likelihood of testing more than one,
>
> and
>
> (b) running a mile, and pointing out that we have a tradition of using
> arch/x86/mm/gup.c as a template for the others, and here I've just
> added a few more lines to that template (which never gets built more
> than once into any kernel).
>
> Both are appealing in their different ways, but I think you can tell
> which I'm leaning towards...
>
> Honestly, I am still struggling between those two; but I think the patch
> as it stands is one thing, and cleanup for commonality should be another
> however weaselly that sounds ("I'll come back to it" - yeah, right).
Yeah, so my worry is this: your patch for example roughly doubles the algorithmic
complexity of mm/gup.c and arch/*/mm/gup.c's ::gup_huge_pmd().
And you want this to add a new feature!
So it really looks like to me this is the last sane chance to unify cheaply, then
add the feature you want. Everyone else in the future will be able to refer to
your example to chicken out! ;-)
Thanks,
Ingo
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:40 +0200 |
| Subject | [PATCH 16/31] kvm: plumb return of hva when resolving page fault. |
| Message-ID | <rkGRA-1Og-21@gated-at.bofh.it> |
| In reply to | #1371950 |
From: Andres Lagar-Cavilla <andreslc@google.com>
So we don't have to redo this work later. Note the hva is not racy,
it is simple arithmetic based on the memslot.
This will be used in the huge tmpfs commits.
Signed-off-by: Andres Lagar-Cavilla <andreslc@google.com>
Signed-off-by: Hugh Dickins <hughd@google.com>
---
Cc'ed to kvm@vger.kernel.org as an FYI: this patch is not expected to
go into the tree in the next few weeks. The context is a huge tmpfs
patchset which implements huge pagecache transparently on tmpfs,
using a team of small pages rather than one compound page:
please refer to linux-mm or linux-kernel for more context.
arch/x86/kvm/mmu.c | 20 ++++++++++++++------
arch/x86/kvm/paging_tmpl.h | 3 ++-
include/linux/kvm_host.h | 2 +-
virt/kvm/kvm_main.c | 14 ++++++++------
4 files changed, 25 insertions(+), 14 deletions(-)
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -2992,7 +2992,8 @@ exit:
}
static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,
- gva_t gva, kvm_pfn_t *pfn, bool write, bool *writable);
+ gva_t gva, kvm_pfn_t *pfn, bool write, bool *writable,
+ unsigned long *hva);
static void make_mmu_pages_available(struct kvm_vcpu *vcpu);
static int nonpaging_map(struct kvm_vcpu *vcpu, gva_t v, u32 error_code,
@@ -3003,6 +3004,7 @@ static int nonpaging_map(struct kvm_vcpu
bool force_pt_level = false;
kvm_pfn_t pfn;
unsigned long mmu_seq;
+ unsigned long hva;
bool map_writable, write = error_code & PFERR_WRITE_MASK;
level = mapping_level(vcpu, gfn, &force_pt_level);
@@ -3024,7 +3026,8 @@ static int nonpaging_map(struct kvm_vcpu
mmu_seq = vcpu->kvm->mmu_notifier_seq;
smp_rmb();
- if (try_async_pf(vcpu, prefault, gfn, v, &pfn, write, &map_writable))
+ if (try_async_pf(vcpu, prefault, gfn, v, &pfn, write,
+ &map_writable, &hva))
return 0;
if (handle_abnormal_pfn(vcpu, v, gfn, pfn, ACC_ALL, &r))
@@ -3487,14 +3490,16 @@ static bool can_do_async_pf(struct kvm_v
}
static bool try_async_pf(struct kvm_vcpu *vcpu, bool prefault, gfn_t gfn,
- gva_t gva, kvm_pfn_t *pfn, bool write, bool *writable)
+ gva_t gva, kvm_pfn_t *pfn, bool write, bool *writable,
+ unsigned long *hva)
{
struct kvm_memory_slot *slot;
bool async;
slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
async = false;
- *pfn = __gfn_to_pfn_memslot(slot, gfn, false, &async, write, writable);
+ *pfn = __gfn_to_pfn_memslot(slot, gfn,
+ false, &async, write, writable, hva);
if (!async)
return false; /* *pfn has correct page already */
@@ -3508,7 +3513,8 @@ static bool try_async_pf(struct kvm_vcpu
return true;
}
- *pfn = __gfn_to_pfn_memslot(slot, gfn, false, NULL, write, writable);
+ *pfn = __gfn_to_pfn_memslot(slot, gfn,
+ false, NULL, write, writable, hva);
return false;
}
@@ -3531,6 +3537,7 @@ static int tdp_page_fault(struct kvm_vcp
bool force_pt_level;
gfn_t gfn = gpa >> PAGE_SHIFT;
unsigned long mmu_seq;
+ unsigned long hva;
int write = error_code & PFERR_WRITE_MASK;
bool map_writable;
@@ -3559,7 +3566,8 @@ static int tdp_page_fault(struct kvm_vcp
mmu_seq = vcpu->kvm->mmu_notifier_seq;
smp_rmb();
- if (try_async_pf(vcpu, prefault, gfn, gpa, &pfn, write, &map_writable))
+ if (try_async_pf(vcpu, prefault, gfn, gpa, &pfn, write,
+ &map_writable, &hva))
return 0;
if (handle_abnormal_pfn(vcpu, 0, gfn, pfn, ACC_ALL, &r))
--- a/arch/x86/kvm/paging_tmpl.h
+++ b/arch/x86/kvm/paging_tmpl.h
@@ -712,6 +712,7 @@ static int FNAME(page_fault)(struct kvm_
int level = PT_PAGE_TABLE_LEVEL;
bool force_pt_level = false;
unsigned long mmu_seq;
+ unsigned long hva;
bool map_writable, is_self_change_mapping;
pgprintk("%s: addr %lx err %x\n", __func__, addr, error_code);
@@ -765,7 +766,7 @@ static int FNAME(page_fault)(struct kvm_
smp_rmb();
if (try_async_pf(vcpu, prefault, walker.gfn, addr, &pfn, write_fault,
- &map_writable))
+ &map_writable, &hva))
return 0;
if (handle_abnormal_pfn(vcpu, mmu_is_nested(vcpu) ? 0 : addr,
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -600,7 +600,7 @@ kvm_pfn_t gfn_to_pfn_memslot(struct kvm_
kvm_pfn_t gfn_to_pfn_memslot_atomic(struct kvm_memory_slot *slot, gfn_t gfn);
kvm_pfn_t __gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn,
bool atomic, bool *async, bool write_fault,
- bool *writable);
+ bool *writable, unsigned long *hva);
void kvm_release_pfn_clean(kvm_pfn_t pfn);
void kvm_set_pfn_dirty(kvm_pfn_t pfn);
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1444,7 +1444,7 @@ exit:
kvm_pfn_t __gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn,
bool atomic, bool *async, bool write_fault,
- bool *writable)
+ bool *writable, unsigned long *hva)
{
unsigned long addr = __gfn_to_hva_many(slot, gfn, NULL, write_fault);
@@ -1466,8 +1466,10 @@ kvm_pfn_t __gfn_to_pfn_memslot(struct kv
writable = NULL;
}
- return hva_to_pfn(addr, atomic, async, write_fault,
- writable);
+ if (hva)
+ *hva = addr;
+
+ return hva_to_pfn(addr, atomic, async, write_fault, writable);
}
EXPORT_SYMBOL_GPL(__gfn_to_pfn_memslot);
@@ -1475,19 +1477,19 @@ kvm_pfn_t gfn_to_pfn_prot(struct kvm *kv
bool *writable)
{
return __gfn_to_pfn_memslot(gfn_to_memslot(kvm, gfn), gfn, false, NULL,
- write_fault, writable);
+ write_fault, writable, NULL);
}
EXPORT_SYMBOL_GPL(gfn_to_pfn_prot);
kvm_pfn_t gfn_to_pfn_memslot(struct kvm_memory_slot *slot, gfn_t gfn)
{
- return __gfn_to_pfn_memslot(slot, gfn, false, NULL, true, NULL);
+ return __gfn_to_pfn_memslot(slot, gfn, false, NULL, true, NULL, NULL);
}
EXPORT_SYMBOL_GPL(gfn_to_pfn_memslot);
kvm_pfn_t gfn_to_pfn_memslot_atomic(struct kvm_memory_slot *slot, gfn_t gfn)
{
- return __gfn_to_pfn_memslot(slot, gfn, true, NULL, true, NULL);
+ return __gfn_to_pfn_memslot(slot, gfn, true, NULL, true, NULL, NULL);
}
EXPORT_SYMBOL_GPL(gfn_to_pfn_memslot_atomic);
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:40 +0200 |
| Subject | [PATCH 14/31] huge tmpfs: fix Mlocked meminfo, track huge & unhuge mlocks |
| Message-ID | <rkGRB-1Og-29@gated-at.bofh.it> |
| In reply to | #1371950 |
Up to this point, the huge tmpfs effort has barely looked at or touched
mm/mlock.c at all: just a PageTeam test to stop __munlock_pagevec_fill()
crashing (or hanging on a non-existent spinlock) on hugepage pmds.
/proc/meminfo's Mlocked count has been whatever happens to be shown
if we do nothing extra: a hugely mapped and mlocked team page would
count as 4kB instead of the 2MB you'd expect; or at least until the
previous (Unevictable) patch, which now requires lruvec locking for
hpage_nr_pages() on a team page (locking not given it in mlock.c),
and varies the amount returned by hpage_nr_pages().
It would be easy to correct the 4kB or variable amount to 2MB
by using an alternative to hpage_nr_pages() here. And it would be
fairly easy to maintain an entirely independent PmdMlocked count,
such that Mlocked+PmdMlocked might amount to (almost) twice RAM
size. But is that what observers of Mlocked want? Probably not.
So we need a huge pmd mlock to count as 2MB, but discount 4kB for
each page within it that is already mlocked by pte somewhere, in
this or another process; and a small pte mlock to count usually as
4kB, but 0 if the team head is already mlocked by pmd somewhere.
Can this be done by maintaining extra counts per team? I did
intend so, but (a) space in team_usage is limited, and (b) mlock
and munlock already involve slow LRU switching, so might as well
keep 4kB and 2MB in synch manually; but most significantly (c):
the trylocking around which mlock was designed, makes it hard
to work out just when a count does need to be incremented.
The hard-won solution looks much simpler than I thought possible,
but an odd interface in its current implementation. Not so much
needed changing, mainly just clear_page_mlock(), mlock_vma_page()
munlock_vma_page() and try_to_"unmap"_one(). The big difference
from before, is that a team head page might be being mlocked as a
4kB page or as a 2MB page, and the called functions cannot tell:
so now need an nr_pages argument. But odd because the PageTeam
case immediately converts that to an iteration count, whereas
the anon THP case keeps it as the weight for a single iteration
(and in the munlock case has to reconfirm it under lruvec lock).
Not very nice, but will do for now: it was so hard to get here,
I'm very reluctant to pull it apart in a hurry.
The TEAM_PMD_MLOCKED flag in team_usage does not play a large part,
just optimizes out the overhead in a couple of cases: we don't want to
make yet another pass down the team, whenever a team is last unmapped,
just to handle the unlikely mlocked-then-truncated case; and we don't
want munlocking one of many parallel huge mlocks to check every page.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
include/linux/pageteam.h | 38 +++++++
mm/huge_memory.c | 15 ++-
mm/internal.h | 26 +++--
mm/mlock.c | 181 +++++++++++++++++++++----------------
mm/rmap.c | 44 +++++---
5 files changed, 196 insertions(+), 108 deletions(-)
--- a/include/linux/pageteam.h
+++ b/include/linux/pageteam.h
@@ -36,8 +36,14 @@ static inline struct page *team_head(str
*/
#define TEAM_LRU_WEIGHT_ONE 1L
#define TEAM_LRU_WEIGHT_MASK ((1L << (HPAGE_PMD_ORDER + 1)) - 1)
+/*
+ * Single bit to indicate whether team is hugely mlocked (like PageMlocked).
+ * Then another bit reserved for experiments with other team flags.
+ */
+#define TEAM_PMD_MLOCKED (1L << (HPAGE_PMD_ORDER + 1))
+#define TEAM_RESERVED_FLAG (1L << (HPAGE_PMD_ORDER + 2))
-#define TEAM_HIGH_COUNTER (1L << (HPAGE_PMD_ORDER + 1))
+#define TEAM_HIGH_COUNTER (1L << (HPAGE_PMD_ORDER + 3))
/*
* Count how many pages of team are instantiated, as it is built up.
*/
@@ -97,6 +103,36 @@ static inline void clear_lru_weight(stru
atomic_long_set(&page->team_usage, 0);
}
+static inline bool team_pmd_mlocked(struct page *head)
+{
+ VM_BUG_ON_PAGE(head != team_head(head), head);
+ return atomic_long_read(&head->team_usage) & TEAM_PMD_MLOCKED;
+}
+
+static inline void set_team_pmd_mlocked(struct page *head)
+{
+ long team_usage;
+
+ VM_BUG_ON_PAGE(head != team_head(head), head);
+ team_usage = atomic_long_read(&head->team_usage);
+ while (!(team_usage & TEAM_PMD_MLOCKED)) {
+ team_usage = atomic_long_cmpxchg(&head->team_usage,
+ team_usage, team_usage | TEAM_PMD_MLOCKED);
+ }
+}
+
+static inline void clear_team_pmd_mlocked(struct page *head)
+{
+ long team_usage;
+
+ VM_BUG_ON_PAGE(head != team_head(head), head);
+ team_usage = atomic_long_read(&head->team_usage);
+ while (team_usage & TEAM_PMD_MLOCKED) {
+ team_usage = atomic_long_cmpxchg(&head->team_usage,
+ team_usage, team_usage & ~TEAM_PMD_MLOCKED);
+ }
+}
+
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
int map_team_by_pmd(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmd, struct page *page);
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1443,8 +1443,8 @@ struct page *follow_trans_huge_pmd(struc
touch_pmd(vma, addr, pmd);
if ((flags & FOLL_MLOCK) && (vma->vm_flags & VM_LOCKED)) {
/*
- * We don't mlock() pte-mapped THPs. This way we can avoid
- * leaking mlocked pages into non-VM_LOCKED VMAs.
+ * We don't mlock() pte-mapped compound THPs. This way we
+ * can avoid leaking mlocked pages into non-VM_LOCKED VMAs.
*
* In most cases the pmd is the only mapping of the page as we
* break COW for the mlock() -- see gup_flags |= FOLL_WRITE for
@@ -1453,12 +1453,16 @@ struct page *follow_trans_huge_pmd(struc
* The only scenario when we have the page shared here is if we
* mlocking read-only mapping shared over fork(). We skip
* mlocking such pages.
+ *
+ * But the huge tmpfs PageTeam case is handled differently:
+ * there are no arbitrary restrictions on mlocking such pages,
+ * and compound_mapcount() returns 0 even when they are mapped.
*/
- if (compound_mapcount(page) == 1 && !PageDoubleMap(page) &&
+ if (compound_mapcount(page) <= 1 && !PageDoubleMap(page) &&
page->mapping && trylock_page(page)) {
lru_add_drain();
if (page->mapping)
- mlock_vma_page(page);
+ mlock_vma_pages(page, HPAGE_PMD_NR);
unlock_page(page);
}
}
@@ -1710,6 +1714,9 @@ int zap_huge_pmd(struct mmu_gather *tlb,
pte_free(tlb->mm, pgtable_trans_huge_withdraw(tlb->mm, pmd));
atomic_long_dec(&tlb->mm->nr_ptes);
spin_unlock(ptl);
+ if (PageTeam(page) &&
+ !team_pmd_mapped(page) && team_pmd_mlocked(page))
+ clear_pages_mlock(page, HPAGE_PMD_NR);
tlb_remove_page(tlb, page);
}
return 1;
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -275,8 +275,16 @@ static inline void munlock_vma_pages_all
/*
* must be called with vma's mmap_sem held for read or write, and page locked.
*/
-extern void mlock_vma_page(struct page *page);
-extern unsigned int munlock_vma_page(struct page *page);
+extern void mlock_vma_pages(struct page *page, int nr_pages);
+static inline void mlock_vma_page(struct page *page)
+{
+ mlock_vma_pages(page, 1);
+}
+extern int munlock_vma_pages(struct page *page, int nr_pages);
+static inline void munlock_vma_page(struct page *page)
+{
+ munlock_vma_pages(page, 1);
+}
/*
* Clear the page's PageMlocked(). This can be useful in a situation where
@@ -287,7 +295,11 @@ extern unsigned int munlock_vma_page(str
* If called for a page that is still mapped by mlocked vmas, all we do
* is revert to lazy LRU behaviour -- semantics are not broken.
*/
-extern void clear_page_mlock(struct page *page);
+extern void clear_pages_mlock(struct page *page, int nr_pages);
+static inline void clear_page_mlock(struct page *page)
+{
+ clear_pages_mlock(page, 1);
+}
/*
* mlock_migrate_page - called only from migrate_misplaced_transhuge_page()
@@ -328,13 +340,7 @@ vma_address(struct page *page, struct vm
return address;
}
-
-#else /* !CONFIG_MMU */
-static inline void clear_page_mlock(struct page *page) { }
-static inline void mlock_vma_page(struct page *page) { }
-static inline void mlock_migrate_page(struct page *new, struct page *old) { }
-
-#endif /* !CONFIG_MMU */
+#endif /* CONFIG_MMU */
/*
* Return the mem_map entry representing the 'offset' subpage within
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -11,6 +11,7 @@
#include <linux/swap.h>
#include <linux/swapops.h>
#include <linux/pagemap.h>
+#include <linux/pageteam.h>
#include <linux/pagevec.h>
#include <linux/mempolicy.h>
#include <linux/syscalls.h>
@@ -51,43 +52,72 @@ EXPORT_SYMBOL(can_do_mlock);
* (see mm/rmap.c).
*/
-/*
- * LRU accounting for clear_page_mlock()
+/**
+ * clear_pages_mlock - clear mlock from a page or pages
+ * @page - page to be unlocked
+ * @nr_pages - usually 1, but HPAGE_PMD_NR if pmd mapping is zapped.
+ *
+ * Clear the page's PageMlocked(). This can be useful in a situation where
+ * we want to unconditionally remove a page from the pagecache -- e.g.,
+ * on truncation or freeing.
+ *
+ * It is legal to call this function for any page, mlocked or not.
+ * If called for a page that is still mapped by mlocked vmas, all we do
+ * is revert to lazy LRU behaviour -- semantics are not broken.
*/
-void clear_page_mlock(struct page *page)
+void clear_pages_mlock(struct page *page, int nr_pages)
{
- if (!TestClearPageMlocked(page))
- return;
+ struct zone *zone = page_zone(page);
+ struct page *endpage = page + 1;
- mod_zone_page_state(page_zone(page), NR_MLOCK,
- -hpage_nr_pages(page));
- count_vm_event(UNEVICTABLE_PGCLEARED);
- if (!isolate_lru_page(page)) {
- putback_lru_page(page);
- } else {
- /*
- * We lost the race. the page already moved to evictable list.
- */
- if (PageUnevictable(page))
+ if (nr_pages > 1 && PageTeam(page)) {
+ clear_team_pmd_mlocked(page); /* page is team head */
+ endpage = page + nr_pages;
+ nr_pages = 1;
+ }
+
+ for (; page < endpage; page++) {
+ if (page_mapped(page))
+ continue;
+ if (!TestClearPageMlocked(page))
+ continue;
+ mod_zone_page_state(zone, NR_MLOCK, -nr_pages);
+ count_vm_event(UNEVICTABLE_PGCLEARED);
+ if (!isolate_lru_page(page))
+ putback_lru_page(page);
+ else if (PageUnevictable(page))
count_vm_event(UNEVICTABLE_PGSTRANDED);
}
}
-/*
- * Mark page as mlocked if not already.
+/**
+ * mlock_vma_pages - mlock a vma page or pages
+ * @page - page to be unlocked
+ * @nr_pages - usually 1, but HPAGE_PMD_NR if pmd mapping is mlocked.
+ *
+ * Mark pages as mlocked if not already.
* If page on LRU, isolate and putback to move to unevictable list.
*/
-void mlock_vma_page(struct page *page)
+void mlock_vma_pages(struct page *page, int nr_pages)
{
- /* Serialize with page migration */
- BUG_ON(!PageLocked(page));
+ struct zone *zone = page_zone(page);
+ struct page *endpage = page + 1;
+ /* Serialize with page migration */
+ VM_BUG_ON_PAGE(!PageLocked(page) && !PageTeam(page), page);
VM_BUG_ON_PAGE(PageTail(page), page);
VM_BUG_ON_PAGE(PageCompound(page) && PageDoubleMap(page), page);
- if (!TestSetPageMlocked(page)) {
- mod_zone_page_state(page_zone(page), NR_MLOCK,
- hpage_nr_pages(page));
+ if (nr_pages > 1 && PageTeam(page)) {
+ set_team_pmd_mlocked(page); /* page is team head */
+ endpage = page + nr_pages;
+ nr_pages = 1;
+ }
+
+ for (; page < endpage; page++) {
+ if (TestSetPageMlocked(page))
+ continue;
+ mod_zone_page_state(zone, NR_MLOCK, nr_pages);
count_vm_event(UNEVICTABLE_PGMLOCKED);
if (!isolate_lru_page(page))
putback_lru_page(page);
@@ -111,6 +141,18 @@ static bool __munlock_isolate_lru_page(s
return true;
}
+ /*
+ * Perform accounting when page isolation fails in munlock.
+ * There is nothing else to do because it means some other task has
+ * already removed the page from the LRU. putback_lru_page() will take
+ * care of removing the page from the unevictable list, if necessary.
+ * vmscan [page_referenced()] will move the page back to the
+ * unevictable list if some other vma has it mlocked.
+ */
+ if (PageUnevictable(page))
+ __count_vm_event(UNEVICTABLE_PGSTRANDED);
+ else
+ __count_vm_event(UNEVICTABLE_PGMUNLOCKED);
return false;
}
@@ -128,7 +170,7 @@ static void __munlock_isolated_page(stru
* Optimization: if the page was mapped just once, that's our mapping
* and we don't need to check all the other vmas.
*/
- if (page_mapcount(page) > 1)
+ if (page_mapcount(page) > 1 || PageTeam(page))
ret = try_to_munlock(page);
/* Did try_to_unlock() succeed or punt? */
@@ -138,29 +180,12 @@ static void __munlock_isolated_page(stru
putback_lru_page(page);
}
-/*
- * Accounting for page isolation fail during munlock
- *
- * Performs accounting when page isolation fails in munlock. There is nothing
- * else to do because it means some other task has already removed the page
- * from the LRU. putback_lru_page() will take care of removing the page from
- * the unevictable list, if necessary. vmscan [page_referenced()] will move
- * the page back to the unevictable list if some other vma has it mlocked.
- */
-static void __munlock_isolation_failed(struct page *page)
-{
- if (PageUnevictable(page))
- __count_vm_event(UNEVICTABLE_PGSTRANDED);
- else
- __count_vm_event(UNEVICTABLE_PGMUNLOCKED);
-}
-
/**
- * munlock_vma_page - munlock a vma page
- * @page - page to be unlocked, either a normal page or THP page head
+ * munlock_vma_pages - munlock a vma page or pages
+ * @page - page to be unlocked
+ * @nr_pages - usually 1, but HPAGE_PMD_NR if pmd mapping is munlocked
*
- * returns the size of the page as a page mask (0 for normal page,
- * HPAGE_PMD_NR - 1 for THP head page)
+ * returns the size of the page (usually 1, but HPAGE_PMD_NR for huge page)
*
* called from munlock()/munmap() path with page supposedly on the LRU.
* When we munlock a page, because the vma where we found the page is being
@@ -173,41 +198,56 @@ static void __munlock_isolation_failed(s
* can't isolate the page, we leave it for putback_lru_page() and vmscan
* [page_referenced()/try_to_unmap()] to deal with.
*/
-unsigned int munlock_vma_page(struct page *page)
+int munlock_vma_pages(struct page *page, int nr_pages)
{
- int nr_pages;
struct zone *zone = page_zone(page);
+ struct page *endpage = page + 1;
+ struct page *head = NULL;
+ int ret = nr_pages;
+ bool isolated;
/* For try_to_munlock() and to serialize with page migration */
- BUG_ON(!PageLocked(page));
-
+ VM_BUG_ON_PAGE(!PageLocked(page), page);
VM_BUG_ON_PAGE(PageTail(page), page);
+ if (nr_pages > 1 && PageTeam(page)) {
+ head = page;
+ clear_team_pmd_mlocked(page); /* page is team head */
+ endpage = page + nr_pages;
+ nr_pages = 1;
+ }
+
/*
- * Serialize with any parallel __split_huge_page_refcount() which
- * might otherwise copy PageMlocked to part of the tail pages before
- * we clear it in the head page. It also stabilizes hpage_nr_pages().
+ * Serialize THP with any parallel __split_huge_page_tail() which
+ * might otherwise copy PageMlocked to some of the tail pages before
+ * we clear it in the head page.
*/
spin_lock_irq(&zone->lru_lock);
+ if (nr_pages > 1 && !PageTransHuge(page))
+ ret = nr_pages = 1;
- nr_pages = hpage_nr_pages(page);
- if (!TestClearPageMlocked(page))
- goto unlock_out;
+ for (; page < endpage; page++) {
+ if (!TestClearPageMlocked(page))
+ continue;
- __mod_zone_page_state(zone, NR_MLOCK, -nr_pages);
-
- if (__munlock_isolate_lru_page(page, true)) {
+ __mod_zone_page_state(zone, NR_MLOCK, -nr_pages);
+ isolated = __munlock_isolate_lru_page(page, true);
spin_unlock_irq(&zone->lru_lock);
- __munlock_isolated_page(page);
- goto out;
- }
- __munlock_isolation_failed(page);
+ if (isolated)
+ __munlock_isolated_page(page);
-unlock_out:
+ /*
+ * If try_to_munlock() found the huge page to be still
+ * mlocked, don't waste more time munlocking and rmap
+ * walking and re-mlocking each of the team's pages.
+ */
+ if (!head || team_pmd_mlocked(head))
+ goto out;
+ spin_lock_irq(&zone->lru_lock);
+ }
spin_unlock_irq(&zone->lru_lock);
-
out:
- return nr_pages - 1;
+ return ret;
}
/*
@@ -300,8 +340,6 @@ static void __munlock_pagevec(struct pag
*/
if (__munlock_isolate_lru_page(page, false))
continue;
- else
- __munlock_isolation_failed(page);
}
/*
@@ -461,13 +499,8 @@ void munlock_vma_pages_range(struct vm_a
put_page(page); /* follow_page_mask() */
} else if (PageTransHuge(page) || PageTeam(page)) {
lock_page(page);
- /*
- * Any THP page found by follow_page_mask() may
- * have gotten split before reaching
- * munlock_vma_page(), so we need to recompute
- * the page_mask here.
- */
- page_mask = munlock_vma_page(page);
+ page_mask = munlock_vma_pages(page,
+ page_mask + 1) - 1;
unlock_page(page);
put_page(page); /* follow_page_mask() */
} else {
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -837,10 +837,15 @@ again:
spin_unlock(ptl);
goto again;
}
- pte = NULL;
+ if (ptep)
+ *ptep = NULL;
goto found;
}
+ /* TTU_MUNLOCK on PageTeam makes a second try for huge pmd only */
+ if (unlikely(!ptep))
+ return false;
+
pte = pte_offset_map(pmd, address);
if (!pte_present(*pte)) {
pte_unmap(pte);
@@ -861,8 +866,9 @@ check_pte:
pte_unmap_unlock(pte, ptl);
return false;
}
-found:
+
*ptep = pte;
+found:
*pmdp = pmd;
*ptlp = ptl;
return true;
@@ -1332,7 +1338,7 @@ static void page_remove_anon_compound_rm
}
if (unlikely(PageMlocked(page)))
- clear_page_mlock(page);
+ clear_pages_mlock(page, HPAGE_PMD_NR);
if (nr) {
__mod_zone_page_state(page_zone(page), NR_ANON_PAGES, -nr);
@@ -1418,8 +1424,17 @@ static int try_to_unmap_one(struct page
goto out;
}
- if (!page_check_address_transhuge(page, mm, address, &pmd, &pte, &ptl))
- goto out;
+ if (!page_check_address_transhuge(page, mm, address,
+ &pmd, &pte, &ptl)) {
+ if (!(flags & TTU_MUNLOCK) || !PageTeam(page))
+ goto out;
+ /* We need also to check whether head is hugely mapped here */
+ pte = NULL;
+ page = team_head(page);
+ if (!page_check_address_transhuge(page, mm, address,
+ &pmd, NULL, &ptl))
+ goto out;
+ }
/*
* If the page is mlock()d, we cannot swap it out.
@@ -1429,7 +1444,7 @@ static int try_to_unmap_one(struct page
if (!(flags & TTU_IGNORE_MLOCK)) {
if (vma->vm_flags & VM_LOCKED) {
/* Holding pte lock, we do *not* need mmap_sem here */
- mlock_vma_page(page);
+ mlock_vma_pages(page, pte ? 1 : HPAGE_PMD_NR);
ret = SWAP_MLOCK;
goto out_unmap;
}
@@ -1635,11 +1650,6 @@ int try_to_unmap(struct page *page, enum
return ret;
}
-static int page_not_mapped(struct page *page)
-{
- return !page_mapped(page);
-};
-
/**
* try_to_munlock - try to munlock a page
* @page: the page to be munlocked
@@ -1657,24 +1667,20 @@ static int page_not_mapped(struct page *
*/
int try_to_munlock(struct page *page)
{
- int ret;
struct rmap_private rp = {
.flags = TTU_MUNLOCK,
.lazyfreed = 0,
};
-
struct rmap_walk_control rwc = {
.rmap_one = try_to_unmap_one,
.arg = &rp,
- .done = page_not_mapped,
.anon_lock = page_lock_anon_vma_read,
-
};
- VM_BUG_ON_PAGE(!PageLocked(page) || PageLRU(page), page);
+ VM_BUG_ON_PAGE(!PageLocked(page) && !PageTeam(page), page);
+ VM_BUG_ON_PAGE(PageLRU(page), page);
- ret = rmap_walk(page, &rwc);
- return ret;
+ return rmap_walk(page, &rwc);
}
void __put_anon_vma(struct anon_vma *anon_vma)
@@ -1789,7 +1795,7 @@ static int rmap_walk_file(struct page *p
* structure at mapping cannot be freed and reused yet,
* so we can safely take mapping->i_mmap_rwsem.
*/
- VM_BUG_ON_PAGE(!PageLocked(page), page);
+ VM_BUG_ON_PAGE(!PageLocked(page) && !PageTeam(page), page);
if (!mapping)
return ret;
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:40 +0200 |
| Subject | [PATCH 15/31] huge tmpfs: fix Mapped meminfo, track huge & unhuge mappings |
| Message-ID | <rkGRB-1Og-35@gated-at.bofh.it> |
| In reply to | #1371950 |
Maintaining Mlocked was the difficult one, but now that it is correctly
tracked, without duplication between the 4kB and 2MB amounts, I think
we have to make a similar effort with Mapped.
But whereas mlock and munlock were already rare and slow operations,
to which we could fairly add a little more overhead in the huge tmpfs
case, ordinary mmap is not something we want to slow down further,
relative to hugetlbfs.
In the Mapped case, I think we can take small or misaligned mmaps of
huge tmpfs files as the exceptional operation, and add a little more
overhead to those, by maintaining another count for them in the head;
and by keeping both hugely and unhugely mapped counts in the one long,
can rely on cmpxchg to manage their racing transitions atomically.
That's good on 64-bit, but there are not enough free bits in a 32-bit
atomic_long_t team_usage to support this: I think we should continue
to permit huge tmpfs on 32-bit, but accept that Mapped may be doubly
counted there. (A more serious problem on 32-bit is that it would,
I think, be possible to overflow the huge mapping counter: protection
against that will need to be added.)
Now that we are maintaining NR_FILE_MAPPED correctly for huge
tmpfs, adjust vmscan's zone_unmapped_file_pages() to exclude
NR_SHMEM_PMDMAPPED, which it clearly would not want included.
Whereas minimum_image_size() in kernel/power/snapshot.c? I have
not grasped the basis for that calculation, so leaving untouched.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
include/linux/memcontrol.h | 5 +
include/linux/pageteam.h | 144 ++++++++++++++++++++++++++++++++---
mm/huge_memory.c | 34 +++++++-
mm/rmap.c | 10 +-
mm/vmscan.c | 6 +
5 files changed, 180 insertions(+), 19 deletions(-)
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -700,6 +700,11 @@ static inline bool mem_cgroup_oom_synchr
return false;
}
+static inline void mem_cgroup_update_page_stat(struct page *page,
+ enum mem_cgroup_stat_index idx, int val)
+{
+}
+
static inline void mem_cgroup_inc_page_stat(struct page *page,
enum mem_cgroup_stat_index idx)
{
--- a/include/linux/pageteam.h
+++ b/include/linux/pageteam.h
@@ -30,6 +30,30 @@ static inline struct page *team_head(str
}
/*
+ * Layout of team head's page->team_usage field, as on x86_64 and arm64_4K:
+ *
+ * 63 32 31 22 21 12 11 10 9 0
+ * +------------+--------------+----------+----------+---------+------------+
+ * | pmd_mapped & instantiated |pte_mapped| reserved | mlocked | lru_weight |
+ * | 42 bits 10 bits | 10 bits | 1 bit | 1 bit | 10 bits |
+ * +------------+--------------+----------+----------+---------+------------+
+ *
+ * TEAM_LRU_WEIGHT_ONE 1 (1<<0)
+ * TEAM_LRU_WEIGHT_MASK 3ff (1<<10)-1
+ * TEAM_PMD_MLOCKED 400 (1<<10)
+ * TEAM_RESERVED_FLAG 800 (1<<11)
+ * TEAM_PTE_COUNTER 1000 (1<<12)
+ * TEAM_PTE_MASK 3ff000 (1<<22)-(1<<12)
+ * TEAM_PAGE_COUNTER 400000 (1<<22)
+ * TEAM_COMPLETE 80000000 (1<<31)
+ * TEAM_MAPPING_COUNTER 400000 (1<<22)
+ * TEAM_PMD_MAPPED 80400000 (1<<31)
+ *
+ * The upper bits count up to TEAM_COMPLETE as pages are instantiated,
+ * and then, above TEAM_COMPLETE, they count huge mappings of the team.
+ * Team tails have team_usage either 1 (lru_weight 1) or 0 (lru_weight 0).
+ */
+/*
* Mask for lower bits of team_usage, giving the weight 0..HPAGE_PMD_NR of the
* page on its LRU: normal pages have weight 1, tails held unevictable until
* head is evicted have weight 0, and the head gathers weight 1..HPAGE_PMD_NR.
@@ -42,8 +66,22 @@ static inline struct page *team_head(str
*/
#define TEAM_PMD_MLOCKED (1L << (HPAGE_PMD_ORDER + 1))
#define TEAM_RESERVED_FLAG (1L << (HPAGE_PMD_ORDER + 2))
-
+#ifdef CONFIG_64BIT
+/*
+ * Count how many pages of team are individually mapped into userspace.
+ */
+#define TEAM_PTE_COUNTER (1L << (HPAGE_PMD_ORDER + 3))
+#define TEAM_HIGH_COUNTER (1L << (2*HPAGE_PMD_ORDER + 4))
+#define TEAM_PTE_MASK (TEAM_HIGH_COUNTER - TEAM_PTE_COUNTER)
+#define team_pte_count(usage) (((usage) & TEAM_PTE_MASK) / TEAM_PTE_COUNTER)
+#else /* 32-bit */
+/*
+ * Not enough bits in atomic_long_t: we prefer not to bloat struct page just to
+ * avoid duplication in Mapped, when a page is mapped both hugely and unhugely.
+ */
#define TEAM_HIGH_COUNTER (1L << (HPAGE_PMD_ORDER + 3))
+#define team_pte_count(usage) 1 /* allows for the extra page_add_file_rmap */
+#endif /* CONFIG_64BIT */
/*
* Count how many pages of team are instantiated, as it is built up.
*/
@@ -66,22 +104,110 @@ static inline bool team_pmd_mapped(struc
/*
* Returns true if this was the first mapping by pmd, whereupon mapped stats
- * need to be updated.
+ * need to be updated. Together with the number of pages which then need
+ * to be accounted (can be ignored when false returned): because some team
+ * members may have been mapped unhugely by pte, so already counted as Mapped.
*/
-static inline bool inc_team_pmd_mapped(struct page *head)
+static inline bool inc_team_pmd_mapped(struct page *head, int *nr_pages)
{
- return atomic_long_add_return(TEAM_MAPPING_COUNTER, &head->team_usage)
- < TEAM_PMD_MAPPED + TEAM_MAPPING_COUNTER;
+ long team_usage;
+
+ team_usage = atomic_long_add_return(TEAM_MAPPING_COUNTER,
+ &head->team_usage);
+ *nr_pages = HPAGE_PMD_NR - team_pte_count(team_usage);
+ return team_usage < TEAM_PMD_MAPPED + TEAM_MAPPING_COUNTER;
}
/*
* Returns true if this was the last mapping by pmd, whereupon mapped stats
- * need to be updated.
+ * need to be updated. Together with the number of pages which then need
+ * to be accounted (can be ignored when false returned): because some team
+ * members may still be mapped unhugely by pte, so remain counted as Mapped.
+ */
+static inline bool dec_team_pmd_mapped(struct page *head, int *nr_pages)
+{
+ long team_usage;
+
+ team_usage = atomic_long_sub_return(TEAM_MAPPING_COUNTER,
+ &head->team_usage);
+ *nr_pages = HPAGE_PMD_NR - team_pte_count(team_usage);
+ return team_usage < TEAM_PMD_MAPPED;
+}
+
+/*
+ * Returns true if this pte mapping is of a non-team page, or of a team page not
+ * covered by an existing huge pmd mapping: whereupon stats need to be updated.
+ * Only called when mapcount goes up from 0 to 1 i.e. _mapcount from -1 to 0.
+ */
+static inline bool inc_team_pte_mapped(struct page *page)
+{
+#ifdef CONFIG_64BIT
+ struct page *head;
+ long team_usage;
+ long old;
+
+ if (likely(!PageTeam(page)))
+ return true;
+ head = team_head(page);
+ team_usage = atomic_long_read(&head->team_usage);
+ for (;;) {
+ /* Is team now being disbanded? Stop once team_usage is reset */
+ if (unlikely(!PageTeam(head) ||
+ team_usage / TEAM_PAGE_COUNTER == 0))
+ return true;
+ /*
+ * XXX: but despite the impressive-looking cmpxchg, gthelen
+ * points out that head might be freed and reused and assigned
+ * a matching value in ->private now: tiny chance, must revisit.
+ */
+ old = atomic_long_cmpxchg(&head->team_usage,
+ team_usage, team_usage + TEAM_PTE_COUNTER);
+ if (likely(old == team_usage))
+ break;
+ team_usage = old;
+ }
+ return team_usage < TEAM_PMD_MAPPED;
+#else /* 32-bit */
+ return true;
+#endif
+}
+
+/*
+ * Returns true if this pte mapping is of a non-team page, or of a team page not
+ * covered by a remaining huge pmd mapping: whereupon stats need to be updated.
+ * Only called when mapcount goes down from 1 to 0 i.e. _mapcount from 0 to -1.
*/
-static inline bool dec_team_pmd_mapped(struct page *head)
+static inline bool dec_team_pte_mapped(struct page *page)
{
- return atomic_long_sub_return(TEAM_MAPPING_COUNTER, &head->team_usage)
- < TEAM_PMD_MAPPED;
+#ifdef CONFIG_64BIT
+ struct page *head;
+ long team_usage;
+ long old;
+
+ if (likely(!PageTeam(page)))
+ return true;
+ head = team_head(page);
+ team_usage = atomic_long_read(&head->team_usage);
+ for (;;) {
+ /* Is team now being disbanded? Stop once team_usage is reset */
+ if (unlikely(!PageTeam(head) ||
+ team_usage / TEAM_PAGE_COUNTER == 0))
+ return true;
+ /*
+ * XXX: but despite the impressive-looking cmpxchg, gthelen
+ * points out that head might be freed and reused and assigned
+ * a matching value in ->private now: tiny chance, must revisit.
+ */
+ old = atomic_long_cmpxchg(&head->team_usage,
+ team_usage, team_usage - TEAM_PTE_COUNTER);
+ if (likely(old == team_usage))
+ break;
+ team_usage = old;
+ }
+ return team_usage < TEAM_PMD_MAPPED;
+#else /* 32-bit */
+ return true;
+#endif
}
static inline void inc_lru_weight(struct page *head)
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1130,9 +1130,11 @@ int copy_huge_pmd(struct mm_struct *dst_
pmdp_set_wrprotect(src_mm, addr, src_pmd);
pmd = pmd_wrprotect(pmd);
} else {
+ int nr_pages; /* not interesting here */
+
VM_BUG_ON_PAGE(!PageTeam(src_page), src_page);
page_dup_rmap(src_page, false);
- inc_team_pmd_mapped(src_page);
+ inc_team_pmd_mapped(src_page, &nr_pages);
}
add_mm_counter(dst_mm, mm_counter(src_page), HPAGE_PMD_NR);
atomic_long_inc(&dst_mm->nr_ptes);
@@ -3499,18 +3501,40 @@ late_initcall(split_huge_pages_debugfs);
static void page_add_team_rmap(struct page *page)
{
+ int nr_pages;
+
VM_BUG_ON_PAGE(PageAnon(page), page);
VM_BUG_ON_PAGE(!PageTeam(page), page);
- if (inc_team_pmd_mapped(page))
- __inc_zone_page_state(page, NR_SHMEM_PMDMAPPED);
+
+ lock_page_memcg(page);
+ if (inc_team_pmd_mapped(page, &nr_pages)) {
+ struct zone *zone = page_zone(page);
+
+ __inc_zone_state(zone, NR_SHMEM_PMDMAPPED);
+ __mod_zone_page_state(zone, NR_FILE_MAPPED, nr_pages);
+ mem_cgroup_update_page_stat(page,
+ MEM_CGROUP_STAT_FILE_MAPPED, nr_pages);
+ }
+ unlock_page_memcg(page);
}
static void page_remove_team_rmap(struct page *page)
{
+ int nr_pages;
+
VM_BUG_ON_PAGE(PageAnon(page), page);
VM_BUG_ON_PAGE(!PageTeam(page), page);
- if (dec_team_pmd_mapped(page))
- __dec_zone_page_state(page, NR_SHMEM_PMDMAPPED);
+
+ lock_page_memcg(page);
+ if (dec_team_pmd_mapped(page, &nr_pages)) {
+ struct zone *zone = page_zone(page);
+
+ __dec_zone_state(zone, NR_SHMEM_PMDMAPPED);
+ __mod_zone_page_state(zone, NR_FILE_MAPPED, -nr_pages);
+ mem_cgroup_update_page_stat(page,
+ MEM_CGROUP_STAT_FILE_MAPPED, -nr_pages);
+ }
+ unlock_page_memcg(page);
}
int map_team_by_pmd(struct vm_area_struct *vma, unsigned long addr,
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1272,7 +1272,8 @@ void page_add_new_anon_rmap(struct page
void page_add_file_rmap(struct page *page)
{
lock_page_memcg(page);
- if (atomic_inc_and_test(&page->_mapcount)) {
+ if (atomic_inc_and_test(&page->_mapcount) &&
+ inc_team_pte_mapped(page)) {
__inc_zone_page_state(page, NR_FILE_MAPPED);
mem_cgroup_inc_page_stat(page, MEM_CGROUP_STAT_FILE_MAPPED);
}
@@ -1299,9 +1300,10 @@ static void page_remove_file_rmap(struct
* these counters are not modified in interrupt context, and
* pte lock(a spinlock) is held, which implies preemption disabled.
*/
- __dec_zone_page_state(page, NR_FILE_MAPPED);
- mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_FILE_MAPPED);
-
+ if (dec_team_pte_mapped(page)) {
+ __dec_zone_page_state(page, NR_FILE_MAPPED);
+ mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_FILE_MAPPED);
+ }
if (unlikely(PageMlocked(page)))
clear_page_mlock(page);
out:
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3685,8 +3685,12 @@ static inline unsigned long zone_unmappe
/*
* It's possible for there to be more file mapped pages than
* accounted for by the pages on the file LRU lists because
- * tmpfs pages accounted for as ANON can also be FILE_MAPPED
+ * tmpfs pages accounted for as ANON can also be FILE_MAPPED.
+ * We don't know how many, beyond the PMDMAPPED excluded below.
*/
+ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
+ file_mapped -= zone_page_state(zone, NR_SHMEM_PMDMAPPED) <<
+ HPAGE_PMD_ORDER;
return (file_lru > file_mapped) ? (file_lru - file_mapped) : 0;
}
[toc] | [prev] | [next] | [standalone]
| From | Hugh Dickins <hughd@google.com> |
|---|---|
| Date | 2016-04-05 23:50 +0200 |
| Subject | [PATCH 18/31] huge tmpfs: mem_cgroup move charge on shmem huge pages |
| Message-ID | <rkH1g-1S9-1@gated-at.bofh.it> |
| In reply to | #1371950 |
Early on, for simplicity, we disabled moving huge tmpfs pages from
one memcg to another (nowadays only required when moving a task into a
memcg having move_charge_at_immigrate exceptionally set). We're about
to add a couple of memcg stats for huge tmpfs, and will need to confront
how to handle moving those stats, so better enable moving the pages now.
Although they're discovered by the pmd's get_mctgt_type_thp(), they
have to be considered page by page, in what's usually the pte scan:
because although the common case is for each member of the team to be
owned by the same memcg, nowhere is that enforced - perhaps one day
we shall need to enforce such a limitation, but not so far.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
mm/memcontrol.c | 103 +++++++++++++++++++++++++---------------------
1 file changed, 58 insertions(+), 45 deletions(-)
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4332,6 +4332,7 @@ static int mem_cgroup_do_precharge(unsig
* 2(MC_TARGET_SWAP): if the swap entry corresponding to this pte is a
* target for charge migration. if @target is not NULL, the entry is stored
* in target->ent.
+ * 3(MC_TARGET_TEAM): if pmd entry is not an anon THP: check it page by page
*
* Called with pte lock held.
*/
@@ -4344,6 +4345,7 @@ enum mc_target_type {
MC_TARGET_NONE = 0,
MC_TARGET_PAGE,
MC_TARGET_SWAP,
+ MC_TARGET_TEAM,
};
static struct page *mc_handle_present_pte(struct vm_area_struct *vma,
@@ -4565,19 +4567,22 @@ static enum mc_target_type get_mctgt_typ
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
/*
- * We don't consider swapping or file mapped pages because THP does not
- * support them for now.
* Caller should make sure that pmd_trans_huge(pmd) is true.
*/
-static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma,
- unsigned long addr, pmd_t pmd, union mc_target *target)
+static enum mc_target_type get_mctgt_type_thp(pmd_t pmd,
+ union mc_target *target, unsigned long *pfn)
{
- struct page *page = NULL;
+ struct page *page;
enum mc_target_type ret = MC_TARGET_NONE;
page = pmd_page(pmd);
- /* Don't attempt to move huge tmpfs pages yet: can be enabled later */
- if (!(mc.flags & MOVE_ANON) || !PageAnon(page))
+ if (!PageAnon(page)) {
+ if (!(mc.flags & MOVE_FILE))
+ return ret;
+ *pfn = page_to_pfn(page);
+ return MC_TARGET_TEAM;
+ }
+ if (!(mc.flags & MOVE_ANON))
return ret;
if (page->mem_cgroup == mc.from) {
ret = MC_TARGET_PAGE;
@@ -4589,8 +4594,8 @@ static enum mc_target_type get_mctgt_typ
return ret;
}
#else
-static inline enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma,
- unsigned long addr, pmd_t pmd, union mc_target *target)
+static inline enum mc_target_type get_mctgt_type_thp(pmd_t pmd,
+ union mc_target *target, unsigned long *pfn)
{
return MC_TARGET_NONE;
}
@@ -4601,24 +4606,33 @@ static int mem_cgroup_count_precharge_pt
struct mm_walk *walk)
{
struct vm_area_struct *vma = walk->vma;
- pte_t *pte;
+ enum mc_target_type target_type;
+ unsigned long uninitialized_var(pfn);
+ pte_t ptent;
+ pte_t *pte = NULL;
spinlock_t *ptl;
ptl = pmd_trans_huge_lock(pmd, vma);
if (ptl) {
- if (get_mctgt_type_thp(vma, addr, *pmd, NULL) == MC_TARGET_PAGE)
+ target_type = get_mctgt_type_thp(*pmd, NULL, &pfn);
+ if (target_type == MC_TARGET_PAGE)
mc.precharge += HPAGE_PMD_NR;
- spin_unlock(ptl);
- return 0;
+ if (target_type != MC_TARGET_TEAM)
+ goto unlock;
+ } else {
+ if (pmd_trans_unstable(pmd))
+ return 0;
+ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
}
-
- if (pmd_trans_unstable(pmd))
- return 0;
- pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
- for (; addr != end; pte++, addr += PAGE_SIZE)
- if (get_mctgt_type(vma, addr, *pte, NULL))
+ for (; addr != end; addr += PAGE_SIZE) {
+ ptent = pte ? *(pte++) : pfn_pte(pfn++, vma->vm_page_prot);
+ if (get_mctgt_type(vma, addr, ptent, NULL))
mc.precharge++; /* increment precharge temporarily */
- pte_unmap_unlock(pte - 1, ptl);
+ }
+ if (pte)
+ pte_unmap(pte - 1);
+unlock:
+ spin_unlock(ptl);
cond_resched();
return 0;
@@ -4787,22 +4801,21 @@ static int mem_cgroup_move_charge_pte_ra
{
int ret = 0;
struct vm_area_struct *vma = walk->vma;
- pte_t *pte;
+ unsigned long uninitialized_var(pfn);
+ pte_t ptent;
+ pte_t *pte = NULL;
spinlock_t *ptl;
enum mc_target_type target_type;
union mc_target target;
struct page *page;
-
+retry:
ptl = pmd_trans_huge_lock(pmd, vma);
if (ptl) {
- if (mc.precharge < HPAGE_PMD_NR) {
- spin_unlock(ptl);
- return 0;
- }
- target_type = get_mctgt_type_thp(vma, addr, *pmd, &target);
+ target_type = get_mctgt_type_thp(*pmd, &target, &pfn);
if (target_type == MC_TARGET_PAGE) {
page = target.page;
- if (!isolate_lru_page(page)) {
+ if (mc.precharge >= HPAGE_PMD_NR &&
+ !isolate_lru_page(page)) {
if (!mem_cgroup_move_account(page, true,
mc.from, mc.to)) {
mc.precharge -= HPAGE_PMD_NR;
@@ -4811,22 +4824,19 @@ static int mem_cgroup_move_charge_pte_ra
putback_lru_page(page);
}
put_page(page);
+ addr = end;
}
- spin_unlock(ptl);
- return 0;
+ if (target_type != MC_TARGET_TEAM)
+ goto unlock;
+ /* addr is not aligned when retrying after precharge ran out */
+ pfn += (addr & (HPAGE_PMD_SIZE-1)) >> PAGE_SHIFT;
+ } else {
+ if (pmd_trans_unstable(pmd))
+ return 0;
+ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
}
-
- if (pmd_trans_unstable(pmd))
- return 0;
-retry:
- pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
- for (; addr != end; addr += PAGE_SIZE) {
- pte_t ptent = *(pte++);
- swp_entry_t ent;
-
- if (!mc.precharge)
- break;
-
+ for (; addr != end && mc.precharge; addr += PAGE_SIZE) {
+ ptent = pte ? *(pte++) : pfn_pte(pfn++, vma->vm_page_prot);
switch (get_mctgt_type(vma, addr, ptent, &target)) {
case MC_TARGET_PAGE:
page = target.page;
@@ -4851,8 +4861,8 @@ put: /* get_mctgt_type() gets the page
put_page(page);
break;
case MC_TARGET_SWAP:
- ent = target.ent;
- if (!mem_cgroup_move_swap_account(ent, mc.from, mc.to)) {
+ if (!mem_cgroup_move_swap_account(target.ent,
+ mc.from, mc.to)) {
mc.precharge--;
/* we fixup refcnts and charges later. */
mc.moved_swap++;
@@ -4862,7 +4872,10 @@ put: /* get_mctgt_type() gets the page
break;
}
}
- pte_unmap_unlock(pte - 1, ptl);
+ if (pte)
+ pte_unmap(pte - 1);
+unlock:
+ spin_unlock(ptl);
cond_resched();
if (addr != end) {
[toc] | [prev] | [next] | [standalone]
Page 1 of 2 [1] 2 Next page →
Back to top | Article view | linux.kernel
csiph-web