Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1340741 > unrolled thread
| Started by | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| First post | 2016-02-23 16:10 +0100 |
| Last post | 2016-02-24 11:50 +0100 |
| Articles | 20 on this page of 48 — 4 participants |
Back to article view | Back to linux.kernel
[RFC PATCH 00/27] Move LRU page reclaim from zones to nodes v2 Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:10 +0100
[PATCH 06/27] mm, vmscan: Begin reclaiming pages on a per-node basis Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:10 +0100
Re: [PATCH 06/27] mm, vmscan: Begin reclaiming pages on a per-node basis Johannes Weiner <hannes@cmpxchg.org> - 2016-02-23 20:00 +0100
Re: [PATCH 06/27] mm, vmscan: Begin reclaiming pages on a per-node basis Johannes Weiner <hannes@cmpxchg.org> - 2016-02-23 20:10 +0100
Re: [PATCH 06/27] mm, vmscan: Begin reclaiming pages on a per-node basis Mel Gorman <mgorman@techsingularity.net> - 2016-02-24 11:30 +0100
[PATCH 15/27] mm, workingset: Make working set detection node-aware Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:10 +0100
Re: [PATCH 15/27] mm, workingset: Make working set detection node-aware Johannes Weiner <hannes@cmpxchg.org> - 2016-02-28 17:20 +0100
[PATCH 13/27] mm, vmscan: Make shrink_node decisions more node-centric Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:10 +0100
[PATCH 09/27] mm, vmscan: Simplify the logic deciding whether kswapd sleeps Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:10 +0100
Re: [PATCH 09/27] mm, vmscan: Simplify the logic deciding whether kswapd sleeps Johannes Weiner <hannes@cmpxchg.org> - 2016-02-28 17:20 +0100
[PATCH 10/27] mm, vmscan: By default have direct reclaim only shrink once per node Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:10 +0100
Re: [PATCH 10/27] mm, vmscan: By default have direct reclaim only shrink once per node Johannes Weiner <hannes@cmpxchg.org> - 2016-02-28 17:20 +0100
[PATCH 12/27] mm: vmscan: Do not reclaim from kswapd if there is any eligible zone Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:10 +0100
[PATCH 07/27] mm, vmscan: Have kswapd only scan based on the highest requested zone Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
Re: [PATCH 07/27] mm, vmscan: Have kswapd only scan based on the highest requested zone Johannes Weiner <hannes@cmpxchg.org> - 2016-02-25 23:20 +0100
[PATCH 19/27] mm: Move most file-based accounting to the node Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
[PATCH 18/27] mm: Rename NR_ANON_PAGES to NR_ANON_MAPPED Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
[PATCH 01/27] mm, page_alloc: Use ac->classzone_idx instead of zone_idx(preferred_zone) Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
Re: [PATCH 01/27] mm, page_alloc: Use ac->classzone_idx instead of zone_idx(preferred_zone) Johannes Weiner <hannes@cmpxchg.org> - 2016-02-23 19:10 +0100
Re: [PATCH 01/27] mm, page_alloc: Use ac->classzone_idx instead of zone_idx(preferred_zone) Vlastimil Babka <vbabka@suse.cz> - 2016-03-03 11:40 +0100
[PATCH 04/27] mm, vmscan: Move lru_lock to the node Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
Re: [PATCH 04/27] mm, vmscan: Move lru_lock to the node Johannes Weiner <hannes@cmpxchg.org> - 2016-02-23 19:50 +0100
[PATCH 17/27] mm: Move page mapped accounting to the node Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
[PATCH 03/27] mm, vmstat: Add infrastructure for per-node vmstats Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
Re: [PATCH 03/27] mm, vmstat: Add infrastructure for per-node vmstats Johannes Weiner <hannes@cmpxchg.org> - 2016-02-23 19:20 +0100
Re: [PATCH 03/27] mm, vmstat: Add infrastructure for per-node vmstats Mel Gorman <mgorman@techsingularity.net> - 2016-02-24 10:20 +0100
[PATCH 21/27] mm, vmscan: Update classzone_idx if buffer_heads_over_limit Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
[PATCH 11/27] mm, vmscan: Clear congestion, dirty and need for compaction on a per-node basis Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
[PATCH 16/27] mm, page_alloc: Consider dirtyable memory in terms of nodes Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
Re: [PATCH 16/27] mm, page_alloc: Consider dirtyable memory in terms of nodes Johannes Weiner <hannes@cmpxchg.org> - 2016-02-28 17:20 +0100
[PATCH 20/27] mm: Move vmscan writes and file write accounting to the node Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
[PATCH 02/27] mm, vmscan: Check if cpusets are enabled during direct reclaim Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:20 +0100
Re: [PATCH 02/27] mm, vmscan: Check if cpusets are enabled during direct reclaim Johannes Weiner <hannes@cmpxchg.org> - 2016-02-23 19:10 +0100
Re: [PATCH 02/27] mm, vmscan: Check if cpusets are enabled during direct reclaim Vlastimil Babka <vbabka@suse.cz> - 2016-03-03 12:40 +0100
[PATCH 27/27] mm: page_alloc: Cache the last node whose dirty limit is reached Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:30 +0100
[PATCH 25/27] mm, vmscan: Add classzone information to tracepoints Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:30 +0100
[PATCH 23/27] mm, vmscan: Account in vmstat for pages skipped during reclaim Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:30 +0100
[PATCH 22/27] mm, vmscan: Only wakeup kswapd once per node for the requested classzone Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:30 +0100
[PATCH 24/27] mm: Convert zone_reclaim to node_reclaim Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:30 +0100
[PATCH 26/27] mm, page_alloc: Remove fair zone allocation policy Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 16:30 +0100
Re: [RFC PATCH 00/27] Move LRU page reclaim from zones to nodes v2 Christoph Lameter <cl@linux.com> - 2016-02-23 18:20 +0100
Re: [PATCH 05/27] mm, vmscan: Move LRU lists to node Johannes Weiner <hannes@cmpxchg.org> - 2016-02-23 19:50 +0100
Re: [RFC PATCH 00/27] Move LRU page reclaim from zones to nodes v2 Johannes Weiner <hannes@cmpxchg.org> - 2016-02-23 21:10 +0100
Re: [RFC PATCH 00/27] Move LRU page reclaim from zones to nodes v2 Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 21:20 +0100
Re: [RFC PATCH 00/27] Move LRU page reclaim from zones to nodes v2 Johannes Weiner <hannes@cmpxchg.org> - 2016-02-23 22:00 +0100
Re: [RFC PATCH 00/27] Move LRU page reclaim from zones to nodes v2 Mel Gorman <mgorman@techsingularity.net> - 2016-02-23 23:00 +0100
Re: [RFC PATCH 00/27] Move LRU page reclaim from zones to nodes v2 Johannes Weiner <hannes@cmpxchg.org> - 2016-02-24 01:20 +0100
Re: [RFC PATCH 00/27] Move LRU page reclaim from zones to nodes v2 Mel Gorman <mgorman@techsingularity.net> - 2016-02-24 11:50 +0100
Page 1 of 3 [1] 2 3 Next page →
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:10 +0100 |
| Subject | [RFC PATCH 00/27] Move LRU page reclaim from zones to nodes v2 |
| Message-ID | <r5mL7-7at-5@gated-at.bofh.it> |
(sorry for the resend, outgoing smtp went haywire so trying again)
This is a revisit of an RFC series from last year that moves LRUs from
the zones to the node. It is based on mmotm from February 9th as it had
to be rebased on top of work there and will not apply cleanly to 4.5-rc*
Conceptually, this is simple but there are a lot of details. Some of the
broad motivations for this are;
1. The residency of a page partially depends on what zone the page was
allocated from. This is partially combatted by the fair zone allocation
policy but that is a partial solution that introduces overhead in the
page allocator paths.
2. Currently, reclaim on node 0 behaves slightly different to node 1. For
example, direct reclaim scans in zonelist order and reclaims even if
the zone is over the high watermark regardless of the age of pages
in that LRU. Kswapd on the other hand starts reclaim on the highest
unbalanced zone. A difference in distribution of file/anon pages due
to when they were allocated results can result in a difference in
again. While the fair zone allocation policy mitigates some of the
problems here, the page reclaim results on a multi-zone node will
always be different to a single-zone node.
it was scheduled on as a result.
3. kswapd and the page allocator scan zones in the opposite order to
avoid interfering with each other but it's sensitive to timing. This
mitigates the page allocator using pages that were allocated very recently
in the ideal case but it's sensitive to timing. When kswapd is allocating
from lower zones then it's great but during the rebalancing of the highest
zone, the page allocator and kswapd interfere with each other. It's worse
if the highest zone is small and difficult to balance.
4. slab shrinkers are node-based which makes it harder to identify the exact
relationship between slab reclaim and LRU reclaim.
The reason we have zone-based reclaim is that we used to have
large highmem zones in common configurations and it was necessary
to quickly find ZONE_NORMAL pages for reclaim. Today, this is much
less of a concern as machines with lots of memory will (or should) use
64-bit kernels. Combinations of 32-bit hardware and 64-bit hardware are
rare. Machines that do use highmem should have relatively low highmem:lowmem
ratios than we worried about in the past.
Conceptually, moving to node LRUs should be easier to understand. The
page allocator plays fewer tricks to game reclaim and reclaim behaves
similarly on all nodes.
It was tested on a UMA (16 cores single socket) and a NUMA machine (48 cores,
2 sockets). However, many of these results are from the UMA machine as the
NUMA machine had a bug that was causing numa balancing to push everything
out to swap. A fix for that issue has already been posted.
In many benchmarks, there is an obvious difference in the number of
allocations from each zone as the fair zone allocation policy is removed
towards the end of the series. For example, this is the allocation stats
when running blogbench that showed no difference in headling performance
mmotm-20160209 nodelru-v2
DMA allocs 0 0
DMA32 allocs 7218763 608067
Normal allocs 12701806 18821286
Movable allocs 0 0
bonnie
------
This was configured to do an IO test with a working set 2*RAM using the
ext4 filesystem. For both machines, there was no significant performance
difference between them but this is the result for the UMA machine
bonnie
4.5.0-rc3 4.5.0-rc3
mmotm-20160209 nodelru-v2
Hmean SeqOut Char 85457.62 ( 0.00%) 85376.69 ( -0.09%)
Hmean SeqOut Block 87031.13 ( 0.00%) 87523.40 ( 0.57%)
Hmean SeqOut Rewrite 36685.66 ( 0.00%) 36006.64 ( -1.85%)
Hmean SeqIn Char 76766.34 ( 0.00%) 75935.63 ( -1.08%)
Hmean SeqIn Block 105405.02 ( 0.00%) 105513.21 ( 0.10%)
Hmean Random seeks 333.03 ( 0.00%) 332.82 ( -0.07%)
Hmean SeqCreate ops 5.00 ( 0.00%) 4.62 ( -7.69%)
Hmean SeqCreate read 4.62 ( 0.00%) 4.62 ( 0.00%)
Hmean SeqCreate del 1622.44 ( 0.00%) 1633.46 ( 0.68%)
Hmean RandCreate ops 5.00 ( 0.00%) 5.00 ( 0.00%)
Hmean RandCreate read 4.62 ( 0.00%) 4.62 ( 0.00%)
Hmean RandCreate del 1664.51 ( 0.00%) 1672.79 ( 0.50%)
4.5.0-rc3 4.5.0-rc3
mmotm-20160209 nodelru-v2
User 892.43 896.96
System 160.86 156.56
Elapsed 5990.52 6005.04
However, the overall VM stats are interesting
4.5.0-rc3 4.5.0-rc3
mmotm-20160209 nodelru-v2
Swap Ins 8 0
Swap Outs 705 52
Allocation stalls 6480 0
DMA allocs 0 0
DMA32 allocs 38287801 35274742
Normal allocs 64983682 67494335
Movable allocs 0 0
Direct pages scanned 1334296 0
Kswapd pages scanned 77617741 78643061
Kswapd pages reclaimed 77493866 78481909
Direct pages reclaimed 1334220 0
Kswapd efficiency 99% 99%
Kswapd velocity 12956.762 13096.176
Direct efficiency 99% 100%
Direct velocity 222.735 0.000
Percentage direct scans 1% 0%
Note that there were no allocation stalls with this patch applied and no
direct reclaim activity.
tiobench
--------
tiobench is a flawed benchmark but it's very important in this case. tiobench
benefited from a bug prior to the fair zone allocation policy that allowed
old pages to be artificially preserved. The visible impact was that performance
exceeded the physical capabilities of the disk. With this patch applied the results are
tiobench Throughput
4.5.0-rc3 4.5.0-rc3
mmotm-20160209 nodelru-v2
Hmean PotentialReadSpeed 91.27 ( 0.00%) 89.89 ( -1.51%)
Hmean SeqRead-MB/sec-1 84.97 ( 0.00%) 84.33 ( -0.75%)
Hmean SeqRead-MB/sec-2 75.18 ( 0.00%) 75.02 ( -0.20%)
Hmean SeqRead-MB/sec-4 77.05 ( 0.00%) 77.07 ( 0.03%)
Hmean SeqRead-MB/sec-8 68.13 ( 0.00%) 67.90 ( -0.33%)
Hmean SeqRead-MB/sec-16 61.64 ( 0.00%) 61.99 ( 0.57%)
Hmean RandRead-MB/sec-1 0.92 ( 0.00%) 0.86 ( -6.49%)
Hmean RandRead-MB/sec-2 1.06 ( 0.00%) 1.09 ( 2.53%)
Hmean RandRead-MB/sec-4 1.49 ( 0.00%) 1.47 ( -1.54%)
Hmean RandRead-MB/sec-8 1.64 ( 0.00%) 1.73 ( 5.72%)
Hmean RandRead-MB/sec-16 2.02 ( 0.00%) 1.91 ( -5.45%)
Hmean SeqWrite-MB/sec-1 83.03 ( 0.00%) 82.91 ( -0.15%)
Hmean SeqWrite-MB/sec-2 77.46 ( 0.00%) 77.43 ( -0.03%)
Hmean SeqWrite-MB/sec-4 80.92 ( 0.00%) 80.90 ( -0.02%)
Hmean SeqWrite-MB/sec-8 77.71 ( 0.00%) 77.36 ( -0.45%)
Hmean SeqWrite-MB/sec-16 79.23 ( 0.00%) 79.36 ( 0.17%)
Hmean RandWrite-MB/sec-1 1.19 ( 0.00%) 1.16 ( -2.29%)
Hmean RandWrite-MB/sec-2 1.00 ( 0.00%) 1.07 ( 7.03%)
Hmean RandWrite-MB/sec-4 0.96 ( 0.00%) 1.05 ( 8.67%)
Hmean RandWrite-MB/sec-8 0.94 ( 0.00%) 0.97 ( 2.76%)
Hmean RandWrite-MB/sec-16 0.95 ( 0.00%) 0.93 ( -2.42%)
Note that the performance is almost identical allowing us to conclude that
the correct reclaim behaviour granted by the fair zone allocation policy
is preserved.
stutter
-------
stutter simulates a simple workload. One part uses a lot of anonymous
memory, a second measures mmap latency and a third copies a large file.
The primary metric is checking for mmap latency.
stutter
4.5.0-rc3 4.5.0-rc3
mmotm-20160209 nodelru-v2
Min mmap 12.5114 ( 0.00%) 13.5315 ( -8.15%)
1st-qrtle mmap 14.4985 ( 0.00%) 14.3907 ( 0.74%)
2nd-qrtle mmap 14.7093 ( 0.00%) 14.5478 ( 1.10%)
3rd-qrtle mmap 15.7381 ( 0.00%) 14.7581 ( 6.23%)
Max-90% mmap 16.4561 ( 0.00%) 15.6516 ( 4.89%)
Max-93% mmap 16.9571 ( 0.00%) 15.8844 ( 6.33%)
Max-95% mmap 17.2948 ( 0.00%) 16.3679 ( 5.36%)
Max-99% mmap 21.1054 ( 0.00%) 19.9593 ( 5.43%)
Max mmap 2815.7509 ( 0.00%) 2717.4201 ( 3.49%)
Mean mmap 16.6965 ( 0.00%) 14.9653 ( 10.37%)
There is a consistent improvement in mmap latency and some of this may be due
to less direct reclaim and more kswapd activity
4.5.0-rc3 4.5.0-rc3
mmotm-20160209 nodelru-v2
Minor Faults 89868559 78842249
Major Faults 1037 899
Swap Ins 362 583
Swap Outs 0 0
Allocation stalls 65758 31410
DMA allocs 0 0
DMA32 allocs 1196649783 2633682376
Normal allocs 2227851590 1110162400
Movable allocs 0 0
Direct pages scanned 28776006 15074415
Kswapd pages scanned 13051818 30529292
Kswapd pages reclaimed 12936208 26704609
Direct pages reclaimed 28774473 15074044
Best1%Mean mmap 14.0438 ( 0.00%) 13.7945 ( 1.77%)
Other pagereclaim workloads were tested but the results are often repetitive
lmbench lat_mmap: no major performance difference, less direct reclaim scanning
parallelio: This measures how much an anonymous memory workload is affected by
large amounts of background IO. Impact on workload is roughly comparable.
fsmark: This created large numbers of zero-length files to target the shrinkers.
Shrinker activity was comparable.
Page allocator intensive workloads showed little difference as the cost
of the fair zone allocation policy does not dominate from a userspace perspective
but a microbench of just the allocator shows a difference
4.5.0-rc3 4.5.0-rc3
mmotm-20160209 nodelru-v2
Min total-odr0-1 1075.00 ( 0.00%) 606.00 ( 43.63%)
Min total-odr0-2 786.00 ( 0.00%) 456.00 ( 41.98%)
Min total-odr0-4 383.00 ( 0.00%) 377.00 ( 1.57%)
Min total-odr0-8 355.00 ( 0.00%) 554.00 (-56.06%)
Min total-odr0-16 312.00 ( 0.00%) 293.00 ( 6.09%)
Min total-odr0-32 309.00 ( 0.00%) 284.00 ( 8.09%)
Min total-odr0-64 283.00 ( 0.00%) 269.00 ( 4.95%)
Min total-odr0-128 292.00 ( 0.00%) 274.00 ( 6.16%)
Min total-odr0-256 305.00 ( 0.00%) 292.00 ( 4.26%)
Min total-odr0-512 335.00 ( 0.00%) 333.00 ( 0.60%)
Min total-odr0-1024 347.00 ( 0.00%) 347.00 ( 0.00%)
Min total-odr0-2048 361.00 ( 0.00%) 356.00 ( 1.39%)
Min total-odr0-4096 371.00 ( 0.00%) 366.00 ( 1.35%)
Min total-odr0-8192 376.00 ( 0.00%) 368.00 ( 2.13%)
Min total-odr0-16384 377.00 ( 0.00%) 368.00 ( 2.39%)
Documentation/cgroup-v1/memcg_test.txt | 4 +-
Documentation/cgroup-v1/memory.txt | 4 +-
arch/s390/appldata/appldata_mem.c | 2 +-
arch/tile/mm/pgtable.c | 18 +-
drivers/base/node.c | 73 +--
drivers/staging/android/lowmemorykiller.c | 12 +-
fs/fs-writeback.c | 4 +-
fs/fuse/file.c | 8 +-
fs/nfs/internal.h | 2 +-
fs/nfs/write.c | 2 +-
fs/proc/meminfo.c | 14 +-
include/linux/backing-dev.h | 2 +-
include/linux/memcontrol.h | 30 +-
include/linux/mm_inline.h | 4 +-
include/linux/mm_types.h | 2 +-
include/linux/mmzone.h | 156 +++---
include/linux/swap.h | 15 +-
include/linux/topology.h | 2 +-
include/linux/vm_event_item.h | 11 +-
include/linux/vmstat.h | 106 +++-
include/linux/writeback.h | 2 +-
include/trace/events/vmscan.h | 40 +-
include/trace/events/writeback.h | 10 +-
kernel/power/snapshot.c | 10 +-
kernel/sysctl.c | 4 +-
mm/backing-dev.c | 15 +-
mm/compaction.c | 28 +-
mm/filemap.c | 14 +-
mm/huge_memory.c | 14 +-
mm/internal.h | 11 +-
mm/memcontrol.c | 235 ++++-----
mm/memory-failure.c | 4 +-
mm/memory_hotplug.c | 7 +-
mm/mempolicy.c | 2 +-
mm/migrate.c | 35 +-
mm/mlock.c | 12 +-
mm/mmap.c | 4 +-
mm/nommu.c | 4 +-
mm/page-writeback.c | 119 ++---
mm/page_alloc.c | 269 +++++-----
mm/page_idle.c | 4 +-
mm/rmap.c | 15 +-
mm/shmem.c | 12 +-
mm/swap.c | 66 +--
mm/swap_state.c | 4 +-
mm/vmscan.c | 828 ++++++++++++++----------------
mm/vmstat.c | 363 ++++++++++---
mm/workingset.c | 51 +-
48 files changed, 1455 insertions(+), 1198 deletions(-)
--
2.6.4
[toc] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:10 +0100 |
| Subject | [PATCH 06/27] mm, vmscan: Begin reclaiming pages on a per-node basis |
| Message-ID | <r5mL8-7at-27@gated-at.bofh.it> |
| In reply to | #1340741 |
This patch makes reclaim decisions on a per-node basis. A reclaimer knows
what zone is required by the allocation request and skips pages from
higher zones. In many cases this will be ok because it's a GFP_HIGHMEM
request of some description. On 64-bit, ZONE_DMA32 requests will cause
some problems but 32-bit devices on 64-bit platforms are increasingly
rare. Historically it would have been a major problem on 32-bit with big
Highmem:Lowmem ratios but such configurations are also now rare and even
where they exist, they are not encouraged. If it really becomes a problem,
it'll manifest as very low reclaim efficiencies.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/vmscan.c | 78 +++++++++++++++++++++++++++++++++++++------------------------
1 file changed, 48 insertions(+), 30 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 760fdea19729..3e423993d374 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -84,6 +84,9 @@ struct scan_control {
/* Scan (total_size >> priority) pages at once */
int priority;
+ /* The highest zone to isolate pages for reclaim from */
+ enum zone_type reclaim_idx;
+
unsigned int may_writepage:1;
/* Can mapped pages be reclaimed? */
@@ -1369,6 +1372,7 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,
struct list_head *src = &lruvec->lists[lru];
unsigned long nr_taken = 0;
unsigned long scan;
+ LIST_HEAD(pages_skipped);
for (scan = 0; scan < nr_to_scan && nr_taken < nr_to_scan &&
!list_empty(src); scan++) {
@@ -1380,6 +1384,11 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,
VM_BUG_ON_PAGE(!PageLRU(page), page);
+ if (page_zonenum(page) > sc->reclaim_idx) {
+ list_move(&page->lru, &pages_skipped);
+ continue;
+ }
+
switch (__isolate_lru_page(page, mode)) {
case 0:
nr_pages = hpage_nr_pages(page);
@@ -1398,6 +1407,15 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan,
}
}
+ /*
+ * Splice any skipped pages to the start of the LRU list. Note that
+ * this disrupts the LRU order when reclaiming for lower zones but
+ * we cannot splice to the tail. If we did then the SWAP_CLUSTER_MAX
+ * scanning would soon rescan the same pages to skip and put the
+ * system at risk of premature OOM.
+ */
+ if (!list_empty(&pages_skipped))
+ list_splice(&pages_skipped, src);
*nr_scanned = scan;
trace_mm_vmscan_lru_isolate(sc->order, nr_to_scan, scan,
nr_taken, mode, is_file_lru(lru));
@@ -1560,7 +1578,7 @@ static int current_may_throttle(void)
}
/*
- * shrink_inactive_list() is a helper for shrink_zone(). It returns the number
+ * shrink_inactive_list() is a helper for shrink_node(). It returns the number
* of reclaimed pages
*/
static noinline_for_stack unsigned long
@@ -2394,12 +2412,14 @@ static inline bool should_continue_reclaim(struct zone *zone,
}
}
-static bool shrink_zone(struct zone *zone, struct scan_control *sc,
- bool is_classzone)
+static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc,
+ enum zone_type reclaim_idx,
+ enum zone_type classzone_idx)
{
struct reclaim_state *reclaim_state = current->reclaim_state;
unsigned long nr_reclaimed, nr_scanned;
bool reclaimable = false;
+ struct zone *zone = &pgdat->node_zones[classzone_idx];
do {
struct mem_cgroup *root = sc->target_mem_cgroup;
@@ -2428,10 +2448,11 @@ static bool shrink_zone(struct zone *zone, struct scan_control *sc,
reclaimed = sc->nr_reclaimed;
scanned = sc->nr_scanned;
+ sc->reclaim_idx = reclaim_idx;
shrink_zone_memcg(zone, memcg, sc, &lru_pages);
zone_lru_pages += lru_pages;
- if (memcg && is_classzone)
+ if (!global_reclaim(sc) && reclaim_idx == classzone_idx)
shrink_slab(sc->gfp_mask, zone_to_nid(zone),
memcg, sc->nr_scanned - scanned,
lru_pages);
@@ -2462,7 +2483,7 @@ static bool shrink_zone(struct zone *zone, struct scan_control *sc,
* Shrink the slab caches in the same proportion that
* the eligible LRU pages were scanned.
*/
- if (global_reclaim(sc) && is_classzone)
+ if (global_reclaim(sc) && reclaim_idx == classzone_idx)
shrink_slab(sc->gfp_mask, zone_to_nid(zone), NULL,
sc->nr_scanned - nr_scanned,
zone_lru_pages);
@@ -2539,14 +2560,14 @@ static inline bool compaction_ready(struct zone *zone, int order)
* If a zone is deemed to be full of pinned pages then just give it a light
* scan then give up on it.
*/
-static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
+static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc,
+ enum zone_type reclaim_idx, enum zone_type classzone_idx)
{
struct zoneref *z;
struct zone *zone;
unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned;
gfp_t orig_mask;
- enum zone_type requested_highidx = gfp_zone(sc->gfp_mask);
/*
* If the number of buffer_heads in the machine exceeds the maximum
@@ -2558,16 +2579,12 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
sc->gfp_mask |= __GFP_HIGHMEM;
for_each_zone_zonelist_nodemask(zone, z, zonelist,
- requested_highidx, sc->nodemask) {
- enum zone_type classzone_idx;
-
- if (!populated_zone(zone))
- continue;
-
- classzone_idx = requested_highidx;
- while (!populated_zone(zone->zone_pgdat->node_zones +
- classzone_idx))
+ classzone_idx, sc->nodemask) {
+ if (!populated_zone(zone)) {
+ reclaim_idx--;
classzone_idx--;
+ continue;
+ }
/*
* Take care memory controller reclaiming has small influence
@@ -2593,7 +2610,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
*/
if (IS_ENABLED(CONFIG_COMPACTION) &&
sc->order > PAGE_ALLOC_COSTLY_ORDER &&
- zonelist_zone_idx(z) <= requested_highidx &&
+ zonelist_zone_idx(z) <= classzone_idx &&
compaction_ready(zone, sc->order)) {
sc->compaction_ready = true;
continue;
@@ -2611,10 +2628,10 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
&nr_soft_scanned);
sc->nr_reclaimed += nr_soft_reclaimed;
sc->nr_scanned += nr_soft_scanned;
- /* need some check for avoid more shrink_zone() */
+ /* need some check for avoid more shrink_node() */
}
- shrink_zone(zone, sc, zone_idx(zone));
+ shrink_node(zone->zone_pgdat, sc, reclaim_idx, classzone_idx);
}
/*
@@ -2646,6 +2663,7 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
int initial_priority = sc->priority;
unsigned long total_scanned = 0;
unsigned long writeback_threshold;
+ enum zone_type classzone_idx = gfp_zone(sc->gfp_mask);
retry:
delayacct_freepages_start();
@@ -2656,7 +2674,7 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
vmpressure_prio(sc->gfp_mask, sc->target_mem_cgroup,
sc->priority);
sc->nr_scanned = 0;
- shrink_zones(zonelist, sc);
+ shrink_zones(zonelist, sc, classzone_idx, classzone_idx);
total_scanned += sc->nr_scanned;
if (sc->nr_reclaimed >= sc->nr_to_reclaim)
@@ -3112,7 +3130,7 @@ static bool kswapd_shrink_zone(struct zone *zone,
balance_gap, classzone_idx))
return true;
- shrink_zone(zone, sc, zone_idx(zone) == classzone_idx);
+ shrink_node(zone->zone_pgdat, sc, zone_idx(zone), classzone_idx);
/* TODO: ANOMALY */
clear_bit(PGDAT_WRITEBACK, &pgdat->flags);
@@ -3161,6 +3179,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
unsigned long nr_soft_scanned;
struct scan_control sc = {
.gfp_mask = GFP_KERNEL,
+ .reclaim_idx = MAX_NR_ZONES - 1,
.order = order,
.priority = DEF_PRIORITY,
.may_writepage = !laptop_mode,
@@ -3231,15 +3250,14 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
sc.may_writepage = 1;
/*
- * Now scan the zone in the dma->highmem direction, stopping
- * at the last zone which needs scanning.
- *
- * We do this because the page allocator works in the opposite
- * direction. This prevents the page allocator from allocating
- * pages behind kswapd's direction of progress, which would
- * cause too much scanning of the lower zones.
+ * Continue scanning in the highmem->dma direction stopping at
+ * the last zone which needs scanning. This may reclaim lowmem
+ * pages that are not necessary for zone balancing but it
+ * preserves LRU ordering. It is assumed that the bulk of
+ * allocation requests can use arbitrary zones with the
+ * possible exception of big highmem:lowmem configurations.
*/
- for (i = 0; i <= end_zone; i++) {
+ for (i = end_zone; i >= end_zone; i--) {
struct zone *zone = pgdat->node_zones + i;
if (!populated_zone(zone))
@@ -3717,7 +3735,7 @@ static int __zone_reclaim(struct zone *zone, gfp_t gfp_mask, unsigned int order)
* priorities until we have enough memory freed.
*/
do {
- shrink_zone(zone, &sc, true);
+ shrink_node(zone->zone_pgdat, &sc, zone_idx(zone), zone_idx(zone));
} while (sc.nr_reclaimed < nr_pages && --sc.priority >= 0);
}
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Johannes Weiner <hannes@cmpxchg.org> |
|---|---|
| Date | 2016-02-23 20:00 +0100 |
| Subject | Re: [PATCH 06/27] mm, vmscan: Begin reclaiming pages on a per-node basis |
| Message-ID | <r5qlJ-WQ-29@gated-at.bofh.it> |
| In reply to | #1340742 |
On Tue, Feb 23, 2016 at 03:04:29PM +0000, Mel Gorman wrote:
> @@ -2428,10 +2448,11 @@ static bool shrink_zone(struct zone *zone, struct scan_control *sc,
> reclaimed = sc->nr_reclaimed;
> scanned = sc->nr_scanned;
>
> + sc->reclaim_idx = reclaim_idx;
> shrink_zone_memcg(zone, memcg, sc, &lru_pages);
> zone_lru_pages += lru_pages;
The setting of sc->reclaim_idx is unexpected here. Why not set it in
the caller and eliminate the reclaim_idx parameter?
> @@ -2558,16 +2579,12 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
> sc->gfp_mask |= __GFP_HIGHMEM;
>
> for_each_zone_zonelist_nodemask(zone, z, zonelist,
> - requested_highidx, sc->nodemask) {
It's unfortunate that we start with the lowest zone here. For Normal
allocations, the most common allocations, this will always have two
full shrink_node() rounds that skip over everything >DMA in the first,
then over everything >DMA32 in the second, even though all pages on
the node are valid reclaim candidates for that allocation.
[toc] | [prev] | [next] | [standalone]
| From | Johannes Weiner <hannes@cmpxchg.org> |
|---|---|
| Date | 2016-02-23 20:10 +0100 |
| Subject | Re: [PATCH 06/27] mm, vmscan: Begin reclaiming pages on a per-node basis |
| Message-ID | <r5qvo-1gK-29@gated-at.bofh.it> |
| In reply to | #1340949 |
On Tue, Feb 23, 2016 at 10:57:22AM -0800, Johannes Weiner wrote:
> > @@ -2558,16 +2579,12 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
> > sc->gfp_mask |= __GFP_HIGHMEM;
> >
> > for_each_zone_zonelist_nodemask(zone, z, zonelist,
> > - requested_highidx, sc->nodemask) {
>
> It's unfortunate that we start with the lowest zone here. For Normal
> allocations, the most common allocations, this will always have two
> full shrink_node() rounds that skip over everything >DMA in the first,
> then over everything >DMA32 in the second, even though all pages on
> the node are valid reclaim candidates for that allocation.
Nevermind, I'm confused about zonelist ordering. It starts with the
highest indexed zones.
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-24 11:30 +0100 |
| Subject | Re: [PATCH 06/27] mm, vmscan: Begin reclaiming pages on a per-node basis |
| Message-ID | <r5ERK-36A-59@gated-at.bofh.it> |
| In reply to | #1340949 |
On Tue, Feb 23, 2016 at 10:57:22AM -0800, Johannes Weiner wrote: > On Tue, Feb 23, 2016 at 03:04:29PM +0000, Mel Gorman wrote: > > @@ -2428,10 +2448,11 @@ static bool shrink_zone(struct zone *zone, struct scan_control *sc, > > reclaimed = sc->nr_reclaimed; > > scanned = sc->nr_scanned; > > > > + sc->reclaim_idx = reclaim_idx; > > shrink_zone_memcg(zone, memcg, sc, &lru_pages); > > zone_lru_pages += lru_pages; > > The setting of sc->reclaim_idx is unexpected here. Why not set it in > the caller and eliminate the reclaim_idx parameter? > Initially because it was easier to develop the patch for but it's good either way. I updated this patch and handled the conflicts. It's now set in the callers. Thanks. -- Mel Gorman SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:10 +0100 |
| Subject | [PATCH 15/27] mm, workingset: Make working set detection node-aware |
| Message-ID | <r5mL8-7at-25@gated-at.bofh.it> |
| In reply to | #1340741 |
Working set and refault detection is still zone-based, fix it.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
include/linux/mmzone.h | 6 +++---
mm/vmstat.c | 6 +++---
mm/workingset.c | 37 +++++++++++++++++--------------------
3 files changed, 23 insertions(+), 26 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ffe8a6e606b9..cef476813581 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -146,9 +146,6 @@ enum zone_stat_item {
NUMA_LOCAL, /* allocation from local node */
NUMA_OTHER, /* allocation from other node */
#endif
- WORKINGSET_REFAULT,
- WORKINGSET_ACTIVATE,
- WORKINGSET_NODERECLAIM,
NR_ANON_TRANSPARENT_HUGEPAGES,
NR_FREE_CMA_PAGES,
NR_VM_ZONE_STAT_ITEMS };
@@ -163,6 +160,9 @@ enum node_stat_item {
NR_ISOLATED_ANON, /* Temporary isolated pages from anon lru */
NR_ISOLATED_FILE, /* Temporary isolated pages from file lru */
NR_PAGES_SCANNED, /* pages scanned since last reclaim */
+ WORKINGSET_REFAULT,
+ WORKINGSET_ACTIVATE,
+ WORKINGSET_NODERECLAIM,
NR_VM_NODE_STAT_ITEMS
};
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 19bd521e161b..0746dd5e1e73 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -970,9 +970,6 @@ const char * const vmstat_text[] = {
"numa_local",
"numa_other",
#endif
- "workingset_refault",
- "workingset_activate",
- "workingset_nodereclaim",
"nr_anon_transparent_hugepages",
"nr_free_cma",
@@ -985,6 +982,9 @@ const char * const vmstat_text[] = {
"nr_isolated_anon",
"nr_isolated_file",
"nr_pages_scanned",
+ "workingset_refault",
+ "workingset_activate",
+ "workingset_nodereclaim",
/* enum writeback_stat_item counters */
"nr_dirty_threshold",
diff --git a/mm/workingset.c b/mm/workingset.c
index d06d69670b5d..82a3fedef0df 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -16,7 +16,7 @@
/*
* Double CLOCK lists
*
- * Per zone, two clock lists are maintained for file pages: the
+ * Per node, two clock lists are maintained for file pages: the
* inactive and the active list. Freshly faulted pages start out at
* the head of the inactive list and page reclaim scans pages from the
* tail. Pages that are accessed multiple times on the inactive list
@@ -141,11 +141,11 @@
*
* Implementation
*
- * For each zone's file LRU lists, a counter for inactive evictions
- * and activations is maintained (zone->inactive_age).
+ * For each node's file LRU lists, a counter for inactive evictions
+ * and activations is maintained (node->inactive_age).
*
* On eviction, a snapshot of this counter (along with some bits to
- * identify the zone) is stored in the now empty page cache radix tree
+ * identify the node) is stored in the now empty page cache radix tree
* slot of the evicted page. This is called a shadow entry.
*
* On cache misses for which there are shadow entries, an eligible
@@ -167,33 +167,30 @@
*/
static unsigned int bucket_order __read_mostly;
-static void *pack_shadow(int memcgid, struct zone *zone, unsigned long eviction)
+static void *pack_shadow(int memcgid, pg_data_t *pgdat, unsigned long eviction)
{
eviction >>= bucket_order;
eviction = (eviction << MEM_CGROUP_ID_SHIFT) | memcgid;
- eviction = (eviction << NODES_SHIFT) | zone_to_nid(zone);
- eviction = (eviction << ZONES_SHIFT) | zone_idx(zone);
+ eviction = (eviction << NODES_SHIFT) | pgdat->node_id;
eviction = (eviction << RADIX_TREE_EXCEPTIONAL_SHIFT);
return (void *)(eviction | RADIX_TREE_EXCEPTIONAL_ENTRY);
}
-static void unpack_shadow(void *shadow, int *memcgidp, struct zone **zonep,
+static void unpack_shadow(void *shadow, int *memcgidp, pg_data_t **pgdat,
unsigned long *evictionp)
{
unsigned long entry = (unsigned long)shadow;
- int memcgid, nid, zid;
+ int memcgid, nid;
entry >>= RADIX_TREE_EXCEPTIONAL_SHIFT;
- zid = entry & ((1UL << ZONES_SHIFT) - 1);
- entry >>= ZONES_SHIFT;
nid = entry & ((1UL << NODES_SHIFT) - 1);
entry >>= NODES_SHIFT;
memcgid = entry & ((1UL << MEM_CGROUP_ID_SHIFT) - 1);
entry >>= MEM_CGROUP_ID_SHIFT;
*memcgidp = memcgid;
- *zonep = NODE_DATA(nid)->node_zones + zid;
+ *pgdat = NODE_DATA(nid);
*evictionp = entry << bucket_order;
}
@@ -220,7 +217,7 @@ void *workingset_eviction(struct address_space *mapping, struct page *page)
lruvec = mem_cgroup_lruvec(zone->zone_pgdat, memcg);
eviction = atomic_long_inc_return(&lruvec->inactive_age);
- return pack_shadow(memcgid, zone, eviction);
+ return pack_shadow(memcgid, zone->zone_pgdat, eviction);
}
/**
@@ -228,7 +225,7 @@ void *workingset_eviction(struct address_space *mapping, struct page *page)
* @shadow: shadow entry of the evicted page
*
* Calculates and evaluates the refault distance of the previously
- * evicted page in the context of the zone it was allocated in.
+ * evicted page in the context of the node it was allocated in.
*
* Returns %true if the page should be activated, %false otherwise.
*/
@@ -240,10 +237,10 @@ bool workingset_refault(void *shadow)
unsigned long eviction;
struct lruvec *lruvec;
unsigned long refault;
- struct zone *zone;
+ struct pglist_data *pgdat;
int memcgid;
- unpack_shadow(shadow, &memcgid, &zone, &eviction);
+ unpack_shadow(shadow, &memcgid, &pgdat, &eviction);
rcu_read_lock();
/*
@@ -267,7 +264,7 @@ bool workingset_refault(void *shadow)
rcu_read_unlock();
return false;
}
- lruvec = mem_cgroup_lruvec(zone->zone_pgdat, memcg);
+ lruvec = mem_cgroup_lruvec(pgdat, memcg);
refault = atomic_long_read(&lruvec->inactive_age);
active_file = lruvec_lru_size(lruvec, LRU_ACTIVE_FILE);
rcu_read_unlock();
@@ -290,10 +287,10 @@ bool workingset_refault(void *shadow)
*/
refault_distance = (refault - eviction) & EVICTION_MASK;
- inc_zone_state(zone, WORKINGSET_REFAULT);
+ inc_node_state(pgdat, WORKINGSET_REFAULT);
if (refault_distance <= active_file) {
- inc_zone_state(zone, WORKINGSET_ACTIVATE);
+ inc_node_state(pgdat, WORKINGSET_ACTIVATE);
return true;
}
return false;
@@ -435,7 +432,7 @@ static enum lru_status shadow_lru_isolate(struct list_head *item,
}
}
BUG_ON(node->count);
- inc_zone_state(page_zone(virt_to_page(node)), WORKINGSET_NODERECLAIM);
+ inc_node_state(page_zone(virt_to_page(node))->zone_pgdat, WORKINGSET_NODERECLAIM);
if (!__radix_tree_delete_node(&mapping->page_tree, node))
BUG();
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Johannes Weiner <hannes@cmpxchg.org> |
|---|---|
| Date | 2016-02-28 17:20 +0100 |
| Subject | Re: [PATCH 15/27] mm, workingset: Make working set detection node-aware |
| Message-ID | <r7ceB-57l-1@gated-at.bofh.it> |
| In reply to | #1340743 |
On Tue, Feb 23, 2016 at 03:04:38PM +0000, Mel Gorman wrote:
> @@ -167,33 +167,30 @@
> */
> static unsigned int bucket_order __read_mostly;
>
> -static void *pack_shadow(int memcgid, struct zone *zone, unsigned long eviction)
> +static void *pack_shadow(int memcgid, pg_data_t *pgdat, unsigned long eviction)
> {
> eviction >>= bucket_order;
> eviction = (eviction << MEM_CGROUP_ID_SHIFT) | memcgid;
> - eviction = (eviction << NODES_SHIFT) | zone_to_nid(zone);
> - eviction = (eviction << ZONES_SHIFT) | zone_idx(zone);
> + eviction = (eviction << NODES_SHIFT) | pgdat->node_id;
> eviction = (eviction << RADIX_TREE_EXCEPTIONAL_SHIFT);
You need to remove ZONES_SHIFT from the EVICTION_SHIFT as well.
Other than that it looks good to me.
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:10 +0100 |
| Subject | [PATCH 13/27] mm, vmscan: Make shrink_node decisions more node-centric |
| Message-ID | <r5mL8-7at-29@gated-at.bofh.it> |
| In reply to | #1340741 |
Earlier patches focused on having direct reclaim and kswapd use data that
is node-centric for reclaiming but shrink_node() itself still uses too much
zone information. This patch removes unnecessary zone-based information
with the most important decision being whether to continue reclaim or
not. Some memcg APIs are adjusted as a result even though memcg itself
still uses some zone information.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
include/linux/memcontrol.h | 9 +++----
include/linux/mmzone.h | 4 ++--
include/linux/swap.h | 2 +-
mm/memcontrol.c | 17 +++++++-------
mm/page_alloc.c | 2 +-
mm/vmscan.c | 58 ++++++++++++++++++++++++++--------------------
mm/workingset.c | 6 ++---
7 files changed, 54 insertions(+), 44 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index f5626a5c88c2..51f4441a69fc 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -306,7 +306,8 @@ void mem_cgroup_uncharge_list(struct list_head *page_list);
void mem_cgroup_migrate(struct page *oldpage, struct page *newpage);
-struct lruvec *mem_cgroup_zone_lruvec(struct zone *, struct mem_cgroup *);
+struct lruvec *mem_cgroup_lruvec(struct pglist_data *, struct zone *zone,
+ struct mem_cgroup *);
struct lruvec *mem_cgroup_page_lruvec(struct page *, struct pglist_data *);
bool task_in_mem_cgroup(struct task_struct *task, struct mem_cgroup *memcg);
@@ -592,10 +593,10 @@ static inline void mem_cgroup_migrate(struct page *old, struct page *new)
{
}
-static inline struct lruvec *mem_cgroup_zone_lruvec(struct zone *zone,
- struct mem_cgroup *memcg)
+static inline struct lruvec *mem_cgroup_lruvec(struct pglist_data *pgdat,
+ struct zone *zone, struct mem_cgroup *memcg)
{
- return zone_lruvec(zone);
+ return node_lruvec(pgdat);
}
static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page,
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 8f164ce06627..ffe8a6e606b9 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -739,9 +739,9 @@ static inline spinlock_t *zone_lru_lock(struct zone *zone)
return &zone->zone_pgdat->lru_lock;
}
-static inline struct lruvec *zone_lruvec(struct zone *zone)
+static inline struct lruvec *node_lruvec(struct pglist_data *pgdat)
{
- return &zone->zone_pgdat->lruvec;
+ return &pgdat->lruvec;
}
static inline unsigned long pgdat_end_pfn(pg_data_t *pgdat)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index a776f6506522..2137416c1f15 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -324,7 +324,7 @@ extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
unsigned long nr_pages,
gfp_t gfp_mask,
bool may_swap);
-extern unsigned long mem_cgroup_shrink_node_zone(struct mem_cgroup *mem,
+extern unsigned long mem_cgroup_shrink_node(struct mem_cgroup *mem,
gfp_t gfp_mask, bool noswap,
struct zone *zone,
unsigned long *nr_scanned);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index e9e35424c57e..1cfb7c40fa36 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -943,22 +943,23 @@ static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
iter = mem_cgroup_iter(NULL, iter, NULL))
/**
- * mem_cgroup_zone_lruvec - get the lru list vector for a zone and memcg
+ * mem_cgroup_lruvec - get the lru list vector for a node or a memcg zone
+ * @node: node of the wanted lruvec
* @zone: zone of the wanted lruvec
* @memcg: memcg of the wanted lruvec
*
- * Returns the lru list vector holding pages for the given @zone and
- * @mem. This can be the global zone lruvec, if the memory controller
+ * Returns the lru list vector holding pages for a given @node or a given
+ * @memcg and @zone. This can be the node lruvec, if the memory controller
* is disabled.
*/
-struct lruvec *mem_cgroup_zone_lruvec(struct zone *zone,
- struct mem_cgroup *memcg)
+struct lruvec *mem_cgroup_lruvec(struct pglist_data *pgdat,
+ struct zone *zone, struct mem_cgroup *memcg)
{
struct mem_cgroup_per_zone *mz;
struct lruvec *lruvec;
if (mem_cgroup_disabled()) {
- lruvec = zone_lruvec(zone);
+ lruvec = node_lruvec(pgdat);
goto out;
}
@@ -1454,8 +1455,8 @@ static int mem_cgroup_soft_reclaim(struct mem_cgroup *root_memcg,
}
continue;
}
- total += mem_cgroup_shrink_node_zone(victim, gfp_mask, false,
- zone, &nr_scanned);
+ total += mem_cgroup_shrink_node(victim, gfp_mask, false,
+ zone, &nr_scanned);
*total_scanned += nr_scanned;
if (!soft_limit_excess(root_memcg))
break;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 34faebad3972..d320524d68c6 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5521,6 +5521,7 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat)
init_waitqueue_head(&pgdat->kcompactd_wait);
#endif
pgdat_page_ext_init(pgdat);
+ lruvec_init(node_lruvec(pgdat));
for (j = 0; j < MAX_NR_ZONES; j++) {
struct zone *zone = pgdat->node_zones + j;
@@ -5585,7 +5586,6 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat)
/* For bootup, initialized properly in watermark setup */
mod_zone_page_state(zone, NR_ALLOC_BATCH, zone->managed_pages);
- lruvec_init(zone_lruvec(zone));
if (!size)
continue;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 39ad2ab76a2b..3f67edc69e04 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2219,10 +2219,11 @@ static inline void init_tlb_ubc(void)
/*
* This is a basic per-zone page freer. Used by both kswapd and direct reclaim.
*/
-static void shrink_zone_memcg(struct zone *zone, struct mem_cgroup *memcg,
+static void shrink_node_memcg(struct pglist_data *pgdat, struct mem_cgroup *memcg,
struct scan_control *sc, unsigned long *lru_pages)
{
- struct lruvec *lruvec = mem_cgroup_zone_lruvec(zone, memcg);
+ struct zone *zone = &pgdat->node_zones[sc->reclaim_idx];
+ struct lruvec *lruvec = mem_cgroup_lruvec(pgdat, zone, memcg);
unsigned long nr[NR_LRU_LISTS];
unsigned long targets[NR_LRU_LISTS];
unsigned long nr_to_scan;
@@ -2355,13 +2356,14 @@ static bool in_reclaim_compaction(struct scan_control *sc)
* calls try_to_compact_zone() that it will have enough free pages to succeed.
* It will give up earlier than that if there is difficulty reclaiming pages.
*/
-static inline bool should_continue_reclaim(struct zone *zone,
+static inline bool should_continue_reclaim(struct pglist_data *pgdat,
unsigned long nr_reclaimed,
unsigned long nr_scanned,
struct scan_control *sc)
{
unsigned long pages_for_compaction;
unsigned long inactive_lru_pages;
+ int z;
/* If not in reclaim/compaction mode, stop */
if (!in_reclaim_compaction(sc))
@@ -2395,21 +2397,27 @@ static inline bool should_continue_reclaim(struct zone *zone,
* inactive lists are large enough, continue reclaiming
*/
pages_for_compaction = (2UL << sc->order);
- inactive_lru_pages = node_page_state(zone->zone_pgdat, NR_INACTIVE_FILE);
+ inactive_lru_pages = node_page_state(pgdat, NR_INACTIVE_FILE);
if (get_nr_swap_pages() > 0)
- inactive_lru_pages += node_page_state(zone->zone_pgdat, NR_INACTIVE_ANON);
+ inactive_lru_pages += node_page_state(pgdat, NR_INACTIVE_ANON);
if (sc->nr_reclaimed < pages_for_compaction &&
inactive_lru_pages > pages_for_compaction)
return true;
/* If compaction would go ahead or the allocation would succeed, stop */
- switch (compaction_suitable(zone, sc->order, 0, 0)) {
- case COMPACT_PARTIAL:
- case COMPACT_CONTINUE:
- return false;
- default:
- return true;
+ for (z = 0; z <= sc->reclaim_idx; z++) {
+ struct zone *zone = &pgdat->node_zones[z];
+
+ switch (compaction_suitable(zone, sc->order, 0, 0)) {
+ case COMPACT_PARTIAL:
+ case COMPACT_CONTINUE:
+ return false;
+ default:
+ /* check next zone */
+ ;
+ }
}
+ return true;
}
static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc,
@@ -2419,15 +2427,14 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc,
struct reclaim_state *reclaim_state = current->reclaim_state;
unsigned long nr_reclaimed, nr_scanned;
bool reclaimable = false;
- struct zone *zone = &pgdat->node_zones[classzone_idx];
do {
struct mem_cgroup *root = sc->target_mem_cgroup;
struct mem_cgroup_reclaim_cookie reclaim = {
- .zone = zone,
+ .zone = &pgdat->node_zones[classzone_idx],
.priority = sc->priority,
};
- unsigned long zone_lru_pages = 0;
+ unsigned long node_lru_pages = 0;
struct mem_cgroup *memcg;
nr_reclaimed = sc->nr_reclaimed;
@@ -2449,11 +2456,11 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc,
scanned = sc->nr_scanned;
sc->reclaim_idx = reclaim_idx;
- shrink_zone_memcg(zone, memcg, sc, &lru_pages);
- zone_lru_pages += lru_pages;
+ shrink_node_memcg(pgdat, memcg, sc, &lru_pages);
+ node_lru_pages += lru_pages;
if (!global_reclaim(sc) && reclaim_idx == classzone_idx)
- shrink_slab(sc->gfp_mask, zone_to_nid(zone),
+ shrink_slab(sc->gfp_mask, pgdat->node_id,
memcg, sc->nr_scanned - scanned,
lru_pages);
@@ -2465,7 +2472,7 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc,
/*
* Direct reclaim and kswapd have to scan all memory
* cgroups to fulfill the overall scan target for the
- * zone.
+ * node.
*
* Limit reclaim, on the other hand, only cares about
* nr_to_reclaim pages to be reclaimed and it will
@@ -2484,9 +2491,9 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc,
* the eligible LRU pages were scanned.
*/
if (global_reclaim(sc) && reclaim_idx == classzone_idx)
- shrink_slab(sc->gfp_mask, zone_to_nid(zone), NULL,
+ shrink_slab(sc->gfp_mask, pgdat->node_id, NULL,
sc->nr_scanned - nr_scanned,
- zone_lru_pages);
+ node_lru_pages);
if (reclaim_state) {
sc->nr_reclaimed += reclaim_state->reclaimed_slab;
@@ -2501,7 +2508,7 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc,
if (sc->nr_reclaimed - nr_reclaimed)
reclaimable = true;
- } while (should_continue_reclaim(zone, sc->nr_reclaimed - nr_reclaimed,
+ } while (should_continue_reclaim(pgdat, sc->nr_reclaimed - nr_reclaimed,
sc->nr_scanned - nr_scanned, sc));
return reclaimable;
@@ -2887,7 +2894,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
#ifdef CONFIG_MEMCG
-unsigned long mem_cgroup_shrink_node_zone(struct mem_cgroup *memcg,
+unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg,
gfp_t gfp_mask, bool noswap,
struct zone *zone,
unsigned long *nr_scanned)
@@ -2897,6 +2904,7 @@ unsigned long mem_cgroup_shrink_node_zone(struct mem_cgroup *memcg,
.target_mem_cgroup = memcg,
.may_writepage = !laptop_mode,
.may_unmap = 1,
+ .reclaim_idx = zone_idx(zone),
.may_swap = !noswap,
};
unsigned long lru_pages;
@@ -2911,11 +2919,11 @@ unsigned long mem_cgroup_shrink_node_zone(struct mem_cgroup *memcg,
/*
* NOTE: Although we can get the priority field, using it
* here is not a good idea, since it limits the pages we can scan.
- * if we don't reclaim here, the shrink_zone from balance_pgdat
+ * if we don't reclaim here, the shrink_node from balance_pgdat
* will pick up pages from other mem cgroup's as well. We hack
* the priority and make it zero.
*/
- shrink_zone_memcg(zone, memcg, &sc, &lru_pages);
+ shrink_node_memcg(zone->zone_pgdat, memcg, &sc, &lru_pages);
trace_mm_vmscan_memcg_softlimit_reclaim_end(sc.nr_reclaimed);
@@ -2973,7 +2981,7 @@ static void age_active_anon(struct pglist_data *pgdat,
memcg = mem_cgroup_iter(NULL, NULL, NULL);
do {
- struct lruvec *lruvec = mem_cgroup_zone_lruvec(zone, memcg);
+ struct lruvec *lruvec = mem_cgroup_lruvec(pgdat, zone, memcg);
if (inactive_anon_is_low(lruvec))
shrink_active_list(SWAP_CLUSTER_MAX, lruvec,
diff --git a/mm/workingset.c b/mm/workingset.c
index 173399b239be..f68cc74a7795 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -218,7 +218,7 @@ void *workingset_eviction(struct address_space *mapping, struct page *page)
VM_BUG_ON_PAGE(page_count(page), page);
VM_BUG_ON_PAGE(!PageLocked(page), page);
- lruvec = mem_cgroup_zone_lruvec(zone, memcg);
+ lruvec = mem_cgroup_lruvec(zone->zone_pgdat, zone, memcg);
eviction = atomic_long_inc_return(&lruvec->inactive_age);
return pack_shadow(memcgid, zone, eviction);
}
@@ -267,7 +267,7 @@ bool workingset_refault(void *shadow)
rcu_read_unlock();
return false;
}
- lruvec = mem_cgroup_zone_lruvec(zone, memcg);
+ lruvec = mem_cgroup_lruvec(zone->zone_pgdat, zone, memcg);
refault = atomic_long_read(&lruvec->inactive_age);
active_file = lruvec_lru_size(lruvec, LRU_ACTIVE_FILE);
rcu_read_unlock();
@@ -317,7 +317,7 @@ void workingset_activation(struct page *page)
*/
if (!mem_cgroup_disabled() && !page_memcg(page))
goto out;
- lruvec = mem_cgroup_zone_lruvec(page_zone(page), page_memcg(page));
+ lruvec = mem_cgroup_lruvec(page_zone(page)->zone_pgdat, page_zone(page), page_memcg(page));
atomic_long_inc(&lruvec->inactive_age);
out:
unlock_page_memcg(page);
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:10 +0100 |
| Subject | [PATCH 09/27] mm, vmscan: Simplify the logic deciding whether kswapd sleeps |
| Message-ID | <r5mL8-7at-31@gated-at.bofh.it> |
| In reply to | #1340741 |
kswapd goes through some complex steps trying to figure out if it
should stay awake based on the classzone_idx and the requested order.
It is unnecessarily complex and passes in an invalid classzone_idx to
balance_pgdat(). What matters most of all is whether a larger order has
been requsted and whether kswapd successfully reclaimed at the previous
order. This patch irons out the logic to check just that and the end result
is less headache inducing.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
include/linux/mmzone.h | 5 ++--
mm/compaction.c | 4 +--
mm/memory_hotplug.c | 5 ++--
mm/page_alloc.c | 2 +-
mm/vmscan.c | 80 +++++++++++++++++++++++++-------------------------
5 files changed, 49 insertions(+), 47 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 93151a9a4f56..8f164ce06627 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -667,8 +667,9 @@ typedef struct pglist_data {
wait_queue_head_t pfmemalloc_wait;
struct task_struct *kswapd; /* Protected by
mem_hotplug_begin/end() */
- int kswapd_max_order;
- enum zone_type classzone_idx;
+ int kswapd_order;
+ enum zone_type kswapd_classzone_idx;
+
#ifdef CONFIG_COMPACTION
int kcompactd_max_order;
enum zone_type kcompactd_classzone_idx;
diff --git a/mm/compaction.c b/mm/compaction.c
index 2ae7c7ea664e..a1274a1f5e66 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1828,8 +1828,8 @@ static void kcompactd_do_work(pg_data_t *pgdat)
*/
if (pgdat->kcompactd_max_order <= cc.order)
pgdat->kcompactd_max_order = 0;
- if (pgdat->classzone_idx >= cc.classzone_idx)
- pgdat->classzone_idx = pgdat->nr_zones - 1;
+ if (pgdat->kswapd_classzone_idx >= cc.classzone_idx)
+ pgdat->kcompactd_classzone_idx = pgdat->nr_zones - 1;
}
void wakeup_kcompactd(pg_data_t *pgdat, int order, int classzone_idx)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index b33fe895a35c..1cb26f8bbda1 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1147,9 +1147,10 @@ static pg_data_t __ref *hotadd_new_pgdat(int nid, u64 start)
arch_refresh_nodedata(nid, pgdat);
} else {
- /* Reset the nr_zones and classzone_idx to 0 before reuse */
+ /* Reset the nr_zones, order and classzone_idx before reuse */
pgdat->nr_zones = 0;
- pgdat->classzone_idx = 0;
+ pgdat->kswapd_order = 0;
+ pgdat->kswapd_classzone_idx = -1;
}
/* we can use NODE_DATA(nid) from here */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index b03c7b5872bf..34faebad3972 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5653,7 +5653,7 @@ void __paginginit free_area_init_node(int nid, unsigned long *zones_size,
unsigned long end_pfn = 0;
/* pg_data_t should be reset to zero when it's allocated */
- WARN_ON(pgdat->nr_zones || pgdat->classzone_idx);
+ WARN_ON(pgdat->nr_zones || pgdat->kswapd_classzone_idx);
reset_deferred_meminit(pgdat);
pgdat->node_id = nid;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 57114c130df1..0417f20aead7 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2750,7 +2750,7 @@ static bool pfmemalloc_watermark_ok(pg_data_t *pgdat)
/* kswapd must be awake if processes are being throttled */
if (!wmark_ok && waitqueue_active(&pgdat->kswapd_wait)) {
- pgdat->classzone_idx = min(pgdat->classzone_idx,
+ pgdat->kswapd_classzone_idx = min(pgdat->kswapd_classzone_idx,
(enum zone_type)ZONE_NORMAL);
wake_up_interruptible(&pgdat->kswapd_wait);
}
@@ -3235,6 +3235,12 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int order,
prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
+ /* If kswapd has not been woken recently, then full sleep */
+ if (classzone_idx == -1) {
+ classzone_idx = balanced_classzone_idx = MAX_NR_ZONES - 1;
+ goto full_sleep;
+ }
+
/* Try to sleep for a short interval */
if (prepare_kswapd_sleep(pgdat, order, remaining,
balanced_classzone_idx)) {
@@ -3243,6 +3249,7 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int order,
prepare_to_wait(&pgdat->kswapd_wait, &wait, TASK_INTERRUPTIBLE);
}
+full_sleep:
/*
* After a short sleep, check if it was a premature sleep. If not, then
* go fully to sleep until explicitly woken up.
@@ -3303,9 +3310,7 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int order,
*/
static int kswapd(void *p)
{
- unsigned long order, new_order;
- int classzone_idx, new_classzone_idx;
- int balanced_classzone_idx;
+ unsigned int order, classzone_idx;
pg_data_t *pgdat = (pg_data_t*)p;
struct task_struct *tsk = current;
@@ -3335,38 +3340,25 @@ static int kswapd(void *p)
tsk->flags |= PF_MEMALLOC | PF_SWAPWRITE | PF_KSWAPD;
set_freezable();
- order = new_order = 0;
- classzone_idx = new_classzone_idx = pgdat->nr_zones - 1;
- balanced_classzone_idx = classzone_idx;
+ pgdat->kswapd_order = order = 0;
+ pgdat->kswapd_classzone_idx = classzone_idx = -1;
for ( ; ; ) {
bool ret;
+kswapd_try_sleep:
+ kswapd_try_to_sleep(pgdat, order, classzone_idx, classzone_idx);
+
/*
- * While we were reclaiming, there might have been another
- * wakeup, so check the values.
+ * Read the new order and classzone_idx which may be -1 if
+ * kswapd_try_to_sleep() woke up after a short timeout instead
+ * of being woken by the page allocator.
*/
- new_order = pgdat->kswapd_max_order;
- new_classzone_idx = pgdat->classzone_idx;
- pgdat->kswapd_max_order = 0;
- pgdat->classzone_idx = pgdat->nr_zones - 1;
-
- if (order < new_order || classzone_idx > new_classzone_idx) {
- /*
- * Don't sleep if someone wants a larger 'order'
- * allocation or has tigher zone constraints
- */
- order = new_order;
- classzone_idx = new_classzone_idx;
- } else {
- kswapd_try_to_sleep(pgdat, order, classzone_idx,
- balanced_classzone_idx);
- order = pgdat->kswapd_max_order;
- classzone_idx = pgdat->classzone_idx;
- new_order = order;
- new_classzone_idx = classzone_idx;
- pgdat->kswapd_max_order = 0;
- pgdat->classzone_idx = pgdat->nr_zones - 1;
- }
+ order = pgdat->kswapd_order;
+ classzone_idx = pgdat->kswapd_classzone_idx;
+ if (classzone_idx == -1)
+ classzone_idx = MAX_NR_ZONES - 1;
+ pgdat->kswapd_order = 0;
+ pgdat->kswapd_classzone_idx = -1;
ret = try_to_freeze();
if (kthread_should_stop())
@@ -3376,11 +3368,19 @@ static int kswapd(void *p)
* We can speed up thawing tasks if we don't call balance_pgdat
* after returning from the refrigerator
*/
- if (!ret) {
- trace_mm_vmscan_kswapd_wake(pgdat->node_id, order);
- balanced_classzone_idx = balance_pgdat(pgdat, order,
- classzone_idx);
- }
+ if (ret)
+ continue;
+
+ /*
+ * Try reclaim the requested order but if that fails
+ * then try sleeping on the basis of the order reclaimed.
+ */
+ trace_mm_vmscan_kswapd_wake(pgdat->node_id, order);
+ if (balance_pgdat(pgdat, order, classzone_idx) < order)
+ goto kswapd_try_sleep;
+
+ order = pgdat->kswapd_order;
+ classzone_idx = pgdat->kswapd_classzone_idx;
}
tsk->flags &= ~(PF_MEMALLOC | PF_SWAPWRITE | PF_KSWAPD);
@@ -3403,10 +3403,10 @@ void wakeup_kswapd(struct zone *zone, int order, enum zone_type classzone_idx)
if (!cpuset_zone_allowed(zone, GFP_KERNEL | __GFP_HARDWALL))
return;
pgdat = zone->zone_pgdat;
- if (pgdat->kswapd_max_order < order) {
- pgdat->kswapd_max_order = order;
- pgdat->classzone_idx = min(pgdat->classzone_idx, classzone_idx);
- }
+ if (pgdat->kswapd_classzone_idx == -1)
+ pgdat->kswapd_classzone_idx = classzone_idx;
+ pgdat->kswapd_classzone_idx = max(pgdat->kswapd_classzone_idx, classzone_idx);
+ pgdat->kswapd_order = max(pgdat->kswapd_order, order);
if (!waitqueue_active(&pgdat->kswapd_wait))
return;
if (zone_balanced(zone, order, 0, 0))
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Johannes Weiner <hannes@cmpxchg.org> |
|---|---|
| Date | 2016-02-28 17:20 +0100 |
| Subject | Re: [PATCH 09/27] mm, vmscan: Simplify the logic deciding whether kswapd sleeps |
| Message-ID | <r7ceB-57l-13@gated-at.bofh.it> |
| In reply to | #1340745 |
On Tue, Feb 23, 2016 at 03:04:32PM +0000, Mel Gorman wrote: > kswapd goes through some complex steps trying to figure out if it > should stay awake based on the classzone_idx and the requested order. > It is unnecessarily complex and passes in an invalid classzone_idx to > balance_pgdat(). What matters most of all is whether a larger order has > been requsted and whether kswapd successfully reclaimed at the previous > order. This patch irons out the logic to check just that and the end result > is less headache inducing. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Johannes Weiner <hannes@cmpxchg.org>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:10 +0100 |
| Subject | [PATCH 10/27] mm, vmscan: By default have direct reclaim only shrink once per node |
| Message-ID | <r5mL9-7at-41@gated-at.bofh.it> |
| In reply to | #1340741 |
Direct reclaim iterates over all zones in the zonelist and shrinking them
but this is in conflict with node-based reclaim. In the default case,
only shrink once per node.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/vmscan.c | 24 ++++++++++++------------
1 file changed, 12 insertions(+), 12 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 0417f20aead7..fe004cb11b71 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2549,14 +2549,6 @@ static inline bool compaction_ready(struct zone *zone, int order)
* try to reclaim pages from zones which will satisfy the caller's allocation
* request.
*
- * We reclaim from a zone even if that zone is over high_wmark_pages(zone).
- * Because:
- * a) The caller may be trying to free *extra* pages to satisfy a higher-order
- * allocation or
- * b) The target zone may be at high_wmark_pages(zone) but the lower zones
- * must go *over* high_wmark_pages(zone) to satisfy the `incremental min'
- * zone defense algorithm.
- *
* If a zone is deemed to be full of pinned pages then just give it a light
* scan then give up on it.
*/
@@ -2568,6 +2560,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc,
unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned;
gfp_t orig_mask;
+ pg_data_t *last_pgdat = NULL;
/*
* If the number of buffer_heads in the machine exceeds the maximum
@@ -2580,11 +2573,18 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc,
for_each_zone_zonelist_nodemask(zone, z, zonelist,
classzone_idx, sc->nodemask) {
- if (!populated_zone(zone)) {
- reclaim_idx--;
- classzone_idx--;
+ BUG_ON(!populated_zone(zone));
+
+ /*
+ * Shrink each node in the zonelist once. If the zonelist is
+ * ordered by zone (not the default) then a node may be
+ * shrunk multiple times but in that case the user prefers
+ * lower zones being preserved
+ */
+ if (zone->zone_pgdat == last_pgdat)
continue;
- }
+ last_pgdat = zone->zone_pgdat;
+ reclaim_idx = zone_idx(zone);
/*
* Take care memory controller reclaiming has small influence
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Johannes Weiner <hannes@cmpxchg.org> |
|---|---|
| Date | 2016-02-28 17:20 +0100 |
| Subject | Re: [PATCH 10/27] mm, vmscan: By default have direct reclaim only shrink once per node |
| Message-ID | <r7ceB-57l-3@gated-at.bofh.it> |
| In reply to | #1340748 |
On Tue, Feb 23, 2016 at 03:04:33PM +0000, Mel Gorman wrote: > Direct reclaim iterates over all zones in the zonelist and shrinking them > but this is in conflict with node-based reclaim. In the default case, > only shrink once per node. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Johannes Weiner <hannes@cmpxchg.org>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:10 +0100 |
| Subject | [PATCH 12/27] mm: vmscan: Do not reclaim from kswapd if there is any eligible zone |
| Message-ID | <r5mL9-7at-49@gated-at.bofh.it> |
| In reply to | #1340741 |
kswapd scans from highest to lowest for a zone that requires balancing.
This was necessary when reclaim was per-zone to fairly age pages on
lower zones. Now that we are reclaiming on a per-node basis, any eligible
zone can be used and pages will still be aged fairly. This patch avoids
reclaiming excessively unless buffer_heads are over the limit and it's
necessary to reclaim from a higher zone than requested by the waker of
kswapd to relieve low memory pressure.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/vmscan.c | 32 +++++++++++++++++++-------------
1 file changed, 19 insertions(+), 13 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 1f5d3829d35e..39ad2ab76a2b 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3123,24 +3123,30 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
sc.nr_reclaimed = 0;
- /* Scan from the highest requested zone to dma */
+ /*
+ * If the number of buffer_heads in the machine exceeds the
+ * maximum allowed level and this node has a highmem zone,
+ * force kswapd to reclaim from it to relieve lowmem pressure.
+ */
+ if (buffer_heads_over_limit) {
+ for (i = MAX_NR_ZONES - 1; i >= 0; i++) {
+ zone = pgdat->node_zones + i;
+ if (!populated_zone(zone))
+ continue;
+
+ if (is_highmem_idx(i))
+ classzone_idx = i;
+ break;
+ }
+ }
+
+ /* Only reclaim if there are no eligible zones */
for (i = classzone_idx; i >= 0; i--) {
zone = pgdat->node_zones + i;
if (!populated_zone(zone))
continue;
- /*
- * If the number of buffer_heads in the machine
- * exceeds the maximum allowed level and this node
- * has a highmem zone, force kswapd to reclaim from
- * it to relieve lowmem pressure.
- */
- if (buffer_heads_over_limit && is_highmem_idx(i)) {
- classzone_idx = i;
- break;
- }
-
- if (!zone_balanced(zone, order, 0, 0)) {
+ if (!zone_balanced(zone, order, 0, classzone_idx)) {
classzone_idx = i;
break;
}
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:20 +0100 |
| Subject | [PATCH 07/27] mm, vmscan: Have kswapd only scan based on the highest requested zone |
| Message-ID | <r5mUN-7eb-1@gated-at.bofh.it> |
| In reply to | #1340741 |
kswapd checks all eligible zones to see if they need balancing even if it was
woken for a lower zone. This made sense when we reclaimed on a per-zone basis
because we wanted to shrink zones fairly so avoid age-inversion problems.
Ideally this is completely unnecessary when reclaiming on a per-node basis.
In theory, there may still be anomalies when all requests are for lower
zones and very old pages are preserved in higher zones but this should be
the exceptional case.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/vmscan.c | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 3e423993d374..1decaa31c4c6 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3193,11 +3193,8 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
sc.nr_reclaimed = 0;
- /*
- * Scan in the highmem->dma direction for the highest
- * zone which needs scanning
- */
- for (i = pgdat->nr_zones - 1; i >= 0; i--) {
+ /* Scan from the highest requested zone to dma */
+ for (i = classzone_idx; i >= 0; i--) {
struct zone *zone = pgdat->node_zones + i;
if (!populated_zone(zone))
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Johannes Weiner <hannes@cmpxchg.org> |
|---|---|
| Date | 2016-02-25 23:20 +0100 |
| Subject | Re: [PATCH 07/27] mm, vmscan: Have kswapd only scan based on the highest requested zone |
| Message-ID | <r6cqm-1Y1-11@gated-at.bofh.it> |
| In reply to | #1340751 |
On Tue, Feb 23, 2016 at 03:04:30PM +0000, Mel Gorman wrote: > kswapd checks all eligible zones to see if they need balancing even if it was > woken for a lower zone. This made sense when we reclaimed on a per-zone basis > because we wanted to shrink zones fairly so avoid age-inversion problems. > Ideally this is completely unnecessary when reclaiming on a per-node basis. > In theory, there may still be anomalies when all requests are for lower > zones and very old pages are preserved in higher zones but this should be > the exceptional case. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Johannes Weiner <hannes@cmpxchg.org> As I mentioned in the other subthread, this should probably be fine, although it would be good, generally, to have some sort of statistics on how much of the age-distorting reclaim is happening, and then maybe mention its existence in the changelog of this patch. Just an idea.
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:20 +0100 |
| Subject | [PATCH 19/27] mm: Move most file-based accounting to the node |
| Message-ID | <r5mUO-7eb-11@gated-at.bofh.it> |
| In reply to | #1340741 |
There are now a number of accounting oddities such as mapped file pages
being accounted for on the node while the total number of file pages are
accounted on the zone. This can be coped with to some extent but it's
confusing so this patch moves the relevant file-based accounted.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
arch/s390/appldata/appldata_mem.c | 2 +-
arch/tile/mm/pgtable.c | 8 +++---
drivers/base/node.c | 10 ++++----
drivers/staging/android/lowmemorykiller.c | 4 +--
fs/fs-writeback.c | 4 +--
fs/fuse/file.c | 8 +++---
fs/nfs/internal.h | 2 +-
fs/nfs/write.c | 2 +-
fs/proc/meminfo.c | 10 ++++----
include/linux/mmzone.h | 12 ++++-----
include/trace/events/writeback.h | 6 ++---
mm/filemap.c | 10 ++++----
mm/migrate.c | 12 ++++-----
mm/mmap.c | 4 +--
mm/nommu.c | 4 +--
mm/page-writeback.c | 42 +++++++++++++------------------
mm/page_alloc.c | 34 ++++++++++++-------------
mm/shmem.c | 12 ++++-----
mm/swap_state.c | 4 +--
mm/vmscan.c | 16 ++++++------
mm/vmstat.c | 12 ++++-----
21 files changed, 105 insertions(+), 113 deletions(-)
diff --git a/arch/s390/appldata/appldata_mem.c b/arch/s390/appldata/appldata_mem.c
index edcf2a706942..598df5708501 100644
--- a/arch/s390/appldata/appldata_mem.c
+++ b/arch/s390/appldata/appldata_mem.c
@@ -102,7 +102,7 @@ static void appldata_get_mem_data(void *data)
mem_data->totalhigh = P2K(val.totalhigh);
mem_data->freehigh = P2K(val.freehigh);
mem_data->bufferram = P2K(val.bufferram);
- mem_data->cached = P2K(global_page_state(NR_FILE_PAGES)
+ mem_data->cached = P2K(global_node_page_state(NR_FILE_PAGES)
- val.bufferram);
si_swapinfo(&val);
diff --git a/arch/tile/mm/pgtable.c b/arch/tile/mm/pgtable.c
index 2e784e84bd6f..dad42acd0f84 100644
--- a/arch/tile/mm/pgtable.c
+++ b/arch/tile/mm/pgtable.c
@@ -49,16 +49,16 @@ void show_mem(unsigned int filter)
global_node_page_state(NR_ACTIVE_FILE)),
(global_node_page_state(NR_INACTIVE_ANON) +
global_node_page_state(NR_INACTIVE_FILE)),
- global_page_state(NR_FILE_DIRTY),
- global_page_state(NR_WRITEBACK),
- global_page_state(NR_UNSTABLE_NFS),
+ global_node_page_state(NR_FILE_DIRTY),
+ global_node_page_state(NR_WRITEBACK),
+ global_node_page_state(NR_UNSTABLE_NFS),
global_page_state(NR_FREE_PAGES),
(global_page_state(NR_SLAB_RECLAIMABLE) +
global_page_state(NR_SLAB_UNRECLAIMABLE)),
global_node_page_state(NR_FILE_MAPPED),
global_page_state(NR_PAGETABLE),
global_page_state(NR_BOUNCE),
- global_page_state(NR_FILE_PAGES),
+ global_node_page_state(NR_FILE_PAGES),
get_nr_swap_pages());
for_each_zone(zone) {
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 897b6bcb36be..ec733919bc6b 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -116,18 +116,18 @@ static ssize_t node_read_meminfo(struct device *dev,
"Node %d AnonHugePages: %8lu kB\n"
#endif
,
- nid, K(sum_zone_node_page_state(nid, NR_FILE_DIRTY)),
- nid, K(sum_zone_node_page_state(nid, NR_WRITEBACK)),
- nid, K(sum_zone_node_page_state(nid, NR_FILE_PAGES)),
+ nid, K(node_page_state(pgdat, NR_FILE_DIRTY)),
+ nid, K(node_page_state(pgdat, NR_WRITEBACK)),
+ nid, K(node_page_state(pgdat, NR_FILE_PAGES)),
nid, K(node_page_state(pgdat, NR_FILE_MAPPED)),
nid, K(node_page_state(pgdat, NR_ANON_MAPPED)),
nid, K(i.sharedram),
nid, sum_zone_node_page_state(nid, NR_KERNEL_STACK) *
THREAD_SIZE / 1024,
nid, K(sum_zone_node_page_state(nid, NR_PAGETABLE)),
- nid, K(sum_zone_node_page_state(nid, NR_UNSTABLE_NFS)),
+ nid, K(node_page_state(pgdat, NR_UNSTABLE_NFS)),
nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)),
- nid, K(sum_zone_node_page_state(nid, NR_WRITEBACK_TEMP)),
+ nid, K(node_page_state(pgdat, NR_WRITEBACK_TEMP)),
nid, K(sum_zone_node_page_state(nid, NR_SLAB_RECLAIMABLE) +
sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
nid, K(sum_zone_node_page_state(nid, NR_SLAB_RECLAIMABLE)),
diff --git a/drivers/staging/android/lowmemorykiller.c b/drivers/staging/android/lowmemorykiller.c
index 7d677791d13a..e0b01b162303 100644
--- a/drivers/staging/android/lowmemorykiller.c
+++ b/drivers/staging/android/lowmemorykiller.c
@@ -89,8 +89,8 @@ static unsigned long lowmem_scan(struct shrinker *s, struct shrink_control *sc)
short selected_oom_score_adj;
int array_size = ARRAY_SIZE(lowmem_adj);
int other_free = global_page_state(NR_FREE_PAGES) - totalreserve_pages;
- int other_file = global_page_state(NR_FILE_PAGES) -
- global_page_state(NR_SHMEM) -
+ int other_file = global_node_page_state(NR_FILE_PAGES) -
+ global_node_page_state(NR_SHMEM) -
total_swapcache_pages();
if (lowmem_adj_size < array_size)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 6915c950e6e8..491990aa78e1 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -1723,8 +1723,8 @@ static struct wb_writeback_work *get_next_work_item(struct bdi_writeback *wb)
*/
static unsigned long get_nr_dirty_pages(void)
{
- return global_page_state(NR_FILE_DIRTY) +
- global_page_state(NR_UNSTABLE_NFS) +
+ return global_node_page_state(NR_FILE_DIRTY) +
+ global_node_page_state(NR_UNSTABLE_NFS) +
get_nr_dirty_inodes();
}
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index b03d253ece15..7ed9f10aa673 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -1446,7 +1446,7 @@ static void fuse_writepage_finish(struct fuse_conn *fc, struct fuse_req *req)
list_del(&req->writepages_entry);
for (i = 0; i < req->num_pages; i++) {
dec_wb_stat(&bdi->wb, WB_WRITEBACK);
- dec_zone_page_state(req->pages[i], NR_WRITEBACK_TEMP);
+ dec_node_page_state(req->pages[i], NR_WRITEBACK_TEMP);
wb_writeout_inc(&bdi->wb);
}
wake_up(&fi->page_waitq);
@@ -1636,7 +1636,7 @@ static int fuse_writepage_locked(struct page *page)
req->inode = inode;
inc_wb_stat(&inode_to_bdi(inode)->wb, WB_WRITEBACK);
- inc_zone_page_state(tmp_page, NR_WRITEBACK_TEMP);
+ inc_node_page_state(tmp_page, NR_WRITEBACK_TEMP);
spin_lock(&fc->lock);
list_add(&req->writepages_entry, &fi->writepages);
@@ -1750,7 +1750,7 @@ static bool fuse_writepage_in_flight(struct fuse_req *new_req,
spin_unlock(&fc->lock);
dec_wb_stat(&bdi->wb, WB_WRITEBACK);
- dec_zone_page_state(page, NR_WRITEBACK_TEMP);
+ dec_node_page_state(page, NR_WRITEBACK_TEMP);
wb_writeout_inc(&bdi->wb);
fuse_writepage_free(fc, new_req);
fuse_request_free(new_req);
@@ -1849,7 +1849,7 @@ static int fuse_writepages_fill(struct page *page,
req->page_descs[req->num_pages].length = PAGE_SIZE;
inc_wb_stat(&inode_to_bdi(inode)->wb, WB_WRITEBACK);
- inc_zone_page_state(tmp_page, NR_WRITEBACK_TEMP);
+ inc_node_page_state(tmp_page, NR_WRITEBACK_TEMP);
err = 0;
if (is_writeback && fuse_writepage_in_flight(req, page)) {
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 9a547aa3ec8e..4a605c36b585 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -626,7 +626,7 @@ void nfs_mark_page_unstable(struct page *page, struct nfs_commit_info *cinfo)
if (!cinfo->dreq) {
struct inode *inode = page_file_mapping(page)->host;
- inc_zone_page_state(page, NR_UNSTABLE_NFS);
+ inc_node_page_state(page, NR_UNSTABLE_NFS);
inc_wb_stat(&inode_to_bdi(inode)->wb, WB_RECLAIMABLE);
__mark_inode_dirty(inode, I_DIRTY_DATASYNC);
}
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 5754835a2886..d1e8d70f644b 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -897,7 +897,7 @@ nfs_mark_request_commit(struct nfs_page *req, struct pnfs_layout_segment *lseg,
static void
nfs_clear_page_commit(struct page *page)
{
- dec_zone_page_state(page, NR_UNSTABLE_NFS);
+ dec_node_page_state(page, NR_UNSTABLE_NFS);
dec_wb_stat(&inode_to_bdi(page_file_mapping(page)->host)->wb,
WB_RECLAIMABLE);
}
diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
index b3583d75b211..2f1ae07cac98 100644
--- a/fs/proc/meminfo.c
+++ b/fs/proc/meminfo.c
@@ -43,7 +43,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
si_swapinfo(&i);
committed = percpu_counter_read_positive(&vm_committed_as);
- cached = global_page_state(NR_FILE_PAGES) -
+ cached = global_node_page_state(NR_FILE_PAGES) -
total_swapcache_pages() - i.bufferram;
if (cached < 0)
cached = 0;
@@ -165,8 +165,8 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
#endif
K(i.totalswap),
K(i.freeswap),
- K(global_page_state(NR_FILE_DIRTY)),
- K(global_page_state(NR_WRITEBACK)),
+ K(global_node_page_state(NR_FILE_DIRTY)),
+ K(global_node_page_state(NR_WRITEBACK)),
K(global_node_page_state(NR_ANON_MAPPED)),
K(global_node_page_state(NR_FILE_MAPPED)),
K(i.sharedram),
@@ -179,9 +179,9 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
#ifdef CONFIG_QUICKLIST
K(quicklist_total_size()),
#endif
- K(global_page_state(NR_UNSTABLE_NFS)),
+ K(global_node_page_state(NR_UNSTABLE_NFS)),
K(global_page_state(NR_BOUNCE)),
- K(global_page_state(NR_WRITEBACK_TEMP)),
+ K(global_node_page_state(NR_WRITEBACK_TEMP)),
K(vm_commit_limit()),
K(committed),
(unsigned long)VMALLOC_TOTAL >> 10,
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 05a273171ab0..3e65e2abd1bb 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -119,20 +119,14 @@ enum zone_stat_item {
NR_FREE_PAGES,
NR_ALLOC_BATCH,
NR_MLOCK, /* mlock()ed pages found and moved off LRU */
- NR_FILE_PAGES,
- NR_FILE_DIRTY,
- NR_WRITEBACK,
NR_SLAB_RECLAIMABLE,
NR_SLAB_UNRECLAIMABLE,
NR_PAGETABLE, /* used for pagetables */
NR_KERNEL_STACK,
/* Second 128 byte cacheline */
- NR_UNSTABLE_NFS, /* NFS unstable pages */
NR_BOUNCE,
NR_VMSCAN_WRITE,
NR_VMSCAN_IMMEDIATE, /* Prioritise for reclaim when writeback ends */
- NR_WRITEBACK_TEMP, /* Writeback using temporary buffers */
- NR_SHMEM, /* shmem pages (included tmpfs/GEM pages) */
NR_DIRTIED, /* page dirtyings since bootup */
NR_WRITTEN, /* page writings since bootup */
#ifdef CONFIG_NUMA
@@ -163,6 +157,12 @@ enum node_stat_item {
NR_ANON_MAPPED, /* Mapped anonymous pages */
NR_FILE_MAPPED, /* pagecache pages mapped into pagetables.
only modified from process context */
+ NR_FILE_PAGES,
+ NR_FILE_DIRTY,
+ NR_WRITEBACK,
+ NR_WRITEBACK_TEMP, /* Writeback using temporary buffers */
+ NR_SHMEM, /* shmem pages (included tmpfs/GEM pages) */
+ NR_UNSTABLE_NFS, /* NFS unstable pages */
NR_VM_NODE_STAT_ITEMS
};
diff --git a/include/trace/events/writeback.h b/include/trace/events/writeback.h
index fff846b512e6..af584b2931d5 100644
--- a/include/trace/events/writeback.h
+++ b/include/trace/events/writeback.h
@@ -442,9 +442,9 @@ TRACE_EVENT(global_dirty_state,
),
TP_fast_assign(
- __entry->nr_dirty = global_page_state(NR_FILE_DIRTY);
- __entry->nr_writeback = global_page_state(NR_WRITEBACK);
- __entry->nr_unstable = global_page_state(NR_UNSTABLE_NFS);
+ __entry->nr_dirty = global_node_page_state(NR_FILE_DIRTY);
+ __entry->nr_writeback = global_node_page_state(NR_WRITEBACK);
+ __entry->nr_unstable = global_node_page_state(NR_UNSTABLE_NFS);
__entry->nr_dirtied = global_page_state(NR_DIRTIED);
__entry->nr_written = global_page_state(NR_WRITTEN);
__entry->background_thresh = background_thresh;
diff --git a/mm/filemap.c b/mm/filemap.c
index 99d921e1e1bb..0fa2f236a0fc 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -200,9 +200,9 @@ void __delete_from_page_cache(struct page *page, void *shadow)
/* hugetlb pages do not participate in page cache accounting. */
if (!PageHuge(page))
- __dec_zone_page_state(page, NR_FILE_PAGES);
+ __dec_node_page_state(page, NR_FILE_PAGES);
if (PageSwapBacked(page))
- __dec_zone_page_state(page, NR_SHMEM);
+ __dec_node_page_state(page, NR_SHMEM);
VM_BUG_ON_PAGE(page_mapped(page), page);
/*
@@ -545,9 +545,9 @@ int replace_page_cache_page(struct page *old, struct page *new, gfp_t gfp_mask)
* hugetlb pages do not participate in page cache accounting.
*/
if (!PageHuge(new))
- __inc_zone_page_state(new, NR_FILE_PAGES);
+ __inc_node_page_state(new, NR_FILE_PAGES);
if (PageSwapBacked(new))
- __inc_zone_page_state(new, NR_SHMEM);
+ __inc_node_page_state(new, NR_SHMEM);
spin_unlock_irqrestore(&mapping->tree_lock, flags);
mem_cgroup_migrate(old, new);
radix_tree_preload_end();
@@ -644,7 +644,7 @@ static int __add_to_page_cache_locked(struct page *page,
/* hugetlb pages do not participate in page cache accounting. */
if (!huge)
- __inc_zone_page_state(page, NR_FILE_PAGES);
+ __inc_node_page_state(page, NR_FILE_PAGES);
spin_unlock_irq(&mapping->tree_lock);
if (!huge)
mem_cgroup_commit_charge(page, memcg, false, false);
diff --git a/mm/migrate.c b/mm/migrate.c
index 0492a035df80..0653ab74bd58 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -413,15 +413,15 @@ int migrate_page_move_mapping(struct address_space *mapping,
* are mapped to swap space.
*/
if (newzone != oldzone) {
- __dec_zone_state(oldzone, NR_FILE_PAGES);
- __inc_zone_state(newzone, NR_FILE_PAGES);
+ __dec_node_state(oldzone->zone_pgdat, NR_FILE_PAGES);
+ __inc_node_state(newzone->zone_pgdat, NR_FILE_PAGES);
if (PageSwapBacked(page) && !PageSwapCache(page)) {
- __dec_zone_state(oldzone, NR_SHMEM);
- __inc_zone_state(newzone, NR_SHMEM);
+ __dec_node_state(oldzone->zone_pgdat, NR_SHMEM);
+ __inc_node_state(newzone->zone_pgdat, NR_SHMEM);
}
if (dirty && mapping_cap_account_dirty(mapping)) {
- __dec_zone_state(oldzone, NR_FILE_DIRTY);
- __inc_zone_state(newzone, NR_FILE_DIRTY);
+ __dec_node_state(oldzone->zone_pgdat, NR_FILE_DIRTY);
+ __inc_node_state(newzone->zone_pgdat, NR_FILE_DIRTY);
}
}
local_irq_enable();
diff --git a/mm/mmap.c b/mm/mmap.c
index e2e9f48b06c2..5acee7f5af62 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -184,7 +184,7 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
if (sysctl_overcommit_memory == OVERCOMMIT_GUESS) {
free = global_page_state(NR_FREE_PAGES);
- free += global_page_state(NR_FILE_PAGES);
+ free += global_node_page_state(NR_FILE_PAGES);
/*
* shmem pages shouldn't be counted as free in this
@@ -192,7 +192,7 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
* that won't affect the overall amount of available
* memory in the system.
*/
- free -= global_page_state(NR_SHMEM);
+ free -= global_node_page_state(NR_SHMEM);
free += get_nr_swap_pages();
diff --git a/mm/nommu.c b/mm/nommu.c
index fbf6f0f1d6c9..fcdd9f3cf632 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -1859,7 +1859,7 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
if (sysctl_overcommit_memory == OVERCOMMIT_GUESS) {
free = global_page_state(NR_FREE_PAGES);
- free += global_page_state(NR_FILE_PAGES);
+ free += global_node_page_state(NR_FILE_PAGES);
/*
* shmem pages shouldn't be counted as free in this
@@ -1867,7 +1867,7 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
* that won't affect the overall amount of available
* memory in the system.
*/
- free -= global_page_state(NR_SHMEM);
+ free -= global_node_page_state(NR_SHMEM);
free += get_nr_swap_pages();
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 4abce6295f7c..f778baeff78e 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -469,20 +469,12 @@ static unsigned long node_dirty_limit(struct pglist_data *pgdat)
*/
bool node_dirty_ok(struct pglist_data *pgdat)
{
- int z;
unsigned long limit = node_dirty_limit(pgdat);
unsigned long nr_pages = 0;
- for (z = 0; z < MAX_NR_ZONES; z++) {
- struct zone *zone = pgdat->node_zones + z;
-
- if (!populated_zone(zone))
- continue;
-
- nr_pages += zone_page_state(zone, NR_FILE_DIRTY);
- nr_pages += zone_page_state(zone, NR_UNSTABLE_NFS);
- nr_pages += zone_page_state(zone, NR_WRITEBACK);
- }
+ nr_pages += node_page_state(pgdat, NR_FILE_DIRTY);
+ nr_pages += node_page_state(pgdat, NR_UNSTABLE_NFS);
+ nr_pages += node_page_state(pgdat, NR_WRITEBACK);
return nr_pages <= limit;
}
@@ -1572,10 +1564,10 @@ static void balance_dirty_pages(struct address_space *mapping,
* written to the server's write cache, but has not yet
* been flushed to permanent storage.
*/
- nr_reclaimable = global_page_state(NR_FILE_DIRTY) +
- global_page_state(NR_UNSTABLE_NFS);
+ nr_reclaimable = global_node_page_state(NR_FILE_DIRTY) +
+ global_node_page_state(NR_UNSTABLE_NFS);
gdtc->avail = global_dirtyable_memory();
- gdtc->dirty = nr_reclaimable + global_page_state(NR_WRITEBACK);
+ gdtc->dirty = nr_reclaimable + global_node_page_state(NR_WRITEBACK);
domain_dirty_limits(gdtc);
@@ -1912,8 +1904,8 @@ bool wb_over_bg_thresh(struct bdi_writeback *wb)
* as we're trying to decide whether to put more under writeback.
*/
gdtc->avail = global_dirtyable_memory();
- gdtc->dirty = global_page_state(NR_FILE_DIRTY) +
- global_page_state(NR_UNSTABLE_NFS);
+ gdtc->dirty = global_node_page_state(NR_FILE_DIRTY) +
+ global_node_page_state(NR_UNSTABLE_NFS);
domain_dirty_limits(gdtc);
if (gdtc->dirty > gdtc->bg_thresh)
@@ -1955,8 +1947,8 @@ void throttle_vm_writeout(gfp_t gfp_mask)
*/
dirty_thresh += dirty_thresh / 10; /* wheeee... */
- if (global_page_state(NR_UNSTABLE_NFS) +
- global_page_state(NR_WRITEBACK) <= dirty_thresh)
+ if (global_node_page_state(NR_UNSTABLE_NFS) +
+ global_node_page_state(NR_WRITEBACK) <= dirty_thresh)
break;
congestion_wait(BLK_RW_ASYNC, HZ/10);
@@ -1984,8 +1976,8 @@ int dirty_writeback_centisecs_handler(struct ctl_table *table, int write,
void laptop_mode_timer_fn(unsigned long data)
{
struct request_queue *q = (struct request_queue *)data;
- int nr_pages = global_page_state(NR_FILE_DIRTY) +
- global_page_state(NR_UNSTABLE_NFS);
+ int nr_pages = global_node_page_state(NR_FILE_DIRTY) +
+ global_node_page_state(NR_UNSTABLE_NFS);
struct bdi_writeback *wb;
/*
@@ -2436,7 +2428,7 @@ void account_page_dirtied(struct page *page, struct address_space *mapping)
wb = inode_to_wb(inode);
mem_cgroup_inc_page_stat(page, MEM_CGROUP_STAT_DIRTY);
- __inc_zone_page_state(page, NR_FILE_DIRTY);
+ __inc_node_page_state(page, NR_FILE_DIRTY);
__inc_zone_page_state(page, NR_DIRTIED);
__inc_wb_stat(wb, WB_RECLAIMABLE);
__inc_wb_stat(wb, WB_DIRTIED);
@@ -2457,7 +2449,7 @@ void account_page_cleaned(struct page *page, struct address_space *mapping,
{
if (mapping_cap_account_dirty(mapping)) {
mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_DIRTY);
- dec_zone_page_state(page, NR_FILE_DIRTY);
+ dec_node_page_state(page, NR_FILE_DIRTY);
dec_wb_stat(wb, WB_RECLAIMABLE);
task_io_account_cancelled_write(PAGE_CACHE_SIZE);
}
@@ -2712,7 +2704,7 @@ int clear_page_dirty_for_io(struct page *page)
wb = unlocked_inode_to_wb_begin(inode, &locked);
if (TestClearPageDirty(page)) {
mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_DIRTY);
- dec_zone_page_state(page, NR_FILE_DIRTY);
+ dec_node_page_state(page, NR_FILE_DIRTY);
dec_wb_stat(wb, WB_RECLAIMABLE);
ret = 1;
}
@@ -2753,7 +2745,7 @@ int test_clear_page_writeback(struct page *page)
}
if (ret) {
mem_cgroup_dec_page_stat(page, MEM_CGROUP_STAT_WRITEBACK);
- dec_zone_page_state(page, NR_WRITEBACK);
+ dec_node_page_state(page, NR_WRITEBACK);
inc_zone_page_state(page, NR_WRITTEN);
}
unlock_page_memcg(page);
@@ -2794,7 +2786,7 @@ int __test_set_page_writeback(struct page *page, bool keep_write)
}
if (!ret) {
mem_cgroup_inc_page_stat(page, MEM_CGROUP_STAT_WRITEBACK);
- inc_zone_page_state(page, NR_WRITEBACK);
+ inc_node_page_state(page, NR_WRITEBACK);
}
unlock_page_memcg(page);
return ret;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 710c1bc26847..4c618662b8dc 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3817,7 +3817,7 @@ static inline void show_node(struct zone *zone)
void si_meminfo(struct sysinfo *val)
{
val->totalram = totalram_pages;
- val->sharedram = global_page_state(NR_SHMEM);
+ val->sharedram = global_node_page_state(NR_SHMEM);
val->freeram = global_page_state(NR_FREE_PAGES);
val->bufferram = nr_blockdev_pages();
val->totalhigh = totalhigh_pages;
@@ -3837,7 +3837,7 @@ void si_meminfo_node(struct sysinfo *val, int nid)
for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++)
managed_pages += pgdat->node_zones[zone_type].managed_pages;
val->totalram = managed_pages;
- val->sharedram = sum_zone_node_page_state(nid, NR_SHMEM);
+ val->sharedram = node_page_state(pgdat, NR_SHMEM);
val->freeram = sum_zone_node_page_state(nid, NR_FREE_PAGES);
#ifdef CONFIG_HIGHMEM
val->totalhigh = pgdat->node_zones[ZONE_HIGHMEM].managed_pages;
@@ -3937,13 +3937,13 @@ void show_free_areas(unsigned int filter)
global_node_page_state(NR_INACTIVE_FILE),
global_node_page_state(NR_ISOLATED_FILE),
global_node_page_state(NR_UNEVICTABLE),
- global_page_state(NR_FILE_DIRTY),
- global_page_state(NR_WRITEBACK),
- global_page_state(NR_UNSTABLE_NFS),
+ global_node_page_state(NR_FILE_DIRTY),
+ global_node_page_state(NR_WRITEBACK),
+ global_node_page_state(NR_UNSTABLE_NFS),
global_page_state(NR_SLAB_RECLAIMABLE),
global_page_state(NR_SLAB_UNRECLAIMABLE),
global_node_page_state(NR_FILE_MAPPED),
- global_page_state(NR_SHMEM),
+ global_node_page_state(NR_SHMEM),
global_page_state(NR_PAGETABLE),
global_page_state(NR_BOUNCE),
global_page_state(NR_FREE_PAGES),
@@ -3960,6 +3960,11 @@ void show_free_areas(unsigned int filter)
" isolated(anon):%lukB"
" isolated(file):%lukB"
" mapped:%lukB"
+ " dirty:%lukB"
+ " writeback:%lukB"
+ " shmem:%lukB"
+ " writeback_tmp:%lukB"
+ " unstable:%lukB"
" all_unreclaimable? %s"
"\n",
pgdat->node_id,
@@ -3971,6 +3976,11 @@ void show_free_areas(unsigned int filter)
K(node_page_state(pgdat, NR_ISOLATED_ANON)),
K(node_page_state(pgdat, NR_ISOLATED_FILE)),
K(node_page_state(pgdat, NR_FILE_MAPPED)),
+ K(node_page_state(pgdat, NR_FILE_DIRTY)),
+ K(node_page_state(pgdat, NR_WRITEBACK)),
+ K(node_page_state(pgdat, NR_SHMEM)),
+ K(node_page_state(pgdat, NR_WRITEBACK_TEMP)),
+ K(node_page_state(pgdat, NR_UNSTABLE_NFS)),
!pgdat_reclaimable(pgdat) ? "yes" : "no");
}
@@ -3993,19 +4003,14 @@ void show_free_areas(unsigned int filter)
" present:%lukB"
" managed:%lukB"
" mlocked:%lukB"
- " dirty:%lukB"
- " writeback:%lukB"
- " shmem:%lukB"
" slab_reclaimable:%lukB"
" slab_unreclaimable:%lukB"
" kernel_stack:%lukB"
" pagetables:%lukB"
- " unstable:%lukB"
" bounce:%lukB"
" free_pcp:%lukB"
" local_pcp:%ukB"
" free_cma:%lukB"
- " writeback_tmp:%lukB"
" node_pages_scanned:%lu"
"\n",
zone->name,
@@ -4016,20 +4021,15 @@ void show_free_areas(unsigned int filter)
K(zone->present_pages),
K(zone->managed_pages),
K(zone_page_state(zone, NR_MLOCK)),
- K(zone_page_state(zone, NR_FILE_DIRTY)),
- K(zone_page_state(zone, NR_WRITEBACK)),
- K(zone_page_state(zone, NR_SHMEM)),
K(zone_page_state(zone, NR_SLAB_RECLAIMABLE)),
K(zone_page_state(zone, NR_SLAB_UNRECLAIMABLE)),
zone_page_state(zone, NR_KERNEL_STACK) *
THREAD_SIZE / 1024,
K(zone_page_state(zone, NR_PAGETABLE)),
- K(zone_page_state(zone, NR_UNSTABLE_NFS)),
K(zone_page_state(zone, NR_BOUNCE)),
K(free_pcp),
K(this_cpu_read(zone->pageset->pcp.count)),
K(zone_page_state(zone, NR_FREE_CMA_PAGES)),
- K(zone_page_state(zone, NR_WRITEBACK_TEMP)),
K(node_page_state(zone->zone_pgdat, NR_PAGES_SCANNED)));
printk("lowmem_reserve[]:");
for (i = 0; i < MAX_NR_ZONES; i++)
@@ -4072,7 +4072,7 @@ void show_free_areas(unsigned int filter)
hugetlb_show_meminfo();
- printk("%ld total pagecache pages\n", global_page_state(NR_FILE_PAGES));
+ printk("%ld total pagecache pages\n", global_node_page_state(NR_FILE_PAGES));
show_swap_cache_info();
}
diff --git a/mm/shmem.c b/mm/shmem.c
index b8e8369676be..2d9c0579e398 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -312,8 +312,8 @@ static int shmem_add_to_page_cache(struct page *page,
page);
if (!error) {
mapping->nrpages++;
- __inc_zone_page_state(page, NR_FILE_PAGES);
- __inc_zone_page_state(page, NR_SHMEM);
+ __inc_node_page_state(page, NR_FILE_PAGES);
+ __inc_node_page_state(page, NR_SHMEM);
spin_unlock_irq(&mapping->tree_lock);
} else {
page->mapping = NULL;
@@ -335,8 +335,8 @@ static void shmem_delete_from_page_cache(struct page *page, void *radswap)
error = shmem_radix_tree_replace(mapping, page->index, page, radswap);
page->mapping = NULL;
mapping->nrpages--;
- __dec_zone_page_state(page, NR_FILE_PAGES);
- __dec_zone_page_state(page, NR_SHMEM);
+ __dec_node_page_state(page, NR_FILE_PAGES);
+ __dec_node_page_state(page, NR_SHMEM);
spin_unlock_irq(&mapping->tree_lock);
page_cache_release(page);
BUG_ON(error);
@@ -1098,8 +1098,8 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp,
error = shmem_radix_tree_replace(swap_mapping, swap_index, oldpage,
newpage);
if (!error) {
- __inc_zone_page_state(newpage, NR_FILE_PAGES);
- __dec_zone_page_state(oldpage, NR_FILE_PAGES);
+ __inc_node_page_state(newpage, NR_FILE_PAGES);
+ __dec_node_page_state(oldpage, NR_FILE_PAGES);
}
spin_unlock_irq(&swap_mapping->tree_lock);
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 69cb2464e7dc..ca49c3e41443 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -95,7 +95,7 @@ int __add_to_swap_cache(struct page *page, swp_entry_t entry)
entry.val, page);
if (likely(!error)) {
address_space->nrpages++;
- __inc_zone_page_state(page, NR_FILE_PAGES);
+ __inc_node_page_state(page, NR_FILE_PAGES);
INC_CACHE_INFO(add_total);
}
spin_unlock_irq(&address_space->tree_lock);
@@ -147,7 +147,7 @@ void __delete_from_swap_cache(struct page *page)
set_page_private(page, 0);
ClearPageSwapCache(page);
address_space->nrpages--;
- __dec_zone_page_state(page, NR_FILE_PAGES);
+ __dec_node_page_state(page, NR_FILE_PAGES);
INC_CACHE_INFO(del_total);
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 20497c5c6bb9..faa724af85d5 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3576,11 +3576,11 @@ int sysctl_min_unmapped_ratio = 1;
*/
int sysctl_min_slab_ratio = 5;
-static inline unsigned long zone_unmapped_file_pages(struct zone *zone)
+static inline unsigned long node_unmapped_file_pages(struct pglist_data *pgdat)
{
- unsigned long file_mapped = node_page_state(zone->zone_pgdat, NR_FILE_MAPPED);
- unsigned long file_lru = node_page_state(zone->zone_pgdat, NR_INACTIVE_FILE) +
- node_page_state(zone->zone_pgdat, NR_ACTIVE_FILE);
+ unsigned long file_mapped = node_page_state(pgdat, NR_FILE_MAPPED);
+ unsigned long file_lru = node_page_state(pgdat, NR_INACTIVE_FILE) +
+ node_page_state(pgdat, NR_ACTIVE_FILE);
/*
* It's possible for there to be more file mapped pages than
@@ -3599,17 +3599,17 @@ static unsigned long zone_pagecache_reclaimable(struct zone *zone)
/*
* If RECLAIM_UNMAP is set, then all file pages are considered
* potentially reclaimable. Otherwise, we have to worry about
- * pages like swapcache and zone_unmapped_file_pages() provides
+ * pages like swapcache and node_unmapped_file_pages() provides
* a better estimate
*/
if (zone_reclaim_mode & RECLAIM_UNMAP)
- nr_pagecache_reclaimable = zone_page_state(zone, NR_FILE_PAGES);
+ nr_pagecache_reclaimable = node_page_state(zone->zone_pgdat, NR_FILE_PAGES);
else
- nr_pagecache_reclaimable = zone_unmapped_file_pages(zone);
+ nr_pagecache_reclaimable = node_unmapped_file_pages(zone->zone_pgdat);
/* If we can't clean pages, remove dirty pages from consideration */
if (!(zone_reclaim_mode & RECLAIM_WRITE))
- delta += zone_page_state(zone, NR_FILE_DIRTY);
+ delta += node_page_state(zone->zone_pgdat, NR_FILE_DIRTY);
/* Watch for any possible underflows due to delta */
if (unlikely(delta > nr_pagecache_reclaimable))
diff --git a/mm/vmstat.c b/mm/vmstat.c
index c60df3090ed7..07a25fdbcf28 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -944,19 +944,13 @@ const char * const vmstat_text[] = {
"nr_free_pages",
"nr_alloc_batch",
"nr_mlock",
- "nr_file_pages",
- "nr_dirty",
- "nr_writeback",
"nr_slab_reclaimable",
"nr_slab_unreclaimable",
"nr_page_table_pages",
"nr_kernel_stack",
- "nr_unstable",
"nr_bounce",
"nr_vmscan_write",
"nr_vmscan_immediate_reclaim",
- "nr_writeback_temp",
- "nr_shmem",
"nr_dirtied",
"nr_written",
@@ -985,6 +979,12 @@ const char * const vmstat_text[] = {
"workingset_nodereclaim",
"nr_anon_pages",
"nr_mapped",
+ "nr_file_pages",
+ "nr_dirty",
+ "nr_writeback",
+ "nr_writeback_temp",
+ "nr_shmem",
+ "nr_unstable",
/* enum writeback_stat_item counters */
"nr_dirty_threshold",
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:20 +0100 |
| Subject | [PATCH 18/27] mm: Rename NR_ANON_PAGES to NR_ANON_MAPPED |
| Message-ID | <r5mUO-7eb-17@gated-at.bofh.it> |
| In reply to | #1340741 |
NR_FILE_PAGES is the number of file pages.
NR_FILE_MAPPED is the number of mapped file pages.
NR_ANON_PAGES is the number of mapped anon pages.
This is unhelpful naming as it's easy to confuse NR_FILE_MAPPED and NR_ANON_PAGES for
mapped pages. This patch renames NR_ANON_PAGES so we have
NR_FILE_PAGES is the number of file pages.
NR_FILE_MAPPED is the number of mapped file pages.
NR_ANON_MAPPED is the number of mapped anon pages.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
drivers/base/node.c | 2 +-
fs/proc/meminfo.c | 2 +-
include/linux/mmzone.h | 2 +-
mm/migrate.c | 2 +-
mm/rmap.c | 8 ++++----
5 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 66aed68a0fdc..897b6bcb36be 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -120,7 +120,7 @@ static ssize_t node_read_meminfo(struct device *dev,
nid, K(sum_zone_node_page_state(nid, NR_WRITEBACK)),
nid, K(sum_zone_node_page_state(nid, NR_FILE_PAGES)),
nid, K(node_page_state(pgdat, NR_FILE_MAPPED)),
- nid, K(node_page_state(pgdat, NR_ANON_PAGES)),
+ nid, K(node_page_state(pgdat, NR_ANON_MAPPED)),
nid, K(i.sharedram),
nid, sum_zone_node_page_state(nid, NR_KERNEL_STACK) *
THREAD_SIZE / 1024,
diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
index 2c0dfee0f29f..b3583d75b211 100644
--- a/fs/proc/meminfo.c
+++ b/fs/proc/meminfo.c
@@ -167,7 +167,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
K(i.freeswap),
K(global_page_state(NR_FILE_DIRTY)),
K(global_page_state(NR_WRITEBACK)),
- K(global_node_page_state(NR_ANON_PAGES)),
+ K(global_node_page_state(NR_ANON_MAPPED)),
K(global_node_page_state(NR_FILE_MAPPED)),
K(i.sharedram),
K(global_page_state(NR_SLAB_RECLAIMABLE) +
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index b901dbfb6cc9..05a273171ab0 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -160,7 +160,7 @@ enum node_stat_item {
WORKINGSET_REFAULT,
WORKINGSET_ACTIVATE,
WORKINGSET_NODERECLAIM,
- NR_ANON_PAGES, /* Mapped anonymous pages */
+ NR_ANON_MAPPED, /* Mapped anonymous pages */
NR_FILE_MAPPED, /* pagecache pages mapped into pagetables.
only modified from process context */
NR_VM_NODE_STAT_ITEMS
diff --git a/mm/migrate.c b/mm/migrate.c
index aea3e350f4fc..0492a035df80 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -409,7 +409,7 @@ int migrate_page_move_mapping(struct address_space *mapping,
* new page and drop references to the old page.
*
* Note that anonymous pages are accounted for
- * via NR_FILE_PAGES and NR_ANON_PAGES if they
+ * via NR_FILE_PAGES and NR_ANON_MAPPED if they
* are mapped to swap space.
*/
if (newzone != oldzone) {
diff --git a/mm/rmap.c b/mm/rmap.c
index 8f405fd9cfe1..fd4dd2cdc390 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1232,7 +1232,7 @@ void do_page_add_anon_rmap(struct page *page,
NR_ANON_TRANSPARENT_HUGEPAGES);
}
__mod_node_page_state(page_zone(page)->zone_pgdat,
- NR_ANON_PAGES, nr);
+ NR_ANON_MAPPED, nr);
}
if (unlikely(PageKsm(page)))
return;
@@ -1276,7 +1276,7 @@ void page_add_new_anon_rmap(struct page *page,
/* increment count (starts at -1) */
atomic_set(&page->_mapcount, 0);
}
- __mod_node_page_state(page_zone(page)->zone_pgdat, NR_ANON_PAGES, nr);
+ __mod_node_page_state(page_zone(page)->zone_pgdat, NR_ANON_MAPPED, nr);
__page_set_anon_rmap(page, vma, address, 1);
}
@@ -1358,7 +1358,7 @@ static void page_remove_anon_compound_rmap(struct page *page)
clear_page_mlock(page);
if (nr) {
- __mod_node_page_state(page_zone(page)->zone_pgdat, NR_ANON_PAGES, -nr);
+ __mod_node_page_state(page_zone(page)->zone_pgdat, NR_ANON_MAPPED, -nr);
deferred_split_huge_page(page);
}
}
@@ -1390,7 +1390,7 @@ void page_remove_rmap(struct page *page, bool compound)
* these counters are not modified in interrupt context, and
* pte lock(a spinlock) is held, which implies preemption disabled.
*/
- __dec_node_page_state(page, NR_ANON_PAGES);
+ __dec_node_page_state(page, NR_ANON_MAPPED);
if (unlikely(PageMlocked(page)))
clear_page_mlock(page);
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-02-23 16:20 +0100 |
| Subject | [PATCH 01/27] mm, page_alloc: Use ac->classzone_idx instead of zone_idx(preferred_zone) |
| Message-ID | <r5mUO-7eb-19@gated-at.bofh.it> |
| In reply to | #1340741 |
ac->classzone_idx is determined by the index of the preferred zone and cached to avoid repeated calculations. wake_all_kswapds() should use it instead of using zone_idx() within a loop. Signed-off-by: Mel Gorman <mgorman@techsingularity.net> --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c849a202b6f1..46ecbfa5b228 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3022,7 +3022,7 @@ static void wake_all_kswapds(unsigned int order, const struct alloc_context *ac) for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->high_zoneidx, ac->nodemask) - wakeup_kswapd(zone, order, zone_idx(ac->preferred_zone)); + wakeup_kswapd(zone, order, ac->classzone_idx); } static inline int -- 2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Johannes Weiner <hannes@cmpxchg.org> |
|---|---|
| Date | 2016-02-23 19:10 +0100 |
| Subject | Re: [PATCH 01/27] mm, page_alloc: Use ac->classzone_idx instead of zone_idx(preferred_zone) |
| Message-ID | <r5pzj-Cq-1@gated-at.bofh.it> |
| In reply to | #1340756 |
On Tue, Feb 23, 2016 at 03:04:24PM +0000, Mel Gorman wrote: > ac->classzone_idx is determined by the index of the preferred zone and cached > to avoid repeated calculations. wake_all_kswapds() should use it instead of > using zone_idx() within a loop. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Johannes Weiner <hannes@cmpxchg.org>
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-03-03 11:40 +0100 |
| Subject | Re: [PATCH 01/27] mm, page_alloc: Use ac->classzone_idx instead of zone_idx(preferred_zone) |
| Message-ID | <r8yPM-5Gt-19@gated-at.bofh.it> |
| In reply to | #1340756 |
On 02/23/2016 04:04 PM, Mel Gorman wrote: > ac->classzone_idx is determined by the index of the preferred zone and cached > to avoid repeated calculations. wake_all_kswapds() should use it instead of > using zone_idx() within a loop. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Vlastimil Babka <vbabka@suse.cz>
[toc] | [prev] | [next] | [standalone]
Page 1 of 3 [1] 2 3 Next page →
Back to top | Article view | linux.kernel
csiph-web