Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1435600 > unrolled thread
| Started by | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| First post | 2016-07-01 22:20 +0200 |
| Last post | 2016-07-05 12:20 +0200 |
| Articles | 12 — 2 participants |
Back to article view | Back to linux.kernel
[PATCH 00/31] Move LRU page reclaim from zones to nodes v8 Mel Gorman <mgorman@techsingularity.net> - 2016-07-01 22:20 +0200
[PATCH 01/31] mm, vmstat: add infrastructure for per-node vmstats Mel Gorman <mgorman@techsingularity.net> - 2016-07-01 22:20 +0200
Re: [PATCH 01/31] mm, vmstat: add infrastructure for per-node vmstats Minchan Kim <minchan@kernel.org> - 2016-07-05 01:50 +0200
Re: [PATCH 01/31] mm, vmstat: add infrastructure for per-node vmstats Mel Gorman <mgorman@techsingularity.net> - 2016-07-05 10:20 +0200
Re: [PATCH 01/31] mm, vmstat: add infrastructure for per-node vmstats Minchan Kim <minchan@kernel.org> - 2016-07-06 02:20 +0200
Re: [PATCH 00/31] Move LRU page reclaim from zones to nodes v8 Minchan Kim <minchan@kernel.org> - 2016-07-04 03:40 +0200
Re: [PATCH 00/31] Move LRU page reclaim from zones to nodes v8 Mel Gorman <mgorman@techsingularity.net> - 2016-07-04 12:00 +0200
Re: [PATCH 00/31] Move LRU page reclaim from zones to nodes v8 Minchan Kim <minchan@kernel.org> - 2016-07-06 04:00 +0200
Re: [PATCH 00/31] Move LRU page reclaim from zones to nodes v8 Minchan Kim <minchan@kernel.org> - 2016-07-04 10:10 +0200
Re: [PATCH 00/31] Move LRU page reclaim from zones to nodes v8 Mel Gorman <mgorman@techsingularity.net> - 2016-07-04 06:40 +0200
Re: [PATCH 03/31] mm, vmscan: move LRU lists to node Minchan Kim <minchan@kernel.org> - 2016-07-05 03:30 +0200
Re: [PATCH 03/31] mm, vmscan: move LRU lists to node Mel Gorman <mgorman@techsingularity.net> - 2016-07-05 12:20 +0200
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-07-01 22:20 +0200 |
| Subject | [PATCH 00/31] Move LRU page reclaim from zones to nodes v8 |
| Message-ID | <rQcVb-2Ax-3@gated-at.bofh.it> |
(Sorry for the resend, I accidentally sent the branch that still had the
Signed-off-by's from mmotm still applied which is incorrect.)
Previous releases double accounted LRU stats on the zone and the node
because it was required by should_reclaim_retry. The last patch in the
series removes the double accounting. It's not integrated with the series
as reviewers may not like the solution. If not, it can be safely dropped
without a major impact to the results.
Changelog since v7
o Rebase onto current mmots
o Avoid double accounting of stats in node and zone
o Kswapd will avoid more reclaim if an eligible zone is available
o Remove some duplications of sc->reclaim_idx and classzone_idx
o Print per-node stats in zoneinfo
Changelog since v6
o Correct reclaim_idx when direct reclaiming for memcg
o Also account LRU pages per zone for compaction/reclaim
o Add page_pgdat helper with more efficient lookup
o Init pgdat LRU lock only once
o Slight optimisation to wake_all_kswapds
o Always wake kcompactd when kswapd is going to sleep
o Rebase to mmotm as of June 15th, 2016
Changelog since v5
o Rebase and adjust to changes
Changelog since v4
o Rebase on top of v3 of page allocator optimisation series
Changelog since v3
o Rebase on top of the page allocator optimisation series
o Remove RFC tag
This is the latest version of a series that moves LRUs from the zones to
the node that is based upon 4.7-rc4 with Andrew's tree applied. While this
is a current rebase, the test results were based on mmotm as of June 23rd.
Conceptually, this series is simple but there are a lot of details. Some
of the broad motivations for this are;
1. The residency of a page partially depends on what zone the page was
allocated from. This is partially combatted by the fair zone allocation
policy but that is a partial solution that introduces overhead in the
page allocator paths.
2. Currently, reclaim on node 0 behaves slightly different to node 1. For
example, direct reclaim scans in zonelist order and reclaims even if
the zone is over the high watermark regardless of the age of pages
in that LRU. Kswapd on the other hand starts reclaim on the highest
unbalanced zone. A difference in distribution of file/anon pages due
to when they were allocated results can result in a difference in
again. While the fair zone allocation policy mitigates some of the
problems here, the page reclaim results on a multi-zone node will
always be different to a single-zone node.
it was scheduled on as a result.
3. kswapd and the page allocator scan zones in the opposite order to
avoid interfering with each other but it's sensitive to timing. This
mitigates the page allocator using pages that were allocated very recently
in the ideal case but it's sensitive to timing. When kswapd is allocating
from lower zones then it's great but during the rebalancing of the highest
zone, the page allocator and kswapd interfere with each other. It's worse
if the highest zone is small and difficult to balance.
4. slab shrinkers are node-based which makes it harder to identify the exact
relationship between slab reclaim and LRU reclaim.
The reason we have zone-based reclaim is that we used to have
large highmem zones in common configurations and it was necessary
to quickly find ZONE_NORMAL pages for reclaim. Today, this is much
less of a concern as machines with lots of memory will (or should) use
64-bit kernels. Combinations of 32-bit hardware and 64-bit hardware are
rare. Machines that do use highmem should have relatively low highmem:lowmem
ratios than we worried about in the past.
Conceptually, moving to node LRUs should be easier to understand. The
page allocator plays fewer tricks to game reclaim and reclaim behaves
similarly on all nodes.
The series has been tested on a 16 core UMA machine and a 2-socket 48
core NUMA machine. The UMA results are presented in most cases as the NUMA
machine behaved similarly.
pagealloc
---------
This is a microbenchmark that shows the benefit of removing the fair zone
allocation policy. It was tested uip to order-4 but only orders 0 and 1 are
shown as the other orders were comparable.
4.7.0-rc4 4.7.0-rc4
mmotm-20160623 nodelru-v8
Min total-odr0-1 490.00 ( 0.00%) 463.00 ( 5.51%)
Min total-odr0-2 349.00 ( 0.00%) 325.00 ( 6.88%)
Min total-odr0-4 288.00 ( 0.00%) 272.00 ( 5.56%)
Min total-odr0-8 250.00 ( 0.00%) 235.00 ( 6.00%)
Min total-odr0-16 234.00 ( 0.00%) 222.00 ( 5.13%)
Min total-odr0-32 223.00 ( 0.00%) 205.00 ( 8.07%)
Min total-odr0-64 217.00 ( 0.00%) 202.00 ( 6.91%)
Min total-odr0-128 214.00 ( 0.00%) 207.00 ( 3.27%)
Min total-odr0-256 242.00 ( 0.00%) 242.00 ( 0.00%)
Min total-odr0-512 272.00 ( 0.00%) 265.00 ( 2.57%)
Min total-odr0-1024 290.00 ( 0.00%) 283.00 ( 2.41%)
Min total-odr0-2048 302.00 ( 0.00%) 296.00 ( 1.99%)
Min total-odr0-4096 311.00 ( 0.00%) 306.00 ( 1.61%)
Min total-odr0-8192 314.00 ( 0.00%) 309.00 ( 1.59%)
Min total-odr0-16384 315.00 ( 0.00%) 309.00 ( 1.90%)
Min total-odr1-1 741.00 ( 0.00%) 716.00 ( 3.37%)
Min total-odr1-2 565.00 ( 0.00%) 524.00 ( 7.26%)
Min total-odr1-4 457.00 ( 0.00%) 427.00 ( 6.56%)
Min total-odr1-8 408.00 ( 0.00%) 371.00 ( 9.07%)
Min total-odr1-16 383.00 ( 0.00%) 344.00 ( 10.18%)
Min total-odr1-32 378.00 ( 0.00%) 334.00 ( 11.64%)
Min total-odr1-64 383.00 ( 0.00%) 334.00 ( 12.79%)
Min total-odr1-128 376.00 ( 0.00%) 342.00 ( 9.04%)
Min total-odr1-256 381.00 ( 0.00%) 343.00 ( 9.97%)
Min total-odr1-512 388.00 ( 0.00%) 349.00 ( 10.05%)
Min total-odr1-1024 386.00 ( 0.00%) 356.00 ( 7.77%)
Min total-odr1-2048 389.00 ( 0.00%) 362.00 ( 6.94%)
Min total-odr1-4096 389.00 ( 0.00%) 362.00 ( 6.94%)
Min total-odr1-8192 389.00 ( 0.00%) 362.00 ( 6.94%)
This shows a steady improvement throughout. The primary benefit is from
reduced system CPU usage which is obvious from the overall times;
4.7.0-rc4 4.7.0-rc4
mmotm-20160623nodelru-v8
User 191.39 191.61
System 2651.24 2504.48
Elapsed 2904.40 2757.01
The vmstats also showed that the fair zone allocation policy was definitely
removed as can be seen here;
4.7.0-rc3 4.7.0-rc3
mmotm-20160623 nodelru-v8
DMA32 allocs 28794771816 0
Normal allocs 48432582848 77227356392
Movable allocs 0 0
tiobench on ext4
----------------
tiobench is a benchmark that artifically benefits if old pages remain resident
while new pages get reclaimed. The fair zone allocation policy mitigates this
problem so pages age fairly. While the benchmark has problems, it is important
that tiobench performance remains constant as it implies that page aging
problems that the fair zone allocation policy fixes are not re-introduced.
4.7.0-rc4 4.7.0-rc4
mmotm-20160623 nodelru-v8
Min PotentialReadSpeed 89.65 ( 0.00%) 90.34 ( 0.77%)
Min SeqRead-MB/sec-1 82.68 ( 0.00%) 83.13 ( 0.54%)
Min SeqRead-MB/sec-2 72.76 ( 0.00%) 72.15 ( -0.84%)
Min SeqRead-MB/sec-4 75.13 ( 0.00%) 74.23 ( -1.20%)
Min SeqRead-MB/sec-8 64.91 ( 0.00%) 65.25 ( 0.52%)
Min SeqRead-MB/sec-16 62.24 ( 0.00%) 62.76 ( 0.84%)
Min RandRead-MB/sec-1 0.88 ( 0.00%) 0.95 ( 7.95%)
Min RandRead-MB/sec-2 0.95 ( 0.00%) 0.94 ( -1.05%)
Min RandRead-MB/sec-4 1.43 ( 0.00%) 1.46 ( 2.10%)
Min RandRead-MB/sec-8 1.61 ( 0.00%) 1.58 ( -1.86%)
Min RandRead-MB/sec-16 1.80 ( 0.00%) 1.93 ( 7.22%)
Min SeqWrite-MB/sec-1 76.41 ( 0.00%) 78.84 ( 3.18%)
Min SeqWrite-MB/sec-2 74.11 ( 0.00%) 73.35 ( -1.03%)
Min SeqWrite-MB/sec-4 80.05 ( 0.00%) 78.69 ( -1.70%)
Min SeqWrite-MB/sec-8 72.88 ( 0.00%) 71.38 ( -2.06%)
Min SeqWrite-MB/sec-16 75.91 ( 0.00%) 75.81 ( -0.13%)
Min RandWrite-MB/sec-1 1.18 ( 0.00%) 1.12 ( -5.08%)
Min RandWrite-MB/sec-2 1.02 ( 0.00%) 1.02 ( 0.00%)
Min RandWrite-MB/sec-4 1.05 ( 0.00%) 0.99 ( -5.71%)
Min RandWrite-MB/sec-8 0.89 ( 0.00%) 0.92 ( 3.37%)
Min RandWrite-MB/sec-16 0.92 ( 0.00%) 0.89 ( -3.26%)
This shows that the series has little or not impact on tiobench which is
desirable. It indicates that the fair zone allocation policy was removed
in a manner that didn't reintroduce one class of page aging bug. There
were only minor differences in overall reclaim activity
4.7.0-rc4 4.7.0-rc4
mmotm-20160623nodelru-v8
Minor Faults 645838 644036
Major Faults 573 593
Swap Ins 0 0
Swap Outs 0 0
Allocation stalls 24 0
DMA allocs 0 0
DMA32 allocs 46041453 44154171
Normal allocs 78053072 79865782
Movable allocs 0 0
Direct pages scanned 10969 54504
Kswapd pages scanned 93375144 93250583
Kswapd pages reclaimed 93372243 93247714
Direct pages reclaimed 10969 54504
Kswapd efficiency 99% 99%
Kswapd velocity 13741.015 13711.950
Direct efficiency 100% 100%
Direct velocity 1.614 8.014
Percentage direct scans 0% 0%
Zone normal velocity 8641.875 13719.964
Zone dma32 velocity 5100.754 0.000
Zone dma velocity 0.000 0.000
Page writes by reclaim 0.000 0.000
Page writes file 0 0
Page writes anon 0 0
Page reclaim immediate 37 54
kswapd activity was roughly comparable. There were differences in direct
reclaim activity but negligible in the context of the overall workload
(velocity of 8 pages per second with the patches applied, 1.6 pages per
second in the baseline kernel).
pgbench read-only large configuration on ext4
---------------------------------------------
pgbench is a database benchmark that can be sensitive to page reclaim
decisions. This also checks if removing the fair zone allocation policy
is safe
pgbench Transactions
4.7.0-rc4 4.7.0-rc4
mmotm-20160623 nodelru-v8
Hmean 1 188.26 ( 0.00%) 189.78 ( 0.81%)
Hmean 5 330.66 ( 0.00%) 328.69 ( -0.59%)
Hmean 12 370.32 ( 0.00%) 380.72 ( 2.81%)
Hmean 21 368.89 ( 0.00%) 369.00 ( 0.03%)
Hmean 30 382.14 ( 0.00%) 360.89 ( -5.56%)
Hmean 32 428.87 ( 0.00%) 432.96 ( 0.95%)
Negligible differences again. As with tiobench, overall reclaim activity
was comparable.
bonnie++ on ext4
----------------
No interesting performance difference, negligible differences on reclaim
stats.
paralleldd on ext4
------------------
This workload uses varying numbers of dd instances to read large amounts of
data from disk.
4.7.0-rc3 4.7.0-rc3
mmotm-20160615 nodelru-v7r17
Amean Elapsd-1 181.57 ( 0.00%) 179.63 ( 1.07%)
Amean Elapsd-3 188.29 ( 0.00%) 183.68 ( 2.45%)
Amean Elapsd-5 188.02 ( 0.00%) 181.73 ( 3.35%)
Amean Elapsd-7 186.07 ( 0.00%) 184.11 ( 1.05%)
Amean Elapsd-12 188.16 ( 0.00%) 183.51 ( 2.47%)
Amean Elapsd-16 189.03 ( 0.00%) 181.27 ( 4.10%)
4.7.0-rc3 4.7.0-rc3
mmotm-20160615nodelru-v7r17
User 1439.23 1433.37
System 8332.31 8216.01
Elapsed 3619.80 3532.69
There is a slight gain in performance, some of which is from the reduced system
CPU usage. There areminor differences in reclaim activity but nothing significant
4.7.0-rc3 4.7.0-rc3
mmotm-20160615nodelru-v7r17
Minor Faults 362486 358215
Major Faults 1143 1113
Swap Ins 26 0
Swap Outs 2920 482
DMA allocs 0 0
DMA32 allocs 31568814 28598887
Normal allocs 46539922 49514444
Movable allocs 0 0
Allocation stalls 0 0
Direct pages scanned 0 0
Kswapd pages scanned 40886878 40849710
Kswapd pages reclaimed 40869923 40835207
Direct pages reclaimed 0 0
Kswapd efficiency 99% 99%
Kswapd velocity 11295.342 11563.344
Direct efficiency 100% 100%
Direct velocity 0.000 0.000
Slabs scanned 131673 126099
Direct inode steals 57 60
Kswapd inode steals 762 18
It basically shows that kswapd was active at roughly the same rate in
both kernels. There was also comparable slab scanning activity and direct
reclaim was avoided in both cases. There appears to be a large difference
in numbers of inodes reclaimed but the workload has few active inodes and
is likely a timing artifact. It's interesting to note that the node-lru
did not swap in any pages but given the low swap activity, it's unlikely
to be significant.
stutter
-------
stutter simulates a simple workload. One part uses a lot of anonymous
memory, a second measures mmap latency and a third copies a large file.
The primary metric is checking for mmap latency.
stutter
4.7.0-rc4 4.7.0-rc4
mmotm-20160623 nodelru-v8
Min mmap 16.6283 ( 0.00%) 16.1394 ( 2.94%)
1st-qrtle mmap 54.7570 ( 0.00%) 55.2975 ( -0.99%)
2nd-qrtle mmap 57.3163 ( 0.00%) 57.5230 ( -0.36%)
3rd-qrtle mmap 58.9976 ( 0.00%) 58.0537 ( 1.60%)
Max-90% mmap 59.7433 ( 0.00%) 58.3910 ( 2.26%)
Max-93% mmap 60.1298 ( 0.00%) 58.4801 ( 2.74%)
Max-95% mmap 73.4112 ( 0.00%) 58.5537 ( 20.24%)
Max-99% mmap 92.8542 ( 0.00%) 58.9673 ( 36.49%)
Max mmap 1440.6569 ( 0.00%) 137.6875 ( 90.44%)
Mean mmap 59.3493 ( 0.00%) 55.5153 ( 6.46%)
Best99%Mean mmap 57.2121 ( 0.00%) 55.4194 ( 3.13%)
Best95%Mean mmap 55.9113 ( 0.00%) 55.2813 ( 1.13%)
Best90%Mean mmap 55.6199 ( 0.00%) 55.1044 ( 0.93%)
Best50%Mean mmap 53.2183 ( 0.00%) 52.8330 ( 0.72%)
Best10%Mean mmap 45.9842 ( 0.00%) 42.3740 ( 7.85%)
Best5%Mean mmap 43.2256 ( 0.00%) 38.8660 ( 10.09%)
Best1%Mean mmap 32.9388 ( 0.00%) 27.7577 ( 15.73%)
This shows a number of improvements with the worst-case outlier greatly
improved.
Some of the vmstats are interesting
4.7.0-rc4 4.7.0-rc4
mmotm-20160623nodelru-v8
Swap Ins 163 239
Swap Outs 0 0
Allocation stalls 2603 0
DMA allocs 0 0
DMA32 allocs 618719206 1303037965
Normal allocs 891235743 229914091
Movable allocs 0 0
Direct pages scanned 216787 3173
Kswapd pages scanned 50719775 41732250
Kswapd pages reclaimed 41541765 41731168
Direct pages reclaimed 209159 3173
Kswapd efficiency 81% 99%
Kswapd velocity 16859.554 14231.043
Direct efficiency 96% 100%
Direct velocity 72.061 1.082
Percentage direct scans 0% 0%
Zone normal velocity 8431.777 14232.125
Zone dma32 velocity 8499.838 0.000
Zone dma velocity 0.000 0.000
Page writes by reclaim 6215049.000 0.000
Page writes file 6215049 0
Page writes anon 0 0
Page reclaim immediate 70673 143
Sector Reads 81940800 81489388
Sector Writes 100158984 99161860
Page rescued immediate 0 0
Slabs scanned 1366954 21196
While this is not guaranteed in all cases, this particular test showed
a large reduction in direct reclaim activity. It's also worth noting
that no page writes were issued from reclaim context.
This series is not without its hazards. There are at least three areas
that I'm concerned with even though I could not reproduce any problems in
that area.
1. Reclaim/compaction is going to be affected because the amount of reclaim is
no longer targetted at a specific zone. Compaction works on a per-zone basis
so there is no guarantee that reclaiming a few THP's worth page pages will
have a positive impact on compaction success rates.
2. The Slab/LRU reclaim ratio is affected because the frequency the shrinkers
are called is now different. This may or may not be a problem but if it
is, it'll be because shrinkers are not called enough and some balancing
is required.
3. The anon/file reclaim ratio may be affected. Pages about to be dirtied are
distributed between zones and the fair zone allocation policy used to do
something very similar for anon. The distribution is now different but not
necessarily in any way that matters but it's still worth bearing in mind.
Documentation/cgroup-v1/memcg_test.txt | 4 +-
Documentation/cgroup-v1/memory.txt | 4 +-
arch/s390/appldata/appldata_mem.c | 2 +-
arch/tile/mm/pgtable.c | 18 +-
drivers/base/node.c | 77 ++-
drivers/staging/android/lowmemorykiller.c | 12 +-
drivers/staging/lustre/lustre/osc/osc_cache.c | 6 +-
fs/fs-writeback.c | 4 +-
fs/fuse/file.c | 8 +-
fs/nfs/internal.h | 2 +-
fs/nfs/write.c | 2 +-
fs/proc/meminfo.c | 20 +-
include/linux/backing-dev.h | 2 +-
include/linux/memcontrol.h | 61 +-
include/linux/mm.h | 5 +
include/linux/mm_inline.h | 35 +-
include/linux/mm_types.h | 2 +-
include/linux/mmzone.h | 155 +++--
include/linux/swap.h | 24 +-
include/linux/topology.h | 2 +-
include/linux/vm_event_item.h | 14 +-
include/linux/vmstat.h | 111 +++-
include/linux/writeback.h | 2 +-
include/trace/events/vmscan.h | 63 +-
include/trace/events/writeback.h | 10 +-
kernel/power/snapshot.c | 10 +-
kernel/sysctl.c | 4 +-
mm/backing-dev.c | 15 +-
mm/compaction.c | 50 +-
mm/filemap.c | 16 +-
mm/huge_memory.c | 12 +-
mm/internal.h | 11 +-
mm/khugepaged.c | 14 +-
mm/memcontrol.c | 215 +++----
mm/memory-failure.c | 4 +-
mm/memory_hotplug.c | 7 +-
mm/mempolicy.c | 2 +-
mm/migrate.c | 35 +-
mm/mlock.c | 12 +-
mm/page-writeback.c | 123 ++--
mm/page_alloc.c | 371 +++++------
mm/page_idle.c | 4 +-
mm/rmap.c | 26 +-
mm/shmem.c | 14 +-
mm/swap.c | 64 +-
mm/swap_state.c | 4 +-
mm/util.c | 4 +-
mm/vmscan.c | 879 +++++++++++++-------------
mm/vmstat.c | 398 +++++++++---
mm/workingset.c | 54 +-
50 files changed, 1674 insertions(+), 1319 deletions(-)
--
2.6.4
Mel Gorman (31):
mm, vmstat: add infrastructure for per-node vmstats
mm, vmscan: move lru_lock to the node
mm, vmscan: move LRU lists to node
mm, vmscan: begin reclaiming pages on a per-node basis
mm, vmscan: have kswapd only scan based on the highest requested zone
mm, vmscan: make kswapd reclaim in terms of nodes
mm, vmscan: remove balance gap
mm, vmscan: simplify the logic deciding whether kswapd sleeps
mm, vmscan: by default have direct reclaim only shrink once per node
mm, vmscan: remove duplicate logic clearing node congestion and dirty
state
mm: vmscan: do not reclaim from kswapd if there is any eligible zone
mm, vmscan: make shrink_node decisions more node-centric
mm, memcg: move memcg limit enforcement from zones to nodes
mm, workingset: make working set detection node-aware
mm, page_alloc: consider dirtyable memory in terms of nodes
mm: move page mapped accounting to the node
mm: rename NR_ANON_PAGES to NR_ANON_MAPPED
mm: move most file-based accounting to the node
mm: move vmscan writes and file write accounting to the node
mm, vmscan: only wakeup kswapd once per node for the requested
classzone
mm, page_alloc: Wake kswapd based on the highest eligible zone
mm: convert zone_reclaim to node_reclaim
mm, vmscan: Avoid passing in classzone_idx unnecessarily to
shrink_node
mm, vmscan: Avoid passing in classzone_idx unnecessarily to
compaction_ready
mm, vmscan: add classzone information to tracepoints
mm, page_alloc: remove fair zone allocation policy
mm: page_alloc: cache the last node whose dirty limit is reached
mm: vmstat: replace __count_zone_vm_events with a zone id equivalent
mm: vmstat: account per-zone stalls and pages skipped during reclaim
mm, vmstat: print node-based stats in zoneinfo file
mm, vmstat: Remove zone and node double accounting by approximating
retries
Documentation/cgroup-v1/memcg_test.txt | 4 +-
Documentation/cgroup-v1/memory.txt | 4 +-
arch/s390/appldata/appldata_mem.c | 2 +-
arch/tile/mm/pgtable.c | 18 +-
drivers/base/node.c | 77 ++-
drivers/staging/android/lowmemorykiller.c | 12 +-
drivers/staging/lustre/lustre/osc/osc_cache.c | 6 +-
fs/fs-writeback.c | 4 +-
fs/fuse/file.c | 8 +-
fs/nfs/internal.h | 2 +-
fs/nfs/write.c | 2 +-
fs/proc/meminfo.c | 20 +-
include/linux/backing-dev.h | 2 +-
include/linux/memcontrol.h | 61 +-
include/linux/mm.h | 5 +
include/linux/mm_inline.h | 35 +-
include/linux/mm_types.h | 2 +-
include/linux/mmzone.h | 155 +++--
include/linux/swap.h | 24 +-
include/linux/topology.h | 2 +-
include/linux/vm_event_item.h | 14 +-
include/linux/vmstat.h | 111 +++-
include/linux/writeback.h | 2 +-
include/trace/events/vmscan.h | 63 +-
include/trace/events/writeback.h | 10 +-
kernel/power/snapshot.c | 10 +-
kernel/sysctl.c | 4 +-
mm/backing-dev.c | 15 +-
mm/compaction.c | 50 +-
mm/filemap.c | 16 +-
mm/huge_memory.c | 12 +-
mm/internal.h | 11 +-
mm/khugepaged.c | 14 +-
mm/memcontrol.c | 215 +++----
mm/memory-failure.c | 4 +-
mm/memory_hotplug.c | 7 +-
mm/mempolicy.c | 2 +-
mm/migrate.c | 35 +-
mm/mlock.c | 12 +-
mm/page-writeback.c | 123 ++--
mm/page_alloc.c | 371 +++++------
mm/page_idle.c | 4 +-
mm/rmap.c | 26 +-
mm/shmem.c | 14 +-
mm/swap.c | 64 +-
mm/swap_state.c | 4 +-
mm/util.c | 4 +-
mm/vmscan.c | 879 +++++++++++++-------------
mm/vmstat.c | 398 +++++++++---
mm/workingset.c | 54 +-
50 files changed, 1674 insertions(+), 1319 deletions(-)
--
2.6.4
[toc] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-07-01 22:20 +0200 |
| Subject | [PATCH 01/31] mm, vmstat: add infrastructure for per-node vmstats |
| Message-ID | <rQd4S-2DT-5@gated-at.bofh.it> |
| In reply to | #1435600 |
VM statistic counters for reclaim decisions are zone-based. If the kernel
is to reclaim on a per-node basis then we need to track per-node
statistics but there is no infrastructure for that. The most notable
change is that the old node_page_state is renamed to
sum_zone_node_page_state. The new node_page_state takes a pglist_data and
uses per-node stats but none exist yet. There is some renaming such as
vm_stat to vm_zone_stat and the addition of vm_node_stat and the renaming
of mod_state to mod_zone_state. Otherwise, this is mostly a mechanical
patch with no functional change. There is a lot of similarity between the
node and zone helpers which is unfortunate but there was no obvious way of
reusing the code and maintaining type safety.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
---
drivers/base/node.c | 76 +++++++------
include/linux/mm.h | 5 +
include/linux/mmzone.h | 13 +++
include/linux/vmstat.h | 92 +++++++++++++---
mm/page_alloc.c | 10 +-
mm/vmstat.c | 282 +++++++++++++++++++++++++++++++++++++++++++++----
mm/workingset.c | 9 +-
7 files changed, 411 insertions(+), 76 deletions(-)
diff --git a/drivers/base/node.c b/drivers/base/node.c
index ed0ef0f69489..92d8e090c5b3 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -74,16 +74,16 @@ static ssize_t node_read_meminfo(struct device *dev,
nid, K(i.totalram),
nid, K(i.freeram),
nid, K(i.totalram - i.freeram),
- nid, K(node_page_state(nid, NR_ACTIVE_ANON) +
- node_page_state(nid, NR_ACTIVE_FILE)),
- nid, K(node_page_state(nid, NR_INACTIVE_ANON) +
- node_page_state(nid, NR_INACTIVE_FILE)),
- nid, K(node_page_state(nid, NR_ACTIVE_ANON)),
- nid, K(node_page_state(nid, NR_INACTIVE_ANON)),
- nid, K(node_page_state(nid, NR_ACTIVE_FILE)),
- nid, K(node_page_state(nid, NR_INACTIVE_FILE)),
- nid, K(node_page_state(nid, NR_UNEVICTABLE)),
- nid, K(node_page_state(nid, NR_MLOCK)));
+ nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_ANON) +
+ sum_zone_node_page_state(nid, NR_ACTIVE_FILE)),
+ nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_ANON) +
+ sum_zone_node_page_state(nid, NR_INACTIVE_FILE)),
+ nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_ANON)),
+ nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_ANON)),
+ nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_FILE)),
+ nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_FILE)),
+ nid, K(sum_zone_node_page_state(nid, NR_UNEVICTABLE)),
+ nid, K(sum_zone_node_page_state(nid, NR_MLOCK)));
#ifdef CONFIG_HIGHMEM
n += sprintf(buf + n,
@@ -117,31 +117,31 @@ static ssize_t node_read_meminfo(struct device *dev,
"Node %d ShmemPmdMapped: %8lu kB\n"
#endif
,
- nid, K(node_page_state(nid, NR_FILE_DIRTY)),
- nid, K(node_page_state(nid, NR_WRITEBACK)),
- nid, K(node_page_state(nid, NR_FILE_PAGES)),
- nid, K(node_page_state(nid, NR_FILE_MAPPED)),
- nid, K(node_page_state(nid, NR_ANON_PAGES)),
+ nid, K(sum_zone_node_page_state(nid, NR_FILE_DIRTY)),
+ nid, K(sum_zone_node_page_state(nid, NR_WRITEBACK)),
+ nid, K(sum_zone_node_page_state(nid, NR_FILE_PAGES)),
+ nid, K(sum_zone_node_page_state(nid, NR_FILE_MAPPED)),
+ nid, K(sum_zone_node_page_state(nid, NR_ANON_PAGES)),
nid, K(i.sharedram),
- nid, node_page_state(nid, NR_KERNEL_STACK) *
+ nid, sum_zone_node_page_state(nid, NR_KERNEL_STACK) *
THREAD_SIZE / 1024,
- nid, K(node_page_state(nid, NR_PAGETABLE)),
- nid, K(node_page_state(nid, NR_UNSTABLE_NFS)),
- nid, K(node_page_state(nid, NR_BOUNCE)),
- nid, K(node_page_state(nid, NR_WRITEBACK_TEMP)),
- nid, K(node_page_state(nid, NR_SLAB_RECLAIMABLE) +
- node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
- nid, K(node_page_state(nid, NR_SLAB_RECLAIMABLE)),
+ nid, K(sum_zone_node_page_state(nid, NR_PAGETABLE)),
+ nid, K(sum_zone_node_page_state(nid, NR_UNSTABLE_NFS)),
+ nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)),
+ nid, K(sum_zone_node_page_state(nid, NR_WRITEBACK_TEMP)),
+ nid, K(sum_zone_node_page_state(nid, NR_SLAB_RECLAIMABLE) +
+ sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
+ nid, K(sum_zone_node_page_state(nid, NR_SLAB_RECLAIMABLE)),
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- nid, K(node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
- nid, K(node_page_state(nid, NR_ANON_THPS) *
+ nid, K(sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)),
+ nid, K(sum_zone_node_page_state(nid, NR_ANON_THPS) *
HPAGE_PMD_NR),
- nid, K(node_page_state(nid, NR_SHMEM_THPS) *
+ nid, K(sum_zone_node_page_state(nid, NR_SHMEM_THPS) *
HPAGE_PMD_NR),
- nid, K(node_page_state(nid, NR_SHMEM_PMDMAPPED) *
+ nid, K(sum_zone_node_page_state(nid, NR_SHMEM_PMDMAPPED) *
HPAGE_PMD_NR));
#else
- nid, K(node_page_state(nid, NR_SLAB_UNRECLAIMABLE)));
+ nid, K(sum_zone_node_page_state(nid, NR_SLAB_UNRECLAIMABLE)));
#endif
n += hugetlb_report_node_meminfo(nid, buf + n);
return n;
@@ -160,12 +160,12 @@ static ssize_t node_read_numastat(struct device *dev,
"interleave_hit %lu\n"
"local_node %lu\n"
"other_node %lu\n",
- node_page_state(dev->id, NUMA_HIT),
- node_page_state(dev->id, NUMA_MISS),
- node_page_state(dev->id, NUMA_FOREIGN),
- node_page_state(dev->id, NUMA_INTERLEAVE_HIT),
- node_page_state(dev->id, NUMA_LOCAL),
- node_page_state(dev->id, NUMA_OTHER));
+ sum_zone_node_page_state(dev->id, NUMA_HIT),
+ sum_zone_node_page_state(dev->id, NUMA_MISS),
+ sum_zone_node_page_state(dev->id, NUMA_FOREIGN),
+ sum_zone_node_page_state(dev->id, NUMA_INTERLEAVE_HIT),
+ sum_zone_node_page_state(dev->id, NUMA_LOCAL),
+ sum_zone_node_page_state(dev->id, NUMA_OTHER));
}
static DEVICE_ATTR(numastat, S_IRUGO, node_read_numastat, NULL);
@@ -173,12 +173,18 @@ static ssize_t node_read_vmstat(struct device *dev,
struct device_attribute *attr, char *buf)
{
int nid = dev->id;
+ struct pglist_data *pgdat = NODE_DATA(nid);
int i;
int n = 0;
for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++)
n += sprintf(buf+n, "%s %lu\n", vmstat_text[i],
- node_page_state(nid, i));
+ sum_zone_node_page_state(nid, i));
+
+ for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++)
+ n += sprintf(buf+n, "%s %lu\n",
+ vmstat_text[i + NR_VM_ZONE_STAT_ITEMS],
+ node_page_state(pgdat, i));
return n;
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index b21e5f30378e..dd79aa2800a3 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -933,6 +933,11 @@ static inline struct zone *page_zone(const struct page *page)
return &NODE_DATA(page_to_nid(page))->node_zones[page_zonenum(page)];
}
+static inline pg_data_t *page_pgdat(const struct page *page)
+{
+ return NODE_DATA(page_to_nid(page));
+}
+
#ifdef SECTION_IN_PAGE_FLAGS
static inline void set_page_section(struct page *page, unsigned long section)
{
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 19425e988bdc..078ecb81e209 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -160,6 +160,10 @@ enum zone_stat_item {
NR_FREE_CMA_PAGES,
NR_VM_ZONE_STAT_ITEMS };
+enum node_stat_item {
+ NR_VM_NODE_STAT_ITEMS
+};
+
/*
* We do arithmetic on the LRU lists in various places in the code,
* so it is important to keep the active lists LRU_ACTIVE higher in
@@ -267,6 +271,11 @@ struct per_cpu_pageset {
#endif
};
+struct per_cpu_nodestat {
+ s8 stat_threshold;
+ s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS];
+};
+
#endif /* !__GENERATING_BOUNDS.H */
enum zone_type {
@@ -695,6 +704,10 @@ typedef struct pglist_data {
struct list_head split_queue;
unsigned long split_queue_len;
#endif
+
+ /* Per-node vmstats */
+ struct per_cpu_nodestat __percpu *per_cpu_nodestats;
+ atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
} pg_data_t;
#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index d2da8e053210..d1744aa3ab9c 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -106,20 +106,38 @@ static inline void vm_events_fold_cpu(int cpu)
zone_idx(zone), delta)
/*
- * Zone based page accounting with per cpu differentials.
+ * Zone and node-based page accounting with per cpu differentials.
*/
-extern atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS];
+extern atomic_long_t vm_zone_stat[NR_VM_ZONE_STAT_ITEMS];
+extern atomic_long_t vm_node_stat[NR_VM_NODE_STAT_ITEMS];
static inline void zone_page_state_add(long x, struct zone *zone,
enum zone_stat_item item)
{
atomic_long_add(x, &zone->vm_stat[item]);
- atomic_long_add(x, &vm_stat[item]);
+ atomic_long_add(x, &vm_zone_stat[item]);
+}
+
+static inline void node_page_state_add(long x, struct pglist_data *pgdat,
+ enum node_stat_item item)
+{
+ atomic_long_add(x, &pgdat->vm_stat[item]);
+ atomic_long_add(x, &vm_node_stat[item]);
}
static inline unsigned long global_page_state(enum zone_stat_item item)
{
- long x = atomic_long_read(&vm_stat[item]);
+ long x = atomic_long_read(&vm_zone_stat[item]);
+#ifdef CONFIG_SMP
+ if (x < 0)
+ x = 0;
+#endif
+ return x;
+}
+
+static inline unsigned long global_node_page_state(enum node_stat_item item)
+{
+ long x = atomic_long_read(&vm_node_stat[item]);
#ifdef CONFIG_SMP
if (x < 0)
x = 0;
@@ -161,31 +179,44 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone,
}
#ifdef CONFIG_NUMA
-
-extern unsigned long node_page_state(int node, enum zone_stat_item item);
-
+extern unsigned long sum_zone_node_page_state(int node,
+ enum zone_stat_item item);
+extern unsigned long node_page_state(struct pglist_data *pgdat,
+ enum node_stat_item item);
#else
-
-#define node_page_state(node, item) global_page_state(item)
-
+#define sum_zone_node_page_state(node, item) global_page_state(item)
+#define node_page_state(node, item) global_node_page_state(item)
#endif /* CONFIG_NUMA */
#define add_zone_page_state(__z, __i, __d) mod_zone_page_state(__z, __i, __d)
#define sub_zone_page_state(__z, __i, __d) mod_zone_page_state(__z, __i, -(__d))
+#define add_node_page_state(__p, __i, __d) mod_node_page_state(__p, __i, __d)
+#define sub_node_page_state(__p, __i, __d) mod_node_page_state(__p, __i, -(__d))
#ifdef CONFIG_SMP
void __mod_zone_page_state(struct zone *, enum zone_stat_item item, long);
void __inc_zone_page_state(struct page *, enum zone_stat_item);
void __dec_zone_page_state(struct page *, enum zone_stat_item);
+void __mod_node_page_state(struct pglist_data *, enum node_stat_item item, long);
+void __inc_node_page_state(struct page *, enum node_stat_item);
+void __dec_node_page_state(struct page *, enum node_stat_item);
+
void mod_zone_page_state(struct zone *, enum zone_stat_item, long);
void inc_zone_page_state(struct page *, enum zone_stat_item);
void dec_zone_page_state(struct page *, enum zone_stat_item);
+void mod_node_page_state(struct pglist_data *, enum node_stat_item, long);
+void inc_node_page_state(struct page *, enum node_stat_item);
+void dec_node_page_state(struct page *, enum node_stat_item);
+
extern void inc_zone_state(struct zone *, enum zone_stat_item);
+extern void inc_node_state(struct pglist_data *, enum node_stat_item);
extern void __inc_zone_state(struct zone *, enum zone_stat_item);
+extern void __inc_node_state(struct pglist_data *, enum node_stat_item);
extern void dec_zone_state(struct zone *, enum zone_stat_item);
extern void __dec_zone_state(struct zone *, enum zone_stat_item);
+extern void __dec_node_state(struct pglist_data *, enum node_stat_item);
void quiet_vmstat(void);
void cpu_vm_stats_fold(int cpu);
@@ -213,16 +244,34 @@ static inline void __mod_zone_page_state(struct zone *zone,
zone_page_state_add(delta, zone, item);
}
+static inline void __mod_node_page_state(struct pglist_data *pgdat,
+ enum node_stat_item item, int delta)
+{
+ node_page_state_add(delta, pgdat, item);
+}
+
static inline void __inc_zone_state(struct zone *zone, enum zone_stat_item item)
{
atomic_long_inc(&zone->vm_stat[item]);
- atomic_long_inc(&vm_stat[item]);
+ atomic_long_inc(&vm_zone_stat[item]);
+}
+
+static inline void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)
+{
+ atomic_long_inc(&pgdat->vm_stat[item]);
+ atomic_long_inc(&vm_node_stat[item]);
}
static inline void __dec_zone_state(struct zone *zone, enum zone_stat_item item)
{
atomic_long_dec(&zone->vm_stat[item]);
- atomic_long_dec(&vm_stat[item]);
+ atomic_long_dec(&vm_zone_stat[item]);
+}
+
+static inline void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item)
+{
+ atomic_long_dec(&pgdat->vm_stat[item]);
+ atomic_long_dec(&vm_node_stat[item]);
}
static inline void __inc_zone_page_state(struct page *page,
@@ -231,12 +280,26 @@ static inline void __inc_zone_page_state(struct page *page,
__inc_zone_state(page_zone(page), item);
}
+static inline void __inc_node_page_state(struct page *page,
+ enum node_stat_item item)
+{
+ __inc_node_state(page_pgdat(page), item);
+}
+
+
static inline void __dec_zone_page_state(struct page *page,
enum zone_stat_item item)
{
__dec_zone_state(page_zone(page), item);
}
+static inline void __dec_node_page_state(struct page *page,
+ enum node_stat_item item)
+{
+ __dec_node_state(page_pgdat(page), item);
+}
+
+
/*
* We only use atomic operations to update counters. So there is no need to
* disable interrupts.
@@ -245,7 +308,12 @@ static inline void __dec_zone_page_state(struct page *page,
#define dec_zone_page_state __dec_zone_page_state
#define mod_zone_page_state __mod_zone_page_state
+#define inc_node_page_state __inc_node_page_state
+#define dec_node_page_state __dec_node_page_state
+#define mod_node_page_state __mod_node_page_state
+
#define inc_zone_state __inc_zone_state
+#define inc_node_state __inc_node_state
#define dec_zone_state __dec_zone_state
#define set_pgdat_percpu_threshold(pgdat, callback) { }
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 403c5dcd24da..34e46c02a406 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4247,8 +4247,8 @@ void si_meminfo_node(struct sysinfo *val, int nid)
for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++)
managed_pages += pgdat->node_zones[zone_type].managed_pages;
val->totalram = managed_pages;
- val->sharedram = node_page_state(nid, NR_SHMEM);
- val->freeram = node_page_state(nid, NR_FREE_PAGES);
+ val->sharedram = sum_zone_node_page_state(nid, NR_SHMEM);
+ val->freeram = sum_zone_node_page_state(nid, NR_FREE_PAGES);
#ifdef CONFIG_HIGHMEM
for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) {
struct zone *zone = &pgdat->node_zones[zone_type];
@@ -5373,6 +5373,11 @@ static void __meminit setup_zone_pageset(struct zone *zone)
zone->pageset = alloc_percpu(struct per_cpu_pageset);
for_each_possible_cpu(cpu)
zone_pageset_init(zone, cpu);
+
+ if (!zone->zone_pgdat->per_cpu_nodestats) {
+ zone->zone_pgdat->per_cpu_nodestats =
+ alloc_percpu(struct per_cpu_nodestat);
+ }
}
/*
@@ -6078,6 +6083,7 @@ void __paginginit free_area_init_node(int nid, unsigned long *zones_size,
reset_deferred_meminit(pgdat);
pgdat->node_id = nid;
pgdat->node_start_pfn = node_start_pfn;
+ pgdat->per_cpu_nodestats = NULL;
#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
get_pfn_range_for_nid(nid, &start_pfn, &end_pfn);
pr_info("Initmem setup node %d [mem %#018Lx-%#018Lx]\n", nid,
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 7997f52935c9..90b0737ee4be 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -86,8 +86,10 @@ void vm_events_fold_cpu(int cpu)
*
* vm_stat contains the global counters
*/
-atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS] __cacheline_aligned_in_smp;
-EXPORT_SYMBOL(vm_stat);
+atomic_long_t vm_zone_stat[NR_VM_ZONE_STAT_ITEMS] __cacheline_aligned_in_smp;
+atomic_long_t vm_node_stat[NR_VM_NODE_STAT_ITEMS] __cacheline_aligned_in_smp;
+EXPORT_SYMBOL(vm_zone_stat);
+EXPORT_SYMBOL(vm_node_stat);
#ifdef CONFIG_SMP
@@ -172,13 +174,17 @@ void refresh_zone_stat_thresholds(void)
int threshold;
for_each_populated_zone(zone) {
+ struct pglist_data *pgdat = zone->zone_pgdat;
unsigned long max_drift, tolerate_drift;
threshold = calculate_normal_threshold(zone);
- for_each_online_cpu(cpu)
+ for_each_online_cpu(cpu) {
per_cpu_ptr(zone->pageset, cpu)->stat_threshold
= threshold;
+ per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold
+ = threshold;
+ }
/*
* Only set percpu_drift_mark if there is a danger that
@@ -238,6 +244,26 @@ void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item,
}
EXPORT_SYMBOL(__mod_zone_page_state);
+void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item,
+ long delta)
+{
+ struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats;
+ s8 __percpu *p = pcp->vm_node_stat_diff + item;
+ long x;
+ long t;
+
+ x = delta + __this_cpu_read(*p);
+
+ t = __this_cpu_read(pcp->stat_threshold);
+
+ if (unlikely(x > t || x < -t)) {
+ node_page_state_add(x, pgdat, item);
+ x = 0;
+ }
+ __this_cpu_write(*p, x);
+}
+EXPORT_SYMBOL(__mod_node_page_state);
+
/*
* Optimized increment and decrement functions.
*
@@ -277,12 +303,34 @@ void __inc_zone_state(struct zone *zone, enum zone_stat_item item)
}
}
+void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)
+{
+ struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats;
+ s8 __percpu *p = pcp->vm_node_stat_diff + item;
+ s8 v, t;
+
+ v = __this_cpu_inc_return(*p);
+ t = __this_cpu_read(pcp->stat_threshold);
+ if (unlikely(v > t)) {
+ s8 overstep = t >> 1;
+
+ node_page_state_add(v + overstep, pgdat, item);
+ __this_cpu_write(*p, -overstep);
+ }
+}
+
void __inc_zone_page_state(struct page *page, enum zone_stat_item item)
{
__inc_zone_state(page_zone(page), item);
}
EXPORT_SYMBOL(__inc_zone_page_state);
+void __inc_node_page_state(struct page *page, enum node_stat_item item)
+{
+ __inc_node_state(page_pgdat(page), item);
+}
+EXPORT_SYMBOL(__inc_node_page_state);
+
void __dec_zone_state(struct zone *zone, enum zone_stat_item item)
{
struct per_cpu_pageset __percpu *pcp = zone->pageset;
@@ -299,12 +347,34 @@ void __dec_zone_state(struct zone *zone, enum zone_stat_item item)
}
}
+void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item)
+{
+ struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats;
+ s8 __percpu *p = pcp->vm_node_stat_diff + item;
+ s8 v, t;
+
+ v = __this_cpu_dec_return(*p);
+ t = __this_cpu_read(pcp->stat_threshold);
+ if (unlikely(v < - t)) {
+ s8 overstep = t >> 1;
+
+ node_page_state_add(v - overstep, pgdat, item);
+ __this_cpu_write(*p, overstep);
+ }
+}
+
void __dec_zone_page_state(struct page *page, enum zone_stat_item item)
{
__dec_zone_state(page_zone(page), item);
}
EXPORT_SYMBOL(__dec_zone_page_state);
+void __dec_node_page_state(struct page *page, enum node_stat_item item)
+{
+ __dec_node_state(page_pgdat(page), item);
+}
+EXPORT_SYMBOL(__dec_node_page_state);
+
#ifdef CONFIG_HAVE_CMPXCHG_LOCAL
/*
* If we have cmpxchg_local support then we do not need to incur the overhead
@@ -318,8 +388,8 @@ EXPORT_SYMBOL(__dec_zone_page_state);
* 1 Overstepping half of threshold
* -1 Overstepping minus half of threshold
*/
-static inline void mod_state(struct zone *zone, enum zone_stat_item item,
- long delta, int overstep_mode)
+static inline void mod_zone_state(struct zone *zone,
+ enum zone_stat_item item, long delta, int overstep_mode)
{
struct per_cpu_pageset __percpu *pcp = zone->pageset;
s8 __percpu *p = pcp->vm_stat_diff + item;
@@ -359,26 +429,88 @@ static inline void mod_state(struct zone *zone, enum zone_stat_item item,
void mod_zone_page_state(struct zone *zone, enum zone_stat_item item,
long delta)
{
- mod_state(zone, item, delta, 0);
+ mod_zone_state(zone, item, delta, 0);
}
EXPORT_SYMBOL(mod_zone_page_state);
void inc_zone_state(struct zone *zone, enum zone_stat_item item)
{
- mod_state(zone, item, 1, 1);
+ mod_zone_state(zone, item, 1, 1);
}
void inc_zone_page_state(struct page *page, enum zone_stat_item item)
{
- mod_state(page_zone(page), item, 1, 1);
+ mod_zone_state(page_zone(page), item, 1, 1);
}
EXPORT_SYMBOL(inc_zone_page_state);
void dec_zone_page_state(struct page *page, enum zone_stat_item item)
{
- mod_state(page_zone(page), item, -1, -1);
+ mod_zone_state(page_zone(page), item, -1, -1);
}
EXPORT_SYMBOL(dec_zone_page_state);
+
+static inline void mod_node_state(struct pglist_data *pgdat,
+ enum node_stat_item item, int delta, int overstep_mode)
+{
+ struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats;
+ s8 __percpu *p = pcp->vm_node_stat_diff + item;
+ long o, n, t, z;
+
+ do {
+ z = 0; /* overflow to node counters */
+
+ /*
+ * The fetching of the stat_threshold is racy. We may apply
+ * a counter threshold to the wrong the cpu if we get
+ * rescheduled while executing here. However, the next
+ * counter update will apply the threshold again and
+ * therefore bring the counter under the threshold again.
+ *
+ * Most of the time the thresholds are the same anyways
+ * for all cpus in a node.
+ */
+ t = this_cpu_read(pcp->stat_threshold);
+
+ o = this_cpu_read(*p);
+ n = delta + o;
+
+ if (n > t || n < -t) {
+ int os = overstep_mode * (t >> 1) ;
+
+ /* Overflow must be added to node counters */
+ z = n + os;
+ n = -os;
+ }
+ } while (this_cpu_cmpxchg(*p, o, n) != o);
+
+ if (z)
+ node_page_state_add(z, pgdat, item);
+}
+
+void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item,
+ long delta)
+{
+ mod_node_state(pgdat, item, delta, 0);
+}
+EXPORT_SYMBOL(mod_node_page_state);
+
+void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)
+{
+ mod_node_state(pgdat, item, 1, 1);
+}
+
+void inc_node_page_state(struct page *page, enum node_stat_item item)
+{
+ mod_node_state(page_pgdat(page), item, 1, 1);
+}
+EXPORT_SYMBOL(inc_node_page_state);
+
+void dec_node_page_state(struct page *page, enum node_stat_item item)
+{
+ mod_node_state(page_pgdat(page), item, -1, -1);
+}
+EXPORT_SYMBOL(dec_node_page_state);
#else
/*
* Use interrupt disable to serialize counter updates
@@ -424,21 +556,69 @@ void dec_zone_page_state(struct page *page, enum zone_stat_item item)
local_irq_restore(flags);
}
EXPORT_SYMBOL(dec_zone_page_state);
-#endif
+void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ __inc_node_state(pgdat, item);
+ local_irq_restore(flags);
+}
+EXPORT_SYMBOL(inc_node_state);
+
+void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item,
+ long delta)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ __mod_node_page_state(pgdat, item, delta);
+ local_irq_restore(flags);
+}
+EXPORT_SYMBOL(mod_node_page_state);
+
+void inc_node_page_state(struct page *page, enum node_stat_item item)
+{
+ unsigned long flags;
+ struct pglist_data *pgdat;
+
+ pgdat = page_pgdat(page);
+ local_irq_save(flags);
+ __inc_node_state(pgdat, item);
+ local_irq_restore(flags);
+}
+EXPORT_SYMBOL(inc_node_page_state);
+
+void dec_node_page_state(struct page *page, enum node_stat_item item)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ __dec_node_page_state(page, item);
+ local_irq_restore(flags);
+}
+EXPORT_SYMBOL(dec_node_page_state);
+#endif
/*
* Fold a differential into the global counters.
* Returns the number of counters updated.
*/
-static int fold_diff(int *diff)
+static int fold_diff(int *zone_diff, int *node_diff)
{
int i;
int changes = 0;
for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++)
- if (diff[i]) {
- atomic_long_add(diff[i], &vm_stat[i]);
+ if (zone_diff[i]) {
+ atomic_long_add(zone_diff[i], &vm_zone_stat[i]);
+ changes++;
+ }
+
+ for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++)
+ if (node_diff[i]) {
+ atomic_long_add(node_diff[i], &vm_node_stat[i]);
changes++;
}
return changes;
@@ -462,9 +642,11 @@ static int fold_diff(int *diff)
*/
static int refresh_cpu_vm_stats(bool do_pagesets)
{
+ struct pglist_data *pgdat;
struct zone *zone;
int i;
- int global_diff[NR_VM_ZONE_STAT_ITEMS] = { 0, };
+ int global_zone_diff[NR_VM_ZONE_STAT_ITEMS] = { 0, };
+ int global_node_diff[NR_VM_NODE_STAT_ITEMS] = { 0, };
int changes = 0;
for_each_populated_zone(zone) {
@@ -477,7 +659,7 @@ static int refresh_cpu_vm_stats(bool do_pagesets)
if (v) {
atomic_long_add(v, &zone->vm_stat[i]);
- global_diff[i] += v;
+ global_zone_diff[i] += v;
#ifdef CONFIG_NUMA
/* 3 seconds idle till flush */
__this_cpu_write(p->expire, 3);
@@ -516,7 +698,22 @@ static int refresh_cpu_vm_stats(bool do_pagesets)
}
#endif
}
- changes += fold_diff(global_diff);
+
+ for_each_online_pgdat(pgdat) {
+ struct per_cpu_nodestat __percpu *p = pgdat->per_cpu_nodestats;
+
+ for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) {
+ int v;
+
+ v = this_cpu_xchg(p->vm_node_stat_diff[i], 0);
+ if (v) {
+ atomic_long_add(v, &pgdat->vm_stat[i]);
+ global_node_diff[i] += v;
+ }
+ }
+ }
+
+ changes += fold_diff(global_zone_diff, global_node_diff);
return changes;
}
@@ -527,9 +724,11 @@ static int refresh_cpu_vm_stats(bool do_pagesets)
*/
void cpu_vm_stats_fold(int cpu)
{
+ struct pglist_data *pgdat;
struct zone *zone;
int i;
- int global_diff[NR_VM_ZONE_STAT_ITEMS] = { 0, };
+ int global_zone_diff[NR_VM_ZONE_STAT_ITEMS] = { 0, };
+ int global_node_diff[NR_VM_NODE_STAT_ITEMS] = { 0, };
for_each_populated_zone(zone) {
struct per_cpu_pageset *p;
@@ -543,11 +742,27 @@ void cpu_vm_stats_fold(int cpu)
v = p->vm_stat_diff[i];
p->vm_stat_diff[i] = 0;
atomic_long_add(v, &zone->vm_stat[i]);
- global_diff[i] += v;
+ global_zone_diff[i] += v;
}
}
- fold_diff(global_diff);
+ for_each_online_pgdat(pgdat) {
+ struct per_cpu_nodestat *p;
+
+ p = per_cpu_ptr(pgdat->per_cpu_nodestats, cpu);
+
+ for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++)
+ if (p->vm_node_stat_diff[i]) {
+ int v;
+
+ v = p->vm_node_stat_diff[i];
+ p->vm_node_stat_diff[i] = 0;
+ atomic_long_add(v, &pgdat->vm_stat[i]);
+ global_node_diff[i] += v;
+ }
+ }
+
+ fold_diff(global_zone_diff, global_node_diff);
}
/*
@@ -563,16 +778,19 @@ void drain_zonestat(struct zone *zone, struct per_cpu_pageset *pset)
int v = pset->vm_stat_diff[i];
pset->vm_stat_diff[i] = 0;
atomic_long_add(v, &zone->vm_stat[i]);
- atomic_long_add(v, &vm_stat[i]);
+ atomic_long_add(v, &vm_zone_stat[i]);
}
}
#endif
#ifdef CONFIG_NUMA
/*
- * Determine the per node value of a stat item.
+ * Determine the per node value of a stat item. This function
+ * is called frequently in a NUMA machine, so try to be as
+ * frugal as possible.
*/
-unsigned long node_page_state(int node, enum zone_stat_item item)
+unsigned long sum_zone_node_page_state(int node,
+ enum zone_stat_item item)
{
struct zone *zones = NODE_DATA(node)->node_zones;
int i;
@@ -584,6 +802,19 @@ unsigned long node_page_state(int node, enum zone_stat_item item)
return count;
}
+/*
+ * Determine the per node value of a stat item.
+ */
+unsigned long node_page_state(struct pglist_data *pgdat,
+ enum node_stat_item item)
+{
+ long x = atomic_long_read(&pgdat->vm_stat[item]);
+#ifdef CONFIG_SMP
+ if (x < 0)
+ x = 0;
+#endif
+ return x;
+}
#endif
#ifdef CONFIG_COMPACTION
@@ -1287,6 +1518,7 @@ static void *vmstat_start(struct seq_file *m, loff_t *pos)
if (*pos >= ARRAY_SIZE(vmstat_text))
return NULL;
stat_items_size = NR_VM_ZONE_STAT_ITEMS * sizeof(unsigned long) +
+ NR_VM_NODE_STAT_ITEMS * sizeof(unsigned long) +
NR_VM_WRITEBACK_STAT_ITEMS * sizeof(unsigned long);
#ifdef CONFIG_VM_EVENT_COUNTERS
@@ -1301,6 +1533,10 @@ static void *vmstat_start(struct seq_file *m, loff_t *pos)
v[i] = global_page_state(i);
v += NR_VM_ZONE_STAT_ITEMS;
+ for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++)
+ v[i] = global_node_page_state(i);
+ v += NR_VM_NODE_STAT_ITEMS;
+
global_dirty_limits(v + NR_DIRTY_BG_THRESHOLD,
v + NR_DIRTY_THRESHOLD);
v += NR_VM_WRITEBACK_STAT_ITEMS;
@@ -1390,7 +1626,7 @@ int vmstat_refresh(struct ctl_table *table, int write,
if (err)
return err;
for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) {
- val = atomic_long_read(&vm_stat[i]);
+ val = atomic_long_read(&vm_zone_stat[i]);
if (val < 0) {
switch (i) {
case NR_ALLOC_BATCH:
diff --git a/mm/workingset.c b/mm/workingset.c
index 8252de4566e9..ba972ac2dfdd 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -351,12 +351,13 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,
shadow_nodes = list_lru_shrink_count(&workingset_shadow_nodes, sc);
local_irq_enable();
- if (memcg_kmem_enabled())
+ if (memcg_kmem_enabled()) {
pages = mem_cgroup_node_nr_lru_pages(sc->memcg, sc->nid,
LRU_ALL_FILE);
- else
- pages = node_page_state(sc->nid, NR_ACTIVE_FILE) +
- node_page_state(sc->nid, NR_INACTIVE_FILE);
+ } else {
+ pages = sum_zone_node_page_state(sc->nid, NR_ACTIVE_FILE) +
+ sum_zone_node_page_state(sc->nid, NR_INACTIVE_FILE);
+ }
/*
* Active cache pages are limited to 50% of memory, and shadow
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-07-05 01:50 +0200 |
| Subject | Re: [PATCH 01/31] mm, vmstat: add infrastructure for per-node vmstats |
| Message-ID | <rRlMJ-3AJ-5@gated-at.bofh.it> |
| In reply to | #1435601 |
On Fri, Jul 01, 2016 at 09:01:09PM +0100, Mel Gorman wrote:
> VM statistic counters for reclaim decisions are zone-based. If the kernel
> is to reclaim on a per-node basis then we need to track per-node
> statistics but there is no infrastructure for that. The most notable
> change is that the old node_page_state is renamed to
> sum_zone_node_page_state. The new node_page_state takes a pglist_data and
> uses per-node stats but none exist yet. There is some renaming such as
> vm_stat to vm_zone_stat and the addition of vm_node_stat and the renaming
> of mod_state to mod_zone_state. Otherwise, this is mostly a mechanical
> patch with no functional change. There is a lot of similarity between the
> node and zone helpers which is unfortunate but there was no obvious way of
> reusing the code and maintaining type safety.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> Acked-by: Vlastimil Babka <vbabka@suse.cz>
> ---
<snip>
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 7997f52935c9..90b0737ee4be 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -86,8 +86,10 @@ void vm_events_fold_cpu(int cpu)
> *
> * vm_stat contains the global counters
> */
> -atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS] __cacheline_aligned_in_smp;
> -EXPORT_SYMBOL(vm_stat);
> +atomic_long_t vm_zone_stat[NR_VM_ZONE_STAT_ITEMS] __cacheline_aligned_in_smp;
> +atomic_long_t vm_node_stat[NR_VM_NODE_STAT_ITEMS] __cacheline_aligned_in_smp;
> +EXPORT_SYMBOL(vm_zone_stat);
> +EXPORT_SYMBOL(vm_node_stat);
>
> #ifdef CONFIG_SMP
>
> @@ -172,13 +174,17 @@ void refresh_zone_stat_thresholds(void)
> int threshold;
>
> for_each_populated_zone(zone) {
> + struct pglist_data *pgdat = zone->zone_pgdat;
> unsigned long max_drift, tolerate_drift;
>
> threshold = calculate_normal_threshold(zone);
>
> - for_each_online_cpu(cpu)
> + for_each_online_cpu(cpu) {
> per_cpu_ptr(zone->pageset, cpu)->stat_threshold
> = threshold;
> + per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold
> + = threshold;
> + }
I didn't see other patches yet so it might fix it then.
per_cpu_nodestats is per node not zone but it use per-zone threshold
and even overwritten by next zones. I don't think it's not intended.
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-07-05 10:20 +0200 |
| Subject | Re: [PATCH 01/31] mm, vmstat: add infrastructure for per-node vmstats |
| Message-ID | <rRtKh-g7-5@gated-at.bofh.it> |
| In reply to | #1436657 |
On Tue, Jul 05, 2016 at 08:50:18AM +0900, Minchan Kim wrote:
> > @@ -172,13 +174,17 @@ void refresh_zone_stat_thresholds(void)
> > int threshold;
> >
> > for_each_populated_zone(zone) {
> > + struct pglist_data *pgdat = zone->zone_pgdat;
> > unsigned long max_drift, tolerate_drift;
> >
> > threshold = calculate_normal_threshold(zone);
> >
> > - for_each_online_cpu(cpu)
> > + for_each_online_cpu(cpu) {
> > per_cpu_ptr(zone->pageset, cpu)->stat_threshold
> > = threshold;
> > + per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold
> > + = threshold;
> > + }
>
> I didn't see other patches yet so it might fix it then.
>
> per_cpu_nodestats is per node not zone but it use per-zone threshold
> and even overwritten by next zones. I don't think it's not intended.
It was intended that the threshold from one zone would be used but now
that you point it out, it would use the threshold for the smallest zone
in the node which is sub-optimal. I applied the patch below on top to
use the threshold from the largest zone. I considered using the sum of
all thresholds but feared it might allow too much per-cpu drift. It can
be switched to the sum if we find a case where vmstat updates are too
high.
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 90b0737ee4be..3345d396a99b 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -169,10 +169,18 @@ int calculate_normal_threshold(struct zone *zone)
*/
void refresh_zone_stat_thresholds(void)
{
+ struct pglist_data *pgdat;
struct zone *zone;
int cpu;
int threshold;
+ /* Zero current pgdat thresholds */
+ for_each_online_pgdat(pgdat) {
+ for_each_online_cpu(cpu) {
+ per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold = 0;
+ }
+ }
+
for_each_populated_zone(zone) {
struct pglist_data *pgdat = zone->zone_pgdat;
unsigned long max_drift, tolerate_drift;
@@ -180,10 +188,15 @@ void refresh_zone_stat_thresholds(void)
threshold = calculate_normal_threshold(zone);
for_each_online_cpu(cpu) {
+ int pgdat_threshold;
+
per_cpu_ptr(zone->pageset, cpu)->stat_threshold
= threshold;
+
+ /* Base nodestat threshold on the largest populated zone. */
+ pgdat_threshold = per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold;
per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold
- = threshold;
+ = max(threshold, pgdat_threshold);
}
/*
--
Mel Gorman
SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-07-06 02:20 +0200 |
| Subject | Re: [PATCH 01/31] mm, vmstat: add infrastructure for per-node vmstats |
| Message-ID | <rRIJj-1Ki-1@gated-at.bofh.it> |
| In reply to | #1436833 |
On Tue, Jul 05, 2016 at 09:14:05AM +0100, Mel Gorman wrote:
> On Tue, Jul 05, 2016 at 08:50:18AM +0900, Minchan Kim wrote:
> > > @@ -172,13 +174,17 @@ void refresh_zone_stat_thresholds(void)
> > > int threshold;
> > >
> > > for_each_populated_zone(zone) {
> > > + struct pglist_data *pgdat = zone->zone_pgdat;
> > > unsigned long max_drift, tolerate_drift;
> > >
> > > threshold = calculate_normal_threshold(zone);
> > >
> > > - for_each_online_cpu(cpu)
> > > + for_each_online_cpu(cpu) {
> > > per_cpu_ptr(zone->pageset, cpu)->stat_threshold
> > > = threshold;
> > > + per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold
> > > + = threshold;
> > > + }
> >
> > I didn't see other patches yet so it might fix it then.
> >
> > per_cpu_nodestats is per node not zone but it use per-zone threshold
> > and even overwritten by next zones. I don't think it's not intended.
>
> It was intended that the threshold from one zone would be used but now
> that you point it out, it would use the threshold for the smallest zone
> in the node which is sub-optimal. I applied the patch below on top to
> use the threshold from the largest zone. I considered using the sum of
> all thresholds but feared it might allow too much per-cpu drift. It can
> be switched to the sum if we find a case where vmstat updates are too
> high.
Fair enough.
[toc] | [prev] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-07-04 03:40 +0200 |
| Message-ID | <rR11D-7HN-5@gated-at.bofh.it> |
| In reply to | #1435600 |
On Fri, Jul 01, 2016 at 09:01:08PM +0100, Mel Gorman wrote: > (Sorry for the resend, I accidentally sent the branch that still had the > Signed-off-by's from mmotm still applied which is incorrect.) > > Previous releases double accounted LRU stats on the zone and the node > because it was required by should_reclaim_retry. The last patch in the > series removes the double accounting. It's not integrated with the series > as reviewers may not like the solution. If not, it can be safely dropped > without a major impact to the results. > > Changelog since v7 > o Rebase onto current mmots > o Avoid double accounting of stats in node and zone > o Kswapd will avoid more reclaim if an eligible zone is available > o Remove some duplications of sc->reclaim_idx and classzone_idx > o Print per-node stats in zoneinfo > > Changelog since v6 > o Correct reclaim_idx when direct reclaiming for memcg > o Also account LRU pages per zone for compaction/reclaim > o Add page_pgdat helper with more efficient lookup > o Init pgdat LRU lock only once > o Slight optimisation to wake_all_kswapds > o Always wake kcompactd when kswapd is going to sleep > o Rebase to mmotm as of June 15th, 2016 > > Changelog since v5 > o Rebase and adjust to changes > > Changelog since v4 > o Rebase on top of v3 of page allocator optimisation series > > Changelog since v3 > o Rebase on top of the page allocator optimisation series > o Remove RFC tag > > This is the latest version of a series that moves LRUs from the zones to > the node that is based upon 4.7-rc4 with Andrew's tree applied. While this > is a current rebase, the test results were based on mmotm as of June 23rd. > Conceptually, this series is simple but there are a lot of details. Some > of the broad motivations for this are; > > 1. The residency of a page partially depends on what zone the page was > allocated from. This is partially combatted by the fair zone allocation > policy but that is a partial solution that introduces overhead in the > page allocator paths. > > 2. Currently, reclaim on node 0 behaves slightly different to node 1. For > example, direct reclaim scans in zonelist order and reclaims even if > the zone is over the high watermark regardless of the age of pages > in that LRU. Kswapd on the other hand starts reclaim on the highest > unbalanced zone. A difference in distribution of file/anon pages due > to when they were allocated results can result in a difference in > again. While the fair zone allocation policy mitigates some of the > problems here, the page reclaim results on a multi-zone node will > always be different to a single-zone node. > it was scheduled on as a result. > > 3. kswapd and the page allocator scan zones in the opposite order to > avoid interfering with each other but it's sensitive to timing. This > mitigates the page allocator using pages that were allocated very recently > in the ideal case but it's sensitive to timing. When kswapd is allocating > from lower zones then it's great but during the rebalancing of the highest > zone, the page allocator and kswapd interfere with each other. It's worse > if the highest zone is small and difficult to balance. > > 4. slab shrinkers are node-based which makes it harder to identify the exact > relationship between slab reclaim and LRU reclaim. > > The reason we have zone-based reclaim is that we used to have > large highmem zones in common configurations and it was necessary > to quickly find ZONE_NORMAL pages for reclaim. Today, this is much > less of a concern as machines with lots of memory will (or should) use > 64-bit kernels. Combinations of 32-bit hardware and 64-bit hardware are > rare. Machines that do use highmem should have relatively low highmem:lowmem > ratios than we worried about in the past. Hello Mel, I agree the direction absolutely. However, I have a concern on highmem system as you already mentioned. Embedded products still use 2 ~ 3 ratio (highmem:lowmem). In such system, LRU churning by skipping other zone pages frequently might be significant for the performance. How big ratio between highmem:lowmem do you think a problem? > > Conceptually, moving to node LRUs should be easier to understand. The > page allocator plays fewer tricks to game reclaim and reclaim behaves > similarly on all nodes. > > The series has been tested on a 16 core UMA machine and a 2-socket 48 > core NUMA machine. The UMA results are presented in most cases as the NUMA > machine behaved similarly. I guess you would already test below with various highmem system(e.g., 2:1, 3:1, 4:1 and so on). If you have, could you mind sharing it? > > pagealloc > --------- > > This is a microbenchmark that shows the benefit of removing the fair zone > allocation policy. It was tested uip to order-4 but only orders 0 and 1 are > shown as the other orders were comparable. > > 4.7.0-rc4 4.7.0-rc4 > mmotm-20160623 nodelru-v8 > Min total-odr0-1 490.00 ( 0.00%) 463.00 ( 5.51%) > Min total-odr0-2 349.00 ( 0.00%) 325.00 ( 6.88%) > Min total-odr0-4 288.00 ( 0.00%) 272.00 ( 5.56%) > Min total-odr0-8 250.00 ( 0.00%) 235.00 ( 6.00%) > Min total-odr0-16 234.00 ( 0.00%) 222.00 ( 5.13%) > Min total-odr0-32 223.00 ( 0.00%) 205.00 ( 8.07%) > Min total-odr0-64 217.00 ( 0.00%) 202.00 ( 6.91%) > Min total-odr0-128 214.00 ( 0.00%) 207.00 ( 3.27%) > Min total-odr0-256 242.00 ( 0.00%) 242.00 ( 0.00%) > Min total-odr0-512 272.00 ( 0.00%) 265.00 ( 2.57%) > Min total-odr0-1024 290.00 ( 0.00%) 283.00 ( 2.41%) > Min total-odr0-2048 302.00 ( 0.00%) 296.00 ( 1.99%) > Min total-odr0-4096 311.00 ( 0.00%) 306.00 ( 1.61%) > Min total-odr0-8192 314.00 ( 0.00%) 309.00 ( 1.59%) > Min total-odr0-16384 315.00 ( 0.00%) 309.00 ( 1.90%) > Min total-odr1-1 741.00 ( 0.00%) 716.00 ( 3.37%) > Min total-odr1-2 565.00 ( 0.00%) 524.00 ( 7.26%) > Min total-odr1-4 457.00 ( 0.00%) 427.00 ( 6.56%) > Min total-odr1-8 408.00 ( 0.00%) 371.00 ( 9.07%) > Min total-odr1-16 383.00 ( 0.00%) 344.00 ( 10.18%) > Min total-odr1-32 378.00 ( 0.00%) 334.00 ( 11.64%) > Min total-odr1-64 383.00 ( 0.00%) 334.00 ( 12.79%) > Min total-odr1-128 376.00 ( 0.00%) 342.00 ( 9.04%) > Min total-odr1-256 381.00 ( 0.00%) 343.00 ( 9.97%) > Min total-odr1-512 388.00 ( 0.00%) 349.00 ( 10.05%) > Min total-odr1-1024 386.00 ( 0.00%) 356.00 ( 7.77%) > Min total-odr1-2048 389.00 ( 0.00%) 362.00 ( 6.94%) > Min total-odr1-4096 389.00 ( 0.00%) 362.00 ( 6.94%) > Min total-odr1-8192 389.00 ( 0.00%) 362.00 ( 6.94%) > > This shows a steady improvement throughout. The primary benefit is from > reduced system CPU usage which is obvious from the overall times; > > 4.7.0-rc4 4.7.0-rc4 > mmotm-20160623nodelru-v8 > User 191.39 191.61 > System 2651.24 2504.48 > Elapsed 2904.40 2757.01 > > The vmstats also showed that the fair zone allocation policy was definitely > removed as can be seen here; > > > 4.7.0-rc3 4.7.0-rc3 > mmotm-20160623 nodelru-v8 > DMA32 allocs 28794771816 0 > Normal allocs 48432582848 77227356392 > Movable allocs 0 0 > > tiobench on ext4 > ---------------- > > tiobench is a benchmark that artifically benefits if old pages remain resident > while new pages get reclaimed. The fair zone allocation policy mitigates this > problem so pages age fairly. While the benchmark has problems, it is important > that tiobench performance remains constant as it implies that page aging > problems that the fair zone allocation policy fixes are not re-introduced. > > 4.7.0-rc4 4.7.0-rc4 > mmotm-20160623 nodelru-v8 > Min PotentialReadSpeed 89.65 ( 0.00%) 90.34 ( 0.77%) > Min SeqRead-MB/sec-1 82.68 ( 0.00%) 83.13 ( 0.54%) > Min SeqRead-MB/sec-2 72.76 ( 0.00%) 72.15 ( -0.84%) > Min SeqRead-MB/sec-4 75.13 ( 0.00%) 74.23 ( -1.20%) > Min SeqRead-MB/sec-8 64.91 ( 0.00%) 65.25 ( 0.52%) > Min SeqRead-MB/sec-16 62.24 ( 0.00%) 62.76 ( 0.84%) > Min RandRead-MB/sec-1 0.88 ( 0.00%) 0.95 ( 7.95%) > Min RandRead-MB/sec-2 0.95 ( 0.00%) 0.94 ( -1.05%) > Min RandRead-MB/sec-4 1.43 ( 0.00%) 1.46 ( 2.10%) > Min RandRead-MB/sec-8 1.61 ( 0.00%) 1.58 ( -1.86%) > Min RandRead-MB/sec-16 1.80 ( 0.00%) 1.93 ( 7.22%) > Min SeqWrite-MB/sec-1 76.41 ( 0.00%) 78.84 ( 3.18%) > Min SeqWrite-MB/sec-2 74.11 ( 0.00%) 73.35 ( -1.03%) > Min SeqWrite-MB/sec-4 80.05 ( 0.00%) 78.69 ( -1.70%) > Min SeqWrite-MB/sec-8 72.88 ( 0.00%) 71.38 ( -2.06%) > Min SeqWrite-MB/sec-16 75.91 ( 0.00%) 75.81 ( -0.13%) > Min RandWrite-MB/sec-1 1.18 ( 0.00%) 1.12 ( -5.08%) > Min RandWrite-MB/sec-2 1.02 ( 0.00%) 1.02 ( 0.00%) > Min RandWrite-MB/sec-4 1.05 ( 0.00%) 0.99 ( -5.71%) > Min RandWrite-MB/sec-8 0.89 ( 0.00%) 0.92 ( 3.37%) > Min RandWrite-MB/sec-16 0.92 ( 0.00%) 0.89 ( -3.26%) > > This shows that the series has little or not impact on tiobench which is > desirable. It indicates that the fair zone allocation policy was removed > in a manner that didn't reintroduce one class of page aging bug. There > were only minor differences in overall reclaim activity > > 4.7.0-rc4 4.7.0-rc4 > mmotm-20160623nodelru-v8 > Minor Faults 645838 644036 > Major Faults 573 593 > Swap Ins 0 0 > Swap Outs 0 0 > Allocation stalls 24 0 > DMA allocs 0 0 > DMA32 allocs 46041453 44154171 > Normal allocs 78053072 79865782 > Movable allocs 0 0 > Direct pages scanned 10969 54504 > Kswapd pages scanned 93375144 93250583 > Kswapd pages reclaimed 93372243 93247714 > Direct pages reclaimed 10969 54504 > Kswapd efficiency 99% 99% > Kswapd velocity 13741.015 13711.950 > Direct efficiency 100% 100% > Direct velocity 1.614 8.014 > Percentage direct scans 0% 0% > Zone normal velocity 8641.875 13719.964 > Zone dma32 velocity 5100.754 0.000 > Zone dma velocity 0.000 0.000 > Page writes by reclaim 0.000 0.000 > Page writes file 0 0 > Page writes anon 0 0 > Page reclaim immediate 37 54 > > kswapd activity was roughly comparable. There were differences in direct > reclaim activity but negligible in the context of the overall workload > (velocity of 8 pages per second with the patches applied, 1.6 pages per > second in the baseline kernel). Hmm, nodelru's allocation stall is zero above but how does direct page scanning/reclaimed happens? Above, DMA32 allocs in nodelru is almost same but zone dma32 velocity is zero. What does it means? > > pgbench read-only large configuration on ext4 > --------------------------------------------- > > pgbench is a database benchmark that can be sensitive to page reclaim > decisions. This also checks if removing the fair zone allocation policy > is safe > > pgbench Transactions > 4.7.0-rc4 4.7.0-rc4 > mmotm-20160623 nodelru-v8 > Hmean 1 188.26 ( 0.00%) 189.78 ( 0.81%) > Hmean 5 330.66 ( 0.00%) 328.69 ( -0.59%) > Hmean 12 370.32 ( 0.00%) 380.72 ( 2.81%) > Hmean 21 368.89 ( 0.00%) 369.00 ( 0.03%) > Hmean 30 382.14 ( 0.00%) 360.89 ( -5.56%) > Hmean 32 428.87 ( 0.00%) 432.96 ( 0.95%) > > Negligible differences again. As with tiobench, overall reclaim activity > was comparable. > > bonnie++ on ext4 > ---------------- > > No interesting performance difference, negligible differences on reclaim > stats. > > paralleldd on ext4 > ------------------ > > This workload uses varying numbers of dd instances to read large amounts of > data from disk. > > 4.7.0-rc3 4.7.0-rc3 > mmotm-20160615 nodelru-v7r17 > Amean Elapsd-1 181.57 ( 0.00%) 179.63 ( 1.07%) > Amean Elapsd-3 188.29 ( 0.00%) 183.68 ( 2.45%) > Amean Elapsd-5 188.02 ( 0.00%) 181.73 ( 3.35%) > Amean Elapsd-7 186.07 ( 0.00%) 184.11 ( 1.05%) > Amean Elapsd-12 188.16 ( 0.00%) 183.51 ( 2.47%) > Amean Elapsd-16 189.03 ( 0.00%) 181.27 ( 4.10%) > > 4.7.0-rc3 4.7.0-rc3 > mmotm-20160615nodelru-v7r17 > User 1439.23 1433.37 > System 8332.31 8216.01 > Elapsed 3619.80 3532.69 > > There is a slight gain in performance, some of which is from the reduced system > CPU usage. There areminor differences in reclaim activity but nothing significant > > 4.7.0-rc3 4.7.0-rc3 > mmotm-20160615nodelru-v7r17 > Minor Faults 362486 358215 > Major Faults 1143 1113 > Swap Ins 26 0 > Swap Outs 2920 482 > DMA allocs 0 0 > DMA32 allocs 31568814 28598887 > Normal allocs 46539922 49514444 > Movable allocs 0 0 > Allocation stalls 0 0 > Direct pages scanned 0 0 > Kswapd pages scanned 40886878 40849710 > Kswapd pages reclaimed 40869923 40835207 > Direct pages reclaimed 0 0 > Kswapd efficiency 99% 99% > Kswapd velocity 11295.342 11563.344 > Direct efficiency 100% 100% > Direct velocity 0.000 0.000 > Slabs scanned 131673 126099 > Direct inode steals 57 60 > Kswapd inode steals 762 18 > > It basically shows that kswapd was active at roughly the same rate in > both kernels. There was also comparable slab scanning activity and direct > reclaim was avoided in both cases. There appears to be a large difference > in numbers of inodes reclaimed but the workload has few active inodes and > is likely a timing artifact. It's interesting to note that the node-lru > did not swap in any pages but given the low swap activity, it's unlikely > to be significant. > > stutter > ------- > > stutter simulates a simple workload. One part uses a lot of anonymous > memory, a second measures mmap latency and a third copies a large file. > The primary metric is checking for mmap latency. > > stutter > 4.7.0-rc4 4.7.0-rc4 > mmotm-20160623 nodelru-v8 > Min mmap 16.6283 ( 0.00%) 16.1394 ( 2.94%) > 1st-qrtle mmap 54.7570 ( 0.00%) 55.2975 ( -0.99%) > 2nd-qrtle mmap 57.3163 ( 0.00%) 57.5230 ( -0.36%) > 3rd-qrtle mmap 58.9976 ( 0.00%) 58.0537 ( 1.60%) > Max-90% mmap 59.7433 ( 0.00%) 58.3910 ( 2.26%) > Max-93% mmap 60.1298 ( 0.00%) 58.4801 ( 2.74%) > Max-95% mmap 73.4112 ( 0.00%) 58.5537 ( 20.24%) > Max-99% mmap 92.8542 ( 0.00%) 58.9673 ( 36.49%) > Max mmap 1440.6569 ( 0.00%) 137.6875 ( 90.44%) > Mean mmap 59.3493 ( 0.00%) 55.5153 ( 6.46%) > Best99%Mean mmap 57.2121 ( 0.00%) 55.4194 ( 3.13%) > Best95%Mean mmap 55.9113 ( 0.00%) 55.2813 ( 1.13%) > Best90%Mean mmap 55.6199 ( 0.00%) 55.1044 ( 0.93%) > Best50%Mean mmap 53.2183 ( 0.00%) 52.8330 ( 0.72%) > Best10%Mean mmap 45.9842 ( 0.00%) 42.3740 ( 7.85%) > Best5%Mean mmap 43.2256 ( 0.00%) 38.8660 ( 10.09%) > Best1%Mean mmap 32.9388 ( 0.00%) 27.7577 ( 15.73%) > > This shows a number of improvements with the worst-case outlier greatly > improved. > > Some of the vmstats are interesting > > 4.7.0-rc4 4.7.0-rc4 > mmotm-20160623nodelru-v8 > Swap Ins 163 239 > Swap Outs 0 0 > Allocation stalls 2603 0 > DMA allocs 0 0 > DMA32 allocs 618719206 1303037965 > Normal allocs 891235743 229914091 > Movable allocs 0 0 > Direct pages scanned 216787 3173 > Kswapd pages scanned 50719775 41732250 > Kswapd pages reclaimed 41541765 41731168 > Direct pages reclaimed 209159 3173 > Kswapd efficiency 81% 99% > Kswapd velocity 16859.554 14231.043 > Direct efficiency 96% 100% > Direct velocity 72.061 1.082 > Percentage direct scans 0% 0% > Zone normal velocity 8431.777 14232.125 > Zone dma32 velocity 8499.838 0.000 > Zone dma velocity 0.000 0.000 > Page writes by reclaim 6215049.000 0.000 > Page writes file 6215049 0 > Page writes anon 0 0 > Page reclaim immediate 70673 143 > Sector Reads 81940800 81489388 > Sector Writes 100158984 99161860 > Page rescued immediate 0 0 > Slabs scanned 1366954 21196 > > While this is not guaranteed in all cases, this particular test showed > a large reduction in direct reclaim activity. It's also worth noting > that no page writes were issued from reclaim context. > > This series is not without its hazards. There are at least three areas > that I'm concerned with even though I could not reproduce any problems in > that area. > > 1. Reclaim/compaction is going to be affected because the amount of reclaim is > no longer targetted at a specific zone. Compaction works on a per-zone basis > so there is no guarantee that reclaiming a few THP's worth page pages will > have a positive impact on compaction success rates. > > 2. The Slab/LRU reclaim ratio is affected because the frequency the shrinkers > are called is now different. This may or may not be a problem but if it > is, it'll be because shrinkers are not called enough and some balancing > is required. > > 3. The anon/file reclaim ratio may be affected. Pages about to be dirtied are > distributed between zones and the fair zone allocation policy used to do > something very similar for anon. The distribution is now different but not > necessarily in any way that matters but it's still worth bearing in mind.
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-07-04 12:00 +0200 |
| Message-ID | <rR8Px-419-73@gated-at.bofh.it> |
| In reply to | #1436104 |
On Mon, Jul 04, 2016 at 05:04:12PM +0900, Minchan Kim wrote: > > > How big ratio between highmem:lowmem do you think a problem? > > > > > > > That's a "how long is a piece of string" type question. The ratio does > > not matter as much as whether the workload is both under memory pressure > > and requires large amounts of lowmem pages. Even on systems with very high > > ratios, it may not be a problem if HIGHPTE is enabled. > > As well page table, pgd/kernelstack/zbud/slab and so on, every kernel > allocations wanted to mask __GFP_HIGHMEM off would be a problem in > 32bit system. > The same point applies -- it depends on the rate of these allocations, not the ratio of highmem:lowmem per se. > It also depends on that how many drivers needed lowmem only we have > in the system. > > I don't know how many such driver in the world. When I simply do grep, > I found several cases which mask __GFP_HIGHMEM off and among them, > I guess DRM might be a popular for us. However, it might be really rare > usecase among various i915 usecases. > It's also perfectly possible that such allocations are long-lived in which case they are not going to cause many skips. Hence why I cannot make a general prediction. > > > > Conceptually, moving to node LRUs should be easier to understand. The > > > > page allocator plays fewer tricks to game reclaim and reclaim behaves > > > > similarly on all nodes. > > > > > > > > The series has been tested on a 16 core UMA machine and a 2-socket 48 > > > > core NUMA machine. The UMA results are presented in most cases as the NUMA > > > > machine behaved similarly. > > > > > > I guess you would already test below with various highmem system(e.g., > > > 2:1, 3:1, 4:1 and so on). If you have, could you mind sharing it? > > > > > > > I haven't that data, the baseline distribution used doesn't even have > > 32-bit support. Even if it was, the results may not be that interesting. > > The workloads used were not necessarily going to trigger lowmem pressure > > as HIGHPTE was set on the 32-bit configs. > > That means we didn't test this on 32-bit with highmem. > No. I tested the skip logic and noticed that when forced on purpose that system CPU usage was higher but it functionally worked. > I'm not sure it's really too rare case to spend a time for testing. > In fact, I really want to test all series to our production system > which is 32bit and highmem but as we know well, most of embedded > system kernel is rather old so backporting needs lots of time and > care. However, if we miss testing in those system at the moment, > we will be suprised after 1~2 years. > It would be appreciated if it could be tested on such platforms if at all possible. Even if I did set up a 32-bit x86 system, it won't have the same allocation/reclaim profile as the platforms you are considering. > I don't know what kinds of benchmark can we can check it so I cannot > insist on it but you might know it. > One method would be to use fsmark with very large numbers of small files to force slab to require low memory. It's not representative of many real workloads unfortunately. Usually such a configuration is for checking the slab shrinker is working as expected. > Okay, do you have any idea to fix it if we see such regression report > in 32-bit system in future? Two options, neither whose complexity is justified without a "real" workload to use as a reference. 1. Long-term isolation of highmem pages when reclaim is lowmem When pages are skipped, they are immediately added back onto the LRU list. If lowmem reclaim persisted for long periods of time, the same highmem pages get continually scanned. The idea would be that lowmem keeps those pages on a separate list until a reclaim for highmem pages arrives that splices the highmem pages back onto the LRU. That would reduce the skip rate, the potential corner case is that highmem pages have to be scanned and reclaimed to free lowmem slab pages. 2. Linear scan lowmem pages if the initial LRU shrink fails This will break LRU ordering but may be preferable and faster during memory pressure than skipping LRU pages. -- Mel Gorman SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-07-06 04:00 +0200 |
| Message-ID | <rRKi6-2FP-7@gated-at.bofh.it> |
| In reply to | #1436220 |
On Mon, Jul 04, 2016 at 10:55:09AM +0100, Mel Gorman wrote: > On Mon, Jul 04, 2016 at 05:04:12PM +0900, Minchan Kim wrote: > > > > How big ratio between highmem:lowmem do you think a problem? > > > > > > > > > > That's a "how long is a piece of string" type question. The ratio does > > > not matter as much as whether the workload is both under memory pressure > > > and requires large amounts of lowmem pages. Even on systems with very high > > > ratios, it may not be a problem if HIGHPTE is enabled. > > > > As well page table, pgd/kernelstack/zbud/slab and so on, every kernel > > allocations wanted to mask __GFP_HIGHMEM off would be a problem in > > 32bit system. > > > > The same point applies -- it depends on the rate of these allocations, > not the ratio of highmem:lowmem per se. > > > It also depends on that how many drivers needed lowmem only we have > > in the system. > > > > I don't know how many such driver in the world. When I simply do grep, > > I found several cases which mask __GFP_HIGHMEM off and among them, > > I guess DRM might be a popular for us. However, it might be really rare > > usecase among various i915 usecases. > > > > It's also perfectly possible that such allocations are long-lived in which > case they are not going to cause many skips. Hence why I cannot make a > general prediction. > > > > > > Conceptually, moving to node LRUs should be easier to understand. The > > > > > page allocator plays fewer tricks to game reclaim and reclaim behaves > > > > > similarly on all nodes. > > > > > > > > > > The series has been tested on a 16 core UMA machine and a 2-socket 48 > > > > > core NUMA machine. The UMA results are presented in most cases as the NUMA > > > > > machine behaved similarly. > > > > > > > > I guess you would already test below with various highmem system(e.g., > > > > 2:1, 3:1, 4:1 and so on). If you have, could you mind sharing it? > > > > > > > > > > I haven't that data, the baseline distribution used doesn't even have > > > 32-bit support. Even if it was, the results may not be that interesting. > > > The workloads used were not necessarily going to trigger lowmem pressure > > > as HIGHPTE was set on the 32-bit configs. > > > > That means we didn't test this on 32-bit with highmem. > > > > No. I tested the skip logic and noticed that when forced on purpose that > system CPU usage was higher but it functionally worked. Yeb, it would work well functionally. I meant not functionally but performance point of view, system cpu usage and majfault rate and so on. > > > I'm not sure it's really too rare case to spend a time for testing. > > In fact, I really want to test all series to our production system > > which is 32bit and highmem but as we know well, most of embedded > > system kernel is rather old so backporting needs lots of time and > > care. However, if we miss testing in those system at the moment, > > we will be suprised after 1~2 years. > > > > It would be appreciated if it could be tested on such platforms if at all > possible. Even if I did set up a 32-bit x86 system, it won't have the same > allocation/reclaim profile as the platforms you are considering. Yeb. I just finished reviewing of all patches and found no *big* problem with my brain so my remanining homework is just testing which would find what my brain have missed. I will give the backporing to old 32-bit production kernel a shot and report if something strange happens. Thanks for great work, Mel! > > > I don't know what kinds of benchmark can we can check it so I cannot > > insist on it but you might know it. > > > > One method would be to use fsmark with very large numbers of small files > to force slab to require low memory. It's not representative of many real > workloads unfortunately. Usually such a configuration is for checking the > slab shrinker is working as expected. Thanks for the suggestion. > > > Okay, do you have any idea to fix it if we see such regression report > > in 32-bit system in future? > > Two options, neither whose complexity is justified without a "real" > workload to use as a reference. > > 1. Long-term isolation of highmem pages when reclaim is lowmem > > When pages are skipped, they are immediately added back onto the LRU > list. If lowmem reclaim persisted for long periods of time, the same > highmem pages get continually scanned. The idea would be that lowmem > keeps those pages on a separate list until a reclaim for highmem pages > arrives that splices the highmem pages back onto the LRU. > > That would reduce the skip rate, the potential corner case is that > highmem pages have to be scanned and reclaimed to free lowmem slab pages. > > 2. Linear scan lowmem pages if the initial LRU shrink fails > > This will break LRU ordering but may be preferable and faster during > memory pressure than skipping LRU pages. Okay. I guess it would be better to include this in descripion of [4/31]. > > -- > Mel Gorman > SUSE Labs > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
[toc] | [prev] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-07-04 10:10 +0200 |
| Message-ID | <rR774-3bg-27@gated-at.bofh.it> |
| In reply to | #1436104 |
On Mon, Jul 04, 2016 at 05:34:05AM +0100, Mel Gorman wrote: > On Mon, Jul 04, 2016 at 10:37:03AM +0900, Minchan Kim wrote: > > > The reason we have zone-based reclaim is that we used to have > > > large highmem zones in common configurations and it was necessary > > > to quickly find ZONE_NORMAL pages for reclaim. Today, this is much > > > less of a concern as machines with lots of memory will (or should) use > > > 64-bit kernels. Combinations of 32-bit hardware and 64-bit hardware are > > > rare. Machines that do use highmem should have relatively low highmem:lowmem > > > ratios than we worried about in the past. > > > > Hello Mel, > > > > I agree the direction absolutely. However, I have a concern on highmem > > system as you already mentioned. > > > > Embedded products still use 2 ~ 3 ratio (highmem:lowmem). > > In such system, LRU churning by skipping other zone pages frequently > > might be significant for the performance. > > > > How big ratio between highmem:lowmem do you think a problem? > > > > That's a "how long is a piece of string" type question. The ratio does > not matter as much as whether the workload is both under memory pressure > and requires large amounts of lowmem pages. Even on systems with very high > ratios, it may not be a problem if HIGHPTE is enabled. As well page table, pgd/kernelstack/zbud/slab and so on, every kernel allocations wanted to mask __GFP_HIGHMEM off would be a problem in 32bit system. It also depends on that how many drivers needed lowmem only we have in the system. I don't know how many such driver in the world. When I simply do grep, I found several cases which mask __GFP_HIGHMEM off and among them, I guess DRM might be a popular for us. However, it might be really rare usecase among various i915 usecases. > > > > > > > Conceptually, moving to node LRUs should be easier to understand. The > > > page allocator plays fewer tricks to game reclaim and reclaim behaves > > > similarly on all nodes. > > > > > > The series has been tested on a 16 core UMA machine and a 2-socket 48 > > > core NUMA machine. The UMA results are presented in most cases as the NUMA > > > machine behaved similarly. > > > > I guess you would already test below with various highmem system(e.g., > > 2:1, 3:1, 4:1 and so on). If you have, could you mind sharing it? > > > > I haven't that data, the baseline distribution used doesn't even have > 32-bit support. Even if it was, the results may not be that interesting. > The workloads used were not necessarily going to trigger lowmem pressure > as HIGHPTE was set on the 32-bit configs. That means we didn't test this on 32-bit with highmem. I'm not sure it's really too rare case to spend a time for testing. In fact, I really want to test all series to our production system which is 32bit and highmem but as we know well, most of embedded system kernel is rather old so backporting needs lots of time and care. However, if we miss testing in those system at the moment, we will be suprised after 1~2 years. I don't know what kinds of benchmark can we can check it so I cannot insist on it but you might know it. Okay, do you have any idea to fix it if we see such regression report in 32-bit system in future?
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-07-04 06:40 +0200 |
| Message-ID | <rR3PP-158-7@gated-at.bofh.it> |
| In reply to | #1436104 |
On Mon, Jul 04, 2016 at 10:37:03AM +0900, Minchan Kim wrote:
> > The reason we have zone-based reclaim is that we used to have
> > large highmem zones in common configurations and it was necessary
> > to quickly find ZONE_NORMAL pages for reclaim. Today, this is much
> > less of a concern as machines with lots of memory will (or should) use
> > 64-bit kernels. Combinations of 32-bit hardware and 64-bit hardware are
> > rare. Machines that do use highmem should have relatively low highmem:lowmem
> > ratios than we worried about in the past.
>
> Hello Mel,
>
> I agree the direction absolutely. However, I have a concern on highmem
> system as you already mentioned.
>
> Embedded products still use 2 ~ 3 ratio (highmem:lowmem).
> In such system, LRU churning by skipping other zone pages frequently
> might be significant for the performance.
>
> How big ratio between highmem:lowmem do you think a problem?
>
That's a "how long is a piece of string" type question. The ratio does
not matter as much as whether the workload is both under memory pressure
and requires large amounts of lowmem pages. Even on systems with very high
ratios, it may not be a problem if HIGHPTE is enabled.
> >
> > Conceptually, moving to node LRUs should be easier to understand. The
> > page allocator plays fewer tricks to game reclaim and reclaim behaves
> > similarly on all nodes.
> >
> > The series has been tested on a 16 core UMA machine and a 2-socket 48
> > core NUMA machine. The UMA results are presented in most cases as the NUMA
> > machine behaved similarly.
>
> I guess you would already test below with various highmem system(e.g.,
> 2:1, 3:1, 4:1 and so on). If you have, could you mind sharing it?
>
I haven't that data, the baseline distribution used doesn't even have
32-bit support. Even if it was, the results may not be that interesting.
The workloads used were not necessarily going to trigger lowmem pressure
as HIGHPTE was set on the 32-bit configs.
The skip logic has been checked and it does work. This was done during
development, by forcing the "wrong" reclaim index to use. It was
noticable in system CPU usage and in the "skip" stats. I didn't preserve
this data.
> > 4.7.0-rc4 4.7.0-rc4
> > mmotm-20160623nodelru-v8
> > Minor Faults 645838 644036
> > Major Faults 573 593
> > Swap Ins 0 0
> > Swap Outs 0 0
> > Allocation stalls 24 0
> > DMA allocs 0 0
> > DMA32 allocs 46041453 44154171
> > Normal allocs 78053072 79865782
> > Movable allocs 0 0
> > Direct pages scanned 10969 54504
> > Kswapd pages scanned 93375144 93250583
> > Kswapd pages reclaimed 93372243 93247714
> > Direct pages reclaimed 10969 54504
> > Kswapd efficiency 99% 99%
> > Kswapd velocity 13741.015 13711.950
> > Direct efficiency 100% 100%
> > Direct velocity 1.614 8.014
> > Percentage direct scans 0% 0%
> > Zone normal velocity 8641.875 13719.964
> > Zone dma32 velocity 5100.754 0.000
> > Zone dma velocity 0.000 0.000
> > Page writes by reclaim 0.000 0.000
> > Page writes file 0 0
> > Page writes anon 0 0
> > Page reclaim immediate 37 54
> >
> > kswapd activity was roughly comparable. There were differences in direct
> > reclaim activity but negligible in the context of the overall workload
> > (velocity of 8 pages per second with the patches applied, 1.6 pages per
> > second in the baseline kernel).
>
> Hmm, nodelru's allocation stall is zero above but how does direct page
> scanning/reclaimed happens?
>
Good spot, it's because I used the wrong comparison script -- one that
doesn't understand the different skip and allocation stats and I was
looking primarily at the scanning activity. This is a correct version
4.7.0-rc4 4.7.0-rc4
mmotm-20160623nodelru-v8r26
Minor Faults 645838 643815
Major Faults 573 493
Swap Ins 0 0
Swap Outs 0 0
DMA allocs 0 0
DMA32 allocs 46041453 44174923
Normal allocs 78053072 79816443
Movable allocs 0 0
Allocation stalls 24 31
Stall zone DMA 0 0
Stall zone DMA32 0 0
Stall zone Normal 0 1
Stall zone HighMem 0 0
Stall zone Movable 0 30
Direct pages scanned 10969 14198
Kswapd pages scanned 93375144 93252534
Kswapd pages reclaimed 93372243 93249856
Direct pages reclaimed 10969 14198
Kswapd efficiency 99% 99%
Kswapd velocity 13741.015 13742.771
Direct efficiency 100% 100%
Direct velocity 1.614 2.092
Percentage direct scans 0% 0%
Page writes by reclaim 0 0
Page writes file 0 0
Page writes anon 0 0
Page reclaim immediate 37 29
The points about kswapd and direct reclaim activity still hold.
> Above, DMA32 allocs in nodelru is almost same but zone dma32 velocity
> is zero. What does it means?
>
It's a consequence of using the wrong script when cutting and pasting
the final data. With node-lru, "zone dma32 velocity" is meaningless and
the reporting script no longer includes it.
--
Mel Gorman
SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-07-05 03:30 +0200 |
| Subject | Re: [PATCH 03/31] mm, vmscan: move LRU lists to node |
| Message-ID | <rRnlw-4AT-3@gated-at.bofh.it> |
| In reply to | #1435600 |
On Fri, Jul 01, 2016 at 09:01:11PM +0100, Mel Gorman wrote:
> This moves the LRU lists from the zone to the node and related data such
> as counters, tracing, congestion tracking and writeback tracking.
> Unfortunately, due to reclaim and compaction retry logic, it is necessary
> to account for the number of LRU pages on both zone and node logic. Most
> reclaim logic is based on the node counters but the retry logic uses the
> zone counters which do not distinguish inactive and inactive sizes. It
active
> would be possible to leave the LRU counters on a per-zone basis but it's a
> heavier calculation across multiple cache lines that is much more frequent
> than the retry checks.
>
> Other than the LRU counters, this is mostly a mechanical patch but note
> that it introduces a number of anomalies. For example, the scans are
> per-zone but using per-node counters. We also mark a node as congested
> when a zone is congested. This causes weird problems that are fixed later
> but is easier to review.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> Acked-by: Vlastimil Babka <vbabka@suse.cz>
> ---
> arch/tile/mm/pgtable.c | 8 +-
> drivers/base/node.c | 19 +--
> drivers/staging/android/lowmemorykiller.c | 8 +-
> include/linux/backing-dev.h | 2 +-
> include/linux/memcontrol.h | 16 +--
> include/linux/mm_inline.h | 21 ++-
> include/linux/mmzone.h | 69 +++++----
> include/linux/swap.h | 1 +
> include/linux/vm_event_item.h | 10 +-
> include/linux/vmstat.h | 17 +++
> include/trace/events/vmscan.h | 12 +-
> kernel/power/snapshot.c | 10 +-
> mm/backing-dev.c | 15 +-
> mm/compaction.c | 18 +--
> mm/huge_memory.c | 2 +-
> mm/internal.h | 2 +-
> mm/khugepaged.c | 4 +-
> mm/memcontrol.c | 17 +--
> mm/memory-failure.c | 4 +-
> mm/memory_hotplug.c | 2 +-
> mm/mempolicy.c | 2 +-
> mm/migrate.c | 21 +--
> mm/mlock.c | 2 +-
> mm/page-writeback.c | 8 +-
> mm/page_alloc.c | 70 ++++-----
> mm/swap.c | 50 +++----
> mm/vmscan.c | 226 +++++++++++++++++-------------
> mm/vmstat.c | 47 ++++---
> mm/workingset.c | 4 +-
> 29 files changed, 387 insertions(+), 300 deletions(-)
>
> diff --git a/arch/tile/mm/pgtable.c b/arch/tile/mm/pgtable.c
> index c4d5bf841a7f..9e389213580d 100644
> --- a/arch/tile/mm/pgtable.c
> +++ b/arch/tile/mm/pgtable.c
> @@ -45,10 +45,10 @@ void show_mem(unsigned int filter)
> struct zone *zone;
>
> pr_err("Active:%lu inactive:%lu dirty:%lu writeback:%lu unstable:%lu free:%lu\n slab:%lu mapped:%lu pagetables:%lu bounce:%lu pagecache:%lu swap:%lu\n",
> - (global_page_state(NR_ACTIVE_ANON) +
> - global_page_state(NR_ACTIVE_FILE)),
> - (global_page_state(NR_INACTIVE_ANON) +
> - global_page_state(NR_INACTIVE_FILE)),
> + (global_node_page_state(NR_ACTIVE_ANON) +
> + global_node_page_state(NR_ACTIVE_FILE)),
> + (global_node_page_state(NR_INACTIVE_ANON) +
> + global_node_page_state(NR_INACTIVE_FILE)),
> global_page_state(NR_FILE_DIRTY),
> global_page_state(NR_WRITEBACK),
> global_page_state(NR_UNSTABLE_NFS),
> diff --git a/drivers/base/node.c b/drivers/base/node.c
> index 92d8e090c5b3..b7f01a4a642d 100644
> --- a/drivers/base/node.c
> +++ b/drivers/base/node.c
> @@ -56,6 +56,7 @@ static ssize_t node_read_meminfo(struct device *dev,
> {
> int n;
> int nid = dev->id;
> + struct pglist_data *pgdat = NODE_DATA(nid);
> struct sysinfo i;
>
> si_meminfo_node(&i, nid);
> @@ -74,15 +75,15 @@ static ssize_t node_read_meminfo(struct device *dev,
> nid, K(i.totalram),
> nid, K(i.freeram),
> nid, K(i.totalram - i.freeram),
> - nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_ANON) +
> - sum_zone_node_page_state(nid, NR_ACTIVE_FILE)),
> - nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_ANON) +
> - sum_zone_node_page_state(nid, NR_INACTIVE_FILE)),
> - nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_ANON)),
> - nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_ANON)),
> - nid, K(sum_zone_node_page_state(nid, NR_ACTIVE_FILE)),
> - nid, K(sum_zone_node_page_state(nid, NR_INACTIVE_FILE)),
> - nid, K(sum_zone_node_page_state(nid, NR_UNEVICTABLE)),
> + nid, K(node_page_state(pgdat, NR_ACTIVE_ANON) +
> + node_page_state(pgdat, NR_ACTIVE_FILE)),
> + nid, K(node_page_state(pgdat, NR_INACTIVE_ANON) +
> + node_page_state(pgdat, NR_INACTIVE_FILE)),
> + nid, K(node_page_state(pgdat, NR_ACTIVE_ANON)),
> + nid, K(node_page_state(pgdat, NR_INACTIVE_ANON)),
> + nid, K(node_page_state(pgdat, NR_ACTIVE_FILE)),
> + nid, K(node_page_state(pgdat, NR_INACTIVE_FILE)),
> + nid, K(node_page_state(pgdat, NR_UNEVICTABLE)),
> nid, K(sum_zone_node_page_state(nid, NR_MLOCK)));
>
> #ifdef CONFIG_HIGHMEM
> diff --git a/drivers/staging/android/lowmemorykiller.c b/drivers/staging/android/lowmemorykiller.c
> index 24d2745e9437..93dbcc38eb0f 100644
> --- a/drivers/staging/android/lowmemorykiller.c
> +++ b/drivers/staging/android/lowmemorykiller.c
> @@ -72,10 +72,10 @@ static unsigned long lowmem_deathpending_timeout;
> static unsigned long lowmem_count(struct shrinker *s,
> struct shrink_control *sc)
> {
> - return global_page_state(NR_ACTIVE_ANON) +
> - global_page_state(NR_ACTIVE_FILE) +
> - global_page_state(NR_INACTIVE_ANON) +
> - global_page_state(NR_INACTIVE_FILE);
> + return global_node_page_state(NR_ACTIVE_ANON) +
> + global_node_page_state(NR_ACTIVE_FILE) +
> + global_node_page_state(NR_INACTIVE_ANON) +
> + global_node_page_state(NR_INACTIVE_FILE);
> }
>
> static unsigned long lowmem_scan(struct shrinker *s, struct shrink_control *sc)
> diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
> index c82794f20110..491a91717788 100644
> --- a/include/linux/backing-dev.h
> +++ b/include/linux/backing-dev.h
> @@ -197,7 +197,7 @@ static inline int wb_congested(struct bdi_writeback *wb, int cong_bits)
> }
>
> long congestion_wait(int sync, long timeout);
> -long wait_iff_congested(struct zone *zone, int sync, long timeout);
> +long wait_iff_congested(struct pglist_data *pgdat, int sync, long timeout);
> int pdflush_proc_obsolete(struct ctl_table *table, int write,
> void __user *buffer, size_t *lenp, loff_t *ppos);
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index 104efa6874db..1927dcb6921e 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -340,7 +340,7 @@ static inline struct lruvec *mem_cgroup_zone_lruvec(struct zone *zone,
> struct lruvec *lruvec;
>
> if (mem_cgroup_disabled()) {
> - lruvec = &zone->lruvec;
> + lruvec = zone_lruvec(zone);
> goto out;
> }
>
> @@ -352,12 +352,12 @@ static inline struct lruvec *mem_cgroup_zone_lruvec(struct zone *zone,
> * we have to be prepared to initialize lruvec->zone here;
lruvec->pgdat
> * and if offlined then reonlined, we need to reinitialize it.
> */
> - if (unlikely(lruvec->zone != zone))
> - lruvec->zone = zone;
> + if (unlikely(lruvec->pgdat != zone->zone_pgdat))
> + lruvec->pgdat = zone->zone_pgdat;
> return lruvec;
> }
>
> -struct lruvec *mem_cgroup_page_lruvec(struct page *, struct zone *);
> +struct lruvec *mem_cgroup_page_lruvec(struct page *, struct pglist_data *);
>
> bool task_in_mem_cgroup(struct task_struct *task, struct mem_cgroup *memcg);
> struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p);
> @@ -438,7 +438,7 @@ static inline bool mem_cgroup_online(struct mem_cgroup *memcg)
> int mem_cgroup_select_victim_node(struct mem_cgroup *memcg);
>
> void mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru,
> - int nr_pages);
> + enum zone_type zid, int nr_pages);
>
> unsigned long mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg,
> int nid, unsigned int lru_mask);
> @@ -613,13 +613,13 @@ static inline void mem_cgroup_migrate(struct page *old, struct page *new)
> static inline struct lruvec *mem_cgroup_zone_lruvec(struct zone *zone,
> struct mem_cgroup *memcg)
> {
> - return &zone->lruvec;
> + return zone_lruvec(zone);
> }
>
> static inline struct lruvec *mem_cgroup_page_lruvec(struct page *page,
> - struct zone *zone)
> + struct pglist_data *pgdat)
> {
> - return &zone->lruvec;
> + return &pgdat->lruvec;
> }
>
> static inline bool mm_match_cgroup(struct mm_struct *mm,
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index 5bd29ba4f174..9aadcc781857 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -23,25 +23,32 @@ static inline int page_is_file_cache(struct page *page)
> }
>
> static __always_inline void __update_lru_size(struct lruvec *lruvec,
> - enum lru_list lru, int nr_pages)
> + enum lru_list lru, enum zone_type zid,
> + int nr_pages)
> {
> - __mod_zone_page_state(lruvec_zone(lruvec), NR_LRU_BASE + lru, nr_pages);
> + struct pglist_data *pgdat = lruvec_pgdat(lruvec);
> +
> + __mod_node_page_state(pgdat, NR_LRU_BASE + lru, nr_pages);
> + __mod_zone_page_state(&pgdat->node_zones[zid],
> + NR_ZONE_LRU_BASE + !!is_file_lru(lru),
> + nr_pages);
> }
>
> static __always_inline void update_lru_size(struct lruvec *lruvec,
> - enum lru_list lru, int nr_pages)
> + enum lru_list lru, enum zone_type zid,
> + int nr_pages)
> {
> #ifdef CONFIG_MEMCG
> - mem_cgroup_update_lru_size(lruvec, lru, nr_pages);
> + mem_cgroup_update_lru_size(lruvec, lru, zid, nr_pages);
> #else
> - __update_lru_size(lruvec, lru, nr_pages);
> + __update_lru_size(lruvec, lru, zid, nr_pages);
> #endif
> }
>
> static __always_inline void add_page_to_lru_list(struct page *page,
> struct lruvec *lruvec, enum lru_list lru)
> {
> - update_lru_size(lruvec, lru, hpage_nr_pages(page));
> + update_lru_size(lruvec, lru, page_zonenum(page), hpage_nr_pages(page));
> list_add(&page->lru, &lruvec->lists[lru]);
> }
>
> @@ -49,7 +56,7 @@ static __always_inline void del_page_from_lru_list(struct page *page,
> struct lruvec *lruvec, enum lru_list lru)
> {
> list_del(&page->lru);
> - update_lru_size(lruvec, lru, -hpage_nr_pages(page));
> + update_lru_size(lruvec, lru, page_zonenum(page), -hpage_nr_pages(page));
> }
>
> /**
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 2d5087e3c034..258c20758e80 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -111,12 +111,9 @@ enum zone_stat_item {
> /* First 128 byte cacheline (assuming 64 bit words) */
> NR_FREE_PAGES,
> NR_ALLOC_BATCH,
> - NR_LRU_BASE,
> - NR_INACTIVE_ANON = NR_LRU_BASE, /* must match order of LRU_[IN]ACTIVE */
> - NR_ACTIVE_ANON, /* " " " " " */
> - NR_INACTIVE_FILE, /* " " " " " */
> - NR_ACTIVE_FILE, /* " " " " " */
> - NR_UNEVICTABLE, /* " " " " " */
> + NR_ZONE_LRU_BASE, /* Used only for compaction and reclaim retry */
> + NR_ZONE_LRU_ANON = NR_ZONE_LRU_BASE,
> + NR_ZONE_LRU_FILE,
> NR_MLOCK, /* mlock()ed pages found and moved off LRU */
> NR_ANON_PAGES, /* Mapped anonymous pages */
> NR_FILE_MAPPED, /* pagecache pages mapped into pagetables.
> @@ -134,12 +131,9 @@ enum zone_stat_item {
> NR_VMSCAN_WRITE,
> NR_VMSCAN_IMMEDIATE, /* Prioritise for reclaim when writeback ends */
> NR_WRITEBACK_TEMP, /* Writeback using temporary buffers */
> - NR_ISOLATED_ANON, /* Temporary isolated pages from anon lru */
> - NR_ISOLATED_FILE, /* Temporary isolated pages from file lru */
> NR_SHMEM, /* shmem pages (included tmpfs/GEM pages) */
> NR_DIRTIED, /* page dirtyings since bootup */
> NR_WRITTEN, /* page writings since bootup */
> - NR_PAGES_SCANNED, /* pages scanned since last reclaim */
> #if IS_ENABLED(CONFIG_ZSMALLOC)
> NR_ZSPAGES, /* allocated in zsmalloc */
> #endif
> @@ -161,6 +155,15 @@ enum zone_stat_item {
> NR_VM_ZONE_STAT_ITEMS };
>
> enum node_stat_item {
> + NR_LRU_BASE,
> + NR_INACTIVE_ANON = NR_LRU_BASE, /* must match order of LRU_[IN]ACTIVE */
> + NR_ACTIVE_ANON, /* " " " " " */
> + NR_INACTIVE_FILE, /* " " " " " */
> + NR_ACTIVE_FILE, /* " " " " " */
> + NR_UNEVICTABLE, /* " " " " " */
> + NR_ISOLATED_ANON, /* Temporary isolated pages from anon lru */
> + NR_ISOLATED_FILE, /* Temporary isolated pages from file lru */
> + NR_PAGES_SCANNED, /* pages scanned since last reclaim */
> NR_VM_NODE_STAT_ITEMS
> };
>
> @@ -219,7 +222,7 @@ struct lruvec {
> /* Evictions & activations on the inactive file list */
> atomic_long_t inactive_age;
> #ifdef CONFIG_MEMCG
> - struct zone *zone;
> + struct pglist_data *pgdat;
> #endif
> };
>
> @@ -357,13 +360,6 @@ struct zone {
> #ifdef CONFIG_NUMA
> int node;
> #endif
> -
> - /*
> - * The target ratio of ACTIVE_ANON to INACTIVE_ANON pages on
> - * this zone's LRU. Maintained by the pageout code.
> - */
> - unsigned int inactive_ratio;
> -
> struct pglist_data *zone_pgdat;
> struct per_cpu_pageset __percpu *pageset;
>
> @@ -495,9 +491,6 @@ struct zone {
>
> /* Write-intensive fields used by page reclaim */
trivial:
We moved lru_lock and lruvec to pgdat so I'm not sure we need ZONE_PADDING,
still.
>
> - /* Fields commonly accessed by the page reclaim scanner */
> - struct lruvec lruvec;
> -
> /*
> * When free pages are below this point, additional steps are taken
> * when reading the number of free pages to avoid per-cpu counter
> @@ -537,17 +530,20 @@ struct zone {
>
> enum zone_flags {
> ZONE_RECLAIM_LOCKED, /* prevents concurrent reclaim */
> - ZONE_CONGESTED, /* zone has many dirty pages backed by
> + ZONE_FAIR_DEPLETED, /* fair zone policy batch depleted */
> +};
> +
> +enum pgdat_flags {
> + PGDAT_CONGESTED, /* zone has many dirty pages backed by
node or pgdat, whatever.
> * a congested BDI
> */
> - ZONE_DIRTY, /* reclaim scanning has recently found
> + PGDAT_DIRTY, /* reclaim scanning has recently found
> * many dirty file pages at the tail
> * of the LRU.
> */
> - ZONE_WRITEBACK, /* reclaim scanning has recently found
> + PGDAT_WRITEBACK, /* reclaim scanning has recently found
> * many pages under writeback
> */
> - ZONE_FAIR_DEPLETED, /* fair zone policy batch depleted */
> };
>
> static inline unsigned long zone_end_pfn(const struct zone *zone)
> @@ -701,12 +697,26 @@ typedef struct pglist_data {
> unsigned long first_deferred_pfn;
> #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
>
> +
Unnecessary change.
> #ifdef CONFIG_TRANSPARENT_HUGEPAGE
> spinlock_t split_queue_lock;
> struct list_head split_queue;
> unsigned long split_queue_len;
> #endif
>
> + /* Fields commonly accessed by the page reclaim scanner */
> + struct lruvec lruvec;
> +
> + /*
> + * The target ratio of ACTIVE_ANON to INACTIVE_ANON pages on
> + * this node's LRU. Maintained by the pageout code.
> + */
> + unsigned int inactive_ratio;
> +
> + unsigned long flags;
> +
> + ZONE_PADDING(_pad2_)
> +
> /* Per-node vmstats */
> struct per_cpu_nodestat __percpu *per_cpu_nodestats;
> atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
> @@ -728,6 +738,11 @@ static inline spinlock_t *zone_lru_lock(struct zone *zone)
> return &zone->zone_pgdat->lru_lock;
> }
>
> +static inline struct lruvec *zone_lruvec(struct zone *zone)
> +{
> + return &zone->zone_pgdat->lruvec;
> +}
> +
> static inline unsigned long pgdat_end_pfn(pg_data_t *pgdat)
> {
> return pgdat->node_start_pfn + pgdat->node_spanned_pages;
> @@ -779,12 +794,12 @@ extern int init_currently_empty_zone(struct zone *zone, unsigned long start_pfn,
>
> extern void lruvec_init(struct lruvec *lruvec);
>
> -static inline struct zone *lruvec_zone(struct lruvec *lruvec)
> +static inline struct pglist_data *lruvec_pgdat(struct lruvec *lruvec)
> {
> #ifdef CONFIG_MEMCG
> - return lruvec->zone;
> + return lruvec->pgdat;
> #else
> - return container_of(lruvec, struct zone, lruvec);
> + return container_of(lruvec, struct pglist_data, lruvec);
> #endif
> }
>
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 0af2bb2028fd..c82f916008b7 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -317,6 +317,7 @@ extern void lru_cache_add_active_or_unevictable(struct page *page,
>
> /* linux/mm/vmscan.c */
> extern unsigned long zone_reclaimable_pages(struct zone *zone);
> +extern unsigned long pgdat_reclaimable_pages(struct pglist_data *pgdat);
> extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
> gfp_t gfp_mask, nodemask_t *mask);
> extern int __isolate_lru_page(struct page *page, isolate_mode_t mode);
> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
> index 42604173f122..1798ff542517 100644
> --- a/include/linux/vm_event_item.h
> +++ b/include/linux/vm_event_item.h
> @@ -26,11 +26,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
> PGFREE, PGACTIVATE, PGDEACTIVATE,
> PGFAULT, PGMAJFAULT,
> PGLAZYFREED,
> - FOR_ALL_ZONES(PGREFILL),
> - FOR_ALL_ZONES(PGSTEAL_KSWAPD),
> - FOR_ALL_ZONES(PGSTEAL_DIRECT),
> - FOR_ALL_ZONES(PGSCAN_KSWAPD),
> - FOR_ALL_ZONES(PGSCAN_DIRECT),
> + PGREFILL,
> + PGSTEAL_KSWAPD,
> + PGSTEAL_DIRECT,
> + PGSCAN_KSWAPD,
> + PGSCAN_DIRECT,
> PGSCAN_DIRECT_THROTTLE,
> #ifdef CONFIG_NUMA
> PGSCAN_ZONE_RECLAIM_FAILED,
> diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
> index d1744aa3ab9c..ced0c3e9da88 100644
> --- a/include/linux/vmstat.h
> +++ b/include/linux/vmstat.h
> @@ -178,6 +178,23 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone,
> return x;
> }
>
> +static inline unsigned long node_page_state_snapshot(pg_data_t *pgdat,
> + enum zone_stat_item item)
enum node_stat_item
> +{
> + long x = atomic_long_read(&pgdat->vm_stat[item]);
> +
> +#ifdef CONFIG_SMP
> + int cpu;
> + for_each_online_cpu(cpu)
> + x += per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->vm_node_stat_diff[item];
> +
> + if (x < 0)
> + x = 0;
> +#endif
> + return x;
> +}
> +
> +
> #ifdef CONFIG_NUMA
> extern unsigned long sum_zone_node_page_state(int node,
> enum zone_stat_item item);
<snip>
> @@ -1147,9 +1147,9 @@ static void free_one_page(struct zone *zone,
> {
> unsigned long nr_scanned;
> spin_lock(&zone->lock);
> - nr_scanned = zone_page_state(zone, NR_PAGES_SCANNED);
> + nr_scanned = node_page_state(zone->zone_pgdat, NR_PAGES_SCANNED);
> if (nr_scanned)
> - __mod_zone_page_state(zone, NR_PAGES_SCANNED, -nr_scanned);
> + __mod_node_page_state(zone->zone_pgdat, NR_PAGES_SCANNED, -nr_scanned);
>
> if (unlikely(has_isolate_pageblock(zone) ||
> is_migrate_isolate(migratetype))) {
> @@ -3526,7 +3526,7 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
>
> available = reclaimable = zone_reclaimable_pages(zone);
> available -= DIV_ROUND_UP(no_progress_loops * available,
> - MAX_RECLAIM_RETRIES);
> + MAX_RECLAIM_RETRIES);
Unnecessary change.
> available += zone_page_state_snapshot(zone, NR_FREE_PAGES);
>
> /*
> @@ -4331,6 +4331,7 @@ void show_free_areas(unsigned int filter)
<snip.
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index e7ffcd259cc4..86a523a761c9 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -191,26 +191,42 @@ static bool sane_reclaim(struct scan_control *sc)
> }
> #endif
>
> +/*
> + * This misses isolated pages which are not accounted for to save counters.
> + * As the data only determines if reclaim or compaction continues, it is
> + * not expected that isolated pages will be a dominating factor.
When I read below commit, one of the reason it was introduced is whether we
should continue to reclaim page or not.
At that time, several people wanted it by my guessing [suggested|acked]-by
so I think we should notice it to them.
Michal?
[9f6c399ddc36, consider isolated pages in zone_reclaimable_pages],
> + */
> unsigned long zone_reclaimable_pages(struct zone *zone)
> {
> unsigned long nr;
>
> - nr = zone_page_state_snapshot(zone, NR_ACTIVE_FILE) +
> - zone_page_state_snapshot(zone, NR_INACTIVE_FILE) +
> - zone_page_state_snapshot(zone, NR_ISOLATED_FILE);
> + nr = zone_page_state_snapshot(zone, NR_ZONE_LRU_FILE);
> + if (get_nr_swap_pages() > 0)
> + nr += zone_page_state_snapshot(zone, NR_ZONE_LRU_ANON);
> +
> + return nr;
> +}
> +
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-07-05 12:20 +0200 |
| Subject | Re: [PATCH 03/31] mm, vmscan: move LRU lists to node |
| Message-ID | <rRvCq-1tJ-37@gated-at.bofh.it> |
| In reply to | #1436713 |
On Tue, Jul 05, 2016 at 10:19:57AM +0900, Minchan Kim wrote:
> On Fri, Jul 01, 2016 at 09:01:11PM +0100, Mel Gorman wrote:
> > This moves the LRU lists from the zone to the node and related data such
> > as counters, tracing, congestion tracking and writeback tracking.
> > Unfortunately, due to reclaim and compaction retry logic, it is necessary
> > to account for the number of LRU pages on both zone and node logic. Most
> > reclaim logic is based on the node counters but the retry logic uses the
> > zone counters which do not distinguish inactive and inactive sizes. It
>
> active
>
Fixed.
> > @@ -352,12 +352,12 @@ static inline struct lruvec *mem_cgroup_zone_lruvec(struct zone *zone,
> > * we have to be prepared to initialize lruvec->zone here;
>
> lruvec->pgdat
>
Fixed.
> > @@ -357,13 +360,6 @@ struct zone {
> > #ifdef CONFIG_NUMA
> > int node;
> > #endif
> > -
> > - /*
> > - * The target ratio of ACTIVE_ANON to INACTIVE_ANON pages on
> > - * this zone's LRU. Maintained by the pageout code.
> > - */
> > - unsigned int inactive_ratio;
> > -
> > struct pglist_data *zone_pgdat;
> > struct per_cpu_pageset __percpu *pageset;
> >
> > @@ -495,9 +491,6 @@ struct zone {
> >
> > /* Write-intensive fields used by page reclaim */
>
> trivial:
> We moved lru_lock and lruvec to pgdat so I'm not sure we need ZONE_PADDING,
> still.
>
It still separates the page allocator structures from compaction and
vmstats. The comment is misleading so I added a patch to clarify what
the padding is doing.
> >
> > - /* Fields commonly accessed by the page reclaim scanner */
> > - struct lruvec lruvec;
> > -
> > /*
> > * When free pages are below this point, additional steps are taken
> > * when reading the number of free pages to avoid per-cpu counter
> > @@ -537,17 +530,20 @@ struct zone {
> >
> > enum zone_flags {
> > ZONE_RECLAIM_LOCKED, /* prevents concurrent reclaim */
> > - ZONE_CONGESTED, /* zone has many dirty pages backed by
> > + ZONE_FAIR_DEPLETED, /* fair zone policy batch depleted */
> > +};
> > +
>
> > +enum pgdat_flags {
> > + PGDAT_CONGESTED, /* zone has many dirty pages backed by
>
> node or pgdat, whatever.
>
Fixed.
> > * a congested BDI
> > */
> > - ZONE_DIRTY, /* reclaim scanning has recently found
> > + PGDAT_DIRTY, /* reclaim scanning has recently found
> > * many dirty file pages at the tail
> > * of the LRU.
> > */
> > - ZONE_WRITEBACK, /* reclaim scanning has recently found
> > + PGDAT_WRITEBACK, /* reclaim scanning has recently found
> > * many pages under writeback
> > */
> > - ZONE_FAIR_DEPLETED, /* fair zone policy batch depleted */
> > };
> >
> > static inline unsigned long zone_end_pfn(const struct zone *zone)
> > @@ -701,12 +697,26 @@ typedef struct pglist_data {
> > unsigned long first_deferred_pfn;
> > #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
> >
> > +
>
> Unnecessary change.
>
Fixed.
> > diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
> > index d1744aa3ab9c..ced0c3e9da88 100644
> > --- a/include/linux/vmstat.h
> > +++ b/include/linux/vmstat.h
> > @@ -178,6 +178,23 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone,
> > return x;
> > }
> >
> > +static inline unsigned long node_page_state_snapshot(pg_data_t *pgdat,
> > + enum zone_stat_item item)
>
> enum node_stat_item
>
Fixed
> > @@ -1147,9 +1147,9 @@ static void free_one_page(struct zone *zone,
> > {
> > unsigned long nr_scanned;
> > spin_lock(&zone->lock);
> > - nr_scanned = zone_page_state(zone, NR_PAGES_SCANNED);
> > + nr_scanned = node_page_state(zone->zone_pgdat, NR_PAGES_SCANNED);
> > if (nr_scanned)
> > - __mod_zone_page_state(zone, NR_PAGES_SCANNED, -nr_scanned);
> > + __mod_node_page_state(zone->zone_pgdat, NR_PAGES_SCANNED, -nr_scanned);
> >
> > if (unlikely(has_isolate_pageblock(zone) ||
> > is_migrate_isolate(migratetype))) {
> > @@ -3526,7 +3526,7 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
> >
> > available = reclaimable = zone_reclaimable_pages(zone);
> > available -= DIV_ROUND_UP(no_progress_loops * available,
> > - MAX_RECLAIM_RETRIES);
> > + MAX_RECLAIM_RETRIES);
>
> Unnecessary change.
>
Fixed.
> > available += zone_page_state_snapshot(zone, NR_FREE_PAGES);
> >
> > /*
> > @@ -4331,6 +4331,7 @@ void show_free_areas(unsigned int filter)
>
> <snip.
>
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index e7ffcd259cc4..86a523a761c9 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -191,26 +191,42 @@ static bool sane_reclaim(struct scan_control *sc)
> > }
> > #endif
> >
> > +/*
> > + * This misses isolated pages which are not accounted for to save counters.
> > + * As the data only determines if reclaim or compaction continues, it is
> > + * not expected that isolated pages will be a dominating factor.
>
> When I read below commit, one of the reason it was introduced is whether we
> should continue to reclaim page or not.
> At that time, several people wanted it by my guessing [suggested|acked]-by
> so I think we should notice it to them.
>
> Michal?
>
Ultimately this gets fixed up at the end of the series when
zone_reclaimable_pages gets removed again to avoid double accounting.
--
Mel Gorman
SUSE Labs
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web