Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1379561 > unrolled thread
| Started by | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| First post | 2016-04-15 11:00 +0200 |
| Last post | 2016-04-26 13:50 +0200 |
| Articles | 20 on this page of 70 — 4 participants |
Back to article view | Back to linux.kernel
[PATCH 00/28] Optimise page alloc/free fast paths v3 Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:00 +0200
[PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:00 +0200
Re: [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 11:40 +0200
Re: [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 12:40 +0200
Re: [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:30 +0200
[PATCH 21/28] mm, page_alloc: Avoid looking up the first zone in a zonelist twice Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 21/28] mm, page_alloc: Avoid looking up the first zone in a zonelist twice Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 19:50 +0200
[PATCH 04/28] mm, page_alloc: Inline zone_statistics Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 04/28] mm, page_alloc: Inline zone_statistics Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 13:20 +0200
[PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 15:50 +0200
Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 17:00 +0200
Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 17:20 +0200
Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 18:30 +0200
[PATCH 19/28] mm, page_alloc: Reduce cost of fair zone allocation policy retry Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 22/28] mm, page_alloc: Remove field from alloc_context Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 14/28] mm, page_alloc: Simplify last cpupid reset Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 14/28] mm, page_alloc: Simplify last cpupid reset Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 15:40 +0200
[PATCH 20/28] mm, page_alloc: Shortcut watermark checks for order-0 pages Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 16/28] mm, page_alloc: Move __GFP_HARDWALL modifications out of the fastpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 16/28] mm, page_alloc: Move __GFP_HARDWALL modifications out of the fastpath Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 16:20 +0200
[PATCH 17/28] mm, page_alloc: Check once if a zone has isolated pageblocks Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 17/28] mm, page_alloc: Check once if a zone has isolated pageblocks Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 16:30 +0200
[PATCH 18/28] mm, page_alloc: Shorten the page allocator fast path Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 18/28] mm, page_alloc: Shorten the page allocator fast path Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 17:30 +0200
[PATCH 26/28] cpuset: use static key better and convert to new API Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
Re: [PATCH 26/28] cpuset: use static key better and convert to new API Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 22:00 +0200
[PATCH 27/28] mm, page_alloc: Defer debugging checks of freed pages until a PCP drain Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
Re: [PATCH 27/28] mm, page_alloc: Defer debugging checks of freed pages until a PCP drain Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 14:00 +0200
[PATCH 2/3] mm, page_alloc: pull out side effects from free_pages_check Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 14:10 +0200
Re: [PATCH 2/3] mm, page_alloc: pull out side effects from free_pages_check Mel Gorman <mgorman@techsingularity.net> - 2016-04-27 14:50 +0200
Re: [PATCH 2/3] mm, page_alloc: pull out side effects from free_pages_check Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 15:10 +0200
[PATCH 3/3] mm, page_alloc: don't duplicate code in free_pcp_prepare Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 14:10 +0200
[PATCH 1/3] mm, page_alloc: un-inline the bad part of free_pages_check Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 14:10 +0200
Re: [PATCH 1/3] mm, page_alloc: un-inline the bad part of free_pages_check Mel Gorman <mgorman@techsingularity.net> - 2016-04-27 14:40 +0200
Re: [PATCH 1/3] mm, page_alloc: un-inline the bad part of free_pages_check Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 15:00 +0200
[PATCH 25/28] mm, page_alloc: Inline pageblock lookup in page free fast paths Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
[PATCH 24/28] mm, page_alloc: Remove unnecessary variable from free_pcppages_bulk Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
[PATCH 23/28] mm, page_alloc: Check multiple page fields with a single branch Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
Re: [PATCH 23/28] mm, page_alloc: Check multiple page fields with a single branch Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 20:50 +0200
Re: [PATCH 23/28] mm, page_alloc: Check multiple page fields with a single branch Mel Gorman <mgorman@techsingularity.net> - 2016-04-27 12:10 +0200
[PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
Re: [PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 16:10 +0200
Re: [PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP Mel Gorman <mgorman@techsingularity.net> - 2016-04-27 17:40 +0200
Re: [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 14:10 +0200
Re: [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 15:10 +0200
Re: [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Andrew Morton <akpm@linux-foundation.org> - 2016-04-26 21:20 +0200
[PATCH 07/28] mm, page_alloc: Avoid unnecessary zone lookups during pageblock operations Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 07/28] mm, page_alloc: Avoid unnecessary zone lookups during pageblock operations Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:30 +0200
[PATCH 02/28] mm, page_alloc: Use new PageAnonHead helper in the free page fast path Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 02/28] mm, page_alloc: Use new PageAnonHead helper in the free page fast path Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 12:00 +0200
[PATCH 11/28] mm, page_alloc: Remove unnecessary initialisation in get_page_from_freelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 06/28] mm, page_alloc: Use __dec_zone_state for order-0 page allocation Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 06/28] mm, page_alloc: Use __dec_zone_state for order-0 page allocation Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:30 +0200
[PATCH 09/28] mm, page_alloc: Convert nr_fair_skipped to bool Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 09/28] mm, page_alloc: Convert nr_fair_skipped to bool Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:40 +0200
[PATCH 08/28] mm, page_alloc: Convert alloc_flags to unsigned Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 03/28] mm, page_alloc: Reduce branches in zone_statistics Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 03/28] mm, page_alloc: Reduce branches in zone_statistics Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 13:20 +0200
[PATCH 05/28] mm, page_alloc: Inline the fast path of the zonelist iterator Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 05/28] mm, page_alloc: Inline the fast path of the zonelist iterator Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 17:00 +0200
Re: [PATCH 05/28] mm, page_alloc: Inline the fast path of the zonelist iterator Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 12:40 +0200
Re: [PATCH 05/28] mm, page_alloc: Inline the fast path of the zonelist iterator Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:10 +0200
[PATCH 10/28] mm, page_alloc: Remove unnecessary local variable in get_page_from_freelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 10/28] mm, page_alloc: Remove unnecessary local variable in get_page_from_freelist Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:40 +0200
Re: [PATCH 00/28] Optimise page alloc/free fast paths v3 Jesper Dangaard Brouer <brouer@redhat.com> - 2016-04-15 14:50 +0200
Re: [PATCH 00/28] Optimise page alloc/free fast paths v3 Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 15:10 +0200
[PATCH 12/28] mm, page_alloc: Remove unnecessary initialisation from __alloc_pages_nodemask() Mel Gorman <mgorman@techsingularity.net> - 2016-04-16 09:30 +0200
Re: [PATCH 12/28] mm, page_alloc: Remove unnecessary initialisation from __alloc_pages_nodemask() Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:50 +0200
Page 1 of 4 [1] 2 3 4 Next page →
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:00 +0200 |
| Subject | [PATCH 00/28] Optimise page alloc/free fast paths v3 |
| Message-ID | <ro7LB-hc-17@gated-at.bofh.it> |
There were no further responses to the last series but I kept going and
added a few more small bits. Most are basic micro-optimisations. The last
two patches weaken debugging checks to improve performance at the cost of
delayed detection of some use-after-free and memory corruption bugs. If
they make people uncomfortable, they can be dropped and the rest of the
series stands on its own.
Changelog since v2
o Add more micro-optimisations
o Weak debugging checks in favor of speed
Changelog since v1
o Fix an unused variable warning
o Throw in a few optimisations in the bulk pcp free path
o Rebase to 4.6-rc3
Another year, another round of page allocator optimisations focusing this
time on the alloc and free fast paths. This should be of help to workloads
that are allocator-intensive from kernel space where the cost of zeroing
is not nceessraily incurred.
The series is motivated by the observation that page alloc microbenchmarks
on multiple machines regressed between 3.12.44 and 4.4. Second, there is
discussions before LSF/MM considering the possibility of adding another
page allocator which is potentially hazardous but a patch series improving
performance is better than whining.
After the series is applied, there are still hazards. In the free paths,
the debugging checking and page zone/pageblock lookups dominate but
there was not an obvious solution to that. In the alloc path, the major
contributers are dealing with zonelists, new page preperation, the fair
zone allocation and numerous statistic updates. The fair zone allocator
is removed by the per-node LRU series if that gets merged so it's nor a
major concern at the moment.
On normal userspace benchmarks, there is little impact as the zeroing cost
is significant but it's visible
aim9
4.6.0-rc3 4.6.0-rc3
vanilla deferalloc-v3
Min page_test 828693.33 ( 0.00%) 887060.00 ( 7.04%)
Min brk_test 4847266.67 ( 0.00%) 4966266.67 ( 2.45%)
Min exec_test 1271.00 ( 0.00%) 1275.67 ( 0.37%)
Min fork_test 12371.75 ( 0.00%) 12380.00 ( 0.07%)
The overall impact on a page allocator microbenchmark for a range of orders
and number of pages allocated in a batch is
4.6.0-rc3 4.6.0-rc3
vanilla deferalloc-v3r7
Min alloc-odr0-1 428.00 ( 0.00%) 316.00 ( 26.17%)
Min alloc-odr0-2 314.00 ( 0.00%) 231.00 ( 26.43%)
Min alloc-odr0-4 256.00 ( 0.00%) 192.00 ( 25.00%)
Min alloc-odr0-8 222.00 ( 0.00%) 166.00 ( 25.23%)
Min alloc-odr0-16 207.00 ( 0.00%) 154.00 ( 25.60%)
Min alloc-odr0-32 197.00 ( 0.00%) 148.00 ( 24.87%)
Min alloc-odr0-64 193.00 ( 0.00%) 144.00 ( 25.39%)
Min alloc-odr0-128 191.00 ( 0.00%) 143.00 ( 25.13%)
Min alloc-odr0-256 203.00 ( 0.00%) 153.00 ( 24.63%)
Min alloc-odr0-512 212.00 ( 0.00%) 165.00 ( 22.17%)
Min alloc-odr0-1024 221.00 ( 0.00%) 172.00 ( 22.17%)
Min alloc-odr0-2048 225.00 ( 0.00%) 179.00 ( 20.44%)
Min alloc-odr0-4096 232.00 ( 0.00%) 185.00 ( 20.26%)
Min alloc-odr0-8192 235.00 ( 0.00%) 187.00 ( 20.43%)
Min alloc-odr0-16384 236.00 ( 0.00%) 188.00 ( 20.34%)
Min alloc-odr1-1 519.00 ( 0.00%) 450.00 ( 13.29%)
Min alloc-odr1-2 391.00 ( 0.00%) 336.00 ( 14.07%)
Min alloc-odr1-4 313.00 ( 0.00%) 268.00 ( 14.38%)
Min alloc-odr1-8 277.00 ( 0.00%) 235.00 ( 15.16%)
Min alloc-odr1-16 256.00 ( 0.00%) 218.00 ( 14.84%)
Min alloc-odr1-32 252.00 ( 0.00%) 212.00 ( 15.87%)
Min alloc-odr1-64 244.00 ( 0.00%) 206.00 ( 15.57%)
Min alloc-odr1-128 244.00 ( 0.00%) 207.00 ( 15.16%)
Min alloc-odr1-256 243.00 ( 0.00%) 207.00 ( 14.81%)
Min alloc-odr1-512 245.00 ( 0.00%) 209.00 ( 14.69%)
Min alloc-odr1-1024 248.00 ( 0.00%) 214.00 ( 13.71%)
Min alloc-odr1-2048 253.00 ( 0.00%) 220.00 ( 13.04%)
Min alloc-odr1-4096 258.00 ( 0.00%) 224.00 ( 13.18%)
Min alloc-odr1-8192 261.00 ( 0.00%) 229.00 ( 12.26%)
Min alloc-odr2-1 560.00 ( 0.00%) 753.00 (-34.46%)
Min alloc-odr2-2 424.00 ( 0.00%) 351.00 ( 17.22%)
Min alloc-odr2-4 339.00 ( 0.00%) 393.00 (-15.93%)
Min alloc-odr2-8 298.00 ( 0.00%) 246.00 ( 17.45%)
Min alloc-odr2-16 276.00 ( 0.00%) 227.00 ( 17.75%)
Min alloc-odr2-32 271.00 ( 0.00%) 221.00 ( 18.45%)
Min alloc-odr2-64 264.00 ( 0.00%) 217.00 ( 17.80%)
Min alloc-odr2-128 264.00 ( 0.00%) 217.00 ( 17.80%)
Min alloc-odr2-256 264.00 ( 0.00%) 218.00 ( 17.42%)
Min alloc-odr2-512 269.00 ( 0.00%) 223.00 ( 17.10%)
Min alloc-odr2-1024 279.00 ( 0.00%) 230.00 ( 17.56%)
Min alloc-odr2-2048 283.00 ( 0.00%) 235.00 ( 16.96%)
Min alloc-odr2-4096 285.00 ( 0.00%) 239.00 ( 16.14%)
Min alloc-odr3-1 629.00 ( 0.00%) 505.00 ( 19.71%)
Min alloc-odr3-2 472.00 ( 0.00%) 374.00 ( 20.76%)
Min alloc-odr3-4 383.00 ( 0.00%) 301.00 ( 21.41%)
Min alloc-odr3-8 341.00 ( 0.00%) 266.00 ( 21.99%)
Min alloc-odr3-16 316.00 ( 0.00%) 248.00 ( 21.52%)
Min alloc-odr3-32 308.00 ( 0.00%) 241.00 ( 21.75%)
Min alloc-odr3-64 305.00 ( 0.00%) 241.00 ( 20.98%)
Min alloc-odr3-128 308.00 ( 0.00%) 244.00 ( 20.78%)
Min alloc-odr3-256 317.00 ( 0.00%) 249.00 ( 21.45%)
Min alloc-odr3-512 327.00 ( 0.00%) 256.00 ( 21.71%)
Min alloc-odr3-1024 331.00 ( 0.00%) 261.00 ( 21.15%)
Min alloc-odr3-2048 333.00 ( 0.00%) 266.00 ( 20.12%)
Min alloc-odr4-1 767.00 ( 0.00%) 572.00 ( 25.42%)
Min alloc-odr4-2 578.00 ( 0.00%) 429.00 ( 25.78%)
Min alloc-odr4-4 474.00 ( 0.00%) 346.00 ( 27.00%)
Min alloc-odr4-8 422.00 ( 0.00%) 310.00 ( 26.54%)
Min alloc-odr4-16 399.00 ( 0.00%) 295.00 ( 26.07%)
Min alloc-odr4-32 392.00 ( 0.00%) 293.00 ( 25.26%)
Min alloc-odr4-64 394.00 ( 0.00%) 293.00 ( 25.63%)
Min alloc-odr4-128 405.00 ( 0.00%) 305.00 ( 24.69%)
Min alloc-odr4-256 417.00 ( 0.00%) 319.00 ( 23.50%)
Min alloc-odr4-512 425.00 ( 0.00%) 326.00 ( 23.29%)
Min alloc-odr4-1024 426.00 ( 0.00%) 329.00 ( 22.77%)
Min free-odr0-1 216.00 ( 0.00%) 178.00 ( 17.59%)
Min free-odr0-2 152.00 ( 0.00%) 125.00 ( 17.76%)
Min free-odr0-4 120.00 ( 0.00%) 99.00 ( 17.50%)
Min free-odr0-8 106.00 ( 0.00%) 85.00 ( 19.81%)
Min free-odr0-16 97.00 ( 0.00%) 80.00 ( 17.53%)
Min free-odr0-32 92.00 ( 0.00%) 76.00 ( 17.39%)
Min free-odr0-64 89.00 ( 0.00%) 74.00 ( 16.85%)
Min free-odr0-128 89.00 ( 0.00%) 73.00 ( 17.98%)
Min free-odr0-256 107.00 ( 0.00%) 90.00 ( 15.89%)
Min free-odr0-512 117.00 ( 0.00%) 108.00 ( 7.69%)
Min free-odr0-1024 125.00 ( 0.00%) 118.00 ( 5.60%)
Min free-odr0-2048 132.00 ( 0.00%) 125.00 ( 5.30%)
Min free-odr0-4096 135.00 ( 0.00%) 130.00 ( 3.70%)
Min free-odr0-8192 137.00 ( 0.00%) 130.00 ( 5.11%)
Min free-odr0-16384 137.00 ( 0.00%) 131.00 ( 4.38%)
Min free-odr1-1 318.00 ( 0.00%) 289.00 ( 9.12%)
Min free-odr1-2 228.00 ( 0.00%) 207.00 ( 9.21%)
Min free-odr1-4 182.00 ( 0.00%) 165.00 ( 9.34%)
Min free-odr1-8 163.00 ( 0.00%) 146.00 ( 10.43%)
Min free-odr1-16 151.00 ( 0.00%) 135.00 ( 10.60%)
Min free-odr1-32 146.00 ( 0.00%) 129.00 ( 11.64%)
Min free-odr1-64 145.00 ( 0.00%) 130.00 ( 10.34%)
Min free-odr1-128 148.00 ( 0.00%) 134.00 ( 9.46%)
Min free-odr1-256 148.00 ( 0.00%) 137.00 ( 7.43%)
Min free-odr1-512 151.00 ( 0.00%) 140.00 ( 7.28%)
Min free-odr1-1024 154.00 ( 0.00%) 143.00 ( 7.14%)
Min free-odr1-2048 156.00 ( 0.00%) 144.00 ( 7.69%)
Min free-odr1-4096 156.00 ( 0.00%) 142.00 ( 8.97%)
Min free-odr1-8192 156.00 ( 0.00%) 140.00 ( 10.26%)
Min free-odr2-1 361.00 ( 0.00%) 457.00 (-26.59%)
Min free-odr2-2 258.00 ( 0.00%) 224.00 ( 13.18%)
Min free-odr2-4 208.00 ( 0.00%) 223.00 ( -7.21%)
Min free-odr2-8 185.00 ( 0.00%) 160.00 ( 13.51%)
Min free-odr2-16 173.00 ( 0.00%) 149.00 ( 13.87%)
Min free-odr2-32 166.00 ( 0.00%) 145.00 ( 12.65%)
Min free-odr2-64 166.00 ( 0.00%) 146.00 ( 12.05%)
Min free-odr2-128 169.00 ( 0.00%) 148.00 ( 12.43%)
Min free-odr2-256 170.00 ( 0.00%) 152.00 ( 10.59%)
Min free-odr2-512 177.00 ( 0.00%) 156.00 ( 11.86%)
Min free-odr2-1024 182.00 ( 0.00%) 162.00 ( 10.99%)
Min free-odr2-2048 181.00 ( 0.00%) 160.00 ( 11.60%)
Min free-odr2-4096 180.00 ( 0.00%) 159.00 ( 11.67%)
Min free-odr3-1 431.00 ( 0.00%) 367.00 ( 14.85%)
Min free-odr3-2 306.00 ( 0.00%) 259.00 ( 15.36%)
Min free-odr3-4 249.00 ( 0.00%) 208.00 ( 16.47%)
Min free-odr3-8 224.00 ( 0.00%) 186.00 ( 16.96%)
Min free-odr3-16 208.00 ( 0.00%) 176.00 ( 15.38%)
Min free-odr3-32 206.00 ( 0.00%) 174.00 ( 15.53%)
Min free-odr3-64 210.00 ( 0.00%) 178.00 ( 15.24%)
Min free-odr3-128 215.00 ( 0.00%) 182.00 ( 15.35%)
Min free-odr3-256 224.00 ( 0.00%) 189.00 ( 15.62%)
Min free-odr3-512 232.00 ( 0.00%) 195.00 ( 15.95%)
Min free-odr3-1024 230.00 ( 0.00%) 195.00 ( 15.22%)
Min free-odr3-2048 229.00 ( 0.00%) 193.00 ( 15.72%)
Min free-odr4-1 561.00 ( 0.00%) 439.00 ( 21.75%)
Min free-odr4-2 418.00 ( 0.00%) 318.00 ( 23.92%)
Min free-odr4-4 339.00 ( 0.00%) 269.00 ( 20.65%)
Min free-odr4-8 299.00 ( 0.00%) 239.00 ( 20.07%)
Min free-odr4-16 289.00 ( 0.00%) 234.00 ( 19.03%)
Min free-odr4-32 291.00 ( 0.00%) 235.00 ( 19.24%)
Min free-odr4-64 298.00 ( 0.00%) 238.00 ( 20.13%)
Min free-odr4-128 308.00 ( 0.00%) 251.00 ( 18.51%)
Min free-odr4-256 321.00 ( 0.00%) 267.00 ( 16.82%)
Min free-odr4-512 327.00 ( 0.00%) 269.00 ( 17.74%)
Min free-odr4-1024 326.00 ( 0.00%) 271.00 ( 16.87%)
Min total-odr0-1 644.00 ( 0.00%) 494.00 ( 23.29%)
Min total-odr0-2 466.00 ( 0.00%) 356.00 ( 23.61%)
Min total-odr0-4 376.00 ( 0.00%) 291.00 ( 22.61%)
Min total-odr0-8 328.00 ( 0.00%) 251.00 ( 23.48%)
Min total-odr0-16 304.00 ( 0.00%) 234.00 ( 23.03%)
Min total-odr0-32 289.00 ( 0.00%) 224.00 ( 22.49%)
Min total-odr0-64 282.00 ( 0.00%) 218.00 ( 22.70%)
Min total-odr0-128 280.00 ( 0.00%) 216.00 ( 22.86%)
Min total-odr0-256 310.00 ( 0.00%) 243.00 ( 21.61%)
Min total-odr0-512 329.00 ( 0.00%) 273.00 ( 17.02%)
Min total-odr0-1024 346.00 ( 0.00%) 290.00 ( 16.18%)
Min total-odr0-2048 357.00 ( 0.00%) 304.00 ( 14.85%)
Min total-odr0-4096 367.00 ( 0.00%) 315.00 ( 14.17%)
Min total-odr0-8192 372.00 ( 0.00%) 317.00 ( 14.78%)
Min total-odr0-16384 373.00 ( 0.00%) 319.00 ( 14.48%)
Min total-odr1-1 838.00 ( 0.00%) 739.00 ( 11.81%)
Min total-odr1-2 619.00 ( 0.00%) 543.00 ( 12.28%)
Min total-odr1-4 495.00 ( 0.00%) 433.00 ( 12.53%)
Min total-odr1-8 440.00 ( 0.00%) 382.00 ( 13.18%)
Min total-odr1-16 407.00 ( 0.00%) 353.00 ( 13.27%)
Min total-odr1-32 398.00 ( 0.00%) 341.00 ( 14.32%)
Min total-odr1-64 389.00 ( 0.00%) 336.00 ( 13.62%)
Min total-odr1-128 392.00 ( 0.00%) 341.00 ( 13.01%)
Min total-odr1-256 391.00 ( 0.00%) 344.00 ( 12.02%)
Min total-odr1-512 396.00 ( 0.00%) 349.00 ( 11.87%)
Min total-odr1-1024 402.00 ( 0.00%) 357.00 ( 11.19%)
Min total-odr1-2048 409.00 ( 0.00%) 364.00 ( 11.00%)
Min total-odr1-4096 414.00 ( 0.00%) 366.00 ( 11.59%)
Min total-odr1-8192 417.00 ( 0.00%) 369.00 ( 11.51%)
Min total-odr2-1 921.00 ( 0.00%) 1210.00 (-31.38%)
Min total-odr2-2 682.00 ( 0.00%) 576.00 ( 15.54%)
Min total-odr2-4 547.00 ( 0.00%) 616.00 (-12.61%)
Min total-odr2-8 483.00 ( 0.00%) 406.00 ( 15.94%)
Min total-odr2-16 449.00 ( 0.00%) 376.00 ( 16.26%)
Min total-odr2-32 437.00 ( 0.00%) 366.00 ( 16.25%)
Min total-odr2-64 431.00 ( 0.00%) 363.00 ( 15.78%)
Min total-odr2-128 433.00 ( 0.00%) 365.00 ( 15.70%)
Min total-odr2-256 434.00 ( 0.00%) 371.00 ( 14.52%)
Min total-odr2-512 446.00 ( 0.00%) 379.00 ( 15.02%)
Min total-odr2-1024 461.00 ( 0.00%) 392.00 ( 14.97%)
Min total-odr2-2048 464.00 ( 0.00%) 395.00 ( 14.87%)
Min total-odr2-4096 465.00 ( 0.00%) 398.00 ( 14.41%)
Min total-odr3-1 1060.00 ( 0.00%) 872.00 ( 17.74%)
Min total-odr3-2 778.00 ( 0.00%) 633.00 ( 18.64%)
Min total-odr3-4 632.00 ( 0.00%) 510.00 ( 19.30%)
Min total-odr3-8 565.00 ( 0.00%) 452.00 ( 20.00%)
Min total-odr3-16 524.00 ( 0.00%) 424.00 ( 19.08%)
Min total-odr3-32 514.00 ( 0.00%) 415.00 ( 19.26%)
Min total-odr3-64 515.00 ( 0.00%) 419.00 ( 18.64%)
Min total-odr3-128 523.00 ( 0.00%) 426.00 ( 18.55%)
Min total-odr3-256 541.00 ( 0.00%) 438.00 ( 19.04%)
Min total-odr3-512 559.00 ( 0.00%) 451.00 ( 19.32%)
Min total-odr3-1024 561.00 ( 0.00%) 456.00 ( 18.72%)
Min total-odr3-2048 562.00 ( 0.00%) 459.00 ( 18.33%)
Min total-odr4-1 1328.00 ( 0.00%) 1011.00 ( 23.87%)
Min total-odr4-2 997.00 ( 0.00%) 747.00 ( 25.08%)
Min total-odr4-4 813.00 ( 0.00%) 615.00 ( 24.35%)
Min total-odr4-8 721.00 ( 0.00%) 550.00 ( 23.72%)
Min total-odr4-16 689.00 ( 0.00%) 529.00 ( 23.22%)
Min total-odr4-32 683.00 ( 0.00%) 528.00 ( 22.69%)
Min total-odr4-64 692.00 ( 0.00%) 531.00 ( 23.27%)
Min total-odr4-128 713.00 ( 0.00%) 556.00 ( 22.02%)
Min total-odr4-256 738.00 ( 0.00%) 586.00 ( 20.60%)
Min total-odr4-512 753.00 ( 0.00%) 595.00 ( 20.98%)
Min total-odr4-1024 752.00 ( 0.00%) 600.00 ( 20.21%)
fs/buffer.c | 10 +-
include/linux/compaction.h | 6 +-
include/linux/cpuset.h | 42 ++-
include/linux/mm.h | 5 +-
include/linux/mmzone.h | 41 ++-
include/linux/page-flags.h | 7 +-
include/linux/vmstat.h | 2 -
kernel/cpuset.c | 14 +-
mm/compaction.c | 16 +-
mm/internal.h | 7 +-
mm/mempolicy.c | 19 +-
mm/mmzone.c | 2 +-
mm/page_alloc.c | 836 +++++++++++++++++++++++++++------------------
mm/page_owner.c | 2 +-
mm/vmstat.c | 27 +-
15 files changed, 602 insertions(+), 434 deletions(-)
--
2.6.4
[toc] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:00 +0200 |
| Subject | [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages |
| Message-ID | <ro7LB-hc-29@gated-at.bofh.it> |
| In reply to | #1379561 |
order-0 pages by definition cannot be compound so avoid the check in the
fast path for those pages.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 25 +++++++++++++++++--------
1 file changed, 17 insertions(+), 8 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 59de90d5d3a3..5d205bcfe10d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1024,24 +1024,33 @@ void __meminit reserve_bootmem_region(unsigned long start, unsigned long end)
static bool free_pages_prepare(struct page *page, unsigned int order)
{
- bool compound = PageCompound(page);
- int i, bad = 0;
+ int bad = 0;
VM_BUG_ON_PAGE(PageTail(page), page);
- VM_BUG_ON_PAGE(compound && compound_order(page) != order, page);
trace_mm_page_free(page, order);
kmemcheck_free_shadow(page, order);
kasan_free_pages(page, order);
+ /*
+ * Check tail pages before head page information is cleared to
+ * avoid checking PageCompound for order-0 pages.
+ */
+ if (order) {
+ bool compound = PageCompound(page);
+ int i;
+
+ VM_BUG_ON_PAGE(compound && compound_order(page) != order, page);
+
+ for (i = 1; i < (1 << order); i++) {
+ if (compound)
+ bad += free_tail_pages_check(page, page + i);
+ bad += free_pages_check(page + i);
+ }
+ }
if (PageAnon(page))
page->mapping = NULL;
bad += free_pages_check(page);
- for (i = 1; i < (1 << order); i++) {
- if (compound)
- bad += free_tail_pages_check(page, page + i);
- bad += free_pages_check(page + i);
- }
if (bad)
return false;
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-25 11:40 +0200 |
| Subject | Re: [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages |
| Message-ID | <rrL9L-3t5-9@gated-at.bofh.it> |
| In reply to | #1379564 |
On 04/15/2016 10:58 AM, Mel Gorman wrote:
> order-0 pages by definition cannot be compound so avoid the check in the
> fast path for those pages.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Suggestion to improve below:
> ---
> mm/page_alloc.c | 25 +++++++++++++++++--------
> 1 file changed, 17 insertions(+), 8 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 59de90d5d3a3..5d205bcfe10d 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1024,24 +1024,33 @@ void __meminit reserve_bootmem_region(unsigned long start, unsigned long end)
>
> static bool free_pages_prepare(struct page *page, unsigned int order)
> {
> - bool compound = PageCompound(page);
> - int i, bad = 0;
> + int bad = 0;
>
> VM_BUG_ON_PAGE(PageTail(page), page);
> - VM_BUG_ON_PAGE(compound && compound_order(page) != order, page);
>
> trace_mm_page_free(page, order);
> kmemcheck_free_shadow(page, order);
> kasan_free_pages(page, order);
>
> + /*
> + * Check tail pages before head page information is cleared to
> + * avoid checking PageCompound for order-0 pages.
> + */
> + if (order) {
Sticking unlikely() here results in:
add/remove: 0/0 grow/shrink: 0/1 up/down: 0/-30 (-30)
function old new delta
free_pages_prepare 771 741 -30
And from brief comparison of disassembly it really seems it's moved the
compound handling towards the end of the function, which should be nicer
for the instruction cache, branch prediction etc. And since this series
is about microoptimization, I think the extra step is worth it.
> + bool compound = PageCompound(page);
> + int i;
> +
> + VM_BUG_ON_PAGE(compound && compound_order(page) != order, page);
> +
> + for (i = 1; i < (1 << order); i++) {
> + if (compound)
> + bad += free_tail_pages_check(page, page + i);
> + bad += free_pages_check(page + i);
> + }
> + }
> if (PageAnon(page))
> page->mapping = NULL;
> bad += free_pages_check(page);
> - for (i = 1; i < (1 << order); i++) {
> - if (compound)
> - bad += free_tail_pages_check(page, page + i);
> - bad += free_pages_check(page + i);
> - }
> if (bad)
> return false;
>
>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-26 12:40 +0200 |
| Subject | Re: [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages |
| Message-ID | <rs8zp-652-49@gated-at.bofh.it> |
| In reply to | #1386172 |
On Mon, Apr 25, 2016 at 11:33:15AM +0200, Vlastimil Babka wrote:
> On 04/15/2016 10:58 AM, Mel Gorman wrote:
> >order-0 pages by definition cannot be compound so avoid the check in the
> >fast path for those pages.
> >
> >Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
>
> Acked-by: Vlastimil Babka <vbabka@suse.cz>
>
> Suggestion to improve below:
>
> >---
> > mm/page_alloc.c | 25 +++++++++++++++++--------
> > 1 file changed, 17 insertions(+), 8 deletions(-)
> >
> >diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> >index 59de90d5d3a3..5d205bcfe10d 100644
> >--- a/mm/page_alloc.c
> >+++ b/mm/page_alloc.c
> >@@ -1024,24 +1024,33 @@ void __meminit reserve_bootmem_region(unsigned long start, unsigned long end)
> >
> > static bool free_pages_prepare(struct page *page, unsigned int order)
> > {
> >- bool compound = PageCompound(page);
> >- int i, bad = 0;
> >+ int bad = 0;
> >
> > VM_BUG_ON_PAGE(PageTail(page), page);
> >- VM_BUG_ON_PAGE(compound && compound_order(page) != order, page);
> >
> > trace_mm_page_free(page, order);
> > kmemcheck_free_shadow(page, order);
> > kasan_free_pages(page, order);
> >
> >+ /*
> >+ * Check tail pages before head page information is cleared to
> >+ * avoid checking PageCompound for order-0 pages.
> >+ */
> >+ if (order) {
>
> Sticking unlikely() here results in:
>
> add/remove: 0/0 grow/shrink: 0/1 up/down: 0/-30 (-30)
> function old new delta
> free_pages_prepare 771 741 -30
>
> And from brief comparison of disassembly it really seems it's moved the
> compound handling towards the end of the function, which should be nicer for
> the instruction cache, branch prediction etc. And since this series is about
> microoptimization, I think the extra step is worth it.
>
I dithered on this a bit and could not convince myself that the order
case really is unlikely. It depends on the situation as we could be
tearing down a large THP-backed mapping. SLUB is also using compound
pages so it's both workload and configuration dependent whether this
path is really likely or not.
--
Mel Gorman
SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 13:30 +0200 |
| Subject | Re: [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages |
| Message-ID | <rs9lM-6K3-11@gated-at.bofh.it> |
| In reply to | #1387353 |
On 04/26/2016 12:33 PM, Mel Gorman wrote: > > I dithered on this a bit and could not convince myself that the order > case really is unlikely. It depends on the situation as we could be > tearing down a large THP-backed mapping. SLUB is also using compound > pages so it's both workload and configuration dependent whether this > path is really likely or not. Hmm I see. But e.g. buffered_rmqueue uses "if (likely(order == 0))" so it would be at least consistent. Also compound pages can amortize the extra cost over more base pages.
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 21/28] mm, page_alloc: Avoid looking up the first zone in a zonelist twice |
| Message-ID | <ro7Vg-Er-13@gated-at.bofh.it> |
| In reply to | #1379561 |
The allocator fast path looks up the first usable zone in a zonelist
and then get_page_from_freelist does the same job in the zonelist
iterator. This patch preserves the necessary information.
4.6.0-rc2 4.6.0-rc2
fastmark-v1r20 initonce-v1r20
Min alloc-odr0-1 364.00 ( 0.00%) 359.00 ( 1.37%)
Min alloc-odr0-2 262.00 ( 0.00%) 260.00 ( 0.76%)
Min alloc-odr0-4 214.00 ( 0.00%) 214.00 ( 0.00%)
Min alloc-odr0-8 186.00 ( 0.00%) 186.00 ( 0.00%)
Min alloc-odr0-16 173.00 ( 0.00%) 173.00 ( 0.00%)
Min alloc-odr0-32 165.00 ( 0.00%) 165.00 ( 0.00%)
Min alloc-odr0-64 161.00 ( 0.00%) 162.00 ( -0.62%)
Min alloc-odr0-128 159.00 ( 0.00%) 161.00 ( -1.26%)
Min alloc-odr0-256 168.00 ( 0.00%) 170.00 ( -1.19%)
Min alloc-odr0-512 180.00 ( 0.00%) 181.00 ( -0.56%)
Min alloc-odr0-1024 190.00 ( 0.00%) 190.00 ( 0.00%)
Min alloc-odr0-2048 196.00 ( 0.00%) 196.00 ( 0.00%)
Min alloc-odr0-4096 202.00 ( 0.00%) 202.00 ( 0.00%)
Min alloc-odr0-8192 206.00 ( 0.00%) 205.00 ( 0.49%)
Min alloc-odr0-16384 206.00 ( 0.00%) 205.00 ( 0.49%)
The benefit is negligible and the results are within the noise but each
cycle counts.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
fs/buffer.c | 10 +++++-----
include/linux/mmzone.h | 18 +++++++++++-------
mm/internal.h | 2 +-
mm/mempolicy.c | 19 ++++++++++---------
mm/page_alloc.c | 32 +++++++++++++++-----------------
5 files changed, 42 insertions(+), 39 deletions(-)
diff --git a/fs/buffer.c b/fs/buffer.c
index af0d9a82a8ed..754813a6962b 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -255,17 +255,17 @@ __find_get_block_slow(struct block_device *bdev, sector_t block)
*/
static void free_more_memory(void)
{
- struct zone *zone;
+ struct zoneref *z;
int nid;
wakeup_flusher_threads(1024, WB_REASON_FREE_MORE_MEM);
yield();
for_each_online_node(nid) {
- (void)first_zones_zonelist(node_zonelist(nid, GFP_NOFS),
- gfp_zone(GFP_NOFS), NULL,
- &zone);
- if (zone)
+
+ z = first_zones_zonelist(node_zonelist(nid, GFP_NOFS),
+ gfp_zone(GFP_NOFS), NULL);
+ if (z->zone)
try_to_free_pages(node_zonelist(nid, GFP_NOFS), 0,
GFP_NOFS, NULL);
}
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index f49bb9add372..bf153ed097d5 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -962,13 +962,10 @@ static __always_inline struct zoneref *next_zones_zonelist(struct zoneref *z,
*/
static inline struct zoneref *first_zones_zonelist(struct zonelist *zonelist,
enum zone_type highest_zoneidx,
- nodemask_t *nodes,
- struct zone **zone)
+ nodemask_t *nodes)
{
- struct zoneref *z = next_zones_zonelist(zonelist->_zonerefs,
+ return next_zones_zonelist(zonelist->_zonerefs,
highest_zoneidx, nodes);
- *zone = zonelist_zone(z);
- return z;
}
/**
@@ -983,10 +980,17 @@ static inline struct zoneref *first_zones_zonelist(struct zonelist *zonelist,
* within a given nodemask
*/
#define for_each_zone_zonelist_nodemask(zone, z, zlist, highidx, nodemask) \
- for (z = first_zones_zonelist(zlist, highidx, nodemask, &zone); \
+ for (z = first_zones_zonelist(zlist, highidx, nodemask), zone = zonelist_zone(z); \
zone; \
z = next_zones_zonelist(++z, highidx, nodemask), \
- zone = zonelist_zone(z)) \
+ zone = zonelist_zone(z))
+
+#define for_next_zone_zonelist_nodemask(zone, z, zlist, highidx, nodemask) \
+ for (zone = z->zone; \
+ zone; \
+ z = next_zones_zonelist(++z, highidx, nodemask), \
+ zone = zonelist_zone(z))
+
/**
* for_each_zone_zonelist - helper macro to iterate over valid zones in a zonelist at or below a given zone index
diff --git a/mm/internal.h b/mm/internal.h
index f6d0a5875ec4..4c2396cd514c 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -102,7 +102,7 @@ extern pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);
struct alloc_context {
struct zonelist *zonelist;
nodemask_t *nodemask;
- struct zone *preferred_zone;
+ struct zoneref *preferred_zoneref;
int classzone_idx;
int migratetype;
enum zone_type high_zoneidx;
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 36cc01bc950a..66d73efba370 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1744,18 +1744,18 @@ unsigned int mempolicy_slab_node(void)
return interleave_nodes(policy);
case MPOL_BIND: {
+ struct zoneref *z;
+
/*
* Follow bind policy behavior and start allocation at the
* first node.
*/
struct zonelist *zonelist;
- struct zone *zone;
enum zone_type highest_zoneidx = gfp_zone(GFP_KERNEL);
zonelist = &NODE_DATA(node)->node_zonelists[0];
- (void)first_zones_zonelist(zonelist, highest_zoneidx,
- &policy->v.nodes,
- &zone);
- return zone ? zone->node : node;
+ z = first_zones_zonelist(zonelist, highest_zoneidx,
+ &policy->v.nodes);
+ return z->zone ? z->zone->node : node;
}
default:
@@ -2284,7 +2284,7 @@ static void sp_free(struct sp_node *n)
int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long addr)
{
struct mempolicy *pol;
- struct zone *zone;
+ struct zoneref *z;
int curnid = page_to_nid(page);
unsigned long pgoff;
int thiscpu = raw_smp_processor_id();
@@ -2316,6 +2316,7 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
break;
case MPOL_BIND:
+
/*
* allows binding to multiple nodes.
* use current page if in policy nodemask,
@@ -2324,11 +2325,11 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
*/
if (node_isset(curnid, pol->v.nodes))
goto out;
- (void)first_zones_zonelist(
+ z = first_zones_zonelist(
node_zonelist(numa_node_id(), GFP_HIGHUSER),
gfp_zone(GFP_HIGHUSER),
- &pol->v.nodes, &zone);
- polnid = zone->node;
+ &pol->v.nodes);
+ polnid = z->zone->node;
break;
default:
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8923d74b1707..897e9d2a8500 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2700,7 +2700,7 @@ static struct page *
get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
const struct alloc_context *ac)
{
- struct zoneref *z;
+ struct zoneref *z = ac->preferred_zoneref;
struct zone *zone;
bool fair_skipped = false;
bool apply_fair = (alloc_flags & ALLOC_FAIR);
@@ -2710,7 +2710,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
* Scan zonelist, looking for a zone with enough free.
* See also __cpuset_node_allowed() comment in kernel/cpuset.c.
*/
- for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->high_zoneidx,
+ for_next_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->high_zoneidx,
ac->nodemask) {
struct page *page;
unsigned long mark;
@@ -2730,7 +2730,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
fair_skipped = true;
continue;
}
- if (!zone_local(ac->preferred_zone, zone)) {
+ if (!zone_local(ac->preferred_zoneref->zone, zone)) {
if (fair_skipped)
goto reset_fair;
apply_fair = false;
@@ -2776,7 +2776,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
goto try_this_zone;
if (zone_reclaim_mode == 0 ||
- !zone_allows_reclaim(ac->preferred_zone, zone))
+ !zone_allows_reclaim(ac->preferred_zoneref->zone, zone))
continue;
ret = zone_reclaim(zone, gfp_mask, order);
@@ -2798,7 +2798,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
}
try_this_zone:
- page = buffered_rmqueue(ac->preferred_zone, zone, order,
+ page = buffered_rmqueue(ac->preferred_zoneref->zone, zone, order,
gfp_mask, alloc_flags, ac->migratetype);
if (page) {
if (prep_new_page(page, order, gfp_mask, alloc_flags))
@@ -2827,7 +2827,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
reset_fair:
apply_fair = false;
fair_skipped = false;
- reset_alloc_batches(ac->preferred_zone);
+ reset_alloc_batches(ac->preferred_zoneref->zone);
goto zonelist_scan;
}
@@ -3114,7 +3114,7 @@ static void wake_all_kswapds(unsigned int order, const struct alloc_context *ac)
for_each_zone_zonelist_nodemask(zone, z, ac->zonelist,
ac->high_zoneidx, ac->nodemask)
- wakeup_kswapd(zone, order, zone_idx(ac->preferred_zone));
+ wakeup_kswapd(zone, order, zonelist_zone_idx(ac->preferred_zoneref));
}
static inline unsigned int
@@ -3334,7 +3334,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
if ((did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) ||
((gfp_mask & __GFP_REPEAT) && pages_reclaimed < (1 << order))) {
/* Wait for some write requests to complete then retry */
- wait_iff_congested(ac->preferred_zone, BLK_RW_ASYNC, HZ/50);
+ wait_iff_congested(ac->preferred_zoneref->zone, BLK_RW_ASYNC, HZ/50);
goto retry;
}
@@ -3372,7 +3372,6 @@ struct page *
__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
struct zonelist *zonelist, nodemask_t *nodemask)
{
- struct zoneref *preferred_zoneref;
struct page *page;
unsigned int cpuset_mems_cookie;
unsigned int alloc_flags = ALLOC_WMARK_LOW|ALLOC_FAIR;
@@ -3408,9 +3407,9 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
ac.spread_dirty_pages = (gfp_mask & __GFP_WRITE);
/* The preferred zone is used for statistics later */
- preferred_zoneref = first_zones_zonelist(ac.zonelist, ac.high_zoneidx,
- ac.nodemask, &ac.preferred_zone);
- ac.classzone_idx = zonelist_zone_idx(preferred_zoneref);
+ ac.preferred_zoneref = first_zones_zonelist(ac.zonelist, ac.high_zoneidx,
+ ac.nodemask);
+ ac.classzone_idx = zonelist_zone_idx(ac.preferred_zoneref);
/* First allocation attempt */
page = get_page_from_freelist(alloc_mask, order, alloc_flags, &ac);
@@ -4439,13 +4438,12 @@ static void build_zonelists(pg_data_t *pgdat)
*/
int local_memory_node(int node)
{
- struct zone *zone;
+ struct zoneref *z;
- (void)first_zones_zonelist(node_zonelist(node, GFP_KERNEL),
+ z = first_zones_zonelist(node_zonelist(node, GFP_KERNEL),
gfp_zone(GFP_KERNEL),
- NULL,
- &zone);
- return zone->node;
+ NULL);
+ return z->zone->node;
}
#endif
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 19:50 +0200 |
| Subject | Re: [PATCH 21/28] mm, page_alloc: Avoid looking up the first zone in a zonelist twice |
| Message-ID | <rsfhw-3db-21@gated-at.bofh.it> |
| In reply to | #1379566 |
On 04/15/2016 11:07 AM, Mel Gorman wrote: > The allocator fast path looks up the first usable zone in a zonelist > and then get_page_from_freelist does the same job in the zonelist > iterator. This patch preserves the necessary information. > > 4.6.0-rc2 4.6.0-rc2 > fastmark-v1r20 initonce-v1r20 > Min alloc-odr0-1 364.00 ( 0.00%) 359.00 ( 1.37%) > Min alloc-odr0-2 262.00 ( 0.00%) 260.00 ( 0.76%) > Min alloc-odr0-4 214.00 ( 0.00%) 214.00 ( 0.00%) > Min alloc-odr0-8 186.00 ( 0.00%) 186.00 ( 0.00%) > Min alloc-odr0-16 173.00 ( 0.00%) 173.00 ( 0.00%) > Min alloc-odr0-32 165.00 ( 0.00%) 165.00 ( 0.00%) > Min alloc-odr0-64 161.00 ( 0.00%) 162.00 ( -0.62%) > Min alloc-odr0-128 159.00 ( 0.00%) 161.00 ( -1.26%) > Min alloc-odr0-256 168.00 ( 0.00%) 170.00 ( -1.19%) > Min alloc-odr0-512 180.00 ( 0.00%) 181.00 ( -0.56%) > Min alloc-odr0-1024 190.00 ( 0.00%) 190.00 ( 0.00%) > Min alloc-odr0-2048 196.00 ( 0.00%) 196.00 ( 0.00%) > Min alloc-odr0-4096 202.00 ( 0.00%) 202.00 ( 0.00%) > Min alloc-odr0-8192 206.00 ( 0.00%) 205.00 ( 0.49%) > Min alloc-odr0-16384 206.00 ( 0.00%) 205.00 ( 0.49%) > > The benefit is negligible and the results are within the noise but each > cycle counts. Hmm this indeed doesn't look too convincing to justify the patch. Also it's adding adding extra pointer dereferences by accessing zone via zoneref, and the next patch does the same with classzone_idx (stack saving shouldn't be that important when the purpose of alloc_context is to have all of it only once on stack). I don't feel strongly enough to NAK, but not convinced to ack either.
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 04/28] mm, page_alloc: Inline zone_statistics |
| Message-ID | <ro7Vg-Er-5@gated-at.bofh.it> |
| In reply to | #1379561 |
zone_statistics has one call-site but it's a public function. Make
it static and inline.
The performance difference on a page allocator microbenchmark is;
4.6.0-rc2 4.6.0-rc2
statbranch-v1r20 statinline-v1r20
Min alloc-odr0-1 419.00 ( 0.00%) 412.00 ( 1.67%)
Min alloc-odr0-2 305.00 ( 0.00%) 301.00 ( 1.31%)
Min alloc-odr0-4 250.00 ( 0.00%) 247.00 ( 1.20%)
Min alloc-odr0-8 219.00 ( 0.00%) 215.00 ( 1.83%)
Min alloc-odr0-16 203.00 ( 0.00%) 199.00 ( 1.97%)
Min alloc-odr0-32 195.00 ( 0.00%) 191.00 ( 2.05%)
Min alloc-odr0-64 191.00 ( 0.00%) 187.00 ( 2.09%)
Min alloc-odr0-128 189.00 ( 0.00%) 185.00 ( 2.12%)
Min alloc-odr0-256 198.00 ( 0.00%) 193.00 ( 2.53%)
Min alloc-odr0-512 210.00 ( 0.00%) 207.00 ( 1.43%)
Min alloc-odr0-1024 216.00 ( 0.00%) 213.00 ( 1.39%)
Min alloc-odr0-2048 221.00 ( 0.00%) 220.00 ( 0.45%)
Min alloc-odr0-4096 227.00 ( 0.00%) 226.00 ( 0.44%)
Min alloc-odr0-8192 232.00 ( 0.00%) 229.00 ( 1.29%)
Min alloc-odr0-16384 232.00 ( 0.00%) 229.00 ( 1.29%)
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
include/linux/vmstat.h | 2 --
mm/page_alloc.c | 31 +++++++++++++++++++++++++++++++
mm/vmstat.c | 29 -----------------------------
3 files changed, 31 insertions(+), 31 deletions(-)
diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 73fae8c4a5fb..152d26b7f972 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -163,12 +163,10 @@ static inline unsigned long zone_page_state_snapshot(struct zone *zone,
#ifdef CONFIG_NUMA
extern unsigned long node_page_state(int node, enum zone_stat_item item);
-extern void zone_statistics(struct zone *, struct zone *, gfp_t gfp);
#else
#define node_page_state(node, item) global_page_state(item)
-#define zone_statistics(_zl, _z, gfp) do { } while (0)
#endif /* CONFIG_NUMA */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6812de41f698..b56c2b2911a2 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2352,6 +2352,37 @@ int split_free_page(struct page *page)
}
/*
+ * Update NUMA hit/miss statistics
+ *
+ * Must be called with interrupts disabled.
+ *
+ * When __GFP_OTHER_NODE is set assume the node of the preferred
+ * zone is the local node. This is useful for daemons who allocate
+ * memory on behalf of other processes.
+ */
+static inline void zone_statistics(struct zone *preferred_zone, struct zone *z,
+ gfp_t flags)
+{
+#ifdef CONFIG_NUMA
+ int local_nid = numa_node_id();
+ enum zone_stat_item local_stat = NUMA_LOCAL;
+
+ if (unlikely(flags & __GFP_OTHER_NODE)) {
+ local_stat = NUMA_OTHER;
+ local_nid = preferred_zone->node;
+ }
+
+ if (z->node == local_nid) {
+ __inc_zone_state(z, NUMA_HIT);
+ __inc_zone_state(z, local_stat);
+ } else {
+ __inc_zone_state(z, NUMA_MISS);
+ __inc_zone_state(preferred_zone, NUMA_FOREIGN);
+ }
+#endif
+}
+
+/*
* Allocate a page from the given zone. Use pcplists for order-0 allocations.
*/
static inline
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 2e58ead9bcf5..a4bda11eac8d 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -570,35 +570,6 @@ void drain_zonestat(struct zone *zone, struct per_cpu_pageset *pset)
#ifdef CONFIG_NUMA
/*
- * zonelist = the list of zones passed to the allocator
- * z = the zone from which the allocation occurred.
- *
- * Must be called with interrupts disabled.
- *
- * When __GFP_OTHER_NODE is set assume the node of the preferred
- * zone is the local node. This is useful for daemons who allocate
- * memory on behalf of other processes.
- */
-void zone_statistics(struct zone *preferred_zone, struct zone *z, gfp_t flags)
-{
- int local_nid = numa_node_id();
- enum zone_stat_item local_stat = NUMA_LOCAL;
-
- if (unlikely(flags & __GFP_OTHER_NODE)) {
- local_stat = NUMA_OTHER;
- local_nid = preferred_zone->node;
- }
-
- if (z->node == local_nid) {
- __inc_zone_state(z, NUMA_HIT);
- __inc_zone_state(z, local_stat);
- } else {
- __inc_zone_state(z, NUMA_MISS);
- __inc_zone_state(preferred_zone, NUMA_FOREIGN);
- }
-}
-
-/*
* Determine the per node value of a stat item.
*/
unsigned long node_page_state(int node, enum zone_stat_item item)
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-25 13:20 +0200 |
| Subject | Re: [PATCH 04/28] mm, page_alloc: Inline zone_statistics |
| Message-ID | <rrMIy-4QJ-23@gated-at.bofh.it> |
| In reply to | #1379567 |
On 04/15/2016 10:58 AM, Mel Gorman wrote: > zone_statistics has one call-site but it's a public function. Make > it static and inline. > > The performance difference on a page allocator microbenchmark is; > > 4.6.0-rc2 4.6.0-rc2 > statbranch-v1r20 statinline-v1r20 > Min alloc-odr0-1 419.00 ( 0.00%) 412.00 ( 1.67%) > Min alloc-odr0-2 305.00 ( 0.00%) 301.00 ( 1.31%) > Min alloc-odr0-4 250.00 ( 0.00%) 247.00 ( 1.20%) > Min alloc-odr0-8 219.00 ( 0.00%) 215.00 ( 1.83%) > Min alloc-odr0-16 203.00 ( 0.00%) 199.00 ( 1.97%) > Min alloc-odr0-32 195.00 ( 0.00%) 191.00 ( 2.05%) > Min alloc-odr0-64 191.00 ( 0.00%) 187.00 ( 2.09%) > Min alloc-odr0-128 189.00 ( 0.00%) 185.00 ( 2.12%) > Min alloc-odr0-256 198.00 ( 0.00%) 193.00 ( 2.53%) > Min alloc-odr0-512 210.00 ( 0.00%) 207.00 ( 1.43%) > Min alloc-odr0-1024 216.00 ( 0.00%) 213.00 ( 1.39%) > Min alloc-odr0-2048 221.00 ( 0.00%) 220.00 ( 0.45%) > Min alloc-odr0-4096 227.00 ( 0.00%) 226.00 ( 0.44%) > Min alloc-odr0-8192 232.00 ( 0.00%) 229.00 ( 1.29%) > Min alloc-odr0-16384 232.00 ( 0.00%) 229.00 ( 1.29%) > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Vlastimil Babka <vbabka@suse.cz>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath |
| Message-ID | <ro7Vg-Er-7@gated-at.bofh.it> |
| In reply to | #1379561 |
There is a debugging check for callers that specify __GFP_DIRECT_RECLAIM from a context that cannot sleep. Triggering this is almost certainly a bug but it's also overhead in the fast path. Move the check to the slow path. It'll be harder to trigger as it'll only be checked when watermarks are depleted but it'll also only be checked in a path that can sleep. Signed-off-by: Mel Gorman <mgorman@techsingularity.net> --- mm/page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 21aaef6ddd7a..9ef2f4ab9ca5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3176,6 +3176,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, return NULL; } + might_sleep_if(gfp_mask & __GFP_DIRECT_RECLAIM); + /* * We also sanity check to catch abuse of atomic reserves being used by * callers that are not in atomic context. @@ -3369,8 +3371,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, lockdep_trace_alloc(gfp_mask); - might_sleep_if(gfp_mask & __GFP_DIRECT_RECLAIM); - if (should_fail_alloc_page(gfp_mask, order)) return NULL; -- 2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 15:50 +0200 |
| Subject | Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath |
| Message-ID | <rsbxg-eJ-19@gated-at.bofh.it> |
| In reply to | #1379568 |
On 04/15/2016 11:07 AM, Mel Gorman wrote: > There is a debugging check for callers that specify __GFP_DIRECT_RECLAIM > from a context that cannot sleep. Triggering this is almost certainly > a bug but it's also overhead in the fast path. For CONFIG_DEBUG_ATOMIC_SLEEP, enabling is asking for the overhead. But for CONFIG_PREEMPT_VOLUNTARY which turns it into _cond_resched(), I guess it's not. > Move the check to the slow > path. It'll be harder to trigger as it'll only be checked when watermarks > are depleted but it'll also only be checked in a path that can sleep. Hmm what about zone_reclaim_mode=1, should the check be also duplicated to that part of get_page_from_freelist()? > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> > --- > mm/page_alloc.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 21aaef6ddd7a..9ef2f4ab9ca5 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -3176,6 +3176,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > return NULL; > } > > + might_sleep_if(gfp_mask & __GFP_DIRECT_RECLAIM); > + > /* > * We also sanity check to catch abuse of atomic reserves being used by > * callers that are not in atomic context. > @@ -3369,8 +3371,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, > > lockdep_trace_alloc(gfp_mask); > > - might_sleep_if(gfp_mask & __GFP_DIRECT_RECLAIM); > - > if (should_fail_alloc_page(gfp_mask, order)) > return NULL; > >
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-26 17:00 +0200 |
| Subject | Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath |
| Message-ID | <rscD0-11Z-25@gated-at.bofh.it> |
| In reply to | #1387490 |
On Tue, Apr 26, 2016 at 03:41:22PM +0200, Vlastimil Babka wrote: > On 04/15/2016 11:07 AM, Mel Gorman wrote: > >There is a debugging check for callers that specify __GFP_DIRECT_RECLAIM > >from a context that cannot sleep. Triggering this is almost certainly > >a bug but it's also overhead in the fast path. > > For CONFIG_DEBUG_ATOMIC_SLEEP, enabling is asking for the overhead. But for > CONFIG_PREEMPT_VOLUNTARY which turns it into _cond_resched(), I guess it's > not. > Either way, it struck me as odd. It does depend on the config and it's marginal so if there is a problem then I can drop it. > >Move the check to the slow > >path. It'll be harder to trigger as it'll only be checked when watermarks > >are depleted but it'll also only be checked in a path that can sleep. > > Hmm what about zone_reclaim_mode=1, should the check be also duplicated to > that part of get_page_from_freelist()? > zone_reclaim has a !gfpflags_allow_blocking() check, does not call cond_resched() before that check so it does not fall into an accidental sleep path. I'm not seeing why the check is necessary there. -- Mel Gorman SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 17:20 +0200 |
| Subject | Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath |
| Message-ID | <rscWm-1qW-25@gated-at.bofh.it> |
| In reply to | #1387577 |
On 04/26/2016 04:50 PM, Mel Gorman wrote: > On Tue, Apr 26, 2016 at 03:41:22PM +0200, Vlastimil Babka wrote: >> On 04/15/2016 11:07 AM, Mel Gorman wrote: >> >There is a debugging check for callers that specify __GFP_DIRECT_RECLAIM >> >from a context that cannot sleep. Triggering this is almost certainly >> >a bug but it's also overhead in the fast path. >> >> For CONFIG_DEBUG_ATOMIC_SLEEP, enabling is asking for the overhead. But for >> CONFIG_PREEMPT_VOLUNTARY which turns it into _cond_resched(), I guess it's >> not. >> > > Either way, it struck me as odd. It does depend on the config and it's > marginal so if there is a problem then I can drop it. What I tried to say is that it makes sense, but it's perhaps non-obvious :) >> >Move the check to the slow >> >path. It'll be harder to trigger as it'll only be checked when watermarks >> >are depleted but it'll also only be checked in a path that can sleep. >> >> Hmm what about zone_reclaim_mode=1, should the check be also duplicated to >> that part of get_page_from_freelist()? >> > > zone_reclaim has a !gfpflags_allow_blocking() check, does not call > cond_resched() before that check so it does not fall into an accidental > sleep path. I'm not seeing why the check is necessary there. Hmm I thought the primary purpose of this might_sleep_if() is to catch those (via the DEBUG_ATOMIC_SLEEP) that do pass __GFP_DIRECT_RECLAIM (which means gfpflags_allow_blocking() will be true and zone_reclaim will proceed), but do so from the wrong context. Am I getting that wrong?
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-26 18:30 +0200 |
| Subject | Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath |
| Message-ID | <rse26-2io-5@gated-at.bofh.it> |
| In reply to | #1387612 |
On Tue, Apr 26, 2016 at 05:16:21PM +0200, Vlastimil Babka wrote: > On 04/26/2016 04:50 PM, Mel Gorman wrote: > >On Tue, Apr 26, 2016 at 03:41:22PM +0200, Vlastimil Babka wrote: > >>On 04/15/2016 11:07 AM, Mel Gorman wrote: > >>>There is a debugging check for callers that specify __GFP_DIRECT_RECLAIM > >>>from a context that cannot sleep. Triggering this is almost certainly > >>>a bug but it's also overhead in the fast path. > >> > >>For CONFIG_DEBUG_ATOMIC_SLEEP, enabling is asking for the overhead. But for > >>CONFIG_PREEMPT_VOLUNTARY which turns it into _cond_resched(), I guess it's > >>not. > >> > > > >Either way, it struck me as odd. It does depend on the config and it's > >marginal so if there is a problem then I can drop it. > > What I tried to say is that it makes sense, but it's perhaps non-obvious :) > > >>>Move the check to the slow > >>>path. It'll be harder to trigger as it'll only be checked when watermarks > >>>are depleted but it'll also only be checked in a path that can sleep. > >> > >>Hmm what about zone_reclaim_mode=1, should the check be also duplicated to > >>that part of get_page_from_freelist()? > >> > > > >zone_reclaim has a !gfpflags_allow_blocking() check, does not call > >cond_resched() before that check so it does not fall into an accidental > >sleep path. I'm not seeing why the check is necessary there. > > Hmm I thought the primary purpose of this might_sleep_if() is to catch those > (via the DEBUG_ATOMIC_SLEEP) that do pass __GFP_DIRECT_RECLAIM (which means > gfpflags_allow_blocking() will be true and zone_reclaim will proceed), It proceeds but fails immediately so what I'm failing to see is why moving the check increases risk. I wanted to remove the check from the path where the problem it's catching cannot happen. It does mean the debugging check is made less frequently but it's still useful. If you feel the safety is preferred then I'll drop the patch. -- Mel Gorman SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 19/28] mm, page_alloc: Reduce cost of fair zone allocation policy retry |
| Message-ID | <ro7Vh-Er-17@gated-at.bofh.it> |
| In reply to | #1379561 |
The fair zone allocation policy is not without cost but it can be reduced
slightly. This patch removes an unnecessary local variable, checks the
likely conditions of the fair zone policy first, uses a bool instead of
a flags check and falls through when a remote node is encountered instead
of doing a full restart. The benefit is marginal but it's there
4.6.0-rc2 4.6.0-rc2
decstat-v1r20 optfair-v1r20
Min alloc-odr0-1 377.00 ( 0.00%) 380.00 ( -0.80%)
Min alloc-odr0-2 273.00 ( 0.00%) 273.00 ( 0.00%)
Min alloc-odr0-4 226.00 ( 0.00%) 227.00 ( -0.44%)
Min alloc-odr0-8 196.00 ( 0.00%) 196.00 ( 0.00%)
Min alloc-odr0-16 183.00 ( 0.00%) 183.00 ( 0.00%)
Min alloc-odr0-32 175.00 ( 0.00%) 173.00 ( 1.14%)
Min alloc-odr0-64 172.00 ( 0.00%) 169.00 ( 1.74%)
Min alloc-odr0-128 170.00 ( 0.00%) 169.00 ( 0.59%)
Min alloc-odr0-256 183.00 ( 0.00%) 180.00 ( 1.64%)
Min alloc-odr0-512 191.00 ( 0.00%) 190.00 ( 0.52%)
Min alloc-odr0-1024 199.00 ( 0.00%) 198.00 ( 0.50%)
Min alloc-odr0-2048 204.00 ( 0.00%) 204.00 ( 0.00%)
Min alloc-odr0-4096 210.00 ( 0.00%) 209.00 ( 0.48%)
Min alloc-odr0-8192 213.00 ( 0.00%) 213.00 ( 0.00%)
Min alloc-odr0-16384 214.00 ( 0.00%) 214.00 ( 0.00%)
The benefit is marginal at best but one of the most important benefits,
avoiding a second search when falling back to another node is not triggered
by this particular test so the benefit for some corner cases is understated.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 32 ++++++++++++++------------------
1 file changed, 14 insertions(+), 18 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 7a5f6ff4ea06..98b443c97be6 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2676,12 +2676,10 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
{
struct zoneref *z;
struct zone *zone;
- bool fair_skipped;
- bool zonelist_rescan;
+ bool fair_skipped = false;
+ bool apply_fair = (alloc_flags & ALLOC_FAIR);
zonelist_scan:
- zonelist_rescan = false;
-
/*
* Scan zonelist, looking for a zone with enough free.
* See also __cpuset_node_allowed() comment in kernel/cpuset.c.
@@ -2701,13 +2699,16 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
* page was allocated in should have no effect on the
* time the page has in memory before being reclaimed.
*/
- if (alloc_flags & ALLOC_FAIR) {
- if (!zone_local(ac->preferred_zone, zone))
- break;
+ if (apply_fair) {
if (test_bit(ZONE_FAIR_DEPLETED, &zone->flags)) {
fair_skipped = true;
continue;
}
+ if (!zone_local(ac->preferred_zone, zone)) {
+ if (fair_skipped)
+ goto reset_fair;
+ apply_fair = false;
+ }
}
/*
* When allocating a page cache page for writing, we
@@ -2796,18 +2797,13 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
* include remote zones now, before entering the slowpath and waking
* kswapd: prefer spilling to a remote zone over swapping locally.
*/
- if (alloc_flags & ALLOC_FAIR) {
- alloc_flags &= ~ALLOC_FAIR;
- if (fair_skipped) {
- zonelist_rescan = true;
- reset_alloc_batches(ac->preferred_zone);
- }
- if (nr_online_nodes > 1)
- zonelist_rescan = true;
- }
-
- if (zonelist_rescan)
+ if (fair_skipped) {
+reset_fair:
+ apply_fair = false;
+ fair_skipped = false;
+ reset_alloc_batches(ac->preferred_zone);
goto zonelist_scan;
+ }
return NULL;
}
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist |
| Message-ID | <ro7Vg-Er-9@gated-at.bofh.it> |
| In reply to | #1379561 |
A check is made for an empty zonelist early in the page allocator fast path but it's unnecessary. When get_page_from_freelist() is called, it'll return NULL immediately. Removing the first check is slower for machines with memoryless nodes but that is a corner case that can live with the overhead. Signed-off-by: Mel Gorman <mgorman@techsingularity.net> --- mm/page_alloc.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index df03ccc7f07c..21aaef6ddd7a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3374,14 +3374,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, if (should_fail_alloc_page(gfp_mask, order)) return NULL; - /* - * Check the zones suitable for the gfp_mask contain at least one - * valid zone. It's possible to have an empty zonelist as a result - * of __GFP_THISNODE and a memoryless node - */ - if (unlikely(!zonelist->_zonerefs->zone)) - return NULL; - if (IS_ENABLED(CONFIG_CMA) && ac.migratetype == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; @@ -3394,8 +3386,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, /* The preferred zone is used for statistics later */ preferred_zoneref = first_zones_zonelist(ac.zonelist, ac.high_zoneidx, ac.nodemask, &ac.preferred_zone); - if (!ac.preferred_zone) - goto out; ac.classzone_idx = zonelist_zone_idx(preferred_zoneref); /* First allocation attempt */ @@ -3418,7 +3408,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, trace_mm_page_alloc(page, order, alloc_mask, ac.migratetype); -out: /* * When updating a task's mems_allowed, it is possible to race with * parallel threads in such a way that an allocation can fail while -- 2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 22/28] mm, page_alloc: Remove field from alloc_context |
| Message-ID | <ro7Vg-Er-15@gated-at.bofh.it> |
| In reply to | #1379570 |
The classzone_idx can be inferred from preferred_zoneref so remove the
unnecessary field and save stack space.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/compaction.c | 4 ++--
mm/internal.h | 3 ++-
mm/page_alloc.c | 7 +++----
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index 244bb669b5a6..c2fb3c61f1b6 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1536,7 +1536,7 @@ unsigned long try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
status = compact_zone_order(zone, order, gfp_mask, mode,
&zone_contended, alloc_flags,
- ac->classzone_idx);
+ ac_classzone_idx(ac));
rc = max(status, rc);
/*
* It takes at least one zone that wasn't lock contended
@@ -1546,7 +1546,7 @@ unsigned long try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
/* If a normal allocation would succeed, stop compacting */
if (zone_watermark_ok(zone, order, low_wmark_pages(zone),
- ac->classzone_idx, alloc_flags)) {
+ ac_classzone_idx(ac), alloc_flags)) {
/*
* We think the allocation will succeed in this zone,
* but it is not certain, hence the false. The caller
diff --git a/mm/internal.h b/mm/internal.h
index 4c2396cd514c..3bf62e085b16 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -103,12 +103,13 @@ struct alloc_context {
struct zonelist *zonelist;
nodemask_t *nodemask;
struct zoneref *preferred_zoneref;
- int classzone_idx;
int migratetype;
enum zone_type high_zoneidx;
bool spread_dirty_pages;
};
+#define ac_classzone_idx(ac) zonelist_zone_idx(ac->preferred_zoneref)
+
/*
* Locate the struct page for both the matching buddy in our
* pair (buddy1) and the combined O(n+1) page they form (page).
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 897e9d2a8500..bc754d32aed6 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2767,7 +2767,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
mark = zone->watermark[alloc_flags & ALLOC_WMARK_MASK];
if (!zone_watermark_fast(zone, order, mark,
- ac->classzone_idx, alloc_flags)) {
+ ac_classzone_idx(ac), alloc_flags)) {
int ret;
/* Checked here to keep the fast path fast */
@@ -2790,7 +2790,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
default:
/* did we reclaim enough */
if (zone_watermark_ok(zone, order, mark,
- ac->classzone_idx, alloc_flags))
+ ac_classzone_idx(ac), alloc_flags))
goto try_this_zone;
continue;
@@ -3114,7 +3114,7 @@ static void wake_all_kswapds(unsigned int order, const struct alloc_context *ac)
for_each_zone_zonelist_nodemask(zone, z, ac->zonelist,
ac->high_zoneidx, ac->nodemask)
- wakeup_kswapd(zone, order, zonelist_zone_idx(ac->preferred_zoneref));
+ wakeup_kswapd(zone, order, ac_classzone_idx(ac));
}
static inline unsigned int
@@ -3409,7 +3409,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
/* The preferred zone is used for statistics later */
ac.preferred_zoneref = first_zones_zonelist(ac.zonelist, ac.high_zoneidx,
ac.nodemask);
- ac.classzone_idx = zonelist_zone_idx(ac.preferred_zoneref);
/* First allocation attempt */
page = get_page_from_freelist(alloc_mask, order, alloc_flags, &ac);
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 14/28] mm, page_alloc: Simplify last cpupid reset |
| Message-ID | <ro7Vh-Er-25@gated-at.bofh.it> |
| In reply to | #1379570 |
The current reset unnecessarily clears flags and makes pointless calculations.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
include/linux/mm.h | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index ffcff53e3b2b..60656db00abd 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -837,10 +837,7 @@ extern int page_cpupid_xchg_last(struct page *page, int cpupid);
static inline void page_cpupid_reset_last(struct page *page)
{
- int cpupid = (1 << LAST_CPUPID_SHIFT) - 1;
-
- page->flags &= ~(LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT);
- page->flags |= (cpupid & LAST_CPUPID_MASK) << LAST_CPUPID_PGSHIFT;
+ page->flags |= LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT;
}
#endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
#else /* !CONFIG_NUMA_BALANCING */
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 15:40 +0200 |
| Subject | Re: [PATCH 14/28] mm, page_alloc: Simplify last cpupid reset |
| Message-ID | <rsbnA-9L-15@gated-at.bofh.it> |
| In reply to | #1379575 |
On 04/15/2016 11:07 AM, Mel Gorman wrote:
> The current reset unnecessarily clears flags and makes pointless calculations.
Ugh, indeed.
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
> ---
> include/linux/mm.h | 5 +----
> 1 file changed, 1 insertion(+), 4 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index ffcff53e3b2b..60656db00abd 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -837,10 +837,7 @@ extern int page_cpupid_xchg_last(struct page *page, int cpupid);
>
> static inline void page_cpupid_reset_last(struct page *page)
> {
> - int cpupid = (1 << LAST_CPUPID_SHIFT) - 1;
> -
> - page->flags &= ~(LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT);
> - page->flags |= (cpupid & LAST_CPUPID_MASK) << LAST_CPUPID_PGSHIFT;
> + page->flags |= LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT;
> }
> #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
> #else /* !CONFIG_NUMA_BALANCING */
>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 20/28] mm, page_alloc: Shortcut watermark checks for order-0 pages |
| Message-ID | <ro7Vi-Er-41@gated-at.bofh.it> |
| In reply to | #1379570 |
Watermarks have to be checked on every allocation including the number of
pages being allocated and whether reserves can be accessed. The reserves
only matter if memory is limited and the free_pages adjustment only applies
to high-order pages. This patch adds a shortcut for order-0 pages that avoids
numerous calculations if there is plenty of free memory yielding the following
performance difference in a page allocator microbenchmark;
4.6.0-rc2 4.6.0-rc2
optfair-v1r20 fastmark-v1r20
Min alloc-odr0-1 380.00 ( 0.00%) 364.00 ( 4.21%)
Min alloc-odr0-2 273.00 ( 0.00%) 262.00 ( 4.03%)
Min alloc-odr0-4 227.00 ( 0.00%) 214.00 ( 5.73%)
Min alloc-odr0-8 196.00 ( 0.00%) 186.00 ( 5.10%)
Min alloc-odr0-16 183.00 ( 0.00%) 173.00 ( 5.46%)
Min alloc-odr0-32 173.00 ( 0.00%) 165.00 ( 4.62%)
Min alloc-odr0-64 169.00 ( 0.00%) 161.00 ( 4.73%)
Min alloc-odr0-128 169.00 ( 0.00%) 159.00 ( 5.92%)
Min alloc-odr0-256 180.00 ( 0.00%) 168.00 ( 6.67%)
Min alloc-odr0-512 190.00 ( 0.00%) 180.00 ( 5.26%)
Min alloc-odr0-1024 198.00 ( 0.00%) 190.00 ( 4.04%)
Min alloc-odr0-2048 204.00 ( 0.00%) 196.00 ( 3.92%)
Min alloc-odr0-4096 209.00 ( 0.00%) 202.00 ( 3.35%)
Min alloc-odr0-8192 213.00 ( 0.00%) 206.00 ( 3.29%)
Min alloc-odr0-16384 214.00 ( 0.00%) 206.00 ( 3.74%)
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 28 +++++++++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 98b443c97be6..8923d74b1707 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2619,6 +2619,32 @@ bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
zone_page_state(z, NR_FREE_PAGES));
}
+static inline bool zone_watermark_fast(struct zone *z, unsigned int order,
+ unsigned long mark, int classzone_idx, unsigned int alloc_flags)
+{
+ long free_pages = zone_page_state(z, NR_FREE_PAGES);
+ long cma_pages = 0;
+
+#ifdef CONFIG_CMA
+ /* If allocation can't use CMA areas don't use free CMA pages */
+ if (!(alloc_flags & ALLOC_CMA))
+ cma_pages = zone_page_state(z, NR_FREE_CMA_PAGES);
+#endif
+
+ /*
+ * Fast check for order-0 only. If this fails then the reserves
+ * need to be calculated. There is a corner case where the check
+ * passes but only the high-order atomic reserve are free. If
+ * the caller is !atomic then it'll uselessly search the free
+ * list. That corner case is then slower but it is harmless.
+ */
+ if (!order && (free_pages - cma_pages) > mark + z->lowmem_reserve[classzone_idx])
+ return true;
+
+ return __zone_watermark_ok(z, order, mark, classzone_idx, alloc_flags,
+ free_pages);
+}
+
bool zone_watermark_ok_safe(struct zone *z, unsigned int order,
unsigned long mark, int classzone_idx)
{
@@ -2740,7 +2766,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
continue;
mark = zone->watermark[alloc_flags & ALLOC_WMARK_MASK];
- if (!zone_watermark_ok(zone, order, mark,
+ if (!zone_watermark_fast(zone, order, mark,
ac->classzone_idx, alloc_flags)) {
int ret;
--
2.6.4
[toc] | [prev] | [next] | [standalone]
Page 1 of 4 [1] 2 3 4 Next page →
Back to top | Article view | linux.kernel
csiph-web