Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1205933 > unrolled thread
| Started by | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| First post | 2015-08-12 12:50 +0200 |
| Last post | 2015-08-20 17:00 +0200 |
| Articles | 7 — 3 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
[PATCH 01/10] mm, page_alloc: Delete the zonelist_cache Mel Gorman <mgorman@techsingularity.net> - 2015-08-12 12:50 +0200
Re: [PATCH 01/10] mm, page_alloc: Delete the zonelist_cache Michal Hocko <mhocko@kernel.org> - 2015-08-20 15:20 +0200
Re: [PATCH 01/10] mm, page_alloc: Delete the zonelist_cache Mel Gorman <mgorman@techsingularity.net> - 2015-08-20 15:50 +0200
Re: [PATCH 01/10] mm, page_alloc: Delete the zonelist_cache Michal Hocko <mhocko@kernel.org> - 2015-08-21 11:30 +0200
Re: [PATCH 01/10] mm, page_alloc: Delete the zonelist_cache Vlastimil Babka <vbabka@suse.cz> - 2015-08-20 15:40 +0200
Re: [PATCH 01/10] mm, page_alloc: Delete the zonelist_cache Mel Gorman <mgorman@techsingularity.net> - 2015-08-20 16:20 +0200
Re: [PATCH 01/10] mm, page_alloc: Delete the zonelist_cache Vlastimil Babka <vbabka@suse.cz> - 2015-08-20 17:00 +0200
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-12 12:50 +0200 |
| Subject | [PATCH 01/10] mm, page_alloc: Delete the zonelist_cache |
| Message-ID | <pWBLz-1Xg-5@gated-at.bofh.it> |
The zonelist cache (zlc) was introduced to skip over zones that were
recently known to be full. This avoided expensive operations such as the
cpuset checks, watermark calculations and zone_reclaim. The situation
today is different and the complexity of zlc is harder to justify.
1) The cpuset checks are no-ops unless a cpuset is active and in general are
a lot cheaper.
2) zone_reclaim is now disabled by default and I suspect that was a large
source of the cost that zlc wanted to avoid. When it is enabled, it's
known to be a major source of stalling when nodes fill up and it's
unwise to hit every other user with the overhead.
3) Watermark checks are expensive to calculate for high-order
allocation requests. Later patches in this series will reduce the cost
of the watermark checking.
4) The most important issue is that in the current implementation it
is possible for a failed THP allocation to mark a zone full for order-0
allocations and cause a fallback to remote nodes.
The last issue could be addressed with additional complexity but as the
benefit of zlc is questionable, it is better to remove it. If stalls
due to zone_reclaim are ever reported then an alternative would be to
introduce deferring logic based on a timeout inside zone_reclaim itself
and leave the page allocator fast paths alone.
The impact on page-allocator microbenchmarks is negligible as they don't
hit the paths where the zlc comes into play. The impact was noticeable
in a workload called "stutter". One part uses a lot of anonymous memory,
a second measures mmap latency and a third copies a large file. In an
ideal world the latency application would not notice the mmap latency.
On a 4-node machine the results of this patch are
4-node machine stutter
4.2.0-rc1 4.2.0-rc1
vanilla nozlc-v1r20
Min mmap 53.9902 ( 0.00%) 49.3629 ( 8.57%)
1st-qrtle mmap 54.6776 ( 0.00%) 54.1201 ( 1.02%)
2nd-qrtle mmap 54.9242 ( 0.00%) 54.5961 ( 0.60%)
3rd-qrtle mmap 55.1817 ( 0.00%) 54.9338 ( 0.45%)
Max-90% mmap 55.3952 ( 0.00%) 55.3929 ( 0.00%)
Max-93% mmap 55.4766 ( 0.00%) 57.5712 ( -3.78%)
Max-95% mmap 55.5522 ( 0.00%) 57.8376 ( -4.11%)
Max-99% mmap 55.7938 ( 0.00%) 63.6180 (-14.02%)
Max mmap 6344.0292 ( 0.00%) 67.2477 ( 98.94%)
Mean mmap 57.3732 ( 0.00%) 54.5680 ( 4.89%)
Note the maximum stall latency which was 6 seconds and becomes 67ms with
this patch applied. However, also note that it is not guaranteed this
benchmark always hits pathelogical cases and the milage varies. There is
a secondary impact with more direct reclaim because zones are now being
considered instead of being skipped by zlc.
4.1.0 4.1.0
vanilla nozlc-v1r4
Swap Ins 838 502
Swap Outs 1149395 2622895
DMA32 allocs 17839113 15863747
Normal allocs 129045707 137847920
Direct pages scanned 4070089 29046893
Kswapd pages scanned 17147837 17140694
Kswapd pages reclaimed 17146691 17139601
Direct pages reclaimed 1888879 4886630
Kswapd efficiency 99% 99%
Kswapd velocity 17523.721 17518.928
Direct efficiency 46% 16%
Direct velocity 4159.306 29687.854
Percentage direct scans 19% 62%
Page writes by reclaim 1149395.000 2622895.000
Page writes file 0 0
Page writes anon 1149395 2622895
The direct page scan and reclaim rates are noticeable. It is possible
this will not be a universal win on all workloads but cycling through
zonelists waiting for zlc->last_full_zap to expire is not the right
decision.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: David Rientjes <rientjes@google.com>
---
include/linux/mmzone.h | 71 ----------------
mm/page_alloc.c | 217 +------------------------------------------------
2 files changed, 2 insertions(+), 286 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 754c25966a0a..decc99a007f5 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -585,75 +585,8 @@ static inline bool zone_is_empty(struct zone *zone)
* [1] : No fallback (__GFP_THISNODE)
*/
#define MAX_ZONELISTS 2
-
-
-/*
- * We cache key information from each zonelist for smaller cache
- * footprint when scanning for free pages in get_page_from_freelist().
- *
- * 1) The BITMAP fullzones tracks which zones in a zonelist have come
- * up short of free memory since the last time (last_fullzone_zap)
- * we zero'd fullzones.
- * 2) The array z_to_n[] maps each zone in the zonelist to its node
- * id, so that we can efficiently evaluate whether that node is
- * set in the current tasks mems_allowed.
- *
- * Both fullzones and z_to_n[] are one-to-one with the zonelist,
- * indexed by a zones offset in the zonelist zones[] array.
- *
- * The get_page_from_freelist() routine does two scans. During the
- * first scan, we skip zones whose corresponding bit in 'fullzones'
- * is set or whose corresponding node in current->mems_allowed (which
- * comes from cpusets) is not set. During the second scan, we bypass
- * this zonelist_cache, to ensure we look methodically at each zone.
- *
- * Once per second, we zero out (zap) fullzones, forcing us to
- * reconsider nodes that might have regained more free memory.
- * The field last_full_zap is the time we last zapped fullzones.
- *
- * This mechanism reduces the amount of time we waste repeatedly
- * reexaming zones for free memory when they just came up low on
- * memory momentarilly ago.
- *
- * The zonelist_cache struct members logically belong in struct
- * zonelist. However, the mempolicy zonelists constructed for
- * MPOL_BIND are intentionally variable length (and usually much
- * shorter). A general purpose mechanism for handling structs with
- * multiple variable length members is more mechanism than we want
- * here. We resort to some special case hackery instead.
- *
- * The MPOL_BIND zonelists don't need this zonelist_cache (in good
- * part because they are shorter), so we put the fixed length stuff
- * at the front of the zonelist struct, ending in a variable length
- * zones[], as is needed by MPOL_BIND.
- *
- * Then we put the optional zonelist cache on the end of the zonelist
- * struct. This optional stuff is found by a 'zlcache_ptr' pointer in
- * the fixed length portion at the front of the struct. This pointer
- * both enables us to find the zonelist cache, and in the case of
- * MPOL_BIND zonelists, (which will just set the zlcache_ptr to NULL)
- * to know that the zonelist cache is not there.
- *
- * The end result is that struct zonelists come in two flavors:
- * 1) The full, fixed length version, shown below, and
- * 2) The custom zonelists for MPOL_BIND.
- * The custom MPOL_BIND zonelists have a NULL zlcache_ptr and no zlcache.
- *
- * Even though there may be multiple CPU cores on a node modifying
- * fullzones or last_full_zap in the same zonelist_cache at the same
- * time, we don't lock it. This is just hint data - if it is wrong now
- * and then, the allocator will still function, perhaps a bit slower.
- */
-
-
-struct zonelist_cache {
- unsigned short z_to_n[MAX_ZONES_PER_ZONELIST]; /* zone->nid */
- DECLARE_BITMAP(fullzones, MAX_ZONES_PER_ZONELIST); /* zone full? */
- unsigned long last_full_zap; /* when last zap'd (jiffies) */
-};
#else
#define MAX_ZONELISTS 1
-struct zonelist_cache;
#endif
/*
@@ -683,11 +616,7 @@ struct zoneref {
* zonelist_node_idx() - Return the index of the node for an entry
*/
struct zonelist {
- struct zonelist_cache *zlcache_ptr; // NULL or &zlcache
struct zoneref _zonerefs[MAX_ZONES_PER_ZONELIST + 1];
-#ifdef CONFIG_NUMA
- struct zonelist_cache zlcache; // optional ...
-#endif
};
#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index ef19f22b2b7d..41c0799b9049 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2250,122 +2250,6 @@ bool zone_watermark_ok_safe(struct zone *z, unsigned int order,
}
#ifdef CONFIG_NUMA
-/*
- * zlc_setup - Setup for "zonelist cache". Uses cached zone data to
- * skip over zones that are not allowed by the cpuset, or that have
- * been recently (in last second) found to be nearly full. See further
- * comments in mmzone.h. Reduces cache footprint of zonelist scans
- * that have to skip over a lot of full or unallowed zones.
- *
- * If the zonelist cache is present in the passed zonelist, then
- * returns a pointer to the allowed node mask (either the current
- * tasks mems_allowed, or node_states[N_MEMORY].)
- *
- * If the zonelist cache is not available for this zonelist, does
- * nothing and returns NULL.
- *
- * If the fullzones BITMAP in the zonelist cache is stale (more than
- * a second since last zap'd) then we zap it out (clear its bits.)
- *
- * We hold off even calling zlc_setup, until after we've checked the
- * first zone in the zonelist, on the theory that most allocations will
- * be satisfied from that first zone, so best to examine that zone as
- * quickly as we can.
- */
-static nodemask_t *zlc_setup(struct zonelist *zonelist, int alloc_flags)
-{
- struct zonelist_cache *zlc; /* cached zonelist speedup info */
- nodemask_t *allowednodes; /* zonelist_cache approximation */
-
- zlc = zonelist->zlcache_ptr;
- if (!zlc)
- return NULL;
-
- if (time_after(jiffies, zlc->last_full_zap + HZ)) {
- bitmap_zero(zlc->fullzones, MAX_ZONES_PER_ZONELIST);
- zlc->last_full_zap = jiffies;
- }
-
- allowednodes = !in_interrupt() && (alloc_flags & ALLOC_CPUSET) ?
- &cpuset_current_mems_allowed :
- &node_states[N_MEMORY];
- return allowednodes;
-}
-
-/*
- * Given 'z' scanning a zonelist, run a couple of quick checks to see
- * if it is worth looking at further for free memory:
- * 1) Check that the zone isn't thought to be full (doesn't have its
- * bit set in the zonelist_cache fullzones BITMAP).
- * 2) Check that the zones node (obtained from the zonelist_cache
- * z_to_n[] mapping) is allowed in the passed in allowednodes mask.
- * Return true (non-zero) if zone is worth looking at further, or
- * else return false (zero) if it is not.
- *
- * This check -ignores- the distinction between various watermarks,
- * such as GFP_HIGH, GFP_ATOMIC, PF_MEMALLOC, ... If a zone is
- * found to be full for any variation of these watermarks, it will
- * be considered full for up to one second by all requests, unless
- * we are so low on memory on all allowed nodes that we are forced
- * into the second scan of the zonelist.
- *
- * In the second scan we ignore this zonelist cache and exactly
- * apply the watermarks to all zones, even it is slower to do so.
- * We are low on memory in the second scan, and should leave no stone
- * unturned looking for a free page.
- */
-static int zlc_zone_worth_trying(struct zonelist *zonelist, struct zoneref *z,
- nodemask_t *allowednodes)
-{
- struct zonelist_cache *zlc; /* cached zonelist speedup info */
- int i; /* index of *z in zonelist zones */
- int n; /* node that zone *z is on */
-
- zlc = zonelist->zlcache_ptr;
- if (!zlc)
- return 1;
-
- i = z - zonelist->_zonerefs;
- n = zlc->z_to_n[i];
-
- /* This zone is worth trying if it is allowed but not full */
- return node_isset(n, *allowednodes) && !test_bit(i, zlc->fullzones);
-}
-
-/*
- * Given 'z' scanning a zonelist, set the corresponding bit in
- * zlc->fullzones, so that subsequent attempts to allocate a page
- * from that zone don't waste time re-examining it.
- */
-static void zlc_mark_zone_full(struct zonelist *zonelist, struct zoneref *z)
-{
- struct zonelist_cache *zlc; /* cached zonelist speedup info */
- int i; /* index of *z in zonelist zones */
-
- zlc = zonelist->zlcache_ptr;
- if (!zlc)
- return;
-
- i = z - zonelist->_zonerefs;
-
- set_bit(i, zlc->fullzones);
-}
-
-/*
- * clear all zones full, called after direct reclaim makes progress so that
- * a zone that was recently full is not skipped over for up to a second
- */
-static void zlc_clear_zones_full(struct zonelist *zonelist)
-{
- struct zonelist_cache *zlc; /* cached zonelist speedup info */
-
- zlc = zonelist->zlcache_ptr;
- if (!zlc)
- return;
-
- bitmap_zero(zlc->fullzones, MAX_ZONES_PER_ZONELIST);
-}
-
static bool zone_local(struct zone *local_zone, struct zone *zone)
{
return local_zone->node == zone->node;
@@ -2376,28 +2260,7 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
return node_distance(zone_to_nid(local_zone), zone_to_nid(zone)) <
RECLAIM_DISTANCE;
}
-
#else /* CONFIG_NUMA */
-
-static nodemask_t *zlc_setup(struct zonelist *zonelist, int alloc_flags)
-{
- return NULL;
-}
-
-static int zlc_zone_worth_trying(struct zonelist *zonelist, struct zoneref *z,
- nodemask_t *allowednodes)
-{
- return 1;
-}
-
-static void zlc_mark_zone_full(struct zonelist *zonelist, struct zoneref *z)
-{
-}
-
-static void zlc_clear_zones_full(struct zonelist *zonelist)
-{
-}
-
static bool zone_local(struct zone *local_zone, struct zone *zone)
{
return true;
@@ -2407,7 +2270,6 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
{
return true;
}
-
#endif /* CONFIG_NUMA */
static void reset_alloc_batches(struct zone *preferred_zone)
@@ -2434,9 +2296,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
struct zoneref *z;
struct page *page = NULL;
struct zone *zone;
- nodemask_t *allowednodes = NULL;/* zonelist_cache approximation */
- int zlc_active = 0; /* set if using zonelist_cache */
- int did_zlc_setup = 0; /* just call zlc_setup() one time */
bool consider_zone_dirty = (alloc_flags & ALLOC_WMARK_LOW) &&
(gfp_mask & __GFP_WRITE);
int nr_fair_skipped = 0;
@@ -2453,9 +2312,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
ac->nodemask) {
unsigned long mark;
- if (IS_ENABLED(CONFIG_NUMA) && zlc_active &&
- !zlc_zone_worth_trying(zonelist, z, allowednodes))
- continue;
if (cpusets_enabled() &&
(alloc_flags & ALLOC_CPUSET) &&
!cpuset_zone_allowed(zone, gfp_mask))
@@ -2513,28 +2369,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
if (alloc_flags & ALLOC_NO_WATERMARKS)
goto try_this_zone;
- if (IS_ENABLED(CONFIG_NUMA) &&
- !did_zlc_setup && nr_online_nodes > 1) {
- /*
- * we do zlc_setup if there are multiple nodes
- * and before considering the first zone allowed
- * by the cpuset.
- */
- allowednodes = zlc_setup(zonelist, alloc_flags);
- zlc_active = 1;
- did_zlc_setup = 1;
- }
-
if (zone_reclaim_mode == 0 ||
!zone_allows_reclaim(ac->preferred_zone, zone))
- goto this_zone_full;
-
- /*
- * As we may have just activated ZLC, check if the first
- * eligible zone has failed zone_reclaim recently.
- */
- if (IS_ENABLED(CONFIG_NUMA) && zlc_active &&
- !zlc_zone_worth_trying(zonelist, z, allowednodes))
continue;
ret = zone_reclaim(zone, gfp_mask, order);
@@ -2551,19 +2387,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
ac->classzone_idx, alloc_flags))
goto try_this_zone;
- /*
- * Failed to reclaim enough to meet watermark.
- * Only mark the zone full if checking the min
- * watermark or if we failed to reclaim just
- * 1<<order pages or else the page allocator
- * fastpath will prematurely mark zones full
- * when the watermark is between the low and
- * min watermarks.
- */
- if (((alloc_flags & ALLOC_WMARK_MASK) == ALLOC_WMARK_MIN) ||
- ret == ZONE_RECLAIM_SOME)
- goto this_zone_full;
-
continue;
}
}
@@ -2576,9 +2399,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
goto try_this_zone;
return page;
}
-this_zone_full:
- if (IS_ENABLED(CONFIG_NUMA) && zlc_active)
- zlc_mark_zone_full(zonelist, z);
}
/*
@@ -2599,12 +2419,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
zonelist_rescan = true;
}
- if (unlikely(IS_ENABLED(CONFIG_NUMA) && zlc_active)) {
- /* Disable zlc cache for second zonelist scan */
- zlc_active = 0;
- zonelist_rescan = true;
- }
-
if (zonelist_rescan)
goto zonelist_scan;
@@ -2844,10 +2658,6 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
if (unlikely(!(*did_some_progress)))
return NULL;
- /* After successful reclaim, reconsider all zones for allocation */
- if (IS_ENABLED(CONFIG_NUMA))
- zlc_clear_zones_full(ac->zonelist);
-
retry:
page = get_page_from_freelist(gfp_mask, order,
alloc_flags & ~ALLOC_NO_WATERMARKS, ac);
@@ -3157,7 +2967,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
gfp_t alloc_mask; /* The gfp_t that was actually used for allocation */
struct alloc_context ac = {
.high_zoneidx = gfp_zone(gfp_mask),
- .nodemask = nodemask,
+ .nodemask = nodemask ? : &cpuset_current_mems_allowed,
.migratetype = gfpflags_to_migratetype(gfp_mask),
};
@@ -3188,8 +2998,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
ac.zonelist = zonelist;
/* The preferred zone is used for statistics later */
preferred_zoneref = first_zones_zonelist(ac.zonelist, ac.high_zoneidx,
- ac.nodemask ? : &cpuset_current_mems_allowed,
- &ac.preferred_zone);
+ ac.nodemask, &ac.preferred_zone);
if (!ac.preferred_zone)
goto out;
ac.classzone_idx = zonelist_zone_idx(preferred_zoneref);
@@ -4169,20 +3978,6 @@ static void build_zonelists(pg_data_t *pgdat)
build_thisnode_zonelists(pgdat);
}
-/* Construct the zonelist performance cache - see further mmzone.h */
-static void build_zonelist_cache(pg_data_t *pgdat)
-{
- struct zonelist *zonelist;
- struct zonelist_cache *zlc;
- struct zoneref *z;
-
- zonelist = &pgdat->node_zonelists[0];
- zonelist->zlcache_ptr = zlc = &zonelist->zlcache;
- bitmap_zero(zlc->fullzones, MAX_ZONES_PER_ZONELIST);
- for (z = zonelist->_zonerefs; z->zone; z++)
- zlc->z_to_n[z - zonelist->_zonerefs] = zonelist_node_idx(z);
-}
-
#ifdef CONFIG_HAVE_MEMORYLESS_NODES
/*
* Return node id of node used for "local" allocations.
@@ -4243,12 +4038,6 @@ static void build_zonelists(pg_data_t *pgdat)
zonelist->_zonerefs[j].zone_idx = 0;
}
-/* non-NUMA variant of zonelist performance cache - just NULL zlcache_ptr */
-static void build_zonelist_cache(pg_data_t *pgdat)
-{
- pgdat->node_zonelists[0].zlcache_ptr = NULL;
-}
-
#endif /* CONFIG_NUMA */
/*
@@ -4289,14 +4078,12 @@ static int __build_all_zonelists(void *data)
if (self && !node_online(self->node_id)) {
build_zonelists(self);
- build_zonelist_cache(self);
}
for_each_online_node(nid) {
pg_data_t *pgdat = NODE_DATA(nid);
build_zonelists(pgdat);
- build_zonelist_cache(pgdat);
}
/*
--
2.4.6
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2015-08-20 15:20 +0200 |
| Message-ID | <pZxV9-5W7-15@gated-at.bofh.it> |
| In reply to | #1205933 |
On Wed 12-08-15 11:45:26, Mel Gorman wrote: [...] > 4-node machine stutter > 4-node machine stutter > 4.2.0-rc1 4.2.0-rc1 > vanilla nozlc-v1r20 > Min mmap 53.9902 ( 0.00%) 49.3629 ( 8.57%) > 1st-qrtle mmap 54.6776 ( 0.00%) 54.1201 ( 1.02%) > 2nd-qrtle mmap 54.9242 ( 0.00%) 54.5961 ( 0.60%) > 3rd-qrtle mmap 55.1817 ( 0.00%) 54.9338 ( 0.45%) > Max-90% mmap 55.3952 ( 0.00%) 55.3929 ( 0.00%) > Max-93% mmap 55.4766 ( 0.00%) 57.5712 ( -3.78%) > Max-95% mmap 55.5522 ( 0.00%) 57.8376 ( -4.11%) > Max-99% mmap 55.7938 ( 0.00%) 63.6180 (-14.02%) > Max mmap 6344.0292 ( 0.00%) 67.2477 ( 98.94%) > Mean mmap 57.3732 ( 0.00%) 54.5680 ( 4.89%) Do you have data for other leads? Because the reclaim counters look quite discouraging to be honest. > 4.1.0 4.1.0 > vanilla nozlc-v1r4 > Swap Ins 838 502 > Swap Outs 1149395 2622895 Twice as much swapouts is a lot. > DMA32 allocs 17839113 15863747 > Normal allocs 129045707 137847920 > Direct pages scanned 4070089 29046893 7x more scanns by direct reclaim also sounds bad. > Kswapd pages scanned 17147837 17140694 while kswapd is doing the same amount of work so we are moving considerable amount of reclaim activity into the direct reclaim > Kswapd pages reclaimed 17146691 17139601 > Direct pages reclaimed 1888879 4886630 > Kswapd efficiency 99% 99% > Kswapd velocity 17523.721 17518.928 > Direct efficiency 46% 16% which is just a wasted effort because the efficiency is really poor. Is this the effect of hammering a single zone which would be skipped otherwise while the allocation would succed from another zone? The latencies were not very much higher to match these numbers though. Is it possible that other parts of the benchmark suffered? The benchmark has measured only mmap part AFAIU. > Direct velocity 4159.306 29687.854 > Percentage direct scans 19% 62% > Page writes by reclaim 1149395.000 2622895.000 > Page writes file 0 0 > Page writes anon 1149395 2622895 > > The direct page scan and reclaim rates are noticeable. It is possible > this will not be a universal win on all workloads but cycling through > zonelists waiting for zlc->last_full_zap to expire is not the right > decision. As much as I would like to see zlc go it seems that it won't be that easy without regressing some loads. Or the numbers -- Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-20 15:50 +0200 |
| Message-ID | <pZyoe-6xg-79@gated-at.bofh.it> |
| In reply to | #1210492 |
On Thu, Aug 20, 2015 at 03:18:43PM +0200, Michal Hocko wrote: > On Wed 12-08-15 11:45:26, Mel Gorman wrote: > [...] > > 4-node machine stutter > > 4-node machine stutter > > 4.2.0-rc1 4.2.0-rc1 > > vanilla nozlc-v1r20 > > Min mmap 53.9902 ( 0.00%) 49.3629 ( 8.57%) > > 1st-qrtle mmap 54.6776 ( 0.00%) 54.1201 ( 1.02%) > > 2nd-qrtle mmap 54.9242 ( 0.00%) 54.5961 ( 0.60%) > > 3rd-qrtle mmap 55.1817 ( 0.00%) 54.9338 ( 0.45%) > > Max-90% mmap 55.3952 ( 0.00%) 55.3929 ( 0.00%) > > Max-93% mmap 55.4766 ( 0.00%) 57.5712 ( -3.78%) > > Max-95% mmap 55.5522 ( 0.00%) 57.8376 ( -4.11%) > > Max-99% mmap 55.7938 ( 0.00%) 63.6180 (-14.02%) > > Max mmap 6344.0292 ( 0.00%) 67.2477 ( 98.94%) > > Mean mmap 57.3732 ( 0.00%) 54.5680 ( 4.89%) > > Do you have data for other leads? Because the reclaim counters look > quite discouraging to be honest. > None of the other workloads showed changes that were worth reporting. > > 4.1.0 4.1.0 > > vanilla nozlc-v1r4 > > Swap Ins 838 502 > > Swap Outs 1149395 2622895 > > Twice as much swapouts is a lot. > > > DMA32 allocs 17839113 15863747 > > Normal allocs 129045707 137847920 > > Direct pages scanned 4070089 29046893 > > 7x more scanns by direct reclaim also sounds bad. > With this benchmark, the results for stutter will be highly variable as it's hammering the system. The intent of the test was to measure stalls at a time when desktop interactivity went to hell during IO and could stall for several minutes. Due to it nature, there is intense reclaim *and* compaction activity going on and there is no point drawing conclusions from the reclaim stats that are inherently good or bad. There will be differences in direct reclaim figures because instead of looping in the page allocator waiting for zlc to clear, it'll enter direct reclaim. In effect, the zlc causes processes to busy loop while kswapd does the work. If it turns out that this is the correct behaviour then we should do that explicitly, not rely on the broken zlc behaviour for the same reason we no longer rely on sprinkling congestion_wait() all over the place. > > Kswapd pages scanned 17147837 17140694 > > while kswapd is doing the same amount of work so we are moving > considerable amount of reclaim activity into the direct reclaim > > > Kswapd pages reclaimed 17146691 17139601 > > Direct pages reclaimed 1888879 4886630 > > Kswapd efficiency 99% 99% > > Kswapd velocity 17523.721 17518.928 > > Direct efficiency 46% 16% > > which is just a wasted effort because the efficiency is really poor. > Is this the effect of hammering a single zone which would be skipped > otherwise while the allocation would succed from another zone? > Very doubtful. It's more likely because the zlc was causing a process to busy loop waiting for kswapd to make forward progress. > The latencies were not very much higher to match these numbers though. > Is it possible that other parts of the benchmark suffered? The benchmark > has measured only mmap part AFAIU. > mmap latency yes but during it, the system is getting hammered and the latency is also affected by whether THPs were used or not. > > Direct velocity 4159.306 29687.854 > > Percentage direct scans 19% 62% > > Page writes by reclaim 1149395.000 2622895.000 > > Page writes file 0 0 > > Page writes anon 1149395 2622895 > > > > The direct page scan and reclaim rates are noticeable. It is possible > > this will not be a universal win on all workloads but cycling through > > zonelists waiting for zlc->last_full_zap to expire is not the right > > decision. > > As much as I would like to see zlc go it seems that it won't be that > easy without regressing some loads. Or the numbers If there are regressions on a real workload then it would be worth considering why busy looping happened to behave better and then solve it correctly. -- Mel Gorman SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Michal Hocko <mhocko@kernel.org> |
|---|---|
| Date | 2015-08-21 11:30 +0200 |
| Message-ID | <pZQO7-7UZ-13@gated-at.bofh.it> |
| In reply to | #1210500 |
On Thu 20-08-15 14:42:40, Mel Gorman wrote: > On Thu, Aug 20, 2015 at 03:18:43PM +0200, Michal Hocko wrote: > > On Wed 12-08-15 11:45:26, Mel Gorman wrote: > > [...] > > > 4-node machine stutter > > > 4-node machine stutter > > > 4.2.0-rc1 4.2.0-rc1 > > > vanilla nozlc-v1r20 > > > Min mmap 53.9902 ( 0.00%) 49.3629 ( 8.57%) > > > 1st-qrtle mmap 54.6776 ( 0.00%) 54.1201 ( 1.02%) > > > 2nd-qrtle mmap 54.9242 ( 0.00%) 54.5961 ( 0.60%) > > > 3rd-qrtle mmap 55.1817 ( 0.00%) 54.9338 ( 0.45%) > > > Max-90% mmap 55.3952 ( 0.00%) 55.3929 ( 0.00%) > > > Max-93% mmap 55.4766 ( 0.00%) 57.5712 ( -3.78%) > > > Max-95% mmap 55.5522 ( 0.00%) 57.8376 ( -4.11%) > > > Max-99% mmap 55.7938 ( 0.00%) 63.6180 (-14.02%) > > > Max mmap 6344.0292 ( 0.00%) 67.2477 ( 98.94%) > > > Mean mmap 57.3732 ( 0.00%) 54.5680 ( 4.89%) > > > > Do you have data for other leads? Because the reclaim counters look > > quite discouraging to be honest. > > > > None of the other workloads showed changes that were worth reporting. OK, that is a good sign. I would agree that an extreme and artificial load shouldn't be considered as a blocker. > > > 4.1.0 4.1.0 > > > vanilla nozlc-v1r4 > > > Swap Ins 838 502 > > > Swap Outs 1149395 2622895 > > > > Twice as much swapouts is a lot. > > > > > DMA32 allocs 17839113 15863747 > > > Normal allocs 129045707 137847920 > > > Direct pages scanned 4070089 29046893 > > > > 7x more scanns by direct reclaim also sounds bad. > > > > With this benchmark, the results for stutter will be highly variable as > it's hammering the system. The intent of the test was to measure stalls at > a time when desktop interactivity went to hell during IO and could stall > for several minutes. Due to it nature, there is intense reclaim *and* > compaction activity going on and there is no point drawing conclusions > from the reclaim stats that are inherently good or bad. > > There will be differences in direct reclaim figures because instead of > looping in the page allocator waiting for zlc to clear, it'll enter direct > reclaim. OK, I haven't considered this. kswapd might be stuck for quite some time but all of them being stuck shouldn't be that likely. But still, this is not a desirable behavior. > In effect, the zlc causes processes to busy loop while kswapd > does the work. If it turns out that this is the correct behaviour then > we should do that explicitly, not rely on the broken zlc behaviour for > the same reason we no longer rely on sprinkling congestion_wait() all > over the place. Fair point. I do agree that this should be done outside of get_page_from_freelist. I am still surprised by the considerable increase of swapouts but that should be handled separately if we see that in the real world loads. That being said Acked-by: Michal Hocko <mhocko@suse.com> -- Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2015-08-20 15:40 +0200 |
| Message-ID | <pZyeA-6kD-125@gated-at.bofh.it> |
| In reply to | #1205933 |
On 08/12/2015 12:45 PM, Mel Gorman wrote:
> The zonelist cache (zlc) was introduced to skip over zones that were
> recently known to be full. This avoided expensive operations such as the
> cpuset checks, watermark calculations and zone_reclaim. The situation
> today is different and the complexity of zlc is harder to justify.
>
> 1) The cpuset checks are no-ops unless a cpuset is active and in general are
> a lot cheaper.
>
> 2) zone_reclaim is now disabled by default and I suspect that was a large
> source of the cost that zlc wanted to avoid. When it is enabled, it's
> known to be a major source of stalling when nodes fill up and it's
> unwise to hit every other user with the overhead.
>
> 3) Watermark checks are expensive to calculate for high-order
> allocation requests. Later patches in this series will reduce the cost
> of the watermark checking.
>
> 4) The most important issue is that in the current implementation it
> is possible for a failed THP allocation to mark a zone full for order-0
> allocations and cause a fallback to remote nodes.
>
> The last issue could be addressed with additional complexity but as the
> benefit of zlc is questionable, it is better to remove it. If stalls
> due to zone_reclaim are ever reported then an alternative would be to
> introduce deferring logic based on a timeout inside zone_reclaim itself
> and leave the page allocator fast paths alone.
>
> The impact on page-allocator microbenchmarks is negligible as they don't
> hit the paths where the zlc comes into play. The impact was noticeable
> in a workload called "stutter". One part uses a lot of anonymous memory,
> a second measures mmap latency and a third copies a large file. In an
> ideal world the latency application would not notice the mmap latency.
> On a 4-node machine the results of this patch are
>
> 4-node machine stutter
> 4.2.0-rc1 4.2.0-rc1
> vanilla nozlc-v1r20
> Min mmap 53.9902 ( 0.00%) 49.3629 ( 8.57%)
> 1st-qrtle mmap 54.6776 ( 0.00%) 54.1201 ( 1.02%)
> 2nd-qrtle mmap 54.9242 ( 0.00%) 54.5961 ( 0.60%)
> 3rd-qrtle mmap 55.1817 ( 0.00%) 54.9338 ( 0.45%)
> Max-90% mmap 55.3952 ( 0.00%) 55.3929 ( 0.00%)
> Max-93% mmap 55.4766 ( 0.00%) 57.5712 ( -3.78%)
> Max-95% mmap 55.5522 ( 0.00%) 57.8376 ( -4.11%)
> Max-99% mmap 55.7938 ( 0.00%) 63.6180 (-14.02%)
> Max mmap 6344.0292 ( 0.00%) 67.2477 ( 98.94%)
> Mean mmap 57.3732 ( 0.00%) 54.5680 ( 4.89%)
>
> Note the maximum stall latency which was 6 seconds and becomes 67ms with
> this patch applied. However, also note that it is not guaranteed this
> benchmark always hits pathelogical cases and the milage varies. There is
> a secondary impact with more direct reclaim because zones are now being
> considered instead of being skipped by zlc.
>
> 4.1.0 4.1.0
> vanilla nozlc-v1r4
> Swap Ins 838 502
> Swap Outs 1149395 2622895
> DMA32 allocs 17839113 15863747
> Normal allocs 129045707 137847920
> Direct pages scanned 4070089 29046893
> Kswapd pages scanned 17147837 17140694
> Kswapd pages reclaimed 17146691 17139601
> Direct pages reclaimed 1888879 4886630
> Kswapd efficiency 99% 99%
> Kswapd velocity 17523.721 17518.928
> Direct efficiency 46% 16%
> Direct velocity 4159.306 29687.854
> Percentage direct scans 19% 62%
> Page writes by reclaim 1149395.000 2622895.000
> Page writes file 0 0
> Page writes anon 1149395 2622895
Interesting, kswapd has no decrease that would counter the increase in
direct reclaim. So there's more reclaim overall. Does it mean that
stutter doesn't like LRU and zlc was disrupting LRU?
> The direct page scan and reclaim rates are noticeable. It is possible
> this will not be a universal win on all workloads but cycling through
> zonelists waiting for zlc->last_full_zap to expire is not the right
> decision.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> Acked-by: David Rientjes <rientjes@google.com>
It doesn't seem that removal of zlc would increase overhead due to
"expensive operations no longer being avoided". Making some corner-case
benchmark(s) worse as a side-effect of different LRU approximation
shouldn't be a show-stopper. Hence
Acked-by: Vlastimil Babka <vbabka@suse.cz>
just git grep found some lines that should be also deleted:
include/linux/mmzone.h: * If zlcache_ptr is not NULL, then it is just
the address of zlcache,
include/linux/mmzone.h: * as explained above. If zlcache_ptr is NULL,
there is no zlcache.
And:
> @@ -3157,7 +2967,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
> gfp_t alloc_mask; /* The gfp_t that was actually used for allocation */
> struct alloc_context ac = {
> .high_zoneidx = gfp_zone(gfp_mask),
> - .nodemask = nodemask,
> + .nodemask = nodemask ? : &cpuset_current_mems_allowed,
> .migratetype = gfpflags_to_migratetype(gfp_mask),
> };
>
> @@ -3188,8 +2998,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
> ac.zonelist = zonelist;
> /* The preferred zone is used for statistics later */
> preferred_zoneref = first_zones_zonelist(ac.zonelist, ac.high_zoneidx,
> - ac.nodemask ? : &cpuset_current_mems_allowed,
> - &ac.preferred_zone);
> + ac.nodemask, &ac.preferred_zone);
> if (!ac.preferred_zone)
> goto out;
> ac.classzone_idx = zonelist_zone_idx(preferred_zoneref);
These hunks appear unrelated to zonelist cache? Also they move the
evaluation of cpuset_current_mems_allowed
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2015-08-20 16:20 +0200 |
| Message-ID | <pZyRc-7lc-1@gated-at.bofh.it> |
| In reply to | #1210497 |
On Thu, Aug 20, 2015 at 03:30:54PM +0200, Vlastimil Babka wrote:
> >Note the maximum stall latency which was 6 seconds and becomes 67ms with
> >this patch applied. However, also note that it is not guaranteed this
> >benchmark always hits pathelogical cases and the milage varies. There is
> >a secondary impact with more direct reclaim because zones are now being
> >considered instead of being skipped by zlc.
> >
> > 4.1.0 4.1.0
> > vanilla nozlc-v1r4
> >Swap Ins 838 502
> >Swap Outs 1149395 2622895
> >DMA32 allocs 17839113 15863747
> >Normal allocs 129045707 137847920
> >Direct pages scanned 4070089 29046893
> >Kswapd pages scanned 17147837 17140694
> >Kswapd pages reclaimed 17146691 17139601
> >Direct pages reclaimed 1888879 4886630
> >Kswapd efficiency 99% 99%
> >Kswapd velocity 17523.721 17518.928
> >Direct efficiency 46% 16%
> >Direct velocity 4159.306 29687.854
> >Percentage direct scans 19% 62%
> >Page writes by reclaim 1149395.000 2622895.000
> >Page writes file 0 0
> >Page writes anon 1149395 2622895
>
> Interesting, kswapd has no decrease that would counter the increase in
> direct reclaim. So there's more reclaim overall. Does it mean that stutter
> doesn't like LRU and zlc was disrupting LRU?
>
The LRU is being heavily disrupted by both reclaim and compaction
activity. The test is not a reliable means of evaluating reclaim decisions
because of the compaction activity. The main purpose of stutter was as a
proxy measure of desktop interactivity during IO.
As the test does THP allocations, it can trigger the case where zlc can
disable a zone for no reason and instead busy loop which is just wrong.
> >The direct page scan and reclaim rates are noticeable. It is possible
> >this will not be a universal win on all workloads but cycling through
> >zonelists waiting for zlc->last_full_zap to expire is not the right
> >decision.
> >
> >Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> >Acked-by: David Rientjes <rientjes@google.com>
>
> It doesn't seem that removal of zlc would increase overhead due to
> "expensive operations no longer being avoided". Making some corner-case
> benchmark(s) worse as a side-effect of different LRU approximation shouldn't
> be a show-stopper. Hence
>
> Acked-by: Vlastimil Babka <vbabka@suse.cz>
>
Thanks.
> just git grep found some lines that should be also deleted:
>
> include/linux/mmzone.h: * If zlcache_ptr is not NULL, then it is just the
> address of zlcache,
> include/linux/mmzone.h: * as explained above. If zlcache_ptr is NULL, there
> is no zlcache.
>
Thanks
> And:
>
> >@@ -3157,7 +2967,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
> > gfp_t alloc_mask; /* The gfp_t that was actually used for allocation */
> > struct alloc_context ac = {
> > .high_zoneidx = gfp_zone(gfp_mask),
> >- .nodemask = nodemask,
> >+ .nodemask = nodemask ? : &cpuset_current_mems_allowed,
> > .migratetype = gfpflags_to_migratetype(gfp_mask),
> > };
> >
> >@@ -3188,8 +2998,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
> > ac.zonelist = zonelist;
> > /* The preferred zone is used for statistics later */
> > preferred_zoneref = first_zones_zonelist(ac.zonelist, ac.high_zoneidx,
> >- ac.nodemask ? : &cpuset_current_mems_allowed,
> >- &ac.preferred_zone);
> >+ ac.nodemask, &ac.preferred_zone);
> > if (!ac.preferred_zone)
> > goto out;
> > ac.classzone_idx = zonelist_zone_idx(preferred_zoneref);
>
> These hunks appear unrelated to zonelist cache? Also they move the
> evaluation of cpuset_current_mems_allowed
They are rebase-related brain damage :(. I'll fix it and retest.
--
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2015-08-20 17:00 +0200 |
| Message-ID | <pZztU-850-17@gated-at.bofh.it> |
| In reply to | #1210517 |
On 08/20/2015 04:17 PM, Mel Gorman wrote: > On Thu, Aug 20, 2015 at 03:30:54PM +0200, Vlastimil Babka wrote: >> These hunks appear unrelated to zonelist cache? Also they move the >> evaluation of cpuset_current_mems_allowed Ah forgot to delete the "Also" part. I wanted to write that it moves the evaluation away from inside the read_mems_allowed_begin() - read_mems_allowed_retry() pair. But then I realized it's just taking a *reference* and not going through cpuset_current_mems_allowed yet, so it's probably OK. Just out of place in this patch. > They are rebase-related brain damage :(. I'll fix it and retest. > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web