Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1276356 > unrolled thread

[PATCH 0/2] 2 zone_pages_reclaimable fixes

Started byMichal Hocko <mhocko@kernel.org>
First post2015-11-24 13:00 +0100
Last post2015-11-25 12:10 +0100
Articles 8 — 4 participants

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/2] 2 zone_pages_reclaimable fixes Michal Hocko <mhocko@kernel.org> - 2015-11-24 13:00 +0100
    [PATCH 2/2] mm, vmscan: do not overestimate anonymous reclaimable pages Michal Hocko <mhocko@kernel.org> - 2015-11-24 13:00 +0100
      Re: [PATCH 2/2] mm, vmscan: do not overestimate anonymous  reclaimable pages Vladimir Davydov <vdavydov@virtuozzo.com> - 2015-11-24 14:10 +0100
        Re: [PATCH 2/2] mm, vmscan: do not overestimate anonymous  reclaimable pages Michal Hocko <mhocko@kernel.org> - 2015-11-24 14:40 +0100
    [PATCH 1/2] mm, vmscan: consider isolated pages in zone_reclaimable_pages Michal Hocko <mhocko@kernel.org> - 2015-11-24 13:00 +0100
      Re: [PATCH 1/2] mm, vmscan: consider isolated pages in  zone_reclaimable_pages Vladimir Davydov <vdavydov@virtuozzo.com> - 2015-11-24 15:00 +0100
      Re: [PATCH 1/2] mm, vmscan: consider isolated pages in  zone_reclaimable_pages Johannes Weiner <hannes@cmpxchg.org> - 2015-11-24 17:10 +0100
      Re: [PATCH 1/2] mm, vmscan: consider isolated pages in  zone_reclaimable_pages David Rientjes <rientjes@google.com> - 2015-11-25 12:10 +0100

#1276356 — [PATCH 0/2] 2 zone_pages_reclaimable fixes

FromMichal Hocko <mhocko@kernel.org>
Date2015-11-24 13:00 +0100
Subject[PATCH 0/2] 2 zone_pages_reclaimable fixes
Message-ID<qykql-7Cj-1@gated-at.bofh.it>
Hi,
Johannes had a valid point [1] that zone_pages_reclaimable should contain
isolated pages as well. This is what the first patch does. While I was
there I've realized that the current logic of this function allows for
a large overestimation of the reclaimable memory with anon >> nr_swap_pages
which would be visible especially when the swap is getting short on space.
I think this is a bug and this is fixed in the second patch.

I do not have any particular workload which would show significant misbehavior
because of the current implementation though. We mostly just happen to scan
longer than necessary because zone_reclaimable would keep us looping longer
but I still think it makes sense to fix this regardless.

[1] http://lkml.kernel.org/r/20151123182447.GF13000%40cmpxchg.org

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1276357 — [PATCH 2/2] mm, vmscan: do not overestimate anonymous reclaimable pages

FromMichal Hocko <mhocko@kernel.org>
Date2015-11-24 13:00 +0100
Subject[PATCH 2/2] mm, vmscan: do not overestimate anonymous reclaimable pages
Message-ID<qykql-7Cj-3@gated-at.bofh.it>
In reply to#1276356
From: Michal Hocko <mhocko@suse.com>

zone_reclaimable_pages considers all anonymous pages on LRUs reclaimable
if there is at least one entry on the swap storage left. This can be
really misleading when the swap is short on space and skew reclaim
decisions based on zone_reclaimable_pages. Fix this by clamping the
number to the minimum of the available swap space and anon LRU pages.

Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/vmscan.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 946d348f5040..646001a1f279 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -195,15 +195,20 @@ static bool sane_reclaim(struct scan_control *sc)
 static unsigned long zone_reclaimable_pages(struct zone *zone)
 {
 	unsigned long nr;
+	long nr_swap = get_nr_swap_pages();
 
 	nr = zone_page_state(zone, NR_ACTIVE_FILE) +
 	     zone_page_state(zone, NR_INACTIVE_FILE) +
 	     zone_page_state(zone, NR_ISOLATED_FILE);
 
-	if (get_nr_swap_pages() > 0)
-		nr += zone_page_state(zone, NR_ACTIVE_ANON) +
-		      zone_page_state(zone, NR_INACTIVE_ANON) +
-		      zone_page_state(zone, NR_ISOLATED_ANON);
+	if (nr_swap > 0) {
+		unsigned long anon;
+
+		anon = zone_page_state(zone, NR_ACTIVE_ANON) +
+		       zone_page_state(zone, NR_INACTIVE_ANON) +
+		       zone_page_state(zone, NR_ISOLATED_ANON);
+		nr += min_t(unsigned long, nr_swap, anon);
+	}
 
 	return nr;
 }
-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276420 — Re: [PATCH 2/2] mm, vmscan: do not overestimate anonymous reclaimable pages

FromVladimir Davydov <vdavydov@virtuozzo.com>
Date2015-11-24 14:10 +0100
SubjectRe: [PATCH 2/2] mm, vmscan: do not overestimate anonymous reclaimable pages
Message-ID<qylw7-5A-29@gated-at.bofh.it>
In reply to#1276357
On Tue, Nov 24, 2015 at 12:55:00PM +0100, Michal Hocko wrote:
> zone_reclaimable_pages considers all anonymous pages on LRUs reclaimable
> if there is at least one entry on the swap storage left. This can be
> really misleading when the swap is short on space and skew reclaim
> decisions based on zone_reclaimable_pages. Fix this by clamping the
> number to the minimum of the available swap space and anon LRU pages.

Suppose there's 100M of swap and 1G of anon pages. This patch makes
zone_reclaimable_pages return 100M instead of 1G in this case. If you
rotate 600M of oldest anon pages, which is quite possible,
zone_reclaimable will start returning false, which is wrong, because
there are still 400M pages that were not even scanned, besides those
600M of rotated pages could have become reclaimable after their ref bits
got cleared.

I think it is the name of zone_reclaimable_pages which is misleading. It
should be called something like "zone_scannable_pages" judging by how it
is used in zone_reclaimable.

Thanks,
Vladimir

> 
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
>  mm/vmscan.c | 13 +++++++++----
>  1 file changed, 9 insertions(+), 4 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 946d348f5040..646001a1f279 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -195,15 +195,20 @@ static bool sane_reclaim(struct scan_control *sc)
>  static unsigned long zone_reclaimable_pages(struct zone *zone)
>  {
>  	unsigned long nr;
> +	long nr_swap = get_nr_swap_pages();
>  
>  	nr = zone_page_state(zone, NR_ACTIVE_FILE) +
>  	     zone_page_state(zone, NR_INACTIVE_FILE) +
>  	     zone_page_state(zone, NR_ISOLATED_FILE);
>  
> -	if (get_nr_swap_pages() > 0)
> -		nr += zone_page_state(zone, NR_ACTIVE_ANON) +
> -		      zone_page_state(zone, NR_INACTIVE_ANON) +
> -		      zone_page_state(zone, NR_ISOLATED_ANON);
> +	if (nr_swap > 0) {
> +		unsigned long anon;
> +
> +		anon = zone_page_state(zone, NR_ACTIVE_ANON) +
> +		       zone_page_state(zone, NR_INACTIVE_ANON) +
> +		       zone_page_state(zone, NR_ISOLATED_ANON);
> +		nr += min_t(unsigned long, nr_swap, anon);
> +	}
>  
>  	return nr;
>  }
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276454 — Re: [PATCH 2/2] mm, vmscan: do not overestimate anonymous reclaimable pages

FromMichal Hocko <mhocko@kernel.org>
Date2015-11-24 14:40 +0100
SubjectRe: [PATCH 2/2] mm, vmscan: do not overestimate anonymous reclaimable pages
Message-ID<qylZ8-g8-21@gated-at.bofh.it>
In reply to#1276420
On Tue 24-11-15 16:07:40, Vladimir Davydov wrote:
> On Tue, Nov 24, 2015 at 12:55:00PM +0100, Michal Hocko wrote:
> > zone_reclaimable_pages considers all anonymous pages on LRUs reclaimable
> > if there is at least one entry on the swap storage left. This can be
> > really misleading when the swap is short on space and skew reclaim
> > decisions based on zone_reclaimable_pages. Fix this by clamping the
> > number to the minimum of the available swap space and anon LRU pages.
> 
> Suppose there's 100M of swap and 1G of anon pages. This patch makes
> zone_reclaimable_pages return 100M instead of 1G in this case. If you
> rotate 600M of oldest anon pages, which is quite possible,
> zone_reclaimable will start returning false, which is wrong, because
> there are still 400M pages that were not even scanned, besides those
> 600M of rotated pages could have become reclaimable after their ref bits
> got cleared.

Uhm, OK, I guess you are right. Making zone_reclaimable less
conservative can lead to hard to expect results. Scratch this patch
please.
 
> I think it is the name of zone_reclaimable_pages which is misleading. It
> should be called something like "zone_scannable_pages" judging by how it
> is used in zone_reclaimable.

Thanks!
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276358 — [PATCH 1/2] mm, vmscan: consider isolated pages in zone_reclaimable_pages

FromMichal Hocko <mhocko@kernel.org>
Date2015-11-24 13:00 +0100
Subject[PATCH 1/2] mm, vmscan: consider isolated pages in zone_reclaimable_pages
Message-ID<qykql-7Cj-5@gated-at.bofh.it>
In reply to#1276356
From: Michal Hocko <mhocko@suse.com>

zone_reclaimable_pages counts how many pages are reclaimable in
the given zone. This currently includes all pages on file lrus and
anon lrus if there is an available swap storage. We do not consider
NR_ISOLATED_{ANON,FILE} counters though which is not correct because
these counters reflect temporarily isolated pages which are still
reclaimable because they either get back to their LRU or get freed
either by the page reclaim or page migration.

The number of these pages might be sufficiently high to confuse users of
zone_reclaimable_pages (e.g. mbind can migrate large ranges of memory at
once).

Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/vmscan.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index a4507ecaefbf..946d348f5040 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -197,11 +197,13 @@ static unsigned long zone_reclaimable_pages(struct zone *zone)
 	unsigned long nr;
 
 	nr = zone_page_state(zone, NR_ACTIVE_FILE) +
-	     zone_page_state(zone, NR_INACTIVE_FILE);
+	     zone_page_state(zone, NR_INACTIVE_FILE) +
+	     zone_page_state(zone, NR_ISOLATED_FILE);
 
 	if (get_nr_swap_pages() > 0)
 		nr += zone_page_state(zone, NR_ACTIVE_ANON) +
-		      zone_page_state(zone, NR_INACTIVE_ANON);
+		      zone_page_state(zone, NR_INACTIVE_ANON) +
+		      zone_page_state(zone, NR_ISOLATED_ANON);
 
 	return nr;
 }
-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276496 — Re: [PATCH 1/2] mm, vmscan: consider isolated pages in zone_reclaimable_pages

FromVladimir Davydov <vdavydov@virtuozzo.com>
Date2015-11-24 15:00 +0100
SubjectRe: [PATCH 1/2] mm, vmscan: consider isolated pages in zone_reclaimable_pages
Message-ID<qymiv-n6-37@gated-at.bofh.it>
In reply to#1276358
On Tue, Nov 24, 2015 at 12:54:59PM +0100, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> zone_reclaimable_pages counts how many pages are reclaimable in
> the given zone. This currently includes all pages on file lrus and
> anon lrus if there is an available swap storage. We do not consider
> NR_ISOLATED_{ANON,FILE} counters though which is not correct because
> these counters reflect temporarily isolated pages which are still
> reclaimable because they either get back to their LRU or get freed
> either by the page reclaim or page migration.
> 
> The number of these pages might be sufficiently high to confuse users of
> zone_reclaimable_pages (e.g. mbind can migrate large ranges of memory at
> once).

Sounds reasonable to me.

Reviewed-by: Vladimir Davydov <vdavydov@virtuozzo.com>

Thanks,
Vladimir

> 
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Signed-off-by: Michal Hocko <mhocko@suse.com>
> ---
>  mm/vmscan.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index a4507ecaefbf..946d348f5040 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -197,11 +197,13 @@ static unsigned long zone_reclaimable_pages(struct zone *zone)
>  	unsigned long nr;
>  
>  	nr = zone_page_state(zone, NR_ACTIVE_FILE) +
> -	     zone_page_state(zone, NR_INACTIVE_FILE);
> +	     zone_page_state(zone, NR_INACTIVE_FILE) +
> +	     zone_page_state(zone, NR_ISOLATED_FILE);
>  
>  	if (get_nr_swap_pages() > 0)
>  		nr += zone_page_state(zone, NR_ACTIVE_ANON) +
> -		      zone_page_state(zone, NR_INACTIVE_ANON);
> +		      zone_page_state(zone, NR_INACTIVE_ANON) +
> +		      zone_page_state(zone, NR_ISOLATED_ANON);
>  
>  	return nr;
>  }
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276600 — Re: [PATCH 1/2] mm, vmscan: consider isolated pages in zone_reclaimable_pages

FromJohannes Weiner <hannes@cmpxchg.org>
Date2015-11-24 17:10 +0100
SubjectRe: [PATCH 1/2] mm, vmscan: consider isolated pages in zone_reclaimable_pages
Message-ID<qyoki-1XH-33@gated-at.bofh.it>
In reply to#1276358
On Tue, Nov 24, 2015 at 12:54:59PM +0100, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
> 
> zone_reclaimable_pages counts how many pages are reclaimable in
> the given zone. This currently includes all pages on file lrus and
> anon lrus if there is an available swap storage. We do not consider
> NR_ISOLATED_{ANON,FILE} counters though which is not correct because
> these counters reflect temporarily isolated pages which are still
> reclaimable because they either get back to their LRU or get freed
> either by the page reclaim or page migration.
> 
> The number of these pages might be sufficiently high to confuse users of
> zone_reclaimable_pages (e.g. mbind can migrate large ranges of memory at
> once).
> 
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Acked-by: Johannes Weiner <hannes@cmpxchg.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1277229 — Re: [PATCH 1/2] mm, vmscan: consider isolated pages in zone_reclaimable_pages

FromDavid Rientjes <rientjes@google.com>
Date2015-11-25 12:10 +0100
SubjectRe: [PATCH 1/2] mm, vmscan: consider isolated pages in zone_reclaimable_pages
Message-ID<qyG7y-5tK-73@gated-at.bofh.it>
In reply to#1276358
On Tue, 24 Nov 2015, Michal Hocko wrote:

> From: Michal Hocko <mhocko@suse.com>
> 
> zone_reclaimable_pages counts how many pages are reclaimable in
> the given zone. This currently includes all pages on file lrus and
> anon lrus if there is an available swap storage. We do not consider
> NR_ISOLATED_{ANON,FILE} counters though which is not correct because
> these counters reflect temporarily isolated pages which are still
> reclaimable because they either get back to their LRU or get freed
> either by the page reclaim or page migration.
> 
> The number of these pages might be sufficiently high to confuse users of
> zone_reclaimable_pages (e.g. mbind can migrate large ranges of memory at
> once).
> 
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Acked-by: David Rientjes <rientjes@google.com>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web