Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1534366 > unrolled thread

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

Started byBoris Zhmurov <bb@kernelpanic.ru>
First post2016-12-01 19:20 +0100
Last post2016-12-02 17:50 +0100
Articles 8 — 3 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and  `mem_cgroup_shrink_node` Boris Zhmurov <bb@kernelpanic.ru> - 2016-12-01 19:20 +0100
    Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and  `mem_cgroup_shrink_node` "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-12-01 20:40 +0100
    Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and  `mem_cgroup_shrink_node` Michal Hocko <mhocko@kernel.org> - 2016-12-02 10:40 +0100
      Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and  `mem_cgroup_shrink_node` "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-12-02 15:00 +0100
    Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and  `mem_cgroup_shrink_node` "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-12-02 17:50 +0100
      Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and  `mem_cgroup_shrink_node` Michal Hocko <mhocko@kernel.org> - 2016-12-02 18:10 +0100
        Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and  `mem_cgroup_shrink_node` "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-12-02 18:20 +0100
    Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and  `mem_cgroup_shrink_node` Boris Zhmurov <bb@kernelpanic.ru> - 2016-12-02 17:50 +0100

#1534366 — Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

FromBoris Zhmurov <bb@kernelpanic.ru>
Date2016-12-01 19:20 +0100
SubjectRe: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`
Message-ID<sJE7D-85V-13@gated-at.bofh.it>

[Multipart message — attachments visible in raw view] — view raw

Michal Hocko 30/11/16 21:25:

>>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help?
>>
>> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree.
>> I can try another portion of patches, no problem :)
> 
> Replacing cond_resched_rcu_qs in shrink_node_memcg by cond_resched would
> be really helpful to tell whether we are missing a real scheduling point
> or whether something more serious is going on here.

Well, I can confirm, that replacing cond_resched_rcu_qs in
shrink_node_memcg by cond_resched also makes dmesg clean from RCU CPU
stall warnings.

I've attached patch (just modification of Paul's patch), that fixes RCU
stall messages in situations, when all memory is used by
couchbase/memcached + fs cache and linux starts to use swap.


-- 
Boris Zhmurov
System/Network Administrator
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"

[toc] | [next] | [standalone]

#1534409

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2016-12-01 20:40 +0100
Message-ID<sJFn3-D4-5@gated-at.bofh.it>
In reply to#1534366
On Thu, Dec 01, 2016 at 09:10:01PM +0300, Boris Zhmurov wrote:
> Michal Hocko 30/11/16 21:25:
> 
> >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help?
> >>
> >> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree.
> >> I can try another portion of patches, no problem :)
> > 
> > Replacing cond_resched_rcu_qs in shrink_node_memcg by cond_resched would
> > be really helpful to tell whether we are missing a real scheduling point
> > or whether something more serious is going on here.
> 
> Well, I can confirm, that replacing cond_resched_rcu_qs in
> shrink_node_memcg by cond_resched also makes dmesg clean from RCU CPU
> stall warnings.
> 
> I've attached patch (just modification of Paul's patch), that fixes RCU
> stall messages in situations, when all memory is used by
> couchbase/memcached + fs cache and linux starts to use swap.
> 
> 
> -- 
> Boris Zhmurov
> System/Network Administrator
> mailto: bb@kernelpanic.ru
> "wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"

> --- a/mm/vmscan.c.orig	2016-11-30 21:52:58.314895320 +0300
> +++ b/mm/vmscan.c	2016-11-30 21:53:29.502895320 +0300
> @@ -2352,6 +2352,7 @@
>  				nr_reclaimed += shrink_list(lru, nr_to_scan,
>  							    lruvec, sc);
>  			}
> +			cond_resched();
>  		}
> 
>  		if (nr_reclaimed < nr_to_reclaim || scan_adjusted)

Nice!

Just to double-check, could you please also test your patch above with
these two commits from -rcu?

d2db185bfee8 ("rcu: Remove short-term CPU kicking")
f8f127e738e3 ("rcu: Add long-term CPU kicking")

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1534770

FromMichal Hocko <mhocko@kernel.org>
Date2016-12-02 10:40 +0100
Message-ID<sJStY-22N-29@gated-at.bofh.it>
In reply to#1534366
On Thu 01-12-16 21:10:01, Boris Zhmurov wrote:
> Michal Hocko 30/11/16 21:25:
> 
> >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help?
> >>
> >> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree.
> >> I can try another portion of patches, no problem :)
> > 
> > Replacing cond_resched_rcu_qs in shrink_node_memcg by cond_resched would
> > be really helpful to tell whether we are missing a real scheduling point
> > or whether something more serious is going on here.
> 
> Well, I can confirm, that replacing cond_resched_rcu_qs in
> shrink_node_memcg by cond_resched also makes dmesg clean from RCU CPU
> stall warnings.
> 
> I've attached patch (just modification of Paul's patch), that fixes RCU
> stall messages in situations, when all memory is used by
> couchbase/memcached + fs cache and linux starts to use swap.

OK, thanks for the confirmation! I will send a patch because it is true
that we do not have any scheduling point if no pages can be isolated
fromm the LRU. This might be what you are seeing.
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1534934

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2016-12-02 15:00 +0100
Message-ID<sJWxA-4wP-33@gated-at.bofh.it>
In reply to#1534770
On Fri, Dec 02, 2016 at 10:37:35AM +0100, Michal Hocko wrote:
> On Thu 01-12-16 21:10:01, Boris Zhmurov wrote:
> > Michal Hocko 30/11/16 21:25:
> > 
> > >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help?
> > >>
> > >> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree.
> > >> I can try another portion of patches, no problem :)
> > > 
> > > Replacing cond_resched_rcu_qs in shrink_node_memcg by cond_resched would
> > > be really helpful to tell whether we are missing a real scheduling point
> > > or whether something more serious is going on here.
> > 
> > Well, I can confirm, that replacing cond_resched_rcu_qs in
> > shrink_node_memcg by cond_resched also makes dmesg clean from RCU CPU
> > stall warnings.
> > 
> > I've attached patch (just modification of Paul's patch), that fixes RCU
> > stall messages in situations, when all memory is used by
> > couchbase/memcached + fs cache and linux starts to use swap.
> 
> OK, thanks for the confirmation! I will send a patch because it is true
> that we do not have any scheduling point if no pages can be isolated
> fromm the LRU. This might be what you are seeing.

Thank you both!

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1535071

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2016-12-02 17:50 +0100
Message-ID<sJZc5-6jb-13@gated-at.bofh.it>
In reply to#1534366
On Fri, Dec 02, 2016 at 07:39:24PM +0300, Boris Zhmurov wrote:
> Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST:
> 
> >> Well, I can confirm, that replacing cond_resched_rcu_qs in 
> >> shrink_node_memcg by cond_resched also makes dmesg clean from RCU 
> >> CPU stall warnings.
> >> 
> >> I've attached patch (just modification of Paul's patch), that
> >> fixes RCU stall messages in situations, when all memory is used by
> >>  couchbase/memcached + fs cache and linux starts to use swap.
> 
> > Nice! Just to double-check, could you please also test your patch
> > above with these two commits from -rcu?
> > 
> > d2db185bfee8 ("rcu: Remove short-term CPU kicking") f8f127e738e3
> > ("rcu: Add long-term CPU kicking")
> > 
> > Thanx, Paul
> 
> 
> Looks like patches d2db185bfee8 and f8f127e738e3 change nothing.
> 
> With cond_resched() in shrink_node_memcg and these two patches dmesg is
> clean. No any RCU CPU stall messages.

Very good!  I have these two patches queued for 4.11.

And thank you again for all the testing!!!

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1535082

FromMichal Hocko <mhocko@kernel.org>
Date2016-12-02 18:10 +0100
Message-ID<sJZvs-6Fu-21@gated-at.bofh.it>
In reply to#1535071
On Fri 02-12-16 08:44:08, Paul E. McKenney wrote:
> On Fri, Dec 02, 2016 at 07:39:24PM +0300, Boris Zhmurov wrote:
> > Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST:
> > 
> > >> Well, I can confirm, that replacing cond_resched_rcu_qs in 
> > >> shrink_node_memcg by cond_resched also makes dmesg clean from RCU 
> > >> CPU stall warnings.
> > >> 
> > >> I've attached patch (just modification of Paul's patch), that
> > >> fixes RCU stall messages in situations, when all memory is used by
> > >>  couchbase/memcached + fs cache and linux starts to use swap.
> > 
> > > Nice! Just to double-check, could you please also test your patch
> > > above with these two commits from -rcu?
> > > 
> > > d2db185bfee8 ("rcu: Remove short-term CPU kicking") f8f127e738e3
> > > ("rcu: Add long-term CPU kicking")
> > > 
> > > Thanx, Paul
> > 
> > 
> > Looks like patches d2db185bfee8 and f8f127e738e3 change nothing.
> > 
> > With cond_resched() in shrink_node_memcg and these two patches dmesg is
> > clean. No any RCU CPU stall messages.
> 
> Very good!  I have these two patches queued for 4.11.

FWIW I have posted the cond_resched patch to Andrew [1]. I didn't CC you
Paul to save you from emails as this is more MM than anything else
related ;)

[1] http://lkml.kernel.org/r/20161202095841.16648-1-mhocko@kernel.org
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1535088

From"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date2016-12-02 18:20 +0100
Message-ID<sJZF8-6IJ-23@gated-at.bofh.it>
In reply to#1535082
On Fri, Dec 02, 2016 at 06:02:49PM +0100, Michal Hocko wrote:
> On Fri 02-12-16 08:44:08, Paul E. McKenney wrote:
> > On Fri, Dec 02, 2016 at 07:39:24PM +0300, Boris Zhmurov wrote:
> > > Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST:
> > > 
> > > >> Well, I can confirm, that replacing cond_resched_rcu_qs in 
> > > >> shrink_node_memcg by cond_resched also makes dmesg clean from RCU 
> > > >> CPU stall warnings.
> > > >> 
> > > >> I've attached patch (just modification of Paul's patch), that
> > > >> fixes RCU stall messages in situations, when all memory is used by
> > > >>  couchbase/memcached + fs cache and linux starts to use swap.
> > > 
> > > > Nice! Just to double-check, could you please also test your patch
> > > > above with these two commits from -rcu?
> > > > 
> > > > d2db185bfee8 ("rcu: Remove short-term CPU kicking") f8f127e738e3
> > > > ("rcu: Add long-term CPU kicking")
> > > > 
> > > > Thanx, Paul
> > > 
> > > 
> > > Looks like patches d2db185bfee8 and f8f127e738e3 change nothing.
> > > 
> > > With cond_resched() in shrink_node_memcg and these two patches dmesg is
> > > clean. No any RCU CPU stall messages.
> > 
> > Very good!  I have these two patches queued for 4.11.
> 
> FWIW I have posted the cond_resched patch to Andrew [1]. I didn't CC you
> Paul to save you from emails as this is more MM than anything else
> related ;)
> 
> [1] http://lkml.kernel.org/r/20161202095841.16648-1-mhocko@kernel.org

Feel free to apply:

Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1535074

FromBoris Zhmurov <bb@kernelpanic.ru>
Date2016-12-02 17:50 +0100
Message-ID<sJZc5-6jb-15@gated-at.bofh.it>
In reply to#1534366
Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST:

>> Well, I can confirm, that replacing cond_resched_rcu_qs in 
>> shrink_node_memcg by cond_resched also makes dmesg clean from RCU 
>> CPU stall warnings.
>> 
>> I've attached patch (just modification of Paul's patch), that
>> fixes RCU stall messages in situations, when all memory is used by
>>  couchbase/memcached + fs cache and linux starts to use swap.

> Nice! Just to double-check, could you please also test your patch
> above with these two commits from -rcu?
> 
> d2db185bfee8 ("rcu: Remove short-term CPU kicking") f8f127e738e3
> ("rcu: Add long-term CPU kicking")
> 
> Thanx, Paul


Looks like patches d2db185bfee8 and f8f127e738e3 change nothing.

With cond_resched() in shrink_node_memcg and these two patches dmesg is
clean. No any RCU CPU stall messages.

Thanks.

-- 
Boris Zhmurov
mailto: bb@kernelpanic.ru
"wget http://kernelpanic.ru/bb_public_key.pgp -O - | gpg --import"

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web