Re: WARN_ON_ONCE() in process_one_work()?

Path	csiph.com!aioe.org!bofh.it!news.nic.it!robomod
From	"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Newsgroups	linux.kernel
Subject	Re: WARN_ON_ONCE() in process_one_work()?
Date	Fri, 05 May 2017 19:20:02 +0200
Message-ID	<tDP3A-4H2-3@gated-at.bofh.it> (permalink)
References	<tCmQ2-39r-27@gated-at.bofh.it> <tCooO-4aC-27@gated-at.bofh.it> <tCoyu-4dM-17@gated-at.bofh.it> <tCoI9-4h9-7@gated-at.bofh.it>
Reply-To	paulmck@linux.vnet.ibm.com
MIME-Version	1.0
Content-Type	text/plain; charset=us-ascii
Content-Disposition	inline
User-Agent	Mutt/1.5.21 (2010-09-15)
X-Tm-As-Gconf	00
X-Cbid	17050517-0048-0000-0000-000001786789
X-Ibm-Spammodules-Versions	BY=3.00007026; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000209; SDB=6.00856594; UDB=6.00424127; IPR=6.00635862; BA=6.00005326; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015326; XFM=3.00000014; UTC=2017-05-05 17:12:01
X-Ibm-Av-Detection	SAVI=unused REMOTE=unused XFE=unused
X-Cbparentid	17050517-0049-0000-0000-000041031107
X-Proofpoint-Virus-Version	vendor=fsecure engine=2.50.10432:,, definitions=2017-05-05_12:,, signatures=0
X-Proofpoint-Spam-Details	rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1705050169
Sender	robomod@news.nic.it
List-ID	<linux-kernel.vger.kernel.org>
X-Mailing-List	linux-kernel@vger.kernel.org
Approved	robomod@news.nic.it
Lines	38
Organization	linux.* mail to news gateway
X-Original-Cc	jiangshanlai@gmail.com, linux-kernel@vger.kernel.org
X-Original-Date	Fri, 5 May 2017 10:11:59 -0700
X-Original-Message-ID	<20170505171159.GA10296@linux.vnet.ibm.com>
X-Original-References	<20170501165747.GA993@linux.vnet.ibm.com> <20170501183807.GA7054@linux.vnet.ibm.com> <20170501184402.GB8921@htj.duckdns.org> <20170501185819.GJ3956@linux.vnet.ibm.com>
X-Original-Sender	linux-kernel-owner@vger.kernel.org
Xref	csiph.com linux.kernel:1636505

Show key headers only | View raw

On Mon, May 01, 2017 at 11:58:19AM -0700, Paul E. McKenney wrote:
> On Mon, May 01, 2017 at 02:44:02PM -0400, Tejun Heo wrote:
> > Hello, Paul.
> > 
> > On Mon, May 01, 2017 at 11:38:07AM -0700, Paul E. McKenney wrote:
> > > On Mon, May 01, 2017 at 09:57:47AM -0700, Paul E. McKenney wrote:
> > > > Hello!
> > > > 
> > > > I am hitting this WARN_ON_ONCE() in process_one_work() and am wondering
> > > > what I did wrong to make this happen:
> > > 
> > > Oh, wait...  Rescuer, it says.  Might this be due to the fact that RCU's
> > > expedited grace periods block within a workqueue handler?  Might this
> > > in turn run the system out of workqueue kthreads?  If this is the likely
> > > cause, my approach would be to rework the expected-grace-period workqueue
> > > handler to return when waiting for the grace period to complete, and to
> > > replace the current wakeup with a schedule_work() or something similar.
> > 
> > That should be completely fine.  It could just be that the rescuer
> > path has a bug around CPU hotplug handling.  Can you please confirm
> > either way on the cpuset usage?
> 
> I have no explicit cpuset usage or affinity of the workqueue handlers
> themselves.
> 
> However, this is thus far only happening in CONFIG_NO_HZ_FULL=y runs, in
> this case, with the kernel boot parameter nohz_full=2-9 out of 16 CPUs.
> IIRC, this sets up a "housekeeping" cpuset that pushes normal tasks away
> from the nohz_full CPUs.
> 
> I do build with CONFIG_HOTPLUG_CPU=y, and the test does a lot of
> hotplugging.  Also, other kthreads (but again, not the workqueue handlers)
> do a lot of explicit CPU-affinity manipulation.

Just following up...  I have hit this bug a couple of times over the
past few days.  Anything I can do to help?

							Thanx, Paul

Back to linux.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Re: WARN_ON_ONCE() in process_one_work()? Tejun Heo <tj@kernel.org> - 2017-05-01 20:50 +0200
  Re: WARN_ON_ONCE() in process_one_work()? "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-05-01 21:00 +0200
    Re: WARN_ON_ONCE() in process_one_work()? "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-05-05 19:20 +0200

csiph-web