Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1331287 > unrolled thread

[PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for unbound pool_workqueue lookup

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for unbound  pool_workqueue lookup Tejun Heo <tj@kernel.org> - 2016-02-10 17:00 +0100
    Re: [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for  unbound pool_workqueue lookup Michal Hocko <mhocko@kernel.org> - 2016-02-15 18:40 +0100
      Re: [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for  unbound pool_workqueue lookup Tejun Heo <tj@kernel.org> - 2016-02-15 19:30 +0100
        Re: [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for  unbound pool_workqueue lookup Michal Hocko <mhocko@kernel.org> - 2016-02-15 22:00 +0100
          Re: [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for  unbound pool_workqueue lookup Tejun Heo <tj@kernel.org> - 2016-02-15 22:10 +0100

#1331287 — [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for unbound pool_workqueue lookup

Hello,

With 874bbfe600a6 reverted, this doesn't matter as much but I'm still
committing it to wq/for-4.5-fixes.  The commit message has been
updated accordingly.

Thanks.

------ 8< ------
From 9354baa208f47bf088c28c1da5f38abe172db5d6 Mon Sep 17 00:00:00 2001
From: Tejun Heo <tj@kernel.org>
Date: Wed, 3 Feb 2016 13:54:25 -0500

When looking up the pool_workqueue to use for an unbound workqueue,
workqueue assumes that the target CPU is always bound to a valid NUMA
node.  However, currently, when a CPU goes offline, the mapping is
destroyed and cpu_to_node() returns NUMA_NO_NODE.

This has always been broken but hasn't triggered often enough before
874bbfe600a6 ("workqueue: make sure delayed work run in local cpu").
After the commit, workqueue forcifully assigns the local CPU for
delayed work items without explicit target CPU to fix a different
issue.  This widens the window where CPU can go offline while a
delayed work item is pending causing delayed work items dispatched
with target CPU set to an already offlined CPU.  The resulting
NUMA_NO_NODE mapping makes workqueue try to queue the work item on a
NULL pool_workqueue and thus crash.

While 874bbfe600a6 has been reverted for a different reason making the
bug less visible again, it can still happen.  Fix it by mapping
NUMA_NO_NODE to the default pool_workqueue from unbound_pwq_by_node().
This is a temporary workaround.  The long term solution is keeping CPU
-> NODE mapping stable across CPU off/online cycles which is being
worked on.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Tang Chen <tangchen@cn.fujitsu.com>
Cc: Rafael J. Wysocki <rafael@kernel.org>
Cc: Len Brown <len.brown@intel.com>
Cc: stable@vger.kernel.org
Link: http://lkml.kernel.org/g/1454424264.11183.46.camel@gmail.com
Link: http://lkml.kernel.org/g/1453702100-2597-1-git-send-email-tangchen@cn.fujitsu.com
---
 kernel/workqueue.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index dc7faad..3ca7ab8 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -570,6 +570,16 @@ static struct pool_workqueue *unbound_pwq_by_node(struct workqueue_struct *wq,
 						  int node)
 {
 	assert_rcu_or_wq_mutex_or_pool_mutex(wq);
+
+	/*
+	 * XXX: @node can be NUMA_NO_NODE if CPU goes offline while a
+	 * delayed item is pending.  The plan is to keep CPU -> NODE
+	 * mapping valid and stable across CPU on/offlines.  Once that
+	 * happens, this workaround can be removed.
+	 */
+	if (unlikely(node == NUMA_NO_NODE))
+		return wq->dfl_pwq;
+
 	return rcu_dereference_raw(wq->numa_pwq_tbl[node]);
 }
 
-- 
2.5.0

[toc] | [next] | [standalone]

#1334655 — Re: [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for unbound pool_workqueue lookup

On Wed 10-02-16 10:55:03, Tejun Heo wrote:
[...]
> --- a/kernel/workqueue.c
> +++ b/kernel/workqueue.c
> @@ -570,6 +570,16 @@ static struct pool_workqueue *unbound_pwq_by_node(struct workqueue_struct *wq,
>  						  int node)
>  {
>  	assert_rcu_or_wq_mutex_or_pool_mutex(wq);
> +
> +	/*
> +	 * XXX: @node can be NUMA_NO_NODE if CPU goes offline while a
> +	 * delayed item is pending.  The plan is to keep CPU -> NODE
> +	 * mapping valid and stable across CPU on/offlines.  Once that
> +	 * happens, this workaround can be removed.

I am not sure this is completely true with the code as is currently.
Don't wee also need to use cpu_to_mem to handle memoryless CPUs?

> +	 */
> +	if (unlikely(node == NUMA_NO_NODE))
> +		return wq->dfl_pwq;
> +
>  	return rcu_dereference_raw(wq->numa_pwq_tbl[node]);
>  }


diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index c579dbab2e36..4785b895b9b5 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -1325,7 +1325,7 @@ static void __queue_work(int cpu, struct workqueue_struct *wq,
 	if (!(wq->flags & WQ_UNBOUND))
 		pwq = per_cpu_ptr(wq->cpu_pwqs, cpu);
 	else
-		pwq = unbound_pwq_by_node(wq, cpu_to_node(cpu));
+		pwq = unbound_pwq_by_node(wq, cpu_to_mem(cpu));
 
 	/*
 	 * If @work was previously on a different pool, it might still be

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1334699 — Re: [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for unbound pool_workqueue lookup

Hello, Michal.

On Mon, Feb 15, 2016 at 06:33:46PM +0100, Michal Hocko wrote:
> On Wed 10-02-16 10:55:03, Tejun Heo wrote:
> [...]
> > --- a/kernel/workqueue.c
> > +++ b/kernel/workqueue.c
> > @@ -570,6 +570,16 @@ static struct pool_workqueue *unbound_pwq_by_node(struct workqueue_struct *wq,
> >  						  int node)
> >  {
> >  	assert_rcu_or_wq_mutex_or_pool_mutex(wq);
> > +
> > +	/*
> > +	 * XXX: @node can be NUMA_NO_NODE if CPU goes offline while a
> > +	 * delayed item is pending.  The plan is to keep CPU -> NODE
> > +	 * mapping valid and stable across CPU on/offlines.  Once that
> > +	 * happens, this workaround can be removed.
> 
> I am not sure this is completely true with the code as is currently.
> Don't wee also need to use cpu_to_mem to handle memoryless CPUs?

I'm not sure.  I think we still wan to distinguish workers for a
memoryless node from its neighboring node with memory.  We don't want
work items for the latter to be randomly distributed to the former
after all.

Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1334807 — Re: [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for unbound pool_workqueue lookup

On Mon 15-02-16 13:21:25, Tejun Heo wrote:
> Hello, Michal.
> 
> On Mon, Feb 15, 2016 at 06:33:46PM +0100, Michal Hocko wrote:
> > On Wed 10-02-16 10:55:03, Tejun Heo wrote:
> > [...]
> > > --- a/kernel/workqueue.c
> > > +++ b/kernel/workqueue.c
> > > @@ -570,6 +570,16 @@ static struct pool_workqueue *unbound_pwq_by_node(struct workqueue_struct *wq,
> > >  						  int node)
> > >  {
> > >  	assert_rcu_or_wq_mutex_or_pool_mutex(wq);
> > > +
> > > +	/*
> > > +	 * XXX: @node can be NUMA_NO_NODE if CPU goes offline while a
> > > +	 * delayed item is pending.  The plan is to keep CPU -> NODE
> > > +	 * mapping valid and stable across CPU on/offlines.  Once that
> > > +	 * happens, this workaround can be removed.
> > 
> > I am not sure this is completely true with the code as is currently.
> > Don't wee also need to use cpu_to_mem to handle memoryless CPUs?
> 
> I'm not sure.  I think we still wan to distinguish workers for a
> memoryless node from its neighboring node with memory.  We don't want
> work items for the latter to be randomly distributed to the former
> after all.

I am not sure I understand. Does that mean that a node with no memory
would have its WQ specific pool? I might be missing something but I
thought that cpu_to_node will return NUMA_NO_NODE if the CPU is memory
less. Or do you expect that cpu_to_node will always return a valid node
id even when it doesn't contain any memory?

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1334825 — Re: [PATCH wq/for-4.5-fixes] workqueue: handle NUMA_NO_NODE for unbound pool_workqueue lookup

Hello, Michal.

On Mon, Feb 15, 2016 at 09:54:38PM +0100, Michal Hocko wrote:
> I am not sure I understand. Does that mean that a node with no memory
> would have its WQ specific pool? I might be missing something but I

Yeah, I think it should.

> thought that cpu_to_node will return NUMA_NO_NODE if the CPU is memory
> less. Or do you expect that cpu_to_node will always return a valid node
> id even when it doesn't contain any memory?

Hmmm... that was what I was expecting.  It's not really looking for
memory node to use but identify affinity there.

Thanks.

-- 
tejun

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web