Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1272146 > unrolled thread

[PATCH v3 07/22] kthread: Detect when a kthread work is used by more workers

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH v3 07/22] kthread: Detect when a kthread work is used by more workers Petr Mladek <pmladek@suse.com> - 2015-11-18 14:30 +0100
    Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by  more workers Tejun Heo <tj@kernel.org> - 2015-11-23 23:30 +0100
      Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by  more workers Petr Mladek <pmladek@suse.com> - 2015-11-24 11:10 +0100
        Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by  more workers Tejun Heo <tj@kernel.org> - 2015-11-24 15:50 +0100
          Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by  more workers Petr Mladek <pmladek@suse.com> - 2015-11-24 17:30 +0100
        Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by  more workers Peter Zijlstra <peterz@infradead.org> - 2015-11-24 16:00 +0100

#1272146 — [PATCH v3 07/22] kthread: Detect when a kthread work is used by more workers

Nothing currently prevents a work from queuing for a kthread worker
when it is already running on another one. This means that the work
might run in parallel on more workers. Also some operations, e.g.
flush or drain are not reliable.

This problem will be even more visible after we add cancel_kthread_work()
function. It will only have "work" as the parameter and will use
worker->lock to synchronize with others.

Well, normally this is not a problem because the API users are sane.
But bugs might happen and users also might be crazy.

This patch adds a warning when we try to insert the work for another
worker. It does not fully prevent the misuse because it would make the
code much more complicated without a big benefit.

Note that we need to clear the information about the current worker
when the work is not longer used. It is important when the worker
is destroyed and later created again. For example, this is
useful when a service might get disabled and enabled via sysfs.

Also note that kthread_work_pending() function will get more
complicated once we add support for a delayed kthread work and
allow to cancel works.

Just for completeness, the patch adds a check for disabled interrupts
and an empty queue.

The patch also puts all the checks into a separate function. It will
be reused when implementing delayed works.

Signed-off-by: Petr Mladek <pmladek@suse.com>
---
 kernel/kthread.c | 45 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 41 insertions(+), 4 deletions(-)

diff --git a/kernel/kthread.c b/kernel/kthread.c
index 1d41e0faef2d..378d2203c8b0 100644
--- a/kernel/kthread.c
+++ b/kernel/kthread.c
@@ -563,6 +563,18 @@ void __init_kthread_worker(struct kthread_worker *worker,
 }
 EXPORT_SYMBOL_GPL(__init_kthread_worker);
 
+/*
+ * Returns true when there is a pending operation for this work.
+ * In particular, it checks if the work is:
+ *	- queued
+ *
+ * This function must be called with locked work.
+ */
+static inline bool kthread_work_pending(const struct kthread_work *work)
+{
+	return !list_empty(&work->node);
+}
+
 /**
  * kthread_worker_fn - kthread function to process kthread_worker
  * @worker_ptr: pointer to initialized kthread_worker
@@ -574,6 +586,9 @@ EXPORT_SYMBOL_GPL(__init_kthread_worker);
  * The works are not allowed to keep any locks, disable preemption or interrupts
  * when they finish. There is defined a safe point for freezing when one work
  * finishes and before a new one is started.
+ *
+ * Also the works must not be handled by more workers at the same time, see also
+ * queue_kthread_work().
  */
 int kthread_worker_fn(void *worker_ptr)
 {
@@ -610,6 +625,12 @@ repeat:
 	if (work) {
 		__set_current_state(TASK_RUNNING);
 		work->func(work);
+
+		spin_lock_irq(&worker->lock);
+		/* Allow to queue the work into another worker */
+		if (!kthread_work_pending(work))
+			work->worker = NULL;
+		spin_unlock_irq(&worker->lock);
 	} else if (!freezing(current))
 		schedule();
 
@@ -696,12 +717,22 @@ create_kthread_worker_on_cpu(int cpu, const char namefmt[])
 }
 EXPORT_SYMBOL(create_kthread_worker_on_cpu);
 
+static void insert_kthread_work_sanity_check(struct kthread_worker *worker,
+					       struct kthread_work *work)
+{
+	lockdep_assert_held(&worker->lock);
+	WARN_ON_ONCE(!irqs_disabled());
+	WARN_ON_ONCE(!list_empty(&work->node));
+	/* Do not use a work with more workers, see queue_kthread_work() */
+	WARN_ON_ONCE(work->worker && work->worker != worker);
+}
+
 /* insert @work before @pos in @worker */
 static void insert_kthread_work(struct kthread_worker *worker,
-			       struct kthread_work *work,
-			       struct list_head *pos)
+				struct kthread_work *work,
+				struct list_head *pos)
 {
-	lockdep_assert_held(&worker->lock);
+	insert_kthread_work_sanity_check(worker, work);
 
 	list_add_tail(&work->node, pos);
 	work->worker = worker;
@@ -717,6 +748,12 @@ static void insert_kthread_work(struct kthread_worker *worker,
  * Queue @work to work processor @task for async execution.  @task
  * must have been created with kthread_worker_create().  Returns %true
  * if @work was successfully queued, %false if it was already pending.
+ *
+ * Never queue a work into a worker when it is being processed by another
+ * one. Otherwise, some operations, e.g. cancel or flush, will not work
+ * correctly or the work might run in parallel. This is not enforced
+ * because it would make the code too complex. There are only warnings
+ * printed when such a situation is detected.
  */
 bool queue_kthread_work(struct kthread_worker *worker,
 			struct kthread_work *work)
@@ -725,7 +762,7 @@ bool queue_kthread_work(struct kthread_worker *worker,
 	unsigned long flags;
 
 	spin_lock_irqsave(&worker->lock, flags);
-	if (list_empty(&work->node)) {
+	if (!kthread_work_pending(work)) {
 		insert_kthread_work(worker, work, &worker->work_list);
 		ret = true;
 	}
-- 
1.8.5.6

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1275931 — Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by more workers

Hello,

On Wed, Nov 18, 2015 at 02:25:12PM +0100, Petr Mladek wrote:
> @@ -610,6 +625,12 @@ repeat:
>  	if (work) {
>  		__set_current_state(TASK_RUNNING);
>  		work->func(work);
> +
> +		spin_lock_irq(&worker->lock);
> +		/* Allow to queue the work into another worker */
> +		if (!kthread_work_pending(work))
> +			work->worker = NULL;
> +		spin_unlock_irq(&worker->lock);

Doesn't this mean that the work item can't be freed from its callback?
That pattern tends to happen regularly.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276263 — Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by more workers

On Mon 2015-11-23 17:27:03, Tejun Heo wrote:
> Hello,
> 
> On Wed, Nov 18, 2015 at 02:25:12PM +0100, Petr Mladek wrote:
> > @@ -610,6 +625,12 @@ repeat:
> >  	if (work) {
> >  		__set_current_state(TASK_RUNNING);
> >  		work->func(work);
> > +
> > +		spin_lock_irq(&worker->lock);
> > +		/* Allow to queue the work into another worker */
> > +		if (!kthread_work_pending(work))
> > +			work->worker = NULL;
> > +		spin_unlock_irq(&worker->lock);
> 
> Doesn't this mean that the work item can't be freed from its callback?
> That pattern tends to happen regularly.

I am not sure if I understand your question. Do you mean switching
work->func during the life time of the struct kthread_work? This
should not be affected by the above code.

The above code allows to queue an _unused_ kthread_work into any
kthread_worker. For example, it is needed for khugepaged,
see http://marc.info/?l=linux-kernel&m=144785344924871&w=2
The work is static but the worker can be started/stopped
(allocated/freed) repeatedly. It means that the work need
to be usable with many workers. But it is associated only
with one worker when being used.

If the work is in use (pending or being proceed), we must not
touch work->worker. Otherwise there might be a race. Because
all the operations with the work are synchronized using
work->worker->lock.

I hope that it makes sense.

Thanks a lot for feedback,
Petr
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276529 — Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by more workers

Hello, Petr.

On Tue, Nov 24, 2015 at 11:06:50AM +0100, Petr Mladek wrote:
> > > @@ -610,6 +625,12 @@ repeat:
> > >  	if (work) {
> > >  		__set_current_state(TASK_RUNNING);
> > >  		work->func(work);
> > > +
> > > +		spin_lock_irq(&worker->lock);
> > > +		/* Allow to queue the work into another worker */
> > > +		if (!kthread_work_pending(work))
> > > +			work->worker = NULL;
> > > +		spin_unlock_irq(&worker->lock);
> > 
> > Doesn't this mean that the work item can't be freed from its callback?
> > That pattern tends to happen regularly.
> 
> I am not sure if I understand your question. Do you mean switching
> work->func during the life time of the struct kthread_work? This
> should not be affected by the above code.

So, something like the following.

void my_work_fn(work)
{
	struct my_struct *s = container_of(work, ...);

	do something with s;
	kfree(s);
}

and the queuer does

	struct my_struct *s = kmalloc(sizeof(*s));

	init s and s->work;
	queue(&s->work);

expecting s to be freed on completion.  IOW, you can't expect the work
item to remain accessible once the work function starts executing.

> The above code allows to queue an _unused_ kthread_work into any
> kthread_worker. For example, it is needed for khugepaged,
> see http://marc.info/?l=linux-kernel&m=144785344924871&w=2
> The work is static but the worker can be started/stopped
> (allocated/freed) repeatedly. It means that the work need
> to be usable with many workers. But it is associated only
> with one worker when being used.

It can just re-init work items when it restarts workers, right?

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276624 — Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by more workers

On Tue 2015-11-24 09:49:42, Tejun Heo wrote:
> Hello, Petr.
> 
> On Tue, Nov 24, 2015 at 11:06:50AM +0100, Petr Mladek wrote:
> > > > @@ -610,6 +625,12 @@ repeat:
> > > >  	if (work) {
> > > >  		__set_current_state(TASK_RUNNING);
> > > >  		work->func(work);
> > > > +
> > > > +		spin_lock_irq(&worker->lock);
> > > > +		/* Allow to queue the work into another worker */
> > > > +		if (!kthread_work_pending(work))
> > > > +			work->worker = NULL;
> > > > +		spin_unlock_irq(&worker->lock);
> > > 
> > > Doesn't this mean that the work item can't be freed from its callback?
> > > That pattern tends to happen regularly.
> > 
> > I am not sure if I understand your question. Do you mean switching
> > work->func during the life time of the struct kthread_work? This
> > should not be affected by the above code.
> 
>IOW, you can't expect the work
> item to remain accessible once the work function starts executing.

I see, I was not aware of this pattern.


> > The above code allows to queue an _unused_ kthread_work into any
> > kthread_worker. For example, it is needed for khugepaged,
> > see http://marc.info/?l=linux-kernel&m=144785344924871&w=2
> > The work is static but the worker can be started/stopped
> > (allocated/freed) repeatedly. It means that the work need
> > to be usable with many workers. But it is associated only
> > with one worker when being used.
>
> It can just re-init work items when it restarts workers, right?

Yes, this would work. It might be slightly inconvenient but
it looks like a good compromise. It helps to keep the API
implementation rather simple and rather secure.

Alternatively, we could allow to queue the work on another worker
if it is not pending. But then we would need to check the pending
status without the worker->lock because work->worker might point
to an already freed worker. We need to check the pending
status in many situations. It might open a can of worms that
I probably do not want to catch.

Thank you and PeterZ for explanation,
Petr
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1276542 — Re: [PATCH v3 07/22] kthread: Detect when a kthread work is used by more workers

On Tue, Nov 24, 2015 at 11:06:50AM +0100, Petr Mladek wrote:
> On Mon 2015-11-23 17:27:03, Tejun Heo wrote:
> > Hello,
> > 
> > On Wed, Nov 18, 2015 at 02:25:12PM +0100, Petr Mladek wrote:
> > > @@ -610,6 +625,12 @@ repeat:
> > >  	if (work) {
> > >  		__set_current_state(TASK_RUNNING);
> > >  		work->func(work);
> > > +
> > > +		spin_lock_irq(&worker->lock);
> > > +		/* Allow to queue the work into another worker */
> > > +		if (!kthread_work_pending(work))
> > > +			work->worker = NULL;
> > > +		spin_unlock_irq(&worker->lock);
> > 
> > Doesn't this mean that the work item can't be freed from its callback?
> > That pattern tends to happen regularly.
> 
> I am not sure if I understand your question. Do you mean switching
> work->func during the life time of the struct kthread_work? This
> should not be affected by the above code.

No, work->func(work) doing: kfree(work).

That is indeed something quite frequently done, and since you now have
references to work after calling func, things would go *boom* rather
quickly.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web