Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1370829 > unrolled thread

Re: [PATCH] nohz_full: Make sched_should_stop_tick() more conservative

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] nohz_full: Make sched_should_stop_tick() more  conservative Rik van Riel <riel@redhat.com> - 2016-04-04 21:20 +0200
    Re: [PATCH] nohz_full: Make sched_should_stop_tick() more conservative Peter Zijlstra <peterz@infradead.org> - 2016-04-04 21:30 +0200
    Re: [PATCH] nohz_full: Make sched_should_stop_tick() more  conservative Rik van Riel <riel@redhat.com> - 2016-04-04 21:40 +0200

#1370829 — Re: [PATCH] nohz_full: Make sched_should_stop_tick() more conservative

[Multipart message — attachments visible in raw view] — view raw

On Fri, 2016-04-01 at 15:42 -0400, Chris Metcalf wrote:
> On arm64, when calling enqueue_task_fair() from migration_cpu_stop(),
> we find the nr_running value updated by add_nr_running(), but the
> cfs.nr_running value has not always yet been updated.  Accordingly,
> the sched_can_stop_tick() false returns true when we are migrating a
> second task onto a core.

I don't get it.

Looking at the enqueue_task_fair(), I see this:

        for_each_sched_entity(se) {
                cfs_rq = cfs_rq_of(se);
                cfs_rq->h_nr_running++;
		...
	}

        if (!se)
                add_nr_running(rq, 1);

What is the difference between cfs_rq->h_nr_running,
and rq->cfs.nr_running?

Why do we have two?

Are we simply testing against the wrong one in
sched_can_stop_tick?

> Correct this by using rq->nr_running instead of rq->cfs.nr_running.
> This should always be more conservative, and reverts the test to the
> form it had before commit 76d92ac305f2 ("sched: Migrate sched to use
> new tick dependency mask model").

That would cause us to run the timer tick while running
a single SCHED_RR real time task, with a single
SCHED_OTHER task sitting in the background (which will
not get run until the SCHED_RR task is done).

I don't think that is the quite behaviour we want.

> Signed-off-by: Chris Metcalf <cmetcalf@mellanox.com>
> ---
> I found this bug because I had a program running in nohz_full
> on a core, and from a different core I called sched_setaffinity()
> to force that task onto the nohz_full core, but I did not end up with
> a kick to the nohz_full core, so tick-based scheduling did not start.
> This is probably bad enough that we should fix it for 4.6.
> 
> Strangely, for some reason, the existing code worked correctly for me
> for tilegx, but not for arm64.  I see that the enqueue_task_fair()
> code calls enqueue_entity(), which calls account_entity_enqueue() to
> adjust cfs.nr_running.  That seemed to happen on tilegx, but not
> arm64.
> Perhaps there is some difference in how the sched_entity stuff is
> done,
> but frankly that took me a little deeper into the CFS stuff than I
> was
> willing to dive in this moment.
> 
> I could also argue that sched/core.c shouldn't have a lot of CFS
> stuff in it anyway, and if we view the FIFO/RR stuff as handling the
> real special cases in sched_can_stop_tick() anyway, then just
> checking
> the core nr_running feels like the right thing to do regardless.
> 
>  kernel/sched/core.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 00649f7ad567..1737d63c65fa 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -599,7 +599,7 @@ bool sched_can_stop_tick(struct rq *rq)
>  	}
>  
>  	/* Normal multitasking need periodic preemption checks */
> -	if (rq->cfs.nr_running > 1)
> +	if (rq->nr_running > 1)
>  		return false;
>  
>  	return true;
-- 
All Rights Reversed.

[toc] | [next] | [standalone]

#1370830 — Re: [PATCH] nohz_full: Make sched_should_stop_tick() more conservative

On 4 April 2016 21:12:23 CEST, Rik van Riel <riel@redhat.com> wrote:

>What is the difference between cfs_rq->h_nr_running,
>and rq->cfs.nr_running?
>
>Why do we have two?


H is for hierarchy. That counts the total of runnable tasks in the entire child hierarchy. Nr_running is the number of se  entities in the current tree.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[toc] | [prev] | [next] | [standalone]

#1370834

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2016-04-04 at 15:31 -0400, Chris Metcalf wrote:
> On 4/4/2016 3:12 PM, Rik van Riel wrote:
> > 
> > On Fri, 2016-04-01 at 15:42 -0400, Chris Metcalf wrote:
> > > 
> > > On arm64, when calling enqueue_task_fair() from
> > > migration_cpu_stop(),
> > > we find the nr_running value updated by add_nr_running(), but the
> > > cfs.nr_running value has not always yet been
> > > updated.  Accordingly,
> > > the sched_can_stop_tick() false returns true when we are
> > > migrating a
> > > second task onto a core.
> > I don't get it.
> > 
> > Looking at the enqueue_task_fair(), I see this:
> > 
> >          for_each_sched_entity(se) {
> >                  cfs_rq = cfs_rq_of(se);
> >                  cfs_rq->h_nr_running++;
> > 		...
> > 	}
> > 
> >          if (!se)
> >                  add_nr_running(rq, 1);
> > 
> > What is the difference between cfs_rq->h_nr_running,
> > and rq->cfs.nr_running?
> > 
> > Why do we have two?
> > Are we simply testing against the wrong one in
> > sched_can_stop_tick?
> It seems that using the non-CFS one is what we want.  I don't know
> whether
> using a different CFS count instead might be more correct.
> 
> Since I'm not sure what causes the difference I see between tile
> (correct)
> and arm64 (incorrect) it's hard for me to speculate.
> 
> > 
> > > 
> > > Correct this by using rq->nr_running instead of rq-
> > > >cfs.nr_running.
> > > This should always be more conservative, and reverts the test to
> > > the
> > > form it had before commit 76d92ac305f2 ("sched: Migrate sched to
> > > use
> > > new tick dependency mask model").
> > That would cause us to run the timer tick while running
> > a single SCHED_RR real time task, with a single
> > SCHED_OTHER task sitting in the background (which will
> > not get run until the SCHED_RR task is done).
> No, because in sched_can_stop_tick(), we first handle the special
> cases of RR or FIFO tasks present.  For example, RR:
> 
>          if (rq->rt.rr_nr_running) {
>                  if (rq->rt.rr_nr_running == 1)
>                          return true;
>                  else
>                          return false;
>          }
> 
> Once we see there's any RR tasks running, the return value
> ignores any possible SCHED_OTHER tasks.  Only after the code
> concludes there are no RR/FIFO tasks do we even look at
> the over nr_running value.

OK, fair enough. I guess both of the RT cases are
covered already.

Patch gets my:

Acked-by: Rik van Riel <riel@redhat.com>

-- 
All Rights Reversed.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web