Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1312199 > unrolled thread

Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task enqueue/dequeue

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Peter Zijlstra <peterz@infradead.org> - 2016-01-19 14:20 +0100
    Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Frederic Weisbecker <fweisbec@gmail.com> - 2016-01-19 18:10 +0100
      Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Peter Zijlstra <peterz@infradead.org> - 2016-01-20 10:20 +0100
        Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Frederic Weisbecker <fweisbec@gmail.com> - 2016-01-20 16:00 +0100
          Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Thomas Gleixner <tglx@linutronix.de> - 2016-01-20 16:20 +0100
            Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Christoph Lameter <cl@linux.com> - 2016-01-20 16:20 +0100
              Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Frederic Weisbecker <fweisbec@gmail.com> - 2016-01-20 18:00 +0100
            Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Frederic Weisbecker <fweisbec@gmail.com> - 2016-01-20 17:50 +0100
          Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Peter Zijlstra <peterz@infradead.org> - 2016-01-20 18:00 +0100
            Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Frederic Weisbecker <fweisbec@gmail.com> - 2016-01-20 18:30 +0100
              Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Peter Zijlstra <peterz@infradead.org> - 2016-01-20 19:30 +0100
                Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task  enqueue/dequeue Frederic Weisbecker <fweisbec@gmail.com> - 2016-01-21 14:30 +0100

#1312199 — Re: [RFC PATCH 4/4] sched: Upload nohz full CPU load on task enqueue/dequeue

On Wed, Jan 13, 2016 at 05:01:31PM +0100, Frederic Weisbecker wrote:
> The full nohz CPU load is currently accounted on tick restart only.
> But there are a few issues with this model:
> 
> _ On tick restart, if cpu_load[0] doesn't contain the load of the actual
>   tickless load that just ran, we are going to account a wrong value.
>   And it is very likely to be so given that cpu_load[0] doesn't have
>   an opportunity to be updated between tick stop and tick restart.
> 
> _ If the runqueue had updates that didn't trigger a tick restart, we
>   are going to miss those CPU load changes.
> 
> A solution to fix this is to update the CPU load everytime we enqueue
> or dequeue a task in the fair runqueue and more than a jiffy occured
> since the last update.

Would not a much better solution be to do this remotely instead of from
one of the hottest functions in the scheduler?

[toc] | [next] | [standalone]

#1312390

On Tue, Jan 19, 2016 at 02:17:08PM +0100, Peter Zijlstra wrote:
> On Wed, Jan 13, 2016 at 05:01:31PM +0100, Frederic Weisbecker wrote:
> > The full nohz CPU load is currently accounted on tick restart only.
> > But there are a few issues with this model:
> > 
> > _ On tick restart, if cpu_load[0] doesn't contain the load of the actual
> >   tickless load that just ran, we are going to account a wrong value.
> >   And it is very likely to be so given that cpu_load[0] doesn't have
> >   an opportunity to be updated between tick stop and tick restart.
> > 
> > _ If the runqueue had updates that didn't trigger a tick restart, we
> >   are going to miss those CPU load changes.
> > 
> > A solution to fix this is to update the CPU load everytime we enqueue
> > or dequeue a task in the fair runqueue and more than a jiffy occured
> > since the last update.
> 
> Would not a much better solution be to do this remotely instead of from
> one of the hottest functions in the scheduler?

The problem with doing this remotely is that we can miss past cpu loads if
there was several enqueue/dequeue operations happening while tickless.

For example if CPU 1 runs sched entity A with a load of 5 (purely theoric)
for 5 sec then it sleeps, entity B runs with a load of 1 and then CPU 2 updates
the load of CPU 1 remotely. cpu_load[0] will be accurate because it's the current
load of CPU 1 (which is the load of entity B), but the other indexes won't take
the decayed load of entity A into account.

Now we can indeed remove the queue time local update and only rely on remote
updates when needed if we can live with a light precision on target_load() and
source_load().

[toc] | [prev] | [next] | [standalone]

#1313021

On Tue, Jan 19, 2016 at 06:03:19PM +0100, Frederic Weisbecker wrote:
> On Tue, Jan 19, 2016 at 02:17:08PM +0100, Peter Zijlstra wrote:
> > On Wed, Jan 13, 2016 at 05:01:31PM +0100, Frederic Weisbecker wrote:
> > > The full nohz CPU load is currently accounted on tick restart only.
> > > But there are a few issues with this model:
> > > 
> > > _ On tick restart, if cpu_load[0] doesn't contain the load of the actual
> > >   tickless load that just ran, we are going to account a wrong value.
> > >   And it is very likely to be so given that cpu_load[0] doesn't have
> > >   an opportunity to be updated between tick stop and tick restart.
> > > 
> > > _ If the runqueue had updates that didn't trigger a tick restart, we
> > >   are going to miss those CPU load changes.
> > > 
> > > A solution to fix this is to update the CPU load everytime we enqueue
> > > or dequeue a task in the fair runqueue and more than a jiffy occured
> > > since the last update.
> > 
> > Would not a much better solution be to do this remotely instead of from
> > one of the hottest functions in the scheduler?
> 
> The problem with doing this remotely is that we can miss past cpu loads if
> there was several enqueue/dequeue operations happening while tickless.

Its a timer based sample, it _always_ and per definition misses
intermediate state.

You can simply do:

	for_each_nohzfull_cpu(cpu) {
		struct rq *rq = rq_of(cpu);

		raw_spin_lock(&rq->lock);
		update_cpu_load_active(rq);
		raw_spin_unlock(&rq->lock);
	}

Also, since when can we have enqueues/dequeues while NOHZ_FULL ? I
thought that was the 1 task 100% cpu case, there are no
enqueues/dequeues there.

[toc] | [prev] | [next] | [standalone]

#1313270

On Wed, Jan 20, 2016 at 10:09:06AM +0100, Peter Zijlstra wrote:
> On Tue, Jan 19, 2016 at 06:03:19PM +0100, Frederic Weisbecker wrote:
> > On Tue, Jan 19, 2016 at 02:17:08PM +0100, Peter Zijlstra wrote:
> > > On Wed, Jan 13, 2016 at 05:01:31PM +0100, Frederic Weisbecker wrote:
> > > > The full nohz CPU load is currently accounted on tick restart only.
> > > > But there are a few issues with this model:
> > > > 
> > > > _ On tick restart, if cpu_load[0] doesn't contain the load of the actual
> > > >   tickless load that just ran, we are going to account a wrong value.
> > > >   And it is very likely to be so given that cpu_load[0] doesn't have
> > > >   an opportunity to be updated between tick stop and tick restart.
> > > > 
> > > > _ If the runqueue had updates that didn't trigger a tick restart, we
> > > >   are going to miss those CPU load changes.
> > > > 
> > > > A solution to fix this is to update the CPU load everytime we enqueue
> > > > or dequeue a task in the fair runqueue and more than a jiffy occured
> > > > since the last update.
> > > 
> > > Would not a much better solution be to do this remotely instead of from
> > > one of the hottest functions in the scheduler?
> > 
> > The problem with doing this remotely is that we can miss past cpu loads if
> > there was several enqueue/dequeue operations happening while tickless.
> 
> Its a timer based sample, it _always_ and per definition misses
> intermediate state.

Sure, but the problem is when these intermediate states are long enough.

> 
> You can simply do:
> 
> 	for_each_nohzfull_cpu(cpu) {
> 		struct rq *rq = rq_of(cpu);
> 
> 		raw_spin_lock(&rq->lock);
> 		update_cpu_load_active(rq);
> 		raw_spin_unlock(&rq->lock);
> 	}

But from where should we do that? Maybe we can do it before we call source/target_load(),
on the selected targets needed by the caller? The problem is that if we do that right
after a task got enqueued on the nohz runqueue, we may accidentally account it as the
whole dynticks frame (I mean, if we get rid of that enqueue/dequeue accounting).

> 
> Also, since when can we have enqueues/dequeues while NOHZ_FULL ? I
> thought that was the 1 task 100% cpu case, there are no
> enqueues/dequeues there.

That's the most optimized case but we can definetly have small moments with more
than one task running. For example if we have a workqueue, or such short and quick tasks.

If the user makes use of full dynticks for soft isolation (for performance, can live
with a few interrupts...), there can be short moments of multitasking.

[toc] | [prev] | [next] | [standalone]

#1313286

On Wed, 20 Jan 2016, Frederic Weisbecker wrote:
> On Wed, Jan 20, 2016 at 10:09:06AM +0100, Peter Zijlstra wrote:
> > Also, since when can we have enqueues/dequeues while NOHZ_FULL ? I
> > thought that was the 1 task 100% cpu case, there are no
> > enqueues/dequeues there.
> 
> That's the most optimized case but we can definetly have small moments with
> more than one task running. For example if we have a workqueue, or such
> short and quick tasks.
>
> If the user makes use of full dynticks for soft isolation (for performance,
> can live with a few interrupts...), there can be short moments of
> multitasking.

Again, you are trying to make the second step after the first one is
completed. We do not even have proper accounting when we have the ONE task
100% case and still you try to solve problems beyond that.

If that ONE task gets interrupted, then accounting should take place.

When there is another runnable task then that nohz state needs to be left. You
can go back to it once the task is alone again.

You are trying to make the complete accounting 'almost' tick independent, but
approaching that from the tick nohz angle is wrong.

When you really want to go there, and I can see why you want that, then you
need to solve this from ground up and such a solution has nothing to do with
any flavour of NOHZ. That simply needs to rework the whole accounting
machinery and rip out the complete tick dependency. Once you have that your
NOHZ business falls into place. Any other approach is just duct tape
engineering.
 
Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1313293

On Wed, 20 Jan 2016, Thomas Gleixner wrote:

> > If the user makes use of full dynticks for soft isolation (for performance,
> > can live with a few interrupts...), there can be short moments of
> > multitasking.

"Soft" isolation? Like soft realtime ... Argh... Please stay away from
corrupting the intents of the nohz full mode.

> Again, you are trying to make the second step after the first one is
> completed. We do not even have proper accounting when we have the ONE task
> 100% case and still you try to solve problems beyond that.

Please lets only deal with the one task issue. I can have "soft" isolation
today by moving daemons etc off certain processors. We want definitely to
run nothing else on the processor and will want to even go further with
the cache allocation techniques in recent processor to limit the cache
disturbances from other processors etc etc.

[toc] | [prev] | [next] | [standalone]

#1313368

On Wed, Jan 20, 2016 at 09:19:36AM -0600, Christoph Lameter wrote:
> On Wed, 20 Jan 2016, Thomas Gleixner wrote:
> 
> > > If the user makes use of full dynticks for soft isolation (for performance,
> > > can live with a few interrupts...), there can be short moments of
> > > multitasking.
> 
> "Soft" isolation? Like soft realtime ... Argh... Please stay away from
> corrupting the intents of the nohz full mode.

Well some people, especially real time, want no disturbance at all because
their workload really depends on that.

Some other (HPC) don't care about little disturbance.

Although HPC users didn't declare themselves yet, the real time case
expect some more sacrifices (see nohz tasks patchset).

> 
> > Again, you are trying to make the second step after the first one is
> > completed. We do not even have proper accounting when we have the ONE task
> > 100% case and still you try to solve problems beyond that.
> 
> Please lets only deal with the one task issue. I can have "soft" isolation
> today by moving daemons etc off certain processors. We want definitely to
> run nothing else on the processor and will want to even go further with
> the cache allocation techniques in recent processor to limit the cache
> disturbances from other processors etc etc.

One task is common to all usecases anyway.

[toc] | [prev] | [next] | [standalone]

#1313353

On Wed, Jan 20, 2016 at 04:11:14PM +0100, Thomas Gleixner wrote:
> On Wed, 20 Jan 2016, Frederic Weisbecker wrote:
> > On Wed, Jan 20, 2016 at 10:09:06AM +0100, Peter Zijlstra wrote:
> > > Also, since when can we have enqueues/dequeues while NOHZ_FULL ? I
> > > thought that was the 1 task 100% cpu case, there are no
> > > enqueues/dequeues there.
> > 
> > That's the most optimized case but we can definetly have small moments with
> > more than one task running. For example if we have a workqueue, or such
> > short and quick tasks.
> >
> > If the user makes use of full dynticks for soft isolation (for performance,
> > can live with a few interrupts...), there can be short moments of
> > multitasking.
> 
> Again, you are trying to make the second step after the first one is
> completed. We do not even have proper accounting when we have the ONE task
> 100% case and still you try to solve problems beyond that.
> 
> If that ONE task gets interrupted, then accounting should take place.
> 
> When there is another runnable task then that nohz state needs to be left. You
> can go back to it once the task is alone again.
> 
> You are trying to make the complete accounting 'almost' tick independent, but
> approaching that from the tick nohz angle is wrong.
> 
> When you really want to go there, and I can see why you want that, then you
> need to solve this from ground up and such a solution has nothing to do with
> any flavour of NOHZ. That simply needs to rework the whole accounting
> machinery and rip out the complete tick dependency. Once you have that your
> NOHZ business falls into place. Any other approach is just duct tape
> engineering.

Alright, let me respawn that series with handling the most simple and common
scenario, which is 100% single task in full dynticks, and anything else keeps
the tick. That will indeed allow us a more incremental approach.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1313369

On Wed, Jan 20, 2016 at 03:54:19PM +0100, Frederic Weisbecker wrote:

> > You can simply do:
> > 
> > 	for_each_nohzfull_cpu(cpu) {
> > 		struct rq *rq = rq_of(cpu);
> > 
> > 		raw_spin_lock(&rq->lock);
> > 		update_cpu_load_active(rq);
> > 		raw_spin_unlock(&rq->lock);
> > 	}
> 
> But from where should we do that? 

house keeper thingy

> Maybe we can do it before we call source/target_load(), on the
> selected targets needed by the caller? The problem is that if we do
> that right after a task got enqueued on the nohz runqueue, we may
> accidentally account it as the whole dynticks frame (I mean, if we get
> rid of that enqueue/dequeue accounting).

Yes so? What if the current tick happens right after a task get
enqueued? Then we account the whole tick as !idle, even tough we might
have been idle for 99% of the time.

Not a problem, this is sampling.

Doing it locally or remotely doesn't matter.

> > Also, since when can we have enqueues/dequeues while NOHZ_FULL ? I
> > thought that was the 1 task 100% cpu case, there are no
> > enqueues/dequeues there.
> 
> That's the most optimized case but we can definetly have small moments
> with more than one task running. For example if we have a workqueue,
> or such short and quick tasks.

The moment you have nr_running>1 the tick comes back on.

[toc] | [prev] | [next] | [standalone]

#1313401

On Wed, Jan 20, 2016 at 05:56:57PM +0100, Peter Zijlstra wrote:
> On Wed, Jan 20, 2016 at 03:54:19PM +0100, Frederic Weisbecker wrote:
>
> > > You can simply do:
> > >
> > >	for_each_nohzfull_cpu(cpu) {
> > > 				   struct rq *rq = rq_of(cpu);
> > >
> > >		raw_spin_lock(&rq->lock);
> > >			update_cpu_load_active(rq);
> > >				raw_spin_unlock(&rq->lock);
> > > 				}
> >
> > But from where should we do that?
>
> house keeper thingy

You mean a periodic call to the above from the housekeepers?

I didn't think about doing that because you nacked that approach with
scheduler_tick(). This isn't much different.

It means the housekeeper is entirely dedicated to full dynticks CPUs.

>
> > Maybe we can do it before we call source/target_load(), on the
> > selected targets needed by the caller? The problem is that if we do
> > that right after a task got enqueued on the nohz runqueue, we may
> > accidentally account it as the whole dynticks frame (I mean, if we get
> > rid of that enqueue/dequeue accounting).
>
> Yes so? What if the current tick happens right after a task get
> enqueued? Then we account the whole tick as !idle, even tough we might
> have been idle for 99% of the time.

Idle is correctly taken care of there because tick_nohz_idle_exit() makes
sure that the whole dynticks load recorded is 0.

>
> Not a problem, this is sampling.

It's ok to have sampling imprecisions indeed but accounting long samples
of singletask time as multitask is rather erratic than imprecise. Now
that's the issue with pure on-demand updates. If we do the remote update
periodically instead, that wouldn't be a problem anymore as it would just
be about precision.

>
> Doing it locally or remotely doesn't matter.
>
> > > Also, since when can we have enqueues/dequeues while NOHZ_FULL ? I
> > > thought that was the 1 task 100% cpu case, there are no
> > > enqueues/dequeues there.
> >
> > That's the most optimized case but we can definetly have small moments
> > with more than one task running. For example if we have a workqueue,
> > or such short and quick tasks.
>
> The moment you have nr_running>1 the tick comes back on.

Sure, but the current nohz frame exit accounting is wrong at it accounts the
newly woken task as the whole tickless load. We need to record the singletask
load on nohz frame entry at least so we can retrieve and account it on nohz exit.

Unless, again, if we do that housekeeping periodic remote update. Then we don't
care locally at all anymore.

[toc] | [prev] | [next] | [standalone]

#1313431

On Wed, Jan 20, 2016 at 06:21:07PM +0100, Frederic Weisbecker wrote:

> You mean a periodic call to the above from the housekeepers?
> 
> I didn't think about doing that because you nacked that approach with
> scheduler_tick(). This isn't much different.

This does _one_ thing, namely load accounting. scheduler_tick() does a
whole bunch of things, some which really do not make much sense to do
remotely.

> It means the housekeeper is entirely dedicated to full dynticks CPUs.

Depends on how many there are, and how many housekeepers. I would
suggest one housekeeper cpu per node or so to make sure this keeps
working.

Then again, these load values aren't super important, esp. not for the
nohz_full case where you typically don't care about load balancing.

[toc] | [prev] | [next] | [standalone]

#1314195

On Wed, Jan 20, 2016 at 07:25:03PM +0100, Peter Zijlstra wrote:
> On Wed, Jan 20, 2016 at 06:21:07PM +0100, Frederic Weisbecker wrote:
> 
> > You mean a periodic call to the above from the housekeepers?
> > 
> > I didn't think about doing that because you nacked that approach with
> > scheduler_tick(). This isn't much different.
> 
> This does _one_ thing, namely load accounting. scheduler_tick() does a
> whole bunch of things, some which really do not make much sense to do
> remotely.

Ok.

> 
> > It means the housekeeper is entirely dedicated to full dynticks CPUs.
> 
> Depends on how many there are, and how many housekeepers. I would
> suggest one housekeeper cpu per node or so to make sure this keeps
> working.

That's possible to do.

> 
> Then again, these load values aren't super important, esp. not for the
> nohz_full case where you typically don't care about load balancing.

In this case maybe we can assume that these stats don't need precision at
all on nohz full and when the full nohz frame starts, we define
all cpu_load[i] = cpu_load[0] = weighted_cpuload(cpu) and we keep it that way until
the nohz full frame ends. This way we avoid the remote update.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web