Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1460057 > unrolled thread

Re: clocksource_watchdog causing scheduling of timers every second (was [v13] support "task_isolation" mode)

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) Frederic Weisbecker <fweisbec@gmail.com> - 2016-08-11 00:20 +0200
    Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) Peter Zijlstra <peterz@infradead.org> - 2016-08-11 10:50 +0200
      Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) Frederic Weisbecker <fweisbec@gmail.com> - 2016-08-11 14:10 +0200
      Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-08-12 00:30 +0200
        Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) Christoph Lameter <cl@linux.com> - 2016-08-12 01:10 +0200
          Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-08-12 01:50 +0200
            Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) Christoph Lameter <cl@linux.com> - 2016-08-12 16:30 +0200
              Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) Frederic Weisbecker <fweisbec@gmail.com> - 2016-08-12 16:30 +0200
                Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2016-08-12 18:20 +0200
                  Re: clocksource_watchdog causing scheduling of timers every second  (was [v13] support "task_isolation" mode) Frederic Weisbecker <fweisbec@gmail.com> - 2016-08-13 17:40 +0200

#1460057 — Re: clocksource_watchdog causing scheduling of timers every second (was [v13] support "task_isolation" mode)

On Wed, Jul 27, 2016 at 08:55:28AM -0500, Christoph Lameter wrote:
> On Mon, 25 Jul 2016, Christoph Lameter wrote:
> 
> > Guess so. I will have a look at this when I get some time again.
> 
> Ok so the problem is the clocksource_watchdog() function in
> kernel/time/clocksource.c. This function is active if
> CONFIG_CLOCKSOURCE_WATCHDOG is defined. It will check the timesources of
> each processor for being within bounds and then reschedule itself on the
> next one.
> 
> The purpose of the function seems to be to determine *if* a clocksource is
> unstable. It does not mean that the clocksource *is* unstable.
> 
> The critical piece of code is this:
> 
>         /*
>          * Cycle through CPUs to check if the CPUs stay synchronized
>          * to each other.
>          */
>         next_cpu = cpumask_next(raw_smp_processor_id(), cpu_online_mask);
>         if (next_cpu >= nr_cpu_ids)
>                 next_cpu = cpumask_first(cpu_online_mask);
>         watchdog_timer.expires += WATCHDOG_INTERVAL;
>         add_timer_on(&watchdog_timer, next_cpu);
> 
> 
> Should we just cycle through the cpus that are not isolated? Otherwise we
> need to have some means to check the clocksources for accuracy remotely
> (probably impossible for TSC etc).
> 
> The WATCHDOG_INTERVAL is 1 second so this causes an interrupt every
> second.
> 
> Note that we are running with the patch that removes the 1 HZ mininum time
> tick. With an older kernel code base (redhat) we can keep the kernel quiet
> for minutes. The clocksource watchdog causes timers to fire again.

I had similar issues, this seems to happen when the tsc is considered not reliable
(which doesn't necessarily mean unstable. I think it has to do with some x86 CPU feature
flag).

IIRC, this _has_ to execute on all online CPUs because every TSCs of running CPUs
are concerned.

I personally override that with passing the tsc=reliable kernel parameter. Of course
use it at your own risk.

But eventually I don't think we can offline that to housekeeping only CPUs.

[toc] | [next] | [standalone]

#1460271

On Thu, Aug 11, 2016 at 12:16:58AM +0200, Frederic Weisbecker wrote:
> I had similar issues, this seems to happen when the tsc is considered not reliable
> (which doesn't necessarily mean unstable. I think it has to do with some x86 CPU feature
> flag).

Right, as per the other email, in general we cannot know/assume the TSC
to be working as intended :/

> IIRC, this _has_ to execute on all online CPUs because every TSCs of running CPUs
> are concerned.

With modern Intel we could run it on one CPU per package I think, but at
the same time, too much in NOHZ_FULL assumes the TSC is indeed sane so
it doesn't make sense to me to keep the watchdog running, when it
triggers it would also have to kill all NOHZ_FULL stuff, which would
probably bring the entire machine down..

Arguably we should issue a boot time warning if NOHZ_FULL is configured
and the TSC watchdog is running.

> I personally override that with passing the tsc=reliable kernel
> parameter. Of course use it at your own risk.

Yes, that is (sadly) our only option. Manually assert our hardware is
solid under the intended workload and then manually disabling the
watchdog.

[toc] | [prev] | [next] | [standalone]

#1460451

On Thu, Aug 11, 2016 at 10:40:02AM +0200, Peter Zijlstra wrote:
> On Thu, Aug 11, 2016 at 12:16:58AM +0200, Frederic Weisbecker wrote:
> > I had similar issues, this seems to happen when the tsc is considered not reliable
> > (which doesn't necessarily mean unstable. I think it has to do with some x86 CPU feature
> > flag).
> 
> Right, as per the other email, in general we cannot know/assume the TSC
> to be working as intended :/

Yeah, I remember you explained me that a little while ago.

> 
> > IIRC, this _has_ to execute on all online CPUs because every TSCs of running CPUs
> > are concerned.
> 
> With modern Intel we could run it on one CPU per package I think, but at
> the same time, too much in NOHZ_FULL assumes the TSC is indeed sane so
> it doesn't make sense to me to keep the watchdog running, when it
> triggers it would also have to kill all NOHZ_FULL stuff, which would
> probably bring the entire machine down..
> 
> Arguably we should issue a boot time warning if NOHZ_FULL is configured
> and the TSC watchdog is running.

That's a very good idea! We do that when tsc is unstable but indeed we can't
seriously run NOHZ_FULL on a non-reliable tsc.

I'll take care of that warning.

> 
> > I personally override that with passing the tsc=reliable kernel
> > parameter. Of course use it at your own risk.
> 
> Yes, that is (sadly) our only option. Manually assert our hardware is
> solid under the intended workload and then manually disabling the
> watchdog.

Right, I'll tell about that in the warning.

Thanks for those details!

[toc] | [prev] | [next] | [standalone]

#1460837

On Thu, Aug 11, 2016 at 10:40:02AM +0200, Peter Zijlstra wrote:
> On Thu, Aug 11, 2016 at 12:16:58AM +0200, Frederic Weisbecker wrote:
> > I had similar issues, this seems to happen when the tsc is considered not reliable
> > (which doesn't necessarily mean unstable. I think it has to do with some x86 CPU feature
> > flag).
> 
> Right, as per the other email, in general we cannot know/assume the TSC
> to be working as intended :/
> 
> > IIRC, this _has_ to execute on all online CPUs because every TSCs of running CPUs
> > are concerned.
> 
> With modern Intel we could run it on one CPU per package I think, but at
> the same time, too much in NOHZ_FULL assumes the TSC is indeed sane so
> it doesn't make sense to me to keep the watchdog running, when it
> triggers it would also have to kill all NOHZ_FULL stuff, which would
> probably bring the entire machine down..

Well, you -could- force a very low priority CPU-bound task to run on
all nohz_full CPUs.  Not necessarily a good idea, but a relatively
non-intrusive response to that particular error condition.

							Thanx, Paul

> Arguably we should issue a boot time warning if NOHZ_FULL is configured
> and the TSC watchdog is running.
> 
> > I personally override that with passing the tsc=reliable kernel
> > parameter. Of course use it at your own risk.
> 
> Yes, that is (sadly) our only option. Manually assert our hardware is
> solid under the intended workload and then manually disabling the
> watchdog.
> 

[toc] | [prev] | [next] | [standalone]

#1460853

On Thu, 11 Aug 2016, Paul E. McKenney wrote:

> > With modern Intel we could run it on one CPU per package I think, but at
> > the same time, too much in NOHZ_FULL assumes the TSC is indeed sane so
> > it doesn't make sense to me to keep the watchdog running, when it
> > triggers it would also have to kill all NOHZ_FULL stuff, which would
> > probably bring the entire machine down..
>
> Well, you -could- force a very low priority CPU-bound task to run on
> all nohz_full CPUs.  Not necessarily a good idea, but a relatively
> non-intrusive response to that particular error condition.

Given that we want the cpu only to run the user task I would think that is
not a good idea.

[toc] | [prev] | [next] | [standalone]

#1460869

On Thu, Aug 11, 2016 at 06:02:34PM -0500, Christoph Lameter wrote:
> On Thu, 11 Aug 2016, Paul E. McKenney wrote:
> 
> > > With modern Intel we could run it on one CPU per package I think, but at
> > > the same time, too much in NOHZ_FULL assumes the TSC is indeed sane so
> > > it doesn't make sense to me to keep the watchdog running, when it
> > > triggers it would also have to kill all NOHZ_FULL stuff, which would
> > > probably bring the entire machine down..
> >
> > Well, you -could- force a very low priority CPU-bound task to run on
> > all nohz_full CPUs.  Not necessarily a good idea, but a relatively
> > non-intrusive response to that particular error condition.
> 
> Given that we want the cpu only to run the user task I would think that is
> not a good idea.

Heh!  The only really good idea is for clocks to be reliably in sync.

But if they go out of sync, what do you want to do instead?

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1461204

On Thu, 11 Aug 2016, Paul E. McKenney wrote:

> Heh!  The only really good idea is for clocks to be reliably in sync.
>
> But if they go out of sync, what do you want to do instead?

For a NOHZ task? Write a message to the syslog and reenable tick.

[toc] | [prev] | [next] | [standalone]

#1461205

On Fri, Aug 12, 2016 at 09:23:13AM -0500, Christoph Lameter wrote:
> On Thu, 11 Aug 2016, Paul E. McKenney wrote:
> 
> > Heh!  The only really good idea is for clocks to be reliably in sync.
> >
> > But if they go out of sync, what do you want to do instead?
> 
> For a NOHZ task? Write a message to the syslog and reenable tick.

Indeed, a strong clocksource is a requirement for a full tickless machine.

[toc] | [prev] | [next] | [standalone]

#1461304

On Fri, Aug 12, 2016 at 04:26:13PM +0200, Frederic Weisbecker wrote:
> On Fri, Aug 12, 2016 at 09:23:13AM -0500, Christoph Lameter wrote:
> > On Thu, 11 Aug 2016, Paul E. McKenney wrote:
> > 
> > > Heh!  The only really good idea is for clocks to be reliably in sync.
> > >
> > > But if they go out of sync, what do you want to do instead?
> > 
> > For a NOHZ task? Write a message to the syslog and reenable tick.

Fair enough!  Kicking off a low-priority task would achieve the latter
but not necessarily the former.  And of course assumes that the worker
thread is at real-time priority with various scheduler anti-starvation
features disabled.

> Indeed, a strong clocksource is a requirement for a full tickless machine.

No disagrement here!  ;-)

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1461630

On Fri, Aug 12, 2016 at 09:19:19AM -0700, Paul E. McKenney wrote:
> On Fri, Aug 12, 2016 at 04:26:13PM +0200, Frederic Weisbecker wrote:
> > On Fri, Aug 12, 2016 at 09:23:13AM -0500, Christoph Lameter wrote:
> > > On Thu, 11 Aug 2016, Paul E. McKenney wrote:
> > > 
> > > > Heh!  The only really good idea is for clocks to be reliably in sync.
> > > >
> > > > But if they go out of sync, what do you want to do instead?
> > > 
> > > For a NOHZ task? Write a message to the syslog and reenable tick.
> 
> Fair enough!  Kicking off a low-priority task would achieve the latter
> but not necessarily the former.  And of course assumes that the worker
> thread is at real-time priority with various scheduler anti-starvation
> features disabled.
> 
> > Indeed, a strong clocksource is a requirement for a full tickless machine.
> 
> No disagrement here!  ;-)

I have a bot in my mind that randomly posts obvious statements about nohz_full
here and then :-)

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web