Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1601796 > unrolled thread

Re: [PATCH] clock: Fix smp_processor_id() in preemptible bug

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] clock: Fix smp_processor_id() in preemptible bug "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-03-16 00:00 +0100
    Re: [PATCH] clock: Fix smp_processor_id() in preemptible bug Peter Zijlstra <peterz@infradead.org> - 2017-03-16 17:00 +0100
      Re: [PATCH] clock: Fix smp_processor_id() in preemptible bug "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-03-16 18:00 +0100
        Re: [PATCH] clock: Fix smp_processor_id() in preemptible bug "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-03-16 22:50 +0100
        Re: [PATCH] clock: Fix smp_processor_id() in preemptible bug "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-03-17 00:30 +0100

#1601796 — Re: [PATCH] clock: Fix smp_processor_id() in preemptible bug

On Mon, Mar 13, 2017 at 01:46:21PM +0100, Peter Zijlstra wrote:
> On Thu, Mar 09, 2017 at 10:37:32AM -0800, Paul E. McKenney wrote:
> > And it does pass light testing.  I will hammer it harder this evening.
> > 
> > So please send a formal patch!
> 
> Changed it a bit...
> 
> ---
> Subject: sched/clock: Some clear_sched_clock_stable() vs hotplug wobbles
> 
> Paul reported two independent problems with clear_sched_clock_stable().
> 
>  - if we tickle it during hotplug (even though the sched_clock was
>    already marked unstable) we'll attempt to schedule_work() and
>    this explodes because RCU isn't watching the new CPU quite yet.
> 
>  - since we run all of __clear_sched_clock_stable() from workqueue
>    context, there's a preempt problem.
> 
> Cure both by only doing the static_branch_disable() from a workqueue,
> and only when it's still stable.
> 
> This leaves the problem what to do about hotplug actually wrecking TSC
> though, because if it was stable and now isn't, then we will want to run
> that work, which then will prod RCU the wrong way.  Bloody hotplug.

Would it help to do the same trick tglx applied to the hot-unplug path,
that is IPIing some other CPU to schedule the workqueue?

							Thanx, Paul

> Reported-by: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
>  kernel/sched/clock.c | 17 ++++++++++++-----
>  1 file changed, 12 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/sched/clock.c b/kernel/sched/clock.c
> index a08795e21628..fec0f58c8dee 100644
> --- a/kernel/sched/clock.c
> +++ b/kernel/sched/clock.c
> @@ -141,7 +141,14 @@ static void __set_sched_clock_stable(void)
>  	tick_dep_clear(TICK_DEP_BIT_CLOCK_UNSTABLE);
>  }
> 
> -static void __clear_sched_clock_stable(struct work_struct *work)
> +static void __sched_clock_work(struct work_struct *work)
> +{
> +	static_branch_disable(&__sched_clock_stable);
> +}
> +
> +static DECLARE_WORK(sched_clock_work, __sched_clock_work);
> +
> +static void __clear_sched_clock_stable(void)
>  {
>  	struct sched_clock_data *scd = this_scd();
> 
> @@ -160,11 +167,11 @@ static void __clear_sched_clock_stable(struct work_struct *work)
>  			scd->tick_gtod, gtod_offset,
>  			scd->tick_raw,  raw_offset);
> 
> -	static_branch_disable(&__sched_clock_stable);
>  	tick_dep_set(TICK_DEP_BIT_CLOCK_UNSTABLE);
> -}
> 
> -static DECLARE_WORK(sched_clock_work, __clear_sched_clock_stable);
> +	if (sched_clock_stable())
> +		schedule_work(&sched_clock_work);
> +}
> 
>  void clear_sched_clock_stable(void)
>  {
> @@ -173,7 +180,7 @@ void clear_sched_clock_stable(void)
>  	smp_mb(); /* matches sched_clock_init_late() */
> 
>  	if (sched_clock_running == 2)
> -		schedule_work(&sched_clock_work);
> +		__clear_sched_clock_stable();
>  }
> 
>  void sched_clock_init_late(void)
> 

[toc] | [next] | [standalone]

#1602593

On Wed, Mar 15, 2017 at 03:58:17PM -0700, Paul E. McKenney wrote:
> On Mon, Mar 13, 2017 at 01:46:21PM +0100, Peter Zijlstra wrote:
> > On Thu, Mar 09, 2017 at 10:37:32AM -0800, Paul E. McKenney wrote:
> > > And it does pass light testing.  I will hammer it harder this evening.
> > > 
> > > So please send a formal patch!
> > 
> > Changed it a bit...
> > 
> > ---
> > Subject: sched/clock: Some clear_sched_clock_stable() vs hotplug wobbles
> > 
> > Paul reported two independent problems with clear_sched_clock_stable().
> > 
> >  - if we tickle it during hotplug (even though the sched_clock was
> >    already marked unstable) we'll attempt to schedule_work() and
> >    this explodes because RCU isn't watching the new CPU quite yet.
> > 
> >  - since we run all of __clear_sched_clock_stable() from workqueue
> >    context, there's a preempt problem.
> > 
> > Cure both by only doing the static_branch_disable() from a workqueue,
> > and only when it's still stable.
> > 
> > This leaves the problem what to do about hotplug actually wrecking TSC
> > though, because if it was stable and now isn't, then we will want to run
> > that work, which then will prod RCU the wrong way.  Bloody hotplug.
> 
> Would it help to do the same trick tglx applied to the hot-unplug path,
> that is IPIing some other CPU to schedule the workqueue?
> 

So I've been looking again; and I don't think its a problem anymore.

The problem you reported here:

 https://lkml.kernel.org/r/20170308221656.GA11949@linux.vnet.ibm.com

Should not happen after commit:

  f94c8d116997 ("sched/clock, x86/tsc: Rework the x86 'unstable' sched_clock() interface")

which landed around 4.11-rc2; so _after_ your kernel (which reported
itself as -rc1).

Because since that commit we'll never call clear_sched_clock_stable() if
tsc_unstable is set.

So I'll have to amend the Changelog somewhat.

[toc] | [prev] | [next] | [standalone]

#1602661

On Thu, Mar 16, 2017 at 04:53:10PM +0100, Peter Zijlstra wrote:
> On Wed, Mar 15, 2017 at 03:58:17PM -0700, Paul E. McKenney wrote:
> > On Mon, Mar 13, 2017 at 01:46:21PM +0100, Peter Zijlstra wrote:
> > > On Thu, Mar 09, 2017 at 10:37:32AM -0800, Paul E. McKenney wrote:
> > > > And it does pass light testing.  I will hammer it harder this evening.
> > > > 
> > > > So please send a formal patch!
> > > 
> > > Changed it a bit...
> > > 
> > > ---
> > > Subject: sched/clock: Some clear_sched_clock_stable() vs hotplug wobbles
> > > 
> > > Paul reported two independent problems with clear_sched_clock_stable().
> > > 
> > >  - if we tickle it during hotplug (even though the sched_clock was
> > >    already marked unstable) we'll attempt to schedule_work() and
> > >    this explodes because RCU isn't watching the new CPU quite yet.
> > > 
> > >  - since we run all of __clear_sched_clock_stable() from workqueue
> > >    context, there's a preempt problem.
> > > 
> > > Cure both by only doing the static_branch_disable() from a workqueue,
> > > and only when it's still stable.
> > > 
> > > This leaves the problem what to do about hotplug actually wrecking TSC
> > > though, because if it was stable and now isn't, then we will want to run
> > > that work, which then will prod RCU the wrong way.  Bloody hotplug.
> > 
> > Would it help to do the same trick tglx applied to the hot-unplug path,
> > that is IPIing some other CPU to schedule the workqueue?
> 
> So I've been looking again; and I don't think its a problem anymore.
> 
> The problem you reported here:
> 
>  https://lkml.kernel.org/r/20170308221656.GA11949@linux.vnet.ibm.com
> 
> Should not happen after commit:
> 
>   f94c8d116997 ("sched/clock, x86/tsc: Rework the x86 'unstable' sched_clock() interface")
> 
> which landed around 4.11-rc2; so _after_ your kernel (which reported
> itself as -rc1).

Very good!  I am re-runnning rcutorture on -rc2 (without your patch)
and will let you know how it goes.

> Because since that commit we'll never call clear_sched_clock_stable() if
> tsc_unstable is set.
> 
> So I'll have to amend the Changelog somewhat.

Sounds good!

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1602858

On Thu, Mar 16, 2017 at 02:00:02PM -0700, Paul E. McKenney wrote:
> On Thu, Mar 16, 2017 at 09:57:48AM -0700, Paul E. McKenney wrote:
> > On Thu, Mar 16, 2017 at 04:53:10PM +0100, Peter Zijlstra wrote:
> > > On Wed, Mar 15, 2017 at 03:58:17PM -0700, Paul E. McKenney wrote:
> > > > On Mon, Mar 13, 2017 at 01:46:21PM +0100, Peter Zijlstra wrote:
> > > > > On Thu, Mar 09, 2017 at 10:37:32AM -0800, Paul E. McKenney wrote:
> > > > > > And it does pass light testing.  I will hammer it harder this evening.
> > > > > > 
> > > > > > So please send a formal patch!
> > > > > 
> > > > > Changed it a bit...
> > > > > 
> > > > > ---
> > > > > Subject: sched/clock: Some clear_sched_clock_stable() vs hotplug wobbles
> > > > > 
> > > > > Paul reported two independent problems with clear_sched_clock_stable().
> > > > > 
> > > > >  - if we tickle it during hotplug (even though the sched_clock was
> > > > >    already marked unstable) we'll attempt to schedule_work() and
> > > > >    this explodes because RCU isn't watching the new CPU quite yet.
> > > > > 
> > > > >  - since we run all of __clear_sched_clock_stable() from workqueue
> > > > >    context, there's a preempt problem.
> > > > > 
> > > > > Cure both by only doing the static_branch_disable() from a workqueue,
> > > > > and only when it's still stable.
> > > > > 
> > > > > This leaves the problem what to do about hotplug actually wrecking TSC
> > > > > though, because if it was stable and now isn't, then we will want to run
> > > > > that work, which then will prod RCU the wrong way.  Bloody hotplug.
> > > > 
> > > > Would it help to do the same trick tglx applied to the hot-unplug path,
> > > > that is IPIing some other CPU to schedule the workqueue?
> > > 
> > > So I've been looking again; and I don't think its a problem anymore.
> > > 
> > > The problem you reported here:
> > > 
> > >  https://lkml.kernel.org/r/20170308221656.GA11949@linux.vnet.ibm.com
> > > 
> > > Should not happen after commit:
> > > 
> > >   f94c8d116997 ("sched/clock, x86/tsc: Rework the x86 'unstable' sched_clock() interface")
> > > 
> > > which landed around 4.11-rc2; so _after_ your kernel (which reported
> > > itself as -rc1).
> > 
> > Very good!  I am re-runnning rcutorture on -rc2 (without your patch)
> > and will let you know how it goes.
> 
> And -rc2 works just fine on rcutorture, thank you again!
> 
> 							Thanx, Paul
> 
> > > Because since that commit we'll never call clear_sched_clock_stable() if
> > > tsc_unstable is set.
> > > 
> > > So I'll have to amend the Changelog somewhat.
> > 
> > Sounds good!

Ah, and I dropped this patch when rebasing to v4.11-rc2 in anticipation
of your Changelog amendments:

	sched/clock: Some clear_sched_clock_stable() vs hotplug wobbles

Please let me know if you would instead prefer me to keep carrying the
current version.

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1602917

On Thu, Mar 16, 2017 at 09:57:48AM -0700, Paul E. McKenney wrote:
> On Thu, Mar 16, 2017 at 04:53:10PM +0100, Peter Zijlstra wrote:
> > On Wed, Mar 15, 2017 at 03:58:17PM -0700, Paul E. McKenney wrote:
> > > On Mon, Mar 13, 2017 at 01:46:21PM +0100, Peter Zijlstra wrote:
> > > > On Thu, Mar 09, 2017 at 10:37:32AM -0800, Paul E. McKenney wrote:
> > > > > And it does pass light testing.  I will hammer it harder this evening.
> > > > > 
> > > > > So please send a formal patch!
> > > > 
> > > > Changed it a bit...
> > > > 
> > > > ---
> > > > Subject: sched/clock: Some clear_sched_clock_stable() vs hotplug wobbles
> > > > 
> > > > Paul reported two independent problems with clear_sched_clock_stable().
> > > > 
> > > >  - if we tickle it during hotplug (even though the sched_clock was
> > > >    already marked unstable) we'll attempt to schedule_work() and
> > > >    this explodes because RCU isn't watching the new CPU quite yet.
> > > > 
> > > >  - since we run all of __clear_sched_clock_stable() from workqueue
> > > >    context, there's a preempt problem.
> > > > 
> > > > Cure both by only doing the static_branch_disable() from a workqueue,
> > > > and only when it's still stable.
> > > > 
> > > > This leaves the problem what to do about hotplug actually wrecking TSC
> > > > though, because if it was stable and now isn't, then we will want to run
> > > > that work, which then will prod RCU the wrong way.  Bloody hotplug.
> > > 
> > > Would it help to do the same trick tglx applied to the hot-unplug path,
> > > that is IPIing some other CPU to schedule the workqueue?
> > 
> > So I've been looking again; and I don't think its a problem anymore.
> > 
> > The problem you reported here:
> > 
> >  https://lkml.kernel.org/r/20170308221656.GA11949@linux.vnet.ibm.com
> > 
> > Should not happen after commit:
> > 
> >   f94c8d116997 ("sched/clock, x86/tsc: Rework the x86 'unstable' sched_clock() interface")
> > 
> > which landed around 4.11-rc2; so _after_ your kernel (which reported
> > itself as -rc1).
> 
> Very good!  I am re-runnning rcutorture on -rc2 (without your patch)
> and will let you know how it goes.

And -rc2 works just fine on rcutorture, thank you again!

							Thanx, Paul

> > Because since that commit we'll never call clear_sched_clock_stable() if
> > tsc_unstable is set.
> > 
> > So I'll have to amend the Changelog somewhat.
> 
> Sounds good!
> 
> 							Thanx, Paul

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web