Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1497484 > unrolled thread

[v4.8-rc1 Regression] sched/fair: Apply more PELT fixes

Started byJoseph Salisbury <joseph.salisbury@canonical.com>
First post2016-10-07 21:40 +0200
Last post2016-10-12 18:30 +0200
Articles 11 — 6 participants

Back to article view | Back to linux.kernel

Contents

  [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Joseph Salisbury <joseph.salisbury@canonical.com> - 2016-10-07 21:40 +0200
    Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Linus Torvalds <torvalds@linux-foundation.org> - 2016-10-07 22:00 +0200
      Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Joseph Salisbury <joseph.salisbury@canonical.com> - 2016-10-07 22:30 +0200
        Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Linus Torvalds <torvalds@linux-foundation.org> - 2016-10-07 22:40 +0200
    Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Peter Zijlstra <peterz@infradead.org> - 2016-10-08 10:10 +0200
      Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Ingo Molnar <mingo@kernel.org> - 2016-10-08 10:50 +0200
        Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-08 13:40 +0200
          Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Mike Galbraith <efault@gmx.de> - 2016-10-08 14:00 +0200
            Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Vincent Guittot <vincent.guittot@linaro.org> - 2016-10-12 14:30 +0200
              Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Joseph Salisbury <joseph.salisbury@canonical.com> - 2016-10-12 17:40 +0200
              Re: [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes Joseph Salisbury <joseph.salisbury@canonical.com> - 2016-10-12 18:30 +0200

#1497484 — [v4.8-rc1 Regression] sched/fair: Apply more PELT fixes

FromJoseph Salisbury <joseph.salisbury@canonical.com>
Date2016-10-07 21:40 +0200
Subject[v4.8-rc1 Regression] sched/fair: Apply more PELT fixes
Message-ID<spJ9T-4Zd-11@gated-at.bofh.it>
Hello Peter,

A kernel bug report was opened against Ubuntu [0].  After a kernel
bisect, it was found that reverting the following commit resolved this bug:

commit 3d30544f02120b884bba2a9466c87dba980e3be5
Author: Peter Zijlstra <peterz@infradead.org>
Date:   Tue Jun 21 14:27:50 2016 +0200

    sched/fair: Apply more PELT fixes

The regression was introduced as of v4.8-rc1.  The bug can be reproduced
on an X1 Carbon with the following:
stress -c $your_total_cpu_cores

I was hoping to get your feedback, since you are the patch author.  Do
you think gathering any additional data will help diagnose this issue,
or would it be best to submit a revert request? 


Thanks,

Joe


[0] http://pad.lv/1627108

[toc] | [next] | [standalone]

#1497490

FromLinus Torvalds <torvalds@linux-foundation.org>
Date2016-10-07 22:00 +0200
Message-ID<spJtf-56Q-9@gated-at.bofh.it>
In reply to#1497484
On Fri, Oct 7, 2016 at 12:38 PM, Joseph Salisbury
<joseph.salisbury@canonical.com> wrote:
>
> A kernel bug report was opened against Ubuntu [0].  After a kernel
> bisect, it was found that reverting the following commit resolved this bug:

Hmm. Interesting, and it sounds like we should revert that unless
somebody figures out *why* following the rules wrt cfq updates causes
problems. But I also wonder what the Ubuntu kernel config is. Does
Ubuntu enable CONFIG_SCHED_AUTOGROUP=y, for example? Because
regardless of any other scheduler issues, autogrouping *should* mean
that when you run some CPU hogger in one session, that should still
balance all CPU time with other sessions..

I'm not seeing anything odd on my xps13, which should have a similar
CPU to the X1 Carbon.

               Linus

[toc] | [prev] | [next] | [standalone]

#1497500

FromJoseph Salisbury <joseph.salisbury@canonical.com>
Date2016-10-07 22:30 +0200
Message-ID<spJWi-5BJ-11@gated-at.bofh.it>
In reply to#1497490
On 10/07/2016 03:57 PM, Linus Torvalds wrote:
> On Fri, Oct 7, 2016 at 12:38 PM, Joseph Salisbury
> <joseph.salisbury@canonical.com> wrote:
>> A kernel bug report was opened against Ubuntu [0].  After a kernel
>> bisect, it was found that reverting the following commit resolved this bug:
> Hmm. Interesting, and it sounds like we should revert that unless
> somebody figures out *why* following the rules wrt cfq updates causes
> problems. But I also wonder what the Ubuntu kernel config is. Does
> Ubuntu enable CONFIG_SCHED_AUTOGROUP=y, for example? Because
> regardless of any other scheduler issues, autogrouping *should* mean
> that when you run some CPU hogger in one session, that should still
> balance all CPU time with other sessions..
>
> I'm not seeing anything odd on my xps13, which should have a similar
> CPU to the X1 Carbon.
>
>                Linus
Hi Linus,

Yes, CONFIG_SCHED_AUTOGROUP is enabled in the Ubuntu kernel.  However,
that config was also enable in the Ubuntu 4.4 kerrnels without seeing
this issue.   I can try disabling the config in the 4.8 based kernel and
see if that changes things.

Thanks,

Joe

[toc] | [prev] | [next] | [standalone]

#1497505

FromLinus Torvalds <torvalds@linux-foundation.org>
Date2016-10-07 22:40 +0200
Message-ID<spK5X-5Ff-3@gated-at.bofh.it>
In reply to#1497500
On Fri, Oct 7, 2016 at 1:22 PM, Joseph Salisbury
<joseph.salisbury@canonical.com> wrote:
>
> Yes, CONFIG_SCHED_AUTOGROUP is enabled in the Ubuntu kernel.  However,
> that config was also enable in the Ubuntu 4.4 kerrnels without seeing
> this issue.   I can try disabling the config in the 4.8 based kernel and
> see if that changes things.

No, that wouldn't make any sense. I just wanted to know that the
option was enabled, because that option really *should* help buffer
other processes from one session that is a CPU hog.

So something is seriously wrong in that situation if other things get
very choppy. Of course, the fact that it apparently happens on one
particular machine only means that it's hard to figure out what
triggers it. Maybe some unlucky combination of cpufreq and thermal
throttling by the hardware, coupled with the scheduler change.

Peter?

                 Linus

[toc] | [prev] | [next] | [standalone]

#1497686

FromPeter Zijlstra <peterz@infradead.org>
Date2016-10-08 10:10 +0200
Message-ID<spURH-4wI-1@gated-at.bofh.it>
In reply to#1497484
On Fri, Oct 07, 2016 at 03:38:23PM -0400, Joseph Salisbury wrote:
> Hello Peter,
> 
> A kernel bug report was opened against Ubuntu [0].  After a kernel
> bisect, it was found that reverting the following commit resolved this bug:
> 
> commit 3d30544f02120b884bba2a9466c87dba980e3be5
> Author: Peter Zijlstra <peterz@infradead.org>
> Date:   Tue Jun 21 14:27:50 2016 +0200
> 
>     sched/fair: Apply more PELT fixes

That commit doesn't revert cleanly, did you take out more?

> The regression was introduced as of v4.8-rc1.  The bug can be reproduced
> on an X1 Carbon with the following:
> stress -c $your_total_cpu_cores
> 
> I was hoping to get your feedback, since you are the patch author.  Do
> you think gathering any additional data will help diagnose this issue,
> or would it be best to submit a revert request? 

I'll try and have a look asap, but I'm traveling next week so it might a
tad slower than normal.

If you could provide a /proc/sched_debug dump while the thing is running
that'd might be useful.

Thanks!

[toc] | [prev] | [next] | [standalone]

#1497690

FromIngo Molnar <mingo@kernel.org>
Date2016-10-08 10:50 +0200
Message-ID<spVup-4Mr-1@gated-at.bofh.it>
In reply to#1497686
* Peter Zijlstra <peterz@infradead.org> wrote:

> On Fri, Oct 07, 2016 at 03:38:23PM -0400, Joseph Salisbury wrote:
> > Hello Peter,
> > 
> > A kernel bug report was opened against Ubuntu [0].  After a kernel
> > bisect, it was found that reverting the following commit resolved this bug:
> > 
> > commit 3d30544f02120b884bba2a9466c87dba980e3be5
> > Author: Peter Zijlstra <peterz@infradead.org>
> > Date:   Tue Jun 21 14:27:50 2016 +0200
> > 
> >     sched/fair: Apply more PELT fixes
> 
> That commit doesn't revert cleanly, did you take out more?

Note that it reverts cleanly from v4.8 - while it does to revert from current 
upstream that did more changes in that area.

I suspect Josheph tested a v4.8-ish kernel.

> I'll try and have a look asap, but I'm traveling next week so it might a
> tad slower than normal.
> 
> If you could provide a /proc/sched_debug dump while the thing is running
> that'd might be useful.

Also, running the latest scheduler bits would be useful:

  git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git

... just on the off chance that another change in this area fixed the bug, plus to 
make it easier to send test patches.

Upstream merge commit af79ad2b1f33 should also be pretty safe to try - it's v4.8 
with the v4.9 scheduler bits applied.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1497712

FromVincent Guittot <vincent.guittot@linaro.org>
Date2016-10-08 13:40 +0200
Message-ID<spY8V-6tk-5@gated-at.bofh.it>
In reply to#1497690
On 8 October 2016 at 10:39, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Peter Zijlstra <peterz@infradead.org> wrote:
>
>> On Fri, Oct 07, 2016 at 03:38:23PM -0400, Joseph Salisbury wrote:
>> > Hello Peter,
>> >
>> > A kernel bug report was opened against Ubuntu [0].  After a kernel
>> > bisect, it was found that reverting the following commit resolved this bug:
>> >
>> > commit 3d30544f02120b884bba2a9466c87dba980e3be5
>> > Author: Peter Zijlstra <peterz@infradead.org>
>> > Date:   Tue Jun 21 14:27:50 2016 +0200
>> >
>> >     sched/fair: Apply more PELT fixes

This patch only speeds up the update of task group load in order to
reflect the new load balance but It should not change the final value
and as a result the final behavior. I will try to reproduce it in my
target later today

>>
>> That commit doesn't revert cleanly, did you take out more?
>
> Note that it reverts cleanly from v4.8 - while it does to revert from current
> upstream that did more changes in that area.
>
> I suspect Josheph tested a v4.8-ish kernel.
>
>> I'll try and have a look asap, but I'm traveling next week so it might a
>> tad slower than normal.
>>
>> If you could provide a /proc/sched_debug dump while the thing is running
>> that'd might be useful.
>
> Also, running the latest scheduler bits would be useful:
>
>   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
>
> ... just on the off chance that another change in this area fixed the bug, plus to
> make it easier to send test patches.
>
> Upstream merge commit af79ad2b1f33 should also be pretty safe to try - it's v4.8
> with the v4.9 scheduler bits applied.
>
> Thanks,
>
>         Ingo

[toc] | [prev] | [next] | [standalone]

#1497717

FromMike Galbraith <efault@gmx.de>
Date2016-10-08 14:00 +0200
Message-ID<spYsi-6A7-21@gated-at.bofh.it>
In reply to#1497712
On Sat, 2016-10-08 at 13:37 +0200, Vincent Guittot wrote:
> On 8 October 2016 at 10:39, Ingo Molnar <mingo@kernel.org> wrote:
> > 
> > * Peter Zijlstra <peterz@infradead.org> wrote:
> > 
> > > On Fri, Oct 07, 2016 at 03:38:23PM -0400, Joseph Salisbury wrote:
> > > > Hello Peter,
> > > > 
> > > > A kernel bug report was opened against Ubuntu [0].  After a
> > > > kernel
> > > > bisect, it was found that reverting the following commit
> > > > resolved this bug:
> > > > 
> > > > commit 3d30544f02120b884bba2a9466c87dba980e3be5
> > > > Author: Peter Zijlstra <peterz@infradead.org>
> > > > Date:   Tue Jun 21 14:27:50 2016 +0200
> > > > 
> > > >     sched/fair: Apply more PELT fixes
> 
> This patch only speeds up the update of task group load in order to
> reflect the new load balance but It should not change the final value
> and as a result the final behavior. I will try to reproduce it in my
> target later today

FWIW, I tried and failed w/wo autogroup on 4.8 and master.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1499515

FromVincent Guittot <vincent.guittot@linaro.org>
Date2016-10-12 14:30 +0200
Message-ID<srqPw-4jZ-17@gated-at.bofh.it>
In reply to#1497717
On 8 October 2016 at 13:49, Mike Galbraith <efault@gmx.de> wrote:
> On Sat, 2016-10-08 at 13:37 +0200, Vincent Guittot wrote:
>> On 8 October 2016 at 10:39, Ingo Molnar <mingo@kernel.org> wrote:
>> >
>> > * Peter Zijlstra <peterz@infradead.org> wrote:
>> >
>> > > On Fri, Oct 07, 2016 at 03:38:23PM -0400, Joseph Salisbury wrote:
>> > > > Hello Peter,
>> > > >
>> > > > A kernel bug report was opened against Ubuntu [0].  After a
>> > > > kernel
>> > > > bisect, it was found that reverting the following commit
>> > > > resolved this bug:
>> > > >
>> > > > commit 3d30544f02120b884bba2a9466c87dba980e3be5
>> > > > Author: Peter Zijlstra <peterz@infradead.org>
>> > > > Date:   Tue Jun 21 14:27:50 2016 +0200
>> > > >
>> > > >     sched/fair: Apply more PELT fixes
>>
>> This patch only speeds up the update of task group load in order to
>> reflect the new load balance but It should not change the final value
>> and as a result the final behavior. I will try to reproduce it in my
>> target later today
>
> FWIW, I tried and failed w/wo autogroup on 4.8 and master.

Me too

Is it possible to get some dump of  /proc/sched_debug while the problem occurs ?

Vincent

>         -Mike

[toc] | [prev] | [next] | [standalone]

#1499777

FromJoseph Salisbury <joseph.salisbury@canonical.com>
Date2016-10-12 17:40 +0200
Message-ID<srtNo-6jC-11@gated-at.bofh.it>
In reply to#1499515
On 10/12/2016 08:20 AM, Vincent Guittot wrote:
> On 8 October 2016 at 13:49, Mike Galbraith <efault@gmx.de> wrote:
>> On Sat, 2016-10-08 at 13:37 +0200, Vincent Guittot wrote:
>>> On 8 October 2016 at 10:39, Ingo Molnar <mingo@kernel.org> wrote:
>>>> * Peter Zijlstra <peterz@infradead.org> wrote:
>>>>
>>>>> On Fri, Oct 07, 2016 at 03:38:23PM -0400, Joseph Salisbury wrote:
>>>>>> Hello Peter,
>>>>>>
>>>>>> A kernel bug report was opened against Ubuntu [0].  After a
>>>>>> kernel
>>>>>> bisect, it was found that reverting the following commit
>>>>>> resolved this bug:
>>>>>>
>>>>>> commit 3d30544f02120b884bba2a9466c87dba980e3be5
>>>>>> Author: Peter Zijlstra <peterz@infradead.org>
>>>>>> Date:   Tue Jun 21 14:27:50 2016 +0200
>>>>>>
>>>>>>     sched/fair: Apply more PELT fixes
>>> This patch only speeds up the update of task group load in order to
>>> reflect the new load balance but It should not change the final value
>>> and as a result the final behavior. I will try to reproduce it in my
>>> target later today
>> FWIW, I tried and failed w/wo autogroup on 4.8 and master.
> Me too
>
> Is it possible to get some dump of  /proc/sched_debug while the problem occurs ?
Yes, I requested that data from the bug reporter.  I also built a test
kernel and requested testing of the repo Ingo pointed me at.  We should
have an update shortly.

Joe


>
> Vincent
>
>>         -Mike

[toc] | [prev] | [next] | [standalone]

#1499809

FromJoseph Salisbury <joseph.salisbury@canonical.com>
Date2016-10-12 18:30 +0200
Message-ID<sruzM-6Ti-7@gated-at.bofh.it>
In reply to#1499515
On 10/12/2016 08:20 AM, Vincent Guittot wrote:
> On 8 October 2016 at 13:49, Mike Galbraith <efault@gmx.de> wrote:
>> On Sat, 2016-10-08 at 13:37 +0200, Vincent Guittot wrote:
>>> On 8 October 2016 at 10:39, Ingo Molnar <mingo@kernel.org> wrote:
>>>> * Peter Zijlstra <peterz@infradead.org> wrote:
>>>>
>>>>> On Fri, Oct 07, 2016 at 03:38:23PM -0400, Joseph Salisbury wrote:
>>>>>> Hello Peter,
>>>>>>
>>>>>> A kernel bug report was opened against Ubuntu [0].  After a
>>>>>> kernel
>>>>>> bisect, it was found that reverting the following commit
>>>>>> resolved this bug:
>>>>>>
>>>>>> commit 3d30544f02120b884bba2a9466c87dba980e3be5
>>>>>> Author: Peter Zijlstra <peterz@infradead.org>
>>>>>> Date:   Tue Jun 21 14:27:50 2016 +0200
>>>>>>
>>>>>>     sched/fair: Apply more PELT fixes
>>> This patch only speeds up the update of task group load in order to
>>> reflect the new load balance but It should not change the final value
>>> and as a result the final behavior. I will try to reproduce it in my
>>> target later today
>> FWIW, I tried and failed w/wo autogroup on 4.8 and master.
> Me too
>
> Is it possible to get some dump of  /proc/sched_debug while the problem occurs ?
>
> Vincent
>
>>         -Mike

The output from /proc/shed_debug can be seen here:
http://paste.ubuntu.com/23312351/

Ingo, the latest scheduler bits also still exhibit the bug:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web