Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1243979 > unrolled thread
| Started by | Meelis Roos <mroos@linux.ee> |
|---|---|
| First post | 2015-10-10 21:20 +0200 |
| Last post | 2015-10-12 03:00 +0200 |
| Articles | 4 — 3 participants |
Back to article view | Back to linux.kernel
4.2: CONFIG_NO_HZ_FULL_ALL effectively disabling non-boot CPUs Meelis Roos <mroos@linux.ee> - 2015-10-10 21:20 +0200
Re: 4.2: CONFIG_NO_HZ_FULL_ALL effectively disabling non-boot CPUs "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2015-10-10 21:30 +0200
Re: 4.2: CONFIG_NO_HZ_FULL_ALL effectively disabling non-boot CPUs Meelis Roos <mroos@linux.ee> - 2015-10-11 09:30 +0200
Re: 4.2: CONFIG_NO_HZ_FULL_ALL effectively disabling non-boot CPUs Frederic Weisbecker <fweisbec@gmail.com> - 2015-10-12 03:00 +0200
| From | Meelis Roos <mroos@linux.ee> |
|---|---|
| Date | 2015-10-10 21:20 +0200 |
| Subject | 4.2: CONFIG_NO_HZ_FULL_ALL effectively disabling non-boot CPUs |
| Message-ID | <qi7Qu-1Ip-11@gated-at.bofh.it> |
Short summary: turning on CONFIG_NO_HZ_FULL_ALL seems to disable all
non-boot CPUs for scheduler.
A couple of days ago I noticed that make -j8 on a 4-core i5 is very slow
(with 4.3.0-rc4+git). Looking at top ('1' for per-cpu states), only
first CPU is loaded and 3 other CPUs are 100% idle. This seems to be a
problem on 3 of my desktop machines (different generation Intel: i5-660,
i5-2400, i3-3220). All the computers run custom kernels.
Further investigation showed that CPU affinity was set to 1 (CPU0 only)
for init and all the children. Kernel threads had affinities 1,2,4,8
and f (seems normal).
Even more interesting was the behaviour after setting affinity to f for
all userland processes and then running make -j4. The other cores were
still idle!
Switching back to 4.2.0 with my config, the problem persisted. 4.2.3 as
packaged by Debian worked fine. 4.0.0 and 4.1.0 with my config worked
also fine. systemd and sysvinit behaved the same and no affinity was
configured for systemd.
So did a kernel config bisection between my kernel config and Debian
config and came to CONFIG_NO_HZ_FULL_ALL. Debian has it off, I had it
on. Turning that off fixed the scheduling and the system spread the
tasks to all the cores.
I do not remember changing this value for a long time, I set them after
the settings were introduced and used it. So it seems it broken in 4.2.0
but was working in 4.1 but I do not have 4.1 config saved anywhere
(many make oldconfigs since).
Bisection between 4.1 and 4.2 is possible but not easy since the
machines are usually actively used when I am near them.
--
Meelis Roos (mroos@linux.ee)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> |
|---|---|
| Date | 2015-10-10 21:30 +0200 |
| Message-ID | <qi80a-1Ti-3@gated-at.bofh.it> |
| In reply to | #1243979 |
On Sat, Oct 10, 2015 at 10:14:25PM +0300, Meelis Roos wrote:
> Short summary: turning on CONFIG_NO_HZ_FULL_ALL seems to disable all
> non-boot CPUs for scheduler.
>
> A couple of days ago I noticed that make -j8 on a 4-core i5 is very slow
> (with 4.3.0-rc4+git). Looking at top ('1' for per-cpu states), only
> first CPU is loaded and 3 other CPUs are 100% idle. This seems to be a
> problem on 3 of my desktop machines (different generation Intel: i5-660,
> i5-2400, i3-3220). All the computers run custom kernels.
>
> Further investigation showed that CPU affinity was set to 1 (CPU0 only)
> for init and all the children. Kernel threads had affinities 1,2,4,8
> and f (seems normal).
>
> Even more interesting was the behaviour after setting affinity to f for
> all userland processes and then running make -j4. The other cores were
> still idle!
>
> Switching back to 4.2.0 with my config, the problem persisted. 4.2.3 as
> packaged by Debian worked fine. 4.0.0 and 4.1.0 with my config worked
> also fine. systemd and sysvinit behaved the same and no affinity was
> configured for systemd.
>
> So did a kernel config bisection between my kernel config and Debian
> config and came to CONFIG_NO_HZ_FULL_ALL. Debian has it off, I had it
> on. Turning that off fixed the scheduling and the system spread the
> tasks to all the cores.
>
> I do not remember changing this value for a long time, I set them after
> the settings were introduced and used it. So it seems it broken in 4.2.0
> but was working in 4.1 but I do not have 4.1 config saved anywhere
> (many make oldconfigs since).
>
> Bisection between 4.1 and 4.2 is possible but not easy since the
> machines are usually actively used when I am near them.
This is expected and intended behavior. The whole point of
CONFIG_NO_HZ_FULL_ALL is to keep everything off of the non-boot CPUs
that is not explicitly placed there. Without CONFIG_NO_HZ_FULL_ALL,
you can use the nohz_full boot parameter to select exactly which
CPUs are to behave this way.
Thanx, Paul
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Meelis Roos <mroos@linux.ee> |
|---|---|
| Date | 2015-10-11 09:30 +0200 |
| Message-ID | <qijeV-1qh-1@gated-at.bofh.it> |
| In reply to | #1243982 |
> > Short summary: turning on CONFIG_NO_HZ_FULL_ALL seems to disable all > > non-boot CPUs for scheduler. [...] > This is expected and intended behavior. The whole point of > CONFIG_NO_HZ_FULL_ALL is to keep everything off of the non-boot CPUs > that is not explicitly placed there. Without CONFIG_NO_HZ_FULL_ALL, > you can use the nohz_full boot parameter to select exactly which > CPUs are to behave this way. OK, thanks - disabled it on my desktops anyway now. Perhaps the Kconfig description of it should explian the consequences more clearly so other people do not stumble upon it like I did? -- Meelis Roos (mroos@linux.ee) -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Frederic Weisbecker <fweisbec@gmail.com> |
|---|---|
| Date | 2015-10-12 03:00 +0200 |
| Message-ID | <qizD4-8hR-7@gated-at.bofh.it> |
| In reply to | #1243982 |
On Sat, Oct 10, 2015 at 12:24:39PM -0700, Paul E. McKenney wrote:
> On Sat, Oct 10, 2015 at 10:14:25PM +0300, Meelis Roos wrote:
> > Short summary: turning on CONFIG_NO_HZ_FULL_ALL seems to disable all
> > non-boot CPUs for scheduler.
> >
> > A couple of days ago I noticed that make -j8 on a 4-core i5 is very slow
> > (with 4.3.0-rc4+git). Looking at top ('1' for per-cpu states), only
> > first CPU is loaded and 3 other CPUs are 100% idle. This seems to be a
> > problem on 3 of my desktop machines (different generation Intel: i5-660,
> > i5-2400, i3-3220). All the computers run custom kernels.
> >
> > Further investigation showed that CPU affinity was set to 1 (CPU0 only)
> > for init and all the children. Kernel threads had affinities 1,2,4,8
> > and f (seems normal).
> >
> > Even more interesting was the behaviour after setting affinity to f for
> > all userland processes and then running make -j4. The other cores were
> > still idle!
> >
> > Switching back to 4.2.0 with my config, the problem persisted. 4.2.3 as
> > packaged by Debian worked fine. 4.0.0 and 4.1.0 with my config worked
> > also fine. systemd and sysvinit behaved the same and no affinity was
> > configured for systemd.
> >
> > So did a kernel config bisection between my kernel config and Debian
> > config and came to CONFIG_NO_HZ_FULL_ALL. Debian has it off, I had it
> > on. Turning that off fixed the scheduling and the system spread the
> > tasks to all the cores.
> >
> > I do not remember changing this value for a long time, I set them after
> > the settings were introduced and used it. So it seems it broken in 4.2.0
> > but was working in 4.1 but I do not have 4.1 config saved anywhere
> > (many make oldconfigs since).
> >
> > Bisection between 4.1 and 4.2 is possible but not easy since the
> > machines are usually actively used when I am near them.
>
> This is expected and intended behavior. The whole point of
> CONFIG_NO_HZ_FULL_ALL is to keep everything off of the non-boot CPUs
> that is not explicitly placed there. Without CONFIG_NO_HZ_FULL_ALL,
> you can use the nohz_full boot parameter to select exactly which
> CPUs are to behave this way.
I'm preparing a revert of this. Many people are complaining about this.
Most of the time it's about accidentally enbling NO_HZ_FULL_ALL and I could
fix this with a warning to avoid time spent by users to chase a non-bug. But Mike
says that CONFIG_NO_HZ_FULL_ALL makes the machine unusable for anything else
than isolation workloads whereas some "normal" workload may be needed as well
by the machine before or after an isolation task.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web