Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1709924 > unrolled thread

Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant from nohz

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Luiz Capitulino <lcapitulino@redhat.com> - 2017-08-11 21:20 +0200
    Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Frederic Weisbecker <fweisbec@gmail.com> - 2017-08-12 16:20 +0200
      Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Luiz Capitulino <lcapitulino@redhat.com> - 2017-08-13 17:20 +0200
        Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Frederic Weisbecker <fweisbec@gmail.com> - 2017-08-14 19:10 +0200
          Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Luiz Capitulino <lcapitulino@redhat.com> - 2017-08-14 19:40 +0200
            Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Mike Galbraith <efault@gmx.de> - 2017-08-14 20:40 +0200
              Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Frederic Weisbecker <fweisbec@gmail.com> - 2017-08-15 15:10 +0200
                Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Mike Galbraith <efault@gmx.de> - 2017-08-15 17:20 +0200
                  Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> - 2017-08-15 17:40 +0200
                    Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Christopher Lameter <cl@linux.com> - 2017-08-15 18:00 +0200
                      Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Mike Galbraith <efault@gmx.de> - 2017-08-15 18:00 +0200
                        Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Christopher Lameter <cl@linux.com> - 2017-08-16 15:10 +0200
                          Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Mike Galbraith <efault@gmx.de> - 2017-08-16 20:10 +0200
                    Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant  from nohz Mike Galbraith <efault@gmx.de> - 2017-08-15 18:00 +0200

#1709924 — Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant from nohz

On Fri, 21 Jul 2017 15:21:28 +0200
Frederic Weisbecker <fweisbec@gmail.com> wrote:

> The housekeeping is currently driven by nohz_full where any CPU that
> is not in the nohz_full range is considered as a housekeeper. This is
> a design mistake because nohz is just a detail among all the existing
> isolation features. Nohz shouldn't imply anything else than tick related
> things.
> 
> We rather want to drive all the isolation features from the housekeeping
> subsystem which is responsible for all the work that can be either
> affined (unpinned workqueues, timers, kthreads, ...) or offloaded
> (scheduler tick, ...).

That makes a lot of sense. I think this is moving in the right
direction. I have a comment below though.

> 
> Let's start with a boot option to define the housekeepers. We should be
> able to further enhance that through cpusets.
> 
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Chris Metcalf <cmetcalf@mellanox.com>
> Cc: Rik van Riel <riel@redhat.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Mike Galbraith <efault@gmx.de>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Wanpeng Li <kernellwp@gmail.com>
> Cc: Luiz Capitulino <lcapitulino@redhat.com>
> ---
>  include/linux/housekeeping.h |  2 --
>  init/main.c                  |  2 --
>  kernel/housekeeping.c        | 22 ++++++++++------------
>  3 files changed, 10 insertions(+), 16 deletions(-)
> 
> diff --git a/include/linux/housekeeping.h b/include/linux/housekeeping.h
> index 320cc2b..ba769c8 100644
> --- a/include/linux/housekeeping.h
> +++ b/include/linux/housekeeping.h
> @@ -11,7 +11,6 @@ extern int housekeeping_any_cpu(void);
>  extern const struct cpumask *housekeeping_cpumask(void);
>  extern void housekeeping_affine(struct task_struct *t);
>  extern bool housekeeping_test_cpu(int cpu);
> -extern void __init housekeeping_init(void);
>  
>  #else
>  
> @@ -26,7 +25,6 @@ static inline const struct cpumask *housekeeping_cpumask(void)
>  }
>  
>  static inline void housekeeping_affine(struct task_struct *t) { }
> -static inline void housekeeping_init(void) { }
>  #endif /* CONFIG_NO_HZ_FULL */
>  
>  static inline bool housekeeping_cpu(int cpu)
> diff --git a/init/main.c b/init/main.c
> index 9904a1e..9789ab7 100644
> --- a/init/main.c
> +++ b/init/main.c
> @@ -46,7 +46,6 @@
>  #include <linux/cgroup.h>
>  #include <linux/efi.h>
>  #include <linux/tick.h>
> -#include <linux/housekeeping.h>
>  #include <linux/interrupt.h>
>  #include <linux/taskstats_kern.h>
>  #include <linux/delayacct.h>
> @@ -608,7 +607,6 @@ asmlinkage __visible void __init start_kernel(void)
>  	early_irq_init();
>  	init_IRQ();
>  	tick_init();
> -	housekeeping_init();
>  	rcu_init_nohz();
>  	init_timers();
>  	hrtimers_init();
> diff --git a/kernel/housekeeping.c b/kernel/housekeeping.c
> index f8be7e6..a54765d 100644
> --- a/kernel/housekeeping.c
> +++ b/kernel/housekeeping.c
> @@ -45,23 +45,21 @@ bool housekeeping_test_cpu(int cpu)
>  	return true;
>  }
>  
> -void __init housekeeping_init(void)
> +/* Parse the boot-time housekeeping CPU list from the kernel parameters. */
> +static int __init housekeeping_setup(char *str)
>  {
> -	if (!tick_nohz_full_enabled())
> -		return;
> -
> -	if (!alloc_cpumask_var(&housekeeping_mask, GFP_KERNEL)) {
> -		WARN(1, "NO_HZ: Can't allocate not-full dynticks cpumask\n");
> -		cpumask_clear(tick_nohz_full_mask);
> -		tick_nohz_full_running = false;
> -		return;
> +	alloc_bootmem_cpumask_var(&housekeeping_mask);
> +	if (cpulist_parse(str, housekeeping_mask) < 0) {
> +		pr_warn("Housekeeping: Incorrect cpumask\n");
> +		free_bootmem_cpumask_var(housekeeping_mask);
> +		return 1;
>  	}
>  
> -	cpumask_andnot(housekeeping_mask,
> -		       cpu_possible_mask, tick_nohz_full_mask);
> -
>  	static_branch_enable(&housekeeping_overriden);
>  
>  	/* We need at least one CPU to handle housekeeping work */
>  	WARN_ON_ONCE(cpumask_empty(housekeeping_mask));
> +
> +	return 1;
>  }
> +__setup("housekeeping=", housekeeping_setup);

Am I right that from now on nohz_full= users will also have
to specify housekeeping= in order to get nohz_full working?
If that's correct, then won't this patch break nohz_full for
existing setups?

Also, I just give this series a try and got this:

[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.13.0-rc4+ root=/dev/mapper/rhel_virtlab508-root ro crashkernel=auto rd.lvm.lv=rhel_virtlab508/root rd.lvm.lv=rhel_virtlab508/swap console=ttyS1,115200 LANG=en_US.UTF-8 housekeeping=0,2,4,6,8,10,12,14,1 isolcpus=15 nohz_full=15 intel_pstate=disable
[    0.000000] static_key_slow_inc used before call to jump_label_init
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at kernel/jump_label.c:108 static_key_slow_inc+0x86/0xa0
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.13.0-rc4+ #2
[    0.000000] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.2.6 06/08/2015
[    0.000000] task: ffffffffb6010480 task.stack: ffffffffb6000000
[    0.000000] RIP: 0010:static_key_slow_inc+0x86/0xa0
[    0.000000] RSP: 0000:ffffffffb6003d98 EFLAGS: 00010046 ORIG_RAX: 0000000000000000
[    0.000000] RAX: 0000000000000037 RBX: ffffffffb66aa780 RCX: ffffffffb6061308
[    0.000000] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000002
[    0.000000] RBP: ffffffffb6003da0 R08: 6b5f636974617473 R09: 00000000000001e4
[    0.000000] R10: 776f6c735f79656b R11: 0000000000000000 R12: ffff972c3ffd1cfe
[    0.000000] R13: ffffffffffffffff R14: 0000000000000000 R15: 000000000000000d
[    0.000000] FS:  0000000000000000(0000) GS:ffff97282ea00000(0000) knlGS:0000000000000000
[    0.000000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.000000] CR2: ffff972905974000 CR3: 0000000545209000 CR4: 00000000000406b0
[    0.000000] Call Trace:
[    0.000000]  static_key_enable+0x1d/0x30
[    0.000000]  housekeeping_setup+0x5a/0x7e
[    0.000000]  unknown_bootoption+0x8b/0x19a
[    0.000000]  parse_args+0x224/0x3b0
[    0.000000]  ? set_init_arg+0x5a/0x5a
[    0.000000]  start_kernel+0x209/0x4cd
[    0.000000]  ? set_init_arg+0x5a/0x5a
[    0.000000]  ? early_idt_handler_array+0x120/0x120
[    0.000000]  x86_64_start_reservations+0x24/0x26
[    0.000000]  x86_64_start_kernel+0x14c/0x16f
[    0.000000]  secondary_startup_64+0x9f/0x9f

[toc] | [next] | [standalone]

#1710289

On Fri, Aug 11, 2017 at 03:09:57PM -0400, Luiz Capitulino wrote:
> On Fri, 21 Jul 2017 15:21:28 +0200
> Frederic Weisbecker <fweisbec@gmail.com> wrote:
> > -void __init housekeeping_init(void)
> > +/* Parse the boot-time housekeeping CPU list from the kernel parameters. */
> > +static int __init housekeeping_setup(char *str)
> >  {
> > -	if (!tick_nohz_full_enabled())
> > -		return;
> > -
> > -	if (!alloc_cpumask_var(&housekeeping_mask, GFP_KERNEL)) {
> > -		WARN(1, "NO_HZ: Can't allocate not-full dynticks cpumask\n");
> > -		cpumask_clear(tick_nohz_full_mask);
> > -		tick_nohz_full_running = false;
> > -		return;
> > +	alloc_bootmem_cpumask_var(&housekeeping_mask);
> > +	if (cpulist_parse(str, housekeeping_mask) < 0) {
> > +		pr_warn("Housekeeping: Incorrect cpumask\n");
> > +		free_bootmem_cpumask_var(housekeeping_mask);
> > +		return 1;
> >  	}
> >  
> > -	cpumask_andnot(housekeeping_mask,
> > -		       cpu_possible_mask, tick_nohz_full_mask);
> > -
> >  	static_branch_enable(&housekeeping_overriden);
> >  
> >  	/* We need at least one CPU to handle housekeeping work */
> >  	WARN_ON_ONCE(cpumask_empty(housekeeping_mask));
> > +
> > +	return 1;
> >  }
> > +__setup("housekeeping=", housekeeping_setup);
> 
> Am I right that from now on nohz_full= users will also have
> to specify housekeeping= in order to get nohz_full working?
> If that's correct, then won't this patch break nohz_full for
> existing setups?

nohz_full= will still work but will only imply tick stop. A few isolation
details that were enabled by nohz_full= won't be handled anymore such as:
unbound timers affinity, watchdog disablement, rcu threads affinity, sched idle
load balancing... Those are now handled by housekeeping=

So yes in a sense, this can break some setup that assume nohz_full= does more
than stopping the tick.

Perhaps I should remove the nohz_full= parameter altogether and let nohz_full controlled
by housekeeping= only. How much can kernel parameters be considered as kernel ABIs?

Also I'm wondering if "housekeeping=" is a clear name for users. "isolation=" or
"cpu_isolation=" would be better and more obvious. Housekeeping based naming would only be
internal implementation detail. And deactivating the tick through "cpu_isolation=" would
be clearer than if we did through "housekeeping=".

Of course the problem is that we already have "isolcpus=". But re-implementing isolcpus
on top of housekeeping might be a good idea. I believe that the current implementation on
top of NULL domains isn't much beloved. A less controversial implementation might even
allow us to control it though cpusets.
 
> 
> Also, I just give this series a try and got this:
> 
> [    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.13.0-rc4+ root=/dev/mapper/rhel_virtlab508-root ro crashkernel=auto rd.lvm.lv=rhel_virtlab508/root rd.lvm.lv=rhel_virtlab508/swap console=ttyS1,115200 LANG=en_US.UTF-8 housekeeping=0,2,4,6,8,10,12,14,1 isolcpus=15 nohz_full=15 intel_pstate=disable
> [    0.000000] static_key_slow_inc used before call to jump_label_init
> [    0.000000] ------------[ cut here ]------------
> [    0.000000] WARNING: CPU: 0 PID: 0 at kernel/jump_label.c:108 static_key_slow_inc+0x86/0xa0

Oops ^_^

Thanks.

[toc] | [prev] | [next] | [standalone]

#1710551

On Sat, 12 Aug 2017 16:10:06 +0200
Frederic Weisbecker <fweisbec@gmail.com> wrote:

> On Fri, Aug 11, 2017 at 03:09:57PM -0400, Luiz Capitulino wrote:
> > On Fri, 21 Jul 2017 15:21:28 +0200
> > Frederic Weisbecker <fweisbec@gmail.com> wrote:  
> > > -void __init housekeeping_init(void)
> > > +/* Parse the boot-time housekeeping CPU list from the kernel parameters. */
> > > +static int __init housekeeping_setup(char *str)
> > >  {
> > > -	if (!tick_nohz_full_enabled())
> > > -		return;
> > > -
> > > -	if (!alloc_cpumask_var(&housekeeping_mask, GFP_KERNEL)) {
> > > -		WARN(1, "NO_HZ: Can't allocate not-full dynticks cpumask\n");
> > > -		cpumask_clear(tick_nohz_full_mask);
> > > -		tick_nohz_full_running = false;
> > > -		return;
> > > +	alloc_bootmem_cpumask_var(&housekeeping_mask);
> > > +	if (cpulist_parse(str, housekeeping_mask) < 0) {
> > > +		pr_warn("Housekeeping: Incorrect cpumask\n");
> > > +		free_bootmem_cpumask_var(housekeeping_mask);
> > > +		return 1;
> > >  	}
> > >  
> > > -	cpumask_andnot(housekeeping_mask,
> > > -		       cpu_possible_mask, tick_nohz_full_mask);
> > > -
> > >  	static_branch_enable(&housekeeping_overriden);
> > >  
> > >  	/* We need at least one CPU to handle housekeeping work */
> > >  	WARN_ON_ONCE(cpumask_empty(housekeeping_mask));
> > > +
> > > +	return 1;
> > >  }
> > > +__setup("housekeeping=", housekeeping_setup);  
> > 
> > Am I right that from now on nohz_full= users will also have
> > to specify housekeeping= in order to get nohz_full working?
> > If that's correct, then won't this patch break nohz_full for
> > existing setups?  
> 
> nohz_full= will still work but will only imply tick stop. A few isolation
> details that were enabled by nohz_full= won't be handled anymore such as:
> unbound timers affinity, watchdog disablement, rcu threads affinity, sched idle
> load balancing... Those are now handled by housekeeping=
> 
> So yes in a sense, this can break some setup that assume nohz_full= does more
> than stopping the tick.

Yes, the problem is that this is how it has always worked. Also,
the breakage will be very subtle and hard to debug.

> Perhaps I should remove the nohz_full= parameter altogether and let nohz_full controlled
> by housekeeping= only. How much can kernel parameters be considered as kernel ABIs?

That's a very good question, I don't have an answer for that.

> Also I'm wondering if "housekeeping=" is a clear name for users. "isolation=" or
> "cpu_isolation=" would be better and more obvious. Housekeeping based naming would only be
> internal implementation detail. And deactivating the tick through "cpu_isolation=" would
> be clearer than if we did through "housekeeping=".

That's exactly my thinking while I was reviewing the series!

> Of course the problem is that we already have "isolcpus=". But re-implementing isolcpus
> on top of housekeeping might be a good idea. I believe that the current implementation on
> top of NULL domains isn't much beloved. A less controversial implementation might even
> allow us to control it though cpusets.

You're completely right. Some people don't use isolcpus= because it
disables load balancing and that may be a problem for setups where
tasks are pinned to a set of CPUs where the number of tasks is greater
than the number of CPUs. However, for the cases where you have a
single task pinned to a CPU, having load balancing taking place adds
an extra latency (I won't remember how much, but I guess it was more
than 10us).

If there's a way to "disable" load balancing from user-space, say
with cpusets, then I think we should keep the isolated CPUs attached
to a domain as you suggest.

Another detail about isolcpus= is that it doesn't isolate the CPU
from kernel threads. That is, unpinned kernel threads are allowed
to run on CPUs not isolated with isolcpus=. We might consider changing
that for a new isolation option.

I know that there are many arguments against isolcpus= and some people
advice using cpusets. The problem with that advice is that isolcpus=
goes a bit beyond isolating a CPU from user-space tasks. One additional
thing is does for example, is pinning the kernel_init() thread to
housekeeping CPUs. This is key, because that thread will create timers
at early boot that will pin themselves to the CPU they run.

Finally, I'm wondering how all this will fit together with TASK_ISOLATION.
One of the questions I ask myself is: can/should the things TASK_ISOLATION
does be done by a kernel command-line parameter instead? Or should we
try to come up with a list of global things to control (eg. the tick,
kernel thread affinity, etc) and per-task controls?

[toc] | [prev] | [next] | [standalone]

#1711261

On Sun, Aug 13, 2017 at 11:13:40AM -0400, Luiz Capitulino wrote:
> On Sat, 12 Aug 2017 16:10:06 +0200
> Frederic Weisbecker <fweisbec@gmail.com> wrote:
> > > Am I right that from now on nohz_full= users will also have
> > > to specify housekeeping= in order to get nohz_full working?
> > > If that's correct, then won't this patch break nohz_full for
> > > existing setups?  
> > 
> > nohz_full= will still work but will only imply tick stop. A few isolation
> > details that were enabled by nohz_full= won't be handled anymore such as:
> > unbound timers affinity, watchdog disablement, rcu threads affinity, sched idle
> > load balancing... Those are now handled by housekeeping=
> > 
> > So yes in a sense, this can break some setup that assume nohz_full= does more
> > than stopping the tick.
> 
> Yes, the problem is that this is how it has always worked. Also,
> the breakage will be very subtle and hard to debug.

[...]

> 
> > Perhaps I should remove the nohz_full= parameter altogether and let nohz_full controlled
> > by housekeeping= only. How much can kernel parameters be considered as kernel ABIs?
> 
> That's a very good question, I don't have an answer for that.

That said, "nohz_full=" never implied too much isolation features so far, and those have
often changed over time, as in RCU. I think unbound timer affinity is the most important
one.

Perhaps we can keep "nohz_full=1-15" as an alias for a future "cpu_isolation=nohz,1-15"
and at least imply unbound timer affinity with it.

> 
> > Also I'm wondering if "housekeeping=" is a clear name for users. "isolation=" or
> > "cpu_isolation=" would be better and more obvious. Housekeeping based naming would only be
> > internal implementation detail. And deactivating the tick through "cpu_isolation=" would
> > be clearer than if we did through "housekeeping=".
> 
> That's exactly my thinking while I was reviewing the series!
> 
> > Of course the problem is that we already have "isolcpus=". But re-implementing isolcpus
> > on top of housekeeping might be a good idea. I believe that the current implementation on
> > top of NULL domains isn't much beloved. A less controversial implementation might even
> > allow us to control it though cpusets.
> 
> You're completely right. Some people don't use isolcpus= because it
> disables load balancing and that may be a problem for setups where
> tasks are pinned to a set of CPUs where the number of tasks is greater
> than the number of CPUs. However, for the cases where you have a
> single task pinned to a CPU, having load balancing taking place adds
> an extra latency (I won't remember how much, but I guess it was more
> than 10us).

What is the source of the load balancing inducing such latency when a single
task is affine to a CPU? If this is idle load balancing, it is now affine to
housekeepers. If this is task wakeup then it's suprising because select_task_rq()
is optimized toward single CPU affinity.

Is there another source I'm overlooking?

> If there's a way to "disable" load balancing from user-space, say
> with cpusets, then I think we should keep the isolated CPUs attached
> to a domain as you suggest.

I'm not sure such a solution would be accepted. The most sensible way
to disable load balancing is still to tune the affinity of tasks. If there
is an off-case overhead with load balancing (ie: when no more than one
task is affine to that CPU) then we should solve that with a fast path.

> Another detail about isolcpus= is that it doesn't isolate the CPU
> from kernel threads. That is, unpinned kernel threads are allowed
> to run on CPUs not isolated with isolcpus=. We might consider changing
> that for a new isolation option.

You mean unpinned kernel threads are allowed to run on isolcpus, right?
That definetly can be solved.

> 
> I know that there are many arguments against isolcpus= and some people
> advice using cpusets. The problem with that advice is that isolcpus=
> goes a bit beyond isolating a CPU from user-space tasks. One additional
> thing is does for example, is pinning the kernel_init() thread to
> housekeeping CPUs. This is key, because that thread will create timers
> at early boot that will pin themselves to the CPU they run.

Right, but also unbound timers are affine to housekeepers, we needed that for
nohz_full.

> Finally, I'm wondering how all this will fit together with TASK_ISOLATION.
> One of the questions I ask myself is: can/should the things TASK_ISOLATION
> does be done by a kernel command-line parameter instead? Or should we
> try to come up with a list of global things to control (eg. the tick,
> kernel thread affinity, etc) and per-task controls?

So I've been thinking a lot about that lately. I told Chris that TASK_ISOLATION
shouldn't be a CPU feature but a task feature. Then I realized that it doesn't work
either, my bad :-)  In the end I think that the most part of it must be a CPU
property: nohz, task isolation, timers and workqueue affinity, etc... Then what's
left for the per task thing is to tell it when it is unexpectingly interrupted by noise.

Therefore I think most of the isolation features should be controlled by
command line and cpusets (through a new cpuset subsystem maybe) then TASK_ISOLATION
through prtcl() for the noise monitoring.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1711273

On Mon, 14 Aug 2017 19:01:09 +0200
Frederic Weisbecker <fweisbec@gmail.com> wrote:

> > > Perhaps I should remove the nohz_full= parameter altogether and let nohz_full controlled
> > > by housekeeping= only. How much can kernel parameters be considered as kernel ABIs?  
> > 
> > That's a very good question, I don't have an answer for that.  
> 
> That said, "nohz_full=" never implied too much isolation features so far, and those have
> often changed over time, as in RCU. I think unbound timer affinity is the most important
> one.
> 
> Perhaps we can keep "nohz_full=1-15" as an alias for a future "cpu_isolation=nohz,1-15"
> and at least imply unbound timer affinity with it.

That would work for me.

> > > Also I'm wondering if "housekeeping=" is a clear name for users. "isolation=" or
> > > "cpu_isolation=" would be better and more obvious. Housekeeping based naming would only be
> > > internal implementation detail. And deactivating the tick through "cpu_isolation=" would
> > > be clearer than if we did through "housekeeping=".  
> > 
> > That's exactly my thinking while I was reviewing the series!
> >   
> > > Of course the problem is that we already have "isolcpus=". But re-implementing isolcpus
> > > on top of housekeeping might be a good idea. I believe that the current implementation on
> > > top of NULL domains isn't much beloved. A less controversial implementation might even
> > > allow us to control it though cpusets.  
> > 
> > You're completely right. Some people don't use isolcpus= because it
> > disables load balancing and that may be a problem for setups where
> > tasks are pinned to a set of CPUs where the number of tasks is greater
> > than the number of CPUs. However, for the cases where you have a
> > single task pinned to a CPU, having load balancing taking place adds
> > an extra latency (I won't remember how much, but I guess it was more
> > than 10us).  
> 
> What is the source of the load balancing inducing such latency when a single
> task is affine to a CPU? If this is idle load balancing, it is now affine to
> housekeepers. If this is task wakeup then it's suprising because select_task_rq()
> is optimized toward single CPU affinity.

I guess it was idle load balancing, but I don't remember because this
was a few years ago. I think this might be reproducible without using
isolcpus=. I'll give it a try shortly and let you know.

> Is there another source I'm overlooking?
> 
> > If there's a way to "disable" load balancing from user-space, say
> > with cpusets, then I think we should keep the isolated CPUs attached
> > to a domain as you suggest.  
> 
> I'm not sure such a solution would be accepted. The most sensible way
> to disable load balancing is still to tune the affinity of tasks. If there
> is an off-case overhead with load balancing (ie: when no more than one
> task is affine to that CPU) then we should solve that with a fast path.

OK, I'll take a look.

> > Another detail about isolcpus= is that it doesn't isolate the CPU
> > from kernel threads. That is, unpinned kernel threads are allowed
> > to run on CPUs not isolated with isolcpus=. We might consider changing
> > that for a new isolation option.  
> 
> You mean unpinned kernel threads are allowed to run on isolcpus, right?

Exactly.

> That definetly can be solved.
> 
> > 
> > I know that there are many arguments against isolcpus= and some people
> > advice using cpusets. The problem with that advice is that isolcpus=
> > goes a bit beyond isolating a CPU from user-space tasks. One additional
> > thing is does for example, is pinning the kernel_init() thread to
> > housekeeping CPUs. This is key, because that thread will create timers
> > at early boot that will pin themselves to the CPU they run.  
> 
> Right, but also unbound timers are affine to housekeepers, we needed that for
> nohz_full.
> 
> > Finally, I'm wondering how all this will fit together with TASK_ISOLATION.
> > One of the questions I ask myself is: can/should the things TASK_ISOLATION
> > does be done by a kernel command-line parameter instead? Or should we
> > try to come up with a list of global things to control (eg. the tick,
> > kernel thread affinity, etc) and per-task controls?  
> 
> So I've been thinking a lot about that lately. I told Chris that TASK_ISOLATION
> shouldn't be a CPU feature but a task feature. Then I realized that it doesn't work
> either, my bad :-)  In the end I think that the most part of it must be a CPU
> property: nohz, task isolation, timers and workqueue affinity, etc... Then what's
> left for the per task thing is to tell it when it is unexpectingly interrupted by noise.
> 
> Therefore I think most of the isolation features should be controlled by
> command line and cpusets (through a new cpuset subsystem maybe) then TASK_ISOLATION
> through prtcl() for the noise monitoring.

I agree.

[toc] | [prev] | [next] | [standalone]

#1711335

On Mon, 2017-08-14 at 13:34 -0400, Luiz Capitulino wrote:
> On Mon, 14 Aug 2017 19:01:09 +0200
> 
> > What is the source of the load balancing inducing such latency when a single
> > task is affine to a CPU? If this is idle load balancing, it is now affine to
> > housekeepers. If this is task wakeup then it's suprising because select_task_rq()
> > is optimized toward single CPU affinity.
> 
> I guess it was idle load balancing, but I don't remember because this
> was a few years ago. I think this might be reproducible without using
> isolcpus=. I'll give it a try shortly and let you know.

idle_balance() can swamp other noise by a couple orders of magnitude,

	-Mike

[toc] | [prev] | [next] | [standalone]

#1712177

On Mon, Aug 14, 2017 at 08:29:46PM +0200, Mike Galbraith wrote:
> On Mon, 2017-08-14 at 13:34 -0400, Luiz Capitulino wrote:
> > On Mon, 14 Aug 2017 19:01:09 +0200
> > 
> > > What is the source of the load balancing inducing such latency when a single
> > > task is affine to a CPU? If this is idle load balancing, it is now affine to
> > > housekeepers. If this is task wakeup then it's suprising because select_task_rq()
> > > is optimized toward single CPU affinity.
> > 
> > I guess it was idle load balancing, but I don't remember because this
> > was a few years ago. I think this might be reproducible without using
> > isolcpus=. I'll give it a try shortly and let you know.
> 
> idle_balance() can swamp other noise by a couple orders of magnitude,

Ah I missed that one. Is there any way we can also lower the overhead there?
It looks unfortunately hard to tell if there is only one task affine to a given CPU,
assertion on top of which we could make a fast exit.

[toc] | [prev] | [next] | [standalone]

#1712239

On Tue, 2017-08-15 at 15:07 +0200, Frederic Weisbecker wrote:
> On Mon, Aug 14, 2017 at 08:29:46PM +0200, Mike Galbraith wrote:
> > On Mon, 2017-08-14 at 13:34 -0400, Luiz Capitulino wrote:
> > > On Mon, 14 Aug 2017 19:01:09 +0200
> > > 
> > > > What is the source of the load balancing inducing such latency when a single
> > > > task is affine to a CPU? If this is idle load balancing, it is now affine to
> > > > housekeepers. If this is task wakeup then it's suprising because select_task_rq()
> > > > is optimized toward single CPU affinity.
> > > 
> > > I guess it was idle load balancing, but I don't remember because this
> > > was a few years ago. I think this might be reproducible without using
> > > isolcpus=. I'll give it a try shortly and let you know.
> > 
> > idle_balance() can swamp other noise by a couple orders of magnitude,
> 
> Ah I missed that one. Is there any way we can also lower the overhead there?

Why?  HPC proggies won't benefit from a partially filled pothole any
more that a ~zero ground clearance formula 1 car would.  The pothole
intolerant either isolate, killing (most) LB, or they meet a wall.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1712262

On Tue, Aug 15, 2017 at 05:15:23PM +0200, Mike Galbraith wrote:
> On Tue, 2017-08-15 at 15:07 +0200, Frederic Weisbecker wrote:
> > On Mon, Aug 14, 2017 at 08:29:46PM +0200, Mike Galbraith wrote:
> > > On Mon, 2017-08-14 at 13:34 -0400, Luiz Capitulino wrote:
> > > > On Mon, 14 Aug 2017 19:01:09 +0200
> > > > 
> > > > > What is the source of the load balancing inducing such latency when a single
> > > > > task is affine to a CPU? If this is idle load balancing, it is now affine to
> > > > > housekeepers. If this is task wakeup then it's suprising because select_task_rq()
> > > > > is optimized toward single CPU affinity.
> > > > 
> > > > I guess it was idle load balancing, but I don't remember because this
> > > > was a few years ago. I think this might be reproducible without using
> > > > isolcpus=. I'll give it a try shortly and let you know.
> > > 
> > > idle_balance() can swamp other noise by a couple orders of magnitude,
> > 
> > Ah I missed that one. Is there any way we can also lower the overhead there?
> 
> Why?  HPC proggies won't benefit from a partially filled pothole any
> more that a ~zero ground clearance formula 1 car would.  The pothole
> intolerant either isolate, killing (most) LB, or they meet a wall.

Don't the HPC guys just disable idle_balance(), or am I out of date again?

							Thanx, Paul

[toc] | [prev] | [next] | [standalone]

#1712287

On Tue, 15 Aug 2017, Paul E. McKenney wrote:

> Don't the HPC guys just disable idle_balance(), or am I out of date again?

Ummm.. Why does idle management matter when your goal is to keep all
processor busy working at maximum throughput?

[toc] | [prev] | [next] | [standalone]

#1712288

On Tue, 2017-08-15 at 10:52 -0500, Christopher Lameter wrote:
> On Tue, 15 Aug 2017, Paul E. McKenney wrote:
> 
> > Don't the HPC guys just disable idle_balance(), or am I out of date again?
> 
> Ummm.. Why does idle management matter when your goal is to keep all
> processor busy working at maximum throughput?

If you _never_ idle, you never have to worry about it.  Is 100% CPU
until the end of time all there is to HPC?

	-Mike

[toc] | [prev] | [next] | [standalone]

#1712930

[Multipart message — attachments visible in raw view] — view raw

On Tue, 15 Aug 2017, Mike Galbraith wrote:

> On Tue, 2017-08-15 at 10:52 -0500, Christopher Lameter wrote:
> > On Tue, 15 Aug 2017, Paul E. McKenney wrote:
> >
> > > Don't the HPC guys just disable idle_balance(), or am I out of date again?
> >
> > Ummm.. Why does idle management matter when your goal is to keep all
> > processor busy working at maximum throughput?
>
> If you _never_ idle, you never have to worry about it.  Is 100% CPU
> until the end of time all there is to HPC?

Most of the time that is true for HPC loads. They may also go through a
I/O throughput constrained processing phase or synchronization phase where
idle activity occurs.

However, there are also low latency loads that are often confused with
HPC. Those are usually waiting idle until an event happens and then have
to react to it in the fastest way possible. After that they go back to
idle. You could call this a RT load (since the term seems to be so
flexible...) but its more event based than the typical idea of realtime
(do something at this and thata time).

[toc] | [prev] | [next] | [standalone]

#1713204

On Tue, 2017-08-15 at 11:26 -0500, Christopher Lameter wrote:
> On Tue, 15 Aug 2017, Mike Galbraith wrote:
> 
> > On Tue, 2017-08-15 at 10:52 -0500, Christopher Lameter wrote:
> > > On Tue, 15 Aug 2017, Paul E. McKenney wrote:
> > >
> > > > Don't the HPC guys just disable idle_balance(), or am I out of date again?
> > >
> > > Ummm.. Why does idle management matter when your goal is to keep all
> > > processor busy working at maximum throughput?
> >
> > If you _never_ idle, you never have to worry about it.  Is 100% CPU
> > until the end of time all there is to HPC?
> 
> Most of the time that is true for HPC loads. They may also go through a
> I/O throughput constrained processing phase or synchronization phase where
> idle activity occurs.

That synchronization is what I tend to get all hung up on pondering HPC
vs RT terminology.  Care and feeding of parallel pipelines has got to
be loaded to the gills with synchronization (intermediate math results
etc), rendering the sum event driven.

Doesn't matter, both acronyms reduce to latency intolerant.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1712293

On Tue, 2017-08-15 at 08:30 -0700, Paul E. McKenney wrote:
> On Tue, Aug 15, 2017 at 05:15:23PM +0200, Mike Galbraith wrote:
> > On Tue, 2017-08-15 at 15:07 +0200, Frederic Weisbecker wrote:
> > > On Mon, Aug 14, 2017 at 08:29:46PM +0200, Mike Galbraith wrote:
> > > > On Mon, 2017-08-14 at 13:34 -0400, Luiz Capitulino wrote:
> > > > > On Mon, 14 Aug 2017 19:01:09 +0200
> > > > > 
> > > > > > What is the source of the load balancing inducing such latency when a single
> > > > > > task is affine to a CPU? If this is idle load balancing, it is now affine to
> > > > > > housekeepers. If this is task wakeup then it's suprising because select_task_rq()
> > > > > > is optimized toward single CPU affinity.
> > > > > 
> > > > > I guess it was idle load balancing, but I don't remember because this
> > > > > was a few years ago. I think this might be reproducible without using
> > > > > isolcpus=. I'll give it a try shortly and let you know.
> > > > 
> > > > idle_balance() can swamp other noise by a couple orders of magnitude,
> > > 
> > > Ah I missed that one. Is there any way we can also lower the overhead there?
> > 
> > Why?  HPC proggies won't benefit from a partially filled pothole any
> > more that a ~zero ground clearance formula 1 car would.  The pothole
> > intolerant either isolate, killing (most) LB, or they meet a wall.
> 
> Don't the HPC guys just disable idle_balance(), or am I out of date again?

They could do just that if what they're doing is not really critical. 
I'm not an HPC guy, so can only speculate.  I don't see much difference
between HPC and RT though, the rules of the game seem to be about the
same (them both being HPC;).. what you can control, you do control.

	-Mike

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web