Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1708600 > unrolled thread

Re: [RFC PATCH 0/9] Introduce housekeeping subsystem

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC PATCH 0/9] Introduce housekeeping subsystem Frederic Weisbecker <fweisbec@gmail.com> - 2017-08-10 15:00 +0200
    Re: [RFC PATCH 0/9] Introduce housekeeping subsystem Mike Galbraith <efault@gmx.de> - 2017-08-11 08:40 +0200
      Re: [RFC PATCH 0/9] Introduce housekeeping subsystem Frederic Weisbecker <fweisbec@gmail.com> - 2017-08-11 17:10 +0200
        Re: [RFC PATCH 0/9] Introduce housekeeping subsystem Luiz Capitulino <lcapitulino@redhat.com> - 2017-08-11 21:30 +0200
      Re: [RFC PATCH 0/9] Introduce housekeeping subsystem Christopher Lameter <cl@linux.com> - 2017-08-11 17:40 +0200

#1708600 — Re: [RFC PATCH 0/9] Introduce housekeeping subsystem

On Fri, Jul 21, 2017 at 03:48:16PM -0400, Chris Metcalf wrote:
> On 7/21/2017 9:21 AM, Frederic Weisbecker wrote:
> >I'm leaving for two weeks so this is food for thoughts in the meantime :)
> >
> >We have a design issue with nohz_full: it drives the isolation features
> >through the *housekeeping*() functions: kthreads, unpinned timers,
> >watchdog, ...
> >
> >But things should work the other way around because the tick is just an
> >isolation feature among others.
> >
> >So we need a housekeeping subsystem to drive all these isolation
> >features, including nohz full in a later iteration. For now this is a
> >basic draft. In the long run this subsystem should also drive the tick
> >offloading (remove residual 1Hz) and all unbound kthreads.
> >
> >git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> >	nohz/0hz
> >
> >HEAD: 68e3af1de5db228bf6c2a5e721bce59a02cfc4e1
> 
> For the series:
> 
> Reviewed-by: Chris Metcalf <cmetcalf@mellanox.com>
> 
> I spotted a few typos that you should grep for and fix for your next
> version:
> "watchog", "Lets/lets" instead of "Let's/let's", "overriden" (should have
> two d's).

Thanks, I'll check those.

> 
> The new housekeeping=MASK boot option seems like it might make it a little
> irritating to specify nohz_full=MASK as well.  I guess if setting
> NO_HZ_FULL_ALL
> implied "all but housekeeping", it becomes a reasonably tidy solution.  To
> make
> this work right you might have to make the housekeeping option early_param
> instead so its value is available early enough.

Good point. But perhaps I should add a new NO_HZ_FULL_BUT_HOUSEKEEPING option.
Otherwise we'll change the meaning of NO_HZ_FULL_ALL way too much, to the point
that its default behaviour will be the exact opposite of the current one: by default
every CPU is housekeeping, so NO_HZ_FULL_ALL would have no effect anymore if we
don't set housekeeping boot option.

Also I plan to add a housekeeping option to offload the residual 1Hz tick from
nohz_full CPUs. So having "housekeeping=0,tick_offload" would make CPU 0 the
housekeeper, make the other CPUs nohz_full and handle their 1hz tick from CPU 0.

Just a few thoughts.

[toc] | [next] | [standalone]

#1709302

On Thu, 2017-08-10 at 09:57 -0400, Chris Metcalf wrote:
> On 8/10/2017 8:54 AM, Frederic Weisbecker wrote:
> > But perhaps I should add a new NO_HZ_FULL_BUT_HOUSEKEEPING option.
> > Otherwise we'll change the meaning of NO_HZ_FULL_ALL way too much, to the point
> > that its default behaviour will be the exact opposite of the current one: by default
> > every CPU is housekeeping, so NO_HZ_FULL_ALL would have no effect anymore if we
> > don't set housekeeping boot option.
> 
> Maybe a CONFIG_HOUSEKEEPING_BOOT_ONLY as a way to restrict housekeeping
> by default to just the boot cpu.  In conjunction with NOHZ_FULL_ALL you would
> then get the expected semantics.

A big box with only the boot cpu for housekeeping is likely screwed.

Personally, I think NOHZ_FULL_ALL should just die.

	-Mike

[toc] | [prev] | [next] | [standalone]

#1709722

On Fri, Aug 11, 2017 at 08:36:28AM +0200, Mike Galbraith wrote:
> On Thu, 2017-08-10 at 09:57 -0400, Chris Metcalf wrote:
> > On 8/10/2017 8:54 AM, Frederic Weisbecker wrote:
> > > But perhaps I should add a new NO_HZ_FULL_BUT_HOUSEKEEPING option.
> > > Otherwise we'll change the meaning of NO_HZ_FULL_ALL way too much, to the point
> > > that its default behaviour will be the exact opposite of the current one: by default
> > > every CPU is housekeeping, so NO_HZ_FULL_ALL would have no effect anymore if we
> > > don't set housekeeping boot option.
> > 
> > Maybe a CONFIG_HOUSEKEEPING_BOOT_ONLY as a way to restrict housekeeping
> > by default to just the boot cpu.  In conjunction with NOHZ_FULL_ALL you would
> > then get the expected semantics.
> 
> A big box with only the boot cpu for housekeeping is likely screwed.

Indeed we probably shouldn't introduce new config that affine housekeeping to a
single CPU.

> Personally, I think NOHZ_FULL_ALL should just die.

Yeah, although it's still useful for automatic boot testing to detect issues
with nohz_full on.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1709931

On Fri, 11 Aug 2017 17:01:30 +0200
Frederic Weisbecker <fweisbec@gmail.com> wrote:

> > Personally, I think NOHZ_FULL_ALL should just die.  
> 
> Yeah, although it's still useful for automatic boot testing to detect issues
> with nohz_full on.

Maybe we could rename/modify it to be a boot-time testing option for
nohz_full?

I think we may want to simplify config and kernel command-line options.
I don't think it's a good idea to break down every single isolation
feature into a kernel command-line option. What we want in the end is
having the ability to remove all the noise from a CPU right? Why not
have a single option for that?

[toc] | [prev] | [next] | [standalone]

#1709754

On Fri, 11 Aug 2017, Chris Metcalf wrote:

> > > Maybe a CONFIG_HOUSEKEEPING_BOOT_ONLY as a way to restrict housekeeping
> > > by default to just the boot cpu.  In conjunction with NOHZ_FULL_ALL you
> > > would
> > > then get the expected semantics.
> > A big box with only the boot cpu for housekeeping is likely screwed.
>
> Fair point - this kind of configuration would be primarily useful for
> dedicated systems that were running a high-traffic-rate networking
> application on many cores, for example.  In this mode you don't end up
> putting a lot of burden on the housekeeping core.  In any case,
> probably not worth adding an additional kernel config for.

The standard server config at this point is a two NUMA node with lots of
cores on each. For such a thing a single housekeeping cpu is usually
sufficient. Having a rather large number of NUMA nodes is unusual.

The question is also what is considered a "large" system at this point?
Lots of cores? Lots of NUMA nodes?

Ah, Chris since you are here: What is happening with the dataplane
patches?

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web