Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1721482 > unrolled thread

Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on housekeeping

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on  housekeeping Peter Zijlstra <peterz@infradead.org> - 2017-08-28 12:20 +0200
    Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on  housekeeping Frederic Weisbecker <fweisbec@gmail.com> - 2017-08-28 15:30 +0200
      Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on  housekeeping Peter Zijlstra <peterz@infradead.org> - 2017-08-28 15:40 +0200
        Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on  housekeeping Frederic Weisbecker <fweisbec@gmail.com> - 2017-08-28 17:30 +0200
          Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on  housekeeping Peter Zijlstra <peterz@infradead.org> - 2017-08-28 18:30 +0200
            Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on  housekeeping Christopher Lameter <cl@linux.com> - 2017-08-28 19:00 +0200
            Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on  housekeeping Frederic Weisbecker <fweisbec@gmail.com> - 2017-08-28 19:40 +0200
              Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on  housekeeping Thomas Gleixner <tglx@linutronix.de> - 2017-08-31 21:00 +0200
                Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on  housekeeping Frederic Weisbecker <fweisbec@gmail.com> - 2017-09-01 01:10 +0200

#1721482 — Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on housekeeping

On Wed, Aug 23, 2017 at 03:51:11AM +0200, Frederic Weisbecker wrote:
> We want to centralize the isolation features on the housekeeping
> subsystem and scheduler isolation is a significant part of it.
> 
> While at it, this is a proposition for a reimplementation of isolcpus=
> that doesn't involve  scheduler domain isolation. Therefore this
> brings a behaviour change: all user tasks inherit init/1 affinity which
> avoid the isolcpus= range. But if a task later overrides its affinity
> which turns out to intersect an isolated CPU, load balancing may occur
> on it.
> 
> OTOH such a reimplementation that doesn't shortcut scheduler internals
> makes a better candidate for an interface extension to cpuset.

Not sure we can do this. It'll break users that rely on the no
scheduling thing, that's a well documented part of isolcpus.

[toc] | [next] | [standalone]

#1721617

On Mon, Aug 28, 2017 at 12:09:57PM +0200, Peter Zijlstra wrote:
> On Wed, Aug 23, 2017 at 03:51:11AM +0200, Frederic Weisbecker wrote:
> > We want to centralize the isolation features on the housekeeping
> > subsystem and scheduler isolation is a significant part of it.
> > 
> > While at it, this is a proposition for a reimplementation of isolcpus=
> > that doesn't involve  scheduler domain isolation. Therefore this
> > brings a behaviour change: all user tasks inherit init/1 affinity which
> > avoid the isolcpus= range. But if a task later overrides its affinity
> > which turns out to intersect an isolated CPU, load balancing may occur
> > on it.
> > 
> > OTOH such a reimplementation that doesn't shortcut scheduler internals
> > makes a better candidate for an interface extension to cpuset.
> 
> Not sure we can do this. It'll break users that rely on the no
> scheduling thing, that's a well documented part of isolcpus.

That was my worry :-s  That NULL domain was probably a design mistake and
I fear we now have to maintain it.

[toc] | [prev] | [next] | [standalone]

#1721627

On Mon, Aug 28, 2017 at 03:23:06PM +0200, Frederic Weisbecker wrote:
> On Mon, Aug 28, 2017 at 12:09:57PM +0200, Peter Zijlstra wrote:
> > On Wed, Aug 23, 2017 at 03:51:11AM +0200, Frederic Weisbecker wrote:
> > > We want to centralize the isolation features on the housekeeping
> > > subsystem and scheduler isolation is a significant part of it.
> > > 
> > > While at it, this is a proposition for a reimplementation of isolcpus=
> > > that doesn't involve  scheduler domain isolation. Therefore this
> > > brings a behaviour change: all user tasks inherit init/1 affinity which
> > > avoid the isolcpus= range. But if a task later overrides its affinity
> > > which turns out to intersect an isolated CPU, load balancing may occur
> > > on it.
> > > 
> > > OTOH such a reimplementation that doesn't shortcut scheduler internals
> > > makes a better candidate for an interface extension to cpuset.
> > 
> > Not sure we can do this. It'll break users that rely on the no
> > scheduling thing, that's a well documented part of isolcpus.
> 
> That was my worry :-s  That NULL domain was probably a design mistake and
> I fear we now have to maintain it.

I'm fairly sure that was very intentional. If you want to isolate stuff
you don't want load-balancing. You get the same NULL domain with cpusets
if you disable balancing for a set of CPUs.

Now, I completely hate the isolcpus feature and wish is a speedy death,
but replacing it with something sensible is difficult because cgroups
:-(

[toc] | [prev] | [next] | [standalone]

#1721743

On Mon, Aug 28, 2017 at 03:31:16PM +0200, Peter Zijlstra wrote:
> On Mon, Aug 28, 2017 at 03:23:06PM +0200, Frederic Weisbecker wrote:
> > On Mon, Aug 28, 2017 at 12:09:57PM +0200, Peter Zijlstra wrote:
> > > On Wed, Aug 23, 2017 at 03:51:11AM +0200, Frederic Weisbecker wrote:
> > > > We want to centralize the isolation features on the housekeeping
> > > > subsystem and scheduler isolation is a significant part of it.
> > > > 
> > > > While at it, this is a proposition for a reimplementation of isolcpus=
> > > > that doesn't involve  scheduler domain isolation. Therefore this
> > > > brings a behaviour change: all user tasks inherit init/1 affinity which
> > > > avoid the isolcpus= range. But if a task later overrides its affinity
> > > > which turns out to intersect an isolated CPU, load balancing may occur
> > > > on it.
> > > > 
> > > > OTOH such a reimplementation that doesn't shortcut scheduler internals
> > > > makes a better candidate for an interface extension to cpuset.
> > > 
> > > Not sure we can do this. It'll break users that rely on the no
> > > scheduling thing, that's a well documented part of isolcpus.
> > 
> > That was my worry :-s  That NULL domain was probably a design mistake and
> > I fear we now have to maintain it.
> 
> I'm fairly sure that was very intentional. If you want to isolate stuff
> you don't want load-balancing.

Yes I guess that was intentional. In fact having NULL domains is convenient
as it also isolates from many things: tasks, workqueues, timers.

Although for example I guess (IIUC) that if you create an unbound timer on a NULL
domain, it will be stuck on it for ever as we can't walk any hierarchy from the
current CPU domain. I'm not sure how much that can apply to unbound workqueues
as well. But the thing is with NULL domains: things can not migrate in and neither
can them migrate out, which is not exactly what CPU isolation wants.

> You get the same NULL domain with cpusets if you disable balancing for a set of CPUs.

Ok, I didn't know that.

> 
> Now, I completely hate the isolcpus feature and wish is a speedy death,
> but replacing it with something sensible is difficult because cgroups
> :-(

Ah, that would break cgroup somehow?

[toc] | [prev] | [next] | [standalone]

#1721788

On Mon, Aug 28, 2017 at 05:27:15PM +0200, Frederic Weisbecker wrote:
> On Mon, Aug 28, 2017 at 03:31:16PM +0200, Peter Zijlstra wrote:

> > I'm fairly sure that was very intentional. If you want to isolate stuff
> > you don't want load-balancing.
> 
> Yes I guess that was intentional. In fact having NULL domains is convenient
> as it also isolates from many things: tasks, workqueues, timers.

Huh, what? That's entirely unrelated to the NULL domain.

The reason people like isolcpus= is that is ensures _nothing_ runs on
those CPUs before you explicitly place something there.

_That_ is what ensures there are no timers etc.. placed on those CPUs.

Once you run something on that CPU, it stays there.

It is also what I dislike about isolcpus, its a boot time feature, if
you want to reconfigure your system you need a reboot.

> Although for example I guess (IIUC) that if you create an unbound
> timer on a NULL domain, it will be stuck on it for ever as we can't
> walk any hierarchy from the current CPU domain.

Not sure what you're on about. Timers have their own hierarchy.

> I'm not sure how much that can apply to unbound workqueues
> as well.

Well, unbound workqueued will not immediately end up on those CPUs,
since they'll have an affinity exlusive of those CPUs per construction.

But IIRC there's an affinity setting for workqueues where you could
force it on if you wanted to.

> But the thing is with NULL domains: things can not migrate in and neither
> can them migrate out, which is not exactly what CPU isolation wants.

No, its exactly what they want. You get what you put in and nothing
more. If you want something else, use cpusets.

> > Now, I completely hate the isolcpus feature and wish is a speedy death,
> > but replacing it with something sensible is difficult because cgroups
> > :-(
> 
> Ah, that would break cgroup somehow?

Well, ideally something like this would start the system with all the
'crap' threads in !root cgroup. But that means cgroupfs needs to be
populated with at least two directories on boot. And current cgroup
cruft doesn't expect that.

[toc] | [prev] | [next] | [standalone]

#1721814

On Mon, 28 Aug 2017, Peter Zijlstra wrote:

> Well, ideally something like this would start the system with all the
> 'crap' threads in !root cgroup. But that means cgroupfs needs to be
> populated with at least two directories on boot. And current cgroup
> cruft doesn't expect that.

Maybe an affinity mask for bootup will take care of that? I once wrote an
init wrapper that restricted the number of cpus for the threads that init
spawns but we can probably do much better.

[toc] | [prev] | [next] | [standalone]

#1721849

On Mon, Aug 28, 2017 at 06:24:16PM +0200, Peter Zijlstra wrote:
> On Mon, Aug 28, 2017 at 05:27:15PM +0200, Frederic Weisbecker wrote:
> > On Mon, Aug 28, 2017 at 03:31:16PM +0200, Peter Zijlstra wrote:
> 
> > > I'm fairly sure that was very intentional. If you want to isolate stuff
> > > you don't want load-balancing.
> > 
> > Yes I guess that was intentional. In fact having NULL domains is convenient
> > as it also isolates from many things: tasks, workqueues, timers.
> 
> Huh, what? That's entirely unrelated to the NULL domain.
> 
> The reason people like isolcpus= is that is ensures _nothing_ runs on
> those CPUs before you explicitly place something there.
> 
> _That_ is what ensures there are no timers etc.. placed on those CPUs.

Sure that's what I meant.

> 
> Once you run something on that CPU, it stays there.
> 
> It is also what I dislike about isolcpus, its a boot time feature, if
> you want to reconfigure your system you need a reboot.

Indeed.

> 
> > Although for example I guess (IIUC) that if you create an unbound
> > timer on a NULL domain, it will be stuck on it for ever as we can't
> > walk any hierarchy from the current CPU domain.
> 
> Not sure what you're on about. Timers have their own hierarchy.

Check out get_nohz_timer_target() which relies on scheduler hierarchies to
look up a CPU to enqueue an unpinned timer on.

> 
> > I'm not sure how much that can apply to unbound workqueues
> > as well.
> 
> Well, unbound workqueued will not immediately end up on those CPUs,
> since they'll have an affinity exlusive of those CPUs per construction.

Ah that's right.

> But IIRC there's an affinity setting for workqueues where you could
> force it on if you wanted to.

Yep: /sys/devices/virtual/workqueue/cpumask

> 
> > But the thing is with NULL domains: things can not migrate in and neither
> > can them migrate out, which is not exactly what CPU isolation wants.
> 
> No, its exactly what they want. You get what you put in and nothing
> more. If you want something else, use cpusets.

That's still a subtle behaviour that involves knowledge of some scheduler
core details. I wish we hadn't exposed such a low level scheduler control
as a general purpose kernel parameter.

Anyway at least that confirms one worry we had: kernel parameters are kernel
ABI that we can't break.

> 
> > > Now, I completely hate the isolcpus feature and wish is a speedy death,
> > > but replacing it with something sensible is difficult because cgroups
> > > :-(
> > 
> > Ah, that would break cgroup somehow?
> 
> Well, ideally something like this would start the system with all the
> 'crap' threads in !root cgroup. But that means cgroupfs needs to be
> populated with at least two directories on boot. And current cgroup
> cruft doesn't expect that.

Ah I see.

Thanks!

[toc] | [prev] | [next] | [standalone]

#1724494

On Mon, 28 Aug 2017, Frederic Weisbecker wrote:
> On Mon, Aug 28, 2017 at 06:24:16PM +0200, Peter Zijlstra wrote:
> > On Mon, Aug 28, 2017 at 05:27:15PM +0200, Frederic Weisbecker wrote:
> > > Although for example I guess (IIUC) that if you create an unbound
> > > timer on a NULL domain, it will be stuck on it for ever as we can't
> > > walk any hierarchy from the current CPU domain.
> > 
> > Not sure what you're on about. Timers have their own hierarchy.
> 
> Check out get_nohz_timer_target() which relies on scheduler hierarchies to
> look up a CPU to enqueue an unpinned timer on.

Which is one of the most idiotic things we have in that code
path. Anna-Maria has posted this series which gets rid of that nonsense, by
queueing the timer on the current cpu into a wheel, which gets pulled in by
others. That makes a lot of sense because most of these timers get canceled
before expiry anyway. But we still need to fix the fallout and the few
corner cases to make that work reliably. We'll do that hopefully sooner
than later.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1724634

On Thu, Aug 31, 2017 at 08:53:56PM +0200, Thomas Gleixner wrote:
> On Mon, 28 Aug 2017, Frederic Weisbecker wrote:
> > On Mon, Aug 28, 2017 at 06:24:16PM +0200, Peter Zijlstra wrote:
> > > On Mon, Aug 28, 2017 at 05:27:15PM +0200, Frederic Weisbecker wrote:
> > > > Although for example I guess (IIUC) that if you create an unbound
> > > > timer on a NULL domain, it will be stuck on it for ever as we can't
> > > > walk any hierarchy from the current CPU domain.
> > > 
> > > Not sure what you're on about. Timers have their own hierarchy.
> > 
> > Check out get_nohz_timer_target() which relies on scheduler hierarchies to
> > look up a CPU to enqueue an unpinned timer on.
> 
> Which is one of the most idiotic things we have in that code
> path. Anna-Maria has posted this series which gets rid of that nonsense, by
> queueing the timer on the current cpu into a wheel, which gets pulled in by
> others. That makes a lot of sense because most of these timers get canceled
> before expiry anyway. But we still need to fix the fallout and the few
> corner cases to make that work reliably. We'll do that hopefully sooner
> than later.

Sure, I definetly agree with that change.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web