Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1515895 > unrolled thread

[PATCH] sched/rt: RT_RUNTIME_GREED sched feature

Back to article view | Back to linux.kernel

Contents

  [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Daniel Bristot de Oliveira <bristot@redhat.com> - 2016-11-07 09:20 +0100
    Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Tommaso Cucinotta <tommaso.cucinotta@sssup.it> - 2016-11-07 11:40 +0100
      Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Daniel Bristot de Oliveira <bristot@redhat.com> - 2016-11-07 15:00 +0100
        Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Tommaso Cucinotta <tommaso.cucinotta@sssup.it> - 2016-11-07 19:10 +0100
          Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Luca Abeni <luca.abeni@unitn.it> - 2016-11-07 19:30 +0100
        Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature luca abeni <lucabe72@gmail.com> - 2016-11-08 09:00 +0100
        Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Juri Lelli <juri.lelli@arm.com> - 2016-11-08 11:40 +0100
    Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Christoph Lameter <cl@linux.com> - 2016-11-07 18:00 +0100
      Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Steven Rostedt <rostedt@goodmis.org> - 2016-11-07 19:40 +0100
        Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Daniel Bristot de Oliveira <daniel@bristot.me> - 2016-11-07 19:50 +0100
          Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Steven Rostedt <rostedt@goodmis.org> - 2016-11-07 20:20 +0100
        Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Christoph Lameter <cl@linux.com> - 2016-11-07 20:40 +0100
          Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Christoph Lameter <cl@linux.com> - 2016-11-07 21:00 +0100
            Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Steven Rostedt <rostedt@goodmis.org> - 2016-11-07 21:10 +0100
              Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Daniel Bristot de Oliveira <daniel@bristot.me> - 2016-11-07 21:10 +0100
                Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Steven Rostedt <rostedt@goodmis.org> - 2016-11-07 21:20 +0100
                  Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Daniel Bristot de Oliveira <daniel@bristot.me> - 2016-11-07 21:40 +0100
                    Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Steven Rostedt <rostedt@goodmis.org> - 2016-11-07 21:50 +0100
                      [PATCH] sched/rt: Change default setup for RT THROTTLING Daniel Bristot de Oliveira <daniel@bristot.me> - 2016-11-08 10:30 +0100
                  Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Christoph Lameter <cl@linux.com> - 2016-11-09 00:50 +0100
          Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Steven Rostedt <rostedt@goodmis.org> - 2016-11-07 21:10 +0100
    Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Clark Williams <williams@redhat.com> - 2016-11-07 19:30 +0100
      Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Steven Rostedt <rostedt@goodmis.org> - 2016-11-07 19:40 +0100
        Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Daniel Bristot de Oliveira <daniel@bristot.me> - 2016-11-07 19:50 +0100
        Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Clark Williams <williams@redhat.com> - 2016-11-07 20:00 +0100
    Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Peter Zijlstra <peterz@infradead.org> - 2016-11-08 13:30 +0100
      Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Steven Rostedt <rostedt@goodmis.org> - 2016-11-08 15:10 +0100
        Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Peter Zijlstra <peterz@infradead.org> - 2016-11-08 18:00 +0100
          Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Steven Rostedt <rostedt@goodmis.org> - 2016-11-08 18:20 +0100
            Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Peter Zijlstra <peterz@infradead.org> - 2016-11-08 19:10 +0100
              Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Daniel Bristot de Oliveira <bristot@redhat.com> - 2016-11-08 20:40 +0100
                Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Peter Zijlstra <peterz@infradead.org> - 2016-11-08 21:00 +0100
                  Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Daniel Bristot de Oliveira <bristot@redhat.com> - 2016-11-09 14:40 +0100
                    Re: [PATCH] sched/rt: RT_RUNTIME_GREED sched feature Christoph Lameter <cl@linux.com> - 2016-11-11 19:50 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1516556

On Mon, 7 Nov 2016 13:30:15 -0600 (CST)
Christoph Lameter <cl@linux.com> wrote:

> SCHED_RR tasks alternately running on on cpu can cause endless deferral of
> kworker threads. With the global effect of the OS processing reserved
> it may be the case that the processor we are executing never gets any
> time. And if that kworker threads role is releasing a mutex (like the
> cgroup_lock) then deadlocks can result.

I believe SCHED_RR tasks will still throttle if they use up too much of
the CPU. But I still don't see how this patch helps your situation.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1516428

[Multipart message — attachments visible in raw view] — view raw

On Mon,  7 Nov 2016 09:17:55 +0100
Daniel Bristot de Oliveira <bristot@redhat.com> wrote:

> The rt throttling mechanism prevents the starvation of non-real-time
> tasks by CPU intensive real-time tasks. In terms of percentage,
> the default behavior allows real-time tasks to run up to 95% of a
> given period, leaving the other 5% of the period for non-real-time
> tasks. In the absence of non-rt tasks, the system goes idle for 5%
> of the period.
> 
> Although this behavior works fine for the purpose of avoiding
> bad real-time tasks that can hang the system, some greed users
> want to allow the real-time task to continue running in the absence
> of non-real-time tasks starving. In other words, they do not want to
> see the system going idle.
> 
> This patch implements the RT_RUNTIME_GREED scheduler feature for greedy
> users (TM). When enabled, this feature will check if non-rt tasks are
> starving before throttling the real-time task. If the real-time task
> becomes throttled, it will be unthrottled as soon as the system goes
> idle, or when the next period starts, whichever comes first.
> 
> This feature is enabled with the following command:
>   # echo RT_RUNTIME_GREED > /sys/kernel/debug/sched_features
> 

I'm still reviewing the patch, but I have to wonder why bother with making it a scheduler feature?

The SCHED_FIFO definition allows a fifo thread to starve others because a fifo task will run until it yields. Throttling was added as a safety valve to allow starved SCHED_OTHER tasks to get some cpu time.  Adding this unconditionally gets us a safety valve for throttling a badly written fifo task, but allows the fifo task to continue to consume cpu cycles if it's not starving anyone. 

Or am I missing something that's blazingly obvious?

Clark

[toc] | [prev] | [next] | [standalone]

#1516451

On Mon, 7 Nov 2016 12:22:21 -0600
Clark Williams <williams@redhat.com> wrote:

> I'm still reviewing the patch, but I have to wonder why bother with making it a scheduler feature?
> 
> The SCHED_FIFO definition allows a fifo thread to starve others
> because a fifo task will run until it yields. Throttling was added as
> a safety valve to allow starved SCHED_OTHER tasks to get some cpu
> time.  Adding this unconditionally gets us a safety valve for
> throttling a badly written fifo task, but allows the fifo task to
> continue to consume cpu cycles if it's not starving anyone. 
> 
> Or am I missing something that's blazingly obvious?

Or I say make it the default. If people want the old behavior, they can
modify SCHED_FEATURES to do so.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1516461

On 11/07/2016 07:30 PM, Steven Rostedt wrote:
>> I'm still reviewing the patch, but I have to wonder why bother with making it a scheduler feature?
>> > 
>> > The SCHED_FIFO definition allows a fifo thread to starve others
>> > because a fifo task will run until it yields. Throttling was added as
>> > a safety valve to allow starved SCHED_OTHER tasks to get some cpu
>> > time.  Adding this unconditionally gets us a safety valve for
>> > throttling a badly written fifo task, but allows the fifo task to
>> > continue to consume cpu cycles if it's not starving anyone. 
>> > 
>> > Or am I missing something that's blazingly obvious?
> Or I say make it the default. If people want the old behavior, they can
> modify SCHED_FEATURES to do so.

I added it as a feature to keep the current behavior by default.
Currently, we have two throttling modes:

With RT_RUNTIME_SHARING (default):
  before throttle, try to borrow some runtime from other CPU.

Without RT_RUNTIME_SHARING:
  throttle the RT task, even if there is nothing else to do.

The problem of the first is that an CPU easily borrow enough runtime to
make the spin-rt-task to run forever, allowing the starvation of the
non-rt-tasks, hence invalidating the mechanism.

The problem of the second is that (with the default values) the CPU will
be idle 5% of the time.

IMHO, the balanced behavior is using GREED option + without
RT_RUNTIME_SHARING: the non-rt tasks will be able to run, while avoiding
CPU going idle.

We can turn it by default setting default options.

Moreover, AFAICS, these sched options are static keys, so they are very
very low overhead conditions.

-- Daniel

[toc] | [prev] | [next] | [standalone]

#1516481

[Multipart message — attachments visible in raw view] — view raw

On Mon, 7 Nov 2016 13:30:46 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Mon, 7 Nov 2016 12:22:21 -0600
> Clark Williams <williams@redhat.com> wrote:
> 
> > I'm still reviewing the patch, but I have to wonder why bother with making it a scheduler feature?
> > 
> > The SCHED_FIFO definition allows a fifo thread to starve others
> > because a fifo task will run until it yields. Throttling was added as
> > a safety valve to allow starved SCHED_OTHER tasks to get some cpu
> > time.  Adding this unconditionally gets us a safety valve for
> > throttling a badly written fifo task, but allows the fifo task to
> > continue to consume cpu cycles if it's not starving anyone. 
> > 
> > Or am I missing something that's blazingly obvious?  
> 
> Or I say make it the default. If people want the old behavior, they can
> modify SCHED_FEATURES to do so.
> 

Ok, I can see wanting the previous behavior. 

Clark

[toc] | [prev] | [next] | [standalone]

#1517127

No, none of this stands a chance of being accepted.

This is making bad code worse.

[toc] | [prev] | [next] | [standalone]

#1517227

On Tue, 8 Nov 2016 12:59:58 +0100
Peter Zijlstra <peterz@infradead.org> wrote:

> No, none of this stands a chance of being accepted.
> 
> This is making bad code worse.

Peter,

Instead of a flat out rejection, can you please provide some
constructive criticism to let those that are working on this know what
would be accepted? And what their next steps should be.

There's obviously a problem with the current code, what steps do you
recommend to fix it?

-- Steve

[toc] | [prev] | [next] | [standalone]

#1517385

On Tue, Nov 08, 2016 at 09:07:40AM -0500, Steven Rostedt wrote:
> On Tue, 8 Nov 2016 12:59:58 +0100
> Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > No, none of this stands a chance of being accepted.
> > 
> > This is making bad code worse.
> 
> Peter,
> 
> Instead of a flat out rejection, can you please provide some
> constructive criticism to let those that are working on this know what
> would be accepted? And what their next steps should be.
> 
> There's obviously a problem with the current code, what steps do you
> recommend to fix it?

You really should already know this.

As stands the current rt cgroup code (and all this throttling code) is a
giant mess (as in, its not actually correct from a RT pov). We should
not make it worse by adding random hacks to it.

The right way to to about doing this is by replacing it with something
better; like the proposed DL server for FIFO tasks -- which is entirely
non-trivial as well, see existing discussion on that.

I'm not entirely sure what this patch was supposed to fix, but it could
be running CFS tasks with higher priority than RT for a window, instead
of throttling RT tasks. This seems fairly ill specified, but something
like that could easily done with an explicit or slack time DL server for
CFS tasks.

[toc] | [prev] | [next] | [standalone]

#1517401

On Tue, 8 Nov 2016 17:51:33 +0100
Peter Zijlstra <peterz@infradead.org> wrote:

> You really should already know this.

I know what we want to do, but there's some momentous problems that
need to be solved first. Until then, we may be forced to continue with
hacks.

> 
> As stands the current rt cgroup code (and all this throttling code) is a
> giant mess (as in, its not actually correct from a RT pov). We should
> not make it worse by adding random hacks to it.
> 
> The right way to to about doing this is by replacing it with something
> better; like the proposed DL server for FIFO tasks -- which is entirely
> non-trivial as well, see existing discussion on that.

Right. The biggest issue that I see is how to assign affinities to
FIFO tasks and use a DL server to keep them from starving other tasks?

> 
> I'm not entirely sure what this patch was supposed to fix, but it could
> be running CFS tasks with higher priority than RT for a window, instead

I'm a bit confused with the above sentence. Do you mean that this patch
causes CFS tasks to run for a period with a higher priority than RT?
Well, currently we have the both CFS tasks and the "idle" task run
higher than RT, but this patch changes that to be just CFS tasks.

> of throttling RT tasks. This seems fairly ill specified, but something
> like that could easily done with an explicit or slack time DL server for
> CFS tasks.

If we can have a DL scheduler that can handle arbitrary affinities,
then all could be solved with that.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1517439

On Tue, Nov 08, 2016 at 12:17:10PM -0500, Steven Rostedt wrote:
> On Tue, 8 Nov 2016 17:51:33 +0100
> Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > You really should already know this.
> 
> I know what we want to do, but there's some momentous problems that
> need to be solved first.

Like what?

> Until then, we may be forced to continue with
> hacks.

Well, the more ill specified hacks we put in, the harder if will be to
replace because people will end up depending on it.

[toc] | [prev] | [next] | [standalone]

#1517520

On 11/08/2016 07:05 PM, Peter Zijlstra wrote:
>> > 
>> > I know what we want to do, but there's some momentous problems that
>> > need to be solved first.
> Like what?

The problem is that using RT_RUNTIME_SHARE a CPU will almost always
borrow enough runtime to make a CPU intensive rt task to run forever...
well not forever, but until the system crash because a kworker starved
in this CPU. Kworkers are sched fair by design and users do not always
have a way to avoid them in an isolated CPU, for example.

The user then can disable RT_RUNTIME_SHARE, but then the user will have
the CPU going idle for (period - runtime) at each period... throwing CPU
time in the trash.

>> > Until then, we may be forced to continue with
>> > hacks.
> Well, the more ill specified hacks we put in, the harder if will be to
> replace because people will end up depending on it.

The proposed patch seems to be the expected behavior for users/rt
throttling - they want a safeguard for fair tasks while allowing -rt
tasks to run as much as possible.

I see (and completely agree) that a DL server for fair/rt task would be
the best way to deal with this problem, but it will take some time until
such solution :-(. We even discussed this at Retis today, but yeah, it
will take sometime even in the best case.

(thinking aloud... a DL Server would react like the proposed patch, in
the sense that it would not be activated without tasks to run and would
return the CPU for other tasks if the tasks inside the server finish
their job before the end of the DL server runtime...)

-- Daniel

[toc] | [prev] | [next] | [standalone]

#1517534

On Tue, Nov 08, 2016 at 08:29:49PM +0100, Daniel Bristot de Oliveira wrote:
> 
> 
> On 11/08/2016 07:05 PM, Peter Zijlstra wrote:
> >> > 
> >> > I know what we want to do, but there's some momentous problems that
> >> > need to be solved first.
> > Like what?
> 
> The problem is that using RT_RUNTIME_SHARE a CPU will almost always
> borrow enough runtime to make a CPU intensive rt task to run forever...
> well not forever, but until the system crash because a kworker starved
> in this CPU. Kworkers are sched fair by design and users do not always
> have a way to avoid them in an isolated CPU, for example.
> 
> The user then can disable RT_RUNTIME_SHARE, but then the user will have
> the CPU going idle for (period - runtime) at each period... throwing CPU
> time in the trash.

So why is this a problem? You really should not be running that much
FIFO tasks to begin with.

So I'm willing to take out (or at least default disable
RT_RUNTIME_SHARE). But other than this, this never really worked to
begin with. So it cannot be a regression. And we've lived this long with
the 'problem'.

And that means this is a 'feature' and that means I say no.

We really should be doing the right thing here, not make a bigger mess.

[toc] | [prev] | [next] | [standalone]

#1518171

On 11/08/2016 08:50 PM, Peter Zijlstra wrote:
>> The problem is that using RT_RUNTIME_SHARE a CPU will almost always
>> > borrow enough runtime to make a CPU intensive rt task to run forever...
>> > well not forever, but until the system crash because a kworker starved
>> > in this CPU. Kworkers are sched fair by design and users do not always
>> > have a way to avoid them in an isolated CPU, for example.
>> > 
>> > The user then can disable RT_RUNTIME_SHARE, but then the user will have
>> > the CPU going idle for (period - runtime) at each period... throwing CPU
>> > time in the trash.
> So why is this a problem? You really should not be running that much
> FIFO tasks to begin with.

I agree that a spinning real-time task is not a good practice, but there
are people using it and they have their own reasons/metrics/evaluations
motivating them.

> So I'm willing to take out (or at least default disable
> RT_RUNTIME_SHARE). But other than this, this never really worked to
> begin with. So it cannot be a regression. And we've lived this long with
> the 'problem'.

I agree! It would work perfectly in the absence of tasks pinned to a
processor, but that is not the case.

Trying to attend the users that want as much CPU time for -rt tasks as
possible, the proposed patch seems to be a better solution when compared
to RT_RUNTIME_SHARE, and it is way simpler! Even though it is not as
perfect as a DL Server would be in the future, it seems to be useful
until there...

> We really should be doing the right thing here, not make a bigger mess.

I see, agree and I am anxious to have it! :-). Tommaso and I discussed
about DL servers implementing such rt throttling. The more complicated
point so far (as Rostedt pointed on another e-mail) will be to have DL
servers with arbitrary affinity, or serving task with arbitrary
affinity. For example, one DL server pinned to each CPU providing
bandwidth for fair tasks to run for (rt_period - rt_runtime)us at each
rt_period... it will take sometime until someone propose it.

-- Daniel

[toc] | [prev] | [next] | [standalone]

#1519962

On Thu, 10 Nov 2016, Daniel Vacek wrote:

> I believe Daniel's patches are the best thing we can do in current
> situation as the behavior now seems rather buggy and does not provide above
> mentioned expectations set when rt throttling was merged with default
> budget of 95% of CPU time. Nor if you configure so that it does (by
> disabling RT_RUNTIME_SHARE), it also forces CPUs to go idle needlessly when
> there still can be rt task running not really starving anyone. At least
> till a proper rework of rt scheduling with DL Server is implemented.

This looks like a fix for a bug and the company I work for is suffering
as a result. Could we please merge that ASAP?

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web