Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait condition

From	"Rafael J. Wysocki" <rjw@rjwysocki.net>
Newsgroups	linux.kernel
Subject	Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait condition
Date	2016-08-04 23:20 +0200
Message-ID	<s2ydA-6EW-33@gated-at.bofh.it> (permalink)
References	<s18Ex-6NS-1@gated-at.bofh.it> <s1QFz-2lk-15@gated-at.bofh.it> <s2gA1-2HH-5@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On Wednesday, August 03, 2016 07:24:18 PM Steve Muckle wrote:
> On Wed, Aug 03, 2016 at 12:38:20AM +0200, Rafael J. Wysocki wrote:
> > On Wed, Aug 3, 2016 at 12:02 AM, Steve Muckle <steve.muckle@linaro.org> wrote:
> > > On Tue, Aug 02, 2016 at 03:37:02AM +0200, Rafael J. Wysocki wrote:
> > >> On Tue, Aug 2, 2016 at 3:22 AM, Steve Muckle <steve.muckle@linaro.org> wrote:
> > >> > On Mon, Aug 01, 2016 at 01:37:23AM +0200, Rafael J. Wysocki wrote:
> > >> > ...
> > >> >> For this purpose, define a new cpufreq_update_util() flag
> > >> >> UUF_IO and modify enqueue_task_fair() to pass that flag to
> > >> >> cpufreq_update_util() in the in_iowait case.  That generally
> > >> >> requires cpufreq_update_util() to be called directly from there,
> > >> >> because update_load_avg() is not likely to be invoked in that
> > >> >> case.
> > >> >
> > >> > I didn't follow why the cpufreq hook won't likely be called if
> > >> > in_iowait is set? AFAICS update_load_avg() gets called in the second loop
> > >> > and calls update_cfs_rq_load_avg (triggers the hook).
> > >>
> > >> In practice it turns out that in the majority of cases when in_iowait
> > >> is set the second loop will not run.
> > >
> > > My understanding of enqueue_task_fair() is that the first loop walks up
> > > the portion of the sched_entity hierarchy that needs to be enqueued, and
> > > the second loop updates the rest of the hierarchy that was already
> > > enqueued.
> > >
> > > Even if the se corresponding to the root cfs_rq needs to be enqueued
> > > (meaning the whole hierarchy is traversed in the first loop and the
> > > second loop does nothing), enqueue_entity() on the root cfs_rq should
> > > result in the cpufreq hook being called, via enqueue_entity() ->
> > > enqueue_entity_load_avg() -> update_cfs_rq_load_avg().
> > 
> > But then it's rather difficult to pass the IO flag to this one, isn't it?
> > 
> > Essentially, the problem is to pass "IO" to cpufreq_update_util() when
> > p->in_iowait is set.
> > 
> > If you can find a clever way to do it without adding an extra call
> > site, that's fine by me, but in any case the extra
> > cpufreq_update_util() invocation should not be too expensive.
> 
> I was under the impression that function pointer calls were more
> expensive, and in the shared policy case there is a nontrivial amount of
> code that is run in schedutil (including taking a spinlock) before we'd
> see sugov_should_update_freq() return false and bail.

That's correct in principle, but we only do that if p->in_iowait is set,
which is somewhat special anyway and doesn't happen every time for sure.

So while there is overhead theoretically, I'm not even sure if it is measurable.

> Agreed that getting knowledge of p->in_iowait down to the existing hook
> is not easy. I spent some time fiddling with that. It seemed doable but
> somewhat gross due to the required flag passing and modifications
> to enqueue_entity, update_load_avg, etc. If it is decided that it is worth
> pursuing I can keep working on it and post a draft.

Well, that's a Peter's call. :-)

> But I also wonder if the hooks are in the best location.  They are
> currently deep in the PELT code. This may make sense from a theoretical
> standpoint, calling them whenever a root cfs_rq utilization changes, but
> it also makes the hooks difficult to correlate (for policy purposes such
> as this iowait change) with higher level logical events like a task
> wakeup. Or load balance where we probably want to call the hook just
> once after a load balance is complete.

I generally agree.  We still need to ensure that the hools will be invoked
frequently enough, though, even if HZ is 100.

> This is also an issue for the remote wakeup case where I currently have
> another invocation of the hook in check_preempt_curr(), so I can know if
> preemption was triggered and skip a remote schedutil update in that case
> to avoid a duplicate IPI.
> 
> It seems to me worth evaluating if a higher level set of hook locations
> could be used. One possibility is higher up in CFS:
> - enqueue_task_fair, dequeue_task_fair
> - scheduler_tick
> - active_load_balance_cpu_stop, load_balance

Agreed, that's worth checking.

> Though this wouldn't solve my issue with check_preempt_curr. That would
> probably require going further up the stack to try_to_wake_up() etc. Not
> yet sure what the other hook locations would be at that level.

That's probably too far away from the root cfs_rq utilization changes IMO.

Thanks,
Rafael

Thread

[RFC][PATCH 0/7] cpufreq / sched: cpufreq_update_util() flags and iowait boosting "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 01:50 +0200
  [RFC][PATCH 4/7] cpufreq / sched: Add flags argument to cpufreq_update_util() "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 01:50 +0200
    Re: [RFC][PATCH 4/7] cpufreq / sched: Add flags argument to  cpufreq_update_util() Dominik Brodowski <linux@dominikbrodowski.net> - 2016-08-01 10:00 +0200
      Re: [RFC][PATCH 4/7] cpufreq / sched: Add flags argument to cpufreq_update_util() "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 17:10 +0200
        Re: [RFC][PATCH 4/7] cpufreq / sched: Add flags argument to  cpufreq_update_util() Steve Muckle <steve.muckle@linaro.org> - 2016-08-01 22:30 +0200
          Re: [RFC][PATCH 4/7] cpufreq / sched: Add flags argument to cpufreq_update_util() "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-02 01:50 +0200
            Re: [RFC][PATCH 4/7] cpufreq / sched: Add flags argument to  cpufreq_update_util() Steve Muckle <steve.muckle@linaro.org> - 2016-08-02 04:10 +0200
  [RFC][PATCH 3/7] cpufreq / sched: Check cpu_of(rq) in cpufreq_update_util() "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 01:50 +0200
    Re: [RFC][PATCH 3/7] cpufreq / sched: Check cpu_of(rq) in  cpufreq_update_util() Dominik Brodowski <linux@dominikbrodowski.net> - 2016-08-01 10:00 +0200
      Re: [RFC][PATCH 3/7] cpufreq / sched: Check cpu_of(rq) in cpufreq_update_util() "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 17:10 +0200
      Re: [RFC][PATCH 3/7] cpufreq / sched: Check cpu_of(rq) in  cpufreq_update_util() Steve Muckle <steve.muckle@linaro.org> - 2016-08-01 21:50 +0200
        Re: [RFC][PATCH 3/7] cpufreq / sched: Check cpu_of(rq) in cpufreq_update_util() "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-02 01:50 +0200
  [RFC][PATCH 1/7] cpufreq / sched: Make schedutil access utilization data directly "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 01:50 +0200
    Re: [RFC][PATCH 1/7] cpufreq / sched: Make schedutil access  utilization data directly Steve Muckle <steve.muckle@linaro.org> - 2016-08-01 21:40 +0200
      Re: [RFC][PATCH 1/7] cpufreq / sched: Make schedutil access utilization data directly "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-02 02:10 +0200
        Re: [RFC][PATCH 1/7] cpufreq / sched: Make schedutil access  utilization data directly Juri Lelli <juri.lelli@arm.com> - 2016-08-02 12:40 +0200
          Re: [RFC][PATCH 1/7] cpufreq / sched: Make schedutil access  utilization data directly Steve Muckle <steve.muckle@linaro.org> - 2016-08-02 16:40 +0200
            Re: [RFC][PATCH 1/7] cpufreq / sched: Make schedutil access  utilization data directly Juri Lelli <juri.lelli@arm.com> - 2016-08-02 17:00 +0200
        Re: [RFC][PATCH 1/7] cpufreq / sched: Make schedutil access  utilization data directly Peter Zijlstra <peterz@infradead.org> - 2016-08-08 12:40 +0200
  [RFC][PATCH 2/7] cpufreq / sched: Drop cpufreq_trigger_update() "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 01:50 +0200
  [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait condition "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 01:50 +0200
    Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait  condition Steve Muckle <steve.muckle@linaro.org> - 2016-08-02 03:30 +0200
      Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait condition "Rafael J. Wysocki" <rafael@kernel.org> - 2016-08-02 03:50 +0200
        Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait  condition Steve Muckle <steve.muckle@linaro.org> - 2016-08-03 00:30 +0200
          Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait condition "Rafael J. Wysocki" <rafael@kernel.org> - 2016-08-03 00:50 +0200
            Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait  condition Steve Muckle <steve.muckle@linaro.org> - 2016-08-04 04:30 +0200
              Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait condition "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-04 23:20 +0200
                Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait  condition Steve Muckle <steve.muckle@linaro.org> - 2016-08-05 00:10 +0200
                Re: [RFC][PATCH 5/7] cpufreq / sched: UUF_IO flag to indicate iowait condition "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-06 23:20 +0200
  [RFC][PATCH 7/7] cpufreq: intel_pstate: Change P-state selection algorithm for Core "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 01:50 +0200
    RE: [RFC][PATCH 7/7] cpufreq: intel_pstate: Change P-state selection algorithm for Core "Doug Smythies" <dsmythies@telus.net> - 2016-08-04 09:00 +0200
      Re: [RFC][PATCH 7/7] cpufreq: intel_pstate: Change P-state selection algorithm for Core "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-06 23:50 +0200
        RE: [RFC][PATCH 7/7] cpufreq: intel_pstate: Change P-state selection algorithm for Core "Doug Smythies" <dsmythies@telus.net> - 2016-08-09 19:20 +0200
  [RFC][PATCH 6/7] cpufreq: schedutil: Add iowait boosting "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 01:50 +0200
    Re: [RFC][PATCH 6/7] cpufreq: schedutil: Add iowait boosting Steve Muckle <steve.muckle@linaro.org> - 2016-08-02 03:40 +0200
      Re: [RFC][PATCH 6/7] cpufreq: schedutil: Add iowait boosting "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-03 01:00 +0200
  RE: [RFC][PATCH 0/7] cpufreq / sched: cpufreq_update_util() flags and iowait boosting "Doug Smythies" <dsmythies@telus.net> - 2016-08-01 17:30 +0200
    Re: [RFC][PATCH 0/7] cpufreq / sched: cpufreq_update_util() flags and iowait boosting "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-01 18:30 +0200
      Re: [RFC][PATCH 0/7] cpufreq / sched: cpufreq_update_util() flags  and iowait boosting Peter Zijlstra <peterz@infradead.org> - 2016-08-08 13:10 +0200
        Re: [RFC][PATCH 0/7] cpufreq / sched: cpufreq_update_util() flags and iowait boosting "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2016-08-08 15:00 +0200

csiph-web