Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1587045 > unrolled thread

Re: [patch 0/3] KVM CPU frequency change hypercalls

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [patch 0/3] KVM CPU frequency change hypercalls Paolo Bonzini <pbonzini@redhat.com> - 2017-02-23 18:50 +0100
    Re: [patch 0/3] KVM CPU frequency change hypercalls Marcelo Tosatti <mtosatti@redhat.com> - 2017-02-24 02:20 +0100
      Re: [patch 0/3] KVM CPU frequency change hypercalls Paolo Bonzini <pbonzini@redhat.com> - 2017-02-24 10:20 +0100
        Re: [patch 0/3] KVM CPU frequency change hypercalls Marcelo Tosatti <mtosatti@redhat.com> - 2017-02-24 13:00 +0100
          Re: [patch 0/3] KVM CPU frequency change hypercalls Paolo Bonzini <pbonzini@redhat.com> - 2017-02-24 13:20 +0100
            Re: [patch 0/3] KVM CPU frequency change hypercalls Marcelo Tosatti <mtosatti@redhat.com> - 2017-02-24 14:10 +0100
              Re: [patch 0/3] KVM CPU frequency change hypercalls Paolo Bonzini <pbonzini@redhat.com> - 2017-02-24 16:40 +0100
                Re: [patch 0/3] KVM CPU frequency change hypercalls "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2017-02-24 18:10 +0100
                Re: [patch 0/3] KVM CPU frequency change hypercalls Marcelo Tosatti <mtosatti@redhat.com> - 2017-02-28 22:20 +0100
                  Re: [patch 0/3] KVM CPU frequency change hypercalls Paolo Bonzini <pbonzini@redhat.com> - 2017-03-01 15:40 +0100
                    Re: [patch 0/3] KVM CPU frequency change hypercalls Marcelo Tosatti <mtosatti@redhat.com> - 2017-03-01 16:30 +0100

#1587045 — Re: [patch 0/3] KVM CPU frequency change hypercalls

On 03/02/2017 20:09, Radim Krcmar wrote:
> One reason why we have a kernel/userspace split is to allow sharing of
> CPU time.  Each application then its state that the kernel keeps track
> of and saves/restores while time-multiplexing.
> 
> Our frequency scaling interface goes against the idea -- guest kernel
> cannot schedule multiple userspaces on the same vCPU, because they could
> conflict by overriding frequency.
> 
> i.e. our feature implies userspace tasks pinned to isolated vCPUs.

That's bad.  This feature is broken by design unless it does proper
save/restore across preemption.

You don't need a hypercall.  Add a cpufreq driver in DPDK that doesn't
use sysfs, and connect it to a daemon in the host through virtio-serial
or vsock.

Paolo

[toc] | [next] | [standalone]

#1587222

On Thu, Feb 23, 2017 at 06:35:24PM +0100, Paolo Bonzini wrote:
> 
> 
> On 03/02/2017 20:09, Radim Krcmar wrote:
> > One reason why we have a kernel/userspace split is to allow sharing of
> > CPU time.  Each application then its state that the kernel keeps track
> > of and saves/restores while time-multiplexing.
> > 
> > Our frequency scaling interface goes against the idea -- guest kernel
> > cannot schedule multiple userspaces on the same vCPU, because they could
> > conflict by overriding frequency.
> > 
> > i.e. our feature implies userspace tasks pinned to isolated vCPUs.

This is how cpufreq-userspace works:

2.2 Governor
------------

On all other cpufreq implementations, these boundaries still need to
be set. Then, a "governor" must be selected. Such a "governor" decides
what speed the processor shall run within the boundaries. One such
"governor" is the "userspace" governor. This one allows the user - or
a yet-to-implement userspace program - to decide what specific speed
the processor shall run at.

> That's bad.  This feature is broken by design unless it does proper
> save/restore across preemption.

Whats the current usecase, or forseeable future usecase, for save/restore
across preemption again? (which would validate the broken by design
claim).

> You don't need a hypercall.  Add a cpufreq driver in DPDK that doesn't
> use sysfs, and connect it to a daemon in the host through virtio-serial
> or vsock.
> 
> Paolo

Hypercalls overcome the problems mentioned in the first email
of the thread, i think you missed them:

"[patch 0/3] KVM CPU frequency change hypercalls"

[toc] | [prev] | [next] | [standalone]

#1587441

On 24/02/2017 00:19, Marcelo Tosatti wrote:
>>> i.e. our feature implies userspace tasks pinned to isolated vCPUs.
> This is how cpufreq-userspace works:
> 
> 2.2 Governor
> ------------
> 
> On all other cpufreq implementations, these boundaries still need to
> be set. Then, a "governor" must be selected. Such a "governor" decides
> what speed the processor shall run within the boundaries. One such
> "governor" is the "userspace" governor. This one allows the user - or
> a yet-to-implement userspace program - to decide what specific speed
> the processor shall run at.

The userspace program sets a policy for the whole system.

>> That's bad.  This feature is broken by design unless it does proper
>> save/restore across preemption.
> 
> Whats the current usecase, or forseeable future usecase, for save/restore
> across preemption again? (which would validate the broken by design
> claim).

Stop a guest that is using cpufreq, start a guest that is not using it.
The second guest's performance now depends on the state that the first
guest left in cpufreq.

I think this is abusing the userspace governor.  Unfortunately cpufreq
governors cannot be stacked.

Paolo

[toc] | [prev] | [next] | [standalone]

#1587545

On Fri, Feb 24, 2017 at 10:18:59AM +0100, Paolo Bonzini wrote:
> 
> 
> On 24/02/2017 00:19, Marcelo Tosatti wrote:
> >>> i.e. our feature implies userspace tasks pinned to isolated vCPUs.
> > This is how cpufreq-userspace works:
> > 
> > 2.2 Governor
> > ------------
> > 
> > On all other cpufreq implementations, these boundaries still need to
> > be set. Then, a "governor" must be selected. Such a "governor" decides
> > what speed the processor shall run within the boundaries. One such
> > "governor" is the "userspace" governor. This one allows the user - or
> > a yet-to-implement userspace program - to decide what specific speed
> > the processor shall run at.
> 
> The userspace program sets a policy for the whole system.

No, its per cpu.

> >> That's bad.  This feature is broken by design unless it does proper
> >> save/restore across preemption.
> > 
> > Whats the current usecase, or forseeable future usecase, for save/restore
> > across preemption again? (which would validate the broken by design
> > claim).
> 
> Stop a guest that is using cpufreq, start a guest that is not using it.
> The second guest's performance now depends on the state that the first
> guest left in cpufreq.

Nothing forbids the host to implement switching with the
current hypercall interface: all you need is a scheduler
hook.

> I think this is abusing the userspace governor.  Unfortunately cpufreq
> governors cannot be stacked.
> 
> Paolo

This is a special usecase where only the app in the guest knows 
whats the most appropriate frequency at a given time.
This is what cpufreq-userspace is supposed to allow userspace to do, 
but in this case "userspace" is the guest, so i don't 
see this as an abuse at all.

Timeshared setups are by definition not deterministic: 
your task A could be interrupted by another task B 
with results similar to a lower frequency being set.

So saying that:

"Our frequency scaling interface goes against the idea -- guest kernel
 cannot schedule multiple userspaces on the same vCPU, because they
 could
 conflict by overriding frequency."

Assumes that, in a timeshared system, an application is guaranteed a
particular frequency. But that does not make sense: its a timeshared
system in the first place, there is no determinism regarding execution
time.

Moreover, there is no notion of "per-task CPU frequency" in Linux
(there could be, this whole governor business with user
being responsible for setting up the governor is pretty sucky
IMO).

[toc] | [prev] | [next] | [standalone]

#1587552

On 24/02/2017 12:50, Marcelo Tosatti wrote:
>>>
>>> On all other cpufreq implementations, these boundaries still need to
>>> be set. Then, a "governor" must be selected. Such a "governor" decides
>>> what speed the processor shall run within the boundaries. One such
>>> "governor" is the "userspace" governor. This one allows the user - or
>>> a yet-to-implement userspace program - to decide what specific speed
>>> the processor shall run at.
>> The userspace program sets a policy for the whole system.
> No, its per cpu.

Yeah, what I mean is that userspace program can be per-CPU, but it looks
at all the processes running on that CPU ("the whole system").  This is
very different from a guest, which is isolated.

>>>> That's bad.  This feature is broken by design unless it does proper
>>>> save/restore across preemption.
>>> Whats the current usecase, or forseeable future usecase, for save/restore
>>> across preemption again? (which would validate the broken by design
>>> claim).
>> Stop a guest that is using cpufreq, start a guest that is not using it.
>> The second guest's performance now depends on the state that the first
>> guest left in cpufreq.
>
> Nothing forbids the host to implement switching with the
> current hypercall interface: all you need is a scheduler
> hook.

Can it be done in vcpu_load/vcpu_put?  But you still would have two
components (KVM and sysfs) potentially fighting over the frequency, and
that's still a bit ugly.

Paolo

[toc] | [prev] | [next] | [standalone]

#1587599

On Fri, Feb 24, 2017 at 01:17:07PM +0100, Paolo Bonzini wrote:
> 
> 
> On 24/02/2017 12:50, Marcelo Tosatti wrote:
> >>>
> >>> On all other cpufreq implementations, these boundaries still need to
> >>> be set. Then, a "governor" must be selected. Such a "governor" decides
> >>> what speed the processor shall run within the boundaries. One such
> >>> "governor" is the "userspace" governor. This one allows the user - or
> >>> a yet-to-implement userspace program - to decide what specific speed
> >>> the processor shall run at.
> >> The userspace program sets a policy for the whole system.
> > No, its per cpu.
> 
> Yeah, what I mean is that userspace program can be per-CPU, but it looks
> at all the processes running on that CPU ("the whole system").  This is
> very different from a guest, which is isolated.
> 
> >>>> That's bad.  This feature is broken by design unless it does proper
> >>>> save/restore across preemption.
> >>> Whats the current usecase, or forseeable future usecase, for save/restore
> >>> across preemption again? (which would validate the broken by design
> >>> claim).
> >> Stop a guest that is using cpufreq, start a guest that is not using it.
> >> The second guest's performance now depends on the state that the first
> >> guest left in cpufreq.
> >
> > Nothing forbids the host to implement switching with the
> > current hypercall interface: all you need is a scheduler
> > hook.
> 
> Can it be done in vcpu_load/vcpu_put?  But you still would have two
> components (KVM and sysfs) potentially fighting over the frequency, and
> that's still a bit ugly.
> 
> Paolo

Change the frequency at vcpu_load/vcpu_put? Yes: call into
cpufreq-userspace. But there is no notion of "per-task frequency" on the
Linux kernel (which was the starting point of this subthread).
 
But if you configure all CPUs in the system as cpufreq-userspace,
then some other (userspace program) has to decide the frequency
for the other CPUs.

Which agent would do that and why? Thats why i initially said "whats the
usecase".

[toc] | [prev] | [next] | [standalone]

#1587724

On 24/02/2017 14:04, Marcelo Tosatti wrote:
>>>>> Whats the current usecase, or forseeable future usecase, for save/restore
>>>>> across preemption again? (which would validate the broken by design
>>>>> claim).
>>>> Stop a guest that is using cpufreq, start a guest that is not using it.
>>>> The second guest's performance now depends on the state that the first
>>>> guest left in cpufreq.
>>> Nothing forbids the host to implement switching with the
>>> current hypercall interface: all you need is a scheduler
>>> hook.
>> Can it be done in vcpu_load/vcpu_put?  But you still would have two
>> components (KVM and sysfs) potentially fighting over the frequency, and
>> that's still a bit ugly.
>
> Change the frequency at vcpu_load/vcpu_put? Yes: call into
> cpufreq-userspace. But there is no notion of "per-task frequency" on the
> Linux kernel (which was the starting point of this subthread).

There isn't, but this patchset is providing a direct path from a task to
cpufreq-userspace.  This is as close as you can get to a per-task frequency.

> But if you configure all CPUs in the system as cpufreq-userspace,
> then some other (userspace program) has to decide the frequency
> for the other CPUs.
> 
> Which agent would do that and why? Thats why i initially said "whats the
> usecase".

You could just pin them at the highest non-TurboBoost frequency until a
guest runs.  That's assuming that they are idle and, because of
isol_cpus/nohz_full, they would be almost always in deep C state anyway.

Paolo

[toc] | [prev] | [next] | [standalone]

#1587809

On Friday, February 24, 2017 04:34:52 PM Paolo Bonzini wrote:
> 
> On 24/02/2017 14:04, Marcelo Tosatti wrote:
> >>>>> Whats the current usecase, or forseeable future usecase, for save/restore
> >>>>> across preemption again? (which would validate the broken by design
> >>>>> claim).
> >>>> Stop a guest that is using cpufreq, start a guest that is not using it.
> >>>> The second guest's performance now depends on the state that the first
> >>>> guest left in cpufreq.
> >>> Nothing forbids the host to implement switching with the
> >>> current hypercall interface: all you need is a scheduler
> >>> hook.
> >> Can it be done in vcpu_load/vcpu_put?  But you still would have two
> >> components (KVM and sysfs) potentially fighting over the frequency, and
> >> that's still a bit ugly.
> >
> > Change the frequency at vcpu_load/vcpu_put? Yes: call into
> > cpufreq-userspace. But there is no notion of "per-task frequency" on the
> > Linux kernel (which was the starting point of this subthread).
> 
> There isn't, but this patchset is providing a direct path from a task to
> cpufreq-userspace.  This is as close as you can get to a per-task frequency.
> 
> > But if you configure all CPUs in the system as cpufreq-userspace,
> > then some other (userspace program) has to decide the frequency
> > for the other CPUs.
> > 
> > Which agent would do that and why? Thats why i initially said "whats the
> > usecase".
> 
> You could just pin them at the highest non-TurboBoost frequency until a
> guest runs.  That's assuming that they are idle and, because of
> isol_cpus/nohz_full, they would be almost always in deep C state anyway.

Good discussion so far, but it should be happening on the linux-pm list.

Would it be possible to repost the patches with a CC to linux-pm?

Thanks,
Rafael

[toc] | [prev] | [next] | [standalone]

#1589790

On Fri, Feb 24, 2017 at 04:34:52PM +0100, Paolo Bonzini wrote:
> 
> 
> On 24/02/2017 14:04, Marcelo Tosatti wrote:
> >>>>> Whats the current usecase, or forseeable future usecase, for save/restore
> >>>>> across preemption again? (which would validate the broken by design
> >>>>> claim).
> >>>> Stop a guest that is using cpufreq, start a guest that is not using it.
> >>>> The second guest's performance now depends on the state that the first
> >>>> guest left in cpufreq.
> >>> Nothing forbids the host to implement switching with the
> >>> current hypercall interface: all you need is a scheduler
> >>> hook.
> >> Can it be done in vcpu_load/vcpu_put?  But you still would have two
> >> components (KVM and sysfs) potentially fighting over the frequency, and
> >> that's still a bit ugly.
> >
> > Change the frequency at vcpu_load/vcpu_put? Yes: call into
> > cpufreq-userspace. But there is no notion of "per-task frequency" on the
> > Linux kernel (which was the starting point of this subthread).
> 
> There isn't, but this patchset is providing a direct path from a task to
> cpufreq-userspace.  This is as close as you can get to a per-task frequency.

Cpufreq-userspace is supposed to be used by tasks in userspace.
Thats why its called "userspace".

> > But if you configure all CPUs in the system as cpufreq-userspace,
> > then some other (userspace program) has to decide the frequency
> > for the other CPUs.
> > 
> > Which agent would do that and why? Thats why i initially said "whats the
> > usecase".
> 
> You could just pin them at the highest non-TurboBoost frequency until a
> guest runs.  That's assuming that they are idle and, because of
> isol_cpus/nohz_full, they would be almost always in deep C state anyway.
> 
> Paolo

The original claim of the thread  was: "this feature (frequency
hypercalls) works for pinned vcpu<->pcpu, pcpu dedicated exclusively
to vcpu case, lets try to extend this to other cases".

Which is a valid and useful direction to go.

However there is no user for multiple vcpus in the same pcpu now.

If there were multiple vcpus, all of them requesting a given
frequency, it would be necessary to:

	1) Maintain frequency of the pcpu to the highest 
	   frequencies.

		OR

	2) Since switching frequencies can take up to 70us (*)
	   (depends on processor), its generally not worthwhile
	   to switch frequencies between task switches.

So its a dead end...

*: http://www.ena-hpc.org/2013/pdf/04.pdf

[toc] | [prev] | [next] | [standalone]

#1590359

On 28/02/2017 03:45, Marcelo Tosatti wrote:
> On Fri, Feb 24, 2017 at 04:34:52PM +0100, Paolo Bonzini wrote:
>>
>>
>> On 24/02/2017 14:04, Marcelo Tosatti wrote:
>>>>>>> Whats the current usecase, or forseeable future usecase, for save/restore
>>>>>>> across preemption again? (which would validate the broken by design
>>>>>>> claim).
>>>>>> Stop a guest that is using cpufreq, start a guest that is not using it.
>>>>>> The second guest's performance now depends on the state that the first
>>>>>> guest left in cpufreq.
>>>>> Nothing forbids the host to implement switching with the
>>>>> current hypercall interface: all you need is a scheduler
>>>>> hook.
>>>> Can it be done in vcpu_load/vcpu_put?  But you still would have two
>>>> components (KVM and sysfs) potentially fighting over the frequency, and
>>>> that's still a bit ugly.
>>>
>>> Change the frequency at vcpu_load/vcpu_put? Yes: call into
>>> cpufreq-userspace. But there is no notion of "per-task frequency" on the
>>> Linux kernel (which was the starting point of this subthread).
>>
>> There isn't, but this patchset is providing a direct path from a task to
>> cpufreq-userspace.  This is as close as you can get to a per-task frequency.
> 
> Cpufreq-userspace is supposed to be used by tasks in userspace.
> Thats why its called "userspace".

I think the intended usecase is to have a daemon handling a systemwide
policy.  Examples are the historical (and now obsolete) users such as
cpufreqd, cpudyn, powernowd, or cpuspeed.  The user alternatively can
play the role of the daemon by writing to sysfs.

I've never seen userspace tasks talking to cpufreq-userspace to set
their own running frequency.  If DPDK does it, that's nasty in my
opinion and we should find an interface that works best for both DPDK
and KVM.  Which should be done on linux-pm like Rafael suggested.

>>> But if you configure all CPUs in the system as cpufreq-userspace,
>>> then some other (userspace program) has to decide the frequency
>>> for the other CPUs.
>>>
>>> Which agent would do that and why? Thats why i initially said "whats the
>>> usecase".
>>
>> You could just pin them at the highest non-TurboBoost frequency until a
>> guest runs.  That's assuming that they are idle and, because of
>> isol_cpus/nohz_full, they would be almost always in deep C state anyway.
> 
> The original claim of the thread  was: "this feature (frequency
> hypercalls) works for pinned vcpu<->pcpu, pcpu dedicated exclusively
> to vcpu case, lets try to extend this to other cases".
> 
> Which is a valid and useful direction to go.
> 
> However there is no user for multiple vcpus in the same pcpu now.

You are still ignoring the case of one guest started after another, or
of another program started on a CPU that formerly was used by KVM.  They
don't have to be multiple users at the same time.

> If there were multiple vcpus, all of them requesting a given
> frequency, it would be necessary to:
> 
> 	1) Maintain frequency of the pcpu to the highest 
> 	   frequencies.
> 
> 		OR
> 
> 	2) Since switching frequencies can take up to 70us (*)
> 	   (depends on processor), its generally not worthwhile
> 	   to switch frequencies between task switches.

Is latency that important, or is rather overhead the one to pay
attention to?  The slides you linked
(http://www.ena-hpc.org/2013/pdf/04.pdf) at page 17 suggest it's around
10us.

One possibility is to do (1) if you have multiple tasks on the run queue
(or fallback to what is specified in sysfs) and (2) if you only have one
task.

Anyway, please repost with Cc to linux-pm so that we can restart the
discussion there.

Paolo

[toc] | [prev] | [next] | [standalone]

#1590411

On Wed, Mar 01, 2017 at 03:21:32PM +0100, Paolo Bonzini wrote:
> 
> 
> On 28/02/2017 03:45, Marcelo Tosatti wrote:
> > On Fri, Feb 24, 2017 at 04:34:52PM +0100, Paolo Bonzini wrote:
> >>
> >>
> >> On 24/02/2017 14:04, Marcelo Tosatti wrote:
> >>>>>>> Whats the current usecase, or forseeable future usecase, for save/restore
> >>>>>>> across preemption again? (which would validate the broken by design
> >>>>>>> claim).
> >>>>>> Stop a guest that is using cpufreq, start a guest that is not using it.
> >>>>>> The second guest's performance now depends on the state that the first
> >>>>>> guest left in cpufreq.
> >>>>> Nothing forbids the host to implement switching with the
> >>>>> current hypercall interface: all you need is a scheduler
> >>>>> hook.
> >>>> Can it be done in vcpu_load/vcpu_put?  But you still would have two
> >>>> components (KVM and sysfs) potentially fighting over the frequency, and
> >>>> that's still a bit ugly.
> >>>
> >>> Change the frequency at vcpu_load/vcpu_put? Yes: call into
> >>> cpufreq-userspace. But there is no notion of "per-task frequency" on the
> >>> Linux kernel (which was the starting point of this subthread).
> >>
> >> There isn't, but this patchset is providing a direct path from a task to
> >> cpufreq-userspace.  This is as close as you can get to a per-task frequency.
> > 
> > Cpufreq-userspace is supposed to be used by tasks in userspace.
> > Thats why its called "userspace".
> 
> I think the intended usecase is to have a daemon handling a systemwide
> policy.  Examples are the historical (and now obsolete) users such as
> cpufreqd, cpudyn, powernowd, or cpuspeed.  The user alternatively can
> play the role of the daemon by writing to sysfs.
> 
> I've never seen userspace tasks talking to cpufreq-userspace to set
> their own running frequency.  If DPDK does it, that's nasty in my
> opinion

Please extend what "nasty" means in detail. I really don't understand
why its nasty.

>  and we should find an interface that works best for both DPDK
> and KVM.  Which should be done on linux-pm like Rafael suggested.
> 
> >>> But if you configure all CPUs in the system as cpufreq-userspace,
> >>> then some other (userspace program) has to decide the frequency
> >>> for the other CPUs.
> >>>
> >>> Which agent would do that and why? Thats why i initially said "whats the
> >>> usecase".
> >>
> >> You could just pin them at the highest non-TurboBoost frequency until a
> >> guest runs.  That's assuming that they are idle and, because of
> >> isol_cpus/nohz_full, they would be almost always in deep C state anyway.
> > 
> > The original claim of the thread  was: "this feature (frequency
> > hypercalls) works for pinned vcpu<->pcpu, pcpu dedicated exclusively
> > to vcpu case, lets try to extend this to other cases".
> > 
> > Which is a valid and useful direction to go.
> > 
> > However there is no user for multiple vcpus in the same pcpu now.
> 
> You are still ignoring the case of one guest started after another, or
> of another program started on a CPU that formerly was used by KVM.  They
> don't have to be multiple users at the same time.

Just have the cpufreq-userspace policy be instantiated while the 
isolated vcpu owns the pcpu. Before/after that, the previous policy 
is in place. 

> > If there were multiple vcpus, all of them requesting a given
> > frequency, it would be necessary to:
> > 
> > 	1) Maintain frequency of the pcpu to the highest 
> > 	   frequencies.
> > 
> > 		OR
> > 
> > 	2) Since switching frequencies can take up to 70us (*)
> > 	   (depends on processor), its generally not worthwhile
> > 	   to switch frequencies between task switches.
> 
> Is latency that important, or is rather overhead the one to pay
> attention to?  The slides you linked
> (http://www.ena-hpc.org/2013/pdf/04.pdf) at page 17 suggest it's around
> 10us.

Ok, be it 10us. 10us overhead on every task context switch is not
acceptable.

> One possibility is to do (1) if you have multiple tasks on the run queue
> (or fallback to what is specified in sysfs) and (2) if you only have one
> task.

Sure, that is alright. But the use-case at hand does not involve 
multiple tasks on the pcpu.

> Anyway, please repost with Cc to linux-pm so that we can restart the
> discussion there.
> 
> Paolo

Done. Can you please reply with a concise summary of what you object to? 

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web