Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1587045 > unrolled thread

Re: [patch 0/3] KVM CPU frequency change hypercalls

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [patch 0/3] KVM CPU frequency change hypercalls Paolo Bonzini <pbonzini@redhat.com> - 2017-02-23 18:50 +0100
    Re: [patch 0/3] KVM CPU frequency change hypercalls Marcelo Tosatti <mtosatti@redhat.com> - 2017-02-24 02:20 +0100
      Re: [patch 0/3] KVM CPU frequency change hypercalls Paolo Bonzini <pbonzini@redhat.com> - 2017-02-24 10:20 +0100
        Re: [patch 0/3] KVM CPU frequency change hypercalls Marcelo Tosatti <mtosatti@redhat.com> - 2017-02-24 13:00 +0100
          Re: [patch 0/3] KVM CPU frequency change hypercalls Paolo Bonzini <pbonzini@redhat.com> - 2017-02-24 13:20 +0100
            Re: [patch 0/3] KVM CPU frequency change hypercalls Marcelo Tosatti <mtosatti@redhat.com> - 2017-02-24 14:10 +0100
              Re: [patch 0/3] KVM CPU frequency change hypercalls Paolo Bonzini <pbonzini@redhat.com> - 2017-02-24 16:40 +0100
                Re: [patch 0/3] KVM CPU frequency change hypercalls "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2017-02-24 18:10 +0100
                Re: [patch 0/3] KVM CPU frequency change hypercalls Marcelo Tosatti <mtosatti@redhat.com> - 2017-02-28 22:20 +0100

#1587045 — Re: [patch 0/3] KVM CPU frequency change hypercalls

On 03/02/2017 20:09, Radim Krcmar wrote:
> One reason why we have a kernel/userspace split is to allow sharing of
> CPU time.  Each application then its state that the kernel keeps track
> of and saves/restores while time-multiplexing.
> 
> Our frequency scaling interface goes against the idea -- guest kernel
> cannot schedule multiple userspaces on the same vCPU, because they could
> conflict by overriding frequency.
> 
> i.e. our feature implies userspace tasks pinned to isolated vCPUs.

That's bad.  This feature is broken by design unless it does proper
save/restore across preemption.

You don't need a hypercall.  Add a cpufreq driver in DPDK that doesn't
use sysfs, and connect it to a daemon in the host through virtio-serial
or vsock.

Paolo

[toc] | [next] | [standalone]

#1587222

On Thu, Feb 23, 2017 at 06:35:24PM +0100, Paolo Bonzini wrote:
> 
> 
> On 03/02/2017 20:09, Radim Krcmar wrote:
> > One reason why we have a kernel/userspace split is to allow sharing of
> > CPU time.  Each application then its state that the kernel keeps track
> > of and saves/restores while time-multiplexing.
> > 
> > Our frequency scaling interface goes against the idea -- guest kernel
> > cannot schedule multiple userspaces on the same vCPU, because they could
> > conflict by overriding frequency.
> > 
> > i.e. our feature implies userspace tasks pinned to isolated vCPUs.

This is how cpufreq-userspace works:

2.2 Governor
------------

On all other cpufreq implementations, these boundaries still need to
be set. Then, a "governor" must be selected. Such a "governor" decides
what speed the processor shall run within the boundaries. One such
"governor" is the "userspace" governor. This one allows the user - or
a yet-to-implement userspace program - to decide what specific speed
the processor shall run at.

> That's bad.  This feature is broken by design unless it does proper
> save/restore across preemption.

Whats the current usecase, or forseeable future usecase, for save/restore
across preemption again? (which would validate the broken by design
claim).

> You don't need a hypercall.  Add a cpufreq driver in DPDK that doesn't
> use sysfs, and connect it to a daemon in the host through virtio-serial
> or vsock.
> 
> Paolo

Hypercalls overcome the problems mentioned in the first email
of the thread, i think you missed them:

"[patch 0/3] KVM CPU frequency change hypercalls"

[toc] | [prev] | [next] | [standalone]

#1587441

On 24/02/2017 00:19, Marcelo Tosatti wrote:
>>> i.e. our feature implies userspace tasks pinned to isolated vCPUs.
> This is how cpufreq-userspace works:
> 
> 2.2 Governor
> ------------
> 
> On all other cpufreq implementations, these boundaries still need to
> be set. Then, a "governor" must be selected. Such a "governor" decides
> what speed the processor shall run within the boundaries. One such
> "governor" is the "userspace" governor. This one allows the user - or
> a yet-to-implement userspace program - to decide what specific speed
> the processor shall run at.

The userspace program sets a policy for the whole system.

>> That's bad.  This feature is broken by design unless it does proper
>> save/restore across preemption.
> 
> Whats the current usecase, or forseeable future usecase, for save/restore
> across preemption again? (which would validate the broken by design
> claim).

Stop a guest that is using cpufreq, start a guest that is not using it.
The second guest's performance now depends on the state that the first
guest left in cpufreq.

I think this is abusing the userspace governor.  Unfortunately cpufreq
governors cannot be stacked.

Paolo

[toc] | [prev] | [next] | [standalone]

#1587545

On Fri, Feb 24, 2017 at 10:18:59AM +0100, Paolo Bonzini wrote:
> 
> 
> On 24/02/2017 00:19, Marcelo Tosatti wrote:
> >>> i.e. our feature implies userspace tasks pinned to isolated vCPUs.
> > This is how cpufreq-userspace works:
> > 
> > 2.2 Governor
> > ------------
> > 
> > On all other cpufreq implementations, these boundaries still need to
> > be set. Then, a "governor" must be selected. Such a "governor" decides
> > what speed the processor shall run within the boundaries. One such
> > "governor" is the "userspace" governor. This one allows the user - or
> > a yet-to-implement userspace program - to decide what specific speed
> > the processor shall run at.
> 
> The userspace program sets a policy for the whole system.

No, its per cpu.

> >> That's bad.  This feature is broken by design unless it does proper
> >> save/restore across preemption.
> > 
> > Whats the current usecase, or forseeable future usecase, for save/restore
> > across preemption again? (which would validate the broken by design
> > claim).
> 
> Stop a guest that is using cpufreq, start a guest that is not using it.
> The second guest's performance now depends on the state that the first
> guest left in cpufreq.

Nothing forbids the host to implement switching with the
current hypercall interface: all you need is a scheduler
hook.

> I think this is abusing the userspace governor.  Unfortunately cpufreq
> governors cannot be stacked.
> 
> Paolo

This is a special usecase where only the app in the guest knows 
whats the most appropriate frequency at a given time.
This is what cpufreq-userspace is supposed to allow userspace to do, 
but in this case "userspace" is the guest, so i don't 
see this as an abuse at all.

Timeshared setups are by definition not deterministic: 
your task A could be interrupted by another task B 
with results similar to a lower frequency being set.

So saying that:

"Our frequency scaling interface goes against the idea -- guest kernel
 cannot schedule multiple userspaces on the same vCPU, because they
 could
 conflict by overriding frequency."

Assumes that, in a timeshared system, an application is guaranteed a
particular frequency. But that does not make sense: its a timeshared
system in the first place, there is no determinism regarding execution
time.

Moreover, there is no notion of "per-task CPU frequency" in Linux
(there could be, this whole governor business with user
being responsible for setting up the governor is pretty sucky
IMO).

[toc] | [prev] | [next] | [standalone]

#1587552

On 24/02/2017 12:50, Marcelo Tosatti wrote:
>>>
>>> On all other cpufreq implementations, these boundaries still need to
>>> be set. Then, a "governor" must be selected. Such a "governor" decides
>>> what speed the processor shall run within the boundaries. One such
>>> "governor" is the "userspace" governor. This one allows the user - or
>>> a yet-to-implement userspace program - to decide what specific speed
>>> the processor shall run at.
>> The userspace program sets a policy for the whole system.
> No, its per cpu.

Yeah, what I mean is that userspace program can be per-CPU, but it looks
at all the processes running on that CPU ("the whole system").  This is
very different from a guest, which is isolated.

>>>> That's bad.  This feature is broken by design unless it does proper
>>>> save/restore across preemption.
>>> Whats the current usecase, or forseeable future usecase, for save/restore
>>> across preemption again? (which would validate the broken by design
>>> claim).
>> Stop a guest that is using cpufreq, start a guest that is not using it.
>> The second guest's performance now depends on the state that the first
>> guest left in cpufreq.
>
> Nothing forbids the host to implement switching with the
> current hypercall interface: all you need is a scheduler
> hook.

Can it be done in vcpu_load/vcpu_put?  But you still would have two
components (KVM and sysfs) potentially fighting over the frequency, and
that's still a bit ugly.

Paolo

[toc] | [prev] | [next] | [standalone]

#1587599

On Fri, Feb 24, 2017 at 01:17:07PM +0100, Paolo Bonzini wrote:
> 
> 
> On 24/02/2017 12:50, Marcelo Tosatti wrote:
> >>>
> >>> On all other cpufreq implementations, these boundaries still need to
> >>> be set. Then, a "governor" must be selected. Such a "governor" decides
> >>> what speed the processor shall run within the boundaries. One such
> >>> "governor" is the "userspace" governor. This one allows the user - or
> >>> a yet-to-implement userspace program - to decide what specific speed
> >>> the processor shall run at.
> >> The userspace program sets a policy for the whole system.
> > No, its per cpu.
> 
> Yeah, what I mean is that userspace program can be per-CPU, but it looks
> at all the processes running on that CPU ("the whole system").  This is
> very different from a guest, which is isolated.
> 
> >>>> That's bad.  This feature is broken by design unless it does proper
> >>>> save/restore across preemption.
> >>> Whats the current usecase, or forseeable future usecase, for save/restore
> >>> across preemption again? (which would validate the broken by design
> >>> claim).
> >> Stop a guest that is using cpufreq, start a guest that is not using it.
> >> The second guest's performance now depends on the state that the first
> >> guest left in cpufreq.
> >
> > Nothing forbids the host to implement switching with the
> > current hypercall interface: all you need is a scheduler
> > hook.
> 
> Can it be done in vcpu_load/vcpu_put?  But you still would have two
> components (KVM and sysfs) potentially fighting over the frequency, and
> that's still a bit ugly.
> 
> Paolo

Change the frequency at vcpu_load/vcpu_put? Yes: call into
cpufreq-userspace. But there is no notion of "per-task frequency" on the
Linux kernel (which was the starting point of this subthread).
 
But if you configure all CPUs in the system as cpufreq-userspace,
then some other (userspace program) has to decide the frequency
for the other CPUs.

Which agent would do that and why? Thats why i initially said "whats the
usecase".

[toc] | [prev] | [next] | [standalone]

#1587724

On 24/02/2017 14:04, Marcelo Tosatti wrote:
>>>>> Whats the current usecase, or forseeable future usecase, for save/restore
>>>>> across preemption again? (which would validate the broken by design
>>>>> claim).
>>>> Stop a guest that is using cpufreq, start a guest that is not using it.
>>>> The second guest's performance now depends on the state that the first
>>>> guest left in cpufreq.
>>> Nothing forbids the host to implement switching with the
>>> current hypercall interface: all you need is a scheduler
>>> hook.
>> Can it be done in vcpu_load/vcpu_put?  But you still would have two
>> components (KVM and sysfs) potentially fighting over the frequency, and
>> that's still a bit ugly.
>
> Change the frequency at vcpu_load/vcpu_put? Yes: call into
> cpufreq-userspace. But there is no notion of "per-task frequency" on the
> Linux kernel (which was the starting point of this subthread).

There isn't, but this patchset is providing a direct path from a task to
cpufreq-userspace.  This is as close as you can get to a per-task frequency.

> But if you configure all CPUs in the system as cpufreq-userspace,
> then some other (userspace program) has to decide the frequency
> for the other CPUs.
> 
> Which agent would do that and why? Thats why i initially said "whats the
> usecase".

You could just pin them at the highest non-TurboBoost frequency until a
guest runs.  That's assuming that they are idle and, because of
isol_cpus/nohz_full, they would be almost always in deep C state anyway.

Paolo

[toc] | [prev] | [next] | [standalone]

#1587809

On Friday, February 24, 2017 04:34:52 PM Paolo Bonzini wrote:
> 
> On 24/02/2017 14:04, Marcelo Tosatti wrote:
> >>>>> Whats the current usecase, or forseeable future usecase, for save/restore
> >>>>> across preemption again? (which would validate the broken by design
> >>>>> claim).
> >>>> Stop a guest that is using cpufreq, start a guest that is not using it.
> >>>> The second guest's performance now depends on the state that the first
> >>>> guest left in cpufreq.
> >>> Nothing forbids the host to implement switching with the
> >>> current hypercall interface: all you need is a scheduler
> >>> hook.
> >> Can it be done in vcpu_load/vcpu_put?  But you still would have two
> >> components (KVM and sysfs) potentially fighting over the frequency, and
> >> that's still a bit ugly.
> >
> > Change the frequency at vcpu_load/vcpu_put? Yes: call into
> > cpufreq-userspace. But there is no notion of "per-task frequency" on the
> > Linux kernel (which was the starting point of this subthread).
> 
> There isn't, but this patchset is providing a direct path from a task to
> cpufreq-userspace.  This is as close as you can get to a per-task frequency.
> 
> > But if you configure all CPUs in the system as cpufreq-userspace,
> > then some other (userspace program) has to decide the frequency
> > for the other CPUs.
> > 
> > Which agent would do that and why? Thats why i initially said "whats the
> > usecase".
> 
> You could just pin them at the highest non-TurboBoost frequency until a
> guest runs.  That's assuming that they are idle and, because of
> isol_cpus/nohz_full, they would be almost always in deep C state anyway.

Good discussion so far, but it should be happening on the linux-pm list.

Would it be possible to repost the patches with a CC to linux-pm?

Thanks,
Rafael

[toc] | [prev] | [next] | [standalone]

#1589790

On Fri, Feb 24, 2017 at 04:34:52PM +0100, Paolo Bonzini wrote:
> 
> 
> On 24/02/2017 14:04, Marcelo Tosatti wrote:
> >>>>> Whats the current usecase, or forseeable future usecase, for save/restore
> >>>>> across preemption again? (which would validate the broken by design
> >>>>> claim).
> >>>> Stop a guest that is using cpufreq, start a guest that is not using it.
> >>>> The second guest's performance now depends on the state that the first
> >>>> guest left in cpufreq.
> >>> Nothing forbids the host to implement switching with the
> >>> current hypercall interface: all you need is a scheduler
> >>> hook.
> >> Can it be done in vcpu_load/vcpu_put?  But you still would have two
> >> components (KVM and sysfs) potentially fighting over the frequency, and
> >> that's still a bit ugly.
> >
> > Change the frequency at vcpu_load/vcpu_put? Yes: call into
> > cpufreq-userspace. But there is no notion of "per-task frequency" on the
> > Linux kernel (which was the starting point of this subthread).
> 
> There isn't, but this patchset is providing a direct path from a task to
> cpufreq-userspace.  This is as close as you can get to a per-task frequency.

Cpufreq-userspace is supposed to be used by tasks in userspace.
Thats why its called "userspace".

> > But if you configure all CPUs in the system as cpufreq-userspace,
> > then some other (userspace program) has to decide the frequency
> > for the other CPUs.
> > 
> > Which agent would do that and why? Thats why i initially said "whats the
> > usecase".
> 
> You could just pin them at the highest non-TurboBoost frequency until a
> guest runs.  That's assuming that they are idle and, because of
> isol_cpus/nohz_full, they would be almost always in deep C state anyway.
> 
> Paolo

The original claim of the thread  was: "this feature (frequency
hypercalls) works for pinned vcpu<->pcpu, pcpu dedicated exclusively
to vcpu case, lets try to extend this to other cases".

Which is a valid and useful direction to go.

However there is no user for multiple vcpus in the same pcpu now.

If there were multiple vcpus, all of them requesting a given
frequency, it would be necessary to:

	1) Maintain frequency of the pcpu to the highest 
	   frequencies.

		OR

	2) Since switching frequencies can take up to 70us (*)
	   (depends on processor), its generally not worthwhile
	   to switch frequencies between task switches.

So its a dead end...

*: http://www.ena-hpc.org/2013/pdf/04.pdf

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web