Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1697085 > unrolled thread

[RFC]Add new mdev interface for QoS

Back to article view | Back to linux.kernel

Contents

  [RFC]Add new mdev interface for QoS "Gao, Ping A" <ping.a.gao@intel.com> - 2017-07-26 15:20 +0200
    Re: [RFC]Add new mdev interface for QoS Alex Williamson <alex.williamson@redhat.com> - 2017-07-26 18:50 +0200
      Re: [RFC]Add new mdev interface for QoS "Gao, Ping A" <ping.a.gao@intel.com> - 2017-07-27 18:10 +0200
      Re: [libvirt] [RFC]Add new mdev interface for QoS "Daniel P. Berrange" <berrange@redhat.com> - 2017-07-27 18:20 +0200
        Re: [libvirt] [RFC]Add new mdev interface for QoS Alex Williamson <alex.williamson@redhat.com> - 2017-07-27 20:10 +0200
          Re: [libvirt] [RFC]Add new mdev interface for QoS "Daniel P. Berrange" <berrange@redhat.com> - 2017-07-28 10:20 +0200

#1697085 — [RFC]Add new mdev interface for QoS

The vfio-mdev provide the capability to let different guest share the
same physical device through mediate sharing, as result it bring a
requirement about how to control the device sharing, we need a QoS
related interface for mdev to management virtual device resource.

E.g. In practical use, vGPUs assigned to different quests almost has
different performance requirements, some guests may need higher priority
for real time usage, some other may need more portion of the GPU
resource to get higher 3D performance, corresponding we can define some
interfaces like weight/cap for overall budget control, priority for
single submission control.

So I suggest to add some common attributes which are vendor agnostic in
mdev core sysfs for QoS purpose.

-Ping

[toc] | [next] | [standalone]

#1697350

[cc +libvir-list]

On Wed, 26 Jul 2017 21:16:59 +0800
"Gao, Ping A" <ping.a.gao@intel.com> wrote:

> The vfio-mdev provide the capability to let different guest share the
> same physical device through mediate sharing, as result it bring a
> requirement about how to control the device sharing, we need a QoS
> related interface for mdev to management virtual device resource.
> 
> E.g. In practical use, vGPUs assigned to different quests almost has
> different performance requirements, some guests may need higher priority
> for real time usage, some other may need more portion of the GPU
> resource to get higher 3D performance, corresponding we can define some
> interfaces like weight/cap for overall budget control, priority for
> single submission control.
> 
> So I suggest to add some common attributes which are vendor agnostic in
> mdev core sysfs for QoS purpose.

I think what you're asking for is just some standardization of a QoS
attribute_group which a vendor can optionally include within the
existing mdev_parent_ops.mdev_attr_groups.  The mdev core will
transparently enable this, but it really only provides the standard,
all of the support code is left for the vendor.  I'm fine with that,
but of course the trouble with and sort of standardization is arriving
at an agreed upon standard.  Are there QoS knobs that are generic
across any mdev device type?  Are there others that are more specific
to vGPU?  Are there existing examples of this that we can steal their
specification?

Also, mdev devices are not necessarily the exclusive users of the
hardware, we can have a native user such as a local X client.  They're
not an mdev user, so we can't support them via the mdev_attr_group.
Does there need to be a per mdev parent QoS attribute_group standard
for somehow defining the QoS of all the child mdev devices, or perhaps
representing the remaining host QoS attributes?

Ultimately libvirt and upper level management tools would be the
consumer of these control knobs, so let's immediately get libvirt
involved in the discussion.  Thanks,

Alex

[toc] | [prev] | [next] | [standalone]

#1698129

On 2017/7/27 0:43, Alex Williamson wrote:
> [cc +libvir-list]
>
> On Wed, 26 Jul 2017 21:16:59 +0800
> "Gao, Ping A" <ping.a.gao@intel.com> wrote:
>
>> The vfio-mdev provide the capability to let different guest share the
>> same physical device through mediate sharing, as result it bring a
>> requirement about how to control the device sharing, we need a QoS
>> related interface for mdev to management virtual device resource.
>>
>> E.g. In practical use, vGPUs assigned to different quests almost has
>> different performance requirements, some guests may need higher priority
>> for real time usage, some other may need more portion of the GPU
>> resource to get higher 3D performance, corresponding we can define some
>> interfaces like weight/cap for overall budget control, priority for
>> single submission control.
>>
>> So I suggest to add some common attributes which are vendor agnostic in
>> mdev core sysfs for QoS purpose.
> I think what you're asking for is just some standardization of a QoS
> attribute_group which a vendor can optionally include within the
> existing mdev_parent_ops.mdev_attr_groups.  The mdev core will
> transparently enable this, but it really only provides the standard,
> all of the support code is left for the vendor.  I'm fine with that,
> but of course the trouble with and sort of standardization is arriving
> at an agreed upon standard.  Are there QoS knobs that are generic
> across any mdev device type?  Are there others that are more specific
> to vGPU?  Are there existing examples of this that we can steal their
> specification?

Yes, you are right, standardization QoS knobs are exactly what I wanted.
Only when it become a part of the mdev framework and libvirt, then QoS
such critical feature can be leveraged by cloud usage. HW vendor only
need to focus on the implementation of the corresponding QoS algorithm
in their back-end driver.

Vfio-mdev framework provide the capability to share the device that lack
of HW virtualization support to guests, no matter the device type,
mediated sharing actually is a time sharing multiplex method, from this
point of view, QoS can be take as a generic way about how to control the
time assignment for virtual mdev device that occupy HW. As result we can
define QoS knob generic across any device type by this way. Even if HW
has build in with some kind of QoS support, I think it's not a problem
for back-end driver to convert mdev standard QoS definition to their
specification to reach the same performance expectation. Seems there are
no examples for us to follow, we need define it from scratch.

I proposal universal QoS control interfaces like below:

Cap: The cap limits the maximum percentage of time a mdev device can own
physical device. e.g. cap=60, means mdev device cannot take over 60% of
total physical resource.

Weight: The weight define proportional control of the mdev device
resource between guests, it’s orthogonal with Cap, to target load
balancing. E.g. if guest 1 should take double mdev device resource
compare with guest 2, need set weight ratio to 2:1.

Priority: The guest who has higher priority will get execution first,
target to some real time usage and speeding interactive response.

Above QoS interfaces cover both overall budget control and single
submission control. I will sent out detail design later once get aligned.

> Also, mdev devices are not necessarily the exclusive users of the
> hardware, we can have a native user such as a local X client.  They're
> not an mdev user, so we can't support them via the mdev_attr_group.
> Does there need to be a per mdev parent QoS attribute_group standard
> for somehow defining the QoS of all the child mdev devices, or perhaps
> representing the remaining host QoS attributes?

That's really an open, if we don't take host workload into consideration
for cloud usage, it's not a problem any more, however such assumption is
not reasonable. Any way if we take mdev devices as clients of host
driver, and host driver provide the capability to divide out a portion
HW resource to mdev devices, then it's only need to take care about the
resource that host assigned for mdev devices. Follow this way QoS for
mdev focus on the relationship between mdev devices no need to take care
the host workload.

-Ping

> Ultimately libvirt and upper level management tools would be the
> consumer of these control knobs, so let's immediately get libvirt
> involved in the discussion.  Thanks,
>
> Alex

[toc] | [prev] | [next] | [standalone]

#1698142 — Re: [libvirt] [RFC]Add new mdev interface for QoS

On Wed, Jul 26, 2017 at 10:43:43AM -0600, Alex Williamson wrote:
> [cc +libvir-list]
> 
> On Wed, 26 Jul 2017 21:16:59 +0800
> "Gao, Ping A" <ping.a.gao@intel.com> wrote:
> 
> > The vfio-mdev provide the capability to let different guest share the
> > same physical device through mediate sharing, as result it bring a
> > requirement about how to control the device sharing, we need a QoS
> > related interface for mdev to management virtual device resource.
> > 
> > E.g. In practical use, vGPUs assigned to different quests almost has
> > different performance requirements, some guests may need higher priority
> > for real time usage, some other may need more portion of the GPU
> > resource to get higher 3D performance, corresponding we can define some
> > interfaces like weight/cap for overall budget control, priority for
> > single submission control.
> > 
> > So I suggest to add some common attributes which are vendor agnostic in
> > mdev core sysfs for QoS purpose.
> 
> I think what you're asking for is just some standardization of a QoS
> attribute_group which a vendor can optionally include within the
> existing mdev_parent_ops.mdev_attr_groups.  The mdev core will
> transparently enable this, but it really only provides the standard,
> all of the support code is left for the vendor.  I'm fine with that,
> but of course the trouble with and sort of standardization is arriving
> at an agreed upon standard.  Are there QoS knobs that are generic
> across any mdev device type?  Are there others that are more specific
> to vGPU?  Are there existing examples of this that we can steal their
> specification?
> 
> Also, mdev devices are not necessarily the exclusive users of the
> hardware, we can have a native user such as a local X client.  They're
> not an mdev user, so we can't support them via the mdev_attr_group.
> Does there need to be a per mdev parent QoS attribute_group standard
> for somehow defining the QoS of all the child mdev devices, or perhaps
> representing the remaining host QoS attributes?
> 
> Ultimately libvirt and upper level management tools would be the
> consumer of these control knobs, so let's immediately get libvirt
> involved in the discussion.  Thanks,

My view on this from libvirt side is pretty much unchanged since the
last time we discussed this.

We would like the kernel maintainers to define standard sets of properties
for mdevs, whether global to all mdevs, or scoped to certain classes of
mdev (eg a class=gpu). These properties would be exported in sysfs, with
one file per property.

Libvirt can then explicitly map each standardized property into a suitable
XML element, to report on which properties are available to use when creating
an mdev. It would then allow them to be set at time of creation, and/or
changed on the fly for existing mdevs.

Specifically we would like to avoid generic passthrough of arbitrary
vendor specific properties.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[toc] | [prev] | [next] | [standalone]

#1698228 — Re: [libvirt] [RFC]Add new mdev interface for QoS

On Thu, 27 Jul 2017 17:17:48 +0100
"Daniel P. Berrange" <berrange@redhat.com> wrote:

> On Wed, Jul 26, 2017 at 10:43:43AM -0600, Alex Williamson wrote:
> > [cc +libvir-list]
> > 
> > On Wed, 26 Jul 2017 21:16:59 +0800
> > "Gao, Ping A" <ping.a.gao@intel.com> wrote:
> >   
> > > The vfio-mdev provide the capability to let different guest share the
> > > same physical device through mediate sharing, as result it bring a
> > > requirement about how to control the device sharing, we need a QoS
> > > related interface for mdev to management virtual device resource.
> > > 
> > > E.g. In practical use, vGPUs assigned to different quests almost has
> > > different performance requirements, some guests may need higher priority
> > > for real time usage, some other may need more portion of the GPU
> > > resource to get higher 3D performance, corresponding we can define some
> > > interfaces like weight/cap for overall budget control, priority for
> > > single submission control.
> > > 
> > > So I suggest to add some common attributes which are vendor agnostic in
> > > mdev core sysfs for QoS purpose.  
> > 
> > I think what you're asking for is just some standardization of a QoS
> > attribute_group which a vendor can optionally include within the
> > existing mdev_parent_ops.mdev_attr_groups.  The mdev core will
> > transparently enable this, but it really only provides the standard,
> > all of the support code is left for the vendor.  I'm fine with that,
> > but of course the trouble with and sort of standardization is arriving
> > at an agreed upon standard.  Are there QoS knobs that are generic
> > across any mdev device type?  Are there others that are more specific
> > to vGPU?  Are there existing examples of this that we can steal their
> > specification?
> > 
> > Also, mdev devices are not necessarily the exclusive users of the
> > hardware, we can have a native user such as a local X client.  They're
> > not an mdev user, so we can't support them via the mdev_attr_group.
> > Does there need to be a per mdev parent QoS attribute_group standard
> > for somehow defining the QoS of all the child mdev devices, or perhaps
> > representing the remaining host QoS attributes?
> > 
> > Ultimately libvirt and upper level management tools would be the
> > consumer of these control knobs, so let's immediately get libvirt
> > involved in the discussion.  Thanks,  
> 
> My view on this from libvirt side is pretty much unchanged since the
> last time we discussed this.
> 
> We would like the kernel maintainers to define standard sets of properties
> for mdevs, whether global to all mdevs, or scoped to certain classes of
> mdev (eg a class=gpu). These properties would be exported in sysfs, with
> one file per property.

Yes, I think that much of the mechanics are obvious (standardized
sysfs layout, one property per file, properties under the device node
in sysfs, etc).  Are you saying that you don't want to be consulted on
which properties are exposed and how they operate and therefore won't
complain regardless of what we implement in the kernel? ;)

I'm hoping that libvirt folks have some experience managing basic
scheduling level QoS attributes and might have some input as to what
sorts of things work well vs what seems like a good idea, but falls
apart or isn't useful in practice.

> Libvirt can then explicitly map each standardized property into a suitable
> XML element, to report on which properties are available to use when creating
> an mdev. It would then allow them to be set at time of creation, and/or
> changed on the fly for existing mdevs.
> 
> Specifically we would like to avoid generic passthrough of arbitrary
> vendor specific properties.

Of course, I think that's the intent here.  Thanks,

Alex

[toc] | [prev] | [next] | [standalone]

#1698581 — Re: [libvirt] [RFC]Add new mdev interface for QoS

On Thu, Jul 27, 2017 at 12:01:58PM -0600, Alex Williamson wrote:
> On Thu, 27 Jul 2017 17:17:48 +0100
> "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > On Wed, Jul 26, 2017 at 10:43:43AM -0600, Alex Williamson wrote:
> > > [cc +libvir-list]
> > > 
> > > On Wed, 26 Jul 2017 21:16:59 +0800
> > > "Gao, Ping A" <ping.a.gao@intel.com> wrote:
> > >   
> > > > The vfio-mdev provide the capability to let different guest share the
> > > > same physical device through mediate sharing, as result it bring a
> > > > requirement about how to control the device sharing, we need a QoS
> > > > related interface for mdev to management virtual device resource.
> > > > 
> > > > E.g. In practical use, vGPUs assigned to different quests almost has
> > > > different performance requirements, some guests may need higher priority
> > > > for real time usage, some other may need more portion of the GPU
> > > > resource to get higher 3D performance, corresponding we can define some
> > > > interfaces like weight/cap for overall budget control, priority for
> > > > single submission control.
> > > > 
> > > > So I suggest to add some common attributes which are vendor agnostic in
> > > > mdev core sysfs for QoS purpose.  
> > > 
> > > I think what you're asking for is just some standardization of a QoS
> > > attribute_group which a vendor can optionally include within the
> > > existing mdev_parent_ops.mdev_attr_groups.  The mdev core will
> > > transparently enable this, but it really only provides the standard,
> > > all of the support code is left for the vendor.  I'm fine with that,
> > > but of course the trouble with and sort of standardization is arriving
> > > at an agreed upon standard.  Are there QoS knobs that are generic
> > > across any mdev device type?  Are there others that are more specific
> > > to vGPU?  Are there existing examples of this that we can steal their
> > > specification?
> > > 
> > > Also, mdev devices are not necessarily the exclusive users of the
> > > hardware, we can have a native user such as a local X client.  They're
> > > not an mdev user, so we can't support them via the mdev_attr_group.
> > > Does there need to be a per mdev parent QoS attribute_group standard
> > > for somehow defining the QoS of all the child mdev devices, or perhaps
> > > representing the remaining host QoS attributes?
> > > 
> > > Ultimately libvirt and upper level management tools would be the
> > > consumer of these control knobs, so let's immediately get libvirt
> > > involved in the discussion.  Thanks,  
> > 
> > My view on this from libvirt side is pretty much unchanged since the
> > last time we discussed this.
> > 
> > We would like the kernel maintainers to define standard sets of properties
> > for mdevs, whether global to all mdevs, or scoped to certain classes of
> > mdev (eg a class=gpu). These properties would be exported in sysfs, with
> > one file per property.
> 
> Yes, I think that much of the mechanics are obvious (standardized
> sysfs layout, one property per file, properties under the device node
> in sysfs, etc).  Are you saying that you don't want to be consulted on
> which properties are exposed and how they operate and therefore won't
> complain regardless of what we implement in the kernel? ;)

Well ultimately the kernel maintainers know what is possible from the
hardware / driver POV, so yeah, I think we can mostly leave it upto
you what individual things need to be exposed - not much different
scenario from all the knobs we already exposed for physical devices.


> I'm hoping that libvirt folks have some experience managing basic
> scheduling level QoS attributes and might have some input as to what
> sorts of things work well vs what seems like a good idea, but falls
> apart or isn't useful in practice.

Sure, happy to give feedback where desired.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web