Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1225426 > unrolled thread

Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the  scheduling domain hierarchy Dario Faggioli <dario.faggioli@citrix.com> - 2015-09-15 19:00 +0200
    Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling  domain hierarchy Juergen Gross <jgross@suse.com> - 2015-09-21 07:50 +0200
      Re: Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the  scheduling domain hierarchy Juergen Gross <jgross@suse.com> - 2015-09-22 06:50 +0200
        Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling  domain hierarchy George Dunlap <george.dunlap@citrix.com> - 2015-09-22 18:30 +0200
          Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling  domain hierarchy Juergen Gross <jgross@suse.com> - 2015-09-23 06:40 +0200
            Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the  scheduling domain hierarchy Dario Faggioli <dario.faggioli@citrix.com> - 2015-09-23 10:40 +0200
              Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling  domain hierarchy Juergen Gross <jgross@suse.com> - 2015-09-23 11:50 +0200
            Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling  domain hierarchy George Dunlap <george.dunlap@citrix.com> - 2015-09-23 12:30 +0200
      Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the  scheduling domain hierarchy Dario Faggioli <dario.faggioli@citrix.com> - 2015-09-23 09:30 +0200
        Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling  domain hierarchy Juergen Gross <jgross@suse.com> - 2015-09-23 09:40 +0200
          Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling  domain hierarchy Boris Ostrovsky <boris.ostrovsky@oracle.com> - 2015-09-23 14:30 +0200

#1225426 — Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2015-08-20 at 20:16 +0200, Juergen Groß wrote:
> On 08/18/2015 05:55 PM, Dario Faggioli wrote:
> > Hey everyone,
> >
> > So, as a followup of what we were discussing in this thread:
> >
> >   [Xen-devel] PV-vNUMA issue: topology is misinterpreted by the guest
> >   http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg03241.html
> >
> > I started looking in more details at scheduling domains in the Linux
> > kernel. Now, that thread was about CPUID and vNUMA, and their weird way
> > of interacting, while this thing I'm proposing here is completely
> > independent from them both.
> >
> > In fact, no matter whether vNUMA is supported and enabled, and no matter
> > whether CPUID is reporting accurate, random, meaningful or completely
> > misleading information, I think that we should do something about how
> > scheduling domains are build.
> >
> > Fact is, unless we use 1:1, and immutable (across all the guest
> > lifetime) pinning, scheduling domains should not be constructed, in
> > Linux, by looking at *any* topology information, because that just does
> > not make any sense, when vcpus move around.
> >
> > Let me state this again (hoping to make myself as clear as possible): no
> > matter in  how much good shape we put CPUID support, no matter how
> > beautifully and consistently that will interact with both vNUMA,
> > licensing requirements and whatever else. It will be always possible for
> > vCPU #0 and vCPU #3 to be scheduled on two SMT threads at time t1, and
> > on two different NUMA nodes at time t2. Hence, the Linux scheduler
> > should really not skew his load balancing logic toward any of those two
> > situations, as neither of them could be considered correct (since
> > nothing is!).
> >
> > For now, this only covers the PV case. HVM case shouldn't be any
> > different, but I haven't looked at how to make the same thing happen in
> > there as well.
> >
> > OVERALL DESCRIPTION
> > ===================
> > What this RFC patch does is, in the Xen PV case, configure scheduling
> > domains in such a way that there is only one of them, spanning all the
> > pCPUs of the guest.
> >
> > Note that the patch deals directly with scheduling domains, and there is
> > no need to alter the masks that will then be used for building and
> > reporting the topology (via CPUID, /proc/cpuinfo, /sysfs, etc.). That is
> > the main difference between it and the patch proposed by Juergen here:
> > http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg05088.html
> >
> > This means that when, in future, we will fix CPUID handling and make it
> > comply with whatever logic or requirements we want, that won't have  any
> > unexpected side effects on scheduling domains.
> >
> > Information about how the scheduling domains are being constructed
> > during boot are available in `dmesg', if the kernel is booted with the
> > 'sched_debug' parameter. It is also possible to look
> > at /proc/sys/kernel/sched_domain/cpu*, and at /proc/schedstat.
> >
> > With the patch applied, only one scheduling domain is created, called
> > the 'VCPU' domain, spanning all the guest's (or Dom0's) vCPUs. You can
> > tell that from the fact that every cpu* folder
> > in /proc/sys/kernel/sched_domain/ only have one subdirectory
> > ('domain0'), with all the tweaks and the tunables for our scheduling
> > domain.
> >
> > EVALUATION
> > ==========
> > I've tested this with UnixBench, and by looking at Xen build time, on a
> > 16, 24 and 48 pCPUs hosts. I've run the benchmarks in Dom0 only, for
> > now, but I plan to re-run them in DomUs soon (Juergen may be doing
> > something similar to this in DomU already, AFAUI).
> >
> > I've run the benchmarks with and without the patch applied ('patched'
> > and 'vanilla', respectively, in the tables below), and with different
> > number of build jobs (in case of the Xen build) or of parallel copy of
> > the benchmarks (in the case of UnixBench).
> >
> > What I get from the numbers is that the patch almost always brings
> > benefits, in some cases even huge ones. There are a couple of cases
> > where we regress, but always only slightly so, especially if comparing
> > that to the magnitude of some of the improvement that we get.
> >
> > Bear also in mind that these results are gathered from Dom0, and without
> > any overcommitment at the vCPU level (i.e., nr. vCPUs == nr pCPUs). If
> > we move things in DomU and do overcommit at the Xen scheduler level, I
> > am expecting even better results.
> >
> ...
> > REQUEST FOR COMMENTS
> > ====================
> > Basically, the kind of feedback I'd be really glad to hear is:
> >   - what you guys thing of the approach,
> 
> Yesterday at the end of the developer meeting we (Andrew, Elena and
> myself) discussed this topic again.
> 
Hey,

Sorry for replying so late, I've been on vacation from right after
XenSummit up until yesterday. :-)

> Regarding a possible future scenario with credit2 eventually supporting
> gang scheduling on hyperthreads (which is desirable due to security
> reasons [side channel attack] and fairness) my patch seems to be more
> suited for that direction than yours. 
>
Ok. Just let me mention that 'Credit2 + gang scheduling' might not be
exactly around the corner (although, we can prioritize working on it if
we want).

In principle, I think it's a really nice idea. I still don't have clear
in mind how we would handle a couple of situations, but let's leave this
aside for now, and stay on-topic.

> Correct me if I'm wrong, but I
> think scheduling domains won't enable the guest kernel's scheduler to
> migrate threads more easily between hyperthreads opposed to other vcpus,
> while my approach can easily be extended to do so.
> 
I'm not sure I understand what you mean here. As far as the (Linux)
scheduler is concerned, your patch and mine do the exact same thing:
they arrange for the scheduling domains, when they're built, during
boot, not to consider hyperthreads or multi-cores.

Mine does it by removing the SMT (and the MC) level from the data
structure in the scheduler that is used as a base for configuring the
scheduling domains. Yours does it by making the topology bitmaps that
are used at each one of those level all look the same. In fact, with
your patch applied, I get the exact same situation as with mine, as far
as scheduling domains are concerned: there is only one scheduling
domain, with a different scheduling group for each vCPU inside it.

In my case, that one scheduling domain is the special one that I define
in xen_sched_domain_topology (in arch/x86/xen/smp.c), in my patch (it's
called PCPU). In your case, it's the DIE scheduling domain, i.e., the
one coming from the last level defined in default_topology (in
kernel/sched/core.c). I'd have to recheck, but ISTR that, since you're
setting all the bitmaps for all the levels to the same value, previous
levels are created, recognised to be all equal, and merged/discarded.

IOW, mine is using a scheduler provided interface explicitly, via
set_sched_topology(), i.e., the way an architecture (and in this case
the architecture would be 'xen') let the scheduler know about its
topology quirks:
 http://lxr.free-electrons.com/ident?i=set_sched_topology
Basically, I'm telling the scheduler <<Hey, you're on Xen, don't bother
looking for hyperthreads, as they don't make any sense!>>.

Yours is changing the topology bitmaps directly. Basically, you're
telling nothing to the scheduler, which then goes down looking for SMTs
and MCs, but finds none.

All this being said, the effect is the same, and the reason why the
scheduling inside the guest changes --between mainline and both mine or
your patch-- is because of scheduling domains, or so it is how I
understood it.

Therefore, I don't really understand why you're saying one approach is
more easily extensible toward anything... What am I missing?

> >   - whether you think, looking at this preliminary set of numbers, that
> >     this is something worth continuing investigating,
> 
> I believe as both approaches lead to the same topology information used
> by the scheduler (all vcpus are regarded as being equal) your numbers
> should apply to my patch as well. Would you mind verifying this?
> 
I'll run some tests, but yes, I 100% expect the numbers to look the
same. Actually, I did a very quick check, for a few cases, already, and
that is indeed the case, but I'll report back when I'll have the full
data set.

> >   - if yes, what other workloads and benchmark it would make sense to
> >     throw at it.
> 
> As you already mentioned an overcommitted host should be looked at as
> well.
> 
Sure.

> Thanks for doing the measurements,
> 
And more of them will be coming. ISTR you telling me in Seattle that you
(or some teammates of yours) were running some benches too... Any output
from that yet? :-)

Thanks and Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[toc] | [next] | [standalone]

#1229058 — Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy

On 09/15/2015 06:50 PM, Dario Faggioli wrote:
> On Thu, 2015-08-20 at 20:16 +0200, Juergen Groß wrote:
>> On 08/18/2015 05:55 PM, Dario Faggioli wrote:
>>> Hey everyone,
>>>
>>> So, as a followup of what we were discussing in this thread:
>>>
>>>    [Xen-devel] PV-vNUMA issue: topology is misinterpreted by the guest
>>>    http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg03241.html
>>>
>>> I started looking in more details at scheduling domains in the Linux
>>> kernel. Now, that thread was about CPUID and vNUMA, and their weird way
>>> of interacting, while this thing I'm proposing here is completely
>>> independent from them both.
>>>
>>> In fact, no matter whether vNUMA is supported and enabled, and no matter
>>> whether CPUID is reporting accurate, random, meaningful or completely
>>> misleading information, I think that we should do something about how
>>> scheduling domains are build.
>>>
>>> Fact is, unless we use 1:1, and immutable (across all the guest
>>> lifetime) pinning, scheduling domains should not be constructed, in
>>> Linux, by looking at *any* topology information, because that just does
>>> not make any sense, when vcpus move around.
>>>
>>> Let me state this again (hoping to make myself as clear as possible): no
>>> matter in  how much good shape we put CPUID support, no matter how
>>> beautifully and consistently that will interact with both vNUMA,
>>> licensing requirements and whatever else. It will be always possible for
>>> vCPU #0 and vCPU #3 to be scheduled on two SMT threads at time t1, and
>>> on two different NUMA nodes at time t2. Hence, the Linux scheduler
>>> should really not skew his load balancing logic toward any of those two
>>> situations, as neither of them could be considered correct (since
>>> nothing is!).
>>>
>>> For now, this only covers the PV case. HVM case shouldn't be any
>>> different, but I haven't looked at how to make the same thing happen in
>>> there as well.
>>>
>>> OVERALL DESCRIPTION
>>> ===================
>>> What this RFC patch does is, in the Xen PV case, configure scheduling
>>> domains in such a way that there is only one of them, spanning all the
>>> pCPUs of the guest.
>>>
>>> Note that the patch deals directly with scheduling domains, and there is
>>> no need to alter the masks that will then be used for building and
>>> reporting the topology (via CPUID, /proc/cpuinfo, /sysfs, etc.). That is
>>> the main difference between it and the patch proposed by Juergen here:
>>> http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg05088.html
>>>
>>> This means that when, in future, we will fix CPUID handling and make it
>>> comply with whatever logic or requirements we want, that won't have  any
>>> unexpected side effects on scheduling domains.
>>>
>>> Information about how the scheduling domains are being constructed
>>> during boot are available in `dmesg', if the kernel is booted with the
>>> 'sched_debug' parameter. It is also possible to look
>>> at /proc/sys/kernel/sched_domain/cpu*, and at /proc/schedstat.
>>>
>>> With the patch applied, only one scheduling domain is created, called
>>> the 'VCPU' domain, spanning all the guest's (or Dom0's) vCPUs. You can
>>> tell that from the fact that every cpu* folder
>>> in /proc/sys/kernel/sched_domain/ only have one subdirectory
>>> ('domain0'), with all the tweaks and the tunables for our scheduling
>>> domain.
>>>
>>> EVALUATION
>>> ==========
>>> I've tested this with UnixBench, and by looking at Xen build time, on a
>>> 16, 24 and 48 pCPUs hosts. I've run the benchmarks in Dom0 only, for
>>> now, but I plan to re-run them in DomUs soon (Juergen may be doing
>>> something similar to this in DomU already, AFAUI).
>>>
>>> I've run the benchmarks with and without the patch applied ('patched'
>>> and 'vanilla', respectively, in the tables below), and with different
>>> number of build jobs (in case of the Xen build) or of parallel copy of
>>> the benchmarks (in the case of UnixBench).
>>>
>>> What I get from the numbers is that the patch almost always brings
>>> benefits, in some cases even huge ones. There are a couple of cases
>>> where we regress, but always only slightly so, especially if comparing
>>> that to the magnitude of some of the improvement that we get.
>>>
>>> Bear also in mind that these results are gathered from Dom0, and without
>>> any overcommitment at the vCPU level (i.e., nr. vCPUs == nr pCPUs). If
>>> we move things in DomU and do overcommit at the Xen scheduler level, I
>>> am expecting even better results.
>>>
>> ...
>>> REQUEST FOR COMMENTS
>>> ====================
>>> Basically, the kind of feedback I'd be really glad to hear is:
>>>    - what you guys thing of the approach,
>>
>> Yesterday at the end of the developer meeting we (Andrew, Elena and
>> myself) discussed this topic again.
>>
> Hey,
>
> Sorry for replying so late, I've been on vacation from right after
> XenSummit up until yesterday. :-)
>
>> Regarding a possible future scenario with credit2 eventually supporting
>> gang scheduling on hyperthreads (which is desirable due to security
>> reasons [side channel attack] and fairness) my patch seems to be more
>> suited for that direction than yours.
>>
> Ok. Just let me mention that 'Credit2 + gang scheduling' might not be
> exactly around the corner (although, we can prioritize working on it if
> we want).
>
> In principle, I think it's a really nice idea. I still don't have clear
> in mind how we would handle a couple of situations, but let's leave this
> aside for now, and stay on-topic.
>
>> Correct me if I'm wrong, but I
>> think scheduling domains won't enable the guest kernel's scheduler to
>> migrate threads more easily between hyperthreads opposed to other vcpus,
>> while my approach can easily be extended to do so.
>>
> I'm not sure I understand what you mean here. As far as the (Linux)
> scheduler is concerned, your patch and mine do the exact same thing:
> they arrange for the scheduling domains, when they're built, during
> boot, not to consider hyperthreads or multi-cores.
>
> Mine does it by removing the SMT (and the MC) level from the data
> structure in the scheduler that is used as a base for configuring the
> scheduling domains. Yours does it by making the topology bitmaps that
> are used at each one of those level all look the same. In fact, with
> your patch applied, I get the exact same situation as with mine, as far
> as scheduling domains are concerned: there is only one scheduling
> domain, with a different scheduling group for each vCPU inside it.

Uuh, nearly.

Your case won't deal correctly with NUMA, as the generic NUMA code is
using set_sched_topology() as well. One of NUMA and Xen will win and
overwrite the other's settings.

To do things correctly you will have to handle NUMA as well.


Juergen

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1229903

On 09/21/2015 07:49 AM, Juergen Gross wrote:
> On 09/15/2015 06:50 PM, Dario Faggioli wrote:
>> On Thu, 2015-08-20 at 20:16 +0200, Juergen Groß wrote:
>>> On 08/18/2015 05:55 PM, Dario Faggioli wrote:
>>>> Hey everyone,
>>>>
>>>> So, as a followup of what we were discussing in this thread:
>>>>
>>>>    [Xen-devel] PV-vNUMA issue: topology is misinterpreted by the guest
>>>>
>>>> http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg03241.html
>>>>
>>>>
>>>> I started looking in more details at scheduling domains in the Linux
>>>> kernel. Now, that thread was about CPUID and vNUMA, and their weird way
>>>> of interacting, while this thing I'm proposing here is completely
>>>> independent from them both.
>>>>
>>>> In fact, no matter whether vNUMA is supported and enabled, and no
>>>> matter
>>>> whether CPUID is reporting accurate, random, meaningful or completely
>>>> misleading information, I think that we should do something about how
>>>> scheduling domains are build.
>>>>
>>>> Fact is, unless we use 1:1, and immutable (across all the guest
>>>> lifetime) pinning, scheduling domains should not be constructed, in
>>>> Linux, by looking at *any* topology information, because that just does
>>>> not make any sense, when vcpus move around.
>>>>
>>>> Let me state this again (hoping to make myself as clear as
>>>> possible): no
>>>> matter in  how much good shape we put CPUID support, no matter how
>>>> beautifully and consistently that will interact with both vNUMA,
>>>> licensing requirements and whatever else. It will be always possible
>>>> for
>>>> vCPU #0 and vCPU #3 to be scheduled on two SMT threads at time t1, and
>>>> on two different NUMA nodes at time t2. Hence, the Linux scheduler
>>>> should really not skew his load balancing logic toward any of those two
>>>> situations, as neither of them could be considered correct (since
>>>> nothing is!).
>>>>
>>>> For now, this only covers the PV case. HVM case shouldn't be any
>>>> different, but I haven't looked at how to make the same thing happen in
>>>> there as well.
>>>>
>>>> OVERALL DESCRIPTION
>>>> ===================
>>>> What this RFC patch does is, in the Xen PV case, configure scheduling
>>>> domains in such a way that there is only one of them, spanning all the
>>>> pCPUs of the guest.
>>>>
>>>> Note that the patch deals directly with scheduling domains, and
>>>> there is
>>>> no need to alter the masks that will then be used for building and
>>>> reporting the topology (via CPUID, /proc/cpuinfo, /sysfs, etc.).
>>>> That is
>>>> the main difference between it and the patch proposed by Juergen here:
>>>> http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg05088.html
>>>>
>>>>
>>>> This means that when, in future, we will fix CPUID handling and make it
>>>> comply with whatever logic or requirements we want, that won't have
>>>> any
>>>> unexpected side effects on scheduling domains.
>>>>
>>>> Information about how the scheduling domains are being constructed
>>>> during boot are available in `dmesg', if the kernel is booted with the
>>>> 'sched_debug' parameter. It is also possible to look
>>>> at /proc/sys/kernel/sched_domain/cpu*, and at /proc/schedstat.
>>>>
>>>> With the patch applied, only one scheduling domain is created, called
>>>> the 'VCPU' domain, spanning all the guest's (or Dom0's) vCPUs. You can
>>>> tell that from the fact that every cpu* folder
>>>> in /proc/sys/kernel/sched_domain/ only have one subdirectory
>>>> ('domain0'), with all the tweaks and the tunables for our scheduling
>>>> domain.
>>>>
>>>> EVALUATION
>>>> ==========
>>>> I've tested this with UnixBench, and by looking at Xen build time, on a
>>>> 16, 24 and 48 pCPUs hosts. I've run the benchmarks in Dom0 only, for
>>>> now, but I plan to re-run them in DomUs soon (Juergen may be doing
>>>> something similar to this in DomU already, AFAUI).
>>>>
>>>> I've run the benchmarks with and without the patch applied ('patched'
>>>> and 'vanilla', respectively, in the tables below), and with different
>>>> number of build jobs (in case of the Xen build) or of parallel copy of
>>>> the benchmarks (in the case of UnixBench).
>>>>
>>>> What I get from the numbers is that the patch almost always brings
>>>> benefits, in some cases even huge ones. There are a couple of cases
>>>> where we regress, but always only slightly so, especially if comparing
>>>> that to the magnitude of some of the improvement that we get.
>>>>
>>>> Bear also in mind that these results are gathered from Dom0, and
>>>> without
>>>> any overcommitment at the vCPU level (i.e., nr. vCPUs == nr pCPUs). If
>>>> we move things in DomU and do overcommit at the Xen scheduler level, I
>>>> am expecting even better results.
>>>>
>>> ...
>>>> REQUEST FOR COMMENTS
>>>> ====================
>>>> Basically, the kind of feedback I'd be really glad to hear is:
>>>>    - what you guys thing of the approach,
>>>
>>> Yesterday at the end of the developer meeting we (Andrew, Elena and
>>> myself) discussed this topic again.
>>>
>> Hey,
>>
>> Sorry for replying so late, I've been on vacation from right after
>> XenSummit up until yesterday. :-)
>>
>>> Regarding a possible future scenario with credit2 eventually supporting
>>> gang scheduling on hyperthreads (which is desirable due to security
>>> reasons [side channel attack] and fairness) my patch seems to be more
>>> suited for that direction than yours.
>>>
>> Ok. Just let me mention that 'Credit2 + gang scheduling' might not be
>> exactly around the corner (although, we can prioritize working on it if
>> we want).
>>
>> In principle, I think it's a really nice idea. I still don't have clear
>> in mind how we would handle a couple of situations, but let's leave this
>> aside for now, and stay on-topic.
>>
>>> Correct me if I'm wrong, but I
>>> think scheduling domains won't enable the guest kernel's scheduler to
>>> migrate threads more easily between hyperthreads opposed to other vcpus,
>>> while my approach can easily be extended to do so.
>>>
>> I'm not sure I understand what you mean here. As far as the (Linux)
>> scheduler is concerned, your patch and mine do the exact same thing:
>> they arrange for the scheduling domains, when they're built, during
>> boot, not to consider hyperthreads or multi-cores.
>>
>> Mine does it by removing the SMT (and the MC) level from the data
>> structure in the scheduler that is used as a base for configuring the
>> scheduling domains. Yours does it by making the topology bitmaps that
>> are used at each one of those level all look the same. In fact, with
>> your patch applied, I get the exact same situation as with mine, as far
>> as scheduling domains are concerned: there is only one scheduling
>> domain, with a different scheduling group for each vCPU inside it.
>
> Uuh, nearly.
>
> Your case won't deal correctly with NUMA, as the generic NUMA code is
> using set_sched_topology() as well. One of NUMA and Xen will win and
> overwrite the other's settings.
>
> To do things correctly you will have to handle NUMA as well.

One other thing I just discovered: there are other consumers of the
topology sibling masks (e.g. topology_sibling_cpumask()) as well.

I think we would want to avoid any optimizations based on those in
drivers as well, not only in the scheduler.


Juergen

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1230442 — Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy

On 09/22/2015 05:42 AM, Juergen Gross wrote:
> One other thing I just discovered: there are other consumers of the
> topology sibling masks (e.g. topology_sibling_cpumask()) as well.
> 
> I think we would want to avoid any optimizations based on those in
> drivers as well, not only in the scheduler.

I'm beginning to lose the thread of the discussion here a bit.

Juergen / Dario, could one of you summarize your two approaches, and the
(alleged) advantages and disadvantages of each one?

Thanks,
 -George
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1231132 — Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy

On 09/22/2015 06:22 PM, George Dunlap wrote:
> On 09/22/2015 05:42 AM, Juergen Gross wrote:
>> One other thing I just discovered: there are other consumers of the
>> topology sibling masks (e.g. topology_sibling_cpumask()) as well.
>>
>> I think we would want to avoid any optimizations based on those in
>> drivers as well, not only in the scheduler.
>
> I'm beginning to lose the thread of the discussion here a bit.
>
> Juergen / Dario, could one of you summarize your two approaches, and the
> (alleged) advantages and disadvantages of each one?

Okay, I'll have a try:

The problem we want to solve:
-----------------------------

The Linux kernel is gathering cpu topology data during boot via the
CPUID instruction on each processor coming online. This data is
primarily used in the scheduler to decide to which cpu a thread should
be migrated when this seems to be necessary. There are other users of
the topology information in the kernel (e.g. some drivers try to do
optimizations like core-specific queues/lists).

When started in a virtualized environment the obtained data is next to
useless or even wrong, as it is reflecting only the status of the time
of booting the system. Scheduling of the (v)cpus done by the hypervisor
is changing the topology beneath the feet of the Linux kernel without
reflecting this in the gathered topology information. So any decisions
taken based on that data will be clueless and possibly just wrong.

The minimal solution is to change the topology data in the kernel in a
way that all cpus are regarded as equal regarding their relation to each
other (e.g. when migrating a thread to another cpu no cpu is preferred
as a target).

The topology information of the CPUID instruction is, however, even
accessible form user mode and might be used for licensing purposes of
any user program (e.g. by limiting the software to run on a specific
number of cores or sockets). So just mangling the data returned by
CPUID in the hypervisor seems not to be a general solution, while we
might want to do it at least optionally in the future.

In the future we might want to support either dynamic topology updates
or be able to tell the kernel to use some of the topology data, e.g.
when pinning vcpus.


Solution 1 (Dario):
-------------------

Don't use the CPUID derived topology information in the Linux scheduler,
but let it use a simple "flat" topology by setting own scheduler domain
data under Xen.

Advantages:
+ very clean solution regarding the scheduler interface
+ scheduler decisions are based on a minimal data set
+ small patch

Disadvantages:
- covers the scheduler only, drivers still use the "wrong" data
- a little bit hacky regarding some NUMA architectures (needs either a
   hook in the code dealing with that architecture or multiple scheduler
   domain data overwrites)
- future enhancements will make the solution less clean (either need
   duplicating scheduler domain data or some new hooks in scheduler
   domain interface)


Solution 2 (Juergen):
---------------------

When booted as a Xen guest modify the topology data built during boot
resulting in the same simple "flat" topology as in Dario's solution.

Advantages:
+ the simple topology is seen by all consumers of topology data as the
   data itself is modified accordingly
+ small patch
+ future enhancements rather easy by selecting which data to modify

Disadvantages:
- interface to scheduler not as clean as in Dario's approach
- scheduler decisions are based on multiple layers of topology data
   where one layer would be enough to describe the topology


Dario, are you okay with this summary?

Juergen
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1231233

[Multipart message — attachments visible in raw view] — view raw

On Wed, 2015-09-23 at 06:36 +0200, Juergen Gross wrote:

> On 09/22/2015 06:22 PM, George Dunlap wrote:
> > Juergen / Dario, could one of you summarize your two approaches, 
> > and the
> > (alleged) advantages and disadvantages of each one?
> 
> Okay, I'll have a try:
> 
Thanks for this! ;-)

> The problem we want to solve:
> -----------------------------
> 
> The Linux kernel is gathering cpu topology data during boot via the
> CPUID instruction on each processor coming online. This data is
> primarily used in the scheduler to decide to which cpu a thread
> should
> be migrated when this seems to be necessary. There are other users of
> the topology information in the kernel (e.g. some drivers try to do
> optimizations like core-specific queues/lists).
> 
> When started in a virtualized environment the obtained data is next
> to
> useless or even wrong, as it is reflecting only the status of the
> time
> of booting the system. Scheduling of the (v)cpus done by the
> hypervisor
> is changing the topology beneath the feet of the Linux kernel without
> reflecting this in the gathered topology information. So any
> decisions
> taken based on that data will be clueless and possibly just wrong.
> 
Exactly.

> The minimal solution is to change the topology data in the kernel in
> a
> way that all cpus are regarded as equal regarding their relation to
> each
> other (e.g. when migrating a thread to another cpu no cpu is
> preferred
> as a target).
> 
> The topology information of the CPUID instruction is, however, even
> accessible form user mode and might be used for licensing purposes of
> any user program (e.g. by limiting the software to run on a specific
> number of cores or sockets). So just mangling the data returned by
> CPUID in the hypervisor seems not to be a general solution, while we
> might want to do it at least optionally in the future.
> 
Yep. It turned out that, although being what started all this, CPUID
handling is a somewhat related but mostly independent problem. :-)

> In the future we might want to support either dynamic topology
> updates
> or be able to tell the kernel to use some of the topology data, e.g.
> when pinning vcpus.
> 
Indeed. At least for the latter. Dynamic looks really difficult to me,
but indeed it would be ideal. Let's see.

> Solution 1 (Dario):
> -------------------
> 
> Don't use the CPUID derived topology information in the Linux
> scheduler,
> but let it use a simple "flat" topology by setting own scheduler
> domain
> data under Xen.
> 
> Advantages:
> + very clean solution regarding the scheduler interface
>
Yes, this is, I think, one of the main advantages of the patch. The
scheduler is offering an interface to architectures to define their
topology requirements and I'm using it, for specifying our topology
requirements: the tool for the job. :-D

> + scheduler decisions are based on a minimal data set
> + small patch
> 
> Disadvantages:
> - covers the scheduler only, drivers still use the "wrong" data
>
This is a good point. It was the patch's purpose, TBH, but it's
certainly true that, if we need something similar elsewhere, we need to
do more.

> - a little bit hacky regarding some NUMA architectures (needs either
> a
>    hook in the code dealing with that architecture or multiple
> scheduler
>    domain data overwrites)
>
As I said in my other email, I'll double check (yes, I also think this
is about AMD boxes with intra-socket NUMA nodes).

> - future enhancements will make the solution less clean (either need
>    duplicating scheduler domain data or some new hooks in scheduler
>    domain interface)
> 
This one, I'm not sure I understand.

> Solution 2 (Juergen):
> ---------------------
> 
> When booted as a Xen guest modify the topology data built during boot
> resulting in the same simple "flat" topology as in Dario's solution.
> 
> Advantages:
> + the simple topology is seen by all consumers of topology data as
> the
>    data itself is modified accordingly
>
Yep, that's a good point.

> + small patch

> + future enhancements rather easy by selecting which data to modify
>
As for the '-' above about this, I'm not really sure what this means.
> 
> Disadvantages:
> - interface to scheduler not as clean as in Dario's approach
> - scheduler decisions are based on multiple layers of topology data
>    where one layer would be enough to describe the topology
> 
This is not too big of a deal, IMO. Not at runtime, at least, as far as
my investigation went for now. Initialization (of scheduling domains)
is a bit clumsy in this case, as scheduling domains are created and
then destroyed/collapsed, but after they are setup, the net effect is
that there's only one scheduling domain with Juergen's patch too,
exactly as with mine.

> Dario, are you okay with this summary?
>
To most of it, yes, and thanks again for it.

Allow me to add a few points, out of the top of my head:

 * we need to check whether the two approaches have the same 
   performance. In principle, they really should, and early results 
   seems to confirm that, but I'd like to run the full set of benches 
   (and I'll do that ASAP);
 * I think we want to run even more benchmarks, and run them in    
   different (over)load conditions to better assess the effect of the 
   change
 * both our patches provides a solution for Xen (for Xen PV guests, at 
   least for now, to be more precise). It is very likely that, e.g., 
   KVM is in a similar situation, hence it may be worth to look for a 
   more general solution, especially if that buys us something (e.g., 
   HVM support made easy?)

Thanks and Regards,
Dario

PS. BTW, Juergen, you're not on IRC, on #xendevel, are you?

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[toc] | [prev] | [next] | [standalone]

#1231295 — Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy

On 09/23/2015 10:30 AM, Dario Faggioli wrote:
> On Wed, 2015-09-23 at 06:36 +0200, Juergen Gross wrote:
>
>> On 09/22/2015 06:22 PM, George Dunlap wrote:
>>> Juergen / Dario, could one of you summarize your two approaches,
>>> and the
>>> (alleged) advantages and disadvantages of each one?
>>
>> Okay, I'll have a try:
>>
> Thanks for this! ;-)
>
>> The problem we want to solve:
>> -----------------------------
>>
>> The Linux kernel is gathering cpu topology data during boot via the
>> CPUID instruction on each processor coming online. This data is
>> primarily used in the scheduler to decide to which cpu a thread
>> should
>> be migrated when this seems to be necessary. There are other users of
>> the topology information in the kernel (e.g. some drivers try to do
>> optimizations like core-specific queues/lists).
>>
>> When started in a virtualized environment the obtained data is next
>> to
>> useless or even wrong, as it is reflecting only the status of the
>> time
>> of booting the system. Scheduling of the (v)cpus done by the
>> hypervisor
>> is changing the topology beneath the feet of the Linux kernel without
>> reflecting this in the gathered topology information. So any
>> decisions
>> taken based on that data will be clueless and possibly just wrong.
>>
> Exactly.
>
>> The minimal solution is to change the topology data in the kernel in
>> a
>> way that all cpus are regarded as equal regarding their relation to
>> each
>> other (e.g. when migrating a thread to another cpu no cpu is
>> preferred
>> as a target).
>>
>> The topology information of the CPUID instruction is, however, even
>> accessible form user mode and might be used for licensing purposes of
>> any user program (e.g. by limiting the software to run on a specific
>> number of cores or sockets). So just mangling the data returned by
>> CPUID in the hypervisor seems not to be a general solution, while we
>> might want to do it at least optionally in the future.
>>
> Yep. It turned out that, although being what started all this, CPUID
> handling is a somewhat related but mostly independent problem. :-)
>
>> In the future we might want to support either dynamic topology
>> updates
>> or be able to tell the kernel to use some of the topology data, e.g.
>> when pinning vcpus.
>>
> Indeed. At least for the latter. Dynamic looks really difficult to me,
> but indeed it would be ideal. Let's see.
>
>> Solution 1 (Dario):
>> -------------------
>>
>> Don't use the CPUID derived topology information in the Linux
>> scheduler,
>> but let it use a simple "flat" topology by setting own scheduler
>> domain
>> data under Xen.
>>
>> Advantages:
>> + very clean solution regarding the scheduler interface
>>
> Yes, this is, I think, one of the main advantages of the patch. The
> scheduler is offering an interface to architectures to define their
> topology requirements and I'm using it, for specifying our topology
> requirements: the tool for the job. :-D
>
>> + scheduler decisions are based on a minimal data set
>> + small patch
>>
>> Disadvantages:
>> - covers the scheduler only, drivers still use the "wrong" data
>>
> This is a good point. It was the patch's purpose, TBH, but it's
> certainly true that, if we need something similar elsewhere, we need to
> do more.
>
>> - a little bit hacky regarding some NUMA architectures (needs either
>> a
>>     hook in the code dealing with that architecture or multiple
>> scheduler
>>     domain data overwrites)
>>
> As I said in my other email, I'll double check (yes, I also think this
> is about AMD boxes with intra-socket NUMA nodes).
>
>> - future enhancements will make the solution less clean (either need
>>     duplicating scheduler domain data or some new hooks in scheduler
>>     domain interface)
>>
> This one, I'm not sure I understand.

What would you do for keeping the topology information of one level,
e.g. hyperthreads, in case we'd have a gang-scheduler in Xen? Either
you would copy the line:

{ cpu_smt_mask, cpu_smt_flags, SD_INIT_NAME(SMT) },

from kernel/sched/core.c into your topology array, or you would add a
way in kernel/sched/core.c to remove all but this entry and add your
entry on top of it.

>
>> Solution 2 (Juergen):
>> ---------------------
>>
>> When booted as a Xen guest modify the topology data built during boot
>> resulting in the same simple "flat" topology as in Dario's solution.
>>
>> Advantages:
>> + the simple topology is seen by all consumers of topology data as
>> the
>>     data itself is modified accordingly
>>
> Yep, that's a good point.
>
>> + small patch
>
>> + future enhancements rather easy by selecting which data to modify
>>
> As for the '-' above about this, I'm not really sure what this means.

In the case mentioned above I just wouldn't zap the
topology_sibling_cpumask in my patch.

>>
>> Disadvantages:
>> - interface to scheduler not as clean as in Dario's approach
>> - scheduler decisions are based on multiple layers of topology data
>>     where one layer would be enough to describe the topology
>>
> This is not too big of a deal, IMO. Not at runtime, at least, as far as
> my investigation went for now. Initialization (of scheduling domains)
> is a bit clumsy in this case, as scheduling domains are created and
> then destroyed/collapsed, but after they are setup, the net effect is
> that there's only one scheduling domain with Juergen's patch too,
> exactly as with mine.
>
>> Dario, are you okay with this summary?
>>
> To most of it, yes, and thanks again for it.
>
> Allow me to add a few points, out of the top of my head:
>
>   * we need to check whether the two approaches have the same
>     performance. In principle, they really should, and early results
>     seems to confirm that, but I'd like to run the full set of benches
>     (and I'll do that ASAP);

Thanks.

>   * I think we want to run even more benchmarks, and run them in
>     different (over)load conditions to better assess the effect of the
>     change
>   * both our patches provides a solution for Xen (for Xen PV guests, at
>     least for now, to be more precise). It is very likely that, e.g.,
>     KVM is in a similar situation, hence it may be worth to look for a
>     more general solution, especially if that buys us something (e.g.,
>     HVM support made easy?)

I wanted to look at this as soon as we've decided which way to go.

I had some discussion with a kvm guy last week and he seemed not to be
convinced they need something else as mangling CPUID (what they already
do).

>
> Thanks and Regards,
> Dario
>
> PS. BTW, Juergen, you're not on IRC, on #xendevel, are you?

I'd like to, but I'd need an invitation. My user name is juergen_gross.


Juergen
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1231312 — Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy

On 09/23/2015 05:36 AM, Juergen Gross wrote:
> On 09/22/2015 06:22 PM, George Dunlap wrote:
>> On 09/22/2015 05:42 AM, Juergen Gross wrote:
>>> One other thing I just discovered: there are other consumers of the
>>> topology sibling masks (e.g. topology_sibling_cpumask()) as well.
>>>
>>> I think we would want to avoid any optimizations based on those in
>>> drivers as well, not only in the scheduler.
>>
>> I'm beginning to lose the thread of the discussion here a bit.
>>
>> Juergen / Dario, could one of you summarize your two approaches, and the
>> (alleged) advantages and disadvantages of each one?
> 
> Okay, I'll have a try:
> 
> The problem we want to solve:
> -----------------------------
> 
> The Linux kernel is gathering cpu topology data during boot via the
> CPUID instruction on each processor coming online. This data is
> primarily used in the scheduler to decide to which cpu a thread should
> be migrated when this seems to be necessary. There are other users of
> the topology information in the kernel (e.g. some drivers try to do
> optimizations like core-specific queues/lists).
> 
> When started in a virtualized environment the obtained data is next to
> useless or even wrong, as it is reflecting only the status of the time
> of booting the system. Scheduling of the (v)cpus done by the hypervisor
> is changing the topology beneath the feet of the Linux kernel without
> reflecting this in the gathered topology information. So any decisions
> taken based on that data will be clueless and possibly just wrong.
> 
> The minimal solution is to change the topology data in the kernel in a
> way that all cpus are regarded as equal regarding their relation to each
> other (e.g. when migrating a thread to another cpu no cpu is preferred
> as a target).
> 
> The topology information of the CPUID instruction is, however, even
> accessible form user mode and might be used for licensing purposes of
> any user program (e.g. by limiting the software to run on a specific
> number of cores or sockets). So just mangling the data returned by
> CPUID in the hypervisor seems not to be a general solution, while we
> might want to do it at least optionally in the future.
> 
> In the future we might want to support either dynamic topology updates
> or be able to tell the kernel to use some of the topology data, e.g.
> when pinning vcpus.
> 
> 
> Solution 1 (Dario):
> -------------------
> 
> Don't use the CPUID derived topology information in the Linux scheduler,
> but let it use a simple "flat" topology by setting own scheduler domain
> data under Xen.
> 
> Advantages:
> + very clean solution regarding the scheduler interface
> + scheduler decisions are based on a minimal data set
> + small patch
> 
> Disadvantages:
> - covers the scheduler only, drivers still use the "wrong" data
> - a little bit hacky regarding some NUMA architectures (needs either a
>   hook in the code dealing with that architecture or multiple scheduler
>   domain data overwrites)
> - future enhancements will make the solution less clean (either need
>   duplicating scheduler domain data or some new hooks in scheduler
>   domain interface)
> 
> 
> Solution 2 (Juergen):
> ---------------------
> 
> When booted as a Xen guest modify the topology data built during boot
> resulting in the same simple "flat" topology as in Dario's solution.
> 
> Advantages:
> + the simple topology is seen by all consumers of topology data as the
>   data itself is modified accordingly
> + small patch
> + future enhancements rather easy by selecting which data to modify
> 
> Disadvantages:
> - interface to scheduler not as clean as in Dario's approach
> - scheduler decisions are based on multiple layers of topology data
>   where one layer would be enough to describe the topology
> 
> 
> Dario, are you okay with this summary?

Thanks -- that's very helpful.

 -George
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1231190

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2015-09-21 at 07:49 +0200, Juergen Gross wrote:
> On 09/15/2015 06:50 PM, Dario Faggioli wrote:
> > On Thu, 2015-08-20 at 20:16 +0200, Juergen Groß wrote:
> > > On 08/18/2015 05:55 PM, Dario Faggioli wrote:
> > > > Hey everyone,
> > > > 
> > > > So, as a followup of what we were discussing in this thread:
> > > > 
> > > >    [Xen-devel] PV-vNUMA issue: topology is misinterpreted by
> > > > the guest
> > > >    http://lists.xenproject.org/archives/html/xen-devel/2015-07/
> > > > msg03241.html
> > > > 
> > > > I started looking in more details at scheduling domains in the
> > > > Linux
> > > > kernel. Now, that thread was about CPUID and vNUMA, and their
> > > > weird way
> > > > of interacting, while this thing I'm proposing here is
> > > > completely
> > > > independent from them both.
> > > > 
> > > > In fact, no matter whether vNUMA is supported and enabled, and
> > > > no matter
> > > > whether CPUID is reporting accurate, random, meaningful or
> > > > completely
> > > > misleading information, I think that we should do something
> > > > about how
> > > > scheduling domains are build.
> > > > 
> > > > Fact is, unless we use 1:1, and immutable (across all the guest
> > > > lifetime) pinning, scheduling domains should not be
> > > > constructed, in
> > > > Linux, by looking at *any* topology information, because that
> > > > just does
> > > > not make any sense, when vcpus move around.
> > > > 
> > > > Let me state this again (hoping to make myself as clear as
> > > > possible): no
> > > > matter in  how much good shape we put CPUID support, no matter
> > > > how
> > > > beautifully and consistently that will interact with both
> > > > vNUMA,
> > > > licensing requirements and whatever else. It will be always
> > > > possible for
> > > > vCPU #0 and vCPU #3 to be scheduled on two SMT threads at time
> > > > t1, and
> > > > on two different NUMA nodes at time t2. Hence, the Linux
> > > > scheduler
> > > > should really not skew his load balancing logic toward any of
> > > > those two
> > > > situations, as neither of them could be considered correct
> > > > (since
> > > > nothing is!).
> > > > 
> > > > For now, this only covers the PV case. HVM case shouldn't be
> > > > any
> > > > different, but I haven't looked at how to make the same thing
> > > > happen in
> > > > there as well.
> > > > 
> > > > OVERALL DESCRIPTION
> > > > ===================
> > > > What this RFC patch does is, in the Xen PV case, configure
> > > > scheduling
> > > > domains in such a way that there is only one of them, spanning
> > > > all the
> > > > pCPUs of the guest.
> > > > 
> > > > Note that the patch deals directly with scheduling domains, and
> > > > there is
> > > > no need to alter the masks that will then be used for building
> > > > and
> > > > reporting the topology (via CPUID, /proc/cpuinfo, /sysfs,
> > > > etc.). That is
> > > > the main difference between it and the patch proposed by
> > > > Juergen here:
> > > > http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg
> > > > 05088.html
> > > > 
> > > > This means that when, in future, we will fix CPUID handling and
> > > > make it
> > > > comply with whatever logic or requirements we want, that won't
> > > > have  any
> > > > unexpected side effects on scheduling domains.
> > > > 
> > > > Information about how the scheduling domains are being
> > > > constructed
> > > > during boot are available in `dmesg', if the kernel is booted
> > > > with the
> > > > 'sched_debug' parameter. It is also possible to look
> > > > at /proc/sys/kernel/sched_domain/cpu*, and at /proc/schedstat.
> > > > 
> > > > With the patch applied, only one scheduling domain is created,
> > > > called
> > > > the 'VCPU' domain, spanning all the guest's (or Dom0's) vCPUs.
> > > > You can
> > > > tell that from the fact that every cpu* folder
> > > > in /proc/sys/kernel/sched_domain/ only have one subdirectory
> > > > ('domain0'), with all the tweaks and the tunables for our
> > > > scheduling
> > > > domain.
> > > > 
> > > > EVALUATION
> > > > ==========
> > > > I've tested this with UnixBench, and by looking at Xen build
> > > > time, on a
> > > > 16, 24 and 48 pCPUs hosts. I've run the benchmarks in Dom0
> > > > only, for
> > > > now, but I plan to re-run them in DomUs soon (Juergen may be
> > > > doing
> > > > something similar to this in DomU already, AFAUI).
> > > > 
> > > > I've run the benchmarks with and without the patch applied
> > > > ('patched'
> > > > and 'vanilla', respectively, in the tables below), and with
> > > > different
> > > > number of build jobs (in case of the Xen build) or of parallel
> > > > copy of
> > > > the benchmarks (in the case of UnixBench).
> > > > 
> > > > What I get from the numbers is that the patch almost always
> > > > brings
> > > > benefits, in some cases even huge ones. There are a couple of
> > > > cases
> > > > where we regress, but always only slightly so, especially if
> > > > comparing
> > > > that to the magnitude of some of the improvement that we get.
> > > > 
> > > > Bear also in mind that these results are gathered from Dom0,
> > > > and without
> > > > any overcommitment at the vCPU level (i.e., nr. vCPUs == nr
> > > > pCPUs). If
> > > > we move things in DomU and do overcommit at the Xen scheduler
> > > > level, I
> > > > am expecting even better results.
> > > > 
> > > ...
> > > > REQUEST FOR COMMENTS
> > > > ====================
> > > > Basically, the kind of feedback I'd be really glad to hear is:
> > > >    - what you guys thing of the approach,
> > > 
> > > Yesterday at the end of the developer meeting we (Andrew, Elena
> > > and
> > > myself) discussed this topic again.
> > > 
> > Hey,
> > 
> > Sorry for replying so late, I've been on vacation from right after
> > XenSummit up until yesterday. :-)
> > 
> > > Regarding a possible future scenario with credit2 eventually
> > > supporting
> > > gang scheduling on hyperthreads (which is desirable due to
> > > security
> > > reasons [side channel attack] and fairness) my patch seems to be
> > > more
> > > suited for that direction than yours.
> > > 
> > Ok. Just let me mention that 'Credit2 + gang scheduling' might not
> > be
> > exactly around the corner (although, we can prioritize working on
> > it if
> > we want).
> > 
> > In principle, I think it's a really nice idea. I still don't have
> > clear
> > in mind how we would handle a couple of situations, but let's leave
> > this
> > aside for now, and stay on-topic.
> > 
> > > Correct me if I'm wrong, but I
> > > think scheduling domains won't enable the guest kernel's
> > > scheduler to
> > > migrate threads more easily between hyperthreads opposed to other
> > > vcpus,
> > > while my approach can easily be extended to do so.
> > > 
> > I'm not sure I understand what you mean here. As far as the (Linux)
> > scheduler is concerned, your patch and mine do the exact same
> > thing:
> > they arrange for the scheduling domains, when they're built, during
> > boot, not to consider hyperthreads or multi-cores.
> > 
> > Mine does it by removing the SMT (and the MC) level from the data
> > structure in the scheduler that is used as a base for configuring
> > the
> > scheduling domains. Yours does it by making the topology bitmaps
> > that
> > are used at each one of those level all look the same. In fact,
> > with
> > your patch applied, I get the exact same situation as with mine, as
> > far
> > as scheduling domains are concerned: there is only one scheduling
> > domain, with a different scheduling group for each vCPU inside it.
> 
> Uuh, nearly.
> 
> Your case won't deal correctly with NUMA, as the generic NUMA code is
> using set_sched_topology() as well. 
>
Mmm... have you tried and seen something like this? AFAICT, the NUMA
related setup steps of scheduling domains happens after the basic (as
in "without taking NUMAness into account") topology has been set
already, and builds on top of it.

It uses set_sched_topology() only in a special case which, I'm not sure
we'd be hitting.

I'm asking because trying this out, right now, is not straightforward,
as PV vNUMA, even with Wei's Linux patches and with either yours or
mine one, still incurs in the CPUID issue... I'll try that ASAP, but
there are a couple of things I've got to finish for the next few days.

> One of NUMA and Xen will win and
> overwrite the other's settings.
> 
Not sure what this means, but as I said, I'll try.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[toc] | [prev] | [next] | [standalone]

#1231199 — Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy

On 09/23/2015 09:24 AM, Dario Faggioli wrote:
> On Mon, 2015-09-21 at 07:49 +0200, Juergen Gross wrote:
>> On 09/15/2015 06:50 PM, Dario Faggioli wrote:
>>> On Thu, 2015-08-20 at 20:16 +0200, Juergen Groß wrote:
>>>> On 08/18/2015 05:55 PM, Dario Faggioli wrote:
>>>>> Hey everyone,
>>>>>
>>>>> So, as a followup of what we were discussing in this thread:
>>>>>
>>>>>     [Xen-devel] PV-vNUMA issue: topology is misinterpreted by
>>>>> the guest
>>>>>     http://lists.xenproject.org/archives/html/xen-devel/2015-07/
>>>>> msg03241.html
>>>>>
>>>>> I started looking in more details at scheduling domains in the
>>>>> Linux
>>>>> kernel. Now, that thread was about CPUID and vNUMA, and their
>>>>> weird way
>>>>> of interacting, while this thing I'm proposing here is
>>>>> completely
>>>>> independent from them both.
>>>>>
>>>>> In fact, no matter whether vNUMA is supported and enabled, and
>>>>> no matter
>>>>> whether CPUID is reporting accurate, random, meaningful or
>>>>> completely
>>>>> misleading information, I think that we should do something
>>>>> about how
>>>>> scheduling domains are build.
>>>>>
>>>>> Fact is, unless we use 1:1, and immutable (across all the guest
>>>>> lifetime) pinning, scheduling domains should not be
>>>>> constructed, in
>>>>> Linux, by looking at *any* topology information, because that
>>>>> just does
>>>>> not make any sense, when vcpus move around.
>>>>>
>>>>> Let me state this again (hoping to make myself as clear as
>>>>> possible): no
>>>>> matter in  how much good shape we put CPUID support, no matter
>>>>> how
>>>>> beautifully and consistently that will interact with both
>>>>> vNUMA,
>>>>> licensing requirements and whatever else. It will be always
>>>>> possible for
>>>>> vCPU #0 and vCPU #3 to be scheduled on two SMT threads at time
>>>>> t1, and
>>>>> on two different NUMA nodes at time t2. Hence, the Linux
>>>>> scheduler
>>>>> should really not skew his load balancing logic toward any of
>>>>> those two
>>>>> situations, as neither of them could be considered correct
>>>>> (since
>>>>> nothing is!).
>>>>>
>>>>> For now, this only covers the PV case. HVM case shouldn't be
>>>>> any
>>>>> different, but I haven't looked at how to make the same thing
>>>>> happen in
>>>>> there as well.
>>>>>
>>>>> OVERALL DESCRIPTION
>>>>> ===================
>>>>> What this RFC patch does is, in the Xen PV case, configure
>>>>> scheduling
>>>>> domains in such a way that there is only one of them, spanning
>>>>> all the
>>>>> pCPUs of the guest.
>>>>>
>>>>> Note that the patch deals directly with scheduling domains, and
>>>>> there is
>>>>> no need to alter the masks that will then be used for building
>>>>> and
>>>>> reporting the topology (via CPUID, /proc/cpuinfo, /sysfs,
>>>>> etc.). That is
>>>>> the main difference between it and the patch proposed by
>>>>> Juergen here:
>>>>> http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg
>>>>> 05088.html
>>>>>
>>>>> This means that when, in future, we will fix CPUID handling and
>>>>> make it
>>>>> comply with whatever logic or requirements we want, that won't
>>>>> have  any
>>>>> unexpected side effects on scheduling domains.
>>>>>
>>>>> Information about how the scheduling domains are being
>>>>> constructed
>>>>> during boot are available in `dmesg', if the kernel is booted
>>>>> with the
>>>>> 'sched_debug' parameter. It is also possible to look
>>>>> at /proc/sys/kernel/sched_domain/cpu*, and at /proc/schedstat.
>>>>>
>>>>> With the patch applied, only one scheduling domain is created,
>>>>> called
>>>>> the 'VCPU' domain, spanning all the guest's (or Dom0's) vCPUs.
>>>>> You can
>>>>> tell that from the fact that every cpu* folder
>>>>> in /proc/sys/kernel/sched_domain/ only have one subdirectory
>>>>> ('domain0'), with all the tweaks and the tunables for our
>>>>> scheduling
>>>>> domain.
>>>>>
>>>>> EVALUATION
>>>>> ==========
>>>>> I've tested this with UnixBench, and by looking at Xen build
>>>>> time, on a
>>>>> 16, 24 and 48 pCPUs hosts. I've run the benchmarks in Dom0
>>>>> only, for
>>>>> now, but I plan to re-run them in DomUs soon (Juergen may be
>>>>> doing
>>>>> something similar to this in DomU already, AFAUI).
>>>>>
>>>>> I've run the benchmarks with and without the patch applied
>>>>> ('patched'
>>>>> and 'vanilla', respectively, in the tables below), and with
>>>>> different
>>>>> number of build jobs (in case of the Xen build) or of parallel
>>>>> copy of
>>>>> the benchmarks (in the case of UnixBench).
>>>>>
>>>>> What I get from the numbers is that the patch almost always
>>>>> brings
>>>>> benefits, in some cases even huge ones. There are a couple of
>>>>> cases
>>>>> where we regress, but always only slightly so, especially if
>>>>> comparing
>>>>> that to the magnitude of some of the improvement that we get.
>>>>>
>>>>> Bear also in mind that these results are gathered from Dom0,
>>>>> and without
>>>>> any overcommitment at the vCPU level (i.e., nr. vCPUs == nr
>>>>> pCPUs). If
>>>>> we move things in DomU and do overcommit at the Xen scheduler
>>>>> level, I
>>>>> am expecting even better results.
>>>>>
>>>> ...
>>>>> REQUEST FOR COMMENTS
>>>>> ====================
>>>>> Basically, the kind of feedback I'd be really glad to hear is:
>>>>>     - what you guys thing of the approach,
>>>>
>>>> Yesterday at the end of the developer meeting we (Andrew, Elena
>>>> and
>>>> myself) discussed this topic again.
>>>>
>>> Hey,
>>>
>>> Sorry for replying so late, I've been on vacation from right after
>>> XenSummit up until yesterday. :-)
>>>
>>>> Regarding a possible future scenario with credit2 eventually
>>>> supporting
>>>> gang scheduling on hyperthreads (which is desirable due to
>>>> security
>>>> reasons [side channel attack] and fairness) my patch seems to be
>>>> more
>>>> suited for that direction than yours.
>>>>
>>> Ok. Just let me mention that 'Credit2 + gang scheduling' might not
>>> be
>>> exactly around the corner (although, we can prioritize working on
>>> it if
>>> we want).
>>>
>>> In principle, I think it's a really nice idea. I still don't have
>>> clear
>>> in mind how we would handle a couple of situations, but let's leave
>>> this
>>> aside for now, and stay on-topic.
>>>
>>>> Correct me if I'm wrong, but I
>>>> think scheduling domains won't enable the guest kernel's
>>>> scheduler to
>>>> migrate threads more easily between hyperthreads opposed to other
>>>> vcpus,
>>>> while my approach can easily be extended to do so.
>>>>
>>> I'm not sure I understand what you mean here. As far as the (Linux)
>>> scheduler is concerned, your patch and mine do the exact same
>>> thing:
>>> they arrange for the scheduling domains, when they're built, during
>>> boot, not to consider hyperthreads or multi-cores.
>>>
>>> Mine does it by removing the SMT (and the MC) level from the data
>>> structure in the scheduler that is used as a base for configuring
>>> the
>>> scheduling domains. Yours does it by making the topology bitmaps
>>> that
>>> are used at each one of those level all look the same. In fact,
>>> with
>>> your patch applied, I get the exact same situation as with mine, as
>>> far
>>> as scheduling domains are concerned: there is only one scheduling
>>> domain, with a different scheduling group for each vCPU inside it.
>>
>> Uuh, nearly.
>>
>> Your case won't deal correctly with NUMA, as the generic NUMA code is
>> using set_sched_topology() as well.
>>
> Mmm... have you tried and seen something like this? AFAICT, the NUMA
> related setup steps of scheduling domains happens after the basic (as
> in "without taking NUMAness into account") topology has been set
> already, and builds on top of it.
>
> It uses set_sched_topology() only in a special case which, I'm not sure
> we'd be hitting.

Depends on the hardware. On some AMD processors one socket covers
multiple NUMA nodes. This is the critical case. set_sched_topology()
will be called on those machines possibly multiple times when bringing
up additional cpus.

> I'm asking because trying this out, right now, is not straightforward,
> as PV vNUMA, even with Wei's Linux patches and with either yours or
> mine one, still incurs in the CPUID issue... I'll try that ASAP, but
> there are a couple of things I've got to finish for the next few days.
>
>> One of NUMA and Xen will win and
>> overwrite the other's settings.
>>
> Not sure what this means, but as I said, I'll try.

Make sure to use the correct hardware (I'm pretty sure this should be
the AMD "Magny-Cours" [1]).


Juergen

[1]: 
http://developer.amd.com/resources/documentation-articles/articles-whitepapers/introduction-to-magny-cours/

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1231409 — Re: [Xen-devel] [PATCH RFC] xen: if on Xen, "flatten" the scheduling domain hierarchy

On 09/23/2015 03:35 AM, Juergen Gross wrote:
>
> Depends on the hardware. On some AMD processors one socket covers
> multiple NUMA nodes. This is the critical case. set_sched_topology()
> will be called on those machines possibly multiple times when bringing
> up additional cpus.
>
>> I'm asking because trying this out, right now, is not straightforward,
>> as PV vNUMA, even with Wei's Linux patches and with either yours or
>> mine one, still incurs in the CPUID issue... I'll try that ASAP, but
>> there are a couple of things I've got to finish for the next few days.
>>
>>> One of NUMA and Xen will win and
>>> overwrite the other's settings.
>>>
>> Not sure what this means, but as I said, I'll try.
>
> Make sure to use the correct hardware (I'm pretty sure this should be
> the AMD "Magny-Cours" [1]).
>
>
> Juergen
>
> [1]: 
> http://developer.amd.com/resources/documentation-articles/articles-whitepapers/introduction-to-magny-cours/
>


There are few family 0x10 and 0x15 processors that are like that. You 
can see whether you have such a system by comparing number of NUMA nodes 
with number of physical IDs, e.g.:

[root@ovs106 ~]# numactl --hardware |grep available
available: 4 nodes (0-3)
[root@ovs106 ~]# grep "physical id" /proc/cpuinfo | uniq
physical id    : 0
physical id    : 1
[root@ovs106 ~]#


-boris
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web