Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1211303 > unrolled thread

Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-21 21:30 +0200
    Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-22 20:40 +0200
      Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Austin S Hemmelgarn <ahferroin7@gmail.com> - 2015-08-24 17:50 +0200
        Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-24 19:10 +0200
          Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Mike Galbraith <umgwanakikbuti@gmail.com> - 2015-08-24 21:30 +0200
          Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Austin S Hemmelgarn <ahferroin7@gmail.com> - 2015-08-24 22:10 +0200
            Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-24 22:30 +0200
              Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-24 23:10 +0200
                Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-24 23:20 +0200
                  Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-24 23:20 +0200
          Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-24 23:00 +0200
            Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-24 23:10 +0200
              Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-24 23:20 +0200
                Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-24 23:30 +0200
                  Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-24 23:50 +0200
                    Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-25 00:10 +0200
                      Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-25 01:00 +0200
                        Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-25 01:20 +0200
                          Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> - 2015-08-25 04:40 +0200
                            Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-25 23:20 +0200
                          Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Ingo Molnar <mingo@kernel.org> - 2015-08-25 11:30 +0200
                            Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Peter Zijlstra <peterz@infradead.org> - 2015-08-25 12:10 +0200
                          Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-25 21:20 +0200
              Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-24 23:20 +0200
      Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-24 23:00 +0200
        Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-24 23:40 +0200
          Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-25 00:00 +0200
            Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-25 00:20 +0200
              Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy Paul Turner <pjt@google.com> - 2015-08-25 01:10 +0200
                Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-08-25 23:10 +0200
                  Re: [PATCH 3/3] sched: Implement interface for cgroup unified  hierarchy Tejun Heo <tj@kernel.org> - 2015-09-02 19:10 +0200

Page 1 of 2  [1] 2  Next page →

#1211303 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

On Tue, Aug 18, 2015 at 1:31 PM, Tejun Heo <tj@kernel.org> wrote:
> Hello, Paul.
>
> On Mon, Aug 17, 2015 at 09:03:30PM -0700, Paul Turner wrote:
>> > 2) Control within an address-space.  For subsystems with fungible resources,
>> > e.g. CPU, it can be useful for an address space to partition its own
>> > threads.  Losing the capability to do this against the CPU controller would
>> > be a large set-back for instance.  Occasionally, it is useful to share these
>> > groupings between address spaces when processes are cooperative, but this is
>> > less of a requirement.
>> >
>> > This is important to us.
>
> Sure, let's build a proper interface for that.  Do you actually need
> sub-hierarchy inside a process?  Can you describe your use case in
> detail and why having hierarchical CPU cycle distribution is essential
> for your use case?
>

One common example here is a thread-pool.  Having a hierarchical
constraint allows users to specify what proportion of time it should
receive, independent of how many threads are placed in the pool.

A very concrete example of the above is a virtual machine in which you
want to guarantee scheduling for the vCPU threads which must schedule
beside many hypervisor support threads.   A hierarchy is the only way
to fix the ratio at which these compete.

An example that's not the cpu controller is that we use cpusets to
expose to applications their "shared" and "private" cores.  (These
sets are dynamic based on what is coscheduled on a given machine.)

>> >> And that's one of the major fuck ups on cgroup's part that must be
>> >> rectified.  Look at the interface being proposed there.  It's exposing
>> >> direct hardware details w/o much abstraction which is fine for a
>> >> system management interface but at the same time it's intended to be
>> >> exposed to individual applications.
>> >
>> > FWIW this is something we've had no significant problems managing with
>> > separate mount mounts and file system protections.  Yes, there are some
>> > potential warts around atomicity; but we've not found them too onerous.
>
> You guys control the whole stack.  Of course, you can get away with an
> interface which are pretty messed up in terms of layering and
> isolation; however, generic kernel interface cannot be designed
> according to that standard.

I feel like two points are being conflated here:

Yes, it is sufficiently generic that it's possible to configure
nonsensical things.

But, it is also possible to lock things down presently.  This is, for
better or worse, the direction that general user-space has also taken
with centralized management daemons such as systemd.

Setting design aside for a moment -- which I fully agree with you that
there is room for large improvement in.  The largest idiosyncrasy
today is that the configuration above does depend on having a stable
mount point for applications to manage their sub-hierarchies.
Migrations would improve this greatly, but this is a bit of a detour
because you're looking to fix the fundamental design rather than
improve the state of the world and that's probably a good thing :)

>
>> > What I don't quite follow here is the assumption that CAT should would be
>> > necessarily exposed to individual applications? What's wrong with subsystems
>> > that are primarily intended only for system management agents, we already
>> > have several of these.
>
> Why would you assume that threads of a process wouldn't want to
> configure it ever?  How is this different from CPU affinity?

In general cache and CPU behave differently.  Generally for it to make
sense between threads in a process they would have to have wholly
disjoint memory, at which point the only sane long-term implementation
is separate processes and the management moves up a level anyway.

That said, there are surely cases in which it might be convenient to
use at a per-thread level to correct a specific performance anomaly.
But at that point, you have certainly reached the level of hammer that
you can coordinate with an external daemon if necessary.

>
>> >> This lack of distinction makes
>> >> people skip the attention that they should be paying when they're
>> >> designing interface exposed to individual programs.  Worse, this makes
>> >> these things fly under the review scrutiny that public API accessible
>> >> to applications usually receives.  Yet, that's what these things end
>> >> up to be.  This just has to stop.  cgroups can't continue to be this
>> >> ghetto shortcut to implementing half-assed APIs.
>> >
>> > I certainly don't disagree on this point :).  But as above, I don't quite
>> > follow why an API being in cgroups must mean it's accessible to an
>> > application controlled by that group.  This has certainly not been a
>> > requirement for our use.
>
> I don't follow what you're trying to way with the above paragraph.
> Are you still talking about CAT?  If so, that use case isn't the only
> one.  I'm pretty sure there are people who would want to configure
> cache allocation at thread level.

I'm not agreeing with you that "in cgroups" means "must be usable by
applications within that hierarchy".  A cgroup subsystem used as a
partitioning API only by system management daemons is entirely
reasonable.  CAT is a reasonable example of this.

>
>> >> What we should be doing is pushing them into the same arena as any
>> >> other publicly accessible API.  I don't think there can be a shortcut
>> >> to this.
>> >
>> > Are you explicitly opposed to non-hierarchical partitions, however?  Cpuset
>> > is [typically] an example of this, where the interface wants to control
>> > unified properties across a set of processes.  Without necessarily being
>> > usefully hierarchical.  (This is just to understand your core position, I'm
>> > not proposing cpuset should shape *anything*.)
>
> I'm having trouble following what you're trying to say.  FWIW, cpuset
> is fully hierarchical.

I think where I was going with this is better addressed above.  Here
all I meant is that it's difficult to construct useful sub-hierarchies
on the cpuset side, especially for memory.   But this is a little
x86-centric so let's drop it.

>
>> >> I don't think we want migration in sub-process hierarchy but in the
>> >> off chance we do the naming can follow the same pid/program
>> >> group/session id scheme, which, again, is a lot easier to deal with
>> >> from applications.
>> >
>> > I don't have many objections with hand-off versus migration above, however,
>> > I think that this is a big drawback.  Threads are expensive to create and
>> > are often cached rather than released.  While migration may be expensive,
>> > creating a more thread is more so.  The important to reconfigure a thread's
>> > personality at run-time is important.
>
> The core problem here is picking the hot path.  If cgroups as a whole
> doesn't pick a position here, controllers have to assume that
> migration might not be a very cold path which naturally leads to
> overall designs and synchronization schemes which concede hot path
> performance to accomodate migration.  We simply can't afford to do
> that - we end up losing way more in way hotter paths for something
> which may be marginally useful in some corner cases.
>
> So, this is a trade-off we're consciously making.  If there are
> common-enough use cases which require jumping across different cgroup
> domains, we'll try to figure out a way to accomodate those but by
> default migration is a very cold and expensive path.
>

The core here was the need for allowing sub-process migration.  I'm
not sure I follow the performance trade-off argument; haven't we
historically seen the opposite?  That migration has been a slow-path
without optimizations and people pushing to make it faster?  This
seems a hard generalization to make for something that's inherently
tied to a particular controller.

I don't care if we try turning that dial back to assume it's a cold
path once more, only that it's supported.

>> >> But those are relative to the current directory per operation and
>> >> there's no way to define a transaction across multiple file
>> >> operations.  There's no way to prevent a process from being migrated
>> >> inbetween openat() and subsequent write().
>> >
>> > A forwarding /proc/thread_self/cgroup accessor, or similar, would be another
>> > way to address some of these issues.
>
> That sounds horrible to me.  What if the process wants to do RMW a
> config?

Locking within a process is easy.

> What if the permissions are different after an intervening
> migration?

This is a side-effect of migration not being properly supported.

> What if the sub-hierarchy no longer exists or has been
> replaced by a hierarchy with the same topology but actualy is a
> different one?

The easy answer is that:  Only a process should be managing its
sub-hierarchy.  That's the nice thing about hierarchies.

The harder answer is:  How do we handle non-fungible resources such as
CPU assignments within a hierarchy?  This is a big part of why I make
arguments for certain partitions being management-software only above.
This is imperfect, but better then where we stand today.

>
>> > I don't quite agree here.  Losing per-thread control within the cpu
>> > controller is likely going to mean that much of it ends up being
>> > reimplemented as some duplicate-in-appearance interface that gets us back to
>> > where we are today.  I recognize that these controllers (cpu, cpuacct) are
>> > square pegs in that per-process makes sense for most other sub-systems; but
>> > unfortunately, their needs and use-cases are real / dependent on their
>> > present form.
>
> Let's build an API which actually looks and behaves like an API which
> is properly isolated from what external agents may do to the process.
> I can't see how that would be "back to where we are today".  All of
> those are pretty critical attributes for a public kernel API and
> utterly broken right now.
>

Sure, but I don't think you can throw out per-thread control for all
controllers to enable this.  Which makes everything else harder.  A
intermediary step in unification might be that we move from N mounts
to 2.  Those that can be managed at the process level, and those that
can't.  It's a compromise, but may allow cleaner abstractions for the
former case.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1211496 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Paul.

On Fri, Aug 21, 2015 at 12:26:30PM -0700, Paul Turner wrote:
...
> A very concrete example of the above is a virtual machine in which you
> want to guarantee scheduling for the vCPU threads which must schedule
> beside many hypervisor support threads.   A hierarchy is the only way
> to fix the ratio at which these compete.

Just to learn more, what sort of hypervisor support threads are we
talking about?  They would have to consume considerable amount of cpu
cycles for problems like this to be relevant and be dynamic in numbers
in a way which letting them competing against vcpus makes sense.  Do
IO helpers meet these criteria?

> An example that's not the cpu controller is that we use cpusets to
> expose to applications their "shared" and "private" cores.  (These
> sets are dynamic based on what is coscheduled on a given machine.)

Can you please go into more details with these?

> > Why would you assume that threads of a process wouldn't want to
> > configure it ever?  How is this different from CPU affinity?
> 
> In general cache and CPU behave differently.  Generally for it to make
> sense between threads in a process they would have to have wholly
> disjoint memory, at which point the only sane long-term implementation
> is separate processes and the management moves up a level anyway.
> 
> That said, there are surely cases in which it might be convenient to
> use at a per-thread level to correct a specific performance anomaly.
> But at that point, you have certainly reached the level of hammer that
> you can coordinate with an external daemon if necessary.

So, I'm not super familiar with all the use cases but the whole cache
allocation thing is almost by nature a specific niche thing and I feel
pretty reluctant to blow off per-thread usages as too niche to worry
about.

> > I don't follow what you're trying to way with the above paragraph.
> > Are you still talking about CAT?  If so, that use case isn't the only
> > one.  I'm pretty sure there are people who would want to configure
> > cache allocation at thread level.
> 
> I'm not agreeing with you that "in cgroups" means "must be usable by
> applications within that hierarchy".  A cgroup subsystem used as a
> partitioning API only by system management daemons is entirely
> reasonable.  CAT is a reasonable example of this.

I see.  The same argument.  I don't think CAT just being system
management thing makes sense.

> > So, this is a trade-off we're consciously making.  If there are
> > common-enough use cases which require jumping across different cgroup
> > domains, we'll try to figure out a way to accomodate those but by
> > default migration is a very cold and expensive path.
> 
> The core here was the need for allowing sub-process migration.  I'm
> not sure I follow the performance trade-off argument; haven't we
> historically seen the opposite?  That migration has been a slow-path
> without optimizations and people pushing to make it faster?  This
> seems a hard generalization to make for something that's inherently
> tied to a particular controller.

It isn't something tied to a particular controller.  Some controllers
may get impacted less by than others but there's an inherent
connection between how dynamic an association is and how expensive the
locking around it needs to be and we need to set up basic behavior and
usage conventions so that different controllers are designed and
implemented assuming similar usage patterns; otherwise, we end up with
the chaotic shit show that we have had where everything behaves
differently and nobody knows what's the right way to do things and we
end up locked into weird requirements which some controller induced
for no good reason but cause significant pain on use cases which
actually matter.

> I don't care if we try turning that dial back to assume it's a cold
> path once more, only that it's supported.

It has always been a cold path and I'm not saying this is gonna be
noticeably worse in the future but usages like bouncing threads on
request-by-request basis are and will be clearly worse than bouncing
to threads which are already in the target domain.

> >> > A forwarding /proc/thread_self/cgroup accessor, or similar, would be another
> >> > way to address some of these issues.
> >
> > That sounds horrible to me.  What if the process wants to do RMW a
> > config?
> 
> Locking within a process is easy.

It's not contained in the process at all.  What if an external entity
decides to migrate the process into another cgroup inbetween?

> > What if the permissions are different after an intervening
> > migration?
> 
> This is a side-effect of migration not being properly supported.
>
> > What if the sub-hierarchy no longer exists or has been
> > replaced by a hierarchy with the same topology but actualy is a
> > different one?
> 
> The easy answer is that:  Only a process should be managing its
> sub-hierarchy.  That's the nice thing about hierarchies.

cgroupfs is a horrible place to implement that part of interface.  It
doesn't make any sense to combine those two into the same hierarchy.
You're agreeing to the identified problem but somehow still suggesting
doing what we've been doing when the root cause of the said problem is
conflating and interlocking these two separate things.

> The harder answer is:  How do we handle non-fungible resources such as
> CPU assignments within a hierarchy?  This is a big part of why I make
> arguments for certain partitions being management-software only above.
> This is imperfect, but better then where we stand today.

I'm not following.  Why is that different?

> > Let's build an API which actually looks and behaves like an API which
> > is properly isolated from what external agents may do to the process.
> > I can't see how that would be "back to where we are today".  All of
> > those are pretty critical attributes for a public kernel API and
> > utterly broken right now.
> 
> Sure, but I don't think you can throw out per-thread control for all
> controllers to enable this.  Which makes everything else harder.  A
> intermediary step in unification might be that we move from N mounts
> to 2.  Those that can be managed at the process level, and those that
> can't.  It's a compromise, but may allow cleaner abstractions for the
> former case.

The transition can already be gradual.  Why would you add yet another
transition step?

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212320 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

[Multipart message — attachments visible in raw view] — view raw

On 2015-08-22 14:29, Tejun Heo wrote:
> Hello, Paul.
>
> On Fri, Aug 21, 2015 at 12:26:30PM -0700, Paul Turner wrote:
> ...
>> A very concrete example of the above is a virtual machine in which you
>> want to guarantee scheduling for the vCPU threads which must schedule
>> beside many hypervisor support threads.   A hierarchy is the only way
>> to fix the ratio at which these compete.
>
> Just to learn more, what sort of hypervisor support threads are we
> talking about?  They would have to consume considerable amount of cpu
> cycles for problems like this to be relevant and be dynamic in numbers
> in a way which letting them competing against vcpus makes sense.  Do
> IO helpers meet these criteria?
>
Depending on the configuration, yes they can.  VirtualBox has some 
rather CPU intensive threads that aren't vCPU threads (their emulated 
APIC thread immediately comes to mind), and so does QEMU depending on 
the emulated hardware configuration (it gets more noticeable when the 
disk images are stored on a SAN and served through iSCSI, NBD, FCoE, or 
ATAoE, which is pretty typical usage for large virtualization 
deployments).  I've seen cases first hand where the vCPU's can make no 
reasonable progress because they are constantly getting crowded out by 
other threads.

The use of the term 'hypervisor support threads' for this is probably 
not the best way of describing the contention, as it's almost always a 
full system virtualization issue, and the contending threads are usually 
storage back-end access threads.

I would argue that there are better ways to deal properly with this 
(Isolate the non vCPU threads on separate physical CPU's from the 
hardware emulation threads), but such methods require large systems to 
be practical at any scale, and many people don't have the budget for 
such large systems, and this way of doing things is much more flexible 
for small scale use cases (for example, someone running one or two VM's 
on a laptop under QEMU or VirtualBox).

[toc] | [prev] | [next] | [standalone]

#1212355 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Austin.

On Mon, Aug 24, 2015 at 11:47:02AM -0400, Austin S Hemmelgarn wrote:
> >Just to learn more, what sort of hypervisor support threads are we
> >talking about?  They would have to consume considerable amount of cpu
> >cycles for problems like this to be relevant and be dynamic in numbers
> >in a way which letting them competing against vcpus makes sense.  Do
> >IO helpers meet these criteria?
> >
> Depending on the configuration, yes they can.  VirtualBox has some rather
> CPU intensive threads that aren't vCPU threads (their emulated APIC thread
> immediately comes to mind), and so does QEMU depending on the emulated

And the number of those threads fluctuate widely and dynamically?

> hardware configuration (it gets more noticeable when the disk images are
> stored on a SAN and served through iSCSI, NBD, FCoE, or ATAoE, which is
> pretty typical usage for large virtualization deployments).  I've seen cases
> first hand where the vCPU's can make no reasonable progress because they are
> constantly getting crowded out by other threads.

That alone doesn't require hierarchical resource distribution tho.
Setting nice levels reasonably is likely to alleviate most of the
problem.

> The use of the term 'hypervisor support threads' for this is probably not
> the best way of describing the contention, as it's almost always a full
> system virtualization issue, and the contending threads are usually storage
> back-end access threads.
> 
> I would argue that there are better ways to deal properly with this (Isolate
> the non vCPU threads on separate physical CPU's from the hardware emulation
> threads), but such methods require large systems to be practical at any
> scale, and many people don't have the budget for such large systems, and
> this way of doing things is much more flexible for small scale use cases
> (for example, someone running one or two VM's on a laptop under QEMU or
> VirtualBox).

I don't know.  "Someone running one or two VM's on a laptop under
QEMU" doesn't really sound like the use case which absolutely requires
hierarchical cpu cycle distribution.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212446 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

On Mon, 2015-08-24 at 13:04 -0400, Tejun Heo wrote:
> Hello, Austin.
> 
> On Mon, Aug 24, 2015 at 11:47:02AM -0400, Austin S Hemmelgarn wrote:
> > >Just to learn more, what sort of hypervisor support threads are we
> > >talking about?  They would have to consume considerable amount of cpu
> > >cycles for problems like this to be relevant and be dynamic in numbers
> > >in a way which letting them competing against vcpus makes sense.  Do
> > >IO helpers meet these criteria?
> > >
> > Depending on the configuration, yes they can.  VirtualBox has some rather
> > CPU intensive threads that aren't vCPU threads (their emulated APIC thread
> > immediately comes to mind), and so does QEMU depending on the emulated
> 
> And the number of those threads fluctuate widely and dynamically?
> 
> > hardware configuration (it gets more noticeable when the disk images are
> > stored on a SAN and served through iSCSI, NBD, FCoE, or ATAoE, which is
> > pretty typical usage for large virtualization deployments).  I've seen cases
> > first hand where the vCPU's can make no reasonable progress because they are
> > constantly getting crowded out by other threads.

Hm. Serious CPU starvation would seem to require quite a few hungry
threads, but even a few IO threads with kick butt hardware under them
could easily tilt fairness heavily in favor of VPUs generating IO.

> That alone doesn't require hierarchical resource distribution tho.
> Setting nice levels reasonably is likely to alleviate most of the
> problem.

Unless the CPU controller is in use.

	-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212467 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

[Multipart message — attachments visible in raw view] — view raw

On 2015-08-24 13:04, Tejun Heo wrote:
> Hello, Austin.
>
> On Mon, Aug 24, 2015 at 11:47:02AM -0400, Austin S Hemmelgarn wrote:
>>> Just to learn more, what sort of hypervisor support threads are we
>>> talking about?  They would have to consume considerable amount of cpu
>>> cycles for problems like this to be relevant and be dynamic in numbers
>>> in a way which letting them competing against vcpus makes sense.  Do
>>> IO helpers meet these criteria?
>>>
>> Depending on the configuration, yes they can.  VirtualBox has some rather
>> CPU intensive threads that aren't vCPU threads (their emulated APIC thread
>> immediately comes to mind), and so does QEMU depending on the emulated
>
> And the number of those threads fluctuate widely and dynamically?
It depends, usually there isn't dynamic fluctuation unless there is a 
lot of hot[un]plugging of virtual devices going on (which can be the 
case for situations with tight host/guest integration), but the number 
of threads can vary widely between configurations (most of the VM's I 
run under QEMU have about 16 threads on average, but I've seen instances 
with more than 100 threads).  The most likely case to cause wide and 
dynamic fluctuations of threads would be systems set up to dynamically 
hot[un]plug vCPU's based on system load (such systems have other issues 
to contend with also, but they do exist).
>> hardware configuration (it gets more noticeable when the disk images are
>> stored on a SAN and served through iSCSI, NBD, FCoE, or ATAoE, which is
>> pretty typical usage for large virtualization deployments).  I've seen cases
>> first hand where the vCPU's can make no reasonable progress because they are
>> constantly getting crowded out by other threads.
>
> That alone doesn't require hierarchical resource distribution tho.
> Setting nice levels reasonably is likely to alleviate most of the
> problem.
In the cases I've dealt with this myself, nice levels didn't cut it, and 
I had to resort to SCHED_RR with particular care to avoid priority 
inversions.
>> The use of the term 'hypervisor support threads' for this is probably not
>> the best way of describing the contention, as it's almost always a full
>> system virtualization issue, and the contending threads are usually storage
>> back-end access threads.
>>
>> I would argue that there are better ways to deal properly with this (Isolate
>> the non vCPU threads on separate physical CPU's from the hardware emulation
>> threads), but such methods require large systems to be practical at any
>> scale, and many people don't have the budget for such large systems, and
>> this way of doing things is much more flexible for small scale use cases
>> (for example, someone running one or two VM's on a laptop under QEMU or
>> VirtualBox).
>
> I don't know.  "Someone running one or two VM's on a laptop under
> QEMU" doesn't really sound like the use case which absolutely requires
> hierarchical cpu cycle distribution.
It depends on the use case.  I never have more than 2 VM's running on my 
laptop (always under QEMU, setting up Xen is kind of pointless ona quad 
core system with only 8G of RAM), and I take extensive advantage of the 
cpu cgroup to partition resources among various services on the host.

[toc] | [prev] | [next] | [standalone]

#1212489 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Austin.

On Mon, Aug 24, 2015 at 04:00:49PM -0400, Austin S Hemmelgarn wrote:
> >That alone doesn't require hierarchical resource distribution tho.
> >Setting nice levels reasonably is likely to alleviate most of the
> >problem.
>
> In the cases I've dealt with this myself, nice levels didn't cut it, and I
> had to resort to SCHED_RR with particular care to avoid priority inversions.

I wonder why.  The difference between -20 and 20 is around 2500x in
terms of weight.  That should have been enough for expressing whatever
precedence the vcpus should have over other threads.

> >I don't know.  "Someone running one or two VM's on a laptop under
> >QEMU" doesn't really sound like the use case which absolutely requires
> >hierarchical cpu cycle distribution.
>
> It depends on the use case.  I never have more than 2 VM's running on my
> laptop (always under QEMU, setting up Xen is kind of pointless ona quad core
> system with only 8G of RAM), and I take extensive advantage of the cpu
> cgroup to partition resources among various services on the host.

Hmmm... I'm trying to understand the usecases where having hierarchy
inside a process are actually required so that we don't end up doing
something complex unnecessarily.  So far, it looks like an easy
alternative for qemu would be teaching it to manage priorities of its
threads given that the threads are mostly static - vcpus going up and
down are explicit operations which can trigger priority adjustments if
necessary, which is unlikely to begin with.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212508

On Mon, Aug 24, 2015 at 1:25 PM, Tejun Heo <tj@kernel.org> wrote:
> Hello, Austin.
>
> On Mon, Aug 24, 2015 at 04:00:49PM -0400, Austin S Hemmelgarn wrote:
>> >That alone doesn't require hierarchical resource distribution tho.
>> >Setting nice levels reasonably is likely to alleviate most of the
>> >problem.
>>
>> In the cases I've dealt with this myself, nice levels didn't cut it, and I
>> had to resort to SCHED_RR with particular care to avoid priority inversions.
>
> I wonder why.  The difference between -20 and 20 is around 2500x in
> terms of weight.  That should have been enough for expressing whatever
> precedence the vcpus should have over other threads.

This strongly perturbs the load-balancer which performs busiest cpu
selection by weight.

Note that also we do not necessarily want total dominance by vCPU
threads, the hypervisor threads are almost always doing work on their
behalf and we want to provision them with _some_ time.  A
sub-hierarchy allows this to be performed in a way that is independent
of how many vCPUs or support threads that are present.

>
>> >I don't know.  "Someone running one or two VM's on a laptop under
>> >QEMU" doesn't really sound like the use case which absolutely requires
>> >hierarchical cpu cycle distribution.
>>
>> It depends on the use case.  I never have more than 2 VM's running on my
>> laptop (always under QEMU, setting up Xen is kind of pointless ona quad core
>> system with only 8G of RAM), and I take extensive advantage of the cpu
>> cgroup to partition resources among various services on the host.
>
> Hmmm... I'm trying to understand the usecases where having hierarchy
> inside a process are actually required so that we don't end up doing
> something complex unnecessarily.  So far, it looks like an easy
> alternative for qemu would be teaching it to manage priorities of its
> threads given that the threads are mostly static - vcpus going up and
> down are explicit operations which can trigger priority adjustments if
> necessary, which is unlikely to begin with.

What you're proposing is both unnecessarily complex and imprecise.
Arbitrating competition between groups of threads is exactly why we
support sub-hierarchies within cpu.

>
> Thanks.
>
> --
> tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212510 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Paul.

On Mon, Aug 24, 2015 at 02:00:54PM -0700, Paul Turner wrote:
> > Hmmm... I'm trying to understand the usecases where having hierarchy
> > inside a process are actually required so that we don't end up doing
> > something complex unnecessarily.  So far, it looks like an easy
> > alternative for qemu would be teaching it to manage priorities of its
> > threads given that the threads are mostly static - vcpus going up and
> > down are explicit operations which can trigger priority adjustments if
> > necessary, which is unlikely to begin with.
> 
> What you're proposing is both unnecessarily complex and imprecise.
> Arbitrating competition between groups of threads is exactly why we
> support sub-hierarchies within cpu.

Sure, and to make that behave half-way acceptable, we'll have to take
on significant amount of effort and likely complexity and I'm trying
to see whether the usecases are actually justifiable.  I get that
priority based solution will be less precise and more complex on the
application side but by how much and does the added precision enough
to justify the extra facilities to support that?  If it is, sure,
let's get to it but it'd be great if the concrete prolem cases are
properly identified and understood.  I'll continue on the other reply.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212516

On Mon, Aug 24, 2015 at 2:12 PM, Tejun Heo <tj@kernel.org> wrote:
> Hello, Paul.
>
> On Mon, Aug 24, 2015 at 02:00:54PM -0700, Paul Turner wrote:
>> > Hmmm... I'm trying to understand the usecases where having hierarchy
>> > inside a process are actually required so that we don't end up doing
>> > something complex unnecessarily.  So far, it looks like an easy
>> > alternative for qemu would be teaching it to manage priorities of its
>> > threads given that the threads are mostly static - vcpus going up and
>> > down are explicit operations which can trigger priority adjustments if
>> > necessary, which is unlikely to begin with.
>>
>> What you're proposing is both unnecessarily complex and imprecise.
>> Arbitrating competition between groups of threads is exactly why we
>> support sub-hierarchies within cpu.
>
> Sure, and to make that behave half-way acceptable, we'll have to take
> on significant amount of effort and likely complexity and I'm trying
> to see whether the usecases are actually justifiable.  I get that
> priority based solution will be less precise and more complex on the
> application side but by how much and does the added precision enough
> to justify the extra facilities to support that?  If it is, sure,
> let's get to it but it'd be great if the concrete prolem cases are
> properly identified and understood.  I'll continue on the other reply.
>

No problem, I think the conversation is absolutely
constructive/important to have and am happy to help drill down.

> Thanks.
>
> --
> tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212498

On Mon, Aug 24, 2015 at 10:04 AM, Tejun Heo <tj@kernel.org> wrote:
> Hello, Austin.
>
> On Mon, Aug 24, 2015 at 11:47:02AM -0400, Austin S Hemmelgarn wrote:
>> >Just to learn more, what sort of hypervisor support threads are we
>> >talking about?  They would have to consume considerable amount of cpu
>> >cycles for problems like this to be relevant and be dynamic in numbers
>> >in a way which letting them competing against vcpus makes sense.  Do
>> >IO helpers meet these criteria?
>> >
>> Depending on the configuration, yes they can.  VirtualBox has some rather
>> CPU intensive threads that aren't vCPU threads (their emulated APIC thread
>> immediately comes to mind), and so does QEMU depending on the emulated
>
> And the number of those threads fluctuate widely and dynamically?
>
>> hardware configuration (it gets more noticeable when the disk images are
>> stored on a SAN and served through iSCSI, NBD, FCoE, or ATAoE, which is
>> pretty typical usage for large virtualization deployments).  I've seen cases
>> first hand where the vCPU's can make no reasonable progress because they are
>> constantly getting crowded out by other threads.
>
> That alone doesn't require hierarchical resource distribution tho.
> Setting nice levels reasonably is likely to alleviate most of the
> problem.

Nice is not sufficient here.  There could be arbitrarily many threads
within the hypervisor that are not actually hosting guest CPU threads.
The only way to have this competition occur at a reasonably fixed
ratio is a sub-hierarchy.

>
>> The use of the term 'hypervisor support threads' for this is probably not
>> the best way of describing the contention, as it's almost always a full
>> system virtualization issue, and the contending threads are usually storage
>> back-end access threads.
>>
>> I would argue that there are better ways to deal properly with this (Isolate
>> the non vCPU threads on separate physical CPU's from the hardware emulation
>> threads), but such methods require large systems to be practical at any
>> scale, and many people don't have the budget for such large systems, and
>> this way of doing things is much more flexible for small scale use cases
>> (for example, someone running one or two VM's on a laptop under QEMU or
>> VirtualBox).
>
> I don't know.  "Someone running one or two VM's on a laptop under
> QEMU" doesn't really sound like the use case which absolutely requires
> hierarchical cpu cycle distribution.
>

We run more than 'one or two' VMs using this configuration.  :)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212509 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello,

On Mon, Aug 24, 2015 at 01:54:08PM -0700, Paul Turner wrote:
> > That alone doesn't require hierarchical resource distribution tho.
> > Setting nice levels reasonably is likely to alleviate most of the
> > problem.
> 
> Nice is not sufficient here.  There could be arbitrarily many threads
> within the hypervisor that are not actually hosting guest CPU threads.
> The only way to have this competition occur at a reasonably fixed
> ratio is a sub-hierarchy.

I get that having hierarchy of threads would be nicer but am having a
bit of difficulty seeing why adjusting priorities of threads wouldn't
be sufficient.  It's not like threads of the same process competing
with each other is a new problem.  People have been dealing with it
for ages.  Hierarchical management can be a nice plus but we want the
problem and proposed solution to be justifiable.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212513 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello,

On Mon, Aug 24, 2015 at 02:10:17PM -0700, Paul Turner wrote:
> Suppose that we have 10 vcpu threads and 100 support threads.
> Suppose that we want the support threads to receive up to 10% of the
> time available to the VM as a whole on that machine.
> 
> If I have one particular support thread that is busy, I want it to
> receive that entire 10% (maybe a guest is pounding on scsi for
> example, or in the thread-pool case, I've passed a single expensive
> computation).  Conversely, suppose the guest is doing lots of
> different things and several support threads are active, I want the
> time to be shared between them.
> 
> There is no way to implement this with nice.  Either a single thread
> can consume 10%, and the group can dominate, or the group cannot
> dominate and the single thread can be starved.

Would it be possible for you to give realistic and concrete examples?
I'm not trying to play down the use cases but concrete examples are
usually helpful at putting things in perspective.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212526

On Mon, Aug 24, 2015 at 2:17 PM, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> On Mon, Aug 24, 2015 at 02:10:17PM -0700, Paul Turner wrote:
>> Suppose that we have 10 vcpu threads and 100 support threads.
>> Suppose that we want the support threads to receive up to 10% of the
>> time available to the VM as a whole on that machine.
>>
>> If I have one particular support thread that is busy, I want it to
>> receive that entire 10% (maybe a guest is pounding on scsi for
>> example, or in the thread-pool case, I've passed a single expensive
>> computation).  Conversely, suppose the guest is doing lots of
>> different things and several support threads are active, I want the
>> time to be shared between them.
>>
>> There is no way to implement this with nice.  Either a single thread
>> can consume 10%, and the group can dominate, or the group cannot
>> dominate and the single thread can be starved.
>
> Would it be possible for you to give realistic and concrete examples?
> I'm not trying to play down the use cases but concrete examples are
> usually helpful at putting things in perspective.

I don't think there's anything that's not realistic or concrete about
the example above.  The "suppose" parts were only for qualifying the
pool sizes for vcpu and non-vcpu threads above since discussion of
implementation using nice is dependent on knowing these counts.


>
> Thanks.
>
> --
> tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212536 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

On Mon, Aug 24, 2015 at 02:19:29PM -0700, Paul Turner wrote:
> > Would it be possible for you to give realistic and concrete examples?
> > I'm not trying to play down the use cases but concrete examples are
> > usually helpful at putting things in perspective.
> 
> I don't think there's anything that's not realistic or concrete about
> the example above.  The "suppose" parts were only for qualifying the
> pool sizes for vcpu and non-vcpu threads above since discussion of
> implementation using nice is dependent on knowing these counts.

Hmm... I was hoping for an actual configurations and usage scenarios.
Preferably something people can set up and play with.  I take that the
CPU intensive helper threads are usually IO workers?  Is the scenario
where the VM is set up with a lot of IO devices and different ones may
consume large amount of CPU cycles at any given point?

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212543

On Mon, Aug 24, 2015 at 2:40 PM, Tejun Heo <tj@kernel.org> wrote:
> On Mon, Aug 24, 2015 at 02:19:29PM -0700, Paul Turner wrote:
>> > Would it be possible for you to give realistic and concrete examples?
>> > I'm not trying to play down the use cases but concrete examples are
>> > usually helpful at putting things in perspective.
>>
>> I don't think there's anything that's not realistic or concrete about
>> the example above.  The "suppose" parts were only for qualifying the
>> pool sizes for vcpu and non-vcpu threads above since discussion of
>> implementation using nice is dependent on knowing these counts.
>
> Hmm... I was hoping for an actual configurations and usage scenarios.
> Preferably something people can set up and play with.

This is much easier to set up and play with synthetically.  Just
create the 10 threads and 100 threads above then experiment with
configurations designed at guaranteeing the set of 100 threads
relatively uniform throughput regardless of how many are active.   I
don't think trying to run a VM stack adds anything except complexity
of reproduction here.

> I take that the
> CPU intensive helper threads are usually IO workers?  Is the scenario
> where the VM is set up with a lot of IO devices and different ones may
> consume large amount of CPU cycles at any given point?

Yes, generally speaking there are a few major classes of IO (flash,
disk, network) that a guest may invoke.  Each of these backends is
separate and chooses its own threading.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212566 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello,

On Mon, Aug 24, 2015 at 03:03:05PM -0700, Paul Turner wrote:
> > Hmm... I was hoping for an actual configurations and usage scenarios.
> > Preferably something people can set up and play with.
> 
> This is much easier to set up and play with synthetically.  Just
> create the 10 threads and 100 threads above then experiment with
> configurations designed at guaranteeing the set of 100 threads
> relatively uniform throughput regardless of how many are active.   I
> don't think trying to run a VM stack adds anything except complexity
> of reproduction here.

Well, but that loses most of details and why such use cases matter to
begin with.  We can imagine up stuff to induce arbitrary set of
requirements.

> > I take that the
> > CPU intensive helper threads are usually IO workers?  Is the scenario
> > where the VM is set up with a lot of IO devices and different ones may
> > consume large amount of CPU cycles at any given point?
> 
> Yes, generally speaking there are a few major classes of IO (flash,
> disk, network) that a guest may invoke.  Each of these backends is
> separate and chooses its own threading.

Hmmm... if that's the case, would limiting iops on those IO devices
(or classes of them) work?  qemu already implements IO limit mechanism
after all.

Anyways, a point here is that threads of the same process competing
isn't a new problem.  There are many ways to make those threads play
nice as the application itself often has to be involved anyway,
especially for something like qemu which is heavily involved in
provisioning resources.

cgroups can be a nice brute-force add-on which lets sysadmins do wild
things but it's inherently hacky and incomplete for coordinating
threads.  For example, what is it gonna do if qemu cloned vcpus and IO
helpers dynamically off of the same parent thread?  It requires
application's cooperation anyway but at the same time is painful to
actually interact from those applications.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212574

On Mon, Aug 24, 2015 at 3:49 PM, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> On Mon, Aug 24, 2015 at 03:03:05PM -0700, Paul Turner wrote:
>> > Hmm... I was hoping for an actual configurations and usage scenarios.
>> > Preferably something people can set up and play with.
>>
>> This is much easier to set up and play with synthetically.  Just
>> create the 10 threads and 100 threads above then experiment with
>> configurations designed at guaranteeing the set of 100 threads
>> relatively uniform throughput regardless of how many are active.   I
>> don't think trying to run a VM stack adds anything except complexity
>> of reproduction here.
>
> Well, but that loses most of details and why such use cases matter to
> begin with.  We can imagine up stuff to induce arbitrary set of
> requirements.

All that's being proved or disproved here is that it's difficult to
coordinate the consumption of asymmetric thread pools using nice.  The
constraints here were drawn from a real-world example.

>
>> > I take that the
>> > CPU intensive helper threads are usually IO workers?  Is the scenario
>> > where the VM is set up with a lot of IO devices and different ones may
>> > consume large amount of CPU cycles at any given point?
>>
>> Yes, generally speaking there are a few major classes of IO (flash,
>> disk, network) that a guest may invoke.  Each of these backends is
>> separate and chooses its own threading.
>
> Hmmm... if that's the case, would limiting iops on those IO devices
> (or classes of them) work?  qemu already implements IO limit mechanism
> after all.

No.

1) They should proceed at the maximum rate that they can that's still
within their provisioning budget.
2) The cost/IO is both inconsistent and changes over time.  Attempting
to micro-optimize every backend for this is infeasible, this is
exactly the type of problem that the scheduler can usefully help
arbitrate.
3) Even pretending (2) is fixable, dynamically dividing these
right-to-work tokens between different I/O device backends is
extremely complex.

>
> Anyways, a point here is that threads of the same process competing
> isn't a new problem.  There are many ways to make those threads play
> nice as the application itself often has to be involved anyway,
> especially for something like qemu which is heavily involved in
> provisioning resources.

It's certainly not a new problem, but it's a real one, and it's
_hard_.  You're proposing removing the best known solution.

>
> cgroups can be a nice brute-force add-on which lets sysadmins do wild
> things but it's inherently hacky and incomplete for coordinating
> threads.  For example, what is it gonna do if qemu cloned vcpus and IO
> helpers dynamically off of the same parent thread?

We're talking about sub-process usage here.  This is the application
coordinating itself, NOT the sysadmin.  Processes are becoming larger
and larger, we need many of the same controls within them that we have
between them.

>  It requires
> application's cooperation anyway but at the same time is painful to
> actually interact from those applications.

As discussed elsewhere on thread this is really not a problem if you
define consistent rules with respect to which parts are managed by
who.  The argument of potential interference is no different to
messing with an application's on-disk configuration behind its back.
Alternate strawmen which greatly improve this from where we are today
have also been proposed.

>
> Thanks.
>
> --
> tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1212668 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

On 2015/08/25 8:15, Paul Turner wrote:
> On Mon, Aug 24, 2015 at 3:49 PM, Tejun Heo <tj@kernel.org> wrote:
>> Hello,
>>
>> On Mon, Aug 24, 2015 at 03:03:05PM -0700, Paul Turner wrote:
>>>> Hmm... I was hoping for an actual configurations and usage scenarios.
>>>> Preferably something people can set up and play with.
>>>
>>> This is much easier to set up and play with synthetically.  Just
>>> create the 10 threads and 100 threads above then experiment with
>>> configurations designed at guaranteeing the set of 100 threads
>>> relatively uniform throughput regardless of how many are active.   I
>>> don't think trying to run a VM stack adds anything except complexity
>>> of reproduction here.
>>
>> Well, but that loses most of details and why such use cases matter to
>> begin with.  We can imagine up stuff to induce arbitrary set of
>> requirements.
>
> All that's being proved or disproved here is that it's difficult to
> coordinate the consumption of asymmetric thread pools using nice.  The
> constraints here were drawn from a real-world example.
>
>>
>>>> I take that the
>>>> CPU intensive helper threads are usually IO workers?  Is the scenario
>>>> where the VM is set up with a lot of IO devices and different ones may
>>>> consume large amount of CPU cycles at any given point?
>>>
>>> Yes, generally speaking there are a few major classes of IO (flash,
>>> disk, network) that a guest may invoke.  Each of these backends is
>>> separate and chooses its own threading.
>>
>> Hmmm... if that's the case, would limiting iops on those IO devices
>> (or classes of them) work?  qemu already implements IO limit mechanism
>> after all.
>
> No.
>
> 1) They should proceed at the maximum rate that they can that's still
> within their provisioning budget.
> 2) The cost/IO is both inconsistent and changes over time.  Attempting
> to micro-optimize every backend for this is infeasible, this is
> exactly the type of problem that the scheduler can usefully help
> arbitrate.
> 3) Even pretending (2) is fixable, dynamically dividing these
> right-to-work tokens between different I/O device backends is
> extremely complex.
>

I think I should explain my customer's use case of qemu + cpuset/cpu (via libvirt)

(1) Isolating hypervisor thread.
    As already discussed, hypervisor threads are isolated by cpuset. But their purpose
    is to avoid _latency_ spike caused by hypervisor behavior. So, "nice" cannot be solution
    as already discussed.

(2) Fixed rate vcpu service.
    With using cpu controller's quota/period feature, my customer creates  vcpu models like
    Low(1GHz), Mid(2GHz), High(3GHz) for IaaS system.

    To do this, each vcpus should be quota-limited independently, with per-thread cpu control.

Especially, the method (1) is used in several enterprise customers for stabilizing their system.

Sub-process control should be provided by some way.

Thanks,
-Kame



>>
>> Anyways, a point here is that threads of the same process competing
>> isn't a new problem.  There are many ways to make those threads play
>> nice as the application itself often has to be involved anyway,
>> especially for something like qemu which is heavily involved in
>> provisioning resources.
>
> It's certainly not a new problem, but it's a real one, and it's
> _hard_.  You're proposing removing the best known solution.
>
>>
>> cgroups can be a nice brute-force add-on which lets sysadmins do wild
>> things but it's inherently hacky and incomplete for coordinating
>> threads.  For example, what is it gonna do if qemu cloned vcpus and IO
>> helpers dynamically off of the same parent thread?
>
> We're talking about sub-process usage here.  This is the application
> coordinating itself, NOT the sysadmin.  Processes are becoming larger
> and larger, we need many of the same controls within them that we have
> between them.
>
>>   It requires
>> application's cooperation anyway but at the same time is painful to
>> actually interact from those applications.
>
> As discussed elsewhere on thread this is really not a problem if you
> define consistent rules with respect to which parts are managed by
> who.  The argument of potential interference is no different to
> messing with an application's on-disk configuration behind its back.
> Alternate strawmen which greatly improve this from where we are today
> have also been proposed.
>
>>
>> Thanks.
>>
>> --
>> tejun
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1213359 — Re: [PATCH 3/3] sched: Implement interface for cgroup unified hierarchy

Hello, Kame.

On Tue, Aug 25, 2015 at 11:36:25AM +0900, Kamezawa Hiroyuki wrote:
> I think I should explain my customer's use case of qemu + cpuset/cpu (via libvirt)
> 
> (1) Isolating hypervisor thread.
>    As already discussed, hypervisor threads are isolated by cpuset. But their purpose
>    is to avoid _latency_ spike caused by hypervisor behavior. So, "nice" cannot be solution
>    as already discussed.
> 
> (2) Fixed rate vcpu service.
>    With using cpu controller's quota/period feature, my customer creates  vcpu models like
>    Low(1GHz), Mid(2GHz), High(3GHz) for IaaS system.
> 
>    To do this, each vcpus should be quota-limited independently, with per-thread cpu control.
> 
> Especially, the method (1) is used in several enterprise customers for stabilizing their system.
> 
> Sub-process control should be provided by some way.

Can you please take a look at the proposal on my reply to Paul's
email?  AFAICS, both of above cases should be fine with that.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web