Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1213509 > unrolled thread

[PATCH 0/3] timer: Improve itimers scalability

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/3] timer: Improve itimers scalability Jason Low <jason.low2@hp.com> - 2015-08-26 05:20 +0200
    [PATCH 2/3] timer: Check thread timers only when there are active thread timers Jason Low <jason.low2@hp.com> - 2015-08-26 05:20 +0200
    [PATCH 1/3] timer: Optimize fastpath_timer_check() Jason Low <jason.low2@hp.com> - 2015-08-26 05:20 +0200
      Re: [PATCH 1/3] timer: Optimize fastpath_timer_check() Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 00:00 +0200
      Re: [PATCH 1/3] timer: Optimize fastpath_timer_check() Davidlohr Bueso <dave@stgolabs.net> - 2015-08-31 17:20 +0200
        Re: [PATCH 1/3] timer: Optimize fastpath_timer_check() Jason Low <jason.low2@hp.com> - 2015-08-31 21:50 +0200
    [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-26 05:20 +0200
      Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Linus Torvalds <torvalds@linux-foundation.org> - 2015-08-26 20:00 +0200
        Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 00:40 +0200
          Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-27 01:00 +0200
      Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 01:00 +0200
        Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-27 01:40 +0200
          Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-27 07:00 +0200
          Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 15:00 +0200
            Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-27 22:40 +0200
              Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 23:20 +0200
    Re: [PATCH 0/3] timer: Improve itimers scalability Andrew Morton <akpm@linux-foundation.org> - 2015-08-26 05:30 +0200
      Re: [PATCH 0/3] timer: Improve itimers scalability Jason Low <jason.low2@hp.com> - 2015-08-26 18:40 +0200
        Re: [PATCH 0/3] timer: Improve itimers scalability Oleg Nesterov <oleg@redhat.com> - 2015-08-26 19:20 +0200
          Re: [PATCH 0/3] timer: Improve itimers scalability Jason Low <jason.low2@hp.com> - 2015-08-27 00:10 +0200
            Re: [PATCH 0/3] timer: Improve itimers scalability Hideaki Kimura <hideaki.kimura@hpe.com> - 2015-08-27 01:20 +0200
              Re: [PATCH 0/3] timer: Improve itimers scalability Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 01:20 +0200
                Re: [PATCH 0/3] timer: Improve itimers scalability Hideaki Kimura <hideaki.kimura@hpe.com> - 2015-08-27 02:00 +0200
                  Re: [PATCH 0/3] timer: Improve itimers scalability Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 15:20 +0200
                    Re: [PATCH 0/3] timer: Improve itimers scalability Steven Rostedt <rostedt@goodmis.org> - 2015-08-27 16:50 +0200
                      Re: [PATCH 0/3] timer: Improve itimers scalability Thomas Gleixner <tglx@linutronix.de> - 2015-08-27 17:20 +0200
                        Re: [PATCH 0/3] timer: Improve itimers scalability Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 17:20 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1214236

Sure, let me elaborate.

Executive summary:
  Yes, enabling a process-wide timer in such a large machine is not 
wise, but sometimes users/applications cannot avoid it.


The issue was observed actually not in a database itself but in a common 
library it links to; gperftools.

The database itself is optimized for many-cores/sockets, so surely it 
avoids putting a process-wide timer or other unscalable things. It just 
links to libprofiler for an optional feature to profile performance 
bottleneck only when the user turns it on. We of course avoid turning 
the feature on unless while we debug/tune the database.

However, libprofiler sets the timer even when the client program doesn't 
invoke any of its functions: libprofiler does it when the shared library 
is loaded. We requested the developer of libprofiler to change the 
behavior, but seems like there is a reason to keep that behavior:
   https://code.google.com/p/gperftools/issues/detail?id=133

Based on this, I think there are two reasons why we should ameliorate 
this issue in kernel layer.


1. In the particular case, it's hard to prevent or even detect the issue 
in user space.

We (a team of low-level database and kernel experts) in fact spent huge 
amount of time to just figure out what's the bottleneck there because 
nothing measurable happens in user space. I pulled out countless hairs.

Also, the user has to de-link the library from the application to 
prevent the itimer installation. Imagine a case where the software is 
proprietary. It won't fly.


2. This is just one example. There could be many other such 
binaries/libraries that do similar things somewhere in a complex 
software stack.

Today we haven't heard of many such cases, but people will start hitting 
it once 100s~1,000s of cores become common.


After applying this patchset, we have observed that the performance hit 
almost completely went away at least for 240 cores. So, it's quite 
beneficial in real world.

-- 
Hideaki Kimura
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1214237

On Wed, Aug 26, 2015 at 03:53:26PM -0700, Hideaki Kimura wrote:
> Sure, let me elaborate.
> 
> Executive summary:
>  Yes, enabling a process-wide timer in such a large machine is not wise, but
> sometimes users/applications cannot avoid it.
> 
> 
> The issue was observed actually not in a database itself but in a common
> library it links to; gperftools.
> 
> The database itself is optimized for many-cores/sockets, so surely it avoids
> putting a process-wide timer or other unscalable things. It just links to
> libprofiler for an optional feature to profile performance bottleneck only
> when the user turns it on. We of course avoid turning the feature on unless
> while we debug/tune the database.
> 
> However, libprofiler sets the timer even when the client program doesn't
> invoke any of its functions: libprofiler does it when the shared library is
> loaded. We requested the developer of libprofiler to change the behavior,
> but seems like there is a reason to keep that behavior:
>   https://code.google.com/p/gperftools/issues/detail?id=133
> 
> Based on this, I think there are two reasons why we should ameliorate this
> issue in kernel layer.
> 
> 
> 1. In the particular case, it's hard to prevent or even detect the issue in
> user space.
> 
> We (a team of low-level database and kernel experts) in fact spent huge
> amount of time to just figure out what's the bottleneck there because
> nothing measurable happens in user space. I pulled out countless hairs.
> 
> Also, the user has to de-link the library from the application to prevent
> the itimer installation. Imagine a case where the software is proprietary.
> It won't fly.
> 
> 
> 2. This is just one example. There could be many other such
> binaries/libraries that do similar things somewhere in a complex software
> stack.
> 
> Today we haven't heard of many such cases, but people will start hitting it
> once 100s~1,000s of cores become common.
> 
> 
> After applying this patchset, we have observed that the performance hit
> almost completely went away at least for 240 cores. So, it's quite
> beneficial in real world.

I can easily imagine that many code incidentally use posix cpu timers when
it's not strictly required. But it doesn't look right to fix the kernel
for that. For this simple reason: posix cpu timers, even after your fix,
should introduce noticeable overhead. All threads of a process with a timer
enqueued in elapse the cputime in a shared atomic variable. Add to that the
overhead of enqueuing the timer, firing it. There is a bunch of scalability
issue there.

> 
> -- 
> Hideaki Kimura
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1214251

On 08/26/2015 04:13 PM, Frederic Weisbecker wrote:
> On Wed, Aug 26, 2015 at 03:53:26PM -0700, Hideaki Kimura wrote:
>> Sure, let me elaborate.
>>
>> Executive summary:
>>   Yes, enabling a process-wide timer in such a large machine is not wise, but
>> sometimes users/applications cannot avoid it.
>>
>>
>> The issue was observed actually not in a database itself but in a common
>> library it links to; gperftools.
>>
>> The database itself is optimized for many-cores/sockets, so surely it avoids
>> putting a process-wide timer or other unscalable things. It just links to
>> libprofiler for an optional feature to profile performance bottleneck only
>> when the user turns it on. We of course avoid turning the feature on unless
>> while we debug/tune the database.
>>
>> However, libprofiler sets the timer even when the client program doesn't
>> invoke any of its functions: libprofiler does it when the shared library is
>> loaded. We requested the developer of libprofiler to change the behavior,
>> but seems like there is a reason to keep that behavior:
>>    https://code.google.com/p/gperftools/issues/detail?id=133
>>
>> Based on this, I think there are two reasons why we should ameliorate this
>> issue in kernel layer.
>>
>>
>> 1. In the particular case, it's hard to prevent or even detect the issue in
>> user space.
>>
>> We (a team of low-level database and kernel experts) in fact spent huge
>> amount of time to just figure out what's the bottleneck there because
>> nothing measurable happens in user space. I pulled out countless hairs.
>>
>> Also, the user has to de-link the library from the application to prevent
>> the itimer installation. Imagine a case where the software is proprietary.
>> It won't fly.
>>
>>
>> 2. This is just one example. There could be many other such
>> binaries/libraries that do similar things somewhere in a complex software
>> stack.
>>
>> Today we haven't heard of many such cases, but people will start hitting it
>> once 100s~1,000s of cores become common.
>>
>>
>> After applying this patchset, we have observed that the performance hit
>> almost completely went away at least for 240 cores. So, it's quite
>> beneficial in real world.
>
> I can easily imagine that many code incidentally use posix cpu timers when
> it's not strictly required. But it doesn't look right to fix the kernel
> for that. For this simple reason: posix cpu timers, even after your fix,
> should introduce noticeable overhead. All threads of a process with a timer
> enqueued in elapse the cputime in a shared atomic variable. Add to that the
> overhead of enqueuing the timer, firing it. There is a bunch of scalability
> issue there.

I totally agree that this is not a perfect solution. If there are 10x 
more cores and sockets, just the atomic fetch_add might be too expensive.

However, it's comparatively/realistically the best thing we can do 
without any drawbacks. We can't magically force all library developers 
to write the most scalable code always.

My point is: this is a safety net, and a very effective one.

-- 
Hideaki Kimura
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1214566

On Wed, Aug 26, 2015 at 04:45:44PM -0700, Hideaki Kimura wrote:
> I totally agree that this is not a perfect solution. If there are 10x more
> cores and sockets, just the atomic fetch_add might be too expensive.
> 
> However, it's comparatively/realistically the best thing we can do without
> any drawbacks. We can't magically force all library developers to write the
> most scalable code always.
> 
> My point is: this is a safety net, and a very effective one.

I mean the problem here is that a library uses an unscalable profiling feature,
unconditionally as soon as you load it without even initializing anything. And
this library is used in production.

At first sight, fixing that in the kernel is only a hack that just reduces a bit
the symptoms.

What is the technical issue that prevents from fixing that in the library itself?
Posix timers can be attached anytime.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1214624

On Thu, 27 Aug 2015 15:18:49 +0200
Frederic Weisbecker <fweisbec@gmail.com> wrote:

> On Wed, Aug 26, 2015 at 04:45:44PM -0700, Hideaki Kimura wrote:
> > I totally agree that this is not a perfect solution. If there are 10x more
> > cores and sockets, just the atomic fetch_add might be too expensive.
> > 
> > However, it's comparatively/realistically the best thing we can do without
> > any drawbacks. We can't magically force all library developers to write the
> > most scalable code always.
> > 
> > My point is: this is a safety net, and a very effective one.
> 
> I mean the problem here is that a library uses an unscalable profiling feature,
> unconditionally as soon as you load it without even initializing anything. And
> this library is used in production.
> 
> At first sight, fixing that in the kernel is only a hack that just reduces a bit
> the symptoms.
> 
> What is the technical issue that prevents from fixing that in the library itself?
> Posix timers can be attached anytime.

I'm curious to what the downside of this patch set is? If we can fix a
problem that should be fixed in userspace, but does not harm the kernel
by doing so, is that bad? (an argument for kdbus? ;-)


As Hideaki noted, this could be a problem in other locations as well
that people have yet to find.

-- Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1214641

On Thu, 27 Aug 2015, Steven Rostedt wrote:
> On Thu, 27 Aug 2015 15:18:49 +0200
> Frederic Weisbecker <fweisbec@gmail.com> wrote:
> 
> > On Wed, Aug 26, 2015 at 04:45:44PM -0700, Hideaki Kimura wrote:
> > > I totally agree that this is not a perfect solution. If there are 10x more
> > > cores and sockets, just the atomic fetch_add might be too expensive.
> > > 
> > > However, it's comparatively/realistically the best thing we can do without
> > > any drawbacks. We can't magically force all library developers to write the
> > > most scalable code always.
> > > 
> > > My point is: this is a safety net, and a very effective one.
> > 
> > I mean the problem here is that a library uses an unscalable profiling feature,
> > unconditionally as soon as you load it without even initializing anything. And
> > this library is used in production.
> > 
> > At first sight, fixing that in the kernel is only a hack that just reduces a bit
> > the symptoms.
> > 
> > What is the technical issue that prevents from fixing that in the library itself?
> > Posix timers can be attached anytime.
> 
> I'm curious to what the downside of this patch set is? If we can fix a
> problem that should be fixed in userspace, but does not harm the kernel
> by doing so, is that bad? (an argument for kdbus? ;-)

The patches are not fixing a problem which should be fixed in user
space. They merily avoid lock contention which happens to be prominent
with that particular library. But avoiding lock contention even for 2
threads is a worthwhile exercise if it does not hurt otherwise. And I
can't see anything what hurts with these patches.

Thanks,

	tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1214646

On Thu, Aug 27, 2015 at 05:09:21PM +0200, Thomas Gleixner wrote:
> On Thu, 27 Aug 2015, Steven Rostedt wrote:
> > On Thu, 27 Aug 2015 15:18:49 +0200
> > Frederic Weisbecker <fweisbec@gmail.com> wrote:
> > 
> > > On Wed, Aug 26, 2015 at 04:45:44PM -0700, Hideaki Kimura wrote:
> > > > I totally agree that this is not a perfect solution. If there are 10x more
> > > > cores and sockets, just the atomic fetch_add might be too expensive.
> > > > 
> > > > However, it's comparatively/realistically the best thing we can do without
> > > > any drawbacks. We can't magically force all library developers to write the
> > > > most scalable code always.
> > > > 
> > > > My point is: this is a safety net, and a very effective one.
> > > 
> > > I mean the problem here is that a library uses an unscalable profiling feature,
> > > unconditionally as soon as you load it without even initializing anything. And
> > > this library is used in production.
> > > 
> > > At first sight, fixing that in the kernel is only a hack that just reduces a bit
> > > the symptoms.
> > > 
> > > What is the technical issue that prevents from fixing that in the library itself?
> > > Posix timers can be attached anytime.
> > 
> > I'm curious to what the downside of this patch set is? If we can fix a
> > problem that should be fixed in userspace, but does not harm the kernel
> > by doing so, is that bad? (an argument for kdbus? ;-)
> 
> The patches are not fixing a problem which should be fixed in user
> space. They merily avoid lock contention which happens to be prominent
> with that particular library. But avoiding lock contention even for 2
> threads is a worthwhile exercise if it does not hurt otherwise. And I
> can't see anything what hurts with these patches.

Sure it shouldn't really hurt anyway, since the presense of elapsing timers
itself is checked locklessly.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web