Re: [PATCH 0/3] timer: Improve itimers scalability

From	Hideaki Kimura <hideaki.kimura@hpe.com>
Newsgroups	linux.kernel
Subject	Re: [PATCH 0/3] timer: Improve itimers scalability
Date	2015-08-27 02:00 +0200
Message-ID	<q1SLL-46X-5@gated-at.bofh.it> (permalink)
References	(2 earlier) <q1LTY-2LN-31@gated-at.bofh.it> <q1MwG-3Km-5@gated-at.bofh.it> <q1R3k-1Qg-45@gated-at.bofh.it> <q1S93-3nY-5@gated-at.bofh.it> <q1S93-3nY-7@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw


On 08/26/2015 04:13 PM, Frederic Weisbecker wrote:
> On Wed, Aug 26, 2015 at 03:53:26PM -0700, Hideaki Kimura wrote:
>> Sure, let me elaborate.
>>
>> Executive summary:
>>   Yes, enabling a process-wide timer in such a large machine is not wise, but
>> sometimes users/applications cannot avoid it.
>>
>>
>> The issue was observed actually not in a database itself but in a common
>> library it links to; gperftools.
>>
>> The database itself is optimized for many-cores/sockets, so surely it avoids
>> putting a process-wide timer or other unscalable things. It just links to
>> libprofiler for an optional feature to profile performance bottleneck only
>> when the user turns it on. We of course avoid turning the feature on unless
>> while we debug/tune the database.
>>
>> However, libprofiler sets the timer even when the client program doesn't
>> invoke any of its functions: libprofiler does it when the shared library is
>> loaded. We requested the developer of libprofiler to change the behavior,
>> but seems like there is a reason to keep that behavior:
>>    https://code.google.com/p/gperftools/issues/detail?id=133
>>
>> Based on this, I think there are two reasons why we should ameliorate this
>> issue in kernel layer.
>>
>>
>> 1. In the particular case, it's hard to prevent or even detect the issue in
>> user space.
>>
>> We (a team of low-level database and kernel experts) in fact spent huge
>> amount of time to just figure out what's the bottleneck there because
>> nothing measurable happens in user space. I pulled out countless hairs.
>>
>> Also, the user has to de-link the library from the application to prevent
>> the itimer installation. Imagine a case where the software is proprietary.
>> It won't fly.
>>
>>
>> 2. This is just one example. There could be many other such
>> binaries/libraries that do similar things somewhere in a complex software
>> stack.
>>
>> Today we haven't heard of many such cases, but people will start hitting it
>> once 100s~1,000s of cores become common.
>>
>>
>> After applying this patchset, we have observed that the performance hit
>> almost completely went away at least for 240 cores. So, it's quite
>> beneficial in real world.
>
> I can easily imagine that many code incidentally use posix cpu timers when
> it's not strictly required. But it doesn't look right to fix the kernel
> for that. For this simple reason: posix cpu timers, even after your fix,
> should introduce noticeable overhead. All threads of a process with a timer
> enqueued in elapse the cputime in a shared atomic variable. Add to that the
> overhead of enqueuing the timer, firing it. There is a bunch of scalability
> issue there.

I totally agree that this is not a perfect solution. If there are 10x 
more cores and sockets, just the atomic fetch_add might be too expensive.

However, it's comparatively/realistically the best thing we can do 
without any drawbacks. We can't magically force all library developers 
to write the most scalable code always.

My point is: this is a safety net, and a very effective one.

-- 
Hideaki Kimura
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Thread

[PATCH 0/3] timer: Improve itimers scalability Jason Low <jason.low2@hp.com> - 2015-08-26 05:20 +0200
  [PATCH 2/3] timer: Check thread timers only when there are active thread timers Jason Low <jason.low2@hp.com> - 2015-08-26 05:20 +0200
  [PATCH 1/3] timer: Optimize fastpath_timer_check() Jason Low <jason.low2@hp.com> - 2015-08-26 05:20 +0200
    Re: [PATCH 1/3] timer: Optimize fastpath_timer_check() Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 00:00 +0200
    Re: [PATCH 1/3] timer: Optimize fastpath_timer_check() Davidlohr Bueso <dave@stgolabs.net> - 2015-08-31 17:20 +0200
      Re: [PATCH 1/3] timer: Optimize fastpath_timer_check() Jason Low <jason.low2@hp.com> - 2015-08-31 21:50 +0200
  [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-26 05:20 +0200
    Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Linus Torvalds <torvalds@linux-foundation.org> - 2015-08-26 20:00 +0200
      Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 00:40 +0200
        Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-27 01:00 +0200
    Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 01:00 +0200
      Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-27 01:40 +0200
        Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-27 07:00 +0200
        Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 15:00 +0200
          Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Jason Low <jason.low2@hp.com> - 2015-08-27 22:40 +0200
            Re: [PATCH 3/3] timer: Reduce unnecessary sighand lock contention Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 23:20 +0200
  Re: [PATCH 0/3] timer: Improve itimers scalability Andrew Morton <akpm@linux-foundation.org> - 2015-08-26 05:30 +0200
    Re: [PATCH 0/3] timer: Improve itimers scalability Jason Low <jason.low2@hp.com> - 2015-08-26 18:40 +0200
      Re: [PATCH 0/3] timer: Improve itimers scalability Oleg Nesterov <oleg@redhat.com> - 2015-08-26 19:20 +0200
        Re: [PATCH 0/3] timer: Improve itimers scalability Jason Low <jason.low2@hp.com> - 2015-08-27 00:10 +0200
          Re: [PATCH 0/3] timer: Improve itimers scalability Hideaki Kimura <hideaki.kimura@hpe.com> - 2015-08-27 01:20 +0200
            Re: [PATCH 0/3] timer: Improve itimers scalability Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 01:20 +0200
              Re: [PATCH 0/3] timer: Improve itimers scalability Hideaki Kimura <hideaki.kimura@hpe.com> - 2015-08-27 02:00 +0200
                Re: [PATCH 0/3] timer: Improve itimers scalability Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 15:20 +0200
                Re: [PATCH 0/3] timer: Improve itimers scalability Steven Rostedt <rostedt@goodmis.org> - 2015-08-27 16:50 +0200
                Re: [PATCH 0/3] timer: Improve itimers scalability Thomas Gleixner <tglx@linutronix.de> - 2015-08-27 17:20 +0200
                Re: [PATCH 0/3] timer: Improve itimers scalability Frederic Weisbecker <fweisbec@gmail.com> - 2015-08-27 17:20 +0200

csiph-web