Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1171190 > unrolled thread
| Started by | Ingo Molnar <mingo@kernel.org> |
|---|---|
| First post | 2015-06-24 10:50 +0200 |
| Last post | 2015-07-02 11:50 +0200 |
| Articles | 5 — 4 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: [RFC][PATCH 00/13] percpu rwsem -v2 Ingo Molnar <mingo@kernel.org> - 2015-06-24 10:50 +0200
Re: [RFC][PATCH 00/13] percpu rwsem -v2 Peter Zijlstra <peterz@infradead.org> - 2015-06-24 11:10 +0200
Re: [RFC][PATCH 00/13] percpu rwsem -v2 Daniel Wagner <daniel.wagner@bmw-carit.de> - 2015-06-24 11:20 +0200
Re: [RFC][PATCH 00/13] percpu rwsem -v2 Linus Torvalds <torvalds@linux-foundation.org> - 2015-07-02 00:00 +0200
Re: [RFC][PATCH 00/13] percpu rwsem -v2 Peter Zijlstra <peterz@infradead.org> - 2015-07-02 11:50 +0200
| From | Ingo Molnar <mingo@kernel.org> |
|---|---|
| Date | 2015-06-24 10:50 +0200 |
| Subject | Re: [RFC][PATCH 00/13] percpu rwsem -v2 |
| Message-ID | <pEOxA-4Lc-7@gated-at.bofh.it> |
* Peter Zijlstra <peterz@infradead.org> wrote: > On Tue, Jun 23, 2015 at 07:50:12PM +0200, Peter Zijlstra wrote: > > On Tue, Jun 23, 2015 at 04:56:39PM +0200, Daniel Wagner wrote: > > > flock02 > > > mean variance sigma max min > > > tip-1 11.8994 0.5874 0.7664 13.2022 8.6324 > > > tip-2 11.7394 0.5252 0.7247 13.2540 9.7513 > > > tip-3 11.8155 0.5288 0.7272 13.2700 9.9480 > > > tip+percpu-rswem-1 15.3601 0.8981 0.9477 16.8116 12.6910 > > > tip+percpu-rswem-2 15.2558 0.8442 0.9188 17.0199 12.9586 > > > tip+percpu-rswem-3 15.5297 0.6386 0.7991 17.4392 12.7992 > > > > I did indeed manage to get flock02 down to a usable level and found: > > Aside from the flock_lock_file function moving up, we also get an > increase in _raw_spin_lock. > > Before: > > 5.17% 5.17% flock02 [kernel.vmlinux] [k] _raw_spin_lock > | > ---_raw_spin_lock > | > |--99.75%-- flock_lock_file_wait > | sys_flock > | entry_SYSCALL_64_fastpath > | flock > --0.25%-- [...] > > > After: > > 7.20% 7.20% flock02 [kernel.vmlinux] [k] _raw_spin_lock > | > ---_raw_spin_lock > | > |--52.23%-- flock_lock_file_wait > | sys_flock > | entry_SYSCALL_64_fastpath > | flock > | > |--25.92%-- flock_lock_file > | flock_lock_file_wait > | sys_flock > | entry_SYSCALL_64_fastpath > | flock > | > |--21.42%-- locks_delete_lock_ctx > | flock_lock_file > | flock_lock_file_wait > | sys_flock > | entry_SYSCALL_64_fastpath > | flock > --0.43%-- [...] > > > And its not at all clear to me why this would be. It looks like > FILE_LOCK_DEFERRED is happening, but I've not yet figured out why that > would be. So I'd suggest to first compare preemption behavior: does the workload context-switch heavily, and is it the exact same context switching rate and are the points of preemption the same as well between the two kernels? [ Such high variance is often caused by (dynamically) unstable load balancing and the workload never finding a good equilibrium. Any observable locking overhead is usually just a second order concern or a symptom. Assuming the workload context switches heavily. ] Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2015-06-24 11:10 +0200 |
| Message-ID | <pEOQV-5nk-3@gated-at.bofh.it> |
| In reply to | #1171190 |
On Wed, Jun 24, 2015 at 10:46:48AM +0200, Ingo Molnar wrote: > > > > flock02 > > > > mean variance sigma max min > > > > tip-1 11.8994 0.5874 0.7664 13.2022 8.6324 > > > > tip-2 11.7394 0.5252 0.7247 13.2540 9.7513 > > > > tip-3 11.8155 0.5288 0.7272 13.2700 9.9480 > > > > tip+percpu-rswem-1 15.3601 0.8981 0.9477 16.8116 12.6910 > > > > tip+percpu-rswem-2 15.2558 0.8442 0.9188 17.0199 12.9586 > > > > tip+percpu-rswem-3 15.5297 0.6386 0.7991 17.4392 12.7992 > [ Such high variance is often caused by (dynamically) unstable load balancing and > the workload never finding a good equilibrium. Any observable locking overhead > is usually just a second order concern or a symptom. Assuming the workload > context switches heavily. ] flock02 is a relatively stable benchmark -- unlike some of the others where the variance is orders of magnitude higher than the avg. But yes, I'll go poke at it more. I just need to hunt down unrelated fail before continuing with this. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Daniel Wagner <daniel.wagner@bmw-carit.de> |
|---|---|
| Date | 2015-06-24 11:20 +0200 |
| Message-ID | <pEP0C-5yL-9@gated-at.bofh.it> |
| In reply to | #1171190 |
On 06/24/2015 10:46 AM, Ingo Molnar wrote:
> So I'd suggest to first compare preemption behavior: does the workload
> context-switch heavily, and is it the exact same context switching rate and are
> the points of preemption the same as well between the two kernels?
If I read this correctly, the answer is yes.
First the 'stable' flock02 test:
perf stat --repeat 5 --pre 'rm -rf /tmp/a' ~/src/lockperf/flock02 -n 128 -l 64 /tmp/a
0.008793148
0.008784990
0.008587804
0.008693641
0.008776946
Performance counter stats for '/home/wagi/src/lockperf/flock02 -n 128 -l 64 /tmp/a' (5 runs):
76.509634 task-clock (msec) # 3.312 CPUs utilized ( +- 0.67% )
2 context-switches # 0.029 K/sec ( +- 26.50% )
128 cpu-migrations # 0.002 M/sec ( +- 0.31% )
5,295 page-faults # 0.069 M/sec ( +- 0.49% )
89,944,154 cycles # 1.176 GHz ( +- 0.66% )
58,670,259 stalled-cycles-frontend # 65.23% frontend cycles idle ( +- 0.88% )
0 stalled-cycles-backend # 0.00% backend cycles idle
76,991,414 instructions # 0.86 insns per cycle
# 0.76 stalled cycles per insn ( +- 0.19% )
15,239,720 branches # 199.187 M/sec ( +- 0.20% )
103,418 branch-misses # 0.68% of all branches ( +- 6.68% )
0.023102895 seconds time elapsed ( +- 1.09% )
And here posix01 which shows high variance:
perf stat --repeat 5 --pre 'rm -rf /tmp/a' ~/src/lockperf/posix01 -n 128 -l 64 /tmp/a
0.006020402
32.510838421
55.516466069
46.794470223
5.097701438
Performance counter stats for '/home/wagi/src/lockperf/posix01 -n 128 -l 64 /tmp/a' (5 runs):
4177.932106 task-clock (msec) # 14.162 CPUs utilized ( +- 34.59% )
70,646 context-switches # 0.017 M/sec ( +- 31.56% )
28,009 cpu-migrations # 0.007 M/sec ( +- 33.55% )
4,834 page-faults # 0.001 M/sec ( +- 0.98% )
7,291,160,968 cycles # 1.745 GHz ( +- 32.17% )
5,216,204,262 stalled-cycles-frontend # 71.54% frontend cycles idle ( +- 32.13% )
0 stalled-cycles-backend # 0.00% backend cycles idle
1,901,289,780 instructions # 0.26 insns per cycle
# 2.74 stalled cycles per insn ( +- 30.80% )
440,415,914 branches # 105.415 M/sec ( +- 31.06% )
1,347,021 branch-misses # 0.31% of all branches ( +- 29.17% )
0.295016987 seconds time elapsed ( +- 32.01% )
BTW, thanks for the perf stat tip. Really handy!
cheers,
daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Linus Torvalds <torvalds@linux-foundation.org> |
|---|---|
| Date | 2015-07-02 00:00 +0200 |
| Message-ID | <pHycW-2j7-3@gated-at.bofh.it> |
| In reply to | #1171221 |
On Tue, Jun 30, 2015 at 10:57 PM, Daniel Wagner <wagi@monom.org> wrote:
>
> And an attempt at visualization:
>
> http://monom.org/posix01/sweep-4.1.0-02756-ge3d06bd.png
> http://monom.org/posix01/sweep-4.1.0-02769-g6ce2591.png
Ugh. The old numbers look (mostly) fairly tight, and then the new ones
are all over the map, and usually much worse.
We've seen this behavior before when switching from a non-sleeping
lock to a sleeping one. The sleeping locks have absolutely horrible
behavior when they get contended, and spend tons of CPU time on the
sleep/wakeup management, based on almost random timing noise. And it
can get orders of magnitude worse if there are any nested locks that
basically trigger trains of that kind of behavior.
In general, sleeping locks are just horribly horribly bad for things
that do small simple operations. Which is what fs/locks.c does.
I'm not convinced it's fixable. Maybe the new rwsem just isn't a good idea.
Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Peter Zijlstra <peterz@infradead.org> |
|---|---|
| Date | 2015-07-02 11:50 +0200 |
| Message-ID | <pHJi2-Xn-7@gated-at.bofh.it> |
| In reply to | #1175569 |
On Wed, Jul 01, 2015 at 02:54:59PM -0700, Linus Torvalds wrote: > On Tue, Jun 30, 2015 at 10:57 PM, Daniel Wagner <wagi@monom.org> wrote: > > > > And an attempt at visualization: > > > > http://monom.org/posix01/sweep-4.1.0-02756-ge3d06bd.png > > http://monom.org/posix01/sweep-4.1.0-02769-g6ce2591.png > > Ugh. The old numbers look (mostly) fairly tight, and then the new ones > are all over the map, and usually much worse. > > We've seen this behavior before when switching from a non-sleeping > lock to a sleeping one. The sleeping locks have absolutely horrible > behavior when they get contended, and spend tons of CPU time on the > sleep/wakeup management, Right, I'm just not seeing how any of that would happen here :/ The read side would only ever block on reading /proc/$something and I'm fairly sure that benchmark doesn't actually touch that file. In any case, I will look into this, I've just not had time yet.. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web