Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1424254 > unrolled thread

Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Back to article view | Back to linux.kernel

Contents

  Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Jirka Hladky <jhladky@redhat.com> - 2016-06-16 18:40 +0200
    Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Peter Zijlstra <peterz@infradead.org> - 2016-06-16 19:30 +0200
      Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Jirka Hladky <jhladky@redhat.com> - 2016-06-17 01:10 +0200
        Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Jirka Hladky <jhladky@redhat.com> - 2016-06-21 15:30 +0200
        Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Peter Zijlstra <peterz@infradead.org> - 2016-06-22 09:20 +0200
          Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Peter Zijlstra <peterz@infradead.org> - 2016-06-22 10:00 +0200
            Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Peter Zijlstra <peterz@infradead.org> - 2016-06-22 10:00 +0200
              Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Jirka Hladky <jhladky@redhat.com> - 2016-06-22 12:10 +0200
                Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Peter Zijlstra <peterz@infradead.org> - 2016-06-22 13:20 +0200
                  Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Jirka Hladky <jhladky@redhat.com> - 2016-06-22 14:40 +0200
                    Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Jirka Hladky <jhladky@redhat.com> - 2016-06-22 14:50 +0200
                      Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Jirka Hladky <jhladky@redhat.com> - 2016-06-22 16:50 +0200
                        Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Peter Zijlstra <peterz@infradead.org> - 2016-06-22 23:00 +0200
          Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and  SPECjvm2008 benchmarks against 4.6 kernel Jirka Hladky <jhladky@redhat.com> - 2016-06-22 10:30 +0200

#1424254 — Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

Hello,

we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008
benchmarks starting from 4.7.0-0.rc0 kernel compared to 4.6 kernel.

We have tested kernels 4.7.0-0.rc1 and 4.7.0-0.rc3 and these are as
well affected.

We have observed the drop on variety of different x86_64 servers with
different configuration (different CPU models, RAM sizes, both with
Hyper Threading ON and OFF, different NUMA configurations (2 and 4
NUMA nodes)

Linpack and Stream benchmarks do not show any performance drop.

The performance drop increases with higher number of threads. The
maximum number of threads in each benchmark is the same as number of
CPUs.

We have opened a BZ to track the progress:
https://bugzilla.kernel.org/show_bug.cgi?id=120481

You can find more details along with graphs and tables there.

Do you have any hints which commit should we try to reverse?

Thanks a lot!
Jirka

[toc] | [next] | [standalone]

#1424296

On Thu, Jun 16, 2016 at 06:38:50PM +0200, Jirka Hladky wrote:
> Hello,
> 
> we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008

Blergh, of course I don't have those.. :/

> benchmarks starting from 4.7.0-0.rc0 kernel compared to 4.6 kernel.
> 
> We have tested kernels 4.7.0-0.rc1 and 4.7.0-0.rc3 and these are as
> well affected.
> 
> We have observed the drop on variety of different x86_64 servers with
> different configuration (different CPU models, RAM sizes, both with
> Hyper Threading ON and OFF, different NUMA configurations (2 and 4
> NUMA nodes)

What kind of config and userspace setup? Do you run this cruft in a
cgroup of sorts?

If so, does it change anything if you run it in the root cgroup?

> Linpack and Stream benchmarks do not show any performance drop.
> 
> The performance drop increases with higher number of threads. The
> maximum number of threads in each benchmark is the same as number of
> CPUs.
> 
> We have opened a BZ to track the progress:
> https://bugzilla.kernel.org/show_bug.cgi?id=120481
> 
> You can find more details along with graphs and tables there.
> 
> Do you have any hints which commit should we try to reverse?

There were only 66 commits or so, and I think we can rule out the
hotplug changes, which should reduce it even further.

You could see what the parent of this one does:

  2159197d6677 sched/core: Enable increased load resolution on 64-bit kernels

If not that, maybe the parent of:

  c58d25f371f5 sched/fair: Move record_wakee()

After that I suppose you'll have to go bisect.

[toc] | [prev] | [next] | [standalone]

#1424526

> > we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008
> Blergh, of course I don't have those.. :/

SPECjvm2008 is publicly available.
https://www.spec.org/download.html

We will prepare a reproducer and attach it to the BZ.

> What kind of config and userspace setup? Do you run this cruft in a
> cgroup of sorts?

 No, we don't do any special setup except to control the number of threads.

Thanks for the hints which commits are most likely the root cause for
this. We will try to find the commit which has caused it.

Jirka



On Thu, Jun 16, 2016 at 7:22 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Thu, Jun 16, 2016 at 06:38:50PM +0200, Jirka Hladky wrote:
>> Hello,
>>
>> we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008
>
> Blergh, of course I don't have those.. :/
>
>> benchmarks starting from 4.7.0-0.rc0 kernel compared to 4.6 kernel.
>>
>> We have tested kernels 4.7.0-0.rc1 and 4.7.0-0.rc3 and these are as
>> well affected.
>>
>> We have observed the drop on variety of different x86_64 servers with
>> different configuration (different CPU models, RAM sizes, both with
>> Hyper Threading ON and OFF, different NUMA configurations (2 and 4
>> NUMA nodes)
>
> What kind of config and userspace setup? Do you run this cruft in a
> cgroup of sorts?
>
> If so, does it change anything if you run it in the root cgroup?
>
>> Linpack and Stream benchmarks do not show any performance drop.
>>
>> The performance drop increases with higher number of threads. The
>> maximum number of threads in each benchmark is the same as number of
>> CPUs.
>>
>> We have opened a BZ to track the progress:
>> https://bugzilla.kernel.org/show_bug.cgi?id=120481
>>
>> You can find more details along with graphs and tables there.
>>
>> Do you have any hints which commit should we try to reverse?
>
> There were only 66 commits or so, and I think we can rule out the
> hotplug changes, which should reduce it even further.
>
> You could see what the parent of this one does:
>
>   2159197d6677 sched/core: Enable increased load resolution on 64-bit kernels
>
> If not that, maybe the parent of:
>
>   c58d25f371f5 sched/fair: Move record_wakee()
>
> After that I suppose you'll have to go bisect.
>

[toc] | [prev] | [next] | [standalone]

#1427740

Hi Peter,

I have an update for this performance issue. I have tested several
kernels, I'm not at the parent of

  2159197d6677 sched/core: Enable increased load resolution on 64-bit kernels

and I still see the performance regression for multithreaded workloads.

There are only 27 commits remaining between v4.6 (last known to be OK)
and current HEAD (6ecdd74962f246dfe8750b7bea481a1c0816315d)
6ecdd74962f246dfe8750b7bea481a1c0816315d    sched/fair: Generalize the
load/util averages resolution definitionq hook unless util changed

See below [0].

Any hint which commit should I try now?

Thanks a lot!
Jirka

[0]
$ git log --pretty=oneline v4.6..HEAD kernel/sched
6ecdd74962f246dfe8750b7bea481a1c0816315d sched/fair: Generalize the
load/util averages resolution definition
2159197d66770ec01f75c93fb11dc66df81fd45b sched/core: Enable increased
load resolution on 64-bit kernels
e7904a28f5331c21d17af638cb477c83662e3cb6 locking/lockdep, sched/core:
Implement a better lock pinning scheme
eb58075149b7f0300ff19142e6245fe75db2a081 sched/core: Introduce 'struct rq_flags'
3e71a462dd483ce508a723356b293731e7d788ea sched/core: Move
task_rq_lock() out of line
64b7aad5798478ffff52e110878ccaae4c3aaa34 Merge branch 'sched/urgent'
into sched/core, to pick up fixes before applying new changes
f98db6013c557c216da5038d9c52045be55cd039 sched/core: Add
switch_mm_irqs_off() and use it in the scheduler
594dd290cf5403a9a5818619dfff42d8e8e0518e sched/cpufreq: Optimize
cpufreq update kicker to avoid update multiple times
fec148c000d0f9ac21679601722811eb60b4cc52 sched/deadline: Fix a bug in
dl_overflow()
9fd81dd5ce0b12341c9f83346f8d32ac68bd3841 sched/fair: Optimize
!CONFIG_NO_HZ_COMMON CPU load updates
1f41906a6fda1114debd3898668bd7ab6470ee41 sched/fair: Correctly handle
nohz ticks CPU load accounting
cee1afce3053e7aa0793fbd5f2e845fa2cef9e33 sched/fair: Gather CPU load
functions under a more conventional namespace
a2c6c91f98247fef0fe75216d607812485aeb0df sched/fair: Call cpufreq hook
in additional paths
41e0d37f7ac81297c07ba311e4ad39465b8c8295 sched/fair: Do not call
cpufreq hook unless util changed
21e96f88776deead303ecd30a17d1d7c2a1776e3 sched/fair: Move cpufreq hook
to update_cfs_rq_load_avg()
1f621e028baf391f6684003e32e009bc934b750f sched/fair: Fix asym packing
to select correct CPU
bd92883051a0228cc34996b8e766111ba10c9aac sched/cpuacct: Check for NULL
when using task_pt_regs()
2c923e94cd9c6acff3b22f0ae29cfe65e2658b40 sched/clock: Make
local_clock()/cpu_clock() inline
c78b17e28cc2c2df74264afc408bdc6aaf3fbcc8 sched/clock: Remove pointless
test in cpu_clock/local_clock
fb90a6e93c0684ab2629a42462400603aa829b9c sched/debug: Don't dump sched
debug info in SysRq-W
2b8c41daba327c633228169e8bd8ec067ab443f8 sched/fair: Initiate a new
task's util avg to a bounded value
1c3de5e19fc96206dd086e634129d08e5f7b1000 sched/fair: Update comments
after a variable rename
47252cfbac03644ee4a3adfa50c77896aa94f2bb sched/core: Add preempt
checks in preempt_schedule() code
bfdb198ccd99472c5bded689699eb30dd06316bb sched/numa: Remove
unnecessary NUMA dequeue update from non-SMP kernels
d02c071183e1c01a76811c878c8a52322201f81f sched/fair: Reset
nr_balance_failed after active balancing
d740037fac7052e49450f6fa1454f1144a103b55 sched/cpuacct: Split usage
accounting into user_usage and sys_usage
5ca3726af7f66a8cc71ce4414cfeb86deb784491 sched/cpuacct: Show all
possible CPUs in cpuacct output

On Fri, Jun 17, 2016 at 1:04 AM, Jirka Hladky <jhladky@redhat.com> wrote:
>> > we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008
>> Blergh, of course I don't have those.. :/
>
> SPECjvm2008 is publicly available.
> https://www.spec.org/download.html
>
> We will prepare a reproducer and attach it to the BZ.
>
>> What kind of config and userspace setup? Do you run this cruft in a
>> cgroup of sorts?
>
>  No, we don't do any special setup except to control the number of threads.
>
> Thanks for the hints which commits are most likely the root cause for
> this. We will try to find the commit which has caused it.
>
> Jirka
>
>
>
> On Thu, Jun 16, 2016 at 7:22 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> On Thu, Jun 16, 2016 at 06:38:50PM +0200, Jirka Hladky wrote:
>>> Hello,
>>>
>>> we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008
>>
>> Blergh, of course I don't have those.. :/
>>
>>> benchmarks starting from 4.7.0-0.rc0 kernel compared to 4.6 kernel.
>>>
>>> We have tested kernels 4.7.0-0.rc1 and 4.7.0-0.rc3 and these are as
>>> well affected.
>>>
>>> We have observed the drop on variety of different x86_64 servers with
>>> different configuration (different CPU models, RAM sizes, both with
>>> Hyper Threading ON and OFF, different NUMA configurations (2 and 4
>>> NUMA nodes)
>>
>> What kind of config and userspace setup? Do you run this cruft in a
>> cgroup of sorts?
>>
>> If so, does it change anything if you run it in the root cgroup?
>>
>>> Linpack and Stream benchmarks do not show any performance drop.
>>>
>>> The performance drop increases with higher number of threads. The
>>> maximum number of threads in each benchmark is the same as number of
>>> CPUs.
>>>
>>> We have opened a BZ to track the progress:
>>> https://bugzilla.kernel.org/show_bug.cgi?id=120481
>>>
>>> You can find more details along with graphs and tables there.
>>>
>>> Do you have any hints which commit should we try to reverse?
>>
>> There were only 66 commits or so, and I think we can rule out the
>> hotplug changes, which should reduce it even further.
>>
>> You could see what the parent of this one does:
>>
>>   2159197d6677 sched/core: Enable increased load resolution on 64-bit kernels
>>
>> If not that, maybe the parent of:
>>
>>   c58d25f371f5 sched/fair: Move record_wakee()
>>
>> After that I suppose you'll have to go bisect.
>>

[toc] | [prev] | [next] | [standalone]

#1428494

On Fri, Jun 17, 2016 at 01:04:23AM +0200, Jirka Hladky wrote:
> > > we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008
> > Blergh, of course I don't have those.. :/
> 
> SPECjvm2008 is publicly available.
> https://www.spec.org/download.html

Urgh, I _so_ hate java.

Why does it have to pop up windows split between my screens, total fail.

In any case, I run it like:

   java -jar SPECjvm2008.jar --benchmarkThreads 40

because I have 40 cpus (2 sockets * 10 cores/socket * 2 threads/core).

It seems to produce numbers, but then ends with a splat:

Error while creating report: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper
java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper
        at java.awt.Toolkit.loadAssistiveTechnologies(Toolkit.java:807)
        at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:886)
        at sun.swing.SwingUtilities2.getSystemMnemonicKeyMask(SwingUtilities2.java:2020)
        at javax.swing.plaf.basic.BasicLookAndFeel.initComponentDefaults(BasicLookAndFeel.java:1158)
        at javax.swing.plaf.metal.MetalLookAndFeel.initComponentDefaults(MetalLookAndFeel.java:431)
        at javax.swing.plaf.basic.BasicLookAndFeel.getDefaults(BasicLookAndFeel.java:148)
        at javax.swing.plaf.metal.MetalLookAndFeel.getDefaults(MetalLookAndFeel.java:1577)
        at javax.swing.UIManager.setLookAndFeel(UIManager.java:539)
        at javax.swing.UIManager.setLookAndFeel(UIManager.java:579)
        at javax.swing.UIManager.initializeDefaultLAF(UIManager.java:1349)
        at javax.swing.UIManager.initialize(UIManager.java:1459)
        at javax.swing.UIManager.maybeInitialize(UIManager.java:1426)
        at javax.swing.UIManager.getDefaults(UIManager.java:659)
        at javax.swing.UIManager.getColor(UIManager.java:701)
        at org.jfree.chart.JFreeChart.<clinit>(JFreeChart.java:246)
        at org.jfree.chart.ChartFactory.createXYLineChart(ChartFactory.java:1478)
        at spec.reporter.BenchmarkChart.<init>(BenchmarkChart.java:47)
        at spec.reporter.ReportGenerator.handleBenchmarkResult(ReportGenerator.java:141)
        at spec.reporter.ReportGenerator.handleBenchmarksResults(ReportGenerator.java:105)
        at spec.reporter.ReportGenerator.<init>(ReportGenerator.java:87)
        at spec.reporter.ReportGenerator.main2(ReportGenerator.java:750)
        at spec.reporter.Reporter.main2(Reporter.java:51)
        at spec.harness.Launch.createReport(Launch.java:307)
        at spec.harness.Launch.runBenchmarkSuite(Launch.java:250)
        at spec.harness.Launch.main(Launch.java:452)

WTF a benchmark needs that crap is beyond me, but whatever, I have
numbers.

I'll try and reproduce.

[toc] | [prev] | [next] | [standalone]

#1428526

On Wed, Jun 22, 2016 at 09:16:01AM +0200, Peter Zijlstra wrote:
> WTF a benchmark needs that crap is beyond me, but whatever, I have
> numbers.

Oh, shaft me harder, its XML shite :/ How is a sane person ever going to
get numbers out.

I'm >.< close to giving up on this site and declaring the thing
-EDONTCARE.

[toc] | [prev] | [next] | [standalone]

#1428529

On Wed, Jun 22, 2016 at 09:49:41AM +0200, Peter Zijlstra wrote:
> On Wed, Jun 22, 2016 at 09:16:01AM +0200, Peter Zijlstra wrote:
> > WTF a benchmark needs that crap is beyond me, but whatever, I have
> > numbers.
> 
> Oh, shaft me harder, its XML shite :/ How is a sane person ever going to
> get numbers out.
> 
> I'm >.< close to giving up on this site and declaring the thing
> -EDONTCARE.

OK, done.. have a look at this:


/tmp/SPECjvm2008/compiler.compiler/compiler/src/share/classes/javax/lang/model/element/Name.java:54: cannot access java.lang.CharSequence
bad class file: spec.benchmarks.compiler.SpecFileManager$CachedFileObject@1c06fce6
bad constant pool tag: 18 at 10
Please remove or make sure it appears in the correct subdirectory of the classpath.
public interface Name extends CharSequence {
                              ^
ERROR: compiler exit code: 1

Warmup (120s) begins: Wed Jun 22 09:45:33 CEST 2016
/tmp/SPECjvm2008/compiler.compiler/compiler/src/share/classes/javax/lang/model/element/Name.java:54: cannot access java.lang.CharSequence
bad class file: spec.benchmarks.compiler.SpecFileManager$CachedFileObject@1c06fce6
bad constant pool tag: 18 at 10
Please remove or make sure it appears in the correct subdirectory of the classpath.
public interface Name extends CharSequence {
                              ^
/tmp/SPECjvm2008/compiler.compiler/compiler/src/share/classes/javax/lang/model/element/Name.java:54: cannot access java.lang.CharSequence
bad class file: spec.benchmarks.compiler.SpecFileManager$CachedFileObject@1c06fce6
bad constant pool tag: 18 at 10
Please remove or make sure it appears in the correct subdirectory of the classpath.



Clearly this stuff just isn't made to be used.


/me goes do something useful.

[toc] | [prev] | [next] | [standalone]

#1428644

Hi Peter,

the performance regression has been caused by this commit

=================================================
commit 6ecdd74962f246dfe8750b7bea481a1c0816315d
Author: Yuyang Du <yuyang.du@intel.com>
Date:   Tue Apr 5 12:12:26 2016 +0800

    sched/fair: Generalize the load/util averages resolution definition
=================================================

Could you please have a look?

Thanks a lot!
Jirka


On Wed, Jun 22, 2016 at 9:54 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, Jun 22, 2016 at 09:49:41AM +0200, Peter Zijlstra wrote:
>> On Wed, Jun 22, 2016 at 09:16:01AM +0200, Peter Zijlstra wrote:
>> > WTF a benchmark needs that crap is beyond me, but whatever, I have
>> > numbers.
>>
>> Oh, shaft me harder, its XML shite :/ How is a sane person ever going to
>> get numbers out.
>>
>> I'm >.< close to giving up on this site and declaring the thing
>> -EDONTCARE.
>
> OK, done.. have a look at this:
>
>
> /tmp/SPECjvm2008/compiler.compiler/compiler/src/share/classes/javax/lang/model/element/Name.java:54: cannot access java.lang.CharSequence
> bad class file: spec.benchmarks.compiler.SpecFileManager$CachedFileObject@1c06fce6
> bad constant pool tag: 18 at 10
> Please remove or make sure it appears in the correct subdirectory of the classpath.
> public interface Name extends CharSequence {
>                               ^
> ERROR: compiler exit code: 1
>
> Warmup (120s) begins: Wed Jun 22 09:45:33 CEST 2016
> /tmp/SPECjvm2008/compiler.compiler/compiler/src/share/classes/javax/lang/model/element/Name.java:54: cannot access java.lang.CharSequence
> bad class file: spec.benchmarks.compiler.SpecFileManager$CachedFileObject@1c06fce6
> bad constant pool tag: 18 at 10
> Please remove or make sure it appears in the correct subdirectory of the classpath.
> public interface Name extends CharSequence {
>                               ^
> /tmp/SPECjvm2008/compiler.compiler/compiler/src/share/classes/javax/lang/model/element/Name.java:54: cannot access java.lang.CharSequence
> bad class file: spec.benchmarks.compiler.SpecFileManager$CachedFileObject@1c06fce6
> bad constant pool tag: 18 at 10
> Please remove or make sure it appears in the correct subdirectory of the classpath.
>
>
>
> Clearly this stuff just isn't made to be used.
>
>
> /me goes do something useful.

[toc] | [prev] | [next] | [standalone]

#1428694

On Wed, Jun 22, 2016 at 11:52:45AM +0200, Jirka Hladky wrote:
> Hi Peter,
> 
> the performance regression has been caused by this commit
> 
> =================================================
> commit 6ecdd74962f246dfe8750b7bea481a1c0816315d
> Author: Yuyang Du <yuyang.du@intel.com>
> Date:   Tue Apr 5 12:12:26 2016 +0800
> 
>     sched/fair: Generalize the load/util averages resolution definition
> =================================================
> 
> Could you please have a look?

That patch looks like a NO-OP to me.

In any case, the good news it that I can run the benchmark, the bad news
is that the patch you fingered doesn't appear to be it.


v4.60:
./4.6.0/2016-Jun-22_11h11m07s.log:Score on xml.transform: 2007.18 ops/m
./4.6.0/2016-Jun-22_11h11m07s.log:Score on xml.validation: 2999.44 ops/m

tip/master:
./4.7.0-rc4-00345-gf6e78bb/2016-Jun-22_11h30m27s.log:Score on xml.transform: 1283.14 ops/m
./4.7.0-rc4-00345-gf6e78bb/2016-Jun-22_11h30m27s.log:Score on xml.validation: 2008.62 ops/m

patch^1
./4.6.0-rc5-00034-g2159197/2016-Jun-22_12h38m50s.log:Score on xml.transform: 1196.18 ops/m
./4.6.0-rc5-00034-g2159197/2016-Jun-22_12h38m50s.log:Score on xml.validation: 2055.11 ops/m

patch^1 + patch
./4.6.0-rc5-00034-g2159197-dirty/2016-Jun-22_12h55m43s.log:Score on xml.transform: 1294.59 ops/m
./4.6.0-rc5-00034-g2159197-dirty/2016-Jun-22_12h55m43s.log:Score on xml.validation: 2140.02 ops/m

[toc] | [prev] | [next] | [standalone]

#1428759

Hi Peter,

crap - I have done bisecting manually (not using git bisect) and I
have probably done some mistake.

Commits (git checkout <commit>) for which I got BAD results:

2159197d66770ec01f75c93fb11dc66df81fd45b
6ecdd74962f246dfe8750b7bea481a1c0816315d

Commits (git checkout <commit>) for which I got GOOD results:
21e96f88776deead303ecd30a17d1d7c2a1776e3
64b7aad5798478ffff52e110878ccaae4c3aaa34
e7904a28f5331c21d17af638cb477c83662e3cb6

I will try to use git bisect now.

Jirka

On Wed, Jun 22, 2016 at 1:12 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, Jun 22, 2016 at 11:52:45AM +0200, Jirka Hladky wrote:
>> Hi Peter,
>>
>> the performance regression has been caused by this commit
>>
>> =================================================
>> commit 6ecdd74962f246dfe8750b7bea481a1c0816315d
>> Author: Yuyang Du <yuyang.du@intel.com>
>> Date:   Tue Apr 5 12:12:26 2016 +0800
>>
>>     sched/fair: Generalize the load/util averages resolution definition
>> =================================================
>>
>> Could you please have a look?
>
> That patch looks like a NO-OP to me.
>
> In any case, the good news it that I can run the benchmark, the bad news
> is that the patch you fingered doesn't appear to be it.
>
>
> v4.60:
> ./4.6.0/2016-Jun-22_11h11m07s.log:Score on xml.transform: 2007.18 ops/m
> ./4.6.0/2016-Jun-22_11h11m07s.log:Score on xml.validation: 2999.44 ops/m
>
> tip/master:
> ./4.7.0-rc4-00345-gf6e78bb/2016-Jun-22_11h30m27s.log:Score on xml.transform: 1283.14 ops/m
> ./4.7.0-rc4-00345-gf6e78bb/2016-Jun-22_11h30m27s.log:Score on xml.validation: 2008.62 ops/m
>
> patch^1
> ./4.6.0-rc5-00034-g2159197/2016-Jun-22_12h38m50s.log:Score on xml.transform: 1196.18 ops/m
> ./4.6.0-rc5-00034-g2159197/2016-Jun-22_12h38m50s.log:Score on xml.validation: 2055.11 ops/m
>
> patch^1 + patch
> ./4.6.0-rc5-00034-g2159197-dirty/2016-Jun-22_12h55m43s.log:Score on xml.transform: 1294.59 ops/m
> ./4.6.0-rc5-00034-g2159197-dirty/2016-Jun-22_12h55m43s.log:Score on xml.validation: 2140.02 ops/m
>
>

[toc] | [prev] | [next] | [standalone]

#1428765

OK, I have reviewed my results once again:

This commit is fine:
64b7aad - Ingo Molnar, 7 weeks ago : Merge branch 'sched/urgent' into
sched/core, to pick up fixes before applying new changes

This version has already a problem:
2159197 - Peter Zijlstra, 8 weeks ago : sched/core: Enable increased
load resolution on 64-bit kernels

git bisect start
git bisect good 64b7aad
git bisect bad 2159197
Bisecting: 1 revision left to test after this (roughly 1 step)
[eb58075149b7f0300ff19142e6245fe75db2a081] sched/core: Introduce
'struct rq_flags'

I should have results pretty soon.

Jirka


On Wed, Jun 22, 2016 at 2:37 PM, Jirka Hladky <jhladky@redhat.com> wrote:
> Hi Peter,
>
> crap - I have done bisecting manually (not using git bisect) and I
> have probably done some mistake.
>
> Commits (git checkout <commit>) for which I got BAD results:
>
> 2159197d66770ec01f75c93fb11dc66df81fd45b
> 6ecdd74962f246dfe8750b7bea481a1c0816315d
>
> Commits (git checkout <commit>) for which I got GOOD results:
> 21e96f88776deead303ecd30a17d1d7c2a1776e3
> 64b7aad5798478ffff52e110878ccaae4c3aaa34
> e7904a28f5331c21d17af638cb477c83662e3cb6
>
> I will try to use git bisect now.
> 
> Jirka
>
> On Wed, Jun 22, 2016 at 1:12 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> On Wed, Jun 22, 2016 at 11:52:45AM +0200, Jirka Hladky wrote:
>>> Hi Peter,
>>>
>>> the performance regression has been caused by this commit
>>>
>>> =================================================
>>> commit 6ecdd74962f246dfe8750b7bea481a1c0816315d
>>> Author: Yuyang Du <yuyang.du@intel.com>
>>> Date:   Tue Apr 5 12:12:26 2016 +0800
>>>
>>>     sched/fair: Generalize the load/util averages resolution definition
>>> =================================================
>>>
>>> Could you please have a look?
>>
>> That patch looks like a NO-OP to me.
>>
>> In any case, the good news it that I can run the benchmark, the bad news
>> is that the patch you fingered doesn't appear to be it.
>>
>>
>> v4.60:
>> ./4.6.0/2016-Jun-22_11h11m07s.log:Score on xml.transform: 2007.18 ops/m
>> ./4.6.0/2016-Jun-22_11h11m07s.log:Score on xml.validation: 2999.44 ops/m
>>
>> tip/master:
>> ./4.7.0-rc4-00345-gf6e78bb/2016-Jun-22_11h30m27s.log:Score on xml.transform: 1283.14 ops/m
>> ./4.7.0-rc4-00345-gf6e78bb/2016-Jun-22_11h30m27s.log:Score on xml.validation: 2008.62 ops/m
>>
>> patch^1
>> ./4.6.0-rc5-00034-g2159197/2016-Jun-22_12h38m50s.log:Score on xml.transform: 1196.18 ops/m
>> ./4.6.0-rc5-00034-g2159197/2016-Jun-22_12h38m50s.log:Score on xml.validation: 2055.11 ops/m
>>
>> patch^1 + patch
>> ./4.6.0-rc5-00034-g2159197-dirty/2016-Jun-22_12h55m43s.log:Score on xml.transform: 1294.59 ops/m
>> ./4.6.0-rc5-00034-g2159197-dirty/2016-Jun-22_12h55m43s.log:Score on xml.validation: 2140.02 ops/m
>>
>>

[toc] | [prev] | [next] | [standalone]

#1428866

Hi Peter,

the kernel I got with bisecting does not work - I'm getting kernel
panic during the boot.

In any case, the regression was introduced between
git bisect good 64b7aad
git bisect bad 2159197

This commit is good:
64b7aad - Ingo Molnar, 7 weeks ago : Merge branch 'sched/urgent' into
sched/core, to pick up fixes before applying new changes

This commit is bad:
2159197 - Peter Zijlstra, 8 weeks ago : sched/core: Enable increased
load resolution on 64-bit kernels

Could you please have a look?

Thanks a lot!
Jirka


On Wed, Jun 22, 2016 at 2:46 PM, Jirka Hladky <jhladky@redhat.com> wrote:
> OK, I have reviewed my results once again:
>
> This commit is fine:
> 64b7aad - Ingo Molnar, 7 weeks ago : Merge branch 'sched/urgent' into
> sched/core, to pick up fixes before applying new changes
>
> This version has already a problem:
> 2159197 - Peter Zijlstra, 8 weeks ago : sched/core: Enable increased
> load resolution on 64-bit kernels
>
> git bisect start
> git bisect good 64b7aad
> git bisect bad 2159197
> Bisecting: 1 revision left to test after this (roughly 1 step)
> [eb58075149b7f0300ff19142e6245fe75db2a081] sched/core: Introduce
> 'struct rq_flags'
>
> I should have results pretty soon.
>
> Jirka
>
>
> On Wed, Jun 22, 2016 at 2:37 PM, Jirka Hladky <jhladky@redhat.com> wrote:
>> Hi Peter,
>>
>> crap - I have done bisecting manually (not using git bisect) and I
>> have probably done some mistake.
>>
>> Commits (git checkout <commit>) for which I got BAD results:
>>
>> 2159197d66770ec01f75c93fb11dc66df81fd45b
>> 6ecdd74962f246dfe8750b7bea481a1c0816315d
>>
>> Commits (git checkout <commit>) for which I got GOOD results:
>> 21e96f88776deead303ecd30a17d1d7c2a1776e3
>> 64b7aad5798478ffff52e110878ccaae4c3aaa34
>> e7904a28f5331c21d17af638cb477c83662e3cb6
>>
>> I will try to use git bisect now.
>> 
>> Jirka
>>
>> On Wed, Jun 22, 2016 at 1:12 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>>> On Wed, Jun 22, 2016 at 11:52:45AM +0200, Jirka Hladky wrote:
>>>> Hi Peter,
>>>>
>>>> the performance regression has been caused by this commit
>>>>
>>>> =================================================
>>>> commit 6ecdd74962f246dfe8750b7bea481a1c0816315d
>>>> Author: Yuyang Du <yuyang.du@intel.com>
>>>> Date:   Tue Apr 5 12:12:26 2016 +0800
>>>>
>>>>     sched/fair: Generalize the load/util averages resolution definition
>>>> =================================================
>>>>
>>>> Could you please have a look?
>>>
>>> That patch looks like a NO-OP to me.
>>>
>>> In any case, the good news it that I can run the benchmark, the bad news
>>> is that the patch you fingered doesn't appear to be it.
>>>
>>>
>>> v4.60:
>>> ./4.6.0/2016-Jun-22_11h11m07s.log:Score on xml.transform: 2007.18 ops/m
>>> ./4.6.0/2016-Jun-22_11h11m07s.log:Score on xml.validation: 2999.44 ops/m
>>>
>>> tip/master:
>>> ./4.7.0-rc4-00345-gf6e78bb/2016-Jun-22_11h30m27s.log:Score on xml.transform: 1283.14 ops/m
>>> ./4.7.0-rc4-00345-gf6e78bb/2016-Jun-22_11h30m27s.log:Score on xml.validation: 2008.62 ops/m
>>>
>>> patch^1
>>> ./4.6.0-rc5-00034-g2159197/2016-Jun-22_12h38m50s.log:Score on xml.transform: 1196.18 ops/m
>>> ./4.6.0-rc5-00034-g2159197/2016-Jun-22_12h38m50s.log:Score on xml.validation: 2055.11 ops/m
>>>
>>> patch^1 + patch
>>> ./4.6.0-rc5-00034-g2159197-dirty/2016-Jun-22_12h55m43s.log:Score on xml.transform: 1294.59 ops/m
>>> ./4.6.0-rc5-00034-g2159197-dirty/2016-Jun-22_12h55m43s.log:Score on xml.validation: 2140.02 ops/m
>>>
>>>

[toc] | [prev] | [next] | [standalone]

#1429109

On Wed, Jun 22, 2016 at 04:41:06PM +0200, Jirka Hladky wrote:
> This commit is bad:
> 2159197 - Peter Zijlstra, 8 weeks ago : sched/core: Enable increased
> load resolution on 64-bit kernels
> 
> Could you please have a look?

Yes, that is indeed the culprit.

The below 'revert' makes it go fast again. I'll try and figure out
what's wrong tomorrow.

---

diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index bf6fea9..e7e312b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -55,7 +55,7 @@ static inline void cpu_load_update_active(struct rq *this_rq) { }
  * Really only required when CONFIG_FAIR_GROUP_SCHED is also set, but to
  * increase coverage and consistency always enable it on 64bit platforms.
  */
-#ifdef CONFIG_64BIT
+#if 0 // def CONFIG_64BIT
 # define NICE_0_LOAD_SHIFT	(SCHED_FIXEDPOINT_SHIFT + SCHED_FIXEDPOINT_SHIFT)
 # define scale_load(w)		((w) << SCHED_FIXEDPOINT_SHIFT)
 # define scale_load_down(w)	((w) >> SCHED_FIXEDPOINT_SHIFT)

[toc] | [prev] | [next] | [standalone]

#1428545

[Multipart message — attachments visible in raw view] — view raw

Hi Peter,

please find the reproducer script attached. My command to reproduce the bug is:

./run-specjvm.sh --benchmarkThreads 32 --iterations 1 --iterationTime
180 --warmuptime 90 xml.transform xml.validation

I run just xml benchmarks to speed up the runtime.

Please check
 https://bugzilla.kernel.org/show_bug.cgi?id=120481#c9
for some details how to run the benchmark.

The benchmark needs Window manager to be installed to create graphs.
However, you can run the script from ssh terminal. I don't know
exactly why is that but I know that Python's matplot library has the
same requirements.

last known good commit: e7904a28f5331c21d17af638cb477c83662e3cb6
first known bad commit: 6ecdd74962f246dfe8750b7bea481a1c0816315d

Last two commits to be checked:

 git log --pretty=oneline
e7904a28f5331c21d17af638cb477c83662e3cb6..6ecdd74962f246dfe8750b7bea481a1c0816315d
6ecdd74962f246dfe8750b7bea481a1c0816315d sched/fair: Generalize the
load/util averages resolution definition
2159197d66770ec01f75c93fb11dc66df81fd45b sched/core: Enable increased
load resolution on 64-bit kernels

I use following command to review the results produced by reproduce.sh script.

find ./ -name "*log" | xargs grep -H Score | grep xml.validation |
grep "[0-9]\{4\}[.][0-9]\{2\} ops/m"

Jirka

On Wed, Jun 22, 2016 at 9:16 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, Jun 17, 2016 at 01:04:23AM +0200, Jirka Hladky wrote:
>> > > we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008
>> > Blergh, of course I don't have those.. :/
>>
>> SPECjvm2008 is publicly available.
>> https://www.spec.org/download.html
>
> Urgh, I _so_ hate java.
>
> Why does it have to pop up windows split between my screens, total fail.
>
> In any case, I run it like:
>
>    java -jar SPECjvm2008.jar --benchmarkThreads 40
>
> because I have 40 cpus (2 sockets * 10 cores/socket * 2 threads/core).
>
> It seems to produce numbers, but then ends with a splat:
>
> Error while creating report: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper
> java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper
>         at java.awt.Toolkit.loadAssistiveTechnologies(Toolkit.java:807)
>         at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:886)
>         at sun.swing.SwingUtilities2.getSystemMnemonicKeyMask(SwingUtilities2.java:2020)
>         at javax.swing.plaf.basic.BasicLookAndFeel.initComponentDefaults(BasicLookAndFeel.java:1158)
>         at javax.swing.plaf.metal.MetalLookAndFeel.initComponentDefaults(MetalLookAndFeel.java:431)
>         at javax.swing.plaf.basic.BasicLookAndFeel.getDefaults(BasicLookAndFeel.java:148)
>         at javax.swing.plaf.metal.MetalLookAndFeel.getDefaults(MetalLookAndFeel.java:1577)
>         at javax.swing.UIManager.setLookAndFeel(UIManager.java:539)
>         at javax.swing.UIManager.setLookAndFeel(UIManager.java:579)
>         at javax.swing.UIManager.initializeDefaultLAF(UIManager.java:1349)
>         at javax.swing.UIManager.initialize(UIManager.java:1459)
>         at javax.swing.UIManager.maybeInitialize(UIManager.java:1426)
>         at javax.swing.UIManager.getDefaults(UIManager.java:659)
>         at javax.swing.UIManager.getColor(UIManager.java:701)
>         at org.jfree.chart.JFreeChart.<clinit>(JFreeChart.java:246)
>         at org.jfree.chart.ChartFactory.createXYLineChart(ChartFactory.java:1478)
>         at spec.reporter.BenchmarkChart.<init>(BenchmarkChart.java:47)
>         at spec.reporter.ReportGenerator.handleBenchmarkResult(ReportGenerator.java:141)
>         at spec.reporter.ReportGenerator.handleBenchmarksResults(ReportGenerator.java:105)
>         at spec.reporter.ReportGenerator.<init>(ReportGenerator.java:87)
>         at spec.reporter.ReportGenerator.main2(ReportGenerator.java:750)
>         at spec.reporter.Reporter.main2(Reporter.java:51)
>         at spec.harness.Launch.createReport(Launch.java:307)
>         at spec.harness.Launch.runBenchmarkSuite(Launch.java:250)
>         at spec.harness.Launch.main(Launch.java:452)
>
> WTF a benchmark needs that crap is beyond me, but whatever, I have
> numbers.
>
> I'll try and reproduce.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web