Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1369909 > unrolled thread

"perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0

Back to article view | Back to linux.kernel

Contents

  "perf: interrupt took too long" messages even with  perf_cpu_time_max_percent==0 Markus Trippelsdorf <markus@trippelsdorf.de> - 2016-04-02 11:40 +0200
    Re: "perf: interrupt took too long" messages even with  perf_cpu_time_max_percent==0 Peter Zijlstra <peterz@infradead.org> - 2016-04-02 13:10 +0200
      Re: "perf: interrupt took too long" messages even with  perf_cpu_time_max_percent==0 Markus Trippelsdorf <markus@trippelsdorf.de> - 2016-04-02 13:20 +0200
        Re: "perf: interrupt took too long" messages even with  perf_cpu_time_max_percent==0 Markus Trippelsdorf <markus@trippelsdorf.de> - 2016-04-02 13:30 +0200
        Re: "perf: interrupt took too long" messages even with  perf_cpu_time_max_percent==0 Peter Zijlstra <peterz@infradead.org> - 2016-04-02 18:40 +0200
      Re: "perf: interrupt took too long" messages even with  perf_cpu_time_max_percent==0 Markus Trippelsdorf <markus@trippelsdorf.de> - 2016-04-13 12:10 +0200

#1369909 — "perf: interrupt took too long" messages even with perf_cpu_time_max_percent==0

Current git kernel sometimes shows:

 perf: interrupt took too long (71 > 52), lowering kernel.perf_event_max_sample_rate to 300
 perf: interrupt took too long (103 > 88), lowering kernel.perf_event_max_sample_rate to 300
 perf: interrupt took too long (130 > 128), lowering kernel.perf_event_max_sample_rate to 300
 perf: interrupt took too long (175 > 162), lowering kernel.perf_event_max_sample_rate to 300
 perf: interrupt took too long (219 > 218), lowering kernel.perf_event_max_sample_rate to 300
...

when running e.g. "perf top" even when  
/proc/sys/kernel/perf_cpu_time_max_percent is set to 0.

-- 
Markus

[toc] | [next] | [standalone]

#1369917

On Sat, Apr 02, 2016 at 11:32:39AM +0200, Markus Trippelsdorf wrote:
> Current git kernel sometimes shows:
> 
>  perf: interrupt took too long (71 > 52), lowering kernel.perf_event_max_sample_rate to 300
>  perf: interrupt took too long (103 > 88), lowering kernel.perf_event_max_sample_rate to 300
>  perf: interrupt took too long (130 > 128), lowering kernel.perf_event_max_sample_rate to 300
>  perf: interrupt took too long (175 > 162), lowering kernel.perf_event_max_sample_rate to 300
>  perf: interrupt took too long (219 > 218), lowering kernel.perf_event_max_sample_rate to 300
> ...
> 
> when running e.g. "perf top" even when  
> /proc/sys/kernel/perf_cpu_time_max_percent is set to 0.


Ah, was 0 also meant to disable it?

Does the below help?

---
 kernel/events/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 8c3b35f2a269..21ba024c9ed1 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -412,7 +412,8 @@ int perf_cpu_time_max_percent_handler(struct ctl_table *table, int write,
 	if (ret || !write)
 		return ret;
 
-	if (sysctl_perf_cpu_time_max_percent == 100) {
+	if (sysctl_perf_cpu_time_max_percent == 100 ||
+	    sysctl_perf_cpu_time_max_percent == 0) {
 		printk(KERN_WARNING
 		       "perf: Dynamic interrupt throttling disabled, can hang your system!\n");
 		WRITE_ONCE(perf_sample_allowed_ns, 0);

[toc] | [prev] | [next] | [standalone]

#1369925

On 2016.04.02 at 13:00 +0200, Peter Zijlstra wrote:
> On Sat, Apr 02, 2016 at 11:32:39AM +0200, Markus Trippelsdorf wrote:
> > Current git kernel sometimes shows:
> >
> >  perf: interrupt took too long (71 > 52), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (103 > 88), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (130 > 128), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (175 > 162), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (219 > 218), lowering kernel.perf_event_max_sample_rate to 300
> > ...
> >
> > when running e.g. "perf top" even when
> > /proc/sys/kernel/perf_cpu_time_max_percent is set to 0.
>
>
> Ah, was 0 also meant to disable it?

Yes. From Documentation/sysctl/kernel.txt:

 perf_cpu_time_max_percent:
 
 Hints to the kernel how much CPU time it should be allowed to
 use to handle perf sampling events.  If the perf subsystem
 is informed that its samples are exceeding this limit, it
 will drop its sampling frequency to attempt to reduce its CPU
 usage.
 
 Some perf sampling happens in NMIs.  If these samples
 unexpectedly take too long to execute, the NMIs can become
 stacked up next to each other so much that nothing else is
 allowed to execute.
 
 0: disable the mechanism.  Do not monitor or correct perf's
    sampling rate no matter how CPU time it takes.
 
 1-100: attempt to throttle perf's sample rate to this
    percentage of CPU.  Note: the kernel calculates an
    "expected" length of each sample event.  100 here means
    100% of that expected length.  Even if this is set to
    100, you may still see sample throttling if this
    length is exceeded.  Set to 0 if you truly do not care
    how much CPU is consumed.

> Does the below help?

Thanks. I will test it later. But 91a612eea9a3 makes the assumption that only
sysctl_perf_cpu_time_max_percent==100 disables the feature also in
kernel/events/core.c.

--
Markus

[toc] | [prev] | [next] | [standalone]

#1369928

On 2016.04.02 at 13:17 +0200, Markus Trippelsdorf wrote:
> > Does the below help?
> 
> Thanks. I will test it later. But 91a612eea9a3 makes the assumption that only
> sysctl_perf_cpu_time_max_percent==100 disables the feature also in
> kernel/events/core.c.

Please ignore the last sentence. It doesn't make sense.

-- 
Markus

[toc] | [prev] | [next] | [standalone]

#1369989

On Sat, Apr 02, 2016 at 01:17:26PM +0200, Markus Trippelsdorf wrote:
> On 2016.04.02 at 13:00 +0200, Peter Zijlstra wrote:

> > Ah, was 0 also meant to disable it?
> 
> Yes. From Documentation/sysctl/kernel.txt:

Oh wow, we have documentation on this! Learn something new every day :-)

[toc] | [prev] | [next] | [standalone]

#1377718

On 2016.04.02 at 13:00 +0200, Peter Zijlstra wrote:
> On Sat, Apr 02, 2016 at 11:32:39AM +0200, Markus Trippelsdorf wrote:
> > Current git kernel sometimes shows:
> > 
> >  perf: interrupt took too long (71 > 52), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (103 > 88), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (130 > 128), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (175 > 162), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (219 > 218), lowering kernel.perf_event_max_sample_rate to 300
> > ...
> > 
> > when running e.g. "perf top" even when  
> > /proc/sys/kernel/perf_cpu_time_max_percent is set to 0.
> 
> 
> Ah, was 0 also meant to disable it?
> 
> Does the below help?

Yes, it obviously fixes the issue.

-- 
Markus

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web