Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1470903 > unrolled thread

Random abnormal high CPU sys usage related to timer

Back to article view | Back to linux.kernel

Contents

  Random abnormal high CPU sys usage related to timer Mac Lin <mkl0301@gmail.com> - 2016-08-26 19:20 +0200
    Re: Random abnormal high CPU sys usage related to timer Vegard Nossum <vegard.nossum@gmail.com> - 2016-08-26 23:30 +0200
      Re: Random abnormal high CPU sys usage related to timer Mac Lin <mkl0301@gmail.com> - 2016-08-27 17:20 +0200
        Re: Random abnormal high CPU sys usage related to timer Thomas Gleixner <tglx@linutronix.de> - 2016-08-29 12:50 +0200

#1470903 — Random abnormal high CPU sys usage related to timer

[Multipart message — attachments visible in raw view] — view raw

Hi all,

We were having issue with our userspace application which
__sometimes__ result in high CPU sys usage at each execution. The high
sys CPU usage persist until the application is killed.

We simplified the application to just creating a timer and its handler
then does nothing, but looping and sleeping for the timer to be
triggered. With top, the CPU that running the application usually
takes almost 0% for sys usage. But sometimes it will occupy certain
amount of sys usage, up to 100% at most of the time on my embedded
device.

On my laptop, Intel Core i5-4200U, running Ubuntu 14.02.2, Linux
3.13.0-45-generic and 4.4.0-34-generic, the issue can be reproduced
with lower sys usage (7~50%). The same can be reproduced with
buildroot+vanilla kernel 4.7 and 3.13.

Restart the application could temporarily fix the the issue, but there
are chances to happen again.

Googling found one issue that seems related, but no further action.

Keystone II Linux: Random High CPU usage - userspace application using
1 full core - Linux forum - Linux - TI E2E Community
http://e2e.ti.com/support/embedded/linux/f/354/p/433791/1553204

I've checked the /proc/timer_stats, /proc/interrupts, and perf, all
the irq counter, timer counter, timer/irq event didn't show any
abnormal value or useful clue. I'm looking forward for any suggestion
desperately.

Attached the test code and log, and script to detect the issue, which
require dstat and taskset.

Best Regards,
Mac Lin

[toc] | [next] | [standalone]

#1471001

On 26 August 2016 at 19:10, Mac Lin <mkl0301@gmail.com> wrote:
> We were having issue with our userspace application which
> __sometimes__ result in high CPU sys usage at each execution. The high
> sys CPU usage persist until the application is killed.
[...]
> Googling found one issue that seems related, but no further action.

Was it this one? https://lkml.org/lkml/2016/8/23/360

If so you may try to revert commit ff9a9b4c4334 ("sched, time: Switch
VIRT_CPU_ACCOUNTING_GEN to jiffy granularity") to see if it helps.


Vegard

[toc] | [prev] | [next] | [standalone]

#1471197

Hi Vegard,
Thanks for the prompt response.
The commit is introduced since 4.6, but the issue can be reproduced at
3.10 (earliest I have ever test). And testing on buildroot+4.7 with
the commit reverted, the issue still happen.

In fact, I did a test that ran a script that keep increase a counter
for 10 seconds on the same CPU. If I ran 2 of it, the number is half
of running 1. But if I ran it while the issue happened, the counter
reported is around the same value as the 1 process case. So I doubt
that it might be an issue of reported number.

Is there other way to ensure the CPU is "really" doing something?

x=0
trap 'echo x=$x;exit 1' SIGHUP SIGINT SIGTERM
while : ; do
        x=$(($x+1));
#       echo $x;
done


On Sat, Aug 27, 2016 at 5:12 AM, Vegard Nossum <vegard.nossum@gmail.com> wrote:
> On 26 August 2016 at 19:10, Mac Lin <mkl0301@gmail.com> wrote:
>> We were having issue with our userspace application which
>> __sometimes__ result in high CPU sys usage at each execution. The high
>> sys CPU usage persist until the application is killed.
> [...]
>> Googling found one issue that seems related, but no further action.
>
> Was it this one? https://lkml.org/lkml/2016/8/23/360
>
> If so you may try to revert commit ff9a9b4c4334 ("sched, time: Switch
> VIRT_CPU_ACCOUNTING_GEN to jiffy granularity") to see if it helps.
>
>
> Vegard

[toc] | [prev] | [next] | [standalone]

#1471708

On Sat, 27 Aug 2016, Mac Lin wrote:
> Hi Vegard,
> Thanks for the prompt response.
> The commit is introduced since 4.6, but the issue can be reproduced at
> 3.10 (earliest I have ever test). And testing on buildroot+4.7 with
> the commit reverted, the issue still happen.
> 
> In fact, I did a test that ran a script that keep increase a counter
> for 10 seconds on the same CPU. If I ran 2 of it, the number is half
> of running 1. But if I ran it while the issue happened, the counter
> reported is around the same value as the 1 process case. So I doubt
> that it might be an issue of reported number.
> 
> Is there other way to ensure the CPU is "really" doing something?

Tracing will tell you exactly what's going on in the system.

Thanks,

	tglx

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web