Bug#888465: CPU usage reporting issues

From	Ryan Thoryk <ryan@thoryk.com>
Newsgroups	linux.debian.bugs.dist, linux.debian.kernel
Subject	Bug#888465: CPU usage reporting issues
Date	2018-01-26 01:20 +0100
Message-ID	<vbZUm-4tF-9@gated-at.bofh.it> (permalink)
References	<vbZ7Y-3UM-9@gated-at.bofh.it> <vbZAZ-47K-1@gated-at.bofh.it> <vbZ7Y-3UM-9@gated-at.bofh.it> <vbZAZ-47K-1@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Cross-posted to 2 groups.

Show all headers | View raw

On 01/25/2018 05:40 PM, Hans van Kranenburg wrote:
> Hi Ryan,
>
> On 01/26/2018 12:20 AM, Ryan Thoryk wrote:
>> Package: linux-image-4.9.0-5-amd64
>> Version: 4.9.65-3+deb9u2
>> Severity: normal
>>
>> I'm having an issue with CPU usage reporting, tested on kernels 4.9.0-3
>> and 4.9.0-5.  The machines are running on Amazon EC2, which could be
>> related.  With the "sar" utility, after some time, the system's "steal"
>> value periodically is 100%,
> This means that your vcpus want to execute work but are not being
> scheduled on a physical cpu core. Either the physical machine gets too
> much work from all the virtual machines that are requesting cpu time, or
> other things are going on, like your virtual machine getting paused
> (e.g. when doing live migration there's a handover moment when it's
> shortly paused and then resumed, this is also visible as a short 100%
> steal spike).
After going over log files, it appears that the issue started when 
Amazon did a live migration of the VM, probably for the Meltdown patching.
>> and the normal CPU user/system values,
>> including idle, are always 0.  When running a cpu-intensive app and
>> using the "top" utility, the user and system values are always 0, the
>> "idle" field stays at 100%, and only the "wait" field increases.
> Sounds a lot like this one:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871608
>
> A patch to fix that cpu accounting breakage (picked from linux 4.15) was
> included in 4.9.65-3. So only for the 4.9.0-3 (which actual version?)
> you could be seeing that one happening.
The versions were both 4.9.30-2+deb9u2 and the latest, 4.9.65-3+deb9u2.  
So basically the kernel never recovered properly after being paused 
during a live migration.
>> The attached file shows the "sar" output around the time the issue
>> started.  This has happened on 2 separate machines (started at different
>> times on each), and a reboot appears to (temporarily) fix the issue.
>> I'm wondering if anyone else has this issue, and if it could be
>> something to do with the hypervisor.
> Because of the mentioned steal time fix that was included in a version
> in between the 2 versions you mention, my first suggestion would be to
> see if the symptoms on the old and new kernel are exactly the same, or
> if they are only similar but different.
>
> Hans
I already rebooted the system running the 4.9.65 kernel, and beforehand, 
the symptoms were the same.  The CPU usage stats went back to normal 
after the reboot.

-- 
Ryan Thoryk
ryan@thoryk.com
ryan@tliquest.net

Back to linux.debian.kernel | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Bug#888465: CPU usage reporting issues Ryan Thoryk <ryan@thoryk.com> - 2018-01-26 00:30 +0100
  Bug#888465: Additional example Ryan Thoryk <ryan@thoryk.com> - 2018-01-26 00:50 +0100
  Bug#888465: CPU usage reporting issues Hans van Kranenburg <hans@knorrie.org> - 2018-01-26 01:00 +0100
    Bug#888465: CPU usage reporting issues Ryan Thoryk <ryan@thoryk.com> - 2018-01-26 01:20 +0100
    Bug#888465: CPU usage reporting issues Ryan Thoryk <ryan@thoryk.com> - 2018-01-26 01:20 +0100

csiph-web