Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #60070 > unrolled thread

Bug#888465: CPU usage reporting issues

Back to article view | Back to linux.debian.kernel

Contents

  Bug#888465: CPU usage reporting issues Ryan Thoryk <ryan@thoryk.com> - 2018-01-26 00:30 +0100
    Bug#888465: Additional example Ryan Thoryk <ryan@thoryk.com> - 2018-01-26 00:50 +0100
    Bug#888465: CPU usage reporting issues Hans van Kranenburg <hans@knorrie.org> - 2018-01-26 01:00 +0100
      Bug#888465: CPU usage reporting issues Ryan Thoryk <ryan@thoryk.com> - 2018-01-26 01:20 +0100
      Bug#888465: CPU usage reporting issues Ryan Thoryk <ryan@thoryk.com> - 2018-01-26 01:20 +0100

#60070 — Bug#888465: CPU usage reporting issues

[Multipart message — attachments visible in raw view] — view raw

Package: linux-image-4.9.0-5-amd64
Version: 4.9.65-3+deb9u2
Severity: normal

Hi,

I'm having an issue with CPU usage reporting, tested on kernels 4.9.0-3 
and 4.9.0-5.  The machines are running on Amazon EC2, which could be 
related.  With the "sar" utility, after some time, the system's "steal" 
value periodically is 100%, and the normal CPU user/system values, 
including idle, are always 0.  When running a cpu-intensive app and 
using the "top" utility, the user and system values are always 0, the 
"idle" field stays at 100%, and only the "wait" field increases.

The attached file shows the "sar" output around the time the issue 
started.  This has happened on 2 separate machines (started at different 
times on each), and a reboot appears to (temporarily) fix the issue.  
I'm wondering if anyone else has this issue, and if it could be 
something to do with the hypervisor.

-- 
Ryan Thoryk
ryan@thoryk.com
ryan@tliquest.net

[toc] | [next] | [standalone]

#60071 — Bug#888465: Additional example

[Multipart message — attachments visible in raw view] — view raw

Here's an additional example, showing the "vmstat" command output while 
running a cpu-intensive program.

-- 
Ryan Thoryk
ryan@thoryk.com
ryan@tliquest.net

[toc] | [prev] | [next] | [standalone]

#60072

Hi Ryan,

On 01/26/2018 12:20 AM, Ryan Thoryk wrote:
> Package: linux-image-4.9.0-5-amd64
> Version: 4.9.65-3+deb9u2
> Severity: normal
> 
> I'm having an issue with CPU usage reporting, tested on kernels 4.9.0-3
> and 4.9.0-5.  The machines are running on Amazon EC2, which could be
> related.  With the "sar" utility, after some time, the system's "steal"
> value periodically is 100%,

This means that your vcpus want to execute work but are not being
scheduled on a physical cpu core. Either the physical machine gets too
much work from all the virtual machines that are requesting cpu time, or
other things are going on, like your virtual machine getting paused
(e.g. when doing live migration there's a handover moment when it's
shortly paused and then resumed, this is also visible as a short 100%
steal spike).

> and the normal CPU user/system values,
> including idle, are always 0.  When running a cpu-intensive app and
> using the "top" utility, the user and system values are always 0, the
> "idle" field stays at 100%, and only the "wait" field increases.

Sounds a lot like this one:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871608

A patch to fix that cpu accounting breakage (picked from linux 4.15) was
included in 4.9.65-3. So only for the 4.9.0-3 (which actual version?)
you could be seeing that one happening.

> The attached file shows the "sar" output around the time the issue
> started.  This has happened on 2 separate machines (started at different
> times on each), and a reboot appears to (temporarily) fix the issue. 
> I'm wondering if anyone else has this issue, and if it could be
> something to do with the hypervisor.

Because of the mentioned steal time fix that was included in a version
in between the 2 versions you mention, my first suggestion would be to
see if the symptoms on the old and new kernel are exactly the same, or
if they are only similar but different.

Hans

[toc] | [prev] | [next] | [standalone]

#60073

(cleaned the post up for readability, sorry about that)

On 01/25/2018 05:40 PM, Hans van Kranenburg wrote:
> This means that your vcpus want to execute work but are not being
> scheduled on a physical cpu core. Either the physical machine gets too
> much work from all the virtual machines that are requesting cpu time, or
> other things are going on, like your virtual machine getting paused
> (e.g. when doing live migration there's a handover moment when it's
> shortly paused and then resumed, this is also visible as a short 100%
> steal spike).

After going over log files, it appears that the issue started when 
Amazon did a live migration of the VM, probably for the Meltdown patching.

> A patch to fix that cpu accounting breakage (picked from linux 4.15) was
> included in 4.9.65-3. So only for the 4.9.0-3 (which actual version?)
> you could be seeing that one happening.

The versions were both 4.9.30-2+deb9u2 and the latest, 4.9.65-3+deb9u2.  
So basically the kernel never recovered properly after being paused 
during a live migration.

> Because of the mentioned steal time fix that was included in a version
> in between the 2 versions you mention, my first suggestion would be to
> see if the symptoms on the old and new kernel are exactly the same, or
> if they are only similar but different.
>
> Hans

I already rebooted the system running the 4.9.65 kernel, and beforehand, 
the symptoms were the same.  The CPU usage stats went back to normal 
after the reboot.

-- 
Ryan Thoryk
ryan@thoryk.com
ryan@tliquest.net

[toc] | [prev] | [next] | [standalone]

#60074

On 01/25/2018 05:40 PM, Hans van Kranenburg wrote:
> Hi Ryan,
>
> On 01/26/2018 12:20 AM, Ryan Thoryk wrote:
>> Package: linux-image-4.9.0-5-amd64
>> Version: 4.9.65-3+deb9u2
>> Severity: normal
>>
>> I'm having an issue with CPU usage reporting, tested on kernels 4.9.0-3
>> and 4.9.0-5.  The machines are running on Amazon EC2, which could be
>> related.  With the "sar" utility, after some time, the system's "steal"
>> value periodically is 100%,
> This means that your vcpus want to execute work but are not being
> scheduled on a physical cpu core. Either the physical machine gets too
> much work from all the virtual machines that are requesting cpu time, or
> other things are going on, like your virtual machine getting paused
> (e.g. when doing live migration there's a handover moment when it's
> shortly paused and then resumed, this is also visible as a short 100%
> steal spike).
After going over log files, it appears that the issue started when 
Amazon did a live migration of the VM, probably for the Meltdown patching.
>> and the normal CPU user/system values,
>> including idle, are always 0.  When running a cpu-intensive app and
>> using the "top" utility, the user and system values are always 0, the
>> "idle" field stays at 100%, and only the "wait" field increases.
> Sounds a lot like this one:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871608
>
> A patch to fix that cpu accounting breakage (picked from linux 4.15) was
> included in 4.9.65-3. So only for the 4.9.0-3 (which actual version?)
> you could be seeing that one happening.
The versions were both 4.9.30-2+deb9u2 and the latest, 4.9.65-3+deb9u2.  
So basically the kernel never recovered properly after being paused 
during a live migration.
>> The attached file shows the "sar" output around the time the issue
>> started.  This has happened on 2 separate machines (started at different
>> times on each), and a reboot appears to (temporarily) fix the issue.
>> I'm wondering if anyone else has this issue, and if it could be
>> something to do with the hypervisor.
> Because of the mentioned steal time fix that was included in a version
> in between the 2 versions you mention, my first suggestion would be to
> see if the symptoms on the old and new kernel are exactly the same, or
> if they are only similar but different.
>
> Hans
I already rebooted the system running the 4.9.65 kernel, and beforehand, 
the symptoms were the same.  The CPU usage stats went back to normal 
after the reboot.

-- 
Ryan Thoryk
ryan@thoryk.com
ryan@tliquest.net

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web