Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #58626 > unrolled thread

Bug#871608: linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully

Back to article view | Back to linux.debian.kernel

Contents

  Bug#871608: linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully Michael Lass <bevan@bi-co.net> - 2017-08-09 23:10 +0200
    Processed: Re: Bug#871608: linux-image-4.9.0-3-amd64: Linux  kernel should handle decreasing cpu steal clock counter gracefully owner@bugs.debian.org (Debian Bug Tracking System) - 2017-09-25 02:10 +0200
    Bug#871608: linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully Ben Hutchings <ben@decadent.org.uk> - 2017-09-25 02:10 +0200
      Bug#871608: linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully Michael Laß <bevan@bi-co.net> - 2017-09-25 09:40 +0200
      Bug#871608: linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully Michael Laß <bevan@bi-co.net> - 2017-10-13 00:40 +0200
    Bug#871608: linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully Hans van Kranenburg <hans.van.kranenburg@mendix.com> - 2017-11-15 14:40 +0100
    Bug#871608: linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully Hans van Kranenburg <hans.van.kranenburg@mendix.com> - 2017-12-01 17:40 +0100

#58626 — Bug#871608: linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully

[Multipart message — attachments visible in raw view] — view raw

Package: src:linux
Version: 4.9.30-2+deb9u3
Severity: normal
Tags: patch

Dear Maintainer,

running Debian Stretch as a paravirtualized guest under Xen, the kernel
obtains its cpu steal time counter from the virtualization host. On some
hosts, occasionally a slight decrease in the cpu steal time is returned
which leads to an overflow of unsigned variables in the kernel and
subsequent errors in steal time accounting (such as backwards running
counters). This renders tools like "top" or "vmstat" broken in a way
that the cpu utilization cannot be determined anymore.

While this is likely a bug in the virtualization environment, the kernel
running as a guest should deal with this gracefully. I attached a patch
to this report which fixes the errors caused by this on the guest.
Kernel versions 4.7 and older, as well as 4.11 and newer should not be
affected by this issue.

Bug #785557 shows that behavior like this is caused by some broken KVM
hosts. I myself experience this on a Xen host which unfortunately I have
no more information about.

A more detailled description of the issue is part of the patch header,
as well as the following blog post:
https://0xstubs.org/debugging-a-flaky-cpu-steal-time-counter-on-a-paravirtualized-xen-guest/

I would appreciate inclusion of this patch in Debian as this issue may
affect other people running on buggy virtualization hosts and the patch
should not influence other systems.

Note that the system I report this from already runs a customly patched
kernel which may influence some of the information below.

-- Package-specific info:
** Version:
Linux version 4.9.0-3-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.30-2+deb9u3+lass1 (2017-08-08)

** Command line:
root=/dev/xvda ro 

** Not tainted

** Kernel log:
Unable to read kernel log; any relevant messages should be attached

** Model information

** Loaded modules:
ipt_REJECT
nf_reject_ipv4
binfmt_misc
xt_multiport
iptable_filter
intel_rapl
sb_edac
edac_core
evdev
kvm_intel
kvm
irqbypass
crct10dif_pclmul
crc32_pclmul
ghash_clmulni_intel
pcspkr
intel_rapl_perf
ip_tables
x_tables
autofs4
ext4
crc16
jbd2
fscrypto
ecb
mbcache
btrfs
crc32c_generic
xor
raid6_pq
crc32c_intel
xen_netfront
xen_blkfront
aesni_intel
aes_x86_64
glue_helper
lrw
gf128mul
ablk_helper
cryptd

** PCI devices:
not available

** USB devices:
not available


-- System Information:
Debian Release: 9.1
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-3-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages linux-image-4.9.0-3-amd64 depends on:
ii  initramfs-tools [linux-initramfs-tool]  0.130
ii  kmod                                    23-2
ii  linux-base                              4.5

Versions of packages linux-image-4.9.0-3-amd64 recommends:
ii  firmware-linux-free  3.4
ii  irqbalance           1.1.0-2.3

Versions of packages linux-image-4.9.0-3-amd64 suggests:
pn  debian-kernel-handbook               <none>
pn  grub-pc | grub-efi-amd64 | extlinux  <none>
pn  linux-doc-4.9                        <none>

Versions of packages linux-image-4.9.0-3-amd64 is related to:
pn  firmware-amd-graphics     <none>
pn  firmware-atheros          <none>
pn  firmware-bnx2             <none>
pn  firmware-bnx2x            <none>
pn  firmware-brcm80211        <none>
pn  firmware-cavium           <none>
pn  firmware-intel-sound      <none>
pn  firmware-intelwimax       <none>
pn  firmware-ipw2x00          <none>
pn  firmware-ivtv             <none>
pn  firmware-iwlwifi          <none>
pn  firmware-libertas         <none>
pn  firmware-linux-nonfree    <none>
pn  firmware-misc-nonfree     <none>
pn  firmware-myricom          <none>
pn  firmware-netxen           <none>
pn  firmware-qlogic           <none>
pn  firmware-realtek          <none>
pn  firmware-samsung          <none>
pn  firmware-siano            <none>
pn  firmware-ti-connectivity  <none>
pn  xen-hypervisor            <none>

-- no debconf information

[toc] | [next] | [standalone]

#59006 — Processed: Re: Bug#871608: linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully

Processing control commands:

> severity -1 important
Bug #871608 [src:linux] linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully
Severity set to 'important' from 'normal'
> tag -1 moreinfo
Bug #871608 [src:linux] linux-image-4.9.0-3-amd64: Linux kernel should handle decreasing cpu steal clock counter gracefully
Added tag(s) moreinfo.

-- 
871608: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871608
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#59007

[Multipart message — attachments visible in raw view] — view raw

Control: severity -1 important
Control: tag -1 moreinfo

On Wed, 2017-08-09 at 22:46 +0200, Michael Lass wrote:
> Package: src:linux
> Version: 4.9.30-2+deb9u3
> Severity: normal
> Tags: patch
> 
> Dear Maintainer,
> 
> running Debian Stretch as a paravirtualized guest under Xen, the kernel
> obtains its cpu steal time counter from the virtualization host. On some
> hosts, occasionally a slight decrease in the cpu steal time is returned
> which leads to an overflow of unsigned variables in the kernel and
> subsequent errors in steal time accounting (such as backwards running
> counters). This renders tools like "top" or "vmstat" broken in a way
> that the cpu utilization cannot be determined anymore.
> 
> While this is likely a bug in the virtualization environment, the kernel
> running as a guest should deal with this gracefully. I attached a patch
> to this report which fixes the errors caused by this on the guest.
> Kernel versions 4.7 and older, as well as 4.11 and newer should not be
> affected by this issue.
[...]

I agree that the kernel ought to work around this, but I'm hesitant to
add a fix that doesn't look like any upstream change.  Why and how do
you think this was fixed in 4.11?

Ben.

-- 
Ben Hutchings
If the facts do not conform to your theory, they must be disposed of.

[toc] | [prev] | [next] | [standalone]

#59018

> Am 25.09.2017 um 02:07 schrieb Ben Hutchings <ben@decadent.org.uk>:
> 
> I agree that the kernel ought to work around this, but I'm hesitant to
> add a fix that doesn't look like any upstream change.  Why and how do
> you think this was fixed in 4.11?

Indeed this patch is not included upstream and won’t be as it does not apply to the current development branch. See the following brief discussion on this on Linux stable: http://www.spinics.net/lists/stable/msg186915.html

The main issue here is the conversion between nsecs and cputime and how this behaves when the calculated difference in steal time overflows, i.e. decreases. In this case cpustat does not overflow and therefore decrease as well but instead increases by a large number. In Linux 4.11 the conversion between nsecs and cputime is entirely gone, which should lead to a backwards running cpustat counter without further issues (I have not actually tested this):

2b1f967d80e8e5d7361f0e1654c842869570f573  sched/cputime: Complete nsec conversion of tick based accounting

This change seems to be only the last commit of a larger change though and I’m not sure if this larger change is suitable for backporting.

I agree that it would be nice to have a patch which has undergone a good review process. Unfortunately I cannot provide that. I also cannot estimate how many people are affected by this issue considering that it did not attract attention over the last year.

Michael

[toc] | [prev] | [next] | [standalone]

#59169

There is currently a discussion on LKML and xen-devel about this issue:
https://lkml.org/lkml/2017/10/10/182

Maybe this will result in some backportable fix.

[toc] | [prev] | [next] | [standalone]

#59398

Hi,

I ran into the same issue with 4.9.51-1 in the guest. My dom0 in this
case is still Jessie with its xen 4.4 version. Users (rightfully) worry
about what's suddenly wrong with their virtual machine, since the steal
values mess up the cpu graphs.

I see that all the discussions linked above have gone silent quickly
without a solution.

A post to the Xen mailing list on Aug 31th did not get any answer yet:
https://lists.xen.org/archives/html/xen-users/2017-08/msg00092.html

I see that the issue got fixed by replacing the code with a new
implementation of the same functionality. I guess this is a scenario
that sometimes happens, not having a ready to go fix available for the
previous LTS kernel.

So, what would be the best step forward here? Should we poke the Xen
people a bit more to find out how to approach this, or get an opinion on
the best and smallest patch to go with?

I'm not an expert in the cpu time accounting area, but I can help
testing etc...

Thanks,

--
Hans van Kranenburg

[toc] | [prev] | [next] | [standalone]

#59526

On 11/16/2017 02:19 PM, Hans van Kranenburg wrote:
> 
> Latest work on this:
> 
> https://patchwork.kernel.org/patch/10035835/
> 
> "Applied to for-linus-4.15."

Ok, I just built a 4.9.65 kernel with this patch on top and the config
from debian (config-4.9.0-4-amd64). It applies without complaints:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5e25f5db6abb96ca8ee2aaedcb863daa6dfcc07a

With the 4.9.51-1 from Stretch I can reliably reproduce the broken CPU
counters after doing live migration with Xen once or maybe twice.

The 4.9.65 + steal time patch survived throwing it around >20 times now,
and there's no sign of any weird behaviour any more.

Counters in /proc/stat keep showing values that still make sense,
instead of suddenly jumping to values like 1174983480817 or 1753913027832...

How do we proceed from here?

-- 
Hans van Kranenburg

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web