Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #229970 > unrolled thread

QEMU-KVM VMs sometime freeze when I run them for a couple of days

Back to article view | Back to linux.debian.user

Contents

  QEMU-KVM VMs sometime freeze when I run them for a couple of days <buz.hrach@seznam.cz> - 2020-12-24 19:50 +0100
    Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days Dan Ritter <dsr@randomstring.org> - 2020-12-24 20:30 +0100
      Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days <buz.hrach@seznam.cz> - 2020-12-25 10:10 +0100
      Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days <buz.hrach@seznam.cz> - 2020-12-27 11:40 +0100
        Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days Nicholas Geovanis <nickgeovanis@gmail.com> - 2020-12-27 16:50 +0100
          Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days <buz.hrach@seznam.cz> - 2021-01-06 14:40 +0100
            Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days Bob McGowan <bob@bnlmcgowan.net> - 2021-01-07 02:40 +0100
    Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days George Shuklin <george.shuklin@gmail.com> - 2020-12-25 11:40 +0100

#229970 — QEMU-KVM VMs sometime freeze when I run them for a couple of days

Hi Debian people ;-),

After having some issues with Fedora last year I decided to reinstall all my servers to Debian 10. I'm supper happy with Debian except one repeating issue I have with QEMU-KVM hosts that is very difficult to reproduce so I would like to discuss it first before I open a new bug. Could you please discuss it with me? ;-)

I noticed that when I run VMs for a long period of time (a couple of days) one or multiple VMs quite often stuck. It is not possible to connect the stuck VMs using virt-manager and their serial consoles don't respond.

It is not possible to shut them down ("virsh shutdown vm"). Sometimes the stuck VMs can be powered down ("virsh destroy vm") but in most cases "virsh destroy" doesn't work. In that case the only thing to do is to shut down rest of running VMs (that do respond) and reboot the host.

This is a kernel message I get in console that tells me that one or multiple VMs are stuck:
~~~
[686811.010084] INFO: task CPU 0/KVM:12193 blocked for more than 120 seconds.
[686811.017040]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686811.024777] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[686811.033012] INFO: task CPU 1/KVM:12194 blocked for more than 120 seconds.
[686811.039921]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686811.047606] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[686811.055803] INFO: task worker:5952 blocked for more than 120 seconds.
[686811.062355]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686811.070048] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[686811.078059] INFO: task worker:7667 blocked for more than 120 seconds.
[686811.084618]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686811.092306] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[686811.100296] INFO: task worker:4903 blocked for more than 120 seconds.
[686811.106849]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686811.114530] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[686811.122512] INFO: task worker:4905 blocked for more than 120 seconds.
[686811.129068]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686811.136765] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[686811.144771] INFO: task worker:4920 blocked for more than 120 seconds.
[686811.151328]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686811.159009] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[686811.167009] INFO: task worker:7328 blocked for more than 120 seconds.
[686811.173576]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686811.181256] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[686931.842071] INFO: task CPU 0/KVM:12193 blocked for more than 120 seconds.
[686931.849028]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686931.856764] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[686931.864997] INFO: task CPU 1/KVM:12194 blocked for more than 120 seconds.
[686931.871908]       Tainted: P           OE     4.19.0-12-amd64 #1 Debian 4.19.152-1
[686931.879586] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
~~~

When I reboot/shutdown the host the reboot/shutdown takes approx. 30min.

This is how it looks like during the reboot / shutdown:
~~~
...
[1051413.325604] libvirt-guests.sh[10107]: error: Failed to shutdown domain de763fd3-043c-4f6f-b7f9-e134907b9f54
[1051413.325964] libvirt-guests.sh[10107]: error: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainShutdown agent=remoteDispatchDomainShutdown)
...
[1053290.120617] reboot: Power down
~~~

Or systemd waits for LUKS encrypted block storage which doesn't make sense since LUKS is used only on SSD with the host OS and VMs run from a different SSDs.
~~~
[  OK  ] Stopped target Local File Systems (Pre).
[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create System Users.
         Stopping Monitoring of LVM_meventd or progress polling...
[  OK  ] Stopped Monitoring of LVM2_ dmeventd or progress polling.
[FAILED] Failed deactivating swap swp.
         Stopping Cryptography Setup for swapluks1...
[  OK  ] Stopped Cryptography Setup for swapluks1.
         Stopping Load/Save Random Seed...
[  OK  ] Removed slice system-systemd\x2dcryptsetup.slice.
[  OK  ] Stopped Load/Save Random Seed.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Reached target Shutdown.
[   ***] (1 of 4) A stop job is running for /dev/dm-1 (18min 6s / no limit)
~~~
and then
~~~
[  OK  ] Stopped Load/Save Random Seed.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Reached target Shutdown.
[ TIME ] Timed out starting Reboot.
[  !!  ] Forcibly rebooting: job timed out
[415391.907610] watchdog: watchdog0: watchdog did not stop!
[415578.747792] reboot: Restarting system
~~~

As I mentioned it is very difficult to reproduce it since it takes days to get into that situation. VMs that are more likely to get stuck are VMs that:

a) have larger virtual disks
b) more intensive storage use (use more IOPs)
c) have more vCPUs

The problem is that VMs with larger disks usually use more IOPs and also have more vCPUs so it is difficult to say what exactly is the issue. Based on my testing I thing that less vCPUs makes it less likely to get stuck but it's difficult to say...

The only thing I'm confident is that the problem is not HW related - it happened both on my SuperMicro with XEON E5 v2 and on other hardware with Intel i7 7th gen.

>From the configuration/software perspective all hosts and all VMs have up-to-date Debian 10. The only less usual piece of software is ZFS 0.8 from buster-backports. All VMs use ZFS volumes (similar to RAW disks).

I super confident using ZFS since I use it since version 0.7 absolutely everywhere including this Debian 10 HP laptop that I use to write this text.

When some of the VMs stuck/freeze the other VMs run just fine and ZFS is functional and fast and I don't see any errors (in "zpool status" after I run "zpool scrub") so I don't think it's ZFS but I want to mention it since it's the only unusual piece of SW/configuration I use.

Btw. this has never happened on my laptop that has same configuration as the server (+Desktop Env.) but I reboot it multiple time a week so that might be an answer...

Please let me know your thoughts. Thank you ;-).

Merry Christmas and Happy New Year ;-).

Kind regards,

Robert Hrach

[toc] | [next] | [standalone]

#229972

buz.hrach@seznam.cz wrote: 
> Hi Debian people ;-),
> 
> After having some issues with Fedora last year I decided to reinstall all my servers to Debian 10. I'm supper happy with Debian except one repeating issue I have with QEMU-KVM hosts that is very difficult to reproduce so I would like to discuss it first before I open a new bug. Could you please discuss it with me? ;-)
> 
> I noticed that when I run VMs for a long period of time (a couple of days) one or multiple VMs quite often stuck. It is not possible to connect the stuck VMs using virt-manager and their serial consoles don't respond.


First question: when they are just a few minutes old, does the
serial console work?


> It is not possible to shut them down ("virsh shutdown vm"). Sometimes the stuck VMs can be powered down ("virsh destroy vm") but in most cases "virsh destroy" doesn't work. In that case the only thing to do is to shut down rest of running VMs (that do respond) and reboot the host.

Second question: when the VMs are a few minutes old, does virsh
shutdown work?

> When I reboot/shutdown the host the reboot/shutdown takes approx. 30min.
> 
> This is how it looks like during the reboot / shutdown:
> ~~~
> [   ***] (1 of 4) A stop job is running for /dev/dm-1 (18min 6s / no limit)

You probably want to change that to 1 minute or so.

> As I mentioned it is very difficult to reproduce it since it takes days to get into that situation. VMs that are more likely to get stuck are VMs that:
> 
> a) have larger virtual disks
> b) more intensive storage use (use more IOPs)
> c) have more vCPUs
> 
> The problem is that VMs with larger disks usually use more IOPs and also have more vCPUs so it is difficult to say what exactly is the issue. Based on my testing I thing that less vCPUs makes it less likely to get stuck but it's difficult to say...
> 
> The only thing I'm confident is that the problem is not HW related - it happened both on my SuperMicro with XEON E5 v2 and on other hardware with Intel i7 7th gen.

Are the VMs set up to match the local hardware definition or be
fully emulated?

And, especially: if they are not using virtio for disk and
network address, try that ASAP.

> Btw. this has never happened on my laptop that has same configuration as the server (+Desktop Env.) but I reboot it multiple time a week so that might be an answer...

Not so much an answer as an explanation why you haven't seen it,
but, sure, that's plausible.

-dsr-

[toc] | [prev] | [next] | [standalone]

#229977

From: Dan Ritter <dsr@randomstring.org>

Hi Dan, Thank you for answering ;-)

> First question: when they are just a few minutes old, does the
> serial console work?

Yes, 100%. I always configure serial console for all VMs both on QEMU (libvirt) and OS side.

> Second question: when the VMs are a few minutes old, does virsh
> shutdown work?

Yes, 100%. In most cases everything works just fine. Until I keep those VMs
running for days... And even then some virtualization hosts are just fine
and one or two has those frozen VMs.

> You probably want to change that to 1 minute or so.

Good point.

> Are the VMs set up to match the local hardware definition or be
> fully emulated?

Matching HW, all virtio, CPU pass-through for everything (=> maximum performance).

> And, especially: if they are not using virtio for disk and
> network address, try that ASAP.

They do.

==================
  <cpu mode='host-passthrough' check='none'/>
...
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>
      <source dev='/dev/..../>
      <target dev='sdX' bus='scsi'/>
    </disk>
...
    <controller type='scsi' index='0' model='virtio-scsi'/>
...
    <interface type='bridge'>
      <mac address='...'/>
      <source bridge='...'/>
      <vlan>
        <tag id='...'/>
      </vlan>
      <virtualport type='openvswitch'/>
      <target dev='...'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <link state='up'/>
    </interface>
===================

Kind regards,
Robert

[toc] | [prev] | [next] | [standalone]

#230022

From: Dan Ritter <dsr@randomstring.org>

> You probably want to change that to 1 minute or so.

Btw. could you please tell me how to do that?

I know how to modify (=> override) systemd unit files but I have no idea how to change default timeout for mounted block devices that I didn't manually create in /etc/systemd/system/...

Thank you.

Robert

[toc] | [prev] | [next] | [standalone]

#230028

[Multipart message — attachments visible in raw view] — view raw

George Shuklin's comment may have intended this question too: Are backups
running somewhere when the VMs "randomly" hang? Not necessarily on the VM
that hangs, but somewhere touching a related filesystem, disk or network
device?

On Sun, Dec 27, 2020, 4:33 AM <buz.hrach@seznam.cz> wrote:

> From: Dan Ritter <dsr@randomstring.org>
>
> > You probably want to change that to 1 minute or so.
>
> Btw. could you please tell me how to do that?
>
> I know how to modify (=> override) systemd unit files but I have no idea
> how to change default timeout for mounted block devices that I didn't
> manually create in /etc/systemd/system/...
>
> Thank you.
>
> Robert
>
>

[toc] | [prev] | [next] | [standalone]

#230376

From: Nicholas Geovanis <nickgeovanis@gmail.com>

> George Shuklin's comment may have intended this question too: Are backups running
> somewhere when the VMs "randomly" hang? Not necessarily on the VM that hangs,
> but somewhere touching a related filesystem, disk or network device?

Hi Nicholas,

Thank you for asking. No backups on the host or inside the guest. And nothing else IO intensive on the host. They're IO intensive tasks in some guests but as I mentioned it's difficult to say if those tasks trigger the freezing.

[toc] | [prev] | [next] | [standalone]

#230398

On 1/6/21 5:33 AM, buz.hrach@seznam.cz wrote:
> From: Nicholas Geovanis <nickgeovanis@gmail.com>
>
>> George Shuklin's comment may have intended this question too: Are backups running
>> somewhere when the VMs "randomly" hang? Not necessarily on the VM that hangs,
>> but somewhere touching a related filesystem, disk or network device?
> Hi Nicholas,
>
> Thank you for asking. No backups on the host or inside the guest. And nothing else IO intensive on the host. They're IO intensive tasks in some guests but as I mentioned it's difficult to say if those tasks trigger the freezing.
>
I've just been working on getting Qemu for powerpc to install NetBSD 9.1 
and had bunches of problems with the VM hanging during the install 
process.  This was using the current default Debian package, 3.1.0.

The Qemu website has source for version 5.2.0, which I downloaded and 
built.  It does not hang.

Perhaps upgrading is the answer?

Bob

[toc] | [prev] | [next] | [standalone]

#229978

On 12/24/20 8:34 PM, buz.hrach@seznam.cz wrote:
> Hi Debian people ;-),
>
> After having some issues with Fedora last year I decided to reinstall all my servers to Debian 10. I'm supper happy with Debian except one repeating issue I have with QEMU-KVM hosts that is very difficult to reproduce so I would like to discuss it first before I open a new bug. Could you please discuss it with me? ;-)
>
> I noticed that when I run VMs for a long period of time (a couple of days) one or multiple VMs quite often stuck. It is not possible to connect the stuck VMs using virt-manager and their serial consoles don't respond.
>
> It is not possible to shut them down ("virsh shutdown vm"). Sometimes the stuck VMs can be powered down ("virsh destroy vm") but in most cases "virsh destroy" doesn't work. In that case the only thing to do is to shut down rest of running VMs (that do respond) and reboot the host.

 From my past experience, we had had very similar issue on old Ubuntu, 
which had chance to show hung tasks and extremely low IO performance 
after snapshot been made.

Turns out, It was mega-obscure bug around race condition somewhere in 
the kernel, because fallocated file for filesystem is not the same as 
dully 'dd-ed' (we've used raw images). There is about 70% chance your 
case in not that bug, but to confirm/reject it, try to run VMs with raw 
images (no qcow2) which was dd if=/dev/zero of=image.img before been 
seeded with OS. If they stop showing this type of behavior, you have 
direction to investigate. If problem persists, then it's something 
different.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web