Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days

From	George Shuklin <george.shuklin@gmail.com>
Newsgroups	linux.debian.user
Subject	Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days
Date	2020-12-25 11:40 +0100
Message-ID	<BpSFQ-4wq-1@gated-at.bofh.it> (permalink)
References	<BpDQt-3vh-1@gated-at.bofh.it>
Organization	linux.* mail to news gateway

Show all headers | View raw

On 12/24/20 8:34 PM, buz.hrach@seznam.cz wrote:
> Hi Debian people ;-),
>
> After having some issues with Fedora last year I decided to reinstall all my servers to Debian 10. I'm supper happy with Debian except one repeating issue I have with QEMU-KVM hosts that is very difficult to reproduce so I would like to discuss it first before I open a new bug. Could you please discuss it with me? ;-)
>
> I noticed that when I run VMs for a long period of time (a couple of days) one or multiple VMs quite often stuck. It is not possible to connect the stuck VMs using virt-manager and their serial consoles don't respond.
>
> It is not possible to shut them down ("virsh shutdown vm"). Sometimes the stuck VMs can be powered down ("virsh destroy vm") but in most cases "virsh destroy" doesn't work. In that case the only thing to do is to shut down rest of running VMs (that do respond) and reboot the host.

 From my past experience, we had had very similar issue on old Ubuntu, 
which had chance to show hung tasks and extremely low IO performance 
after snapshot been made.

Turns out, It was mega-obscure bug around race condition somewhere in 
the kernel, because fallocated file for filesystem is not the same as 
dully 'dd-ed' (we've used raw images). There is about 70% chance your 
case in not that bug, but to confirm/reject it, try to run VMs with raw 
images (no qcow2) which was dd if=/dev/zero of=image.img before been 
seeded with OS. If they stop showing this type of behavior, you have 
direction to investigate. If problem persists, then it's something 
different.

Back to linux.debian.user | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

QEMU-KVM VMs sometime freeze when I run them for a couple of days <buz.hrach@seznam.cz> - 2020-12-24 19:50 +0100
  Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days Dan Ritter <dsr@randomstring.org> - 2020-12-24 20:30 +0100
    Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days <buz.hrach@seznam.cz> - 2020-12-25 10:10 +0100
    Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days <buz.hrach@seznam.cz> - 2020-12-27 11:40 +0100
      Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days Nicholas Geovanis <nickgeovanis@gmail.com> - 2020-12-27 16:50 +0100
        Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days <buz.hrach@seznam.cz> - 2021-01-06 14:40 +0100
          Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days Bob McGowan <bob@bnlmcgowan.net> - 2021-01-07 02:40 +0100
  Re: QEMU-KVM VMs sometime freeze when I run them for a couple of days George Shuklin <george.shuklin@gmail.com> - 2020-12-25 11:40 +0100

csiph-web