Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #67179 > unrolled thread

5.6 kernel crashes host when using VFIO on KVM guests

Back to article view | Back to linux.debian.kernel

Contents

  5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-05-25 22:40 +0200
    Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Salvatore Bonaccorso <carnil@debian.org> - 2020-05-27 21:40 +0200
      Processed: Re: 5.6 kernel crashes host when using VFIO on KVM guests "Debian Bug Tracking System" <owner@bugs.debian.org> - 2020-05-27 21:40 +0200
      Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-05-27 22:50 +0200
        Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-06-12 00:30 +0200
          Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-06-14 00:50 +0200
            Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-06-28 02:00 +0200
              Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-06-28 02:10 +0200
                Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Salvatore Bonaccorso <carnil@debian.org> - 2020-06-28 10:50 +0200
                  Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <simon@the-jedi.co.uk> - 2020-06-28 12:10 +0200
                    Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-07-03 23:50 +0200
                      Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-07-19 00:20 +0200
                        Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-07-19 01:40 +0200
                          Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-07-19 01:50 +0200
                            Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests Simon John <debian@the-jedi.co.uk> - 2020-07-19 18:00 +0200
      Bug#961676: marked as done (5.6 kernel crashes host when using  VFIO on KVM guests) "Debian Bug Tracking System" <owner@bugs.debian.org> - 2020-07-19 18:20 +0200

#67179 — 5.6 kernel crashes host when using VFIO on KVM guests

Sorry to email directly but I've tried reportbug but it doesn't seem to 
work with the kernel packages.

The 5.6.0-1 kernel in Sid when used in conjunction with VFIO (PCI 
passthrough) guests in Qemu-KVM causes KVM to crash with no useful logs 
- the guest partially starts. Eventually the host slows down and needs 
to be powered off. Doesn't affect non-VFIO guests using spice/headless, 
only when passing through a PCIe graphics card or USB keyboard/mouse.

5.6.0-2 that just landed makes this infinitely worse - as soon as you 
start a VFIO guest the host hard crashes.

5.3/5.4/5.5 kernels do not have this problem with the same version of 
Qemu and reverting from qemu 5.0 to 4.2 doesn't help, so it must be the 
5.6 kernel causing the issue.

Not sure what logs or any useful information I can supply you, there's 
some interesting comments on this reddit thread suggesting a couple of 
bugs that have fixes upstream:

https://www.reddit.com/r/VFIO/comments/glfgqs/qemu_5_vfio_no_longer_works_on_debian/

I'm using macos catalina and win10 guests to reproduce the issue, and 
using Intel not AMD chips.

Cheers.

-- 
Simon John

[toc] | [next] | [standalone]

#67193 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

Source: linux
Version: 5.6.7-1
Control: submitter -1 debian@the-jedi.co.uk

Forwarding this report as bug in the BTS.

On Mon, May 25, 2020 at 09:03:43PM +0100, Simon John wrote:
> Sorry to email directly but I've tried reportbug but it doesn't seem to work
> with the kernel packages.

What exactly did not work?

> The 5.6.0-1 kernel in Sid when used in conjunction with VFIO (PCI
> passthrough) guests in Qemu-KVM causes KVM to crash with no useful logs -
> the guest partially starts. Eventually the host slows down and needs to be
> powered off. Doesn't affect non-VFIO guests using spice/headless, only when
> passing through a PCIe graphics card or USB keyboard/mouse.
> 
> 5.6.0-2 that just landed makes this infinitely worse - as soon as you start
> a VFIO guest the host hard crashes.
> 
> 5.3/5.4/5.5 kernels do not have this problem with the same version of Qemu
> and reverting from qemu 5.0 to 4.2 doesn't help, so it must be the 5.6
> kernel causing the issue.
> 
> Not sure what logs or any useful information I can supply you, there's some
> interesting comments on this reddit thread suggesting a couple of bugs that
> have fixes upstream:
> 
> https://www.reddit.com/r/VFIO/comments/glfgqs/qemu_5_vfio_no_longer_works_on_debian/
> 
> I'm using macos catalina and win10 guests to reproduce the issue, and using
> Intel not AMD chips.
> 
> Cheers.
> 
> -- 
> Simon John

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#67194 — Processed: Re: 5.6 kernel crashes host when using VFIO on KVM guests

Processing control commands:

> submitter -1 debian@the-jedi.co.uk
Bug #961676 [src:linux] 5.6 kernel crashes host when using VFIO on KVM guests
Changed Bug submitter to 'debian@the-jedi.co.uk' from 'Salvatore Bonaccorso <carnil@debian.org>'.

-- 
961676: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961676
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#67195 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

On Wed, 27 May 2020 21:31:46 +0200 Salvatore Bonaccorso 
<carnil@debian.org> wrote:
> Source: linux
> Version: 5.6.7-1
> Control: submitter -1 debian@the-jedi.co.uk
> 
> Forwarding this report as bug in the BTS.
> 
> On Mon, May 25, 2020 at 09:03:43PM +0100, Simon John wrote:
> > Sorry to email directly but I've tried reportbug but it doesn't seem to work
> > with the kernel packages.
> 
> What exactly did not work?

I tried to report a bug in 5.5 kernel regarding HDMI audio not working 
after a while until you kill pulseaudio but reportbug did not create a 
valid report i guess as it never got created - i tried using the email 
template too.

same problem when trying to report this bug.

Anyway, at least this bug has been created now, hopefully will get 
looked into.

> > The 5.6.0-1 kernel in Sid when used in conjunction with VFIO (PCI
> > passthrough) guests in Qemu-KVM causes KVM to crash with no useful logs -
> > the guest partially starts. Eventually the host slows down and needs to be
> > powered off. Doesn't affect non-VFIO guests using spice/headless, only when
> > passing through a PCIe graphics card or USB keyboard/mouse.
> > 
> > 5.6.0-2 that just landed makes this infinitely worse - as soon as you start
> > a VFIO guest the host hard crashes.
> > 
> > 5.3/5.4/5.5 kernels do not have this problem with the same version of Qemu
> > and reverting from qemu 5.0 to 4.2 doesn't help, so it must be the 5.6
> > kernel causing the issue.
> > 
> > Not sure what logs or any useful information I can supply you, there's some
> > interesting comments on this reddit thread suggesting a couple of bugs that
> > have fixes upstream:
> > 
> > https://www.reddit.com/r/VFIO/comments/glfgqs/qemu_5_vfio_no_longer_works_on_debian/
> > 
> > I'm using macos catalina and win10 guests to reproduce the issue, and using
> > Intel not AMD chips.
> > 
> > Cheers.
> > 
> > -- 
> > Simon John
> 
> Regards,
> Salvatore
> 
> 

Best regards.

-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67290 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

The new linux-image-5.6.0-2-amd64 (5.6.14-2) from today is no better, 
only change is its signed I guess?

Also I noticed the 5.3/5.4/5.5 kernels have all been removed from my 
system and the servers so I have no way to boot back into a kernel that 
doesn't break VFIO.

Has this issue been looked at or is 5.7 due soon?

Regards.

-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67304 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

I tried to get some sort of useful logging today, best I could do was 
running strace.

So I noticed that virsh exited 0 but didn't instantly crash, it took 2-3 
seconds more.

Photo of screen with strace output here:

	https://i.imgur.com/kOSEXXA.jpg

Managed to install 5.5.0-2 from snapshot.debian.org which works fine.

Regards.

-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67411 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

Bug still exists in today's 5.7 kernel, however it took long enough to 
trigger that it managed to write some logs:

https://gist.github.com/sej7278/766043a69c76308f84cfa14b3f3a924f

"BUG: soft lockup - CPU#0 stuck for 22s!" seems to go back to 2013 or 
earlier e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1038929

kvm-pit seemed to be eating a lot of cpu, qemu went defunct, macos guest 
got a little further into its boot sequence.

Can we upgrade this bug so it gets some visibility, that's two major 
kernel versions its not been fixed in now, let alone commented on.

-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67412 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

This looks a likely culprit:

https://bugzilla.kernel.org/show_bug.cgi?id=207489

-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67413 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

Hi Simon,

On Sun, Jun 28, 2020 at 01:01:01AM +0100, Simon John wrote:
> This looks a likely culprit:
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=207489

The issue you are seeing now seems different, though afaics.

The fixing commit from the above reference, 8be8f932e3db ("kvm:
ioapic: Restrict lazy EOI update to edge-triggered interrupts") was
applied in v5.7-rc3, which was as well backported to v5.6.13.

Now you kernel is tained, can you check if you see the issue as well
when not loading the modules which taint the kernel?

Next, if it still does show up, does it show up as well in current
mainline?

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#67414 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

On Sun, 28 Jun 2020 10:41:41 +0200 Salvatore Bonaccorso 
<carnil@debian.org> wrote:
> Hi Simon,

Hi Salvatore, thanks for looking into this.

> On Sun, Jun 28, 2020 at 01:01:01AM +0100, Simon John wrote:
> > This looks a likely culprit:
> > 
> > https://bugzilla.kernel.org/show_bug.cgi?id=207489
> 
> The issue you are seeing now seems different, though afaics.
> 
> The fixing commit from the above reference, 8be8f932e3db ("kvm:
> ioapic: Restrict lazy EOI update to edge-triggered interrupts") was
> applied in v5.7-rc3, which was as well backported to v5.6.13.

Yes, I was going to patch the kernel but noticed its already got that 
patch. I also tried passing through only one pci device and not the usb 
device but it didn't help, so probably isn't that issue.

> Now you kernel is tained, can you check if you see the issue as well
> when not loading the modules which taint the kernel?

I removed virtualbox which was tainting the kernel, confirmed cat 
/proc/sys/kernel/tainted returned 0 after a reboot.

I still managed to trigger the issue though, which then tainted the 
kernel itself, output from kernel-chktaint:

Kernel is "tainted" for the following reasons:
  * kernel died recently, i.e. there was an OOPS or BUG (#7)
  * kernel issued warning (#9)
  * soft lockup occurred (#14)
For a more detailed explanation of the various taint flags see
  Documentation/admin-guide/tainted-kernels.rst in the the Linux kernel 
sources
  or https://kernel.org/doc/html/latest/admin-guide/tainted-kernels.html
Raw taint value as int/string: 17024/'G      D W    L   '

The kvm-pit process was using 100% of a cpu core, i've never even 
noticed that process before, looks like the culprit now?:

watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kvm-pit/2888:2913]

CPU: 7 PID: 2913 Comm: kvm-pit/2888 Tainted: G      D W 
5.7.0-1-amd64 #1 Debian 5.7.6-1

I've added a strace and dmesg to the gist here:

https://gist.github.com/sej7278/766043a69c76308f84cfa14b3f3a924f

Any other diagnostics I can run?

> Next, if it still does show up, does it show up as well in current
> mainline?

Not sure what you mean there, do you mean upstream kernel.org kernel - 
i'm not sure how i'd run that.

Regards.

-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67457 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

Done some more digging.

I can boot my macos vm using QXL instead of a passthrough GPU, even if i 
passthrough the USB device (not pci controller). So its definitely only 
passing through PCI devices like graphics cards that breaks things.

So this works in qemu terms:

     -usb -device usb-host,hostbus=1,hostaddr=3 \

This breaks things:

     -device vfio-pci,host=02:00.0,multifunction=on \
     -device vfio-pci,host=02:00.1

Oddly enough an Ubuntu 20.04 vm works fine with the gpu and usb passed 
through.

Added the qemu shell scripts to the gist:

https://gist.github.com/sej7278/766043a69c76308f84cfa14b3f3a924f

Regards.

-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67561 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

Where would I look for some follow-up on this - upstream kernel.org?


-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67562 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

I tried mainline 5.8.0-rc5 and I couldn't even get into gnome shell 
after the gdm3 password prompt, just a black screen!

I tried running virsh start which worked but my monitor never got a 
signal. It also didn't crash the host or qemu though, although virsh 
destroy finished instantly, so i assume the guest didn't even start.

regards.

-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67563 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

Ah no, turns out I had disabled the gpu passthrough, when i re-enabled 
it i just got a kernel panic from 5.8.0-rc5


On 19/07/2020 00:34, Simon John wrote:
> I tried mainline 5.8.0-rc5 and I couldn't even get into gnome shell
> after the gdm3 password prompt, just a black screen!
> 
> I tried running virsh start which worked but my monitor never got a
> signal. It also didn't crash the host or qemu though, although virsh
> destroy finished instantly, so i assume the guest didn't even start.
> 
> regards.
> 


-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67565 — Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests

I think we can close this as #notabug.

I did a fresh install and its working on 5.7

It worked 100% fine on qemu, but when I imported it into virt-manager it 
crashed, until i removed all of the clock bits, so i think maybe there's 
a bug in the clock/pit parts of libvirt that qemu doesn't seem to use.

Regards.

-- 
Simon John

[toc] | [prev] | [next] | [standalone]

#67566 — Bug#961676: marked as done (5.6 kernel crashes host when using VFIO on KVM guests)

[Multipart message — attachments visible in raw view] — view raw

Your message dated Sun, 19 Jul 2020 18:02:00 +0200
with message-id <20200719160200.GB234193@photino.stro.at>
and subject line Re: Bug#961676: 5.6 kernel crashes host when using VFIO on KVM guests
has caused the Debian Bug report #961676,
regarding 5.6 kernel crashes host when using VFIO on KVM guests
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
961676: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961676
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web