Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1360068 > unrolled thread

Major KVM issues with kernel 4.5 on the host

Back to article view | Back to linux.kernel

Contents

  Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-03-17 18:20 +0100
    Re: Major KVM issues with kernel 4.5 on the host Borislav Petkov <bp@alien8.de> - 2016-03-17 19:20 +0100
      Re: Major KVM issues with kernel 4.5 on the host Paolo Bonzini <pbonzini@redhat.com> - 2016-03-18 11:10 +0100
      Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-03-18 19:50 +0100
        Re: Major KVM issues with kernel 4.5 on the host Borislav Petkov <bp@alien8.de> - 2016-03-18 23:10 +0100
          Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-03-19 01:10 +0100
            Re: Major KVM issues with kernel 4.5 on the host Borislav Petkov <bp@alien8.de> - 2016-03-20 14:40 +0100
              Re: Major KVM issues with kernel 4.5 on the host Andrey Korolyov <andrey@xdel.ru> - 2016-03-20 18:20 +0100
                Re: Major KVM issues with kernel 4.5 on the host Borislav Petkov <bp@alien8.de> - 2016-03-20 19:30 +0100
                  Re: Major KVM issues with kernel 4.5 on the host Andrey Korolyov <andrey@xdel.ru> - 2016-03-20 20:00 +0100
                    Re: Major KVM issues with kernel 4.5 on the host Borislav Petkov <bp@alien8.de> - 2016-03-20 20:00 +0100
            Re: Major KVM issues with kernel 4.5 on the host Paolo Bonzini <pbonzini@redhat.com> - 2016-03-21 10:10 +0100

#1360068 — Major KVM issues with kernel 4.5 on the host

Hi,

I have a (semi-productive[1]) system ("host") running Debian unstable.
On this system, a few VMs (Debian unstable, Debian testing) ("vm1",
"vm2", "vm3") are running. I roll my own kernels and take vanilla
upstream sources. No distribution patches.

Since host was updated to Kernel 4.5, the VMs have started acting up.
All of them. The range of strangeness begins with "relocation error,
system halted" on system startup, corrupted data files on disk,
filesystems remounted read-only, libraries rejected with "invalid ELF
format", binaries segfaulting all of a sudden. Downgrading host to
kernel 4.4.5 magically fixed all those issues.

Going back to 4.5 lets the issues reappear. Here, for example, ext4 fs
errors, logged in one of the VMs:

Mar 17 17:39:57 spinturn kernel: EXT4-fs error (device dm-0): ext4_lookup:1602: inode #415065: comm aide: deleted inode referenced: 546538
Mar 17 17:39:57 spinturn kernel: EXT4-fs error (device dm-0): ext4_lookup:1602: inode #415065: comm aide: deleted inode referenced: 546530
Mar 17 17:39:57 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4269: inode #546543: comm aide: bad extra_isize (44800 != 256)
Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4466: inode #546568: comm aide: bogus i_mode (144)
Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_lookup:1602: inode #546548: comm aide: deleted inode referenced: 546564
Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_lookup:1602: inode #546548: comm aide: deleted inode referenced: 546562
Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4269: inode #546563: comm aide: bad extra_isize (6464 != 256)
Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4466: inode #546561: comm aide: bogus i_mode (0)
Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4269: inode #546529: comm aide: bad extra_isize (1152 != 256)
Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_xattr_block_get:297: inode #546359: comm aide: bad block 677784

I'm going to try reproducing the issue on a less "important" machine
so that bisecting is less painful, but maybe you guys have an idea
what's going wrong here.

jftr, kernel 4.5 in guest and in standalone systems seems to be
unproblematic.

Greetings
Marc


[1] my main workstation, running enough services for the local network
that disturbances in its operation cause reasonable discomfort, but not the
Enterprise kind of "productive"

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [next] | [standalone]

#1360117

+ kvm ML.

Do you have any funky messages in host's dmesg ? Can you upload a full
dmesg from both a good and a bad host kernel?

On Thu, Mar 17, 2016 at 05:54:35PM +0100, Marc Haber wrote:
> Hi,
> 
> I have a (semi-productive[1]) system ("host") running Debian unstable.
> On this system, a few VMs (Debian unstable, Debian testing) ("vm1",
> "vm2", "vm3") are running. I roll my own kernels and take vanilla
> upstream sources. No distribution patches.
> 
> Since host was updated to Kernel 4.5, the VMs have started acting up.
> All of them. The range of strangeness begins with "relocation error,
> system halted" on system startup, corrupted data files on disk,
> filesystems remounted read-only, libraries rejected with "invalid ELF
> format", binaries segfaulting all of a sudden. Downgrading host to
> kernel 4.4.5 magically fixed all those issues.
> 
> Going back to 4.5 lets the issues reappear. Here, for example, ext4 fs
> errors, logged in one of the VMs:
> 
> Mar 17 17:39:57 spinturn kernel: EXT4-fs error (device dm-0): ext4_lookup:1602: inode #415065: comm aide: deleted inode referenced: 546538
> Mar 17 17:39:57 spinturn kernel: EXT4-fs error (device dm-0): ext4_lookup:1602: inode #415065: comm aide: deleted inode referenced: 546530
> Mar 17 17:39:57 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4269: inode #546543: comm aide: bad extra_isize (44800 != 256)
> Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4466: inode #546568: comm aide: bogus i_mode (144)
> Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_lookup:1602: inode #546548: comm aide: deleted inode referenced: 546564
> Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_lookup:1602: inode #546548: comm aide: deleted inode referenced: 546562
> Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4269: inode #546563: comm aide: bad extra_isize (6464 != 256)
> Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4466: inode #546561: comm aide: bogus i_mode (0)
> Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_iget:4269: inode #546529: comm aide: bad extra_isize (1152 != 256)
> Mar 17 17:39:58 spinturn kernel: EXT4-fs error (device dm-0): ext4_xattr_block_get:297: inode #546359: comm aide: bad block 677784
> 
> I'm going to try reproducing the issue on a less "important" machine
> so that bisecting is less painful, but maybe you guys have an idea
> what's going wrong here.
> 
> jftr, kernel 4.5 in guest and in standalone systems seems to be
> unproblematic.
> 
> Greetings
> Marc
> 
> 
> [1] my main workstation, running enough services for the local network
> that disturbances in its operation cause reasonable discomfort, but not the
> Enterprise kind of "productive"
> 
> -- 
> -----------------------------------------------------------------------------
> Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
> Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
> Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
> 

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1360487

On 17/03/2016 19:11, Borislav Petkov wrote:
> I'm going to try reproducing the issue on a less "important" machine
> so that bisecting is less painful, but maybe you guys have an idea
> what's going wrong here.

No idea, sorry. :(  Bisecting would be great.  I'll also try reproducing
and bisecting next week, in the meanwhile just having the host dmesg
would help a lot.

Paolo

[toc] | [prev] | [next] | [standalone]

#1360867

Hi Borislav,

On Thu, Mar 17, 2016 at 07:11:28PM +0100, Borislav Petkov wrote:
> Do you have any funky messages in host's dmesg ?

Not that I see.

> Can you upload a full dmesg from both a good and a bad host kernel?

http://q.bofh.de/~mh/stuff/20160317-fan-syslog-kvm-4.4.5
http://q.bofh.de/~mh/stuff/20160317-fan-syslog-kvm-4.5

Hope this helps.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1360959

On Fri, Mar 18, 2016 at 07:49:29PM +0100, Marc Haber wrote:
> http://q.bofh.de/~mh/stuff/20160317-fan-syslog-kvm-4.4.5

This one I got.

> http://q.bofh.de/~mh/stuff/20160317-fan-syslog-kvm-4.5

This one doesn't want:

HTTP request sent, awaiting response... 403 Forbidden
2016-03-18 22:57:46 ERROR 403: Forbidden.

So I have a similar system to yours, I'll try to reproduce on it with
4.5.

Anything special you're doing to cause the host kernel to barf which I
should do here?

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1361015

Hi Borislav,

On Fri, Mar 18, 2016 at 11:04:29PM +0100, Borislav Petkov wrote:
> On Fri, Mar 18, 2016 at 07:49:29PM +0100, Marc Haber wrote:
> > http://q.bofh.de/~mh/stuff/20160317-fan-syslog-kvm-4.4.5
> 
> This one I got.
> 
> > http://q.bofh.de/~mh/stuff/20160317-fan-syslog-kvm-4.5
> 
> This one doesn't want:
> 
> HTTP request sent, awaiting response... 403 Forbidden
> 2016-03-18 22:57:46 ERROR 403: Forbidden.

Idiot me. File permissions fixed.

> Anything special you're doing to cause the host kernel to barf which I
> should do here?

Booting Debian Linux, apt-get update, apt-get upgrade, and run aide
(which builds checksums for the entire filesystem, a rather disk-bound
activity).

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1361283

On Sat, Mar 19, 2016 at 01:08:37AM +0100, Marc Haber wrote:
> Booting Debian Linux, apt-get update, apt-get upgrade, and run aide
> (which builds checksums for the entire filesystem, a rather disk-bound
> activity).

So I did that and aide ran a whole init and check all the way through
and all fine. I don't see anything out of the ordinary in your dmesg
outputs either.

The next things we should look like is:

* diff .configs - there might be something there

* try to reproduce on debian testing or even stable. I have had similar
issues with debian unstable in the past.

* something else which I'm not thinking of it right now.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1361327

On Sun, Mar 20, 2016 at 4:31 PM, Borislav Petkov <bp@alien8.de> wrote:
> On Sat, Mar 19, 2016 at 01:08:37AM +0100, Marc Haber wrote:
>> Booting Debian Linux, apt-get update, apt-get upgrade, and run aide
>> (which builds checksums for the entire filesystem, a rather disk-bound
>> activity).
>
> So I did that and aide ran a whole init and check all the way through
> and all fine. I don't see anything out of the ordinary in your dmesg
> outputs either.
>
> The next things we should look like is:
>
> * diff .configs - there might be something there
>
> * try to reproduce on debian testing or even stable. I have had similar
> issues with debian unstable in the past.
>
> * something else which I'm not thinking of it right now.
>
> --
> Regards/Gruss,
>     Boris.
>

Kinda naive question - do you run same ucode version as Marc on his device?

[toc] | [prev] | [next] | [standalone]

#1361347

On Sun, Mar 20, 2016 at 08:14:58PM +0300, Andrey Korolyov wrote:
> Kinda naive question - do you run same ucode version as Marc on his device?

Yeah, we both have 0x010000dc.

In case you're referring to the recent faulty AMD microcode patch -
it doesn't apply here. The boxes in question are family 0x10 and the
microcode patch is for family 0x15.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1361399

On Sun, Mar 20, 2016 at 9:25 PM, Borislav Petkov <bp@alien8.de> wrote:
> On Sun, Mar 20, 2016 at 08:14:58PM +0300, Andrey Korolyov wrote:
>> Kinda naive question - do you run same ucode version as Marc on his device?
>
> Yeah, we both have 0x010000dc.
>
> In case you're referring to the recent faulty AMD microcode patch -
> it doesn't apply here. The boxes in question are family 0x10 and the
> microcode patch is for family 0x15.
>

Yes, I suggested that the issue could fall over a different family as
well to expose explicit corruption of a guest pages (as opposed to a
generic corruption in a known case). Since there is no direct evidence
of what exactly (data or pgt) is getting corrupted, would disabling
npt for a testing purposes be helpful?

[toc] | [prev] | [next] | [standalone]

#1361407

On Sun, Mar 20, 2016 at 09:42:15PM +0300, Andrey Korolyov wrote:
> Yes, I suggested that the issue could fall over a different family as
> well to expose explicit corruption of a guest pages (as opposed to a
> generic corruption in a known case).

Probably, but I don't think it is microcode patch related.

> Since there is no direct evidence of what exactly (data or pgt) is
> getting corrupted, would disabling npt for a testing purposes be
> helpful?

So I'm not sure what even happens here yet. I haven't seen anything out
of the ordinary in Marc's dmesg and I wasn't able to reproduce either.
So would it be good to try with "npt=0"? Sure, why not.

Marc, you could give that a try to see if it changes anything...

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1361628

On 19/03/2016 01:08, Marc Haber wrote:
>> > 
>>> > > http://q.bofh.de/~mh/stuff/20160317-fan-syslog-kvm-4.5
>> > 
>> > This one doesn't want:
>> > 
>> > HTTP request sent, awaiting response... 403 Forbidden
>> > 2016-03-18 22:57:46 ERROR 403: Forbidden.
> Idiot me. File permissions fixed.
> 
>> > Anything special you're doing to cause the host kernel to barf which I
>> > should do here?
> Booting Debian Linux, apt-get update, apt-get upgrade, and run aide
> (which builds checksums for the entire filesystem, a rather disk-bound
> activity).

Ok, so this is AMD.  I'll take a look.

Paolo

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web