Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1378280 > unrolled thread

Re: Major KVM issues with kernel 4.5 on the host

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Major KVM issues with kernel 4.5 on the host Paolo Bonzini <pbonzini@redhat.com> - 2016-04-13 22:40 +0200
    Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-13 23:00 +0200
    Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-14 00:40 +0200
      Re: Major KVM issues with kernel 4.5 on the host Paolo Bonzini <pbonzini@redhat.com> - 2016-04-14 03:20 +0200
        Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-14 07:30 +0200
        Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-14 08:10 +0200
        Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-14 18:50 +0200
          Re: Major KVM issues with kernel 4.5 on the host Paolo Bonzini <pbonzini@redhat.com> - 2016-04-14 19:40 +0200
            Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-14 19:50 +0200

#1378280 — Re: Major KVM issues with kernel 4.5 on the host

On 13/04/2016 20:37, Marc Haber wrote:
> On Fri, Mar 18, 2016 at 11:01:46AM +0100, Paolo Bonzini wrote:
>> On 17/03/2016 19:11, Borislav Petkov wrote:
>>> I'm going to try reproducing the issue on a less "important" machine
>>> so that bisecting is less painful, but maybe you guys have an idea
>>> what's going wrong here.
>>
>> No idea, sorry. :(  Bisecting would be great.
> 
> Working on that now.
> 
>>   I'll also try reproducing and bisecting next week, in the meanwhile
>>   just having the host dmesg would help a lot.
> 
> Attached. I hope the message will get through to the list.

Didn't help, but a fresh look at the list of 4.5 patches helped.
What the hell was I thinking, I missed write_rdtscp_aux who
obviously uses MSR_TSC_AUX.

diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index 31346a3f20a5..1481dea15844 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -39,6 +39,7 @@
 #include <asm/kvm_para.h>
 
 #include <asm/virtext.h>
+#include <asm/vgtod.h>
 #include "trace.h"
 
 #define __ex(x) __kvm_handle_fault_on_reboot(x)
@@ -1240,9 +1241,6 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 			wrmsrl(MSR_AMD64_TSC_RATIO, tsc_ratio);
 		}
 	}
-	/* This assumes that the kernel never uses MSR_TSC_AUX */
-	if (static_cpu_has(X86_FEATURE_RDTSCP))
-		wrmsrl(MSR_TSC_AUX, svm->tsc_aux);
 }
 
 static void svm_vcpu_put(struct kvm_vcpu *vcpu)
@@ -3847,6 +3845,8 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
 	svm->vmcb->save.cr2 = vcpu->arch.cr2;
 
 	clgi();
+	if (static_cpu_has(X86_FEATURE_RDTSCP))
+		wrmsrl(MSR_TSC_AUX, svm->tsc_aux);
 
 	local_irq_enable();
 
@@ -3923,6 +3923,8 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
 #endif
 		);
 
+	if (static_cpu_has(X86_FEATURE_RDTSCP))
+		wrmsrl(MSR_TSC_AUX, __getcpu());
 #ifdef CONFIG_X86_64
 	wrmsrl(MSR_GS_BASE, svm->host.gs_base);
 #else


Paolo

[toc] | [next] | [standalone]

#1378294

On Wed, Apr 13, 2016 at 10:36:34PM +0200, Paolo Bonzini wrote:
> Didn't help, but a fresh look at the list of 4.5 patches helped.
> What the hell was I thinking, I missed write_rdtscp_aux who
> obviously uses MSR_TSC_AUX.

So you want me to apply that to 4.5 od 4.5.1 and try that?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1378325

On Wed, Apr 13, 2016 at 10:36:34PM +0200, Paolo Bonzini wrote:
> Didn't help, but a fresh look at the list of 4.5 patches helped.
> What the hell was I thinking, I missed write_rdtscp_aux who
> obviously uses MSR_TSC_AUX.

I applied this patch to 4.5, which didn't go cleanly, I had to do it
manually, and there is no change in behavior. Sometimes, the Vm just
crashes, but most times the filesystem is remounted ro.

[   84.658968] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27903
[   84.664877] Aborting journal on device dm-0-8.
[   84.667992] EXT4-fs (dm-0): Remounting filesystem read-only
[   84.670972] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal
[   84.763331] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27898
[   84.825412] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27895
[   84.907959] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27893
[   84.915187] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27900
[   84.961062] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27889
[   84.983700] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27891
[   98.315538] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #23567: comm aide: deleted inode referenced: 27897
[   98.323606] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #23567: comm aide: deleted inode referenced: 27904
[   99.889927] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27892
[   99.893823] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27901
[   99.901140] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27890
[   99.904898] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27896
[   99.909758] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27899
[   99.914394] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27894
[  207.132045] serial8250: too much work for irq4
[  207.220043] serial8250: too much work for irq4
[  207.312028] serial8250: too much work for irq4


Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1378383

On 14/04/2016 00:29, Marc Haber wrote:
> On Wed, Apr 13, 2016 at 10:36:34PM +0200, Paolo Bonzini wrote:
>> Didn't help, but a fresh look at the list of 4.5 patches helped.
>> What the hell was I thinking, I missed write_rdtscp_aux who
>> obviously uses MSR_TSC_AUX.
> 
> I applied this patch to 4.5, which didn't go cleanly, I had to do it
> manually, and there is no change in behavior. Sometimes, the Vm just
> crashes, but most times the filesystem is remounted ro.

Ok, then I guess bisection is needed.  Please first try commit
45bdbcfdf241.  If it fails, then the bug come together with KVM's merge
window changes for 4.5-rc1.  Please apply the patch I sent here when
bisection is past 46896c73c1a4dde527c3a3cc43379deeb41985a1 (which means
that probably that should be the commit you try second; the bisection
then becomes much easier).

Thanks,

Paolo

> [   84.658968] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27903
> [   84.664877] Aborting journal on device dm-0-8.
> [   84.667992] EXT4-fs (dm-0): Remounting filesystem read-only
> [   84.670972] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal
> [   84.763331] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27898
> [   84.825412] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27895
> [   84.907959] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27893
> [   84.915187] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27900
> [   84.961062] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27889
> [   84.983700] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #7669: comm aide: deleted inode referenced: 27891
> [   98.315538] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #23567: comm aide: deleted inode referenced: 27897
> [   98.323606] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #23567: comm aide: deleted inode referenced: 27904
> [   99.889927] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27892
> [   99.893823] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27901
> [   99.901140] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27890
> [   99.904898] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27896
> [   99.909758] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27899
> [   99.914394] EXT4-fs error (device dm-0): ext4_lookup:1602: inode #4650: comm aide: deleted inode referenced: 27894
> [  207.132045] serial8250: too much work for irq4
> [  207.220043] serial8250: too much work for irq4
> [  207.312028] serial8250: too much work for irq4
> 
> 
> Greetings
> Marc
> 

[toc] | [prev] | [next] | [standalone]

#1378484

On Thu, Apr 14, 2016 at 03:16:29AM +0200, Paolo Bonzini wrote:
> On 14/04/2016 00:29, Marc Haber wrote:
> > On Wed, Apr 13, 2016 at 10:36:34PM +0200, Paolo Bonzini wrote:
> >> Didn't help, but a fresh look at the list of 4.5 patches helped.
> >> What the hell was I thinking, I missed write_rdtscp_aux who
> >> obviously uses MSR_TSC_AUX.
> > 
> > I applied this patch to 4.5, which didn't go cleanly, I had to do it
> > manually, and there is no change in behavior. Sometimes, the Vm just
> > crashes, but most times the filesystem is remounted ro.
> 
> Ok, then I guess bisection is needed.  Please first try commit
> 45bdbcfdf241.  If it fails, then the bug come together with KVM's merge
> window changes for 4.5-rc1.  Please apply the patch I sent here when
> bisection is past 46896c73c1a4dde527c3a3cc43379deeb41985a1 (which means
> that probably that should be the commit you try second; the bisection
> then becomes much easier).

I have never bisected this deeply. Can you please give more advice,
with which two commits to start? And how do I find out whether I am
"past" a commit? I am als not a git expert, a few command lines would
be appreciated.

Things have not become any easier this night; 4.5-rc7 ran for more
than three hours before it failed :-(

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1378504

On Thu, Apr 14, 2016 at 03:16:29AM +0200, Paolo Bonzini wrote:
> Ok, then I guess bisection is needed.  Please first try commit
> 45bdbcfdf241.

That kernel labels itself as "4.4.0-rc5+", is that correct?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1379092

On Thu, Apr 14, 2016 at 03:16:29AM +0200, Paolo Bonzini wrote:
> Ok, then I guess bisection is needed.  Please first try commit
> 45bdbcfdf241.

I did git checkout 45bdbcfdf241 and built the resulting kernel
4.4.0-rc5. This one has now been running for ten hours, which is
threefold the longest time that a faulty kernel has held before a VM
experienced corruption. So I guess, that one is fine.

Since 4.5.0-rc1 is bad, I guess I do:

git checkout 45bdbcfdf241
git bisect start
git bisect good
git bisect bad v4.5.0-rc1

right?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1379148

On 14/04/2016 18:47, Marc Haber wrote:
>> > Ok, then I guess bisection is needed.  Please first try commit
>> > 45bdbcfdf241.
> I did git checkout 45bdbcfdf241 and built the resulting kernel
> 4.4.0-rc5. This one has now been running for ten hours, which is
> threefold the longest time that a faulty kernel has held before a VM
> experienced corruption. So I guess, that one is fine.

Interesting, this means it's not a KVM bug.  You can ignore my patch
from yesterday (though we'll get it in anyway).

> Since 4.5.0-rc1 is bad, I guess I do:
> 
> git checkout 45bdbcfdf241
> git bisect start
> git bisect good
> git bisect bad v4.5.0-rc1

This is correct but you also want to do

git bisect good 4.4.0
git bisect good 4.4.0-rc5

so that bisection basically works through the commits in the merge window.

Thanks,

Paolo

[toc] | [prev] | [next] | [standalone]

#1379152

On Thu, Apr 14, 2016 at 07:30:43PM +0200, Paolo Bonzini wrote:
> On 14/04/2016 18:47, Marc Haber wrote:
> >> > Ok, then I guess bisection is needed.  Please first try commit
> >> > 45bdbcfdf241.
> > I did git checkout 45bdbcfdf241 and built the resulting kernel
> > 4.4.0-rc5. This one has now been running for ten hours, which is
> > threefold the longest time that a faulty kernel has held before a VM
> > experienced corruption. So I guess, that one is fine.
> 
> Interesting, this means it's not a KVM bug.  You can ignore my patch
> from yesterday (though we'll get it in anyway).
> 
> > Since 4.5.0-rc1 is bad, I guess I do:
> > 
> > git checkout 45bdbcfdf241
> > git bisect start
> > git bisect good
> > git bisect bad v4.5.0-rc1
> 
> This is correct but you also want to do
> 
> git bisect good 4.4.0
> git bisect good 4.4.0-rc5
> 
> so that bisection basically works through the commits in the merge window.

So I start over from this:

[47/544]mh@fan:~/linux/debug/linux$ git checkout 45bdbcfdf241
HEAD is now at 45bdbcf... kvm: x86: Fix vmwrite to SECONDARY_VM_EXEC_CONTROL
[48/545]mh@fan:~/linux/debug/linux$ git bisect start
[49/546]mh@fan:~/linux/debug/linux$ git bisect good
[50/547]mh@fan:~/linux/debug/linux$ git bisect bad v4.5-rc1
Bisecting: 5761 revisions left to test after this (roughly 13 steps)
[cbd88cd4c07f9361914ab7fd7e21c9227986fe68] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
[51/548]mh@fan:~/linux/debug/linux$ git bisect good v4.4
Bisecting: 5468 revisions left to test after this (roughly 12 steps)
[f9a03ae123c92c1f45cd2ca88d0f6edd787be78c] Merge tag 'for-f2fs-4.5' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs
[52/549]mh@fan:~/linux/debug/linux$ git bisect good v4.4-rc5
Bisecting: 5468 revisions left to test after this (roughly 12 steps)
[f9a03ae123c92c1f45cd2ca88d0f6edd787be78c] Merge tag 'for-f2fs-4.5' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs
[53/550]mh@fan:~/linux/debug/linux$

This is going to take a few days as detecting a "bad" version may take
a few hours.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web