Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1288719 > unrolled thread

[PATCH] kvm: x86: move tracepoints outside extended quiescent state

Started byPaolo Bonzini <pbonzini@redhat.com>
First post2015-12-10 18:40 +0100
Last post2015-12-11 13:30 +0100
Articles 7 — 2 participants

Back to article view | Back to linux.kernel

Contents

  [PATCH] kvm: x86: move tracepoints outside extended quiescent state Paolo Bonzini <pbonzini@redhat.com> - 2015-12-10 18:40 +0100
    Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent  state Borislav Petkov <bp@alien8.de> - 2015-12-10 19:10 +0100
      Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent  state Paolo Bonzini <pbonzini@redhat.com> - 2015-12-10 19:20 +0100
        Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent  state Borislav Petkov <bp@alien8.de> - 2015-12-11 11:30 +0100
          Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent  state Paolo Bonzini <pbonzini@redhat.com> - 2015-12-11 11:50 +0100
            Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent  state Paolo Bonzini <pbonzini@redhat.com> - 2015-12-11 13:20 +0100
            Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent  state Borislav Petkov <bp@alien8.de> - 2015-12-11 13:30 +0100

#1288719 — [PATCH] kvm: x86: move tracepoints outside extended quiescent state

FromPaolo Bonzini <pbonzini@redhat.com>
Date2015-12-10 18:40 +0100
Subject[PATCH] kvm: x86: move tracepoints outside extended quiescent state
Message-ID<qEdma-OR-17@gated-at.bofh.it>
Invoking tracepoints within kvm_guest_enter/kvm_guest_exit causes a
lockdep splat.

Cc: stable@vger.kernel.org
Reported-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/svm.c | 4 ++--
 arch/x86/kvm/vmx.c | 3 ++-
 arch/x86/kvm/x86.c | 2 +-
 3 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index 83a1c643f9a5..899c40f826dd 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -3422,6 +3422,8 @@ static int handle_exit(struct kvm_vcpu *vcpu)
 	struct kvm_run *kvm_run = vcpu->run;
 	u32 exit_code = svm->vmcb->control.exit_code;
 
+	trace_kvm_exit(exit_code, vcpu, KVM_ISA_SVM);
+
 	if (!is_cr_intercept(svm, INTERCEPT_CR0_WRITE))
 		vcpu->arch.cr0 = svm->vmcb->save.cr0;
 	if (npt_enabled)
@@ -3892,8 +3894,6 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
 	vcpu->arch.regs[VCPU_REGS_RSP] = svm->vmcb->save.rsp;
 	vcpu->arch.regs[VCPU_REGS_RIP] = svm->vmcb->save.rip;
 
-	trace_kvm_exit(svm->vmcb->control.exit_code, vcpu, KVM_ISA_SVM);
-
 	if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
 		kvm_before_handle_nmi(&svm->vcpu);
 
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index af823a388c19..6b5605607849 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -8042,6 +8042,8 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu)
 	u32 exit_reason = vmx->exit_reason;
 	u32 vectoring_info = vmx->idt_vectoring_info;
 
+	trace_kvm_exit(exit_reason, vcpu, KVM_ISA_VMX);
+
 	/*
 	 * Flush logged GPAs PML buffer, this will make dirty_bitmap more
 	 * updated. Another good is, in kvm_vm_ioctl_get_dirty_log, before
@@ -8668,7 +8670,6 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
 	vmx->loaded_vmcs->launched = 1;
 
 	vmx->exit_reason = vmcs_read32(VM_EXIT_REASON);
-	trace_kvm_exit(vmx->exit_reason, vcpu, KVM_ISA_VMX);
 
 	/*
 	 * the KVM_REQ_EVENT optimization bit is only on for one entry, and if
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index eed32283d22c..2a8c035ec7fe 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6515,6 +6515,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	if (req_immediate_exit)
 		smp_send_reschedule(vcpu->cpu);
 
+	trace_kvm_entry(vcpu->vcpu_id);
 	__kvm_guest_enter();
 
 	if (unlikely(vcpu->arch.switch_db_regs)) {
@@ -6527,7 +6528,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_RELOAD;
 	}
 
-	trace_kvm_entry(vcpu->vcpu_id);
 	wait_lapic_expire(vcpu);
 	kvm_x86_ops->run(vcpu);
 
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1288764 — Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent state

FromBorislav Petkov <bp@alien8.de>
Date2015-12-10 19:10 +0100
SubjectRe: [PATCH] kvm: x86: move tracepoints outside extended quiescent state
Message-ID<qEdPc-1ez-3@gated-at.bofh.it>
In reply to#1288719
On Thu, Dec 10, 2015 at 06:38:57PM +0100, Paolo Bonzini wrote:
> Invoking tracepoints within kvm_guest_enter/kvm_guest_exit causes a
> lockdep splat.
> 
> Cc: stable@vger.kernel.org
> Reported-by: Borislav Petkov <bp@alien8.de>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  arch/x86/kvm/svm.c | 4 ++--
>  arch/x86/kvm/vmx.c | 3 ++-
>  arch/x86/kvm/x86.c | 2 +-
>  3 files changed, 5 insertions(+), 4 deletions(-)

Looks like you missed some...

[  144.296364] kvm: zapping shadow pages for mmio generation wraparound
[  164.699053] kvm: zapping shadow pages for mmio generation wraparound
[  312.115767] kvm: zapping shadow pages for mmio generation wraparound
[  432.277585] kvm: zapping shadow pages for mmio generation wraparound

[  434.547820] ===============================
[  434.552020] [ INFO: suspicious RCU usage. ]
[  434.556223] 4.4.0-rc4+ #1 Not tainted
[  434.559886] -------------------------------
[  434.564072] arch/x86/kvm/trace.h:971 suspicious rcu_dereference_check() usage!
[  434.571303] 
               other info that might help us debug this:

[  434.579324] 
               RCU used illegally from idle CPU!
               rcu_scheduler_active = 1, debug_locks = 0
[  434.590209] RCU used illegally from extended quiescent state!
[  434.595971] 1 lock held by qemu-system-x86/2402:
[  434.600596]  #0:  (&vcpu->mutex){+.+.+.}, at: [<ffffffffa020872c>] vcpu_load+0x1c/0x80 [kvm]
[  434.609146] 
               stack backtrace:
[  434.613526] CPU: 4 PID: 2402 Comm: qemu-system-x86 Not tainted 4.4.0-rc4+ #1
[  434.620583] Hardware name: Dell Inc. Precision T3600/0PTTT9, BIOS A13 05/11/2014
[  434.627987]  0000000000000001 ffff88042f79fcf0 ffffffff813c2cfc ffff880435aa0000
[  434.635443]  ffff88042f79fd20 ffffffff810c5157 ffff88042fd48000 0000000295a85563
[  434.642886]  0000000295de483f 0000000000000000 ffff88042f79fd58 ffffffffa023ec6e
[  434.650334] Call Trace:
[  434.652804]  [<ffffffff813c2cfc>] dump_stack+0x4e/0x82
[  434.657950]  [<ffffffff810c5157>] lockdep_rcu_suspicious+0xe7/0x120
[  434.664239]  [<ffffffffa023ec6e>] wait_lapic_expire+0xfe/0x1e0 [kvm]
[  434.670606]  [<ffffffffa022183e>] kvm_arch_vcpu_ioctl_run+0x76e/0x19c0 [kvm]
[  434.677674]  [<ffffffffa0221982>] ? kvm_arch_vcpu_ioctl_run+0x8b2/0x19c0 [kvm]
[  434.684905]  [<ffffffff81858acc>] ? mutex_lock_killable_nested+0x29c/0x4c0
[  434.691792]  [<ffffffffa0208b32>] kvm_vcpu_ioctl+0x342/0x700 [kvm]
[  434.697984]  [<ffffffff810c4a7d>] ? __lock_is_held+0x4d/0x70
[  434.703655]  [<ffffffff812351ae>] ? __fget+0xfe/0x200
[  434.708719]  [<ffffffff812291f1>] do_vfs_ioctl+0x301/0x550
[  434.714208]  [<ffffffff8123531a>] ? __fget_light+0x2a/0x90
[  434.719700]  [<ffffffff81229481>] SyS_ioctl+0x41/0x70
[  434.724754]  [<ffffffff8185cb36>] entry_SYSCALL_64_fastpath+0x16/0x7a
[  437.411818] kvm [2400]: vcpu0 unhandled rdmsr: 0x606
[  437.416898] kvm [2400]: vcpu0 unhandled rdmsr: 0x34


-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1288781 — Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent state

FromPaolo Bonzini <pbonzini@redhat.com>
Date2015-12-10 19:20 +0100
SubjectRe: [PATCH] kvm: x86: move tracepoints outside extended quiescent state
Message-ID<qEdYS-1hY-13@gated-at.bofh.it>
In reply to#1288764

On 10/12/2015 19:09, Borislav Petkov wrote:
> On Thu, Dec 10, 2015 at 06:38:57PM +0100, Paolo Bonzini wrote:
>> Invoking tracepoints within kvm_guest_enter/kvm_guest_exit causes a
>> lockdep splat.
>>
>> Cc: stable@vger.kernel.org
>> Reported-by: Borislav Petkov <bp@alien8.de>
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> ---
>>  arch/x86/kvm/svm.c | 4 ++--
>>  arch/x86/kvm/vmx.c | 3 ++-
>>  arch/x86/kvm/x86.c | 2 +-
>>  3 files changed, 5 insertions(+), 4 deletions(-)
> 
> Looks like you missed some...

Yeah, wait_lapic_expire also have to be moved before __kvm_guest_enter.

Paolo

> [  144.296364] kvm: zapping shadow pages for mmio generation wraparound
> [  164.699053] kvm: zapping shadow pages for mmio generation wraparound
> [  312.115767] kvm: zapping shadow pages for mmio generation wraparound
> [  432.277585] kvm: zapping shadow pages for mmio generation wraparound
> 
> [  434.547820] ===============================
> [  434.552020] [ INFO: suspicious RCU usage. ]
> [  434.556223] 4.4.0-rc4+ #1 Not tainted
> [  434.559886] -------------------------------
> [  434.564072] arch/x86/kvm/trace.h:971 suspicious rcu_dereference_check() usage!
> [  434.571303] 
>                other info that might help us debug this:
> 
> [  434.579324] 
>                RCU used illegally from idle CPU!
>                rcu_scheduler_active = 1, debug_locks = 0
> [  434.590209] RCU used illegally from extended quiescent state!
> [  434.595971] 1 lock held by qemu-system-x86/2402:
> [  434.600596]  #0:  (&vcpu->mutex){+.+.+.}, at: [<ffffffffa020872c>] vcpu_load+0x1c/0x80 [kvm]
> [  434.609146] 
>                stack backtrace:
> [  434.613526] CPU: 4 PID: 2402 Comm: qemu-system-x86 Not tainted 4.4.0-rc4+ #1
> [  434.620583] Hardware name: Dell Inc. Precision T3600/0PTTT9, BIOS A13 05/11/2014
> [  434.627987]  0000000000000001 ffff88042f79fcf0 ffffffff813c2cfc ffff880435aa0000
> [  434.635443]  ffff88042f79fd20 ffffffff810c5157 ffff88042fd48000 0000000295a85563
> [  434.642886]  0000000295de483f 0000000000000000 ffff88042f79fd58 ffffffffa023ec6e
> [  434.650334] Call Trace:
> [  434.652804]  [<ffffffff813c2cfc>] dump_stack+0x4e/0x82
> [  434.657950]  [<ffffffff810c5157>] lockdep_rcu_suspicious+0xe7/0x120
> [  434.664239]  [<ffffffffa023ec6e>] wait_lapic_expire+0xfe/0x1e0 [kvm]
> [  434.670606]  [<ffffffffa022183e>] kvm_arch_vcpu_ioctl_run+0x76e/0x19c0 [kvm]
> [  434.677674]  [<ffffffffa0221982>] ? kvm_arch_vcpu_ioctl_run+0x8b2/0x19c0 [kvm]
> [  434.684905]  [<ffffffff81858acc>] ? mutex_lock_killable_nested+0x29c/0x4c0
> [  434.691792]  [<ffffffffa0208b32>] kvm_vcpu_ioctl+0x342/0x700 [kvm]
> [  434.697984]  [<ffffffff810c4a7d>] ? __lock_is_held+0x4d/0x70
> [  434.703655]  [<ffffffff812351ae>] ? __fget+0xfe/0x200
> [  434.708719]  [<ffffffff812291f1>] do_vfs_ioctl+0x301/0x550
> [  434.714208]  [<ffffffff8123531a>] ? __fget_light+0x2a/0x90
> [  434.719700]  [<ffffffff81229481>] SyS_ioctl+0x41/0x70
> [  434.724754]  [<ffffffff8185cb36>] entry_SYSCALL_64_fastpath+0x16/0x7a
> [  437.411818] kvm [2400]: vcpu0 unhandled rdmsr: 0x606
> [  437.416898] kvm [2400]: vcpu0 unhandled rdmsr: 0x34
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1289396 — Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent state

FromBorislav Petkov <bp@alien8.de>
Date2015-12-11 11:30 +0100
SubjectRe: [PATCH] kvm: x86: move tracepoints outside extended quiescent state
Message-ID<qEt7A-2X2-27@gated-at.bofh.it>
In reply to#1288781
On Thu, Dec 10, 2015 at 07:15:19PM +0100, Paolo Bonzini wrote:
> Yeah, wait_lapic_expire also have to be moved before __kvm_guest_enter.

Yeah, v2 doesn't splat on the Intel box anymore but the AMD box still
has, and it is a different problem. With the v2 applied, it still
explodes, see below.

And I'm willing to bet good money on that shadow pages fun. The

[  959.466549] kernel tried to execute NX-protected page - exploit attempt? (uid: 1000)

line basically says that we're pagefaulting when trying to fetch
instructions, i.e., we're trying to execute something from a page, rIP
points to 0xffff8800b9f9bdf0 and that is most likely a page belonging to
kvm, which, however, is for some reason not executable (anymore?).

Could it have anything to do with that zapping of shadow pages, per
chance?

Can I disable the zapping and see if it still triggers? Or should I try
modprobing kvm with "npt=0" or so?

/me goes and tries it...

Nope, that doesn't help - it still splats.

Hmmm...

[  849.272337] kvm: zapping shadow pages for mmio generation wraparound
[  933.813871] kvm: zapping shadow pages for mmio generation wraparound
[  959.466549] kernel tried to execute NX-protected page - exploit attempt? (uid: 1000)
[  959.474369] BUG: unable to handle kernel paging request at ffff8800b9f9bdf0
[  959.481407] IP: [<ffff8800b9f9bdf0>] 0xffff8800b9f9bdf0
[  959.486677] PGD 2d7e067 PUD 43efff067 PMD 80000000b9e001e3 
[  959.492338] Oops: 0011 [#1] PREEMPT SMP 
[  959.496340] Modules linked in: tun sha256_ssse3 sha256_generic drbg binfmt_misc ipv6 vfat fat fuse dm_crypt dm_mod kvm_amd kvm irqbypass crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd amd64_edac_mod fam15h_power k10temp edac_mce_amd amdkfd amd_iommu_v2 radeon acpi_cpufreq
[  959.524023] CPU: 3 PID: 3798 Comm: qemu-system-x86 Not tainted 4.4.0-rc4+ #8
[  959.531127] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 EVO R2.0, BIOS 1503 01/16/2013
[  959.541113] task: ffff8800b7ca5e00 ti: ffff8800b9f98000 task.ti: ffff8800b9f98000
[  959.548625] RIP: 0010:[<ffff8800b9f9bdf0>]  [<ffff8800b9f9bdf0>] 0xffff8800b9f9bdf0
[  959.556338] RSP: 0018:ffff8800b9f9bde0  EFLAGS: 00010206
[  959.561676] RAX: 000003993d0f82ee RBX: ffff8800b7d48000 RCX: 0000000000000001
[  959.568844] RDX: 0000039900000000 RSI: ffffffffa02bdc7b RDI: ffff8800b7d48000
[  959.576010] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[  959.583177] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
[  959.590346] R13: ffff8800b7d48000 R14: 0000000000000000 R15: 0000000000000000
[  959.597513] FS:  00007f7fae580700(0000) GS:ffff88042cc00000(0000) knlGS:0000000000000000
[  959.605643] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  959.611414] CR2: ffff8800b9f9bdf0 CR3: 000000041b5fe000 CR4: 00000000000406e0
[  959.618579] Stack:
[  959.620607]  ffffffffa02d5e17 ffff8800b7d48000 ffff8800b9f9be08 ffffffffa02bdb1f
[  959.628104]  0000000000000000 ffff8800b9f9be98 ffffffffa02bdc7b ffff8804242a4400
[  959.635601]  0000000000000070 0000000000004000 ffffffff81a3c1e0 ffff8800b7ca5e00
[  959.643114] Call Trace:
[  959.645599]  [<ffffffffa02d5e17>] ? kvm_arch_vcpu_put+0x17/0x40 [kvm]
[  959.652081]  [<ffffffffa02bdb1f>] ? vcpu_put+0x1f/0x60 [kvm]
[  959.657782]  [<ffffffffa02bdc7b>] ? kvm_vcpu_ioctl+0x11b/0x6f0 [kvm]
[  959.664169]  [<ffffffff811a0930>] ? do_vfs_ioctl+0x2e0/0x540
[  959.669855]  [<ffffffff811ac8e9>] ? __fget_light+0x29/0x90
[  959.675364]  [<ffffffff811a0bdc>] ? SyS_ioctl+0x4c/0x90
[  959.680618]  [<ffffffff816e2d5b>] ? entry_SYSCALL_64_fastpath+0x16/0x6f
[  959.687263] Code: 00 00 00 06 02 01 00 00 00 00 00 e0 bd f9 b9 00 88 ff ff 18 00 00 00 00 00 00 00 17 5e 2d a0 ff ff ff ff 00 80 d4 b7 00 88 ff ff <08> be f9 b9 00 88 ff ff 1f db 2b a0 ff ff ff ff 00 00 00 00 00 
[  959.707506] RIP  [<ffff8800b9f9bdf0>] 0xffff8800b9f9bdf0
[  959.712862]  RSP <ffff8800b9f9bde0>
[  959.716373] CR2: ffff8800b9f9bdf0
[  959.735764] ---[ end trace 6826bd13f6e235cd ]---
[  959.740465] note: qemu-system-x86[3798] exited with preempt_count 1
[  979.163010] kvm: zapping shadow pages for mmio generation wraparound

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1289408 — Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent state

FromPaolo Bonzini <pbonzini@redhat.com>
Date2015-12-11 11:50 +0100
SubjectRe: [PATCH] kvm: x86: move tracepoints outside extended quiescent state
Message-ID<qEtqW-35O-11@gated-at.bofh.it>
In reply to#1289396

On 11/12/2015 11:22, Borislav Petkov wrote:
> On Thu, Dec 10, 2015 at 07:15:19PM +0100, Paolo Bonzini wrote:
>> Yeah, wait_lapic_expire also have to be moved before __kvm_guest_enter.
> 
> Yeah, v2 doesn't splat on the Intel box anymore but the AMD box still
> has, and it is a different problem. With the v2 applied, it still
> explodes, see below.

Yes, I didn't expect it to fix anything.  I just wanted to pinpoint it
to kvm-amd.

> And I'm willing to bet good money on that shadow pages fun.

You can disable it (well, make it take a few days to appear) with this:

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 484079efea5b..a9070e260c72 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -496,7 +496,7 @@ static struct kvm_memslots *kvm_alloc_memslots(void)
 	 * Init kvm generation close to the maximum to easily test the
 	 * code of handling generation number wrap-around.
 	 */
-	slots->generation = -150;
+	slots->generation = 0;
 	for (i = 0; i < KVM_MEM_SLOTS_NUM; i++)
 		slots->id_to_index[i] = slots->memslots[i].id = i;

but it would not be AMD-specific.

Anyway if this theory is true:

> [  959.466549] kernel tried to execute NX-protected page - exploit attempt? (uid: 1000)
> 
> line basically says that we're pagefaulting when trying to fetch
> instructions, i.e., we're trying to execute something from a page, rIP
> points to 0xffff8800b9f9bdf0 and that is most likely a page belonging to
> kvm, which, however, is for some reason not executable (anymore?).

It would be a kvm hypervisor page, not a kvm guest page, hence unrelated
to the zapping thing.

Can you grab the kallsyms before making it crash?  I will get to it next
week.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1289517 — Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent state

FromPaolo Bonzini <pbonzini@redhat.com>
Date2015-12-11 13:20 +0100
SubjectRe: [PATCH] kvm: x86: move tracepoints outside extended quiescent state
Message-ID<qEuQ2-486-25@gated-at.bofh.it>
In reply to#1289408

On 11/12/2015 12:41, Borislav Petkov wrote:
> On Fri, Dec 11, 2015 at 11:41:30AM +0100, Paolo Bonzini wrote:
>> It would be a kvm hypervisor page, not a kvm guest page, hence unrelated
>> to the zapping thing.
> 
> Ah right, guest pages should be userspace addresses, come to think of
> it.
> 
>> Can you grab the kallsyms before making it crash?
> 
> Attached. It was a different corruption this time, see below. This time
> we don't even have a page table, PGD is 0, rIP is 1. (Fun :-))

Hmm, you had:

- RIP=0 in the original report (start_this_handle)
- RIP=0 in the second (mutex_lock_nested in ext4)
- RIP=1 now

The more interesting one is the other one which doesn't have a small RIP,
because it has RIP that is slightly larger than the stack pointer, meaning
it's likely a frame pointer.  And this means in turn that the call trace
is correct, and the bug might have happened closer to the actual corruption.

[  959.548625] RIP: 0010:[<ffff8800b9f9bdf0>]  [<ffff8800b9f9bdf0>] 0xffff8800b9f9bdf0
[  959.556338] RSP: 0018:ffff8800b9f9bde0  EFLAGS: 00010206
[  959.618579] Stack:
[  959.620607]  ffffffffa02d5e17 ffff8800b7d48000 ffff8800b9f9be08 ffffffffa02bdb1f
[  959.628104]  0000000000000000 ffff8800b9f9be98 ffffffffa02bdc7b ffff8804242a4400
[  959.635601]  0000000000000070 0000000000004000 ffffffff81a3c1e0 ffff8800b7ca5e00
[  959.643114] Call Trace:
[  959.645599]  [<ffffffffa02d5e17>] ? kvm_arch_vcpu_put+0x17/0x40 [kvm]
[  959.652081]  [<ffffffffa02bdb1f>] ? vcpu_put+0x1f/0x60 [kvm]
[  959.657782]  [<ffffffffa02bdc7b>] ? kvm_vcpu_ioctl+0x11b/0x6f0 [kvm]
[  959.664169]  [<ffffffff811a0930>] ? do_vfs_ioctl+0x2e0/0x540
[  959.669855]  [<ffffffff811ac8e9>] ? __fget_light+0x29/0x90
[  959.675364]  [<ffffffff811a0bdc>] ? SyS_ioctl+0x4c/0x90
[  959.680618]  [<ffffffff816e2d5b>] ? entry_SYSCALL_64_fastpath+0x16/0x6f

My wild guess is that RSP is getting corrupted, but I guess I'll have to try
to reproduce to figure out what happens.

The last thing I need from you (hopefully) is a Kconfig.  If you have some
time, it would be great to check if you can reproduce it with an older kernel
version---trying 4.4-rc1 and 4.3 would be great.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1289519 — Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent state

FromBorislav Petkov <bp@alien8.de>
Date2015-12-11 13:30 +0100
SubjectRe: [PATCH] kvm: x86: move tracepoints outside extended quiescent state
Message-ID<qEuZI-4cP-13@gated-at.bofh.it>
In reply to#1289408
On Fri, Dec 11, 2015 at 11:41:30AM +0100, Paolo Bonzini wrote:
> You can disable it (well, make it take a few days to appear) with this:
> 
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 484079efea5b..a9070e260c72 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -496,7 +496,7 @@ static struct kvm_memslots *kvm_alloc_memslots(void)
>  	 * Init kvm generation close to the maximum to easily test the
>  	 * code of handling generation number wrap-around.
>  	 */
> -	slots->generation = -150;
> +	slots->generation = 0;
>  	for (i = 0; i < KVM_MEM_SLOTS_NUM; i++)
>  		slots->id_to_index[i] = slots->memslots[i].id = i;
> 
> but it would not be AMD-specific.

Yeah, that didn't help. This time the splat is a bit more interesting:

qemu process segfaulted at a kernel address - ffffffff816e2db1 - which
is the last insn of entry_SYSCALL_64_fastpath:

ffffffff816e2d45 <entry_SYSCALL_64_fastpath>:
ffffffff816e2d45:       25 ff ff ff bf          and    $0xbfffffff,%eax
ffffffff816e2d4a:       3d 21 02 00 00          cmp    $0x221,%eax

...

ffffffff816e2d9e:       4c 8b 9c 24 90 00 00    mov    0x90(%rsp),%r11
ffffffff816e2da5:       00 
ffffffff816e2da6:       48 8b a4 24 98 00 00    mov    0x98(%rsp),%rsp
ffffffff816e2dad:       00 
ffffffff816e2dae:       0f 01 f8                swapgs
ffffffff816e2db1:       48 0f 07                sysretq

Yap, at SYSRET.

Andy might find this a little amusing :-)

[  459.130565] qemu-system-x86[3724]: segfault at ffffffff816e2db1 ip ffffffff816e2db1 sp 00007fd593ffe970 error 15
[  512.578297] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  512.586189] IP: [<          (null)>]           (null)
[  512.591266] PGD 0 
[  512.593303] Oops: 0010 [#1] PREEMPT SMP 
[  512.597283] Modules linked in: tun sha256_ssse3 sha256_generic drbg binfmt_misc ipv6 vfat fat fuse dm_crypt dm_mod kvm_amd kvm irqbypass crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd amd64_edac_mod k10temp edac_mce_amd fam15h_power amdkfd amd_iommu_v2 radeon acpi_cpufreq
[  512.601698] CPU: 5 PID: 3787 Comm: qemu-system-x86 Not tainted 4.4.0-rc4+ #8
[  512.601699] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 EVO R2.0, BIOS 1503 01/16/2013
[  512.601700] task: ffff8800b5d04680 ti: ffff88041104c000 task.ti: ffff88041104c000
[  512.601701] RIP: 0010:[<0000000000000000>] 
[  512.601701]  [<          (null)>]           (null)
[  512.601702] RSP: 0018:ffff88041104fcc0  EFLAGS: 00010212
[  512.601703] RAX: 0000000000000040 RBX: ffff8804110b4000 RCX: 0000000000000000
[  512.601703] RDX: 0000000000000000 RSI: ffff8804110b4000 RDI: ffff88041104fc20
[  512.601704] RBP: ffff88041104fcc8 R08: 0000000000000001 R09: 0000000000000000
[  512.601704] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff
[  512.601705] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
[  512.601706] FS:  00007fb0deaf4700(0000) GS:ffff88042d000000(0000) knlGS:0000000000000000
[  512.601706] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  512.601707] CR2: 0000000000000000 CR3: 0000000428118000 CR4: 00000000000406e0
[  512.601707] Stack:
[  512.601708]  ffffffffa02b2d3c
[  512.601708]  ffff88041104fdd8
[  512.601709]  ffffffffa02b2d0e
[  512.601709]  ffffffffa02b2c9a

[  512.601710]  ffff8804110b4430
[  512.601710]  0000000000000000
[  512.601710]  0000000100000004
[  512.601711]  ffffffffa02e9bb0

[  512.601711]  0000000400000002
[  512.601711]  0000000000000000
[  512.601712]  0000000000000000
[  512.601712]  0000000000000000

[  512.601713] Call Trace:
[  512.601729]  [<ffffffffa02b2d3c>] ? kvm_set_irq+0x13c/0x250 [kvm]
[  512.601736]  [<ffffffffa02b2d0e>] kvm_set_irq+0x10e/0x250 [kvm]
[  512.601744]  [<ffffffffa02b2c9a>] ? kvm_set_irq+0x9a/0x250 [kvm]
[  512.601756]  [<ffffffffa02e9bb0>] ? kvm_set_msi_irq+0x1b0/0x1b0 [kvm]
[  512.601767]  [<ffffffffa02e9bd0>] ? kvm_set_ioapic_irq+0x20/0x20 [kvm]
[  512.601776]  [<ffffffffa02c2612>] kvm_vm_ioctl_irq_line+0x32/0x40 [kvm]
[  512.601783]  [<ffffffffa02af97b>] kvm_vm_ioctl+0x5eb/0x820 [kvm]
[  512.601786]  [<ffffffff810c6a25>] ? rcu_read_lock_held+0x45/0x60
[  512.601788]  [<ffffffff811a0930>] do_vfs_ioctl+0x2e0/0x540
[  512.601790]  [<ffffffff811ac8e9>] ? __fget_light+0x29/0x90
[  512.601791]  [<ffffffff811a0bdc>] SyS_ioctl+0x4c/0x90
[  512.601794]  [<ffffffff816e2d5b>] entry_SYSCALL_64_fastpath+0x16/0x6f
[  512.601797] Code:  Bad RIP value.
[  512.601798] RIP  [<          (null)>]           (null)
[  512.601798]  RSP <ffff88041104fcc0>
[  512.601799] CR2: 0000000000000000
[  512.609862] ---[ end trace ae4f00b514141891 ]---

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web