Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1587244 > unrolled thread

Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration

Started by"Herongguang (Stephen)" <herongguang.he@huawei.com>
First post2017-02-24 03:30 +0100
Last post2017-02-24 16:20 +0100
Articles 9 — 5 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration "Herongguang (Stephen)" <herongguang.he@huawei.com> - 2017-02-24 03:30 +0100
    Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration "Herongguang (Stephen)" <herongguang.he@huawei.com> - 2017-02-24 10:40 +0100
      Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration Greg KH <gregkh@linuxfoundation.org> - 2017-02-24 11:10 +0100
      Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration Greg KH <gregkh@linuxfoundation.org> - 2017-02-24 11:10 +0100
        Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration Paolo Bonzini <pbonzini@redhat.com> - 2017-02-24 11:10 +0100
    Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration Paolo Bonzini <pbonzini@redhat.com> - 2017-02-24 16:20 +0100
      Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration "Herongguang (Stephen)" <herongguang.he@huawei.com> - 2017-02-25 02:50 +0100
        Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration "Huang, Kai" <kai.huang@linux.intel.com> - 2017-02-27 08:50 +0100
    Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration Chris Friesen <chris.friesen@windriver.com> - 2017-02-24 16:20 +0100

#1587244 — Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration

From"Herongguang (Stephen)" <herongguang.he@huawei.com>
Date2017-02-24 03:30 +0100
SubjectRe: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Message-ID<tedNU-1Pt-3@gated-at.bofh.it>

On 2017/2/22 22:43, Paolo Bonzini wrote:
>
>
> On 22/02/2017 14:31, Chris Friesen wrote:
>>>>
>>>
>>> Can you reproduce it with kernel 4.8+?  I'm suspecting commmit
>>> 4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
>>> 2016-07-14) to be the fix.
>>
>> I can't easily try with a newer kernel, the software package we're using
>> has kernel patches that would have to be ported.
>>
>> I'm at a conference, don't really have time to set up a pair of test
>> machines from scratch with a custom kernel.
>
> Hopefully Gaohuai and Rongguang can help with this too.
>
> Paolo
>
> .
>
Yes, we are looking into and testing this.

I think this can result in any memory corruption, if VM1 writes its
PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
is not registered yet), then VM1 is destroyed (hence its PML buffer
is freed back to kernel), after that, VM2 starts migration, so CPU
logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.

As its severity, this commit (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
is eligible to back port to kernel stable.

[toc] | [next] | [standalone]

#1587458

From"Herongguang (Stephen)" <herongguang.he@huawei.com>
Date2017-02-24 10:40 +0100
Message-ID<tekw2-6Db-3@gated-at.bofh.it>
In reply to#1587244

On 2017/2/24 10:23, Herongguang (Stephen) wrote:
>
>
> On 2017/2/22 22:43, Paolo Bonzini wrote:
>>
>>
>> On 22/02/2017 14:31, Chris Friesen wrote:
>>>>>
>>>>
>>>> Can you reproduce it with kernel 4.8+?  I'm suspecting commmit
>>>> 4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
>>>> 2016-07-14) to be the fix.
>>>
>>> I can't easily try with a newer kernel, the software package we're using
>>> has kernel patches that would have to be ported.
>>>
>>> I'm at a conference, don't really have time to set up a pair of test
>>> machines from scratch with a custom kernel.
>>
>> Hopefully Gaohuai and Rongguang can help with this too.
>>
>> Paolo
>>
>> .
>>
> Yes, we are looking into and testing this.
>
> I think this can result in any memory corruption, if VM1 writes its
> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
> is not registered yet), then VM1 is destroyed (hence its PML buffer
> is freed back to kernel), after that, VM2 starts migration, so CPU
> logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.
>
> As its severity, this commit (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
> is eligible to back port to kernel stable.

Hi, Greg, can you cherry pick commit 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec to 4.4-y?

[toc] | [prev] | [next] | [standalone]

#1587488

FromGreg KH <gregkh@linuxfoundation.org>
Date2017-02-24 11:10 +0100
Message-ID<tekZ4-72W-13@gated-at.bofh.it>
In reply to#1587458
On Fri, Feb 24, 2017 at 11:00:32AM +0100, Paolo Bonzini wrote:
> 
> 
> On 24/02/2017 10:59, Greg KH wrote:
> > On Fri, Feb 24, 2017 at 05:35:17PM +0800, Herongguang (Stephen) wrote:
> >>
> >>
> >> On 2017/2/24 10:23, Herongguang (Stephen) wrote:
> >>>
> >>>
> >>> On 2017/2/22 22:43, Paolo Bonzini wrote:
> >>>>
> >>>>
> >>>> On 22/02/2017 14:31, Chris Friesen wrote:
> >>>>>>>
> >>>>>>
> >>>>>> Can you reproduce it with kernel 4.8+?  I'm suspecting commmit
> >>>>>> 4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
> >>>>>> 2016-07-14) to be the fix.
> >>>>>
> >>>>> I can't easily try with a newer kernel, the software package we're using
> >>>>> has kernel patches that would have to be ported.
> >>>>>
> >>>>> I'm at a conference, don't really have time to set up a pair of test
> >>>>> machines from scratch with a custom kernel.
> >>>>
> >>>> Hopefully Gaohuai and Rongguang can help with this too.
> >>>>
> >>>> Paolo
> >>>>
> >>>> .
> >>>>
> >>> Yes, we are looking into and testing this.
> >>>
> >>> I think this can result in any memory corruption, if VM1 writes its
> >>> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
> >>> is not registered yet), then VM1 is destroyed (hence its PML buffer
> >>> is freed back to kernel), after that, VM2 starts migration, so CPU
> >>> logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.
> >>>
> >>> As its severity, this commit (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
> >>> is eligible to back port to kernel stable.
> >>
> >> Hi, Greg, can you cherry pick commit 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec to 4.4-y?
> > 
> > If the KVM maintainers say it is ok to do so, yes, I will.
> 
> Yes, he beat me by minutes. :)

Heh, ok, I'll go add it to the recently-announced 4.4.52-rc1 release.

thanks,

greg k-h

[toc] | [prev] | [next] | [standalone]

#1587490

FromGreg KH <gregkh@linuxfoundation.org>
Date2017-02-24 11:10 +0100
Message-ID<tekZ4-72W-15@gated-at.bofh.it>
In reply to#1587458
On Fri, Feb 24, 2017 at 05:35:17PM +0800, Herongguang (Stephen) wrote:
> 
> 
> On 2017/2/24 10:23, Herongguang (Stephen) wrote:
> > 
> > 
> > On 2017/2/22 22:43, Paolo Bonzini wrote:
> > > 
> > > 
> > > On 22/02/2017 14:31, Chris Friesen wrote:
> > > > > > 
> > > > > 
> > > > > Can you reproduce it with kernel 4.8+?  I'm suspecting commmit
> > > > > 4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
> > > > > 2016-07-14) to be the fix.
> > > > 
> > > > I can't easily try with a newer kernel, the software package we're using
> > > > has kernel patches that would have to be ported.
> > > > 
> > > > I'm at a conference, don't really have time to set up a pair of test
> > > > machines from scratch with a custom kernel.
> > > 
> > > Hopefully Gaohuai and Rongguang can help with this too.
> > > 
> > > Paolo
> > > 
> > > .
> > > 
> > Yes, we are looking into and testing this.
> > 
> > I think this can result in any memory corruption, if VM1 writes its
> > PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
> > is not registered yet), then VM1 is destroyed (hence its PML buffer
> > is freed back to kernel), after that, VM2 starts migration, so CPU
> > logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.
> > 
> > As its severity, this commit (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
> > is eligible to back port to kernel stable.
> 
> Hi, Greg, can you cherry pick commit 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec to 4.4-y?

If the KVM maintainers say it is ok to do so, yes, I will.

thanks,

greg k-h

[toc] | [prev] | [next] | [standalone]

#1587491

FromPaolo Bonzini <pbonzini@redhat.com>
Date2017-02-24 11:10 +0100
Message-ID<tekZ4-72W-17@gated-at.bofh.it>
In reply to#1587490

On 24/02/2017 10:59, Greg KH wrote:
> On Fri, Feb 24, 2017 at 05:35:17PM +0800, Herongguang (Stephen) wrote:
>>
>>
>> On 2017/2/24 10:23, Herongguang (Stephen) wrote:
>>>
>>>
>>> On 2017/2/22 22:43, Paolo Bonzini wrote:
>>>>
>>>>
>>>> On 22/02/2017 14:31, Chris Friesen wrote:
>>>>>>>
>>>>>>
>>>>>> Can you reproduce it with kernel 4.8+?  I'm suspecting commmit
>>>>>> 4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
>>>>>> 2016-07-14) to be the fix.
>>>>>
>>>>> I can't easily try with a newer kernel, the software package we're using
>>>>> has kernel patches that would have to be ported.
>>>>>
>>>>> I'm at a conference, don't really have time to set up a pair of test
>>>>> machines from scratch with a custom kernel.
>>>>
>>>> Hopefully Gaohuai and Rongguang can help with this too.
>>>>
>>>> Paolo
>>>>
>>>> .
>>>>
>>> Yes, we are looking into and testing this.
>>>
>>> I think this can result in any memory corruption, if VM1 writes its
>>> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
>>> is not registered yet), then VM1 is destroyed (hence its PML buffer
>>> is freed back to kernel), after that, VM2 starts migration, so CPU
>>> logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.
>>>
>>> As its severity, this commit (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
>>> is eligible to back port to kernel stable.
>>
>> Hi, Greg, can you cherry pick commit 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec to 4.4-y?
> 
> If the KVM maintainers say it is ok to do so, yes, I will.

Yes, he beat me by minutes. :)

Paolo

[toc] | [prev] | [next] | [standalone]

#1587705

FromPaolo Bonzini <pbonzini@redhat.com>
Date2017-02-24 16:20 +0100
Message-ID<tepP3-1Yk-1@gated-at.bofh.it>
In reply to#1587244

On 24/02/2017 16:10, Chris Friesen wrote:
> On 02/23/2017 08:23 PM, Herongguang (Stephen) wrote:
> 
>> On 2017/2/22 22:43, Paolo Bonzini wrote:
> 
>>> Hopefully Gaohuai and Rongguang can help with this too.
>>>
>>> Paolo
>>
>> Yes, we are looking into and testing this.
>>
>> I think this can result in any memory corruption, if VM1 writes its
>> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
>> is not registered yet), then VM1 is destroyed (hence its PML buffer
>> is freed back to kernel), after that, VM2 starts migration, so CPU
>> logs VM2’s dirty GFNS into a freed memory, results in any memory
>> corruption.
>>
>> As its severity, this commit
>> (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
>>
>> is eligible to back port to kernel stable.
> 
> Are we expecting that fix to resolve the original issue, or is it a
> separate issue that needs fixing in stable?

It should be the original issue.

Paolo

[toc] | [prev] | [next] | [standalone]

#1588037

From"Herongguang (Stephen)" <herongguang.he@huawei.com>
Date2017-02-25 02:50 +0100
Message-ID<tezEK-t9-21@gated-at.bofh.it>
In reply to#1587705

On 2017/2/24 23:14, Paolo Bonzini wrote:
>
>
> On 24/02/2017 16:10, Chris Friesen wrote:
>> On 02/23/2017 08:23 PM, Herongguang (Stephen) wrote:
>>
>>> On 2017/2/22 22:43, Paolo Bonzini wrote:
>>
>>>> Hopefully Gaohuai and Rongguang can help with this too.
>>>>
>>>> Paolo
>>>
>>> Yes, we are looking into and testing this.
>>>
>>> I think this can result in any memory corruption, if VM1 writes its
>>> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
>>> is not registered yet), then VM1 is destroyed (hence its PML buffer
>>> is freed back to kernel), after that, VM2 starts migration, so CPU
>>> logs VM2’s dirty GFNS into a freed memory, results in any memory
>>> corruption.
>>>
>>> As its severity, this commit
>>> (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
>>>
>>> is eligible to back port to kernel stable.
>>
>> Are we expecting that fix to resolve the original issue, or is it a
>> separate issue that needs fixing in stable?
>
> It should be the original issue.
>
> Paolo
>
> .
>
Yes, I agree, though we are still testing.

[toc] | [prev] | [next] | [standalone]

#1588532

From"Huang, Kai" <kai.huang@linux.intel.com>
Date2017-02-27 08:50 +0100
Message-ID<tfoee-2mQ-3@gated-at.bofh.it>
In reply to#1588037

On 2/25/2017 2:44 PM, Herongguang (Stephen) wrote:
>
>
> On 2017/2/24 23:14, Paolo Bonzini wrote:
>>
>>
>> On 24/02/2017 16:10, Chris Friesen wrote:
>>> On 02/23/2017 08:23 PM, Herongguang (Stephen) wrote:
>>>
>>>> On 2017/2/22 22:43, Paolo Bonzini wrote:
>>>
>>>>> Hopefully Gaohuai and Rongguang can help with this too.
>>>>>
>>>>> Paolo
>>>>
>>>> Yes, we are looking into and testing this.
>>>>
>>>> I think this can result in any memory corruption, if VM1 writes its
>>>> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
>>>> is not registered yet), then VM1 is destroyed (hence its PML buffer
>>>> is freed back to kernel), after that, VM2 starts migration, so CPU
>>>> logs VM2’s dirty GFNS into a freed memory, results in any memory
>>>> corruption.
>>>>
>>>> As its severity, this commit
>>>> (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
>>>>
>>>>
>>>> is eligible to back port to kernel stable.
>>>
>>> Are we expecting that fix to resolve the original issue, or is it a
>>> separate issue that needs fixing in stable?
>>
>> It should be the original issue.
>>
>> Paolo
>>
>> .
>>
> Yes, I agree, though we are still testing.
>
>
Hi Stephen,

Sorry for late reply. I was taking the whole week off last week. How's 
the test going?

Thanks,
-Kai

[toc] | [prev] | [next] | [standalone]

#1587706

FromChris Friesen <chris.friesen@windriver.com>
Date2017-02-24 16:20 +0100
Message-ID<tepP3-1Yk-3@gated-at.bofh.it>
In reply to#1587244
On 02/23/2017 08:23 PM, Herongguang (Stephen) wrote:

> On 2017/2/22 22:43, Paolo Bonzini wrote:

>> Hopefully Gaohuai and Rongguang can help with this too.
>>
>> Paolo
>>
>> .
>>
> Yes, we are looking into and testing this.
>
> I think this can result in any memory corruption, if VM1 writes its
> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
> is not registered yet), then VM1 is destroyed (hence its PML buffer
> is freed back to kernel), after that, VM2 starts migration, so CPU
> logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.
>
> As its severity, this commit
> (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
>
> is eligible to back port to kernel stable.
>

Are we expecting that fix to resolve the original issue, or is it a separate 
issue that needs fixing in stable?

Chris

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web