Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1383938 > unrolled thread

Re: Major KVM issues with kernel 4.5 on the host

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-21 10:50 +0200
    Re: Major KVM issues with kernel 4.5 on the host Borislav Petkov <bp@alien8.de> - 2016-04-21 14:40 +0200
      Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-21 17:00 +0200
        Re: Major KVM issues with kernel 4.5 on the host Borislav Petkov <bp@alien8.de> - 2016-04-21 19:00 +0200
          Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-21 22:10 +0200
            Re: Major KVM issues with kernel 4.5 on the host Borislav Petkov <bp@alien8.de> - 2016-04-23 18:10 +0200
              Re: Major KVM issues with kernel 4.5 on the host Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-04-23 20:50 +0200
                Re: Major KVM issues with kernel 4.5 on the host "Dr. David Alan Gilbert" <dave@treblig.org> - 2016-04-23 21:20 +0200
                Re: Major KVM issues with kernel 4.5 on the host Borislav Petkov <bp@alien8.de> - 2016-04-24 02:00 +0200

#1383938 — Re: Major KVM issues with kernel 4.5 on the host

On Thu, Apr 14, 2016 at 07:22:20AM +0200, Marc Haber wrote:
> On Thu, Apr 14, 2016 at 03:16:29AM +0200, Paolo Bonzini wrote:
> > On 14/04/2016 00:29, Marc Haber wrote:
> > > On Wed, Apr 13, 2016 at 10:36:34PM +0200, Paolo Bonzini wrote:
> > >> Didn't help, but a fresh look at the list of 4.5 patches helped.
> > >> What the hell was I thinking, I missed write_rdtscp_aux who
> > >> obviously uses MSR_TSC_AUX.
> > > 
> > > I applied this patch to 4.5, which didn't go cleanly, I had to do it
> > > manually, and there is no change in behavior. Sometimes, the Vm just
> > > crashes, but most times the filesystem is remounted ro.
> > 
> > Ok, then I guess bisection is needed.  Please first try commit
> > 45bdbcfdf241.  If it fails, then the bug come together with KVM's merge
> > window changes for 4.5-rc1.  Please apply the patch I sent here when
> > bisection is past 46896c73c1a4dde527c3a3cc43379deeb41985a1 (which means
> > that probably that should be the commit you try second; the bisection
> > then becomes much easier).
> 
> I have never bisected this deeply. Can you please give more advice,
> with which two commits to start? And how do I find out whether I am
> "past" a commit? I am als not a git expert, a few command lines would
> be appreciated.

I have tried bisecting, and finally bisect says that the bad commit is
0e749e54244eec87b2a3cd0a4314e60bc6781115 dax: increase granularity of dax_clear_blocks() operations

However, a kernel built after
$ git checkout 0e749e54244eec87b2a3cd0a4314e60bc6781115
seems to be fine, at least my VM is running for 15 hours now.

I guess I need to start over again with git bisect good
0e749e54244eec87b2a3cd0a4314e60bc6781115 and git bisect bad v4.5.

Currently, I cannot explain how this has happened, I must have flagged
an actually good kernel as bad from my understanding of git bisect.

Can you give advice how to continue here?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [next] | [standalone]

#1384161

On Thu, Apr 21, 2016 at 10:39:48AM +0200, Marc Haber wrote:
> Currently, I cannot explain how this has happened, I must have flagged
> an actually good kernel as bad from my understanding of git bisect.
> 
> Can you give advice how to continue here?

Yap, sounds like you marked a bisection step incorrectly, which lead
into the wrong direction. How reliable is your reproducer?

Also, do the bisection as Paolo suggested:

* try 45bdbcfdf241.

* then do

$ git bisect start v4.5-rc1 v4.4

which marks -rc1 as bad and 4.4 as good.

While you're doing that bisect, do what Paolo said by applying the diff
here

  https://lkml.kernel.org/r/570EADD2.8030300@redhat.com

when the bisection point you're at at each step contains

  46896c73c1a4 ("KVM: svm: add support for RDTSCP")

You should apply the above hunk by doing

$ patch -p1 --dry-run -i /tmp/hunk

If it applies fine, you then apply it

$ patch -p1 -i /tmp/hunk

All clear?

If not, do not hesitate to ask.

Thanks

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1384321

On Thu, Apr 21, 2016 at 02:37:11PM +0200, Borislav Petkov wrote:
> On Thu, Apr 21, 2016 at 10:39:48AM +0200, Marc Haber wrote:
> > Currently, I cannot explain how this has happened, I must have flagged
> > an actually good kernel as bad from my understanding of git bisect.
> > 
> > Can you give advice how to continue here?
> 
> Yap, sounds like you marked a bisection step incorrectly, which lead
> into the wrong direction. How reliable is your reproducer?

Usually, the crash or filesystem corruption happens in the first 15 to
30 minutes. I have had one instance running three hours before
corrupting, I have therefore upped the run time to nine hours before
saying "this kernel is good".

What bothers me is that since I ended up with a "suspect" commit that
actually results in a "good" kernel (running for 22 hours now), I must
have said "bad" to an actually "good" kernel, which means that I had
an unrelated crash or corruption. Is that reasoning correct?

> Also, do the bisection as Paolo suggested:
> 
> * try 45bdbcfdf241.

That one qualified as "good" six days ago. I'll retry, maybe I just
didn't wait long enough.

"Trying" means make oldconfig, make deb-pkg in my case right? Does it
matter what I answer to the numerous config questions that keep coming
up during the oldconfig step?

> * then do
> 
> $ git bisect start v4.5-rc1 v4.4
> 
> which marks -rc1 as bad and 4.4 as good.

Would it help to explicitly mark
0e749e54244eec87b2a3cd0a4314e60bc6781115 as good so that the knowledge
gained during the last week is not completely lost?

> While you're doing that bisect, do what Paolo said by applying the diff
> here
> 
>   https://lkml.kernel.org/r/570EADD2.8030300@redhat.com
> 
> when the bisection point you're at at each step contains
> 
>   46896c73c1a4 ("KVM: svm: add support for RDTSCP")
> 
> You should apply the above hunk by doing
> 
> $ patch -p1 --dry-run -i /tmp/hunk
> 
> If it applies fine, you then apply it
> 
> $ patch -p1 -i /tmp/hunk
> 
> All clear?

So I need to git log | grep 46896c73c1a4 and apply the patch again
each time the commit is found?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1384431

On Thu, Apr 21, 2016 at 04:50:05PM +0200, Marc Haber wrote:
> What bothers me is that since I ended up with a "suspect" commit that
> actually results in a "good" kernel (running for 22 hours now), I must
> have said "bad" to an actually "good" kernel, which means that I had
> an unrelated crash or corruption. Is that reasoning correct?

Hmm, did that "unrelated crash or corruption" have the same symptoms as
the original one?

> That one qualified as "good" six days ago. I'll retry, maybe I just
> didn't wait long enough.

So if the trigger time is varying so much, I'd try to double that to
make sure I'm fairly certain about each commit I'm testing.

Also, this is a single box we're talking about, right? And you're sure
it hasn't had any corruption issues so far?

I see you have amd64_edac loading, so it must have ECC DIMMs. Have you
had any reports in the past of ECC errors in dmesg? Or other MCEs,
lockups, etc? Can you grep your logs for stuff like "hardware error",
"mce", "edac" etc? Do a case-insensitive search.

> "Trying" means make oldconfig, make deb-pkg in my case right? Does it
> matter what I answer to the numerous config questions that keep coming
> up during the oldconfig step?

What I do is:

$ git bisect <good|bad>

to mark the current commit after having tested it. Then I do

$ yes "" | make oldconfig

to set the new config options. Then

$ make -j7
$ make modules_install install

and reboot into the new kernel. Kernel name will possibly change each
time so I write down on paper which kernel I'm testing. You can verify
when booting it by doing:

$ dmesg | head
[    0.000000] Linux version 4.6.0-rc2+ (boris@pd) (gcc version 5.3.1 20160101 (Debian 5.3.1-5) ) #1 SMP PREEMPT Wed Apr 6 20:22:51 CEST 2016
...

that date at the end of the line and number "#1" should be current.
Number is also in .version and gets issued when you finish building:

Kernel: arch/x86/boot/bzImage is ready  (#1)

> Would it help to explicitly mark
> 0e749e54244eec87b2a3cd0a4314e60bc6781115 as good so that the knowledge
> gained during the last week is not completely lost?

I'd do the whole thing again, just to be sure.

I know, bisection is very time-consuming :-\ And it is particularly
annoying if it is done on the box I'm normally using daily.

> So I need to git log | grep 46896c73c1a4 and apply the patch again
> each time the commit is found?

I think you can let git do that for ya:

$ git branch --contains 46896c73c1a4
* (HEAD detached at 46896c73c1a4)

that lists that the current checked out HEAD contains that commit. If you do

$ git checkout 46896c73c1a4~1

then that "(HEAD detached..." line is not in the list of branches
containing it.

HTH.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1384545

On Thu, Apr 21, 2016 at 06:51:06PM +0200, Borislav Petkov wrote:
> On Thu, Apr 21, 2016 at 04:50:05PM +0200, Marc Haber wrote:
> > What bothers me is that since I ended up with a "suspect" commit that
> > actually results in a "good" kernel (running for 22 hours now), I must
> > have said "bad" to an actually "good" kernel, which means that I had
> > an unrelated crash or corruption. Is that reasoning correct?
> 
> Hmm, did that "unrelated crash or corruption" have the same symptoms as
> the original one?

Yes, but there are two symptoms. The VM either suffers file system
issues (garbage read from files, or an aborted ext4 journal and
following ro remount) or it stops dead in its tracks.


> > That one qualified as "good" six days ago. I'll retry, maybe I just
> > didn't wait long enough.
> 
> So if the trigger time is varying so much, I'd try to double that to
> make sure I'm fairly certain about each commit I'm testing.

The longest trigger time I have seen was three hours, I tripled that
to nine hours, that probably was not enough.

> Also, this is a single box we're talking about, right? And you're sure
> it hasn't had any corruption issues so far?

It is a single box, and it runs perfectly with kernel 4.4.

> I see you have amd64_edac loading, so it must have ECC DIMMs. Have you
> had any reports in the past of ECC errors in dmesg? Or other MCEs,
> lockups, etc? Can you grep your logs for stuff like "hardware error",
> "mce", "edac" etc? Do a case-insensitive search.

The box reports about one correctable error per week, so I probably
have a faulty DIMM, but since the issue only surfaces in VMs while the
host system is in perfect working order...

And yes, I am pondering to simply replace the box with an Intel CPU.

I see "mce: CPU supports 6 MCE banks" once for each reboot, and about
30 "Machine check events logged" since January. How do I see which
events were logged?

> > "Trying" means make oldconfig, make deb-pkg in my case right? Does it
> > matter what I answer to the numerous config questions that keep coming
> > up during the oldconfig step?
> 
> What I do is:
> 
> $ git bisect <good|bad>
> 
> to mark the current commit after having tested it. Then I do
> 
> $ yes "" | make oldconfig
> 
> to set the new config options.

So you basically select the default for new options.

>  Then
> 
> $ make -j7
> $ make modules_install install
> 
> and reboot into the new kernel. Kernel name will possibly change each
> time so I write down on paper which kernel I'm testing.

I go the way of Debian packages since it is easier to handle the
crypto file systems when the machine is booting up.

And yes, I think about doing a test reinstall on unencrypted disk to
find out whether encryption plays a role, but I currently need the
machine to urgently to take it out of serice for half a month, and,
again, the host system is in perfect working order, it is just VMs
that barf.

>  You can verify when booting it by doing:
> 
> $ dmesg | head
> [    0.000000] Linux version 4.6.0-rc2+ (boris@pd) (gcc version 5.3.1 20160101 (Debian 5.3.1-5) ) #1 SMP PREEMPT Wed Apr 6 20:22:51 CEST 2016
> ...
> 
> that date at the end of the line and number "#1" should be current.

I check the date of the package I am installing and the date stamp of
the kernels being installed to /boot. I'm reasonably sure I have that
under control.

> > Would it help to explicitly mark
> > 0e749e54244eec87b2a3cd0a4314e60bc6781115 as good so that the knowledge
> > gained during the last week is not completely lost?
> 
> I'd do the whole thing again, just to be sure.
> 
> I know, bisection is very time-consuming :-\ And it is particularly
> annoying if it is done on the box I'm normally using daily.

... and if testing a "good" kernel means a day.

> > So I need to git log | grep 46896c73c1a4 and apply the patch again
> > each time the commit is found?
> 
> I think you can let git do that for ya:
> 
> $ git branch --contains 46896c73c1a4
> * (HEAD detached at 46896c73c1a4)
> 
> that lists that the current checked out HEAD contains that commit. If you do
> 
> $ git checkout 46896c73c1a4~1
> 
> then that "(HEAD detached..." line is not in the list of branches
> containing it.

And whenever 46896c73c1a4 is present, I need to apply Paolo's patch,
right?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1385683

On Thu, Apr 21, 2016 at 10:04:33PM +0200, Marc Haber wrote:
> Yes, but there are two symptoms. The VM either suffers file system
> issues (garbage read from files, or an aborted ext4 journal and
> following ro remount) or it stops dead in its tracks.

Stops dead? What does that mean exactly? Box is wedged solid and it
doesn't react to any key presses?

Because if so, this could really be a DRAM going bad and a correctable
error turning into an uncorrectable. How old is the DRAM in that box?
Judging by your CPU, it should be a couple of years...

> The longest trigger time I have seen was three hours, I tripled that
> to nine hours, that probably was not enough.

So enlarge even more I guess.

> The box reports about one correctable error per week, so I probably
> have a faulty DIMM, but since the issue only surfaces in VMs while the
> host system is in perfect working order...

So it could be that correctable error turns into an uncorrectable one at
some point. But then you should be getting an exception...

> And yes, I am pondering to simply replace the box with an Intel CPU.

Your CPU is fine, from what I've seen so far.

> I see "mce: CPU supports 6 MCE banks" once for each reboot, and about
> 30 "Machine check events logged" since January. How do I see which
> events were logged?

Hmm, you have

[   18.149300] MCE: In-kernel MCE decoding enabled.

that's CONFIG_EDAC_DECODE_MCE, so you should have some "Hardware Error"
lines in dmesg, I'd guess, decoding the errors.

> So you basically select the default for new options.

Yap.

> I go the way of Debian packages since it is easier to handle the
> crypto file systems when the machine is booting up.

As long as you're testing the correct bisection kernels...

> And yes, I think about doing a test reinstall on unencrypted disk to
> find out whether encryption plays a role, but I currently need the
> machine to urgently to take it out of serice for half a month, and,
> again, the host system is in perfect working order, it is just VMs
> that barf.

Yeah, I can't reproduce it here and I have a very similar box to yours
which is otherwise idle, more or less.

Another fact which points to potentially DIMM going bad...

> I check the date of the package I am installing and the date stamp of
> the kernels being installed to /boot. I'm reasonably sure I have that
> under control.

Good.

> ... and if testing a "good" kernel means a day.

Yeah, it is annoying. In a perfect world, we all should have two
identical boxes so that we use one as a workstation and the second for
testing when the first one, the workstation barfs. I should bring that
up with my manager next time... :-)

> And whenever 46896c73c1a4 is present, I need to apply Paolo's patch,
> right?

Yap.

Thanks.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1385704

On Sat, Apr 23, 2016 at 06:04:29PM +0200, Borislav Petkov wrote:
> On Thu, Apr 21, 2016 at 10:04:33PM +0200, Marc Haber wrote:
> > Yes, but there are two symptoms. The VM either suffers file system
> > issues (garbage read from files, or an aborted ext4 journal and
> > following ro remount) or it stops dead in its tracks.
> 
> Stops dead? What does that mean exactly? Box is wedged solid and it
> doesn't react to any key presses?

No ping, no reaction on serial console, no reaction on virtual
console, no syslog entries.

> Because if so, this could really be a DRAM going bad and a correctable
> error turning into an uncorrectable. How old is the DRAM in that box?
> Judging by your CPU, it should be a couple of years...

Uncorrectable errors would still be identified by the ECC hardware,
and the box wouldn't be perfectly fine with an "old" kernel.

> > The box reports about one correctable error per week, so I probably
> > have a faulty DIMM, but since the issue only surfaces in VMs while the
> > host system is in perfect working order...
> 
> So it could be that correctable error turns into an uncorrectable one at
> some point. But then you should be getting an exception...

Yes, that would be in the logs.

> > And yes, I am pondering to simply replace the box with an Intel CPU.
> 
> Your CPU is fine, from what I've seen so far.

But we still postulate that the issue does only show on older AMD
CPUs. Otherwise, I wouldn't be the only one making this experience.

> > I go the way of Debian packages since it is easier to handle the
> > crypto file systems when the machine is booting up.
> 
> As long as you're testing the correct bisection kernels...

I am reasonably sure about that, yes.

> > And yes, I think about doing a test reinstall on unencrypted disk to
> > find out whether encryption plays a role, but I currently need the
> > machine to urgently to take it out of serice for half a month, and,
> > again, the host system is in perfect working order, it is just VMs
> > that barf.
> 
> Yeah, I can't reproduce it here and I have a very similar box to yours
> which is otherwise idle, more or less.
> 
> Another fact which points to potentially DIMM going bad...

Do you want me to memtest for 24 hours?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1385712

* Marc Haber (mh+linux-kernel@zugschlus.de) wrote:
> On Sat, Apr 23, 2016 at 06:04:29PM +0200, Borislav Petkov wrote:
> > On Thu, Apr 21, 2016 at 10:04:33PM +0200, Marc Haber wrote:
> > > Yes, but there are two symptoms. The VM either suffers file system
> > > issues (garbage read from files, or an aborted ext4 journal and
> > > following ro remount) or it stops dead in its tracks.
> > 
> > Stops dead? What does that mean exactly? Box is wedged solid and it
> > doesn't react to any key presses?
> 
> No ping, no reaction on serial console, no reaction on virtual
> console, no syslog entries.
> 
> > Because if so, this could really be a DRAM going bad and a correctable
> > error turning into an uncorrectable. How old is the DRAM in that box?
> > Judging by your CPU, it should be a couple of years...
> 
> Uncorrectable errors would still be identified by the ECC hardware,
> and the box wouldn't be perfectly fine with an "old" kernel.

Hmm, your problem does sound like bad hardware, but....
If you've got a nice reliable crash, can you try turning transparent huge pages
off on the host;
   echo never > /sys/kernel/mm/transparent_hugepage/enabled

Dave

> > > The box reports about one correctable error per week, so I probably
> > > have a faulty DIMM, but since the issue only surfaces in VMs while the
> > > host system is in perfect working order...
> > 
> > So it could be that correctable error turns into an uncorrectable one at
> > some point. But then you should be getting an exception...
> 
> Yes, that would be in the logs.
> 
> > > And yes, I am pondering to simply replace the box with an Intel CPU.
> > 
> > Your CPU is fine, from what I've seen so far.
> 
> But we still postulate that the issue does only show on older AMD
> CPUs. Otherwise, I wouldn't be the only one making this experience.
> 
> > > I go the way of Debian packages since it is easier to handle the
> > > crypto file systems when the machine is booting up.
> > 
> > As long as you're testing the correct bisection kernels...
> 
> I am reasonably sure about that, yes.
> 
> > > And yes, I think about doing a test reinstall on unencrypted disk to
> > > find out whether encryption plays a role, but I currently need the
> > > machine to urgently to take it out of serice for half a month, and,
> > > again, the host system is in perfect working order, it is just VMs
> > > that barf.
> > 
> > Yeah, I can't reproduce it here and I have a very similar box to yours
> > which is otherwise idle, more or less.
> > 
> > Another fact which points to potentially DIMM going bad...
> 
> Do you want me to memtest for 24 hours?
> 
> Greetings
> Marc
> 
> -- 
> -----------------------------------------------------------------------------
> Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
> Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
> Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
-- 
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    |       Running GNU/Linux       | Happy  \ 
\        dave @ treblig.org |                               | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/

[toc] | [prev] | [next] | [standalone]

#1385756

On Sat, Apr 23, 2016 at 08:43:41PM +0200, Marc Haber wrote:
> Uncorrectable errors would still be identified by the ECC hardware,

Not if the hardware decides to syncflood so that we don't even get to
run the #MC handler...

> and the box wouldn't be perfectly fine with an "old" kernel.

Maybe the "old" kernel is not causing all the required ingredients to
come together for the uncorrectable error to happen. But yeah, I agree,
the fact that 4.4 is fine kinda doesn't fit with the uncorrectable error
theory.

> Yes, that would be in the logs.

Presumably. And see above.

> But we still postulate that the issue does only show on older AMD
> CPUs. Otherwise, I wouldn't be the only one making this experience.

It actually shows only on this one system. At least I'm not aware of any
other report of the same issue. My system with a F10h, rev E is just
fine.

> Do you want me to memtest for 24 hours?

Yeah, that memtest crap never triggers any ECCs. But if you're bored,
why not...

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web