Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1400349 > unrolled thread

transparent huge pages breaks KVM on AMD.

Started byMarc Haber <mh+linux-kernel@zugschlus.de>
First post2016-05-12 22:30 +0200
Last post2016-05-13 16:10 +0200
Articles 15 — 4 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  transparent huge pages breaks KVM on AMD. Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-05-12 22:30 +0200
    Re: transparent huge pages breaks KVM on AMD. "Kirill A. Shutemov" <kirill@shutemov.name> - 2016-05-12 22:30 +0200
      Re: transparent huge pages breaks KVM on AMD. Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-05-12 22:40 +0200
        Re: transparent huge pages breaks KVM on AMD. "Kirill A. Shutemov" <kirill@shutemov.name> - 2016-05-12 22:50 +0200
          Re: transparent huge pages breaks KVM on AMD. Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-05-13 07:30 +0200
            Re: transparent huge pages breaks KVM on AMD. Borislav Petkov <bp@alien8.de> - 2016-05-13 10:10 +0200
              Re: transparent huge pages breaks KVM on AMD. Borislav Petkov <bp@alien8.de> - 2016-05-13 10:20 +0200
                Re: transparent huge pages breaks KVM on AMD. Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-05-13 15:30 +0200
                  Re: transparent huge pages breaks KVM on AMD. Borislav Petkov <bp@alien8.de> - 2016-05-13 18:10 +0200
                Re: transparent huge pages breaks KVM on AMD. Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-05-14 08:30 +0200
              Re: transparent huge pages breaks KVM on AMD. Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-05-13 11:10 +0200
                Re: transparent huge pages breaks KVM on AMD. Borislav Petkov <bp@alien8.de> - 2016-05-13 11:20 +0200
              Re: transparent huge pages breaks KVM on AMD. Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-05-13 17:00 +0200
    Re: transparent huge pages breaks KVM on AMD. "Dr. David Alan Gilbert" <dave@treblig.org> - 2016-05-13 10:40 +0200
      Re: transparent huge pages breaks KVM on AMD. Marc Haber <mh+linux-kernel@zugschlus.de> - 2016-05-13 16:10 +0200

#1400349 — transparent huge pages breaks KVM on AMD.

FromMarc Haber <mh+linux-kernel@zugschlus.de>
Date2016-05-12 22:30 +0200
Subjecttransparent huge pages breaks KVM on AMD.
Message-ID<ry5p8-4Cj-3@gated-at.bofh.it>
Hi David,

On Sat, Apr 23, 2016 at 07:52:46PM +0100, Dr. David Alan Gilbert wrote:
> Hmm, your problem does sound like bad hardware, but....
> If you've got a nice reliable crash, can you try turning transparent huge pages
> off on the host;
>    echo never > /sys/kernel/mm/transparent_hugepage/enabled

I must have missed this hint in the middle of the "your hardware is
bad" avalance that came over me.

I spent two weeks bisecting "good" kernels since during the repeated
reconfigurations, transparent huge pages got turned off in kernel
configuration. After running each kernel for 24 hours, I eventually
ended up with a working 4.5 kernel. The configuration diff was short,
showing transparent huge pages, and - finally - upon re-reading the
thread I found your hint.

I have now the result that 4.5, 4.5.1 and 4.5.4 corrupt KVM guest
memory reliably in the first hour of running under disk load, causing
the VM to either drop dead in the water, or to read randomness from
disk. Rebooting fixes the VM. This happens as soon as transparent huge
pages are turned on in the host.

Turning off transparent huge pages by echo never >
/sys/kernel/mm/transparent_hugepage/enabled fixes the issue even
without rebooting the host. Start up the VM again and it works just
fine.

Is this an issue in (a) transparent huge pages, (b) KVM or (c) qemu?
Where should this issue be forwarded? Or do we just accept it and turn
transparent huge pages off?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [next] | [standalone]

#1400352

From"Kirill A. Shutemov" <kirill@shutemov.name>
Date2016-05-12 22:30 +0200
Message-ID<ry5p8-4Cj-13@gated-at.bofh.it>
In reply to#1400349
On Thu, May 12, 2016 at 10:20:09PM +0200, Marc Haber wrote:
> Hi David,
> 
> On Sat, Apr 23, 2016 at 07:52:46PM +0100, Dr. David Alan Gilbert wrote:
> > Hmm, your problem does sound like bad hardware, but....
> > If you've got a nice reliable crash, can you try turning transparent huge pages
> > off on the host;
> >    echo never > /sys/kernel/mm/transparent_hugepage/enabled
> 
> I must have missed this hint in the middle of the "your hardware is
> bad" avalance that came over me.
> 
> I spent two weeks bisecting "good" kernels since during the repeated
> reconfigurations, transparent huge pages got turned off in kernel
> configuration. After running each kernel for 24 hours, I eventually
> ended up with a working 4.5 kernel. The configuration diff was short,
> showing transparent huge pages, and - finally - upon re-reading the
> thread I found your hint.
> 
> I have now the result that 4.5, 4.5.1 and 4.5.4 corrupt KVM guest
> memory reliably in the first hour of running under disk load, causing
> the VM to either drop dead in the water, or to read randomness from
> disk. Rebooting fixes the VM. This happens as soon as transparent huge
> pages are turned on in the host.
> 
> Turning off transparent huge pages by echo never >
> /sys/kernel/mm/transparent_hugepage/enabled fixes the issue even
> without rebooting the host. Start up the VM again and it works just
> fine.
> 
> Is this an issue in (a) transparent huge pages, (b) KVM or (c) qemu?
> Where should this issue be forwarded? Or do we just accept it and turn
> transparent huge pages off?

Could you test this:

http://lkml.kernel.org/r/1463070742-18401-1-git-send-email-aarcange@redhat.com

?

-- 
 Kirill A. Shutemov

[toc] | [prev] | [next] | [standalone]

#1400359

FromMarc Haber <mh+linux-kernel@zugschlus.de>
Date2016-05-12 22:40 +0200
Message-ID<ry5yO-4H0-15@gated-at.bofh.it>
In reply to#1400352
On Thu, May 12, 2016 at 11:24:02PM +0300, Kirill A. Shutemov wrote:
> http://lkml.kernel.org/r/1463070742-18401-1-git-send-email-aarcange@redhat.com

Is this in v4.6-rc7?

If so, can I just test v4.6-rc7?

If not so, would it be a valid approach to first check plain v4.6-rc7
and then patched v4.6-rc7?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1400363

From"Kirill A. Shutemov" <kirill@shutemov.name>
Date2016-05-12 22:50 +0200
Message-ID<ry5It-4Oz-1@gated-at.bofh.it>
In reply to#1400359
On Thu, May 12, 2016 at 10:34:57PM +0200, Marc Haber wrote:
> On Thu, May 12, 2016 at 11:24:02PM +0300, Kirill A. Shutemov wrote:
> > http://lkml.kernel.org/r/1463070742-18401-1-git-send-email-aarcange@redhat.com
> 
> Is this in v4.6-rc7?

No.

> If so, can I just test v4.6-rc7?
> 
> If not so, would it be a valid approach to first check plain v4.6-rc7
> and then patched v4.6-rc7?

It haven't hit -mm tree yet.

But I guess it should apply cleanly to v4.5. Or at least without major
conflicts.

Andrew, it looks like third bug report for the bug. It would be nice to
propogate it quicker, if possible.

-- 
 Kirill A. Shutemov

[toc] | [prev] | [next] | [standalone]

#1400501

FromMarc Haber <mh+linux-kernel@zugschlus.de>
Date2016-05-13 07:30 +0200
Message-ID<rydPI-4W7-5@gated-at.bofh.it>
In reply to#1400363
On Thu, May 12, 2016 at 11:42:16PM +0300, Kirill A. Shutemov wrote:
> But I guess it should apply cleanly to v4.5. Or at least without major
> conflicts.

[11/511]mh@fan:~/linux/debug/linux$ curl 'http://marc.info/?l=linux-rdma&m=146307074800836&w=2' | patch -p1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12529    0 12529    0     0   9844      0 --:--:--  0:00:01 --:--:--  9849
patching file include/linux/mm.h
Hunk #1 succeeded at 456 with fuzz 1 (offset -44 lines).
patching file include/linux/swap.h
Hunk #2 FAILED at 513.
1 out of 2 hunks FAILED -- saving rejects to file include/linux/swap.h.rej
patching file mm/huge_memory.c
Hunk #1 FAILED at 1298.
Hunk #2 FAILED at 2079.
Hunk #3 succeeded at 3340 (offset 117 lines).
2 out of 3 hunks FAILED -- saving rejects to file mm/huge_memory.c.rej
patching file mm/memory.c
Hunk #1 FAILED at 2373.
Hunk #2 succeeded at 2331 with fuzz 2 (offset -56 lines).
Hunk #3 FAILED at 2622.
2 out of 3 hunks FAILED -- saving rejects to file mm/memory.c.rej
patching file mm/swapfile.c
Hunk #1 FAILED at 922.
1 out of 1 hunk FAILED -- saving rejects to file mm/swapfile.c.rej
[12/512]mh@fan:~/linux/debug/linux$

It doesn't, and it doesn't apply to 4.6-rc3 as well:

[17/517]mh@fan:~/linux/debug/linux$ git checkout v4.6-rc3
Checking out files: 100% (9945/9945), done.
Previous HEAD position was b562e44... Linux 4.5
HEAD is now at bf16200... Linux 4.6-rc3
[18/518]mh@fan:~/linux/debug/linux$ curl 'http://marc.info/?l=linux-rdma&m=146307074800836&w=2' | patch -p1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12529    0 12529    0     0   9692      0 --:--:--  0:00:01 --:--:--  9697
patching file include/linux/mm.h
patching file include/linux/swap.h
Hunk #2 FAILED at 513.
1 out of 2 hunks FAILED -- saving rejects to file include/linux/swap.h.rej
patching file mm/huge_memory.c
Hunk #1 FAILED at 1298.
Hunk #2 FAILED at 2079.
Hunk #3 succeeded at 3225 (offset 2 lines).
2 out of 3 hunks FAILED -- saving rejects to file mm/huge_memory.c.rej
patching file mm/memory.c
Hunk #1 FAILED at 2373.
Hunk #2 succeeded at 2354 (offset -33 lines).
Hunk #3 FAILED at 2622.
2 out of 3 hunks FAILED -- saving rejects to file mm/memory.c.rej
patching file mm/swapfile.c
Hunk #1 FAILED at 922.
1 out of 1 hunk FAILED -- saving rejects to file mm/swapfile.c.rej
[19/519]mh@fan:~/linux/debug/linux$

How do I apply this?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1400589

FromBorislav Petkov <bp@alien8.de>
Date2016-05-13 10:10 +0200
Message-ID<rygky-7Xv-25@gated-at.bofh.it>
In reply to#1400501

[Multipart message — attachments visible in raw view] — view raw

On Fri, May 13, 2016 at 07:23:34AM +0200, Marc Haber wrote:
> How do I apply this?

I'm attaching it.

$ patch -p1 --dry-run -i /tmp/01-mm-thp-calculate_the_mapcount_correctly_for_thp_pages_during_wp_faults.patch
checking file include/linux/mm.h
checking file include/linux/swap.h
checking file mm/huge_memory.c
checking file mm/memory.c
checking file mm/swapfile.c
$ patch -p1 -i /tmp/01-mm-thp-calculate_the_mapcount_correctly_for_thp_pages_during_wp_faults.patch
patching file include/linux/mm.h
patching file include/linux/swap.h
patching file mm/huge_memory.c
patching file mm/memory.c
patching file mm/swapfile.c

The --dry-run is to check whether it applies first.

That's on 4.6-rc7+ here.

HTH.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1400601

FromBorislav Petkov <bp@alien8.de>
Date2016-05-13 10:20 +0200
Message-ID<rygue-822-11@gated-at.bofh.it>
In reply to#1400589

[Multipart message — attachments visible in raw view] — view raw

Try this one better - it fixes an unitialized var.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1400794

FromMarc Haber <mh+linux-kernel@zugschlus.de>
Date2016-05-13 15:30 +0200
Message-ID<rylke-41J-11@gated-at.bofh.it>
In reply to#1400601
On Fri, May 13, 2016 at 10:09:52AM +0200, Borislav Petkov wrote:
> Try this one better - it fixes an unitialized var.

Instead, or in addiiton?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1400878

FromBorislav Petkov <bp@alien8.de>
Date2016-05-13 18:10 +0200
Message-ID<rynP4-70e-19@gated-at.bofh.it>
In reply to#1400794
On Fri, May 13, 2016 at 03:21:56PM +0200, Marc Haber wrote:
> Instead, or in addiiton?

Instead. You'll notice that it doesn't apply if you try "in addition".

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1401067

FromMarc Haber <mh+linux-kernel@zugschlus.de>
Date2016-05-14 08:30 +0200
Message-ID<ryBfk-38X-15@gated-at.bofh.it>
In reply to#1400601
On Fri, May 13, 2016 at 10:09:52AM +0200, Borislav Petkov wrote:
> Try this one better - it fixes an unitialized var.

Nosireebob, VMs crash even with this patch in the host as soon as the
host has THP enabled.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1400628

FromMarc Haber <mh+linux-kernel@zugschlus.de>
Date2016-05-13 11:10 +0200
Message-ID<ryhgC-qB-11@gated-at.bofh.it>
In reply to#1400589
On Fri, May 13, 2016 at 10:07:45AM +0200, Borislav Petkov wrote:
> On Fri, May 13, 2016 at 07:23:34AM +0200, Marc Haber wrote:
> > How do I apply this?
> 
> I'm attaching it.

Ok, stupid me, I thought that one could simply curl the web page. Too
bad that list archives keep mangling patches :-(

It applies now to 4.5 as well.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1400634

FromBorislav Petkov <bp@alien8.de>
Date2016-05-13 11:20 +0200
Message-ID<ryhqi-uZ-17@gated-at.bofh.it>
In reply to#1400628
On Fri, May 13, 2016 at 11:08:46AM +0200, Marc Haber wrote:
> It applies now to 4.5 as well.

Yeah, I tried getting the raw message from marc.info but then it said:

patch unexpectedly ends in middle of line
Hunk #1 succeeded at 922 with fuzz 1.

The attached versions I sent you are from my lkml mbox - the only reason
I keep it :-)

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[toc] | [prev] | [next] | [standalone]

#1400845

FromMarc Haber <mh+linux-kernel@zugschlus.de>
Date2016-05-13 17:00 +0200
Message-ID<rymJk-5r8-15@gated-at.bofh.it>
In reply to#1400589
On Fri, May 13, 2016 at 10:07:45AM +0200, Borislav Petkov wrote:
> On Fri, May 13, 2016 at 07:23:34AM +0200, Marc Haber wrote:
> > How do I apply this?
> 
> I'm attaching it.

Had the VM crashing twice with this patch applied, THP==madvise on the
host and THP==never in the VM.

Now trying the other patch, assuming that it's intended to be used
_instead_ of this one.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [next] | [standalone]

#1400613

From"Dr. David Alan Gilbert" <dave@treblig.org>
Date2016-05-13 10:40 +0200
Message-ID<rygNA-8e3-9@gated-at.bofh.it>
In reply to#1400349
* Marc Haber (mh+linux-kernel@zugschlus.de) wrote:
> Hi David,
> 
> On Sat, Apr 23, 2016 at 07:52:46PM +0100, Dr. David Alan Gilbert wrote:
> > Hmm, your problem does sound like bad hardware, but....
> > If you've got a nice reliable crash, can you try turning transparent huge pages
> > off on the host;
> >    echo never > /sys/kernel/mm/transparent_hugepage/enabled
> 
> I must have missed this hint in the middle of the "your hardware is
> bad" avalance that came over me.
> 
> I spent two weeks bisecting "good" kernels since during the repeated
> reconfigurations, transparent huge pages got turned off in kernel
> configuration. After running each kernel for 24 hours, I eventually
> ended up with a working 4.5 kernel. The configuration diff was short,
> showing transparent huge pages, and - finally - upon re-reading the
> thread I found your hint.

OK, good.  When I sent that mail I'd hit a THP bug but in a
corner of migration and at the time we didn't know why and there was
no reason to think it would cause any other symptoms, but since it was
also between 4.4 and 4.5 it did seem worth mentioning as a long shot,
but it was no more than a long shot.

> I have now the result that 4.5, 4.5.1 and 4.5.4 corrupt KVM guest
> memory reliably in the first hour of running under disk load, causing
> the VM to either drop dead in the water, or to read randomness from
> disk. Rebooting fixes the VM. This happens as soon as transparent huge
> pages are turned on in the host.
> 
> Turning off transparent huge pages by echo never >
> /sys/kernel/mm/transparent_hugepage/enabled fixes the issue even
> without rebooting the host. Start up the VM again and it works just
> fine.
> 
> Is this an issue in (a) transparent huge pages, (b) KVM or (c) qemu?
> Where should this issue be forwarded? Or do we just accept it and turn
> transparent huge pages off?

Try Andrea's fix for (a).

Dave

> 
> Greetings
> Marc
> 
> -- 
> -----------------------------------------------------------------------------
> Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
> Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
> Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
-- 
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    |       Running GNU/Linux       | Happy  \ 
\        dave @ treblig.org |                               | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/

[toc] | [prev] | [next] | [standalone]

#1400811

FromMarc Haber <mh+linux-kernel@zugschlus.de>
Date2016-05-13 16:10 +0200
Message-ID<rylWW-4R7-5@gated-at.bofh.it>
In reply to#1400613
On Fri, May 13, 2016 at 09:35:45AM +0100, Dr. David Alan Gilbert wrote:
> also between 4.4 and 4.5 it did seem worth mentioning as a long shot,
> but it was no more than a long shot.

It was however helpful. I'd have bisected kernel configuration instead
of using the runtime control first, and seeing your long shot two
weeks earlier, it'd have saved myself those two weeks of tedious
bisecting.

> Try Andrea's fix for (a).

In the works.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web