Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1619123 > unrolled thread

Re: KASLR causes intermittent boot failures on some systems

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: KASLR causes intermittent boot failures on some systems Kees Cook <keescook@chromium.org> - 2017-04-07 23:30 +0200
    Re: KASLR causes intermittent boot failures on some systems Jeff Moyer <jmoyer@redhat.com> - 2017-04-10 18:00 +0200
      Re: KASLR causes intermittent boot failures on some systems Kees Cook <keescook@chromium.org> - 2017-04-10 20:20 +0200
        Re: KASLR causes intermittent boot failures on some systems Jeff Moyer <jmoyer@redhat.com> - 2017-04-10 20:30 +0200
          Re: KASLR causes intermittent boot failures on some systems Kees Cook <keescook@chromium.org> - 2017-04-10 21:10 +0200
            Re: KASLR causes intermittent boot failures on some systems Jeff Moyer <jmoyer@redhat.com> - 2017-04-10 21:20 +0200

#1619123 — Re: KASLR causes intermittent boot failures on some systems

On Fri, Apr 7, 2017 at 7:41 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
> Hi,
>
> commit 021182e52fe01 ("x86/mm: Enable KASLR for physical mapping memory
> regions") causes some of my systems with persistent memory (whether real
> or emulated) to fail to boot with a couple of different crash
> signatures.  The first signature is a NMI watchdog lockup of all but 1
> cpu, which causes much difficulty in extracting useful information from
> the console.  The second variant is an invalid paging request, listed
> below.

Just to rule out some of the stuff in the boot path, does booting with
"nokaslr" solve this? (i.e. I want to figure out if this is from some
of the rearrangements done that are exposed under that commit, or if
it is genuinely the randomization that is killing the systems...)

> On some systems, I haven't hit this problem at all.  Other systems
> experience a failed boot maybe 20-30% of the time.  To reproduce it,
> configure some emulated pmem on your system.  You can find directions
> for that here: https://nvdimm.wiki.kernel.org/
>
> Install ndctl (https://github.com/pmem/ndctl).
> Configure the namespace:
> # ndctl create-namespace -f -e namespace0.0 -m memory
>
> Then just reboot several times (5 should be enough), and hopefully
> you'll hit the issue.
>
> I've attached both my .config and the dmesg output from a successful
> boot at the end of this mail.

Thanks! Considering I know nothing about pmem (yet), I bet there is
some oversight in what's happening with how KASLR scans for available
memory areas. I'll carve out some time next week to look into this.

-Kees

-- 
Kees Cook
Pixel Security

[toc] | [next] | [standalone]

#1620134

Kees Cook <keescook@chromium.org> writes:

> On Fri, Apr 7, 2017 at 7:41 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
>> Hi,
>>
>> commit 021182e52fe01 ("x86/mm: Enable KASLR for physical mapping memory
>> regions") causes some of my systems with persistent memory (whether real
>> or emulated) to fail to boot with a couple of different crash
>> signatures.  The first signature is a NMI watchdog lockup of all but 1
>> cpu, which causes much difficulty in extracting useful information from
>> the console.  The second variant is an invalid paging request, listed
>> below.
>
> Just to rule out some of the stuff in the boot path, does booting with
> "nokaslr" solve this? (i.e. I want to figure out if this is from some
> of the rearrangements done that are exposed under that commit, or if
> it is genuinely the randomization that is killing the systems...)

Adding "nokaslr" to the boot line does indeed make the problem go away.

Cheers,
Jeff

[toc] | [prev] | [next] | [standalone]

#1620582

On Mon, Apr 10, 2017 at 8:49 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
> Kees Cook <keescook@chromium.org> writes:
>
>> On Fri, Apr 7, 2017 at 7:41 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
>>> Hi,
>>>
>>> commit 021182e52fe01 ("x86/mm: Enable KASLR for physical mapping memory
>>> regions") causes some of my systems with persistent memory (whether real
>>> or emulated) to fail to boot with a couple of different crash
>>> signatures.  The first signature is a NMI watchdog lockup of all but 1
>>> cpu, which causes much difficulty in extracting useful information from
>>> the console.  The second variant is an invalid paging request, listed
>>> below.
>>
>> Just to rule out some of the stuff in the boot path, does booting with
>> "nokaslr" solve this? (i.e. I want to figure out if this is from some
>> of the rearrangements done that are exposed under that commit, or if
>> it is genuinely the randomization that is killing the systems...)
>
> Adding "nokaslr" to the boot line does indeed make the problem go away.

Are you booting with a memmap= flag?

-Kees

-- 
Kees Cook
Pixel Security

[toc] | [prev] | [next] | [standalone]

#1620601

Kees Cook <keescook@chromium.org> writes:

> On Mon, Apr 10, 2017 at 8:49 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
>> Kees Cook <keescook@chromium.org> writes:
>>
>>> On Fri, Apr 7, 2017 at 7:41 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
>>>> Hi,
>>>>
>>>> commit 021182e52fe01 ("x86/mm: Enable KASLR for physical mapping memory
>>>> regions") causes some of my systems with persistent memory (whether real
>>>> or emulated) to fail to boot with a couple of different crash
>>>> signatures.  The first signature is a NMI watchdog lockup of all but 1
>>>> cpu, which causes much difficulty in extracting useful information from
>>>> the console.  The second variant is an invalid paging request, listed
>>>> below.
>>>
>>> Just to rule out some of the stuff in the boot path, does booting with
>>> "nokaslr" solve this? (i.e. I want to figure out if this is from some
>>> of the rearrangements done that are exposed under that commit, or if
>>> it is genuinely the randomization that is killing the systems...)
>>
>> Adding "nokaslr" to the boot line does indeed make the problem go away.
>
> Are you booting with a memmap= flag?

From my first email:

[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.11.0-rc5+
root=/dev/mapper/rhel_intel--lizardhead--04-root ro memmap=192G!1024G
crashkernel=auto rd.lvm.lv=rhel_intel-lizardhead-04/root
rd.lvm.lv=rhel_intel-lizardhead-04/swap console=ttyS0,115200n81
LANG=en_US.UTF-8

Did you not receive the attachments?

Cheers,
Jeff

[toc] | [prev] | [next] | [standalone]

#1620633

On Mon, Apr 10, 2017 at 11:22 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
> Kees Cook <keescook@chromium.org> writes:
>
>> On Mon, Apr 10, 2017 at 8:49 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
>>> Kees Cook <keescook@chromium.org> writes:
>>>
>>>> On Fri, Apr 7, 2017 at 7:41 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
>>>>> Hi,
>>>>>
>>>>> commit 021182e52fe01 ("x86/mm: Enable KASLR for physical mapping memory
>>>>> regions") causes some of my systems with persistent memory (whether real
>>>>> or emulated) to fail to boot with a couple of different crash
>>>>> signatures.  The first signature is a NMI watchdog lockup of all but 1
>>>>> cpu, which causes much difficulty in extracting useful information from
>>>>> the console.  The second variant is an invalid paging request, listed
>>>>> below.
>>>>
>>>> Just to rule out some of the stuff in the boot path, does booting with
>>>> "nokaslr" solve this? (i.e. I want to figure out if this is from some
>>>> of the rearrangements done that are exposed under that commit, or if
>>>> it is genuinely the randomization that is killing the systems...)
>>>
>>> Adding "nokaslr" to the boot line does indeed make the problem go away.
>>
>> Are you booting with a memmap= flag?
>
> From my first email:
>
> [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.11.0-rc5+
> root=/dev/mapper/rhel_intel--lizardhead--04-root ro memmap=192G!1024G
> crashkernel=auto rd.lvm.lv=rhel_intel-lizardhead-04/root
> rd.lvm.lv=rhel_intel-lizardhead-04/swap console=ttyS0,115200n81
> LANG=en_US.UTF-8
>
> Did you not receive the attachments?

I see it now, thanks!

The memmap parsing was added in -rc1 (f28442497b5ca), so I'd expect
that to be handled. Hmmm.

-Kees

-- 
Kees Cook
Pixel Security

[toc] | [prev] | [next] | [standalone]

#1620636

Kees Cook <keescook@chromium.org> writes:

> On Mon, Apr 10, 2017 at 11:22 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
>> Kees Cook <keescook@chromium.org> writes:
>>
>>> On Mon, Apr 10, 2017 at 8:49 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
>>>> Kees Cook <keescook@chromium.org> writes:
>>>>
>>>>> On Fri, Apr 7, 2017 at 7:41 AM, Jeff Moyer <jmoyer@redhat.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> commit 021182e52fe01 ("x86/mm: Enable KASLR for physical mapping memory
>>>>>> regions") causes some of my systems with persistent memory (whether real
>>>>>> or emulated) to fail to boot with a couple of different crash
>>>>>> signatures.  The first signature is a NMI watchdog lockup of all but 1
>>>>>> cpu, which causes much difficulty in extracting useful information from
>>>>>> the console.  The second variant is an invalid paging request, listed
>>>>>> below.
>>>>>
>>>>> Just to rule out some of the stuff in the boot path, does booting with
>>>>> "nokaslr" solve this? (i.e. I want to figure out if this is from some
>>>>> of the rearrangements done that are exposed under that commit, or if
>>>>> it is genuinely the randomization that is killing the systems...)
>>>>
>>>> Adding "nokaslr" to the boot line does indeed make the problem go away.
>>>
>>> Are you booting with a memmap= flag?
>>
>> From my first email:
>>
>> [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.11.0-rc5+
>> root=/dev/mapper/rhel_intel--lizardhead--04-root ro memmap=192G!1024G
>> crashkernel=auto rd.lvm.lv=rhel_intel-lizardhead-04/root
>> rd.lvm.lv=rhel_intel-lizardhead-04/swap console=ttyS0,115200n81
>> LANG=en_US.UTF-8
>>
>> Did you not receive the attachments?
>
> I see it now, thanks!
>
> The memmap parsing was added in -rc1 (f28442497b5ca), so I'd expect
> that to be handled. Hmmm.

I can also reproduce this on a system with real persistent memory, which
does not require the memmap parameter.

Cheers,
Jeff

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web