Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1504509 > unrolled thread

4.9-rc1 boot regression, ambiguous bisect result

Started byDan Williams <dan.j.williams@intel.com>
First post2016-10-20 06:10 +0200
Last post2016-10-22 01:30 +0200
Articles 10 — 3 participants

Back to article view | Back to linux.kernel

Contents

  4.9-rc1 boot regression, ambiguous bisect result Dan Williams <dan.j.williams@intel.com> - 2016-10-20 06:10 +0200
    Re: 4.9-rc1 boot regression, ambiguous bisect result Ingo Molnar <mingo@kernel.org> - 2016-10-20 08:30 +0200
    Re: 4.9-rc1 boot regression, ambiguous bisect result Matt Fleming <matt@codeblueprint.co.uk> - 2016-10-20 14:30 +0200
      Re: 4.9-rc1 boot regression, ambiguous bisect result Dan Williams <dan.j.williams@intel.com> - 2016-10-20 17:30 +0200
        Re: 4.9-rc1 boot regression, ambiguous bisect result Dan Williams <dan.j.williams@intel.com> - 2016-10-20 21:40 +0200
          Re: 4.9-rc1 boot regression, ambiguous bisect result Ingo Molnar <mingo@kernel.org> - 2016-10-21 09:10 +0200
            Re: 4.9-rc1 boot regression, ambiguous bisect result Dan Williams <dan.j.williams@intel.com> - 2016-10-21 20:30 +0200
          Re: 4.9-rc1 boot regression, ambiguous bisect result Matt Fleming <matt@codeblueprint.co.uk> - 2016-10-21 17:50 +0200
            Re: 4.9-rc1 boot regression, ambiguous bisect result Matt Fleming <matt@codeblueprint.co.uk> - 2016-10-21 22:30 +0200
              Re: 4.9-rc1 boot regression, ambiguous bisect result Dan Williams <dan.j.williams@intel.com> - 2016-10-22 01:30 +0200

#1504509 — 4.9-rc1 boot regression, ambiguous bisect result

FromDan Williams <dan.j.williams@intel.com>
Date2016-10-20 06:10 +0200
Subject4.9-rc1 boot regression, ambiguous bisect result
Message-ID<sucQ1-2td-1@gated-at.bofh.it>
Hi,

I am currently unable to boot a Yoga 900 with latest mainline, but 4.8 boots.

The symptom is a reboot before the video console is available.

I bisected to commit 816e76129ed5 "efi: Allow drivers to reserve boot
services forever".  However, that commit is known to be broken.  The
proposed fix, commit 92dc33501bfb "x86/efi: Round EFI memmap
reservations to EFI_PAGE_SIZE", also exhibits the reboot problem.

During the bisect some of the stopping points landed on commits that
caused the boot process to hang rather than cause a reboot.  The
commits that resulted in a hang are marked "git bisect skip" in this
log: https://gist.github.com/djbw/1b501daa98192a42ae848f03bb59c30e

I'll try treating those hangs as bad bisect results and re-run the
full bisect tomorrow.  In the meantime I wonder if the bisect log
implicates a better regression candidate?

[toc] | [next] | [standalone]

#1504538

FromIngo Molnar <mingo@kernel.org>
Date2016-10-20 08:30 +0200
Message-ID<suf1v-3Po-3@gated-at.bofh.it>
In reply to#1504509
* Dan Williams <dan.j.williams@intel.com> wrote:

> Hi,
> 
> I am currently unable to boot a Yoga 900 with latest mainline, but 4.8 boots.
> 
> The symptom is a reboot before the video console is available.
> 
> I bisected to commit 816e76129ed5 "efi: Allow drivers to reserve boot
> services forever".  However, that commit is known to be broken.  The
> proposed fix, commit 92dc33501bfb "x86/efi: Round EFI memmap
> reservations to EFI_PAGE_SIZE", also exhibits the reboot problem.
> 
> During the bisect some of the stopping points landed on commits that
> caused the boot process to hang rather than cause a reboot.  The
> commits that resulted in a hang are marked "git bisect skip" in this
> log: https://gist.github.com/djbw/1b501daa98192a42ae848f03bb59c30e
> 
> I'll try treating those hangs as bad bisect results and re-run the
> full bisect tomorrow.  In the meantime I wonder if the bisect log
> implicates a better regression candidate?

You could also try reverts of the suspicious commits, and then, if the reverted 
kernel works fine, create a more linear history by cherry-picking them in the 
right order - and then be able to pinpoint the bad commit with a higher 
confidence.

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1504834

FromMatt Fleming <matt@codeblueprint.co.uk>
Date2016-10-20 14:30 +0200
Message-ID<sukDU-7vO-21@gated-at.bofh.it>
In reply to#1504509
On Wed, 19 Oct, at 09:04:29PM, Dan Williams wrote:
> Hi,
> 
> I am currently unable to boot a Yoga 900 with latest mainline, but 4.8 boots.
> 
> The symptom is a reboot before the video console is available.
> 
> I bisected to commit 816e76129ed5 "efi: Allow drivers to reserve boot
> services forever".  However, that commit is known to be broken.  The
> proposed fix, commit 92dc33501bfb "x86/efi: Round EFI memmap
> reservations to EFI_PAGE_SIZE", also exhibits the reboot problem.
> 
> During the bisect some of the stopping points landed on commits that
> caused the boot process to hang rather than cause a reboot.  The
> commits that resulted in a hang are marked "git bisect skip" in this
> log: https://gist.github.com/djbw/1b501daa98192a42ae848f03bb59c30e
> 
> I'll try treating those hangs as bad bisect results and re-run the
> full bisect tomorrow.  In the meantime I wonder if the bisect log
> implicates a better regression candidate?

Could you mail the dmesg output when booting a known working kernel
with efi=debug ?

[toc] | [prev] | [next] | [standalone]

#1505034

FromDan Williams <dan.j.williams@intel.com>
Date2016-10-20 17:30 +0200
Message-ID<suns6-Ws-7@gated-at.bofh.it>
In reply to#1504834
On Thu, Oct 20, 2016 at 5:29 AM, Matt Fleming <matt@codeblueprint.co.uk> wrote:
> On Wed, 19 Oct, at 09:04:29PM, Dan Williams wrote:
>> Hi,
>>
>> I am currently unable to boot a Yoga 900 with latest mainline, but 4.8 boots.
>>
>> The symptom is a reboot before the video console is available.
>>
>> I bisected to commit 816e76129ed5 "efi: Allow drivers to reserve boot
>> services forever".  However, that commit is known to be broken.  The
>> proposed fix, commit 92dc33501bfb "x86/efi: Round EFI memmap
>> reservations to EFI_PAGE_SIZE", also exhibits the reboot problem.
>>
>> During the bisect some of the stopping points landed on commits that
>> caused the boot process to hang rather than cause a reboot.  The
>> commits that resulted in a hang are marked "git bisect skip" in this
>> log: https://gist.github.com/djbw/1b501daa98192a42ae848f03bb59c30e
>>
>> I'll try treating those hangs as bad bisect results and re-run the
>> full bisect tomorrow.  In the meantime I wonder if the bisect log
>> implicates a better regression candidate?
>
> Could you mail the dmesg output when booting a known working kernel
> with efi=debug ?

Here it is:

https://gist.github.com/djbw/cae05e721b159d5ad7b146d7a93f5fa2

[toc] | [prev] | [next] | [standalone]

#1505209

FromDan Williams <dan.j.williams@intel.com>
Date2016-10-20 21:40 +0200
Message-ID<surm1-3pA-9@gated-at.bofh.it>
In reply to#1505034
On Thu, Oct 20, 2016 at 8:22 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Thu, Oct 20, 2016 at 5:29 AM, Matt Fleming <matt@codeblueprint.co.uk> wrote:
>> On Wed, 19 Oct, at 09:04:29PM, Dan Williams wrote:
>>> Hi,
>>>
>>> I am currently unable to boot a Yoga 900 with latest mainline, but 4.8 boots.
>>>
>>> The symptom is a reboot before the video console is available.
>>>
>>> I bisected to commit 816e76129ed5 "efi: Allow drivers to reserve boot
>>> services forever".  However, that commit is known to be broken.  The
>>> proposed fix, commit 92dc33501bfb "x86/efi: Round EFI memmap
>>> reservations to EFI_PAGE_SIZE", also exhibits the reboot problem.
>>>
>>> During the bisect some of the stopping points landed on commits that
>>> caused the boot process to hang rather than cause a reboot.  The
>>> commits that resulted in a hang are marked "git bisect skip" in this
>>> log: https://gist.github.com/djbw/1b501daa98192a42ae848f03bb59c30e
>>>
>>> I'll try treating those hangs as bad bisect results and re-run the
>>> full bisect tomorrow.  In the meantime I wonder if the bisect log
>>> implicates a better regression candidate?
>>
>> Could you mail the dmesg output when booting a known working kernel
>> with efi=debug ?
>
> Here it is:
>
> https://gist.github.com/djbw/cae05e721b159d5ad7b146d7a93f5fa2

I am able to build a kernel and boot the platform with the following
set of reverts:

      Revert "x86/efi: Round EFI memmap reservations to EFI_PAGE_SIZE"
      Revert "x86/efi-bgrt: Use efi_mem_reserve() to avoid copying image data"
      Revert "efi/esrt: Use efi_mem_reserve() and avoid a kmalloc()"
      Revert "efi: Allow drivers to reserve boot services forever"

[toc] | [prev] | [next] | [standalone]

#1505515

FromIngo Molnar <mingo@kernel.org>
Date2016-10-21 09:10 +0200
Message-ID<suC7L-2gV-3@gated-at.bofh.it>
In reply to#1505209
* Dan Williams <dan.j.williams@intel.com> wrote:

> On Thu, Oct 20, 2016 at 8:22 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> > On Thu, Oct 20, 2016 at 5:29 AM, Matt Fleming <matt@codeblueprint.co.uk> wrote:
> >> On Wed, 19 Oct, at 09:04:29PM, Dan Williams wrote:
> >>> Hi,
> >>>
> >>> I am currently unable to boot a Yoga 900 with latest mainline, but 4.8 boots.
> >>>
> >>> The symptom is a reboot before the video console is available.
> >>>
> >>> I bisected to commit 816e76129ed5 "efi: Allow drivers to reserve boot
> >>> services forever".  However, that commit is known to be broken.  The
> >>> proposed fix, commit 92dc33501bfb "x86/efi: Round EFI memmap
> >>> reservations to EFI_PAGE_SIZE", also exhibits the reboot problem.
> >>>
> >>> During the bisect some of the stopping points landed on commits that
> >>> caused the boot process to hang rather than cause a reboot.  The
> >>> commits that resulted in a hang are marked "git bisect skip" in this
> >>> log: https://gist.github.com/djbw/1b501daa98192a42ae848f03bb59c30e
> >>>
> >>> I'll try treating those hangs as bad bisect results and re-run the
> >>> full bisect tomorrow.  In the meantime I wonder if the bisect log
> >>> implicates a better regression candidate?
> >>
> >> Could you mail the dmesg output when booting a known working kernel
> >> with efi=debug ?
> >
> > Here it is:
> >
> > https://gist.github.com/djbw/cae05e721b159d5ad7b146d7a93f5fa2
> 
> I am able to build a kernel and boot the platform with the following
> set of reverts:
> 
>       Revert "x86/efi: Round EFI memmap reservations to EFI_PAGE_SIZE"
>       Revert "x86/efi-bgrt: Use efi_mem_reserve() to avoid copying image data"
>       Revert "efi/esrt: Use efi_mem_reserve() and avoid a kmalloc()"
>       Revert "efi: Allow drivers to reserve boot services forever"

Could you please describe the bootup behavior after each revert? I.e. wild guess:

       vanilla kernel:
   # spontaneous reboot
       + Revert "x86/efi: Round EFI memmap reservations to EFI_PAGE_SIZE":
   # spontaneous reboot
       + Revert "x86/efi-bgrt: Use efi_mem_reserve() to avoid copying image data":
   # hang
       + Revert "efi/esrt: Use efi_mem_reserve() and avoid a kmalloc()":
   # hang
       + Revert "efi: Allow drivers to reserve boot services forever":
   == works

?

Thanks,

	Ingo

[toc] | [prev] | [next] | [standalone]

#1506210

FromDan Williams <dan.j.williams@intel.com>
Date2016-10-21 20:30 +0200
Message-ID<suMJP-VN-19@gated-at.bofh.it>
In reply to#1505515
On Fri, Oct 21, 2016 at 12:00 AM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dan Williams <dan.j.williams@intel.com> wrote:
>
>> On Thu, Oct 20, 2016 at 8:22 AM, Dan Williams <dan.j.williams@intel.com> wrote:
>> > On Thu, Oct 20, 2016 at 5:29 AM, Matt Fleming <matt@codeblueprint.co.uk> wrote:
>> >> On Wed, 19 Oct, at 09:04:29PM, Dan Williams wrote:
>> >>> Hi,
>> >>>
>> >>> I am currently unable to boot a Yoga 900 with latest mainline, but 4.8 boots.
>> >>>
>> >>> The symptom is a reboot before the video console is available.
>> >>>
>> >>> I bisected to commit 816e76129ed5 "efi: Allow drivers to reserve boot
>> >>> services forever".  However, that commit is known to be broken.  The
>> >>> proposed fix, commit 92dc33501bfb "x86/efi: Round EFI memmap
>> >>> reservations to EFI_PAGE_SIZE", also exhibits the reboot problem.
>> >>>
>> >>> During the bisect some of the stopping points landed on commits that
>> >>> caused the boot process to hang rather than cause a reboot.  The
>> >>> commits that resulted in a hang are marked "git bisect skip" in this
>> >>> log: https://gist.github.com/djbw/1b501daa98192a42ae848f03bb59c30e
>> >>>
>> >>> I'll try treating those hangs as bad bisect results and re-run the
>> >>> full bisect tomorrow.  In the meantime I wonder if the bisect log
>> >>> implicates a better regression candidate?
>> >>
>> >> Could you mail the dmesg output when booting a known working kernel
>> >> with efi=debug ?
>> >
>> > Here it is:
>> >
>> > https://gist.github.com/djbw/cae05e721b159d5ad7b146d7a93f5fa2
>>
>> I am able to build a kernel and boot the platform with the following
>> set of reverts:
>>
>>       Revert "x86/efi: Round EFI memmap reservations to EFI_PAGE_SIZE"
>>       Revert "x86/efi-bgrt: Use efi_mem_reserve() to avoid copying image data"
>>       Revert "efi/esrt: Use efi_mem_reserve() and avoid a kmalloc()"
>>       Revert "efi: Allow drivers to reserve boot services forever"
>
> Could you please describe the bootup behavior after each revert? I.e. wild guess:
>
>        vanilla kernel:
>    # spontaneous reboot
>        + Revert "x86/efi: Round EFI memmap reservations to EFI_PAGE_SIZE":
>    # spontaneous reboot
>        + Revert "x86/efi-bgrt: Use efi_mem_reserve() to avoid copying image data":
>    # hang
>        + Revert "efi/esrt: Use efi_mem_reserve() and avoid a kmalloc()":
>    # hang
>        + Revert "efi: Allow drivers to reserve boot services forever":
>    == works
>
> ?

In this case all but the last revert produce the same result, instant
reboot after loading the kernel.  I have not been able to pinpoint
what changes that behavior to the hang conditions I saw mid-bisect.

The first three reverts are just there to get the kernel to build
again after reverting "efi: Allow drivers to reserve boot services
forever"

[toc] | [prev] | [next] | [standalone]

#1506094

FromMatt Fleming <matt@codeblueprint.co.uk>
Date2016-10-21 17:50 +0200
Message-ID<suKf0-7AT-21@gated-at.bofh.it>
In reply to#1505209
On Thu, 20 Oct, at 12:37:16PM, Dan Williams wrote:
> 
> I am able to build a kernel and boot the platform with the following
> set of reverts:
> 
>       Revert "x86/efi: Round EFI memmap reservations to EFI_PAGE_SIZE"
>       Revert "x86/efi-bgrt: Use efi_mem_reserve() to avoid copying image data"
>       Revert "efi/esrt: Use efi_mem_reserve() and avoid a kmalloc()"
>       Revert "efi: Allow drivers to reserve boot services forever"

FYI, I've been able to reproduce some crash when using your EFI memory
map layout under Qemu and forcing the ESRT driver to reserve the space.

It looks like the new EFI memmap we allocate as part of the
reservation is smaller than the old one - which is backwards.

Still debugging...

[toc] | [prev] | [next] | [standalone]

#1506268

FromMatt Fleming <matt@codeblueprint.co.uk>
Date2016-10-21 22:30 +0200
Message-ID<suOBY-2ad-15@gated-at.bofh.it>
In reply to#1506094
On Fri, 21 Oct, at 04:41:29PM, Matt Fleming wrote:
> 
> FYI, I've been able to reproduce some crash when using your EFI memory
> map layout under Qemu and forcing the ESRT driver to reserve the space.
 
Nope, that was a bug in my hack. I can't get Qemu to crash while using
your memory map layout.

Any chance you can insert "while(1)" loops into the EFI boot paths for
a kernel that is known to reboot or trigger a triple fault in kernels
that hang, so that we can narrow in on the issue. See,

  http://www.codeblueprint.co.uk/2015/04/early-x86-linux-boot-debug-tricks.html

[toc] | [prev] | [next] | [standalone]

#1506352

FromDan Williams <dan.j.williams@intel.com>
Date2016-10-22 01:30 +0200
Message-ID<suRq9-3X6-7@gated-at.bofh.it>
In reply to#1506268
On Fri, Oct 21, 2016 at 1:20 PM, Matt Fleming <matt@codeblueprint.co.uk> wrote:
> On Fri, 21 Oct, at 04:41:29PM, Matt Fleming wrote:
>>
>> FYI, I've been able to reproduce some crash when using your EFI memory
>> map layout under Qemu and forcing the ESRT driver to reserve the space.
>
> Nope, that was a bug in my hack. I can't get Qemu to crash while using
> your memory map layout.
>
> Any chance you can insert "while(1)" loops into the EFI boot paths for
> a kernel that is known to reboot or trigger a triple fault in kernels
> that hang, so that we can narrow in on the issue. See,
>
>   http://www.codeblueprint.co.uk/2015/04/early-x86-linux-boot-debug-tricks.html

I can take a look, but it will not be until Monday when I have
physical access to the system again.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web