Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1637245 > unrolled thread

Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table until per-cpu is set up

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table  until per-cpu is set up Jan Kiszka <jan.kiszka@siemens.com> - 2017-05-08 08:40 +0200
    Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table  until per-cpu is set up Andy Shevchenko <andy.shevchenko@gmail.com> - 2017-05-08 11:40 +0200
      Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table  until per-cpu is set up Andy Lutomirski <luto@kernel.org> - 2017-05-08 13:30 +0200
        Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table  until per-cpu is set up Jan Kiszka <jan.kiszka@siemens.com> - 2017-05-08 14:40 +0200
          Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table  until per-cpu is set up Andy Shevchenko <andy.shevchenko@gmail.com> - 2017-05-08 16:50 +0200
            Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table  until per-cpu is set up Jan Kiszka <jan.kiszka@siemens.com> - 2017-05-08 17:30 +0200
          Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table  until per-cpu is set up Jan Kiszka <jan.kiszka@siemens.com> - 2017-05-08 20:00 +0200
            Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table  until per-cpu is set up Andy Lutomirski <luto@kernel.org> - 2017-05-09 02:10 +0200

#1637245 — Re: [tip:x86/mm] x86/boot/32: Defer resyncing initial_page_table until per-cpu is set up

On 2017-03-23 10:14, tip-bot for Andy Lutomirski wrote:
> The x86 smpboot trampoline expects initial_page_table to have the
> GDT mapped.  If the GDT ends up in a virtually mapped per-cpu page,
> then it won't be in the page tables at all until perc-pu areas are
> set up.  The result will be a triple fault the first time that the
> CPU attempts to access the GDT after LGDT loads the perc-pu GDT.
> 
> This appears to be an old bug, but somehow the GDT fixmap rework
> is triggering it.  This seems to have something to do with the
> memory layout.
> 
> Signed-off-by: Andy Lutomirski <luto@kernel.org>
> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: Brian Gerst <brgerst@gmail.com>
> Cc: Denys Vlasenko <dvlasenk@redhat.com>
> Cc: H. Peter Anvin <hpa@zytor.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>
> Cc: Matt Fleming <matt@codeblueprint.co.uk>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Garnier <thgarnie@google.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: linux-efi@vger.kernel.org
> Link: http://lkml.kernel.org/r/a553264a5972c6a86f9b5caac237470a0c74a720.1490218061.git.luto@kernel.org
> Signed-off-by: Ingo Molnar <mingo@kernel.org>
> ---
>  arch/x86/kernel/setup.c        | 15 ---------------
>  arch/x86/kernel/setup_percpu.c | 21 +++++++++++++++++++++
>  2 files changed, 21 insertions(+), 15 deletions(-)
> 
> diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
> index 4bf0c89..56b1177 100644
> --- a/arch/x86/kernel/setup.c
> +++ b/arch/x86/kernel/setup.c
> @@ -1226,21 +1226,6 @@ void __init setup_arch(char **cmdline_p)
>  
>  	kasan_init();
>  
> -#ifdef CONFIG_X86_32
> -	/* sync back kernel address range */
> -	clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY,
> -			swapper_pg_dir     + KERNEL_PGD_BOUNDARY,
> -			KERNEL_PGD_PTRS);
> -
> -	/*
> -	 * sync back low identity map too.  It is used for example
> -	 * in the 32-bit EFI stub.
> -	 */
> -	clone_pgd_range(initial_page_table,
> -			swapper_pg_dir     + KERNEL_PGD_BOUNDARY,
> -			min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY));
> -#endif
> -
>  	tboot_probe();
>  
>  	map_vsyscall();
> diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c
> index 11338b0..bb1e8cc 100644
> --- a/arch/x86/kernel/setup_percpu.c
> +++ b/arch/x86/kernel/setup_percpu.c
> @@ -288,4 +288,25 @@ void __init setup_per_cpu_areas(void)
>  
>  	/* Setup cpu initialized, callin, callout masks */
>  	setup_cpu_local_masks();
> +
> +#ifdef CONFIG_X86_32
> +	/*
> +	 * Sync back kernel address range.  We want to make sure that
> +	 * all kernel mappings, including percpu mappings, are available
> +	 * in the smpboot asm.  We can't reliably pick up percpu
> +	 * mappings using vmalloc_fault(), because exception dispatch
> +	 * needs percpu data.
> +	 */
> +	clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY,
> +			swapper_pg_dir     + KERNEL_PGD_BOUNDARY,
> +			KERNEL_PGD_PTRS);
> +
> +	/*
> +	 * sync back low identity map too.  It is used for example
> +	 * in the 32-bit EFI stub.
> +	 */
> +	clone_pgd_range(initial_page_table,
> +			swapper_pg_dir     + KERNEL_PGD_BOUNDARY,
> +			min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY));
> +#endif
>  }
> 

This breaks the boot on our Intel Quark platform (IOT2000, similar to
Galileo Gen2). Reverting it over master makes it work again. Any idea
what goes wrong? Let me know how I can help debugging this.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[toc] | [next] | [standalone]

#1637328

On Mon, May 8, 2017 at 9:31 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 2017-03-23 10:14, tip-bot for Andy Lutomirski wrote:
>> The x86 smpboot trampoline expects initial_page_table to have the
>> GDT mapped.  If the GDT ends up in a virtually mapped per-cpu page,
>> then it won't be in the page tables at all until perc-pu areas are
>> set up.  The result will be a triple fault the first time that the
>> CPU attempts to access the GDT after LGDT loads the perc-pu GDT.
>>
>> This appears to be an old bug, but somehow the GDT fixmap rework
>> is triggering it.  This seems to have something to do with the
>> memory layout.

> This breaks the boot on our Intel Quark platform (IOT2000, similar to
> Galileo Gen2). Reverting it over master makes it work again. Any idea
> what goes wrong? Let me know how I can help debugging this.

JFYI: As of today linux-next when _kexec:ed_ works fine to me

Perhaps I can test this later with direct boot from SD card.

-- 
With Best Regards,
Andy Shevchenko

[toc] | [prev] | [next] | [standalone]

#1637393

On Mon, May 8, 2017 at 2:32 AM, Andy Shevchenko
<andy.shevchenko@gmail.com> wrote:
> On Mon, May 8, 2017 at 9:31 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> On 2017-03-23 10:14, tip-bot for Andy Lutomirski wrote:
>>> The x86 smpboot trampoline expects initial_page_table to have the
>>> GDT mapped.  If the GDT ends up in a virtually mapped per-cpu page,
>>> then it won't be in the page tables at all until perc-pu areas are
>>> set up.  The result will be a triple fault the first time that the
>>> CPU attempts to access the GDT after LGDT loads the perc-pu GDT.
>>>
>>> This appears to be an old bug, but somehow the GDT fixmap rework
>>> is triggering it.  This seems to have something to do with the
>>> memory layout.
>
>> This breaks the boot on our Intel Quark platform (IOT2000, similar to
>> Galileo Gen2). Reverting it over master makes it work again. Any idea
>> what goes wrong? Let me know how I can help debugging this.
>
> JFYI: As of today linux-next when _kexec:ed_ works fine to me
>
> Perhaps I can test this later with direct boot from SD card.
>

The most likely explanation is that there's some code that needs the
page table synced and runs before setup_per_cpu_areas().  The relevant
init code is:

    setup_arch(&command_line);
    mm_init_cpumask(&init_mm);
    setup_command_line(command_line);
    setup_nr_cpu_ids();
    setup_per_cpu_areas();

so I didn't move it very far.  It would be awesome if we could get a
backtrace when the failure happens, but it's likely to be a triple
fault.  Is this an EFI boot?  I bet the failure is in efi_init().

Could you try reverting just the deletions in the patch?  I.e. try a
kernel with both the old and the new copies of the code I moved.

--Andy

[toc] | [prev] | [next] | [standalone]

#1637420

On 2017-05-08 13:21, Andy Lutomirski wrote:
> On Mon, May 8, 2017 at 2:32 AM, Andy Shevchenko
> <andy.shevchenko@gmail.com> wrote:
>> On Mon, May 8, 2017 at 9:31 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>> On 2017-03-23 10:14, tip-bot for Andy Lutomirski wrote:
>>>> The x86 smpboot trampoline expects initial_page_table to have the
>>>> GDT mapped.  If the GDT ends up in a virtually mapped per-cpu page,
>>>> then it won't be in the page tables at all until perc-pu areas are
>>>> set up.  The result will be a triple fault the first time that the
>>>> CPU attempts to access the GDT after LGDT loads the perc-pu GDT.
>>>>
>>>> This appears to be an old bug, but somehow the GDT fixmap rework
>>>> is triggering it.  This seems to have something to do with the
>>>> memory layout.
>>
>>> This breaks the boot on our Intel Quark platform (IOT2000, similar to
>>> Galileo Gen2). Reverting it over master makes it work again. Any idea
>>> what goes wrong? Let me know how I can help debugging this.
>>
>> JFYI: As of today linux-next when _kexec:ed_ works fine to me
>>
>> Perhaps I can test this later with direct boot from SD card.
>>
> 
> The most likely explanation is that there's some code that needs the
> page table synced and runs before setup_per_cpu_areas().  The relevant
> init code is:
> 
>     setup_arch(&command_line);
>     mm_init_cpumask(&init_mm);
>     setup_command_line(command_line);
>     setup_nr_cpu_ids();
>     setup_per_cpu_areas();
> 
> so I didn't move it very far.  It would be awesome if we could get a
> backtrace when the failure happens, but it's likely to be a triple
> fault.  Is this an EFI boot?  I bet the failure is in efi_init().

Yes, it's an EFI thing. Unfortunately, I didn't make
earlycon/earlyprintk work yet.

> 
> Could you try reverting just the deletions in the patch?  I.e. try a
> kernel with both the old and the new copies of the code I moved.

Let me try that later. I can also move the new code around to nail down
the dependency.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[toc] | [prev] | [next] | [standalone]

#1637479

On Mon, May 8, 2017 at 3:34 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 2017-05-08 13:21, Andy Lutomirski wrote:

> Yes, it's an EFI thing. Unfortunately, I didn't make
> earlycon/earlyprintk work yet.

Quark does use HS UART for the console and we have no support for such
in x86/earlyprintk. earlycon should work if you add proper parameters
to the command line (including port base address, no ttySx), though I
don't remember if it makes really "early".

-- 
With Best Regards,
Andy Shevchenko

[toc] | [prev] | [next] | [standalone]

#1637517

On 2017-05-08 16:45, Andy Shevchenko wrote:
> On Mon, May 8, 2017 at 3:34 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> On 2017-05-08 13:21, Andy Lutomirski wrote:
> 
>> Yes, it's an EFI thing. Unfortunately, I didn't make
>> earlycon/earlyprintk work yet.
> 
> Quark does use HS UART for the console and we have no support for such
> in x86/earlyprintk. earlycon should work if you add proper parameters
> to the command line (including port base address, no ttySx), though I
> don't remember if it makes really "early".

Ah, that explains it. I've tried earlycon, also with the apparently
correct params, but it didn't print anything in this case. And
earlyprintk=efi seems to use a framebuffer, while our devices are
headless...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[toc] | [prev] | [next] | [standalone]

#1637625

On 2017-05-08 14:34, Jan Kiszka wrote:
> On 2017-05-08 13:21, Andy Lutomirski wrote:
>> On Mon, May 8, 2017 at 2:32 AM, Andy Shevchenko
>> <andy.shevchenko@gmail.com> wrote:
>>> On Mon, May 8, 2017 at 9:31 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>> On 2017-03-23 10:14, tip-bot for Andy Lutomirski wrote:
>>>>> The x86 smpboot trampoline expects initial_page_table to have the
>>>>> GDT mapped.  If the GDT ends up in a virtually mapped per-cpu page,
>>>>> then it won't be in the page tables at all until perc-pu areas are
>>>>> set up.  The result will be a triple fault the first time that the
>>>>> CPU attempts to access the GDT after LGDT loads the perc-pu GDT.
>>>>>
>>>>> This appears to be an old bug, but somehow the GDT fixmap rework
>>>>> is triggering it.  This seems to have something to do with the
>>>>> memory layout.
>>>
>>>> This breaks the boot on our Intel Quark platform (IOT2000, similar to
>>>> Galileo Gen2). Reverting it over master makes it work again. Any idea
>>>> what goes wrong? Let me know how I can help debugging this.
>>>
>>> JFYI: As of today linux-next when _kexec:ed_ works fine to me
>>>
>>> Perhaps I can test this later with direct boot from SD card.
>>>
>>
>> The most likely explanation is that there's some code that needs the
>> page table synced and runs before setup_per_cpu_areas().  The relevant
>> init code is:
>>
>>     setup_arch(&command_line);
>>     mm_init_cpumask(&init_mm);
>>     setup_command_line(command_line);
>>     setup_nr_cpu_ids();
>>     setup_per_cpu_areas();
>>
>> so I didn't move it very far.  It would be awesome if we could get a
>> backtrace when the failure happens, but it's likely to be a triple
>> fault.  Is this an EFI boot?  I bet the failure is in efi_init().
> 
> Yes, it's an EFI thing. Unfortunately, I didn't make
> earlycon/earlyprintk work yet.
> 
>>
>> Could you try reverting just the deletions in the patch?  I.e. try a
>> kernel with both the old and the new copies of the code I moved.
> 
> Let me try that later. I can also move the new code around to nail down
> the dependency.
> 

I found the reason: your patch is very discriminating! Not the whole
world is multicore yet. ;)

setup_per_cpu_areas() is taken from mm/percpu.c in case of !CONFIG_SMP.
So the new home for the resync is not even built.

Any suggestions how to refactor things instead?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

[toc] | [prev] | [next] | [standalone]

#1637793

On Mon, May 8, 2017 at 10:53 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 2017-05-08 14:34, Jan Kiszka wrote:
>> On 2017-05-08 13:21, Andy Lutomirski wrote:
>>> On Mon, May 8, 2017 at 2:32 AM, Andy Shevchenko
>>> <andy.shevchenko@gmail.com> wrote:
>>>> On Mon, May 8, 2017 at 9:31 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>>> On 2017-03-23 10:14, tip-bot for Andy Lutomirski wrote:
>>>>>> The x86 smpboot trampoline expects initial_page_table to have the
>>>>>> GDT mapped.  If the GDT ends up in a virtually mapped per-cpu page,
>>>>>> then it won't be in the page tables at all until perc-pu areas are
>>>>>> set up.  The result will be a triple fault the first time that the
>>>>>> CPU attempts to access the GDT after LGDT loads the perc-pu GDT.
>>>>>>
>>>>>> This appears to be an old bug, but somehow the GDT fixmap rework
>>>>>> is triggering it.  This seems to have something to do with the
>>>>>> memory layout.
>>>>
>>>>> This breaks the boot on our Intel Quark platform (IOT2000, similar to
>>>>> Galileo Gen2). Reverting it over master makes it work again. Any idea
>>>>> what goes wrong? Let me know how I can help debugging this.
>>>>
>>>> JFYI: As of today linux-next when _kexec:ed_ works fine to me
>>>>
>>>> Perhaps I can test this later with direct boot from SD card.
>>>>
>>>
>>> The most likely explanation is that there's some code that needs the
>>> page table synced and runs before setup_per_cpu_areas().  The relevant
>>> init code is:
>>>
>>>     setup_arch(&command_line);
>>>     mm_init_cpumask(&init_mm);
>>>     setup_command_line(command_line);
>>>     setup_nr_cpu_ids();
>>>     setup_per_cpu_areas();
>>>
>>> so I didn't move it very far.  It would be awesome if we could get a
>>> backtrace when the failure happens, but it's likely to be a triple
>>> fault.  Is this an EFI boot?  I bet the failure is in efi_init().
>>
>> Yes, it's an EFI thing. Unfortunately, I didn't make
>> earlycon/earlyprintk work yet.
>>
>>>
>>> Could you try reverting just the deletions in the patch?  I.e. try a
>>> kernel with both the old and the new copies of the code I moved.
>>
>> Let me try that later. I can also move the new code around to nail down
>> the dependency.
>>
>
> I found the reason: your patch is very discriminating! Not the whole
> world is multicore yet. ;)
>
> setup_per_cpu_areas() is taken from mm/percpu.c in case of !CONFIG_SMP.
> So the new home for the resync is not even built.

D'oh!

>
> Any suggestions how to refactor things instead?

efi_init() seems okay, but it makes me nervous.  I think the partial
revert is the right fix.  Patch coming.

>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
> Corporate Competence Center Embedded Linux

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web