Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1610504 > unrolled thread

Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing and suspend-to-ram

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing  and suspend-to-ram Paul Menzel <pmenzel@molgen.mpg.de> - 2017-03-28 12:00 +0200
    Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing  and suspend-to-ram Steven Rostedt <rostedt@goodmis.org> - 2017-03-28 17:50 +0200
      Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing  and suspend-to-ram Josh Poimboeuf <jpoimboe@redhat.com> - 2017-03-28 18:00 +0200
        Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing and suspend-to-ram "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2017-03-28 23:20 +0200
          Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing  and suspend-to-ram Josh Poimboeuf <jpoimboe@redhat.com> - 2017-03-28 23:50 +0200
            Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing and suspend-to-ram "Rafael J. Wysocki" <rjw@rjwysocki.net> - 2017-03-29 00:00 +0200

#1610504 — Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing and suspend-to-ram

Dear Josh,


On 03/27/17 17:01, Paul Menzel wrote:

> On 03/27/17 16:54, Josh Poimboeuf wrote:
>> On x86-32, with CONFIG_FIRMWARE and multiple CPUs, if you enable
>> function graph tracing and then suspend to RAM, it will triple fault and
>> reboot when it resumes.
>>
>> The first fault happens when booting a secondary CPU:
>>
>> startup_32_smp()
>>   load_ucode_ap()
>>     prepare_ftrace_return()
>>       ftrace_graph_is_dead()
>>         (accesses 'kill_ftrace_graph')
>>
>> The early head_32.S code calls into load_ucode_ap(), which has an an
>> ftrace hook, so it calls prepare_ftrace_return(), which calls
>> ftrace_graph_is_dead(), which tries to access the global
>> 'kill_ftrace_graph' variable with a virtual address, causing a fault
>> because the CPU is still in real mode.
>>
>> The fix is to add a check in prepare_ftrace_return() to make sure it's
>> running in protected mode before continuing.  The check makes sure the
>> stack pointer is a virtual kernel address.  It's a bit of a hack, but
>> it's not very intrusive and it works well enough.
>>
>> For reference, here are a few other ways this could have potentially
>> been fixed:
>>
>> - Move startup_32_smp()'s call to load_ucode_ap() down to *after* paging
>>   is enabled.  (No idea what that would break.)
>>
>> - Track down load_ucode_ap()'s entire callee tree and mark all the
>>   functions 'notrace'.  (Probably not realistic.)
>>
>> - Pause graph tracing in ftrace_suspend_notifier_call() or bringup_cpu()
>>   or __cpu_up(), and ensure that the pause facility can be queried from
>>   real mode.
>>
>> Reported-by: Paul Menzel <pmenzel@molgen.mpg.de>
>> Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
>> ---
>>  arch/x86/kernel/ftrace.c | 11 +++++++++++
>>  1 file changed, 11 insertions(+)
>
> Thank you for debugging this. It’s great that you were able to reproduce
> this in QEMU. Hopefully, that’ll make for an easy test case. ;-)
>
>> diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
>> index 8f3d9cf..1c5c4e2 100644
>> --- a/arch/x86/kernel/ftrace.c
>> +++ b/arch/x86/kernel/ftrace.c
>> @@ -983,6 +983,17 @@ void prepare_ftrace_return(unsigned long
>> self_addr, unsigned long *parent,
>>      unsigned long return_hooker = (unsigned long)
>>                  &return_to_handler;
>>
>> +    /*
>> +     * When resuming from suspend-to-ram, this function can be
>> indirectly
>> +     * called from early CPU startup code while the CPU is in real mode,
>> +     * which would fail miserably.  Make sure the stack pointer is a
>> +     * virtual address.
>> +     *
>> +     * This check isn't as accurate as virt_addr_valid(), but it
>> should be
>> +     * good enough for this purpose, and it's fast.
>> +     */
>> +    if (unlikely((long)__builtin_frame_address(0) >= 0)) return;
>
> The coding style requires the `return;` to be on a separate line.
>
>> +
>>      if (unlikely(ftrace_graph_is_dead()))
>>          return;
>
> I’ll test your change this evening.

With both patches applied `./analyze_suspend.py -config 
suspend-callgraph.cfg -filter i915` succeeds on a Lenovo X60t, so 
suspend and resume work perfectly, when tracing is enabled.

Tested-by: Paul Menzel <pmenzel@molgen.mpg.de>

It’d be awesome, if you could tag both patches for inclusion into the 
stable Linux Kernel series.


Kind regards,

Paul

[toc] | [next] | [standalone]

#1611128

On Tue, 28 Mar 2017 11:51:45 +0200
Paul Menzel <pmenzel@molgen.mpg.de> wrote:

> With both patches applied `./analyze_suspend.py -config 
> suspend-callgraph.cfg -filter i915` succeeds on a Lenovo X60t, so 
> suspend and resume work perfectly, when tracing is enabled.
> 
> Tested-by: Paul Menzel <pmenzel@molgen.mpg.de>
> 
> It’d be awesome, if you could tag both patches for inclusion into the 
> stable Linux Kernel series.

As long as they are not dependent on my patch series, I'm fine with
these going to stable.

-- Steve

[toc] | [prev] | [next] | [standalone]

#1611132

On Tue, Mar 28, 2017 at 11:39:41AM -0400, Steven Rostedt wrote:
> On Tue, 28 Mar 2017 11:51:45 +0200
> Paul Menzel <pmenzel@molgen.mpg.de> wrote:
> 
> > With both patches applied `./analyze_suspend.py -config 
> > suspend-callgraph.cfg -filter i915` succeeds on a Lenovo X60t, so 
> > suspend and resume work perfectly, when tracing is enabled.
> > 
> > Tested-by: Paul Menzel <pmenzel@molgen.mpg.de>
> > 
> > It’d be awesome, if you could tag both patches for inclusion into the 
> > stable Linux Kernel series.
> 
> As long as they are not dependent on my patch series, I'm fine with
> these going to stable.

Stable sounds fine to me too.  Both patches are independent of your
x86-32 fentry patch set.

-- 
Josh

[toc] | [prev] | [next] | [standalone]

#1611365 — Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing and suspend-to-ram

On Tuesday, March 28, 2017 10:55:46 AM Josh Poimboeuf wrote:
> On Tue, Mar 28, 2017 at 11:39:41AM -0400, Steven Rostedt wrote:
> > On Tue, 28 Mar 2017 11:51:45 +0200
> > Paul Menzel <pmenzel@molgen.mpg.de> wrote:
> > 
> > > With both patches applied `./analyze_suspend.py -config 
> > > suspend-callgraph.cfg -filter i915` succeeds on a Lenovo X60t, so 
> > > suspend and resume work perfectly, when tracing is enabled.
> > > 
> > > Tested-by: Paul Menzel <pmenzel@molgen.mpg.de>
> > > 
> > > It’d be awesome, if you could tag both patches for inclusion into the 
> > > stable Linux Kernel series.
> > 
> > As long as they are not dependent on my patch series, I'm fine with
> > these going to stable.
> 
> Stable sounds fine to me too.  Both patches are independent of your
> x86-32 fentry patch set.

Does https://patchwork.kernel.org/patch/9628301/ need to go into any particular
-stable series or just all of them?

Or should a Fixes: tag be added to it?

Thanks,
Rafael

[toc] | [prev] | [next] | [standalone]

#1611387

On Tue, Mar 28, 2017 at 11:12:42PM +0200, Rafael J. Wysocki wrote:
> On Tuesday, March 28, 2017 10:55:46 AM Josh Poimboeuf wrote:
> > On Tue, Mar 28, 2017 at 11:39:41AM -0400, Steven Rostedt wrote:
> > > On Tue, 28 Mar 2017 11:51:45 +0200
> > > Paul Menzel <pmenzel@molgen.mpg.de> wrote:
> > > 
> > > > With both patches applied `./analyze_suspend.py -config 
> > > > suspend-callgraph.cfg -filter i915` succeeds on a Lenovo X60t, so 
> > > > suspend and resume work perfectly, when tracing is enabled.
> > > > 
> > > > Tested-by: Paul Menzel <pmenzel@molgen.mpg.de>
> > > > 
> > > > It’d be awesome, if you could tag both patches for inclusion into the 
> > > > stable Linux Kernel series.
> > > 
> > > As long as they are not dependent on my patch series, I'm fine with
> > > these going to stable.
> > 
> > Stable sounds fine to me too.  Both patches are independent of your
> > x86-32 fentry patch set.
> 
> Does https://patchwork.kernel.org/patch/9628301/ need to go into any particular
> -stable series or just all of them?
> 
> Or should a Fixes: tag be added to it?

As far as I can tell this issue has been around since the function_graph
tracer was introduced in 2008:

  15e6cb3673ea ("tracing: add a tracer to catch execution time of kernel functions")

(Though only for gcc >= 4.4.)

Not sure if it's overkill to specify 'Fixes' for an 8+ year old bug?  I
guess it can't hurt anything.

I think it can go in all of the stable branches.

-- 
Josh

[toc] | [prev] | [next] | [standalone]

#1611390 — Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing and suspend-to-ram

On Tuesday, March 28, 2017 04:42:18 PM Josh Poimboeuf wrote:
> On Tue, Mar 28, 2017 at 11:12:42PM +0200, Rafael J. Wysocki wrote:
> > On Tuesday, March 28, 2017 10:55:46 AM Josh Poimboeuf wrote:
> > > On Tue, Mar 28, 2017 at 11:39:41AM -0400, Steven Rostedt wrote:
> > > > On Tue, 28 Mar 2017 11:51:45 +0200
> > > > Paul Menzel <pmenzel@molgen.mpg.de> wrote:
> > > > 
> > > > > With both patches applied `./analyze_suspend.py -config 
> > > > > suspend-callgraph.cfg -filter i915` succeeds on a Lenovo X60t, so 
> > > > > suspend and resume work perfectly, when tracing is enabled.
> > > > > 
> > > > > Tested-by: Paul Menzel <pmenzel@molgen.mpg.de>
> > > > > 
> > > > > It’d be awesome, if you could tag both patches for inclusion into the 
> > > > > stable Linux Kernel series.
> > > > 
> > > > As long as they are not dependent on my patch series, I'm fine with
> > > > these going to stable.
> > > 
> > > Stable sounds fine to me too.  Both patches are independent of your
> > > x86-32 fentry patch set.
> > 
> > Does https://patchwork.kernel.org/patch/9628301/ need to go into any particular
> > -stable series or just all of them?
> > 
> > Or should a Fixes: tag be added to it?
> 
> As far as I can tell this issue has been around since the function_graph
> tracer was introduced in 2008:
> 
>   15e6cb3673ea ("tracing: add a tracer to catch execution time of kernel functions")
> 
> (Though only for gcc >= 4.4.)
> 
> Not sure if it's overkill to specify 'Fixes' for an 8+ year old bug?  I
> guess it can't hurt anything.
> 
> I think it can go in all of the stable branches.

OK, thanks!

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web