Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1330667 > unrolled thread

[PATCH] arm64: use raw_smp_processor_id in stack backtrace dump

Back to article view | Back to linux.kernel

Contents

  [PATCH] arm64: use raw_smp_processor_id in stack backtrace dump Yang Shi <yang.shi@linaro.org> - 2016-02-09 22:50 +0100
    Re: [PATCH] arm64: use raw_smp_processor_id in stack backtrace dump Will Deacon <will.deacon@arm.com> - 2016-02-10 11:30 +0100
      Re: [PATCH] arm64: use raw_smp_processor_id in stack backtrace dump James Morse <james.morse@arm.com> - 2016-02-10 13:00 +0100
        Re: [PATCH] arm64: use raw_smp_processor_id in stack backtrace dump Will Deacon <will.deacon@arm.com> - 2016-02-10 13:20 +0100
          Re: [PATCH] arm64: use raw_smp_processor_id in stack backtrace dump "Shi, Yang" <yang.shi@linaro.org> - 2016-02-10 19:20 +0100
            Re: [PATCH] arm64: use raw_smp_processor_id in stack backtrace dump James Morse <james.morse@arm.com> - 2016-02-11 11:50 +0100
              Re: [PATCH] arm64: use raw_smp_processor_id in stack backtrace dump "Shi, Yang" <yang.shi@linaro.org> - 2016-02-11 18:40 +0100

#1330667 — [PATCH] arm64: use raw_smp_processor_id in stack backtrace dump

dump_backtrace may be called in kthread context, which is not bound to a single
cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug
report:

BUG: using smp_processor_id() in preemptible [00000000] code: khungtaskd/71
caller is debug_smp_processor_id+0x1c/0x28
CPU: 3 PID: 71 Comm: khungtaskd Not tainted 4.5.0-rc1 #144
Hardware name: Freescale Layerscape 2085a RDB Board (DT)
Call trace:
[<ffffffc00008d120>] dump_backtrace+0x0/0x290
[<ffffffc00008d3d4>] show_stack+0x24/0x30
[<ffffffc0006bfd1c>] dump_stack+0x8c/0xd8
[<ffffffc0006fe56c>] check_preemption_disabled+0x184/0x188
[<ffffffc0006fe58c>] debug_smp_processor_id+0x1c/0x28
[<ffffffc00008d1a4>] dump_backtrace+0x84/0x290
[<ffffffc00008d3d4>] show_stack+0x24/0x30
[<ffffffc00012aa5c>] sched_show_task+0x16c/0x280
[<ffffffc0001f3df8>] watchdog+0x560/0x708
[<ffffffc000111bb8>] kthread+0x1b0/0x1d0
[<ffffffc0000864d0>] ret_from_fork+0x10/0x40

Replace it to raw version to prevent from the race condition.

Signed-off-by: Yang Shi <yang.shi@linaro.org>
---
 arch/arm64/kernel/traps.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index cbedd72..9abe236 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -146,7 +146,7 @@ static void dump_instr(const char *lvl, struct pt_regs *regs)
 static void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
 {
 	struct stackframe frame;
-	unsigned long irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
+	unsigned long irq_stack_ptr = IRQ_STACK_PTR(raw_smp_processor_id());
 	int skip;
 
 	pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk);
-- 
2.0.2

[toc] | [next] | [standalone]

#1331037

On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote:
> dump_backtrace may be called in kthread context, which is not bound to a single
> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug
> report:

If we're preemptible here, it means that our irq_stack_ptr is potentially
bogus. Whilst this isn't an issue for kthreads, it does feel like we
could make this slightly more robust in the face of potential frame
corruption. Maybe just zero the IRQ stack pointer if we're in preemptible
context?

Will

> BUG: using smp_processor_id() in preemptible [00000000] code: khungtaskd/71
> caller is debug_smp_processor_id+0x1c/0x28
> CPU: 3 PID: 71 Comm: khungtaskd Not tainted 4.5.0-rc1 #144
> Hardware name: Freescale Layerscape 2085a RDB Board (DT)
> Call trace:
> [<ffffffc00008d120>] dump_backtrace+0x0/0x290
> [<ffffffc00008d3d4>] show_stack+0x24/0x30
> [<ffffffc0006bfd1c>] dump_stack+0x8c/0xd8
> [<ffffffc0006fe56c>] check_preemption_disabled+0x184/0x188
> [<ffffffc0006fe58c>] debug_smp_processor_id+0x1c/0x28
> [<ffffffc00008d1a4>] dump_backtrace+0x84/0x290
> [<ffffffc00008d3d4>] show_stack+0x24/0x30
> [<ffffffc00012aa5c>] sched_show_task+0x16c/0x280
> [<ffffffc0001f3df8>] watchdog+0x560/0x708
> [<ffffffc000111bb8>] kthread+0x1b0/0x1d0
> [<ffffffc0000864d0>] ret_from_fork+0x10/0x40
> 
> Replace it to raw version to prevent from the race condition.
> 
> Signed-off-by: Yang Shi <yang.shi@linaro.org>
> ---
>  arch/arm64/kernel/traps.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index cbedd72..9abe236 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -146,7 +146,7 @@ static void dump_instr(const char *lvl, struct pt_regs *regs)
>  static void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
>  {
>  	struct stackframe frame;
> -	unsigned long irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
> +	unsigned long irq_stack_ptr = IRQ_STACK_PTR(raw_smp_processor_id());
>  	int skip;
>  
>  	pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk);
> -- 
> 2.0.2
> 

[toc] | [prev] | [next] | [standalone]

#1331093

On 10/02/16 10:29, Will Deacon wrote:
> On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote:
>> dump_backtrace may be called in kthread context, which is not bound to a single
>> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug
>> report:
> 
> If we're preemptible here, it means that our irq_stack_ptr is potentially
> bogus. Whilst this isn't an issue for kthreads, it does feel like we
> could make this slightly more robust in the face of potential frame
> corruption. Maybe just zero the IRQ stack pointer if we're in preemptible
> context?

Switching between stacks is only valid if we are tracing ourselves while on the
irq_stack, we should probably prevent it for other tasks too.

Something like (untested):
---------------------
if (tsk == current && in_atomic())
	irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
else
	irq_stack_ptr = 0;
---------------------

This would work when we trace ourselves while on the irq_stack, but break*
tracing a running task on a remote cpu (khungtaskd doesn't do this).

The same fix would apply to unwind_frame(), we have 'tsk' in both functions.

Thoughts?


James


* If this were to ever happen, we would fail to switch to the original stack if
  tracing a remote irq stack, and print the wrong exception args. I'm not aware
  of any users of this, (panic() sends an IPI).

[toc] | [prev] | [next] | [standalone]

#1331106

On Wed, Feb 10, 2016 at 11:52:31AM +0000, James Morse wrote:
> On 10/02/16 10:29, Will Deacon wrote:
> > On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote:
> >> dump_backtrace may be called in kthread context, which is not bound to a single
> >> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug
> >> report:
> > 
> > If we're preemptible here, it means that our irq_stack_ptr is potentially
> > bogus. Whilst this isn't an issue for kthreads, it does feel like we
> > could make this slightly more robust in the face of potential frame
> > corruption. Maybe just zero the IRQ stack pointer if we're in preemptible
> > context?
> 
> Switching between stacks is only valid if we are tracing ourselves while on the
> irq_stack, we should probably prevent it for other tasks too.
> 
> Something like (untested):
> ---------------------
> if (tsk == current && in_atomic())
> 	irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
> else
> 	irq_stack_ptr = 0;
> ---------------------
> 
> This would work when we trace ourselves while on the irq_stack, but break*
> tracing a running task on a remote cpu (khungtaskd doesn't do this).
> 
> The same fix would apply to unwind_frame(), we have 'tsk' in both functions.
> 
> Thoughts?

in_atomic is a misnomer:

  https://lwn.net/Articles/274695/

;)

So we might be better off zeroing the pointer if tsk != current ||
preemptible(). But yeah, I think we're in general agreement about this.

Will

[toc] | [prev] | [next] | [standalone]

#1331420

On 2/10/2016 4:10 AM, Will Deacon wrote:
> On Wed, Feb 10, 2016 at 11:52:31AM +0000, James Morse wrote:
>> On 10/02/16 10:29, Will Deacon wrote:
>>> On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote:
>>>> dump_backtrace may be called in kthread context, which is not bound to a single
>>>> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug
>>>> report:
>>>
>>> If we're preemptible here, it means that our irq_stack_ptr is potentially
>>> bogus. Whilst this isn't an issue for kthreads, it does feel like we
>>> could make this slightly more robust in the face of potential frame
>>> corruption. Maybe just zero the IRQ stack pointer if we're in preemptible
>>> context?
>>
>> Switching between stacks is only valid if we are tracing ourselves while on the
>> irq_stack, we should probably prevent it for other tasks too.
>>
>> Something like (untested):
>> ---------------------
>> if (tsk == current && in_atomic())
>> 	irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());

One follow up question, is it possible to have both tsk != current and 
on_irq_stack is true at the same time? If it is possible, this may be a 
problem in unwind_frame called by profile_pc which has tsk being NULL.

Thanks,
Yang

>> else
>> 	irq_stack_ptr = 0;
>> ---------------------
>>
>> This would work when we trace ourselves while on the irq_stack, but break*
>> tracing a running task on a remote cpu (khungtaskd doesn't do this).
>>
>> The same fix would apply to unwind_frame(), we have 'tsk' in both functions.
>>
>> Thoughts?
>
> in_atomic is a misnomer:
>
>    https://lwn.net/Articles/274695/
>
> ;)
>
> So we might be better off zeroing the pointer if tsk != current ||
> preemptible(). But yeah, I think we're in general agreement about this.
>
> Will
>

[toc] | [prev] | [next] | [standalone]

#1331812

Hi!

On 10/02/16 18:12, Shi, Yang wrote:
> On 2/10/2016 4:10 AM, Will Deacon wrote:
>> On Wed, Feb 10, 2016 at 11:52:31AM +0000, James Morse wrote:
>>> On 10/02/16 10:29, Will Deacon wrote:
>>>> On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote:
>>>>> dump_backtrace may be called in kthread context, which is not bound to a
>>>>> single
>>>>> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug
>>>>> report:
>>>>
>>>> If we're preemptible here, it means that our irq_stack_ptr is potentially
>>>> bogus. Whilst this isn't an issue for kthreads, it does feel like we
>>>> could make this slightly more robust in the face of potential frame
>>>> corruption. Maybe just zero the IRQ stack pointer if we're in preemptible
>>>> context?
>>>
>>> Switching between stacks is only valid if we are tracing ourselves while on the
>>> irq_stack, we should probably prevent it for other tasks too.
>>>
>>> Something like (untested):
>>> ---------------------
>>> if (tsk == current && in_atomic())
>>>     irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
> 
> One follow up question, is it possible to have both tsk != current and
> on_irq_stack is true at the same time?

No. If you are tracing an irq stack, it must be your own stack.

If this weren't the case, it would be the stack of a running task on a remote
CPU, and you would be racing with the remote CPU changing the values you are
reading. Fortunately nothing tries to do this.

(The third case would be tracing a sleeping irq stack - this doesn't happen
 either, as we switch back to the original stack before calling schedule()).


> If it is possible, this may be a problem
> in unwind_frame called by profile_pc which has tsk being NULL.

Ah, well spotted. I guess there should also be a != NULL comparison thrown into
the mix. I don't think it will be a problem for profile_pc() as it should always
find a !in_lock_functions() frame before it needs to switch stack, (which we are
preventing it from doing). If this ever did happen, it will return 0.


Thanks,

James

[toc] | [prev] | [next] | [standalone]

#1332265

On 2/11/2016 2:41 AM, James Morse wrote:
> Hi!
>
> On 10/02/16 18:12, Shi, Yang wrote:
>> On 2/10/2016 4:10 AM, Will Deacon wrote:
>>> On Wed, Feb 10, 2016 at 11:52:31AM +0000, James Morse wrote:
>>>> On 10/02/16 10:29, Will Deacon wrote:
>>>>> On Tue, Feb 09, 2016 at 01:26:22PM -0800, Yang Shi wrote:
>>>>>> dump_backtrace may be called in kthread context, which is not bound to a
>>>>>> single
>>>>>> cpu, i.e. khungtaskd, then calling smp_processor_id may trigger the below bug
>>>>>> report:
>>>>>
>>>>> If we're preemptible here, it means that our irq_stack_ptr is potentially
>>>>> bogus. Whilst this isn't an issue for kthreads, it does feel like we
>>>>> could make this slightly more robust in the face of potential frame
>>>>> corruption. Maybe just zero the IRQ stack pointer if we're in preemptible
>>>>> context?
>>>>
>>>> Switching between stacks is only valid if we are tracing ourselves while on the
>>>> irq_stack, we should probably prevent it for other tasks too.
>>>>
>>>> Something like (untested):
>>>> ---------------------
>>>> if (tsk == current && in_atomic())
>>>>      irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
>>
>> One follow up question, is it possible to have both tsk != current and
>> on_irq_stack is true at the same time?
>
> No. If you are tracing an irq stack, it must be your own stack.
>
> If this weren't the case, it would be the stack of a running task on a remote
> CPU, and you would be racing with the remote CPU changing the values you are
> reading. Fortunately nothing tries to do this.
>
> (The third case would be tracing a sleeping irq stack - this doesn't happen
>   either, as we switch back to the original stack before calling schedule()).
>
>
>> If it is possible, this may be a problem
>> in unwind_frame called by profile_pc which has tsk being NULL.
>
> Ah, well spotted. I guess there should also be a != NULL comparison thrown into
> the mix. I don't think it will be a problem for profile_pc() as it should always
> find a !in_lock_functions() frame before it needs to switch stack, (which we are
> preventing it from doing). If this ever did happen, it will return 0.

Thanks for the elaboration. I changed the logic a little bit to:

if (tsk == current && !preemptible())
                 irq_stack_ptr = IRQ_STACK_PTR(smp_processor_id());
else
                 irq_stack_ptr = 0;

In this way, the NULL pointer will be covered by "else" too.

v2 patch will be sent out soon once I'm done some smoke testing.

Yang

>
>
> Thanks,
>
> James
>

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web