Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1590521 > unrolled thread

Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP on cpu 1

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Thomas Gleixner <tglx@linutronix.de> - 2017-03-01 18:10 +0100
    Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Thomas Gleixner <tglx@linutronix.de> - 2017-03-02 02:40 +0100
      Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Thomas Gleixner <tglx@linutronix.de> - 2017-03-02 19:10 +0100
        Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Meelis Roos <mroos@linux.ee> - 2017-03-03 02:40 +0100
        Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Meelis Roos <mroos@linux.ee> - 2017-03-03 12:00 +0100
        Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Thomas Gleixner <tglx@linutronix.de> - 2017-03-03 14:10 +0100
          Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Meelis Roos <mroos@linux.ee> - 2017-03-05 21:50 +0100
            Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Frederic Weisbecker <fweisbec@gmail.com> - 2017-03-06 00:30 +0100
              Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Meelis Roos <mroos@linux.ee> - 2017-03-06 12:50 +0100
    Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP  on cpu 1 Meelis Roos <mroos@linux.ee> - 2017-03-02 04:40 +0100

#1590521 — Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP on cpu 1

On Wed, 1 Mar 2017, Meelis Roos wrote:

> This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> different, all the others were from arch_cpu_idle. Sometime ecey couple 
> of seconds (after some activity), sometimes nothing for a long time 
> (idle, no SSH logins).

The only watchdog related patch which hit after 4.10 is:

 8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread

Can you try to revert that for a start? I'm not seeing why it should be the
culprit from a quick glance, but ...

Thanks,

	tglx

[toc] | [next] | [standalone]

#1590784

On Thu, 2 Mar 2017, Meelis Roos wrote:

> > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> > > problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> > > CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> > > different, all the others were from arch_cpu_idle. Sometime ecey couple 
> > > of seconds (after some activity), sometimes nothing for a long time 
> > > (idle, no SSH logins).
> > 
> > The only watchdog related patch which hit after 4.10 is:
> > 
> >  8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> > 
> > Can you try to revert that for a start? I'm not seeing why it should be the
> > culprit from a quick glance, but ...
> 
> Reverting this patch does not help.

I did not expect that, but excluding it was a valid shot in the
dark. Thanmks for trying.

To be honest, I have no idea what causes that at the moment, but I will
come back to you tomorrow after thinking it through (with brain awake) how
to debug this.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1591366

On Wed, 1 Mar 2017, Thomas Gleixner wrote:
> On Thu, 2 Mar 2017, Meelis Roos wrote:
> 
> > > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> > > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> > > > problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> > > > CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> > > > different, all the others were from arch_cpu_idle. Sometime ecey couple 
> > > > of seconds (after some activity), sometimes nothing for a long time 
> > > > (idle, no SSH logins).
> > > 
> > > The only watchdog related patch which hit after 4.10 is:
> > > 
> > >  8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> > > 
> > > Can you try to revert that for a start? I'm not seeing why it should be the
> > > culprit from a quick glance, but ...
> > 
> > Reverting this patch does not help.
> 
> I did not expect that, but excluding it was a valid shot in the
> dark. Thanmks for trying.
> 
> To be honest, I have no idea what causes that at the moment, but I will
> come back to you tomorrow after thinking it through (with brain awake) how
> to debug this.

Went through the related changes which came in during the merge window. One
which affects the per cpu timers is: 914122c389d0

Can you try to revert that one please?

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1591609

> > > > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> > > > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> > > > > problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> > > > > CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> > > > > different, all the others were from arch_cpu_idle. Sometime ecey couple 
> > > > > of seconds (after some activity), sometimes nothing for a long time 
> > > > > (idle, no SSH logins).
> > > > 
> > > > The only watchdog related patch which hit after 4.10 is:
> > > > 
> > > >  8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> > > > 
> > > > Can you try to revert that for a start? I'm not seeing why it should be the
> > > > culprit from a quick glance, but ...
> > > 
> > > Reverting this patch does not help.
> > 
> > I did not expect that, but excluding it was a valid shot in the
> > dark. Thanmks for trying.
> > 
> > To be honest, I have no idea what causes that at the moment, but I will
> > come back to you tomorrow after thinking it through (with brain awake) how
> > to debug this.
> 
> Went through the related changes which came in during the merge window. One
> which affects the per cpu timers is: 914122c389d0
> 
> Can you try to revert that one please?

Reverted, no effect.

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [next] | [standalone]

#1591858

> On Thu, 2 Mar 2017, Thomas Gleixner wrote:
> > On Wed, 1 Mar 2017, Thomas Gleixner wrote:
> > > On Thu, 2 Mar 2017, Meelis Roos wrote:
> > > 
> > > > > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> > > > > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> > > > > > problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> > > > > > CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> > > > > > different, all the others were from arch_cpu_idle. Sometime ecey couple 
> > > > > > of seconds (after some activity), sometimes nothing for a long time 
> > > > > > (idle, no SSH logins).
> > > > > 
> > > > > The only watchdog related patch which hit after 4.10 is:
> > > > > 
> > > > >  8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> > > > > 
> > > > > Can you try to revert that for a start? I'm not seeing why it should be the
> > > > > culprit from a quick glance, but ...
> > > > 
> > > > Reverting this patch does not help.
> > > 
> > > I did not expect that, but excluding it was a valid shot in the
> > > dark. Thanmks for trying.
> > > 
> > > To be honest, I have no idea what causes that at the moment, but I will
> > > come back to you tomorrow after thinking it through (with brain awake) how
> > > to debug this.
> > 
> > Went through the related changes which came in during the merge window. One
> > which affects the per cpu timers is: 914122c389d0
> > 
> > Can you try to revert that one please?
> 
> Running out of obvious culprits. Any chance that you can do a bisect or
> this too painful on that box?

I started bisect yesterday on the P4 that has one similar occurrance.

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [next] | [standalone]

#1591925

On Thu, 2 Mar 2017, Thomas Gleixner wrote:
> On Wed, 1 Mar 2017, Thomas Gleixner wrote:
> > On Thu, 2 Mar 2017, Meelis Roos wrote:
> > 
> > > > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> > > > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> > > > > problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> > > > > CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> > > > > different, all the others were from arch_cpu_idle. Sometime ecey couple 
> > > > > of seconds (after some activity), sometimes nothing for a long time 
> > > > > (idle, no SSH logins).
> > > > 
> > > > The only watchdog related patch which hit after 4.10 is:
> > > > 
> > > >  8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> > > > 
> > > > Can you try to revert that for a start? I'm not seeing why it should be the
> > > > culprit from a quick glance, but ...
> > > 
> > > Reverting this patch does not help.
> > 
> > I did not expect that, but excluding it was a valid shot in the
> > dark. Thanmks for trying.
> > 
> > To be honest, I have no idea what causes that at the moment, but I will
> > come back to you tomorrow after thinking it through (with brain awake) how
> > to debug this.
> 
> Went through the related changes which came in during the merge window. One
> which affects the per cpu timers is: 914122c389d0
> 
> Can you try to revert that one please?

Running out of obvious culprits. Any chance that you can do a bisect or
this too painful on that box?

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1592839

Added some CC-s because of bisect find. Whole context should be still 
here.

> > > > > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> > > > > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> > > > > > problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> > > > > > CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> > > > > > different, all the others were from arch_cpu_idle. Sometime ecey couple 
> > > > > > of seconds (after some activity), sometimes nothing for a long time 
> > > > > > (idle, no SSH logins).
> > > > > 
> > > > > The only watchdog related patch which hit after 4.10 is:
> > > > > 
> > > > >  8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> > > > > 
> > > > > Can you try to revert that for a start? I'm not seeing why it should be the
> > > > > culprit from a quick glance, but ...
> > > > 
> > > > Reverting this patch does not help.
> > > 
> > > I did not expect that, but excluding it was a valid shot in the
> > > dark. Thanmks for trying.
> > > 
> > > To be honest, I have no idea what causes that at the moment, but I will
> > > come back to you tomorrow after thinking it through (with brain awake) how
> > > to debug this.
> > 
> > Went through the related changes which came in during the merge window. One
> > which affects the per cpu timers is: 914122c389d0
> > 
> > Can you try to revert that one please?
> 
> Running out of obvious culprits. Any chance that you can do a bisect or
> this too painful on that box?

Done on a P4 where the problem also appeared. The bisecting resulted in 
this commit. Does it seem realistic? I will also try if this help son 
the old PPro.

93825f2ec736f30e034ab7c9d56b42849c5b00da is the first bad commit
commit 93825f2ec736f30e034ab7c9d56b42849c5b00da
Author: Frederic Weisbecker <fweisbec@gmail.com>
Date:   Tue Jan 31 04:09:16 2017 +0100

    jiffies: Reuse TICK_NSEC instead of NSEC_PER_JIFFY
    
    NSEC_PER_JIFFY is an ad-hoc redefinition of TICK_NSEC. Let's rather
    use a unique and well maintained version.
    
    Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
    Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
    Cc: Paul Mackerras <paulus@samba.org>
    Cc: Michael Ellerman <mpe@ellerman.id.au>
    Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
    Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
    Cc: Tony Luck <tony.luck@intel.com>
    Cc: Fenghua Yu <fenghua.yu@intel.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: Stanislaw Gruszka <sgruszka@redhat.com>
    Cc: Wanpeng Li <wanpeng.li@hotmail.com>
    Link: http://lkml.kernel.org/r/1485832191-26889-1-git-send-email-fweisbec@gmail.com
    Signed-off-by: Ingo Molnar <mingo@kernel.org>

:040000 040000 219afc6bb9e757532791f93ed05ddf68a7124cb0 fb9cc12366f970c1bc4872cc38b1c4df5ce9594d M      kernel

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [next] | [standalone]

#1592860

On Sun, Mar 05, 2017 at 10:48:50PM +0200, Meelis Roos wrote:
> Added some CC-s because of bisect find. Whole context should be still 
> here.
> 
> > > > > > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> > > > > > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> > > > > > > problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> > > > > > > CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> > > > > > > different, all the others were from arch_cpu_idle. Sometime ecey couple 
> > > > > > > of seconds (after some activity), sometimes nothing for a long time 
> > > > > > > (idle, no SSH logins).
> > > > > > 
> > > > > > The only watchdog related patch which hit after 4.10 is:
> > > > > > 
> > > > > >  8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> > > > > > 
> > > > > > Can you try to revert that for a start? I'm not seeing why it should be the
> > > > > > culprit from a quick glance, but ...
> > > > > 
> > > > > Reverting this patch does not help.
> > > > 
> > > > I did not expect that, but excluding it was a valid shot in the
> > > > dark. Thanmks for trying.
> > > > 
> > > > To be honest, I have no idea what causes that at the moment, but I will
> > > > come back to you tomorrow after thinking it through (with brain awake) how
> > > > to debug this.
> > > 
> > > Went through the related changes which came in during the merge window. One
> > > which affects the per cpu timers is: 914122c389d0
> > > 
> > > Can you try to revert that one please?
> > 
> > Running out of obvious culprits. Any chance that you can do a bisect or
> > this too painful on that box?
> 
> Done on a P4 where the problem also appeared. The bisecting resulted in 
> this commit. Does it seem realistic? I will also try if this help son 
> the old PPro.
> 
> 93825f2ec736f30e034ab7c9d56b42849c5b00da is the first bad commit
> commit 93825f2ec736f30e034ab7c9d56b42849c5b00da
> Author: Frederic Weisbecker <fweisbec@gmail.com>
> Date:   Tue Jan 31 04:09:16 2017 +0100
> 
>     jiffies: Reuse TICK_NSEC instead of NSEC_PER_JIFFY
>     
>     NSEC_PER_JIFFY is an ad-hoc redefinition of TICK_NSEC. Let's rather
>     use a unique and well maintained version.
>     
>     Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
>     Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
>     Cc: Paul Mackerras <paulus@samba.org>
>     Cc: Michael Ellerman <mpe@ellerman.id.au>
>     Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
>     Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
>     Cc: Tony Luck <tony.luck@intel.com>
>     Cc: Fenghua Yu <fenghua.yu@intel.com>
>     Cc: Peter Zijlstra <peterz@infradead.org>
>     Cc: Rik van Riel <riel@redhat.com>
>     Cc: Stanislaw Gruszka <sgruszka@redhat.com>
>     Cc: Wanpeng Li <wanpeng.li@hotmail.com>
>     Link: http://lkml.kernel.org/r/1485832191-26889-1-git-send-email-fweisbec@gmail.com
>     Signed-off-by: Ingo Molnar <mingo@kernel.org>

Ouch, looking at that patch again, I probably had a delusional moment when I wrote this:

diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
index a4a0e47..7906b3f 100644
--- a/kernel/time/jiffies.c
+++ b/kernel/time/jiffies.c
@@ -125,7 +125,7 @@ int register_refined_jiffies(long cycles_per_second)
 	shift_hz += cycles_per_tick/2;
 	do_div(shift_hz, cycles_per_tick);
 	/* Calculate nsec_per_tick using shift_hz */
-	nsec_per_tick = (u64)NSEC_PER_SEC << 8;
+	nsec_per_tick = (u64)TICK_NSEC << 8;
 	nsec_per_tick += (u32)shift_hz/2;
 	do_div(nsec_per_tick, (u32)shift_hz);
 


Could you please retry after reverting this specific chunk? (that would be the very fix
to apply).

Thanks!

[toc] | [prev] | [next] | [standalone]

#1593246

> On Sun, Mar 05, 2017 at 10:48:50PM +0200, Meelis Roos wrote:
> > Added some CC-s because of bisect find. Whole context should be still 
> > here.
> > 
> > > > > > > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> > > > > > > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> > > > > > > > problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> > > > > > > > CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> > > > > > > > different, all the others were from arch_cpu_idle. Sometime ecey couple 
> > > > > > > > of seconds (after some activity), sometimes nothing for a long time 
> > > > > > > > (idle, no SSH logins).
> > > > > > > 
> > > > > > > The only watchdog related patch which hit after 4.10 is:
> > > > > > > 
> > > > > > >  8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> > > > > > > 
> > > > > > > Can you try to revert that for a start? I'm not seeing why it should be the
> > > > > > > culprit from a quick glance, but ...
> > > > > > 
> > > > > > Reverting this patch does not help.
> > > > > 
> > > > > I did not expect that, but excluding it was a valid shot in the
> > > > > dark. Thanmks for trying.
> > > > > 
> > > > > To be honest, I have no idea what causes that at the moment, but I will
> > > > > come back to you tomorrow after thinking it through (with brain awake) how
> > > > > to debug this.
> > > > 
> > > > Went through the related changes which came in during the merge window. One
> > > > which affects the per cpu timers is: 914122c389d0
> > > > 
> > > > Can you try to revert that one please?
> > > 
> > > Running out of obvious culprits. Any chance that you can do a bisect or
> > > this too painful on that box?
> > 
> > Done on a P4 where the problem also appeared. The bisecting resulted in 
> > this commit. Does it seem realistic? I will also try if this help son 
> > the old PPro.
> > 
> > 93825f2ec736f30e034ab7c9d56b42849c5b00da is the first bad commit
> > commit 93825f2ec736f30e034ab7c9d56b42849c5b00da
> > Author: Frederic Weisbecker <fweisbec@gmail.com>
> > Date:   Tue Jan 31 04:09:16 2017 +0100
> > 
> >     jiffies: Reuse TICK_NSEC instead of NSEC_PER_JIFFY

[...]
> Ouch, looking at that patch again, I probably had a delusional moment when I wrote this:
> 
> diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
> index a4a0e47..7906b3f 100644
> --- a/kernel/time/jiffies.c
> +++ b/kernel/time/jiffies.c
> @@ -125,7 +125,7 @@ int register_refined_jiffies(long cycles_per_second)
>  	shift_hz += cycles_per_tick/2;
>  	do_div(shift_hz, cycles_per_tick);
>  	/* Calculate nsec_per_tick using shift_hz */
> -	nsec_per_tick = (u64)NSEC_PER_SEC << 8;
> +	nsec_per_tick = (u64)TICK_NSEC << 8;
>  	nsec_per_tick += (u32)shift_hz/2;
>  	do_div(nsec_per_tick, (u32)shift_hz);
>  
> 
> 
> Could you please retry after reverting this specific chunk? (that would be the very fix
> to apply).

Reverting this chunk fixed both the PPro and the P4.

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [next] | [standalone]

#1590826

> > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, 
> > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a 
> > problem. Ocassionally NMI watchdog kicks in and discovers one of the 
> > CPUs in LOCKUP. The system keeps running fine. The first lockup was 
> > different, all the others were from arch_cpu_idle. Sometime ecey couple 
> > of seconds (after some activity), sometimes nothing for a long time 
> > (idle, no SSH logins).
> 
> The only watchdog related patch which hit after 4.10 is:
> 
>  8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread
> 
> Can you try to revert that for a start? I'm not seeing why it should be the
> culprit from a quick glance, but ...

Reverting this patch does not help.

-- 
Meelis Roos (mroos@linux.ee)

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web