Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1313528 > unrolled thread

Crashes with 874bbfe600a6 in 3.18.25

Back to article view | Back to linux.kernel

Contents

  Crashes with 874bbfe600a6 in 3.18.25 Jan Kara <jack@suse.cz> - 2016-01-20 22:20 +0100
    Re: Crashes with 874bbfe600a6 in 3.18.25 Shaohua Li <shli@fb.com> - 2016-01-20 22:40 +0100
      Re: Crashes with 874bbfe600a6 in 3.18.25 Jan Kara <jack@suse.cz> - 2016-01-21 11:00 +0100
        Re: Crashes with 874bbfe600a6 in 3.18.25 Sasha Levin <sasha.levin@oracle.com> - 2016-01-21 14:40 +0100
        Re: Crashes with 874bbfe600a6 in 3.18.25 Sasha Levin <sasha.levin@oracle.com> - 2016-01-22 02:20 +0100
          Re: Crashes with 874bbfe600a6 in 3.18.25 Tejun Heo <tj@kernel.org> - 2016-01-22 17:10 +0100
            Re: Crashes with 874bbfe600a6 in 3.18.25 Ben Hutchings <ben@decadent.org.uk> - 2016-01-23 03:30 +0100
              Re: Crashes with 874bbfe600a6 in 3.18.25 Thomas Gleixner <tglx@linutronix.de> - 2016-01-23 17:20 +0100
                Re: Crashes with 874bbfe600a6 in 3.18.25 Jan Kara <jack@suse.cz> - 2016-01-26 10:40 +0100
                  Re: Crashes with 874bbfe600a6 in 3.18.25 Thomas Gleixner <tglx@linutronix.de> - 2016-01-26 11:00 +0100
                  Re: Crashes with 874bbfe600a6 in 3.18.25 Petr Mladek <pmladek@suse.com> - 2016-01-26 12:20 +0100
                    Re: Crashes with 874bbfe600a6 in 3.18.25 Thomas Gleixner <tglx@linutronix.de> - 2016-01-26 14:20 +0100

#1313528 — Crashes with 874bbfe600a6 in 3.18.25

[Multipart message — attachments visible in raw view] — view raw

Hello,

a friend of mine started seeing crashes with 3.18.25 kernel - once
appropriate load is put on the machine it crashes within minutes. He
tracked down that reverting commit 874bbfe600a6 (this is the commit ID from
Linus' tree, in stable tree the commit ID is 1e7af294dd03) "workqueue: make
sure delayed work run in local cpu" makes the kernel stable again. I'm
attaching screenshot of the crash - sadly the initial part is missing but
it seems that we crashed when processing timers on otherwise idle CPU. This
is a production machine so experimentation is not easy but if we really
need more information it may be possible to reproduce the issue again and
gather it.

Anyone has idea what is going on? I was looking into the code for a while
but so far I have no good explanation.  It would be good to understand the
cause instead of just blindly reverting the commit from stable tree...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [next] | [standalone]

#1313531

On Wed, Jan 20, 2016 at 10:19:26PM +0100, Jan Kara wrote:
> Hello,
> 
> a friend of mine started seeing crashes with 3.18.25 kernel - once
> appropriate load is put on the machine it crashes within minutes. He
> tracked down that reverting commit 874bbfe600a6 (this is the commit ID from
> Linus' tree, in stable tree the commit ID is 1e7af294dd03) "workqueue: make
> sure delayed work run in local cpu" makes the kernel stable again. I'm
> attaching screenshot of the crash - sadly the initial part is missing but
> it seems that we crashed when processing timers on otherwise idle CPU. This
> is a production machine so experimentation is not easy but if we really
> need more information it may be possible to reproduce the issue again and
> gather it.
> 
> Anyone has idea what is going on? I was looking into the code for a while
> but so far I have no good explanation.  It would be good to understand the
> cause instead of just blindly reverting the commit from stable tree...

Tejun fixed a bug in timer: 22b886dd10180939. is it included in 3.18.25?

Thanks,
Shaohua

[toc] | [prev] | [next] | [standalone]

#1314055

On Wed 20-01-16 13:39:01, Shaohua Li wrote:
> On Wed, Jan 20, 2016 at 10:19:26PM +0100, Jan Kara wrote:
> > Hello,
> > 
> > a friend of mine started seeing crashes with 3.18.25 kernel - once
> > appropriate load is put on the machine it crashes within minutes. He
> > tracked down that reverting commit 874bbfe600a6 (this is the commit ID from
> > Linus' tree, in stable tree the commit ID is 1e7af294dd03) "workqueue: make
> > sure delayed work run in local cpu" makes the kernel stable again. I'm
> > attaching screenshot of the crash - sadly the initial part is missing but
> > it seems that we crashed when processing timers on otherwise idle CPU. This
> > is a production machine so experimentation is not easy but if we really
> > need more information it may be possible to reproduce the issue again and
> > gather it.
> > 
> > Anyone has idea what is going on? I was looking into the code for a while
> > but so far I have no good explanation.  It would be good to understand the
> > cause instead of just blindly reverting the commit from stable tree...
> 
> Tejun fixed a bug in timer: 22b886dd10180939. is it included in 3.18.25?

That doesn't seem to be included in 3.18-stable although it was CCed to stable.
Sasha?

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1314197

On 01/21/2016 04:52 AM, Jan Kara wrote:
> On Wed 20-01-16 13:39:01, Shaohua Li wrote:
>> > On Wed, Jan 20, 2016 at 10:19:26PM +0100, Jan Kara wrote:
>>> > > Hello,
>>> > > 
>>> > > a friend of mine started seeing crashes with 3.18.25 kernel - once
>>> > > appropriate load is put on the machine it crashes within minutes. He
>>> > > tracked down that reverting commit 874bbfe600a6 (this is the commit ID from
>>> > > Linus' tree, in stable tree the commit ID is 1e7af294dd03) "workqueue: make
>>> > > sure delayed work run in local cpu" makes the kernel stable again. I'm
>>> > > attaching screenshot of the crash - sadly the initial part is missing but
>>> > > it seems that we crashed when processing timers on otherwise idle CPU. This
>>> > > is a production machine so experimentation is not easy but if we really
>>> > > need more information it may be possible to reproduce the issue again and
>>> > > gather it.
>>> > > 
>>> > > Anyone has idea what is going on? I was looking into the code for a while
>>> > > but so far I have no good explanation.  It would be good to understand the
>>> > > cause instead of just blindly reverting the commit from stable tree...
>> > 
>> > Tejun fixed a bug in timer: 22b886dd10180939. is it included in 3.18.25?
> That doesn't seem to be included in 3.18-stable although it was CCed to stable.
> Sasha?

Yup, that's not in. I'll include it and release it along with 3.18.26 on the weekend.


Thanks,
Sasha

[toc] | [prev] | [next] | [standalone]

#1314682

On 01/21/2016 04:52 AM, Jan Kara wrote:
> On Wed 20-01-16 13:39:01, Shaohua Li wrote:
>> On Wed, Jan 20, 2016 at 10:19:26PM +0100, Jan Kara wrote:
>>> Hello,
>>>
>>> a friend of mine started seeing crashes with 3.18.25 kernel - once
>>> appropriate load is put on the machine it crashes within minutes. He
>>> tracked down that reverting commit 874bbfe600a6 (this is the commit ID from
>>> Linus' tree, in stable tree the commit ID is 1e7af294dd03) "workqueue: make
>>> sure delayed work run in local cpu" makes the kernel stable again. I'm
>>> attaching screenshot of the crash - sadly the initial part is missing but
>>> it seems that we crashed when processing timers on otherwise idle CPU. This
>>> is a production machine so experimentation is not easy but if we really
>>> need more information it may be possible to reproduce the issue again and
>>> gather it.
>>>
>>> Anyone has idea what is going on? I was looking into the code for a while
>>> but so far I have no good explanation.  It would be good to understand the
>>> cause instead of just blindly reverting the commit from stable tree...
>>
>> Tejun fixed a bug in timer: 22b886dd10180939. is it included in 3.18.25?
> 
> That doesn't seem to be included in 3.18-stable although it was CCed to stable.
> Sasha?

Looks like it requires more than trivial backport (I think). Tejun?


Thanks,
Sasha

[toc] | [prev] | [next] | [standalone]

#1315097

(cc'ing Thomas)

On Thu, Jan 21, 2016 at 08:10:20PM -0500, Sasha Levin wrote:
> On 01/21/2016 04:52 AM, Jan Kara wrote:
> > On Wed 20-01-16 13:39:01, Shaohua Li wrote:
> >> On Wed, Jan 20, 2016 at 10:19:26PM +0100, Jan Kara wrote:
> >>> Hello,
> >>>
> >>> a friend of mine started seeing crashes with 3.18.25 kernel - once
> >>> appropriate load is put on the machine it crashes within minutes. He
> >>> tracked down that reverting commit 874bbfe600a6 (this is the commit ID from
> >>> Linus' tree, in stable tree the commit ID is 1e7af294dd03) "workqueue: make
> >>> sure delayed work run in local cpu" makes the kernel stable again. I'm
> >>> attaching screenshot of the crash - sadly the initial part is missing but
> >>> it seems that we crashed when processing timers on otherwise idle CPU. This
> >>> is a production machine so experimentation is not easy but if we really
> >>> need more information it may be possible to reproduce the issue again and
> >>> gather it.
> >>>
> >>> Anyone has idea what is going on? I was looking into the code for a while
> >>> but so far I have no good explanation.  It would be good to understand the
> >>> cause instead of just blindly reverting the commit from stable tree...
> >>
> >> Tejun fixed a bug in timer: 22b886dd10180939. is it included in 3.18.25?
> > 
> > That doesn't seem to be included in 3.18-stable although it was CCed to stable.
> > Sasha?
> 
> Looks like it requires more than trivial backport (I think). Tejun?

The timer migration has changed quite a bit.  Given that we've never
seen vmstat work crashing in 3.18 era, I wonder whether the right
thing to do here is reverting 874bbfe600a6 from 3.18 stable?

Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1315488

[Multipart message — attachments visible in raw view] — view raw

On Fri, 2016-01-22 at 11:09 -0500, Tejun Heo wrote:
> (cc'ing Thomas)
> 
> On Thu, Jan 21, 2016 at 08:10:20PM -0500, Sasha Levin wrote:
> > On 01/21/2016 04:52 AM, Jan Kara wrote:
> > > On Wed 20-01-16 13:39:01, Shaohua Li wrote:
> > > > On Wed, Jan 20, 2016 at 10:19:26PM +0100, Jan Kara wrote:
> > > > > Hello,
> > > > > 
> > > > > a friend of mine started seeing crashes with 3.18.25 kernel - once
> > > > > appropriate load is put on the machine it crashes within minutes. He
> > > > > tracked down that reverting commit 874bbfe600a6 (this is the commit ID from
> > > > > Linus' tree, in stable tree the commit ID is 1e7af294dd03) "workqueue: make
> > > > > sure delayed work run in local cpu" makes the kernel stable again. I'm
> > > > > attaching screenshot of the crash - sadly the initial part is missing but
> > > > > it seems that we crashed when processing timers on otherwise idle CPU. This
> > > > > is a production machine so experimentation is not easy but if we really
> > > > > need more information it may be possible to reproduce the issue again and
> > > > > gather it.
> > > > > 
> > > > > Anyone has idea what is going on? I was looking into the code for a while
> > > > > but so far I have no good explanation.  It would be good to understand the
> > > > > cause instead of just blindly reverting the commit from stable tree...
> > > > 
> > > > Tejun fixed a bug in timer: 22b886dd10180939. is it included in 3.18.25?
> > > 
> > > That doesn't seem to be included in 3.18-stable although it was CCed to stable.
> > > Sasha?
> > 
> > Looks like it requires more than trivial backport (I think). Tejun?
> 
> The timer migration has changed quite a bit.  Given that we've never
> seen vmstat work crashing in 3.18 era, I wonder whether the right
> thing to do here is reverting 874bbfe600a6 from 3.18 stable?

It's not just 3.18 that has this; 874bbfe600a6 was backported to all
stable branches from 3.10 onward.  Only the 4.2-ckt branch has
22b886dd10180939.

Ben.

-- 
Ben Hutchings
Life is what happens to you while you're busy making other plans.
                                                               - John Lennon

[toc] | [prev] | [next] | [standalone]

#1315673

[Multipart message — attachments visible in raw view] — view raw

On Sat, 23 Jan 2016, Ben Hutchings wrote:
> On Fri, 2016-01-22 at 11:09 -0500, Tejun Heo wrote:
> > > Looks like it requires more than trivial backport (I think). Tejun?
> > 
> > The timer migration has changed quite a bit.  Given that we've never
> > seen vmstat work crashing in 3.18 era, I wonder whether the right
> > thing to do here is reverting 874bbfe600a6 from 3.18 stable?
> 
> It's not just 3.18 that has this; 874bbfe600a6 was backported to all
> stable branches from 3.10 onward.  Only the 4.2-ckt branch has
> 22b886dd10180939.

22b886dd10180939 fixes a bug which was introduced with the timer wheel
overhaul in 4.2. So only 4.2/3 should have it backported.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1317673

On Sat 23-01-16 17:11:54, Thomas Gleixner wrote:
> On Sat, 23 Jan 2016, Ben Hutchings wrote:
> > On Fri, 2016-01-22 at 11:09 -0500, Tejun Heo wrote:
> > > > Looks like it requires more than trivial backport (I think). Tejun?
> > > 
> > > The timer migration has changed quite a bit.  Given that we've never
> > > seen vmstat work crashing in 3.18 era, I wonder whether the right
> > > thing to do here is reverting 874bbfe600a6 from 3.18 stable?
> > 
> > It's not just 3.18 that has this; 874bbfe600a6 was backported to all
> > stable branches from 3.10 onward.  Only the 4.2-ckt branch has
> > 22b886dd10180939.
> 
> 22b886dd10180939 fixes a bug which was introduced with the timer wheel
> overhaul in 4.2. So only 4.2/3 should have it backported.

Thanks for explanation. So do I understand right that timers are always run
on the calling CPU in kernels prior to 4.2 and thus commit 874bbfe600a6 (to
run timer for delayed work on the calling CPU) doesn't make sense there? If
that is true than reverting the commit from older stable kernels is
probably the easiest way to resolve the crashes.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[toc] | [prev] | [next] | [standalone]

#1317685

[Multipart message — attachments visible in raw view] — view raw

On Tue, 26 Jan 2016, Jan Kara wrote:
> On Sat 23-01-16 17:11:54, Thomas Gleixner wrote:
> > On Sat, 23 Jan 2016, Ben Hutchings wrote:
> > > On Fri, 2016-01-22 at 11:09 -0500, Tejun Heo wrote:
> > > > > Looks like it requires more than trivial backport (I think). Tejun?
> > > > 
> > > > The timer migration has changed quite a bit.  Given that we've never
> > > > seen vmstat work crashing in 3.18 era, I wonder whether the right
> > > > thing to do here is reverting 874bbfe600a6 from 3.18 stable?
> > > 
> > > It's not just 3.18 that has this; 874bbfe600a6 was backported to all
> > > stable branches from 3.10 onward.  Only the 4.2-ckt branch has
> > > 22b886dd10180939.
> > 
> > 22b886dd10180939 fixes a bug which was introduced with the timer wheel
> > overhaul in 4.2. So only 4.2/3 should have it backported.
> 
> Thanks for explanation. So do I understand right that timers are always run
> on the calling CPU in kernels prior to 4.2 and thus commit 874bbfe600a6 (to
> run timer for delayed work on the calling CPU) doesn't make sense there? If
> that is true than reverting the commit from older stable kernels is
> probably the easiest way to resolve the crashes.

I was merily referring to 22b886dd10180939 which is a bug fix for things we
reworked in the timer wheel core code in 4.2. It's completely unrelated to the
problem at hand.

Non pinned timers can be migrated due to power saving decisions since
2.6.36. What changed over time is how the decision is made, but the general
principle still applies.

The timer code was completely unchanged between 3.18 and 4.0 and even with the
larger overhaul in 4.2 we did not change the migration logic. We merily
changed the internal implementation of the timer wheel.

I have no idea how 874bbfe600a6 can result in crashing on older kernels. Can
you ask the reporter to enable DEBUG_OBJECTS so we might get an idea what goes
wrong with that timer.

Thanks,

	tglx

[toc] | [prev] | [next] | [standalone]

#1317758

On Tue 2016-01-26 10:34:00, Jan Kara wrote:
> On Sat 23-01-16 17:11:54, Thomas Gleixner wrote:
> > On Sat, 23 Jan 2016, Ben Hutchings wrote:
> > > On Fri, 2016-01-22 at 11:09 -0500, Tejun Heo wrote:
> > > > > Looks like it requires more than trivial backport (I think). Tejun?
> > > > 
> > > > The timer migration has changed quite a bit.  Given that we've never
> > > > seen vmstat work crashing in 3.18 era, I wonder whether the right
> > > > thing to do here is reverting 874bbfe600a6 from 3.18 stable?
> > > 
> > > It's not just 3.18 that has this; 874bbfe600a6 was backported to all
> > > stable branches from 3.10 onward.  Only the 4.2-ckt branch has
> > > 22b886dd10180939.
> > 
> > 22b886dd10180939 fixes a bug which was introduced with the timer wheel
> > overhaul in 4.2. So only 4.2/3 should have it backported.
> 
> Thanks for explanation. So do I understand right that timers are always run
> on the calling CPU in kernels prior to 4.2 and thus commit 874bbfe600a6 (to
> run timer for delayed work on the calling CPU) doesn't make sense there? If
> that is true than reverting the commit from older stable kernels is
> probably the easiest way to resolve the crashes.

The commit 874bbfe600a6 ("workqueue: make sure delayed work run in
local cpu") forces the timer to run on the local CPU. It might be correct
for vmstat. But I wonder if it might break some other delayed work
user that depends on running on different CPU.

Best Regards,
Petr

[toc] | [prev] | [next] | [standalone]

#1317890

[Multipart message — attachments visible in raw view] — view raw

On Tue, 26 Jan 2016, Petr Mladek wrote:
> On Tue 2016-01-26 10:34:00, Jan Kara wrote:
> > On Sat 23-01-16 17:11:54, Thomas Gleixner wrote:
> > > On Sat, 23 Jan 2016, Ben Hutchings wrote:
> > > > On Fri, 2016-01-22 at 11:09 -0500, Tejun Heo wrote:
> > > > > > Looks like it requires more than trivial backport (I think). Tejun?
> > > > > 
> > > > > The timer migration has changed quite a bit.  Given that we've never
> > > > > seen vmstat work crashing in 3.18 era, I wonder whether the right
> > > > > thing to do here is reverting 874bbfe600a6 from 3.18 stable?
> > > > 
> > > > It's not just 3.18 that has this; 874bbfe600a6 was backported to all
> > > > stable branches from 3.10 onward.  Only the 4.2-ckt branch has
> > > > 22b886dd10180939.
> > > 
> > > 22b886dd10180939 fixes a bug which was introduced with the timer wheel
> > > overhaul in 4.2. So only 4.2/3 should have it backported.
> > 
> > Thanks for explanation. So do I understand right that timers are always run
> > on the calling CPU in kernels prior to 4.2 and thus commit 874bbfe600a6 (to
> > run timer for delayed work on the calling CPU) doesn't make sense there? If
> > that is true than reverting the commit from older stable kernels is
> > probably the easiest way to resolve the crashes.
> 
> The commit 874bbfe600a6 ("workqueue: make sure delayed work run in
> local cpu") forces the timer to run on the local CPU. It might be correct
> for vmstat. But I wonder if it might break some other delayed work
> user that depends on running on different CPU.

The default of add_timer() is to run on the current cpu. It only moves the
timer to a different cpu when the power saving code says so. So 874bbfe600a6
enforces that the timer runs on the cpu on which queue_delayed_work() is
called, but before that commit it was likely that the timer was queued on the
calling cpu. So there is nothing which can depend on running on a different
CPU, except callers of queue_delayed_work_on() which provide the target cpu
explicitely. 874bbfe600a6 does not affect those callers at all.

Now, what's different is:

+       if (cpu == WORK_CPU_UNBOUND)
+               cpu = raw_smp_processor_id();
        dwork->cpu = cpu;

So before that change dwork->cpu was set to WORK_CPU_UNBOUND. Now it's set to
the current cpu, but I can't see how that matters.

Thanks,

	tglx

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web