Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #59576 > unrolled thread

Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51 Salvatore Bonaccorso <carnil@debian.org> - 2017-12-09 21:20 +0100
    Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51 Salvatore Bonaccorso <carnil@debian.org> - 2017-12-11 09:30 +0100
      Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51 Bernhard Schmidt <berni@birkenwald.de> - 2017-12-11 10:00 +0100
        Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51 Salvatore Bonaccorso <carnil@debian.org> - 2017-12-11 17:00 +0100
          Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51 Salvatore Bonaccorso <carnil@debian.org> - 2017-12-11 22:10 +0100
            Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51 Ben Hutchings <ben@decadent.org.uk> - 2017-12-11 23:00 +0100
              Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51 Salvatore Bonaccorso <carnil@debian.org> - 2017-12-11 23:20 +0100
                Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51 Salvatore Bonaccorso <carnil@debian.org> - 2017-12-12 08:40 +0100

#59576 — Bug#883938: linux-image-3.16.0-4-amd64: Kernel panic on boot after upgrading to debian 8.10 kernel 3.16.51

Hi

> I only have a terrible java KVM app available for seeing the console, and from
> what I can see, it logs with timestamps [0.811] through [0.841] (typing in
> from an image - excuse any typos, and leaving out long hexadecimal numbers
> that might not be interesting)

here is one captured from a console (for completeness):

[    0.425445] general protection fault: 0000 [#1] SMP
[    0.485003] Modules linked in:
[    0.521588] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G        W     3.16.0-4-amd64 #1 Debian 3.16.51-2
[    0.630874] Hardware name: Sun Microsystems     Sun Fire X4140/Sun Fire X4140, BIOS 0ABMN068 10/26/2009
[    0.743251] task: ffff88021f4db2d0 ti: ffff88021f4f0000 task.ti: ffff88021f4f0000
[    0.832756] RIP: 0010:[<ffffffff8109be3d>]  [<ffffffff8109be3d>] build_sched_domains+0x72d/0xcf0
[    0.937939] RSP: 0000:ffff88021f4f3df8  EFLAGS: 00010216
[    1.001457] RAX: 0000ffff00000000 RBX: 0000000000000000 RCX: 0000000000000004
[    1.086807] RDX: 0000000000016d48 RSI: 0000000000000000 RDI: 0000000000000200
[    1.172154] RBP: ffff88021f6b2598 R08: ffff88021f6b3960 R09: 0000000000000124
[    1.257488] R10: 0000000000000000 R11: ffff88021f4f3b06 R12: ffff88021f6b3940
[    1.342845] R13: 0000000000000200 R14: ffff88021f6a20c0 R15: 0000000000000200
[    1.428191] FS:  0000000000000000(0000) GS:ffff880227c00000(0000) knlGS:0000000000000000
[    1.524975] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    1.593668] CR2: ffff880427fff000 CR3: 0000000001813000 CR4: 00000000000007f0
[    1.679030] Stack:
[    1.703015]  ffff880200000000 ffff88021f6b3958 ffff88021f6b2500 ffff88021f6a20c0
[    1.791889]  0000000000000000 0000000000000000 0000000000000000 ffff880426f3c080
[    1.880812]  0000000000000000 000000000000f1c8 ffffffff00000000 0000000000000000
[    1.969682] Call Trace:
[    1.998903]  [<ffffffff8192332c>] ? sched_init_smp+0x398/0x452
[    2.068660]  [<ffffffff815244de>] ? mutex_lock+0xe/0x2a
[    2.131133]  [<ffffffff81069b23>] ? put_online_cpus+0x23/0x80
[    2.199847]  [<ffffffff810f284c>] ? stop_machine+0x2c/0x40
[    2.265451]  [<ffffffff8190415e>] ? kernel_init_freeable+0xdd/0x1e1
[    2.340404]  [<ffffffff81512c00>] ? rest_init+0x80/0x80
[    2.402867]  [<ffffffff81512c0a>] ? kernel_init+0xa/0xf0
[    2.466382]  [<ffffffff81525c58>] ? ret_from_fork+0x58/0x90
[    2.533018]  [<ffffffff81512c00>] ? rest_init+0x80/0x80
[    2.595491] Code: c0 0f 85 46 05 00 00 48 8b 74 24 08 48 c7 c2 00 dd a6 81 bf ff ff ff ff e8 91 78 21 00 48 98 49 8b 56 10 48 8b 04 c5 a0 1e 8e 81 <48> 8b 14 10 b8 01 00 00 00 49 89 54 24 10 f0 0
f c1 02 85 c0 75
[    2.827913] RIP  [<ffffffff8109be3d>] build_sched_domains+0x72d/0xcf0
[    2.905062]  RSP <ffff88021f4f3df8>
[    2.946759] ---[ end trace d66b66b984085567 ]---
[    3.001983] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    3.001983]
[    3.111285] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    3.111285]

Salvatore

[toc] | [next] | [standalone]

#59599

The issue seems not present when rolling back to 3.16.48-1 (this is
one kernel version which was only present in jessie-proposed-update).

Can someone confirm? If yes it has to be a change between 3.16.48-1
and 3.16.51-2.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#59601

On Mon, Dec 11, 2017 at 09:23:45AM +0100, Salvatore Bonaccorso wrote:

> The issue seems not present when rolling back to 3.16.48-1 (this is
> one kernel version which was only present in jessie-proposed-update).
> 
> Can someone confirm? If yes it has to be a change between 3.16.48-1
> and 3.16.51-2.

Yes, I can confirm that 3.16.48-1 boots fine on my affected servers.

Bernhard

[toc] | [prev] | [next] | [standalone]

#59610

Hi

The issue is as well happening with 3.16.51 vanilla (and config
generated with localmodconfig, make deb-pkg built).

Trying to bisect now.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#59618

Hi

here the bisect log (unless overseen something with the last good commit):

git bisect start
# good: [3717265153a66c2ecbd745ea8eef213289b4a55e] Linux 3.16.48
git bisect good 3717265153a66c2ecbd745ea8eef213289b4a55e
# bad: [c45c05f42d5d3baf5d18e648c064788381fcfa1c] Linux 3.16.51
git bisect bad c45c05f42d5d3baf5d18e648c064788381fcfa1c
# bad: [36284647e2096b8bcc238772b644151967a5189b] parisc: pci memory bar assignment fails with 64bit kernels on dino/cujo
git bisect bad 36284647e2096b8bcc238772b644151967a5189b
# bad: [bfc20d54acae250fde0c8d56fc57fd41ea6dbe99] Revert "ACPI / EC: Add support to disallow QR_EC to be issued before completing previous QR_EC"
git bisect bad bfc20d54acae250fde0c8d56fc57fd41ea6dbe99
# bad: [b657f2742871a071a015328ec45c26679998096c] PCI/PM: Restore the status of PCI devices across hibernation
git bisect bad b657f2742871a071a015328ec45c26679998096c
# bad: [6d8c76588c1b03d63c8b76eaabac7a537e2d6714] drm/msm/hdmi: Use bitwise operators when building register values
git bisect bad 6d8c76588c1b03d63c8b76eaabac7a537e2d6714
# bad: [95ac78eafd026fb38c38a8f312acdd6fd8aee747] kvm: x86: Guest BNDCFGS requires guest MPX support
git bisect bad 95ac78eafd026fb38c38a8f312acdd6fd8aee747
# bad: [54749559969963ea435013122941643a90f1c6b3] f2fs: load inode's flag from disk
git bisect bad 54749559969963ea435013122941643a90f1c6b3
# good: [49d4283d847987fccbd7c7ce8f59f0fd765702ff] sched/topology: Fix building of overlapping sched-groups
git bisect good 49d4283d847987fccbd7c7ce8f59f0fd765702ff
# bad: [5a7aba3f27f7262cc5fad1b85804ea0ad9f4275c] sched/topology: Fix overlapping sched_group_capacity
git bisect bad 5a7aba3f27f7262cc5fad1b85804ea0ad9f4275c
# bad: [00c978eada13e2cd1bc7da485e99ab5fb7c3418c] sched/topology: Fix overlapping sched_group_mask
git bisect bad 00c978eada13e2cd1bc7da485e99ab5fb7c3418c
# first bad commit: [00c978eada13e2cd1bc7da485e99ab5fb7c3418c] sched/topology: Fix overlapping sched_group_mask

With 00c978eada13e2cd1bc7da485e99ab5fb7c3418c the systems boots, but warns
once:

----cut---------cut---------cut---------cut---------cut---------cut-----
[...]
[    1.958685] smpboot: CPU0: Quad-Core AMD Opteron(tm) Processor 2356 (fam: 10, model: 02, steppi
ng: 03)
[    2.176232] Performance Events: AMD PMU driver.
[    2.230503] ... version:                0
[    2.278424] ... bit width:              48
[    2.327396] ... generic registers:      4
[    2.375308] ... value mask:             0000ffffffffffff
[    2.438837] ... max period:             00007fffffffffff
[    2.502353] ... fixed-purpose events:   0
[    2.550271] ... event mask:             000000000000000f
[    2.615975] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    2.711926] x86: Booting SMP configuration:
[    2.761931] .... node  #0, CPUs:        #1  #2  #3
[    2.859347] .... node  #1, CPUs:    #4  #5  #6  #7
[    3.064352] x86: Booted up 2 nodes, 8 CPUs
[    3.115572] smpboot: Total of 8 processors activated (36803.45 BogoMIPS)
[    3.200017] ------------[ cut here ]------------
[    3.255328] WARNING: CPU: 0 PID: 1 at kernel/sched/core.c:5807 build_sched_domains+0x7da/0xc80()
[    3.360528] Modules linked in:
[    3.397169] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.16.48+ #10
[    3.471097] Hardware name: Sun Microsystems     Sun Fire X4140/Sun Fire X4140, BIOS 0ABMN068 10/26/2009
[    3.583508]  0000000000000000 ffffffff81514999 0000000000000000 0000000000000009
[    3.672426]  ffffffff810699f7 ffff88021d699598 ffff88021d69ab40 ffff88021d69ab58
[    3.761306]  0000000000000000 ffff88041db520c0 ffffffff8109bb0a 0000000000000000
[    3.850185] Call Trace:
[    3.879420]  [<ffffffff81514999>] ? dump_stack+0x5d/0x78
[    3.942942]  [<ffffffff810699f7>] ? warn_slowpath_common+0x77/0x90
[    4.016855]  [<ffffffff8109bb0a>] ? build_sched_domains+0x7da/0xc80
[    4.091806]  [<ffffffff8192da09>] ? sched_init_smp+0x38b/0x444
[    4.161558]  [<ffffffff81518d9e>] ? mutex_lock+0xe/0x30
[    4.224020]  [<ffffffff81069bb3>] ? put_online_cpus+0x23/0x90
[    4.292744]  [<ffffffff810f239c>] ? stop_machine+0x2c/0x40
[    4.358340]  [<ffffffff8190f042>] ? kernel_init_freeable+0x106/0x207
[    4.434331]  [<ffffffff81509f60>] ? rest_init+0x80/0x80
[    4.496793]  [<ffffffff81509f6a>] ? kernel_init+0xa/0xf0
[    4.560319]  [<ffffffff8151a858>] ? ret_from_fork+0x58/0x90
[    4.626955]  [<ffffffff81509f60>] ? rest_init+0x80/0x80
[    4.689436] ---[ end trace 4340fcfda5f7e5a6 ]---
[    4.745076] devtmpfs: initialized
[    4.791668] PM: Registering ACPI NVS region [mem 0xd7fbe000-0xd7feffff] (204800 bytes)
[    4.886661] futex hash table entries: 8192 (order: 7, 524288 bytes)
[...]
----cut---------cut---------cut---------cut---------cut---------cut-----

With the subsequent commit:

----cut---------cut---------cut---------cut---------cut---------cut-----
[...]
[    1.958675] smpboot: CPU0: Quad-Core AMD Opteron(tm) Processor 2356 (fam: 10, model: 02, stepping: 03)
[    2.176232] Performance Events: AMD PMU driver.
[    2.230510] ... version:                0
[    2.278440] ... bit width:              48
[    2.327401] ... generic registers:      4
[    2.375315] ... value mask:             0000ffffffffffff
[    2.438833] ... max period:             00007fffffffffff
[    2.502359] ... fixed-purpose events:   0
[    2.550268] ... event mask:             000000000000000f
[    2.616064] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[    2.712042] x86: Booting SMP configuration:
[    2.762042] .... node  #0, CPUs:        #1  #2  #3
[    2.859422] .... node  #1, CPUs:    #4  #5  #6  #7
[    3.064345] x86: Booted up 2 nodes, 8 CPUs
[    3.115476] smpboot: Total of 8 processors activated (36803.96 BogoMIPS)
[    3.199912] ------------[ cut here ]------------
[    3.255229] WARNING: CPU: 0 PID: 1 at kernel/sched/core.c:5811 build_sched_domains+0x90f/0xcd0()
[    3.360432] Modules linked in:
[    3.397072] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.16.48+ #11
[    3.470991] Hardware name: Sun Microsystems     Sun Fire X4140/Sun Fire X4140, BIOS 0ABMN068 10/26/2009
[    3.583409]  0000000000000000 ffffffff81514999 0000000000000000 0000000000000009
[    3.672327]  ffffffff810699f7 0000000000000000 ffff88021d699598 ffff88021d69ab40
[    3.761207]  0000000000000200 ffff88041db520c0 ffffffff8109bc3f 0000000000000000
[    3.850097] Call Trace:
[    3.879321]  [<ffffffff81514999>] ? dump_stack+0x5d/0x78
[    3.942844]  [<ffffffff810699f7>] ? warn_slowpath_common+0x77/0x90
[    4.016747]  [<ffffffff8109bc3f>] ? build_sched_domains+0x90f/0xcd0
[    4.091708]  [<ffffffff8192da09>] ? sched_init_smp+0x38b/0x444
[    4.161458]  [<ffffffff81518d9e>] ? mutex_lock+0xe/0x30
[    4.223932]  [<ffffffff81069bb3>] ? put_online_cpus+0x23/0x90
[    4.292651]  [<ffffffff810f23ec>] ? stop_machine+0x2c/0x40
[    4.358241]  [<ffffffff8190f042>] ? kernel_init_freeable+0x106/0x207
[    4.434234]  [<ffffffff81509f60>] ? rest_init+0x80/0x80
[    4.496696]  [<ffffffff81509f6a>] ? kernel_init+0xa/0xf0
[    4.560222]  [<ffffffff8151a858>] ? ret_from_fork+0x58/0x90
[    4.626855]  [<ffffffff81509f60>] ? rest_init+0x80/0x80
[    4.689339] ---[ end trace 4ad3f0f59e8e3f1f ]---
[    4.744533] general protection fault: 0000 [#1] SMP 
[    4.804084] Modules linked in:
[    4.840686] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G        W     3.16.48+ #11
[    4.927105] Hardware name: Sun Microsystems     Sun Fire X4140/Sun Fire X4140, BIOS 0ABMN068 10/26/2009
[    5.039590] task: ffff88021d4db2b0 ti: ffff88021d4dc000 task.ti: ffff88021d4dc000
[    5.129191] RIP: 0010:[<ffffffff8109ba35>]  [<ffffffff8109ba35>] build_sched_domains+0x705/0xcd0
[    5.234486] RSP: 0000:ffff88021d4dfe00  EFLAGS: 00010216
[    5.297978] RAX: 0000ffff00000000 RBX: 0000000000000000 RCX: 0000000000000004
[    5.383331] RDX: 0000000000016ec8 RSI: 0000000000000000 RDI: 0000000000000200
[    5.468666] RBP: ffff88021d699598 R08: ffff88021d69ab60 R09: 0000000000000126
[    5.554013] R10: ffffffff81a70fd0 R11: 0000000000000125 R12: ffff88021d69ab40
[    5.639366] R13: 0000000000000200 R14: ffff88041db520c0 R15: 0000000000000200
[    5.724713] FS:  0000000000000000(0000) GS:ffff880227c00000(0000) knlGS:0000000000000000
[    5.821522] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    5.890297] CR2: ffff880427fff000 CR3: 0000000001812000 CR4: 00000000000007f0
[    5.975657] Stack:
[    5.999651]  0000000000000000 ffff88021d69ab58 ffff88021d699500 ffff88041db520c0
[    6.088516]  0000000000000000 0000000000000000 ffff880227829640 0000000000000000
[    6.177397]  0000000000000000 000000000000f228 ffffffff00000000 0000000000000000
[    6.266277] Call Trace:
[    6.295505]  [<ffffffff8192da09>] ? sched_init_smp+0x38b/0x444
[    6.365261]  [<ffffffff81518d9e>] ? mutex_lock+0xe/0x30
[    6.427739]  [<ffffffff81069bb3>] ? put_online_cpus+0x23/0x90
[    6.496449]  [<ffffffff810f23ec>] ? stop_machine+0x2c/0x40
[    6.562044]  [<ffffffff8190f042>] ? kernel_init_freeable+0x106/0x207
[    6.638036]  [<ffffffff81509f60>] ? rest_init+0x80/0x80
[    6.700508]  [<ffffffff81509f6a>] ? kernel_init+0xa/0xf0
[    6.764025]  [<ffffffff8151a858>] ? ret_from_fork+0x58/0x90
[    6.830661]  [<ffffffff81509f60>] ? rest_init+0x80/0x80
[    6.893134] Code: c0 0f 85 10 02 00 00 48 8b 74 24 08 48 c7 c2 80 71 a7 81 bf ff ff ff ff e8 79 c8 21 00 48 98 49 8b 56 10 48 8b 04 c5 80 c1 8e 81 <48> 8b 14 10 b8 01 00 00 00 49 89 54 24 10 f0 0f c1 02 85 c0 75 
[    7.125556] RIP  [<ffffffff8109ba35>] build_sched_domains+0x705/0xcd0
[    7.202704]  RSP <ffff88021d4dfe00>
[    7.244399] ---[ end trace 4ad3f0f59e8e3f20 ]---
[    7.299618] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    7.299618] 
[    7.408926] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    7.408926]
[...]
----cut---------cut---------cut---------cut---------cut---------cut-----

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#59619

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2017-12-11 at 21:29 +0000, Ben Hutchings wrote:
> There were several commits interspersed with sched/topology fixes
> upstream that I thought were cleanup and therefore didn't backport to
> the 3.16 stable branch.  Now I suspect that at least some of them are
> needed.
> 
> I'm attaching backports of 3 of the commits that I left out.  Can you
> test whether they fix the regression for you?
> 
> If this doesn't help, I think we may just have to revert the
> sched/topology fixes.

Sorry, the first patch should be this one and not
0001-sched-topology-Simplify-build_overlap_sched_groups.patch.

Ben.

-- 
Ben Hutchings
A free society is one where it is safe to be unpopular. - Adlai
Stevenson

[toc] | [prev] | [next] | [standalone]

#59620

Hi Ben,

On Mon, Dec 11, 2017 at 09:47:49PM +0000, Ben Hutchings wrote:
> On Mon, 2017-12-11 at 21:29 +0000, Ben Hutchings wrote:
> > There were several commits interspersed with sched/topology fixes
> > upstream that I thought were cleanup and therefore didn't backport to
> > the 3.16 stable branch.  Now I suspect that at least some of them are
> > needed.
> > 
> > I'm attaching backports of 3 of the commits that I left out.  Can you
> > test whether they fix the regression for you?

I built v3.16.51 with the three patches on top, and looks good I have
not seen any further problems with those.

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#59628

On Mon, Dec 11, 2017 at 11:10:42PM +0100, Salvatore Bonaccorso wrote:
> Hi Ben,
> 
> On Mon, Dec 11, 2017 at 09:47:49PM +0000, Ben Hutchings wrote:
> > On Mon, 2017-12-11 at 21:29 +0000, Ben Hutchings wrote:
> > > There were several commits interspersed with sched/topology fixes
> > > upstream that I thought were cleanup and therefore didn't backport to
> > > the 3.16 stable branch.  Now I suspect that at least some of them are
> > > needed.
> > > 
> > > I'm attaching backports of 3 of the commits that I left out.  Can you
> > > test whether they fix the regression for you?
> 
> I built v3.16.51 with the three patches on top, and looks good I have
> no  seen any further problems with those.

To confirm, I now as well bootet two affected systems with you
testkernel containg the patches and they boot now.

Regards,
Salvatore

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web