Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #63369 > unrolled thread

Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders Bananapi and Lamobo R1 unbootable

Back to article view | Back to linux.debian.kernel

Contents

  Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders Bananapi and Lamobo R1 unbootable Jürgen Löb <j.loeb@loeb-it.de> - 2019-02-16 19:10 +0100
    Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders Bananapi and Lamobo R1 unbootable Reco <bugs@enotuniq.net> - 2019-02-17 12:50 +0100
    Bug#922478: have yet to find an armhf board that works with 4.9.144-3 Vagrant Cascadian <vagrant@debian.org> - 2019-02-17 19:00 +0100
      Bug#922478: have yet to find an armhf board that works with 4.9.144-3 Adrian Bunk <bunk@debian.org> - 2019-02-17 23:00 +0100
    Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders Bananapi and Lamobo R1 unbootable Reco <bugs@enotuniq.net> - 2019-02-17 20:00 +0100
      Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders Bananapi and Lamobo R1 unbootable Cyril Brulebois <kibi@debian.org> - 2019-02-17 20:40 +0100
    Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders Bananapi and Lamobo R1 unbootable Steve McIntyre <steve@einval.com> - 2019-02-18 13:40 +0100
    Bug#922478: upgrade linux-image-4.9.0-8-686 from 4.9.130 to 4.9.144 on ALIX-Board is unbootable Werner Opriel <w.opr@gmx.de> - 2019-02-18 13:50 +0100
    Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders Bananapi and Lamobo R1 unbootable "Timo Sigurdsson" <public_timo.s@silentcreek.de> - 2019-02-18 14:00 +0100
    Bug#922478: have yet to find an armhf board that works with 4.9.144-3 Steve McIntyre <steve@einval.com> - 2019-02-18 17:30 +0100
    Bug#922478: Re : Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders many systems unbootable jean.tampa@free.fr - 2019-02-18 17:40 +0100
    Bug#922478: have yet to find an armhf board that works with 4.9.144-3 Neil Williams <codehelp@debian.org> - 2019-02-18 17:40 +0100
    Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders many systems unbootable Neil Williams <codehelp@debian.org> - 2019-02-18 18:00 +0100
    Bug#922478: have yet to find an armhf board that works with 4.9.144-3 "Timo Sigurdsson" <public_timo.s@silentcreek.de> - 2019-02-18 19:10 +0100
    Processed: linux: diff for NMU version 4.9.144-3.1 "Debian Bug Tracking System" <owner@bugs.debian.org> - 2019-02-19 12:50 +0100

#63369 — Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders Bananapi and Lamobo R1 unbootable

[Multipart message — attachments visible in raw view] — view raw

Package: linux-image-4.9.0-8-armmp-lpae

Version: 4.9.144-3

Severity: serious

Updated my Lamobo R1 board with apt update;apt upgrade

After the update uboot is struck at "Starting kernel". There is no
further output after "Starting kernel". Same happens on Bananapi 1
board. Unfortunately there is no more useful information.


I have been able to recover by downgrading to a backup kernel by
mounting the boot partiton on the sd card. Then:

dd if=boot.scr of=boot.script bs=72 skip=1 (extract script)

replaced  in boot.script:

setenv fk_kvers '4.9.0-8-armmp-lpae'

with

setenv fk_kvers '4.9.0-7-armmp-lpae'  (backup kernel that has been
available on my boot partition)

then:

mkimage -C none -A arm -T script -d boot.script boot.scr


Afterwards I have been able to boot the system with the old kernel
version. Then I was able to restore the previous version (4.9.130-2) with:

dpkg -i linux-image-4.9.0-8-armmp-lpae_4.9.130-2_armhf.deb (luckily
found the older package)

Upgrading to 4.9.144-3 again after this results in the unbootable
behavior again. Thus for sure the upgrade to 4.9.144-3 is causing the
problem.

-----------------------
Hiermit widerspreche ich/wir der Nutzung oder Uebermittlung 
meiner/unserer Daten fuer Werbezwecke oder fuer die Markt- oder 
Meinungsforschung gem. Par. 28 Abs. 3 Bundesdatenschutzgesetz.

[toc] | [next] | [standalone]

#63375

	Hi all.

I'd like to add that plain armmp (non-lpae) is broken too.
At least for Armada385/Caiman and QEMU's virt.

Reco

[toc] | [prev] | [next] | [standalone]

#63379 — Bug#922478: have yet to find an armhf board that works with 4.9.144-3

[Multipart message — attachments visible in raw view] — view raw

After upgrading to the latest 4.9.x kernel in sid, all of the armhf
boards running this kernel failed to boot.

Adding to the list:

imx6: Cubox-i4pro, Cubox-i4x4, Wandboard Quad
exynos5: Odroid-XU4
exynos4: Odroid-U3
rk3328: firefly-rk3288
sunxi A20: Cubietruck


So it clearly impacts a wide variety of systems...


live well,
  vagrant

[toc] | [prev] | [next] | [standalone]

#63386 — Bug#922478: have yet to find an armhf board that works with 4.9.144-3

On Sun, Feb 17, 2019 at 09:52:48AM -0800, Vagrant Cascadian wrote:
> After upgrading to the latest 4.9.x kernel in sid, all of the armhf
> boards running this kernel failed to boot.
> 
> Adding to the list:
> 
> imx6: Cubox-i4pro, Cubox-i4x4, Wandboard Quad
> exynos5: Odroid-XU4
> exynos4: Odroid-U3
> rk3328: firefly-rk3288
> sunxi A20: Cubietruck
> 
> 
> So it clearly impacts a wide variety of systems...

debian/patches/debian/arm-avoid-abi-change-in-4.9.139.patch changes
the order of struct processor but lacks a corresponding change to 
arch/arm/mm/proc-macros.S

> live well,
>   vagrant

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

[toc] | [prev] | [next] | [standalone]

#63380

	Hi.

On Sun, Feb 17, 2019 at 07:38:18PM +0100, Cyril Brulebois wrote:
> Hi folks,
> 
> Jürgen Löb <j.loeb@loeb-it.de> (2019-02-16):
> > Package: linux-image-4.9.0-8-armmp-lpae
> > Version: 4.9.144-3
> > Severity: serious
> > 
> > Updated my Lamobo R1 board with apt update;apt upgrade
> > 
> > After the update uboot is struck at "Starting kernel". There is no
> > further output after "Starting kernel". Same happens on Bananapi 1
> > board. Unfortunately there is no more useful information.
> […]
> 
> Summing up, it looks like everybody in cc is confirming the regression
> happens between 4.9.130-2 and 4.9.144-3, with and without lpae, on
> various boards. Any chance you could check what happens with the
> 4.9.135-1 intermediary version that can be found on snapshot.debian.org?

Did this already in QEMU (virt board).
4.9.135-1 works.
4.9.144-1 (next one) is broken.

The problem is - 4.9.144-1 introduced large amount of changes, including
two Spectre mitigations.

My attempts to build a kernel with CONFIG_SPECTRE=n yielded unbootable
kernels, which may mean that:

a) Spectre mitigations are not related to the problem.
b) My kernel-rebuilding skill could use some improvement.

Reco

[toc] | [prev] | [next] | [standalone]

#63381

[Multipart message — attachments visible in raw view] — view raw

Hi,

Reco <bugs@enotuniq.net> (2019-02-17):
> Did this already in QEMU (virt board).
> 4.9.135-1 works.
> 4.9.144-1 (next one) is broken.

Is there any chance you could share how to get such a VM set up in QEMU?

I'd be happy to try a few kernel builds, but having a quick way to check
whether a given kernel build is OK/KO would be much appreciated.
 

Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[toc] | [prev] | [next] | [standalone]

#63390

On Mon, Feb 18, 2019 at 11:28:10AM +0000, Neil Williams wrote:
>On Sun, 17 Feb 2019 21:26:41 +0100 (CET) "Timo Sigurdsson"
>> 
>> So, I also tested 4.9.135-1 on a Bananapi board and can confirm it
>> works.
>> 
>> I would suspect the issue is caused by Debian's kernel configuration
>> or changes. The Kernel CI project has ARM hardware, including the
>> Bananapi board and does tests of stable kernel updates to verify that
>> the kernel boots. At least with multi_v7_defconfig and
>> sunxi_defconfig, upstream 4.9.144 does boot on Allwinner-based
>> hardware, see:
>> https://kernelci.org/soc/allwinner/job/stable-rc/kernel/v4.9.144/

Yup. The bug is clearly coming from a local Debian patch.

>Is it feasible to have a script in devscripts or similar which maps the
>version of the kernel *Candidate* to KernelCI URLs for the same
>version?
>
>Can we correlate Debian kernel versions to something like
>https://kernelci.org/boot/all/job/stable-rc/branch/linux-4.9.y/kernel/v4.9.144/
>or
>https://kernelci.org/boot/all/job/stable-rc/kernel/v4.9.144/ ?

Sure, maybe. I've suggested kernelci as a useful thing to help here,
but we really need to be testing kernels complete with all the Debian
patches to...

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.

[toc] | [prev] | [next] | [standalone]

#63391 — Bug#922478: upgrade linux-image-4.9.0-8-686 from 4.9.130 to 4.9.144 on ALIX-Board is unbootable

Hi all.

I'd like to add that 4.9.0-8-686 Version is broken too.
At least for my ALIX-Board ALIX.2D2 from PC Engines.

(CPU: 500 MHz AMD Geode LX800)

W.O.

[toc] | [prev] | [next] | [standalone]

#63392

Hi,

On Mon, 18 Feb 2019 11:28:10 +0000, Neil Williams <codehelp@debian.org> wrote:
> Is it feasible to have a script in devscripts or similar which maps the
> version of the kernel *Candidate* to KernelCI URLs for the same
> version?
> 
> Can we correlate Debian kernel versions to something like
> https://kernelci.org/boot/all/job/stable-rc/branch/linux-4.9.y/kernel/v4.9.144/
> or
> https://kernelci.org/boot/all/job/stable-rc/kernel/v4.9.144/ ?

I just had another look and found they also have a job for the stable releases rather than the release candidates:
https://kernelci.org/build/stable/branch/linux-4.9.y/kernel/v4.9.144/

So, as long as the Debian kernel is based on a longterm kernel which is still supported upstream, the mapping should work.

What might be worth a thought as well, though, is to have such automated testing of the Debian kernels as well. Either by asking the Kernel CI project whether they'd be willing to build and test Debian kernels, too, or by setting up an infrastructure similar to theirs just for Debian. Now, I wouldn't expect Debian to have as much hardware to test on, but in this particular case it would have helped already to test the kernel in a virtualized setup. Based on this thread, it seems to me that the armhf kernels haven't received any boot testing prior to release. If that's really the case, I guess something along these lines might help Debian substantially.


Cheers,

Timo

[toc] | [prev] | [next] | [standalone]

#63395 — Bug#922478: have yet to find an armhf board that works with 4.9.144-3

On Mon, Feb 18, 2019 at 05:09:12PM +0100, Cyril Brulebois wrote:
>Control: tag -1 patch
>
>Adrian Bunk <bunk@debian.org> (2019-02-17):
>> On Sun, Feb 17, 2019 at 09:52:48AM -0800, Vagrant Cascadian wrote:
>> > After upgrading to the latest 4.9.x kernel in sid, all of the armhf
>> > boards running this kernel failed to boot.
>> > 
>> > Adding to the list:
>> > 
>> > imx6: Cubox-i4pro, Cubox-i4x4, Wandboard Quad
>> > exynos5: Odroid-XU4
>> > exynos4: Odroid-U3
>> > rk3328: firefly-rk3288
>> > sunxi A20: Cubietruck
>> > 
>> > 
>> > So it clearly impacts a wide variety of systems...
>> 
>> debian/patches/debian/arm-avoid-abi-change-in-4.9.139.patch changes
>> the order of struct processor but lacks a corresponding change to 
>> arch/arm/mm/proc-macros.S
>
>Based on this suggestion and Julien's suggested patch on IRC a couple
>hours ago, I've tested the attached patch successfully (as in: from a
>busy loop in qemu-system-arm to the “expected” kernel panic, as
>discussed in another subthread).
>
>I've uploaded linux-image binaries (armmp and armmp-lpae) here, which
>were cross-built through sbuild, thanks to Vagrant's suggestion on IRC:
>  https://people.debian.org/~kibi/linux-bug-922478/
>
>which is:
>  DEBIAN_KERNEL_DISABLE_DEBUG=yes sbuild -d stretch-proposed-updates -c stretch-amd64-sbuild --build=amd64 --profiles='pkg.linux.notools nodoc nopython cross pkg.linux.nosource' --host=armhf linux_4.9.144-4.dsc
>
>Checking this on real hardware would be great, trying to put everyone
>involved in the loop through cc.
>
>
>Cheers,
>-- 
>Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
>D-I release manager -- Release team member -- Freelance Consultant

>From 07f237b3911a685b18a7584456ace1293636bcc7 Mon Sep 17 00:00:00 2001
>From: Cyril Brulebois <cyril@debamax.com>
>Date: Mon, 18 Feb 2019 13:49:50 +0100
>Subject: [PATCH] Update debian/arm-avoid-abi-change-in-4.9.139.patch (Closes:
> #922478).
>
>This makes it take into account the function pointers reordering in
>arch/arm/mm/proc-macros.S (addition of check_bugs), fixing the failure
>to boot many armhf devices.
>
>With thanks to Adrian Bunk for the hint, and Julien Cristau for the
>prospective patch.

...

Just to confirm: installed this on the Armada XP on my desk (copy of
the buildd hardware we're using) and it works just fine.

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
"Every time you use Tcl, God kills a kitten." -- Malcolm Ray

[toc] | [prev] | [next] | [standalone]

#63396 — Bug#922478: Re : Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders many systems unbootable

Hi all,

Same issue here with 4.9.144-3 on my Olimex OLinuXino-LIME2 board 
(Allwinner A20 cpu) :
u-boot is struck at "Starting kernel", nothing happens after that.

This board boots ok with 4.9.130-2 and also with 4.9.135-1 (found at 
snapshot.debian.org).

I'm using the armmp kernel variant (non-lpae), if that helps I could 
easily try with lpae.
More generally please let me know if I can help pinpointing the issue by 
testing some more
kernel images. Note that I've never played with (cross-)compiling armhf 
kernels myself, though.

Despite this specific issue, thanks for all this great work, it's so 
cool to have Debian
supported out-of-the box on such little boards !

Jean

[toc] | [prev] | [next] | [standalone]

#63397 — Bug#922478: have yet to find an armhf board that works with 4.9.144-3

[Multipart message — attachments visible in raw view] — view raw

On Mon, 18 Feb 2019 17:09:12 +0100
Cyril Brulebois <kibi@debian.org> wrote:

> Control: tag -1 patch
> 
> Adrian Bunk <bunk@debian.org> (2019-02-17):
> > On Sun, Feb 17, 2019 at 09:52:48AM -0800, Vagrant Cascadian wrote:  
> > > After upgrading to the latest 4.9.x kernel in sid, all of the
> > > armhf boards running this kernel failed to boot.
> > > 
> > > Adding to the list:
> > > 
> > > imx6: Cubox-i4pro, Cubox-i4x4, Wandboard Quad
> > > exynos5: Odroid-XU4
> > > exynos4: Odroid-U3
> > > rk3328: firefly-rk3288
> > > sunxi A20: Cubietruck
> > > 
> > > 
> > > So it clearly impacts a wide variety of systems...  
> > 
> > debian/patches/debian/arm-avoid-abi-change-in-4.9.139.patch changes
> > the order of struct processor but lacks a corresponding change to 
> > arch/arm/mm/proc-macros.S  
> 
> Based on this suggestion and Julien's suggested patch on IRC a couple
> hours ago, I've tested the attached patch successfully (as in: from a
> busy loop in qemu-system-arm to the “expected” kernel panic, as
> discussed in another subthread).
> 
> I've uploaded linux-image binaries (armmp and armmp-lpae) here, which
> were cross-built through sbuild, thanks to Vagrant's suggestion on
> IRC: https://people.debian.org/~kibi/linux-bug-922478/
> 
> which is:
>   DEBIAN_KERNEL_DISABLE_DEBUG=yes sbuild -d stretch-proposed-updates
> -c stretch-amd64-sbuild --build=amd64 --profiles='pkg.linux.notools
> nodoc nopython cross pkg.linux.nosource' --host=armhf
> linux_4.9.144-4.dsc
> 
> Checking this on real hardware would be great, trying to put everyone
> involved in the loop through cc.

Passes on a cubietruck in LAVA:

https://staging.validation.linaro.org/scheduler/job/249113



-- 

Neil Williams
home@codehelp.co.uk

[toc] | [prev] | [next] | [standalone]

#63399 — Bug#922478: upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders many systems unbootable

[Multipart message — attachments visible in raw view] — view raw

On Mon, 18 Feb 2019 17:45:25 +0100
jean.tampa@free.fr wrote:

> On Mon, Feb 18, 2019 at 05:09:12PM +0100, Cyril Brulebois wrote:
> 
> > I've uploaded linux-image binaries (armmp and armmp-lpae) here,
> > which were cross-built through sbuild, thanks to Vagrant's
> > suggestion on IRC:
> > https://people.debian.org/~kibi/linux-bug-922478/  
> [...]
> > Checking this on real hardware would be great, trying to put
> > everyone involved in the loop through cc.  

I didn't get a chance to prove the original bug in LAVA, so now I've
got that result too:
Original bug:
https://staging.validation.linaro.org/scheduler/job/249115
Fails to boot, as others have reported.

Fixed kernel build:
https://staging.validation.linaro.org/scheduler/job/249113
Kernel boots correctly and job runs to completion.

To test, I downloaded the built .deb files and extracted the contents
and made a modules.tar.gz from ./lib/. The jobs above include
SHA256 checksums for the extracted files and the modules.tar.gz and I
uploaded the files to:
http://people.linaro.org/~neil.williams/linux-bug-922478/ (fixed
build) and
http://people.linaro.org/~neil.williams/linux-bug-922478/original/
(buggy build)

 
> So I've just tried this linux-image-4.9.0-8-armmp_4.9.144-4 on my 
> OLinuXino-LIME2, it boots ok, everything seems to work well.
> 
> Thanks !
> 
> Jean


-- 

Neil Williams
home@codehelp.co.uk

[toc] | [prev] | [next] | [standalone]

#63403 — Bug#922478: have yet to find an armhf board that works with 4.9.144-3

Hi Cyril,

Cyril Brulebois schrieb am 18.02.2019 17:09:

> Based on this suggestion and Julien's suggested patch on IRC a couple
> hours ago, I've tested the attached patch successfully (as in: from a
> busy loop in qemu-system-arm to the “expected” kernel panic, as
> discussed in another subthread).
> 
> I've uploaded linux-image binaries (armmp and armmp-lpae) here, which
> were cross-built through sbuild, thanks to Vagrant's suggestion on IRC:
>  https://people.debian.org/~kibi/linux-bug-922478/
> 
> which is:
>  DEBIAN_KERNEL_DISABLE_DEBUG=yes sbuild -d stretch-proposed-updates -c
>  stretch-amd64-sbuild --build=amd64 --profiles='pkg.linux.notools nodoc
>  nopython cross pkg.linux.nosource' --host=armhf linux_4.9.144-4.dsc
> 
> Checking this on real hardware would be great, trying to put everyone
> involved in the loop through cc.
> 

Thanks for the patch and effort. I can confirm your kernel package boots fine on my LeMaker BananaPi.
  $ uname -a
  Linux bananapi 4.9.0-8-armmp-lpae #1 SMP Debian 4.9.144-4 (2019-02-18) armv7l GNU/Linux

journalctl also shows no errors that haven't been there before.


Regards,

Timo

[toc] | [prev] | [next] | [standalone]

#63411 — Processed: linux: diff for NMU version 4.9.144-3.1

Processing control commands:

> tags 922478 + pending
Bug #922478 [src:linux] upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders many systems unbootable
Bug #922532 [src:linux] upgrade linux-image-4.9.0-8-armmp-lpae:armhf from 4.9.130-2 to 4.9.144-3 renders many systems unbootable
Bug #922547 [src:linux] linux-image-4.9.0-8-armmp: Failed to boot after upgrading to linux-image-4.9.0-8-armmp version 4.9.144-3
Added tag(s) pending.
Added tag(s) pending.
Added tag(s) pending.

-- 
922478: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922478
922532: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922532
922547: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922547
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web