Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #72571 > unrolled thread

Bug#991967: linux-src 4.19.194-3 breaks Xen Dom0 powerdown and reboot

Back to article view | Back to linux.debian.kernel

Contents

  Bug#991967: linux-src 4.19.194-3 breaks Xen Dom0 powerdown and reboot Elliott Mitchell <ehem+debian@m5p.com> - 2021-08-06 21:30 +0200
    Processed: Re: Bug#991967: linux-src 4.19.194-3 breaks Xen Dom0  powerdown and reboot "Debian Bug Tracking System" <owner@bugs.debian.org> - 2021-08-07 08:50 +0200
    Bug#991967: linux-src 4.19.194-3 breaks Xen Dom0 powerdown and reboot Salvatore Bonaccorso <carnil@debian.org> - 2021-08-07 08:50 +0200
    Bug#991967: #991967: Simply ACPI powerdown/reset issue? Elliott Mitchell <ehem+debian@m5p.com> - 2021-09-11 04:10 +0200
      Bug#991967: #991967: Simply ACPI powerdown/reset issue? Salvatore Bonaccorso <carnil@debian.org> - 2021-09-11 13:40 +0200
        Bug#991967: #991967: Simply ACPI powerdown/reset issue? Elliott Mitchell <ehem+debian@m5p.com> - 2021-09-13 00:10 +0200
    Bug#991967: #991967: Simply ACPI powerdown/reset issue? Elliott Mitchell <ehem+debian@m5p.com> - 2021-09-19 17:10 +0200
    Bug#991967: Simply ACPI powerdown/reset issue? Diederik de Haas <didi.debian@cknow.org> - 2021-09-19 19:30 +0200
    Bug#991967: #991967: Simply ACPI powerdown/reset issue? Elliott Mitchell <ehem+debian@m5p.com> - 2021-09-20 06:40 +0200
    Bug#991967: #991967: Simply ACPI powerdown/reset issue? Elliott Mitchell <ehem+debian@m5p.com> - 2021-09-21 01:20 +0200
      Bug#991967: #991967: Simply ACPI powerdown/reset issue? Diederik de Haas <didi.debian@cknow.org> - 2021-09-21 01:50 +0200
    Bug#991967: #991967: Simply ACPI powerdown/reset issue? Elliott Mitchell <ehem+debian@m5p.com> - 2021-09-21 04:40 +0200
    Bug#991967: Simply ACPI powerdown/reset issue? Elliott Mitchell <ehem+debian@m5p.com> - 2021-09-26 05:40 +0200
      Bug#991967: Simply ACPI powerdown/reset issue? Diederik de Haas <didi.debian@cknow.org> - 2021-09-26 14:00 +0200
      Bug#991967: Simply ACPI powerdown/reset issue? Hans van Kranenburg <hans@knorrie.org> - 2021-10-04 12:20 +0200
    Bug#991967: (Presently) Not in 5.10 source Elliott Mitchell <ehem+debian@m5p.com> - 2021-12-07 03:10 +0100

#72571 — Bug#991967: linux-src 4.19.194-3 breaks Xen Dom0 powerdown and reboot

Package: src:linux
Version: 4.19.194-3
Control: affects -1 src:xen

SSIA.  Previous versions of 4.19 had no issues (4.19.181-1 according to
notes), but this cropped up with 4.19.194-3 (-1 and -2 weren't tested).

When a Xen domain 0 tries to reboot or powerdown the computer, it hangs
with the display off, but the power supply is active.

I'm rebuilding from source, so I imagine this also effects
linux-image-4.19.0-17-amd64.

Seems .194 caused multiple problems for Xen given 990642.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[toc] | [next] | [standalone]

#72573 — Processed: Re: Bug#991967: linux-src 4.19.194-3 breaks Xen Dom0 powerdown and reboot

Processing control commands:

> tags -1 + moreinfo
Bug #991967 [src:linux] linux-src 4.19.194-3 breaks Xen Dom0 powerdown and reboot
Added tag(s) moreinfo.

-- 
991967: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991967
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [next] | [standalone]

#72574

Control: tags -1 + moreinfo

Hi,

On Fri, Aug 06, 2021 at 11:50:54AM -0700, Elliott Mitchell wrote:
> Package: src:linux
> Version: 4.19.194-3
> Control: affects -1 src:xen
> 
> SSIA.  Previous versions of 4.19 had no issues (4.19.181-1 according to
> notes), but this cropped up with 4.19.194-3 (-1 and -2 weren't tested).
> 
> When a Xen domain 0 tries to reboot or powerdown the computer, it hangs
> with the display off, but the power supply is active.
> 
> I'm rebuilding from source, so I imagine this also effects
> linux-image-4.19.0-17-amd64.

Can you please try to bisect which commit introduced the issue? Does
it affect as well current upstream 4.19.201?

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#72879 — Bug#991967: #991967: Simply ACPI powerdown/reset issue?

An experiment lead to a potential alternative explanation for #991967.
The issue may be ACPI (non-UEFI) powerdown/reset was broken at
4.19.194-3.  Presence of Xen on the system may be unrelated.

Failing that, it could be Xen and non-UEFI systems are effected.  (Xen
was tried on a UEFI system and the issue wasn't observed)


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[toc] | [prev] | [next] | [standalone]

#72885 — Bug#991967: #991967: Simply ACPI powerdown/reset issue?

Hi Elliott,

On Fri, Sep 10, 2021 at 06:47:12PM -0700, Elliott Mitchell wrote:
> An experiment lead to a potential alternative explanation for #991967.
> The issue may be ACPI (non-UEFI) powerdown/reset was broken at
> 4.19.194-3.  Presence of Xen on the system may be unrelated.
> 
> Failing that, it could be Xen and non-UEFI systems are effected.  (Xen
> was tried on a UEFI system and the issue wasn't observed)

Following up on https://bugs.debian.org/991967#12

Did you succeeded in bisecting the issue as you seem to have it
reproducible?

Regards,
Salvatore

[toc] | [prev] | [next] | [standalone]

#72902 — Bug#991967: #991967: Simply ACPI powerdown/reset issue?

On Sat, Sep 11, 2021 at 01:29:12PM +0200, Salvatore Bonaccorso wrote:
> On Fri, Sep 10, 2021 at 06:47:12PM -0700, Elliott Mitchell wrote:
> > An experiment lead to a potential alternative explanation for #991967.
> > The issue may be ACPI (non-UEFI) powerdown/reset was broken at
> > 4.19.194-3.  Presence of Xen on the system may be unrelated.
> > 
> > Failing that, it could be Xen and non-UEFI systems are effected.  (Xen
> > was tried on a UEFI system and the issue wasn't observed)
> 
> Following up on https://bugs.debian.org/991967#12
> 
> Did you succeeded in bisecting the issue as you seem to have it
> reproducible?

Problem is that is rather a lot of kernel builds, which also means a lot
of downtime...   Right now distribution update seems worthy of greater
attention.

The one notable bit is the one I sent in the last message.  The system
does NOT have UEFI, and a test system with UEFI seemed to have no
problem.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[toc] | [prev] | [next] | [standalone]

#73007 — Bug#991967: #991967: Simply ACPI powerdown/reset issue?

On Sun, Sep 19, 2021 at 01:05:56AM -0400, Chuck Zmudzinski wrote:
> On Sat, 11 Sep 2021 13:29:12 +0200 Salvatore Bonaccorso 
> <carnil@debian.org> wrote:
>  >
>  > On Fri, Sep 10, 2021 at 06:47:12PM -0700, Elliott Mitchell wrote:
>  > > An experiment lead to a potential alternative explanation for #991967.
>  > > The issue may be ACPI (non-UEFI) powerdown/reset was broken at
>  > > 4.19.194-3. Presence of Xen on the system may be unrelated.
>  > >
>  > > Failing that, it could be Xen and non-UEFI systems are effected. (Xen
>  > > was tried on a UEFI system and the issue wasn't observed)
>  >
>  > Following up on https://bugs.debian.org/991967#12
>  >
>  > Did you succeeded in bisecting the issue as you seem to have it
>  > reproducible?
> 
> I noticed this bug on bullseye ever since I have been
> running bullseye as a dom0, but my testing indicates
> there is no problem with src:linux but the problem
> appeared in src:xen with the 4.14 version of xen on
> bullseye.
> 
> I ask Elliott if you are only seeing the problem on Debian's
> xen-4.14 hypervisor? Also, which architecture, arm or
> amd64? I only see the problem on the Debian xen-4.14
> hypervisor, and I have only tested on amd64, and I
> have found a fix for my amd64 system which is as
> follows:
> 
> Motherboard: ASRock B85M Pro4, BIOS P2.50 12/11/2015,
> with a Haswell CPU (core i5-4590S)
> 
> xen hypervisor version: 4.14.2+25-gb6a8c4f72d-2, amd64
> 
> linux kernel version: 5.10.46-4 (the current amd64 kernel
> for bullseye)

Nope.  As per the report the problem appeared with kernel 4.19.194-3 and
at the time using Xen 4.11.

The kernel you're listing is rather more recent, which might suggest a
patch which had been backported from 5.x to 4.19.

I could believe a Xen security update being the trigger though (I don't
recall there being one at the right time, but I wouldn't rule it out).


> Boot system: EFI, not using secure boot, booting xen
> hypervisor and dom0 bullseye with grub-efi package for
> bullseye, and it boots the xen-4.14-amd64.gz file, not
> the xen-4.14-amd64.efi file.
> 
> I also tested a buster dom0 with the 4.19 series kernel
> on the xen-4.14 hypervisor from bullseye and saw the
> problem, but I did not see the problem with either
> a buster (linux 4.19) or bullseye (linux 5.10) dom0 on
> the xen-4.11 hypervisor, so I think the problem is
> with the Debian version of the xen-4.14 hypervisor,
> not with src:linux.

Just to make sure, the kernel you were testing was 4.19.194-3?  The
issue didn't manifest with kernels earlier than that.

Could be we're seeing distinct bugs.


> This patch does affect amd64 acpi code, and is probably causing
> the problem on my amd64 system, so my build of the xen-4.14
> hypervisor without this patch fixed the problem.

While that commit modifies the code path the processor takes, the
modified path appears identical.


> I also would inquire with the Debian Xen Team about why they
> are backporting patches from the upstream xen unstable
> branch into Debian's 4.14 package that is currently shipping
> on Debian stable (bullseye). IMHO, the aforementioned
> patches that are not in the stable 4.14 branch upstream
> should not be included in the xen package for Debian stable.

Some people are asking for those.  Those are bugfixes for an extremely
popular device which panics on boot without the patches.


Meanwhile turned out between 5.10.0 and 5.10.30 the ARM64 device-trees
were modified in a way which broke Xen 4.14 on ARM64.  The change
violated Linux's own standards for device-trees, yet still appeared in a
stable branch.

In other news, if you see device-trees compared to ACPI tables, they're
not very comparable.  99% of ACPI tables work for all versions of all
OSes.  Any given device-tree is only likely to work for a single version
of a single OS.  While a useful abstraction for portions of kernel code,
device-trees are utter garbage compared to ACPI tables.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[toc] | [prev] | [next] | [standalone]

#73008 — Bug#991967: Simply ACPI powerdown/reset issue?

[Multipart message — attachments visible in raw view] — view raw

Adding pkg-xen-devel@lists.alioth.debian.org into the loop.

Chuck Zmudzinski replied to the bug and later replied to his own reply. 
To give full context, I've added the original reply in full and Chuck's reply 
to that (as it only quoted part of the context there).

On zondag 19 september 2021 07:05:56 CEST Chuck Zmudzinski wrote:
> On Sat, 11 Sep 2021 13:29:12 +0200 Salvatore Bonaccorso
> 
> <carnil@debian.org> wrote:
>  > Hi Elliott,
>  > 
>  > On Fri, Sep 10, 2021 at 06:47:12PM -0700, Elliott Mitchell wrote:
>  > > An experiment lead to a potential alternative explanation for #991967.
>  > > The issue may be ACPI (non-UEFI) powerdown/reset was broken at
>  > > 4.19.194-3. Presence of Xen on the system may be unrelated.
>  > > 
>  > > Failing that, it could be Xen and non-UEFI systems are effected. (Xen
>  > > was tried on a UEFI system and the issue wasn't observed)
>  > 
>  > Following up on https://bugs.debian.org/991967#12
>  > 
>  > Did you succeeded in bisecting the issue as you seem to have it
>  > reproducible?
>  > 
>  > Regards,
>  > Salvatore
> 
> Hello Elliott and Salvatore,
> 
> I noticed this bug on bullseye ever since I have been
> running bullseye as a dom0, but my testing indicates
> there is no problem with src:linux but the problem
> appeared in src:xen with the 4.14 version of xen on
> bullseye.
> 
> I ask Elliott if you are only seeing the problem on Debian's
> xen-4.14 hypervisor? Also, which architecture, arm or
> amd64? I only see the problem on the Debian xen-4.14
> hypervisor, and I have only tested on amd64, and I
> have found a fix for my amd64 system which is as
> follows:
> 
> Motherboard: ASRock B85M Pro4, BIOS P2.50 12/11/2015,
> with a Haswell CPU (core i5-4590S)
> 
> xen hypervisor version: 4.14.2+25-gb6a8c4f72d-2, amd64
> 
> linux kernel version: 5.10.46-4 (the current amd64 kernel
> for bullseye)
> 
> Boot system: EFI, not using secure boot, booting xen
> hypervisor and dom0 bullseye with grub-efi package for
> bullseye, and it boots the xen-4.14-amd64.gz file, not
> the xen-4.14-amd64.efi file.
> 
> I also tested a buster dom0 with the 4.19 series kernel
> on the xen-4.14 hypervisor from bullseye and saw the
> problem, but I did not see the problem with either
> a buster (linux 4.19) or bullseye (linux 5.10) dom0 on
> the xen-4.11 hypervisor, so I think the problem is
> with the Debian version of the xen-4.14 hypervisor,
> not with src:linux.
> 
> I also found a fix in src:xen:
> 
> I noticed the series of patches in debian/patches of the
> 4.14.2+25-gb6a8c4f72d-2 version of src:xen (and
> earlier versions of xen-4.14 on Debian) have several patches
> backported from the unstable branch of xen upstream. By
> removing some of these patches from the patches
> series of the src:xen package, the dom0 shuts down
> as expected on my ASRock Haswell motherboard.
> 
> I rebuilt the src:xen package after removing the following
> patches from the debian/patches series and the result
> was that the computer shuts down as expected if I boot
> using the patched hypervisor:
> 
> 0027-xen-rpi4-implement-watchdog-based-reset.patch
> 0028-tools-python-Pass-linker-to-Python-build-process.patch
> 0029-xen-arm-acpi-Don-t-fail-if-SPCR-table-is-absent.patch
> 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch
> 0031-xen-arm-acpi-The-fixmap-area-should-always-be-cleare.patch
> 0032-xen-arm-Check-if-the-platform-is-not-using-ACPI-befo.patch
> 0033-xen-arm-Introduce-fw_unreserved_regions-and-use-it.patch
> 0034-xen-arm-acpi-add-BAD_MADT_GICC_ENTRY-macro.patch
> 0035-xen-arm-traps-Don-t-panic-when-receiving-an-unknown-.patch
> 
> Most of these patches seem unrelated to the amd64
> architecture and instead affect the arm architecture, and
> removing all these patches is probably more than is needed to
> fix this bug, but I removed them all because I could not find
> them upstream on the 4.14 branch but instead only saw them
> on the xen unstable branch upstream (I did not check if they are
> on the 4.15 branch upstream), and I wanted to test
> a true upstream 4.14 version without these seemingly
> aggressive patches added by Debian from the unstable
> branch of xen upstream, and I discovered by being
> more conservative and not adding these patches from the
> unstable branch upstream fixed the problem!
> 
> I suspect the following patch is the culprit for problems
> shutting down on the amd64 architecture:
> 
> 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch
> 
> The commit log for this patch states:
> 
> From: Julien Grall <jgrall@amazon.com>
> Date: Sat, 26 Sep 2020 17:44:29 +0100
> Subject: xen/acpi: Rework acpi_os_map_memory() and acpi_os_unmap_memory()
> 
> The functions acpi_os_{un,}map_memory() are meant to be arch-agnostic
> while the __acpi_os_{un,}map_memory() are meant to be arch-specific.
> 
> Currently, the former are still containing x86 specific code.
> 
> To avoid this rather strange split, the generic helpers are reworked so
> they are arch-agnostic. This requires the introduction of a new helper
> __acpi_os_unmap_memory() that will undo any mapping done by
> __acpi_os_map_memory().
> 
> Currently, the arch-helper for unmap is basically a no-op so it only
> returns whether the mapping was arch specific. But this will change
> in the future.
> 
> Note that the x86 version of acpi_os_map_memory() was already able to
> able the 1MB region. Hence why there is no addition of new code.
> 
> Signed-off-by: Julien Grall <jgrall@amazon.com>
> Reviewed-by: Rahul Singh <rahul.singh@arm.com>
> Reviewed-by: Jan Beulich <jbeulich@suse.com>
> Acked-by: Stefano Stabellini <sstabellini@kernel.org>
> Tested-by: Rahul Singh <rahul.singh@arm.com>
> Tested-by: Elliott Mitchell <ehem+xen@m5p.com>
> (cherry picked from commit 1c4aa69ca1e1fad20b2158051eb152276d1eb973)
> ---------------------------------------------------
> 
> This patch does affect amd64 acpi code, and is probably causing
> the problem on my amd64 system, so my build of the xen-4.14
> hypervisor without this patch fixed the problem.
> 
> I think this bug should be re-classified as a bug in src:xen.
> 
> I also would inquire with the Debian Xen Team about why they
> are backporting patches from the upstream xen unstable
> branch into Debian's 4.14 package that is currently shipping
> on Debian stable (bullseye). IMHO, the aforementioned
> patches that are not in the stable 4.14 branch upstream
> should not be included in the xen package for Debian stable.
> 
> Regards,
> 
> Chuck Zmudzinski

On zondag 19 september 2021 14:44:01 CEST Chuck Zmudzinski wrote:
> As a follow-up to my last comment on this bug, the
> problems I see with my bullseye amd64 dom0 point to
> problems with ACPI powerdown/reset issue, but only on
> the Debian version of Xen-4.14. I do not see the problem
> on any version of the linux kernel, neither on bare metal
> nor on the Debian version of the Xen-4.11 hypervisor
> from buster. For example, the problem manifests itself
> on the Debian Xen-4.14 hypervisor with the Debian
> dom0 reaching the systemd power off target but the
> power does not actually turn off. Moreover, I can only
> recover by manually resetting the computer by pressing
> the physical reset button on the computer or removing
> power by physically unplugging the computer.
> 
> One slight difference I see from what Elliott reported -
> not only does the power supply remain powered after
> shutdown, but also messages on the console about
> powering down remain on the display monitor after
> reaching the systemd power down target and power
> to the display/monitor also persists.
> 
> For my amd64 system, this bug would be probably fixed
> on Debian stable by having a separate Xen-4.14 package
> for Debian stable that removes at least the following
> patches from the debian/patches series of the current
> Xen-4.14 package for stable:
> 
> 0027-xen-rpi4-implement-watchdog-based-reset.patch
> 0029-xen-arm-acpi-Don-t-fail-if-SPCR-table-is-absent.patch
> 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch
> 0031-xen-arm-acpi-The-fixmap-area-should-always-be-cleare.patch
> 0032-xen-arm-Check-if-the-platform-is-not-using-ACPI-befo.patch
> 0033-xen-arm-Introduce-fw_unreserved_regions-and-use-it.patch
> 0034-xen-arm-acpi-add-BAD_MADT_GICC_ENTRY-macro.patch
> 0035-xen-arm-traps-Don-t-panic-when-receiving-an-unknown-.patch
> 
> The 0028-tools-python-Pass-linker-to-Python-build-process.patch
> is probably not related to this bug, but I have not verified that
> the bug is fixed without removing that patch also. I would
> defer to more knowledgeable people about the problems with
> building Xen on Debian using various versions of python to decide
> whether or not to remove the 0028-tools-python... patch.
> 
> I think perhaps the aforementioned patches to xen/arm and
> xen/acpi would be suitable for testing a Debian Xen package
> targeting bookworm/testing or sid/unstable, but not for
> Debian bullseye/stable. As it is now, it appears the Debian Xen
> Team is not making any distinction between stable, testing,
> and unstable for its current Xen-4.14 package, and IMHO
> that is the root cause of this bug on Debian stable.
> 
> If the Debian Xen Team wants to experiment with patches
> from the unstable branch of upstream Xen on a Debian
> version of Xen-4.14, I respectfully ask that it do so only on
> bookworm/testing or unstable/sid and ship a separate
> more conservative package for bullseye/stable that is
> closer to the official upstream Xen 4.14.x version than
> the package that is currently shipping on bullseye/stable.

I don't have an opinion on whether the analyses is correct.

I can tell that after I upgraded my server to Testing which is now Bullseye, 
my server running Xen (4.14) does no longer power off.
It seems it does the whole shutdown procedure successfully, but it does not 
shut the machine off. I have an iKVM module in my server in which I can 
forcefully shut it off (remotely) and I use that as a workaround.

I upgraded the whole machine back then so there were a LOT of potential causes 
and as the machine is off most of the time and I didn't/don't know how to 
debug/bi-sect the issue, I resorted to my workaround. But it is a workaround 
and a regression from what it was when the machine ran Buster.

Cheers,
  Diederik

[toc] | [prev] | [next] | [standalone]

#73013 — Bug#991967: #991967: Simply ACPI powerdown/reset issue?

On Sun, Sep 19, 2021 at 01:05:56AM -0400, Chuck Zmudzinski wrote:
> xen hypervisor version: 4.14.2+25-gb6a8c4f72d-2, amd64
> 
> linux kernel version: 5.10.46-4 (the current amd64 kernel
> for bullseye)
> 
> Boot system: EFI, not using secure boot, booting xen
> hypervisor and dom0 bullseye with grub-efi package for
> bullseye, and it boots the xen-4.14-amd64.gz file, not
> the xen-4.14-amd64.efi file.

> I also tested a buster dom0 with the 4.19 series kernel
> on the xen-4.14 hypervisor from bullseye and saw the
> problem, but I did not see the problem with either
> a buster (linux 4.19) or bullseye (linux 5.10) dom0 on
> the xen-4.11 hypervisor, so I think the problem is
> with the Debian version of the xen-4.14 hypervisor,
> not with src:linux.

You're referencing several software versions which are mismatches for
#991967.  #991967 was observed with Xen 4.11 and Linux kernel 4.19.194-3,
but not Linux kernel 4.19.181.

The fact it correlates with a Linux kernel update rather strongly points
to the Linux kernel.  I could believe the situation is partially the
fault of both though.


> I suspect the following patch is the culprit for problems
> shutting down on the amd64 architecture:
> 
> 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch

> This patch does affect amd64 acpi code, and is probably causing
> the problem on my amd64 system, so my build of the xen-4.14
> hypervisor without this patch fixed the problem.

Of the ones listed that is the only one which has any overlap with x86
code.  The next reproduction step is `apt-get source xen &&
patch -p1 -R < 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch
&& dpkg-buildpackage -b`.  Then try with this to confirm that patch
is what does it.

Thing is that delta is rather small.  I don't have a simulator, but that
is rather small to be the culprit.


> I think this bug should be re-classified as a bug in src:xen.

There could be a separate bug in src:xen, but that is not #991967.

> I also would inquire with the Debian Xen Team about why they
> are backporting patches from the upstream xen unstable
> branch into Debian's 4.14 package that is currently shipping
> on Debian stable (bullseye). IMHO, the aforementioned
> patches that are not in the stable 4.14 branch upstream
> should not be included in the xen package for Debian stable.

It was requested since someone trying to have Xen operational on a device
needed those for operation.  Rather a lot of bugfix or very small
standalone feature patches get cherry-picked.


Presently I haven't been convinced this is a Xen bug (though it does
effect Xen installations).

Any chance you've got the tools to build and try a 5.5.0 or 5.10.0 Linux
kernel?  I'm suspecting got incorrectly backported on the Linux side
(alternatively the Xen project seems a bit poor at keeping needed patches
in Linux).


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[toc] | [prev] | [next] | [standalone]

#73019 — Bug#991967: #991967: Simply ACPI powerdown/reset issue?

On Mon, Sep 20, 2021 at 06:29:49PM -0400, Chuck Zmudzinski wrote:
> On 9/20/21 1:43 PM, Chuck Zmudzinski wrote:
> >
> > On 9/20/21 12:27 AM, Elliott Mitchell wrote:
> >> On Sun, Sep 19, 2021 at 01:05:56AM -0400, Chuck Zmudzinski wrote:
> >>
> >>> I suspect the following patch is the culprit for problems
> >>> shutting down on the amd64 architecture:
> >>>
> >>> 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch
> >>> This patch does affect amd64 acpi code, and is probably causing
> >>> the problem on my amd64 system, so my build of the xen-4.14
> >>> hypervisor without this patch fixed the problem.
> >> Of the ones listed that is the only one which has any overlap with x86
> >> code.?? The next reproduction step is `apt-get source xen &&
> >> patch -p1 -R < 
> >> 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch
> >> && dpkg-buildpackage -b`.?? Then try with this to confirm that patch
> >> is what does it.
> >>
> >> Thing is that delta is rather small.?? I don't have a simulator, but that
> >> is rather small to be the culprit.
> >
> > I just tested the build with
> > patch -p1 -R < 
> > 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch
> > applied before building the package and I can confirm that this is the 
> > patch
> > causing the trouble for dom0 poweroff on x86/amd64. Reverting this patch
> > fixes it on my amd64 system. But this would probably break the arm build.
> >
> > I think one possible fix would require modifying
> > 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch
> > so it only applies at runtime to the arm architecture. I will try some
> > modifications to the patch instead of removing it, and if I get something
> > that works on amd64 and also might work on arm, I will post it
> > for Elliott to try.
> 
> I have an encouraging result. I found a very simple patch
> to xen/arch/x86/acpi/lib.c that fixes the dom0 poweroff
> bug on my system and it should not affect the arm patches
> at all:
> --------------------------------------------------------------
> This patch partially reverts previous patch
> 0030-xen-acpi-Rework-acpi_os_map_memory-and-acpi_os_unmap.patch
> 
> This hopefully fixes #911976
> 
> --- a/xen/arch/x86/acpi/lib.c?????? 2021-09-20 16:49:08.000000000 -0400
> +++ b/xen/arch/x86/acpi/lib.c?????? 2021-09-20 16:25:05.572038000 -0400
> @@ -46,10 +46,6 @@
>  ???????? if ((phys + size) <= (1 * 1024 * 1024))
>  ???????? ?????? return __va(phys);
> 
> -?????? /* No further arch specific implementation after early boot */
> -?????? if (system_state >= SYS_STATE_boot)
> -?????? ?????? return NULL;
> -
>  ???????? offset = phys & (PAGE_SIZE - 1);
>  ???????? mapped_size = PAGE_SIZE - offset;
>  ???????? set_fixmap(FIX_ACPI_END, phys);
> ----------------------------------------------------------------------
> 
> Can you try this patch to src:xen and see if your
> arm devices are OK with it?

Merely having the path is a sufficiently strong indicator for me to
simply wave it past.  I though would suggest Debian should instead
cherry-pick commit 0f089bbf43ecce6f27576cb548ba4341d0ec46a8.

This is available as a patch at:

https://xenbits.xen.org/gitweb/?p=xen.git;a=patch;h=0f089bbf43ecce6f27576cb548ba4341d0ec46a8


The other commit I would suggest being picked by src:xen is
5a4087004d1adbbb223925f3306db0e5824a2bdc

This is for device-tree funkiness which got added between linux-5.10.0
and linux-5.10.y (if the Debian kernel team wants to maintain a fix in
Debian's kernel source, that works too).

BTW have I mentioned I've become rather skeptical of device-trees being
a usable way of representing hardware information?


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[toc] | [prev] | [next] | [standalone]

#73020 — Bug#991967: #991967: Simply ACPI powerdown/reset issue?

[Multipart message — attachments visible in raw view] — view raw

On dinsdag 21 september 2021 01:15:15 CEST Elliott Mitchell wrote:
> Merely having the path is a sufficiently strong indicator for me to
> simply wave it past.  I though would suggest Debian should instead
> cherry-pick commit 0f089bbf43ecce6f27576cb548ba4341d0ec46a8.
> 
> This is available as a patch at:
> 
> https://xenbits.xen.org/gitweb/?p=xen.git;a=patch;h=0f089bbf43ecce6f27576cb548ba4341d0ec46a8

You probably then also want the following commit, which is a fix on that patch:
https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=bc141e8ca56200bdd0a12e04a6ebff3c19d6c27b

Found that via the following url/query:
https://xenbits.xen.org/gitweb/?p=xen.git&a=search&h=HEAD&st=commit&s=x86%2FACPI

I don't know whether others should be used from that as well.

[toc] | [prev] | [next] | [standalone]

#73026 — Bug#991967: #991967: Simply ACPI powerdown/reset issue?

On Mon, Sep 20, 2021 at 10:23:39PM -0400, Chuck Zmudzinski wrote:
> 
> On 9/20/21 7:39 PM, Diederik de Haas wrote:
> > On dinsdag 21 september 2021 01:15:15 CEST Elliott Mitchell wrote:
> >> Merely having the path is a sufficiently strong indicator for me to
> >> simply wave it past.  I though would suggest Debian should instead
> >> cherry-pick commit 0f089bbf43ecce6f27576cb548ba4341d0ec46a8.
> >>
> >> This is available as a patch at:
> >>
> >> https://xenbits.xen.org/gitweb/?p=xen.git;a=patch;h=0f089bbf43ecce6f27576cb548ba4341d0ec46a8
> > You probably then also want the following commit, which is a fix on that patch:
> > https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=bc141e8ca56200bdd0a12e04a6ebff3c19d6c27b
> >
> > Found that via the following url/query:
> > https://xenbits.xen.org/gitweb/?p=xen.git&a=search&h=HEAD&st=commit&s=x86%2FACPI
> >
> > I don't know whether others should be used from that as well.
> 
> I tried these two commits (adapted for the xen-4.14 branch) but this
> approach did not fix the bug - with these patches applied the dom0
> did not power down.
> 
> My advice for the Debian Xen Team is to consult with upstream and
> get their advice on whether or not it is advisable for Debian to
> retain the patches from the Xen-4.16 branch that have been
> added to the Debian 4.14 package in an attempt to support
> some arm devices that panic during on an unpatched Xen-4.14.
> If upstream cannot help Debian backport fixes for arm panics
> from Xen-4.16/unstable to Xen-4.14 stable, I think the Debian
> Xen team should remove aggressive patches that really have now
> turned the Debian Xen-4.14 package into a Frankenstein version
> that is a mixture of Xen-4.14 and Xen-4.16, and decide that support
> for those arm devices must wait until Debian gets Xen 4.16 up
> and running on the unstable and hopefully soon, testing distribution.

It is still not established you're running into #991967.  Unless the one
you're pointing towards was backported to the Xen 4.11 packages (which I
doubt) it cannot explain #991967, since at the time 4.11 was in use.

Could be this is a second bug with symptoms similar to #991967.  Now
that a fix for the second bug has been identified, you might try a
4.19.181-1 kernel and see whether that fixes things.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[toc] | [prev] | [next] | [standalone]

#73067 — Bug#991967: Simply ACPI powerdown/reset issue?

On Tue, Sep 21, 2021 at 06:33:20AM -0400, Chuck Zmudzinski wrote:
> I presume you are suggesting I try booting 4.19.181-1 on the
> current version of Xen-4.14 for bullseye as a dom0. I am not
> inclined to try it until an official Debian developer endorses
> your opinion that the bug I am seeing is distinct
> from #991967, at which point I will report the bug I am
> seeing as a new bug.

Chuck Zmudzinski you are getting rather close to my threshold for calling
harrassment.  You're not /quite/ there, but I'm concerned.


Since the purpose of the bug reports is to find and diagnose bugs, I did
a bit of experimentation and made some observations.

I checked out the Debian Xen source via git.  I got the current
"master" branch which is presently the candidate 4.14.3-1 version,
which includes urgent fixes.  The hash is:
e7a17db0305c8de891b366ad37777528e5a43015

On top of this I cherry-picked 3 commits from Xen's main branch:
5a4087004d1adbbb223925f3306db0e5824a2bdc
0f089bbf43ecce6f27576cb548ba4341d0ec46a8
bc141e8ca56200bdd0a12e04a6ebff3c19d6c27b

(these can be retrieved via Xen's gitweb at
https://xenbits.xen.org/gitweb/?p=xen.git;a=patch;h=<$hash> which is
suitable for the `git am` command)

With these I built 4.14.3-1 and then tried kernels 4.19.181-1 and
4.19.194-3 (this system is presently mostly on oldstable).  The results
were:

Xen 4.14.3-1 with Linux 4.19.181-1: system reboots were successful

Xen 4.14.3-1 with Linux 4.19.194-3: system reboots hung

Unfortunately I was too quick at installing the rebuilt 4.14.3-1 and I
missed trying the vanilla Debian 4.14.2+25-gb6a8c4f72d-2 with
Linux 4.19.181-1.  I believe this combination would have hung during
reboot.


As such, I believe there are in fact two distinct bugs being observed.
The presence of EITHER of these is sufficient to cause hangs during
powerdown or reboot.

First, some patch originally from Linux's main branch breaks Xen reboots
was backported somewhere between 4.19.181-1 and 4.19.194-3.  This may
either have been introduced before 5.10 diverged from main, or may also
have been backported to 5.10.  THIS is Debian bug #991967.

Second, the Xen patch 3c428e9ecb1f290689080c11e0c37b793425bef1 which is
valuable to ARM devices breaks reboots and powerdowns on x86.  This is
correctly fixed by 0f089bbf43ecce6f27576cb548ba4341d0ec46a8.  Presently
this has no Debian bug report.


The first is presently unidentified, someone enthusiastic either needs to
read git logs/source code, or bisect and build to find where it got
broken.

The second we seem to have a fix.  The only question is how many patches
to cherry pick?  bc141e8ca562 is non-urgent as it is merely superficial
and not needed for functionality.
5a4087004d1a is a workaround for Linux kernel breakage, but how likely
are we to see that fixed in the Linux kernel packages?  The fix is
well-contained and needed for some highly popular ARM devices.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[toc] | [prev] | [next] | [standalone]

#73076 — Bug#991967: Simply ACPI powerdown/reset issue?

[Multipart message — attachments visible in raw view] — view raw

Hi Elliott,

On zondag 26 september 2021 05:27:07 CEST Elliott Mitchell wrote:
> I checked out the Debian Xen source via git.  I got the current
> "master" branch which is presently the candidate 4.14.3-1 version,
> which includes urgent fixes.  The hash is:
> e7a17db0305c8de891b366ad37777528e5a43015
> 
> On top of this I cherry-picked 3 commits from Xen's main branch:
> 5a4087004d1adbbb223925f3306db0e5824a2bdc
> 0f089bbf43ecce6f27576cb548ba4341d0ec46a8
> bc141e8ca56200bdd0a12e04a6ebff3c19d6c27b

Shutdown on my Xen server broke for me between 4.14.0+80-gd101b417b7-1 and 
4.14.0+88-g1d1d1f5391-1 (too) and 'Knorrie' and I have been doing some 
experiments. We identified the 0f089bbf43 commit too, but also 2 other ones:

8b6d55c1261820bb9db8d867ce9ee77397d05203
f390941a92f102ebbbbce1b54be206a602187fd7

https://salsa.debian.org/xen-team/debian-xen/-/commits/knorrie/for-diederik-3-fixes/
is a branch Knorrie prepared for me with those 3 patches applied.
I did 'git checkout' on that branch and then a 'dpkg-buildpackage -b' and 
installed the built .deb files and rebooted. After that, shutdown worked again :)

So you may want to take a look at those patches too.

HTH,
  Diederik

[toc] | [prev] | [next] | [standalone]

#73198 — Bug#991967: Simply ACPI powerdown/reset issue?

Hi Elliot and others,

Also including #994899 for once, since that's the bug number for the Xen
issue now.

On 9/26/21 5:27 AM, Elliott Mitchell wrote:
> On Tue, Sep 21, 2021 at 06:33:20AM -0400, Chuck Zmudzinski wrote:
>> I presume you are suggesting I try booting 4.19.181-1 on the
>> current version of Xen-4.14 for bullseye as a dom0. I am not
>> inclined to try it until an official Debian developer endorses
>> your opinion that the bug I am seeing is distinct
>> from #991967, at which point I will report the bug I am
>> seeing as a new bug.
> 
> Chuck Zmudzinski you are getting rather close to my threshold for calling
> harrassment.  You're not /quite/ there, but I'm concerned.
> 
> 
> Since the purpose of the bug reports is to find and diagnose bugs, I did
> a bit of experimentation and made some observations.
> 
> I checked out the Debian Xen source via git.  I got the current
> "master" branch which is presently the candidate 4.14.3-1 version,
> which includes urgent fixes.  The hash is:
> e7a17db0305c8de891b366ad37777528e5a43015
> 
> On top of this I cherry-picked 3 commits from Xen's main branch:
> 5a4087004d1adbbb223925f3306db0e5824a2bdc
> 0f089bbf43ecce6f27576cb548ba4341d0ec46a8
> bc141e8ca56200bdd0a12e04a6ebff3c19d6c27b
> 
> (these can be retrieved via Xen's gitweb at
> https://xenbits.xen.org/gitweb/?p=xen.git;a=patch;h=<$hash> which is
> suitable for the `git am` command)
> 
> With these I built 4.14.3-1 and then tried kernels 4.19.181-1 and
> 4.19.194-3 (this system is presently mostly on oldstable).  The results
> were:
> 
> Xen 4.14.3-1 with Linux 4.19.181-1: system reboots were successful
> 
> Xen 4.14.3-1 with Linux 4.19.194-3: system reboots hung

Ok, so it included 0f089bbf43, which is probably the most important of
the 3 fixes that we need indeed. And, it's good that the above
difference is still visible afterwards, since it confirms that we're
looking at two distinct problems.

> Unfortunately I was too quick at installing the rebuilt 4.14.3-1 and I
> missed trying the vanilla Debian 4.14.2+25-gb6a8c4f72d-2 with
> Linux 4.19.181-1.  I believe this combination would have hung during
> reboot.

The Xen related breakage was introduced in 4.14.0+88-g1d1d1f5391-2, so
with that combination, I would expect you would experience both of the
bugs at the same time, yes.

> As such, I believe there are in fact two distinct bugs being observed.
> The presence of EITHER of these is sufficient to cause hangs during
> powerdown or reboot.
> 
> First, some patch originally from Linux's main branch breaks Xen reboots
> was backported somewhere between 4.19.181-1 and 4.19.194-3.  This may
> either have been introduced before 5.10 diverged from main, or may also
> have been backported to 5.10.  THIS is Debian bug #991967.
> 
> Second, the Xen patch 3c428e9ecb1f290689080c11e0c37b793425bef1 which is
> valuable to ARM devices breaks reboots and powerdowns on x86.  This is
> correctly fixed by 0f089bbf43ecce6f27576cb548ba4341d0ec46a8.  Presently
> this has no Debian bug report.

Correct. Thanks a lot for your help with hunting down and confirming this.

And now we have #994899 for it. So, I would like to kindly ask everyone
to stop hijacking this one, #991967, for discussing the Xen problem.

> The first is presently unidentified, someone enthusiastic either needs to
> read git logs/source code, or bisect and build to find where it got
> broken.
> 
> The second we seem to have a fix.  The only question is how many patches
> to cherry pick?  bc141e8ca562 is non-urgent as it is merely superficial
> and not needed for functionality.
> 5a4087004d1a is a workaround for Linux kernel breakage, but how likely
> are we to see that fixed in the Linux kernel packages?  The fix is
> well-contained and needed for some highly popular ARM devices.

Diederik also helped with testing changes, and when combining results,
the best thing we can do is pick the 4 changes that were initially
posted in Nov 2020 as "x86: ACPI and DMI table mapping fixes", and ended
up in Xen 4.15 as well.

---- >8 ----

commit 8b6d55c1261820bb9db8d867ce9ee77397d05203
Author: Jan Beulich <jbeulich@suse.com>
Date:   Tue Nov 24 11:26:02 2020 +0100

    x86/ACPI: fix mapping of FACS

commit f390941a92f102ebbbbce1b54be206a602187fd7
Author: Jan Beulich <jbeulich@suse.com>
Date:   Tue Nov 24 11:26:34 2020 +0100

    x86/DMI: fix table mapping when one lives above 1Mb

commit 0f089bbf43ecce6f27576cb548ba4341d0ec46a8
Author: Jan Beulich <jbeulich@suse.com>
Date:   Tue Jan 5 13:09:55 2021 +0100

    x86/ACPI: fix S3 wakeup vector mapping

commit 16ca5b3f873f17f4fbdaecf46c133e1aa3d623b2
Author: Jan Beulich <jbeulich@suse.com>
Date:   Tue Jan 5 13:11:04 2021 +0100

    x86/ACPI: don't invalidate S5 data when S3 wakeup vector cannot be
determined

---- >8 ----

The 4th one is not explicitly tagged with Fixes: 1c4aa69ca1e1, but I
agree with Diederik that we should keep them all together.

I do not know if this is also the thing Chuck tested in the end, but I'm
a bit lost in the walls of text that were produced in these two bugs.

https://salsa.debian.org/xen-team/debian-xen/-/merge_requests/14

These fixes were actually posted before 4.14.0+88-g1d1d1f5391-2
happened. It's unfortunate that we did not notice it, since the above
could have been part of the package that was in the archive when Debian
11 released. Or if anyone owning the specific type of hardware had ran
into it during testing during the freeze, we could also have found them
in time. But yeah, that happens.

Diederik, I think we should omit the 5th one, since it's a cosmetics
commit, which also starts touching (older) code unrelated to this issue.

What I plan to do is include these as regression fixes in the next
package update. The issue is only affecting a subset of hardware types.
There's a workaround (pull the plug), the fixes are known. There is no
security risk, there is no data corruption or unexpected crashes during
normal operation.

Hans

[toc] | [prev] | [next] | [standalone]

#73903 — Bug#991967: (Presently) Not in 5.10 source

Having finally gotten to test this, the issue does NOT effect 5.10.70-1.
So far I've only gotten to try reboot, but that went fine.

Might have been an ACPI or Xen mismerge into 4.19.  Alas this may simply
disappear into history.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web