Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #71705 > unrolled thread

Bug#983357: Netinst crashes xen domU when loading kernel

Back to article view | Back to linux.debian.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#983357: Netinst crashes xen domU when loading kernel Phillip Susi <phill@thesusis.net> - 2021-05-19 15:40 +0200
    Bug#983357: Netinst crashes xen domU when loading kernel Cyril Brulebois <kibi@debian.org> - 2021-05-24 06:30 +0200
      Bug#983357: Netinst crashes xen domU when loading kernel Michael Biebl <biebl@debian.org> - 2021-05-24 09:40 +0200
        Bug#983357: Netinst crashes xen domU when loading kernel Phillip Susi <phill@thesusis.net> - 2021-05-25 21:50 +0200
        Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel Ben Hutchings <ben@decadent.org.uk> - 2021-08-24 19:20 +0200
          Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel Phillip Susi <phill@thesusis.net> - 2021-08-24 21:50 +0200
            Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel Ben Hutchings <ben@decadent.org.uk> - 2021-08-25 01:20 +0200
              Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel Ben Hutchings <ben@decadent.org.uk> - 2021-08-25 19:00 +0200
              Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel Phillip Susi <phill@thesusis.net> - 2021-08-25 22:30 +0200
          Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel Ben Hutchings <ben@decadent.org.uk> - 2021-08-25 17:00 +0200

#71705 — Bug#983357: Netinst crashes xen domU when loading kernel

The discussion upstream does not seem to be converging on a proper fix
in the kernel, so I'm going to clone this bug and suggest that
debian-installer patch the start-udev script to ignore the failure of
the udevadm trigger command.

To summarize: init ends up calling start-udev which calls udevadm
trigger to cold plug all devices.  Both scripts are set -e.  The Xen
Virtual Keyboard driver and at least one other driver have always failed
to trigger due to having absurdly long modalias, but the error used to
be ignored.  The kernel now returns the error to udevadm, so it exits
with an error, so start-udev exits with an error, so init exits with an
error, causing the kernel to panic.

[toc] | [next] | [standalone]

#71840

[Multipart message — attachments visible in raw view] — view raw

Hi Phillip,

And thanks for debugging this… I must confess I've never touched
anything Xen related and I'd like to keep it that way in the near
future. ;-)

Phillip Susi <phill@thesusis.net> (2021-05-19):
> The discussion upstream does not seem to be converging on a proper fix
> in the kernel, so I'm going to clone this bug and suggest that
> debian-installer patch the start-udev script to ignore the failure of
> the udevadm trigger command.
> 
> To summarize: init ends up calling start-udev which calls udevadm
> trigger to cold plug all devices.  Both scripts are set -e.  The Xen
> Virtual Keyboard driver and at least one other driver have always failed
> to trigger due to having absurdly long modalias, but the error used to
> be ignored.  The kernel now returns the error to udevadm, so it exits
> with an error, so start-udev exits with an error, so init exits with an
> error, causing the kernel to panic.

Well, it's a little more complicated:
 - start-udev is actually a script shipped by the udev udeb, i.e. the
   responsibility of systemd maintainers;
 - based on a quick grep, the installer contains two calls to that
   start-udev script, in the rootskel source package:
    + rootskel/src/init            (shipped as /init in the udeb)
    + rootskel/src/sbin/init-linux (shipped as /sbin/init in the udeb)

I'd be happy to have a comment from systemd maintainers before thinking
about patching rootskel. :)


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[toc] | [prev] | [next] | [standalone]

#71846

Hi Phillip

Am 24.05.2021 um 06:19 schrieb Cyril Brulebois:
>> trigger to cold plug all devices.  Both scripts are set -e.  The Xen
>> Virtual Keyboard driver and at least one other driver have always failed
>> to trigger due to having absurdly long modalias, but the error used to
>> be ignored.  The kernel now returns the error to udevadm

So this is a change in behaviour in the kernel?
What happens if you boot the installed system? Does udevadm trigger fail 
there as well?

I feel a bit uneasy changing the udev start script this late in the 
release cycle (especially when it appears like covering up an issue 
someplace else).

I'll let Marco make the judgement on this though, as he has the most 
experience with those udev udeb start scripts as the original author.

Michael

[toc] | [prev] | [next] | [standalone]

#71883

Michael Biebl writes:

> So this is a change in behaviour in the kernel?

Yes, this commit fixed the kernel to report the error instead of
silently failing:

commit df44b479654f62b478c18ee4d8bc4e9f897a9844
Author: Peter Rajnoha <prajnoha@redhat.com>
Date:   Wed Dec 5 12:27:44 2018 +0100

    kobject: return error code if writing /sys/.../uevent fails
    
    Propagate error code back to userspace if writing the /sys/.../uevent
    file fails. Before, the write operation always returned with success,
    even if we failed to recognize the input string or if we failed to
    generate the uevent itself.
    
    With the error codes properly propagated back to userspace, we are
    able to react in userspace accordingly by not assuming and awaiting
    a uevent that is not delivered.
    
    Signed-off-by: Peter Rajnoha <prajnoha@redhat.com>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

> What happens if you boot the installed system? Does udevadm trigger fail 
> there as well?

Yes, it does; that is how I was able to track down the problem.

> I feel a bit uneasy changing the udev start script this late in the 
> release cycle (especially when it appears like covering up an issue 
> someplace else).
>
> I'll let Marco make the judgement on this though, as he has the most 
> experience with those udev udeb start scripts as the original author.

So far I have been removing the -e from the shbang line in the
start-udev script and remastering the iso so I can get it to boot.  It
would probably be a better idea to just add a || true to the udevadm
trigger call.  I feel fairly certain that no matter what the cause of
the coldplug failure, the user is going to be better off ignoring it and
trying to proceed than a kernel panic.

[toc] | [prev] | [next] | [standalone]

#72735 — Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel

[Multipart message — attachments visible in raw view] — view raw

On Tue, 2021-08-24 at 10:56 -0400, Chuck Zmudzinski wrote:
> On 5/24/2021 3:30 AM, Michael Biebl wrote:
> > Hi Phillip
> > 
> > Am 24.05.2021 um 06:19 schrieb Cyril Brulebois:
> > > > trigger to cold plug all devices.  Both scripts are set -e.  The Xen
> > > > Virtual Keyboard driver and at least one other driver have always 
> > > > failed
> > > > to trigger due to having absurdly long modalias, but the error used to
> > > > be ignored.  The kernel now returns the error to udevadm
> > 
> > So this is a change in behaviour in the kernel?
> > What happens if you boot the installed system? Does udevadm trigger 
> > fail there as well?
> > 
> > I feel a bit uneasy changing the udev start script this late in the 
> > release cycle (especially when it appears like covering up an issue 
> > someplace else).
> > 
> > I'll let Marco make the judgement on this though, as he has the most 
> > experience with those udev udeb start scripts as the original author.
> > 
> > Michael
> > 
> 
> After reviewing Philip's message at
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983357#43
> 
> which seems to point to the root cause of this bug, I can add:
> 
> On my Xen HVM DomU I see the absurdly long modalias for the Xen
> Virtual keyboard that seems to be causing this crash in sysfs at
> 
> /sys/devices/virtual/input/input2/modalias
> 
> But at /sys/devices/vkbd-0/modalias, I see just 'xen:vkbd', which would
> probably not result in an error in the udev script if this was also
> written as the modalias at /sys/devices/virtual/input/input2/modalias
>
> So the Xen virtual keyboard appears more than once in sysfs, and
> modalias is not the same in the different places. This seems
> to be a problem.

They are two different devices, and they should have different
modaliases.

Linux has code for discovering devices on each kind of bus, including
virtual buses, and that code creates "bus devices" such as vkbd-0.  At
this point the kernel doesn't know what the device is capable of.  The
modalias for a bus device carries some identifying information that can
be used to select a driver module for it.

The driver does know what the device is capable of, and how to use it.
It will normally create one or more "class devices" that support a
particular set of operations; in this case input device operations. 
Class devices typically don't have modaliases, since they don't need
another layer of drivers on top.  However, for input devices the
modalias carries information about the device's capabilities.  These
may trigger loading of the evdev or joydev module.

> I understand the correct way to fix this bug is by modifying the
> Xxen virtual keyboard (and any other devices that might cause
> this crash) and not the start-udev script on the netinst
> installation media, which is so far the only available workaround.
> Hopefully Xen will accept a fix if we can come up with a fix.
[...]

I think a proper fix would be one of:

a. If the Xen virtual keyboard driver is advertising capabilities it
   doesn't have, stop it doing that.
b. Change the implementation of modalias attributes to allow longer
   values.

It's not clear to me whether the Xen driver is advertising correctly or
not.  If it is, then the solution should be b, but that may be too
disruptive a change to the kernel.  So a reasonable workaround might
be:

c. Change the input subsystem to limit the length of the
   capabilities part of the modalias.


Ben.

-- 
Ben Hutchings
73.46% of all statistics are made up.

[toc] | [prev] | [next] | [standalone]

#72736 — Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel

Ben Hutchings <ben@decadent.org.uk> writes:

> I think a proper fix would be one of:
>
> a. If the Xen virtual keyboard driver is advertising capabilities it
>    doesn't have, stop it doing that.
> b. Change the implementation of modalias attributes to allow longer
>    values.
>
> It's not clear to me whether the Xen driver is advertising correctly or
> not.  If it is, then the solution should be b, but that may be too
> disruptive a change to the kernel.  So a reasonable workaround might
> be:
>
> c. Change the input subsystem to limit the length of the
>    capabilities part of the modalias.

The problem with a) is that the Xen keyboard is not a physical keyboard
and so it has no way of knowing what keys it actually has.  It is a fake
input device designed to pass through whatever input the Xen hypervisor
sends down.  As such, any key could come in.  If it doesn't advertise
that it has all of these keys, then they would not be accepted by
libinput when the hypervisor sends them down.

This seems to be the heart of the problem: libinput was designed
assuming that all keyboards can and must report what keys are actually
present, and then libinput tries to cram that information into the
modalias rather than some other sysfs attribute as it should ( or not at
all... I still don't see how this information is actually supposed to be
useful to userspace ).

As for b), the problem isn't with the modalias attribute itself, but
when the kernel tries to copy it into the environment block for the udev
callout.  The environment block is only a single page, and so limited to
4 KB.  And that's for everything else that goes into the environment,
not just the modalias.

[toc] | [prev] | [next] | [standalone]

#72737 — Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel

[Multipart message — attachments visible in raw view] — view raw

On Tue, Aug 24, 2021 at 03:27:19PM -0400, Phillip Susi wrote:
> 
> Ben Hutchings <ben@decadent.org.uk> writes:
> 
> > I think a proper fix would be one of:
> >
> > a. If the Xen virtual keyboard driver is advertising capabilities it
> >    doesn't have, stop it doing that.
> > b. Change the implementation of modalias attributes to allow longer
> >    values.
> >
> > It's not clear to me whether the Xen driver is advertising correctly or
> > not.  If it is, then the solution should be b, but that may be too
> > disruptive a change to the kernel.  So a reasonable workaround might
> > be:
> >
> > c. Change the input subsystem to limit the length of the
> >    capabilities part of the modalias.
> 
> The problem with a) is that the Xen keyboard is not a physical keyboard
> and so it has no way of knowing what keys it actually has.  It is a fake
> input device designed to pass through whatever input the Xen hypervisor
> sends down.  As such, any key could come in.  If it doesn't advertise
> that it has all of these keys, then they would not be accepted by
> libinput when the hypervisor sends them down.

Right, that's what I feared.

xen-kbdfront is setting the bits for keys in the ranges [KEY_ESC,
KEY_UNKNOWN) and [KEY_OK, KEY_MAX), which I think works out to 654
keys and 2362 bytes in the modalias.

> This seems to be the heart of the problem: libinput was designed
> assuming that all keyboards can and must report what keys are actually
> present, and then libinput tries to cram that information into the
> modalias rather than some other sysfs attribute as it should ( or not at
> all... I still don't see how this information is actually supposed to be
> useful to userspace ).

I think modaliases aren't intended to be interpreted by user-space,
other than processing wildcards when matching to modules.

For input devices, the same information is available through other
variables in the uevent, in a more compact form.  The information *is*
useful for user-space; e.g. in initramfs-tools we recognise keyboard
devices and add their drivers to the initramfs but ignore other input
devices.

> As for b), the problem isn't with the modalias attribute itself, but
> when the kernel tries to copy it into the environment block for the udev
> callout.  The environment block is only a single page, and so limited to
> 4 KB.  And that's for everything else that goes into the environment,
> not just the modalias.

Text-based sysfs attributes are limited to a page, but udev receives
uevents through netlink, not sysfs.

The current limit on the environment of a uevent appears to be 2 KB
(UEVENT_BUFFER_SIZE defined in <linux/kobject.h>).  That seems like it
*might* be easier to change, so long as user-space doesn't have a
similar limit.

I looked into systemd/udev, and it seems to use an 8 KB buffer for
receiving uevents:

https://sources.debian.org/src/systemd/247.9-1/src/libsystemd/sd-device/device-monitor.c/?hl=390#L390

But as a first step I think increasing the kernel buffer size to 4 KB
would be enough.  Perhaps someone could test whether this patch to the
domU kernel makes udev happier:

--- a/include/linux/kobject.h
+++ b/include/linux/kobject.h
@@ -30,7 +30,7 @@
 
 #define UEVENT_HELPER_PATH_LEN		256
 #define UEVENT_NUM_ENVP			64	/* number of env pointers */
-#define UEVENT_BUFFER_SIZE		2048	/* buffer for the variables */
+#define UEVENT_BUFFER_SIZE		4096	/* buffer for the variables */
 
 #ifdef CONFIG_UEVENT_HELPER
 /* path to the userspace helper executed on an event */
--- END ---

?

Ben.

-- 
Ben Hutchings
Design a system any fool can use, and only a fool will want to use it.

[toc] | [prev] | [next] | [standalone]

#72748 — Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel

[Multipart message — attachments visible in raw view] — view raw

On Wed, 2021-08-25 at 12:45 -0400, Chuck Zmudzinski wrote:
[...]
> 
> I will try it in my bullseye Xen HVM DomU.
> 
> I am not sure how to rebuild the installation media with a patched
> systemd, but I can patch my installed Xen HVM DomU system
> with a patched systemd with the increased buffer size and see if the
> Coldplug failure early in the boot process goes away. If so, then it
> is likely this patch to systemd would also fix the installation media.
[...]

Sorry for not being clear - this is a patch for the kernel. 
Instructions for rebuilding the kernel package are at
<https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official>.

I agree that you should check whether this fixes the coldplug error
before we try rebuilding the installer.

Ben.

-- 
Ben Hutchings
Design a system any fool can use, and only a fool will want to use it.

[toc] | [prev] | [next] | [standalone]

#72755 — Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel

Chuck Zmudzinski <brchuckz@netscape.net> writes:
> If it doesn't work, I am also willing to try approach a by patching
> the Linux kernel xen-kbdfront driver by removing the for loops that
> advertise those 654 keys. I tend to agree with Philip that this is
> totally unnecessary, but I suppose I could be wrong about that.
> I read the discussion Philip had with the Xen developers and they
> seemed to want to keep the Xen keyboard driver as it is.

That was the first thing I tried and the libinput maintainer pointed out
that if you don't advertise the keys, you can't use the keys.  In other
words, somebody presses that key on their keyboard and the domU won't
recognize it.

[toc] | [prev] | [next] | [standalone]

#72746 — Bug#983357: Bug#988776: Bug#983357: Netinst crashes xen domU when loading kernel

[Multipart message — attachments visible in raw view] — view raw

On Tue, 2021-08-24 at 15:19 -0400, Chuck Zmudzinski wrote:
> On 8/24/2021 1:12 PM, Ben Hutchings wrote:
[...]

> > I think a proper fix would be one of:
> > 
> > a. If the Xen virtual keyboard driver is advertising capabilities it
> >     doesn't have, stop it doing that.
> > b. Change the implementation of modalias attributes to allow longer
> >     values.
> > 
> > It's not clear to me whether the Xen driver is advertising correctly or
> > not.  If it is, then the solution should be b, but that may be too
> > disruptive a change to the kernel.  So a reasonable workaround might
> > be:
> > 
> > c. Change the input subsystem to limit the length of the
> >     capabilities part of the modalias.
> > 
> > 
> > Ben.
> > 
> 
> So workaround c would not involve disruptions to the kernel or
> systemd? Workaround c seems too disruptive for stable to me,
> but maybe could go into unstable and eventually into testing.

I don't think it would be very disruptive.  It might require a kernel
ABI bump, but we do those regularly during a stable release.  And this
bug is severe enough that I think a fix would be suitable for Debian
stable.

> A problem with the approach of fixing this bug in the Xen
> keyboard driver is that the fix must be implemented in the underlying
> Dom0 system, which could be almost anything - another Linux distro
> or Debian stable or oldstable. Any fix upstream would probably get into
> a bullseye Dom0, but not oldstable Dom0, but perhaps it could be
> provided as a backport for anyone who is still on oldstable for their
> Xen Dom0.
[...]

I agree that we need to fix this for domU independently of any protocol
change to allow discovery of which keys the underlying input device
has.  So we can't solve this with approach a.


Ben.

-- 
Ben Hutchings
Design a system any fool can use, and only a fool will want to use it.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web