Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #59318 > unrolled thread

Bug#880502: lxc: cannot start container (kernel related?)

Back to article view | Back to linux.debian.kernel

Contents

  Bug#880502: lxc: cannot start container (kernel related?) Antonio Terceiro <terceiro@debian.org> - 2017-11-01 14:40 +0100
    Bug#880502: [pkg-lxc-devel] Bug#880502: lxc: cannot start container with kernel 4.13.10 Evgeni Golov <evgeni@debian.org> - 2017-11-01 15:50 +0100
      Bug#880502: [pkg-lxc-devel] Bug#880502: lxc: cannot start container with kernel 4.13.10 Ben Hutchings <ben@decadent.org.uk> - 2017-11-01 22:20 +0100
    Bug#880502: [pkg-apparmor] Bug#880502: [pkg-lxc-devel] Bug#880502: lxc: cannot start container with kernel 4.13.10 Christian Boltz <debian-bugs@cboltz.de> - 2017-11-02 19:30 +0100
      Bug#880502: lxc: cannot start container with kernel 4.13.10 Evgeni Golov <evgeni@debian.org> - 2017-11-02 20:20 +0100
        Bug#880502: [pkg-apparmor] Bug#880502: lxc: cannot start container with kernel 4.13.10 Felix Geyer <fgeyer@debian.org> - 2017-11-03 19:40 +0100
          Bug#880502: [pkg-apparmor] Bug#880502: lxc: cannot start container with kernel 4.13.10 intrigeri <intrigeri@debian.org> - 2017-11-05 13:30 +0100
    Bug#880502: [pkg-apparmor] Bug#880502: [pkg-lxc-devel] Bug#880502: lxc: cannot start container with kernel 4.13.10 intrigeri <intrigeri@debian.org> - 2017-11-05 13:20 +0100

#59318 — Bug#880502: lxc: cannot start container (kernel related?)

[Multipart message — attachments visible in raw view] — view raw

Package: lxc
Version: 1:2.0.9-3
Severity: serious

I'm filing this in lxc initially as I don't know exactly where the issue
is yet. We will probably want to reassign it.

Something other than lxc itself changed recently in unstable which makes
lxc not able to start a Debian container:

# lxc-start -n autopkgtest-sid-amd64
lxc-start: lxccontainer.c: wait_on_daemonized_start: 754 Received container state "ABORTING" instead of "RUNNING"
lxc-start: tools/lxc_start.c: main: 368 The container failed to start.
lxc-start: tools/lxc_start.c: main: 370 To get more details, run the container in foreground mode.
lxc-start: tools/lxc_start.c: main: 372 Additional information can be obtained by setting the --logfile and --logpriority options.
# cat /var/lib/lxc/autopkgtest-sid-amd64/autopkgtest-sid-amd64.log
      lxc-start 20171101123914.655 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:220 - If you really want to start this container, set
      lxc-start 20171101123914.655 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:221 - lxc.aa_allow_incomplete = 1
      lxc-start 20171101123914.655 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:222 - in your container configuration file
      lxc-start 20171101123914.655 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
      lxc-start 20171101123914.701 ERROR    lxc_container - lxccontainer.c:wait_on_daemonized_start:754 - Received container state "ABORTING" instead of "RUNNING"
      lxc-start 20171101123914.701 ERROR    lxc_start - start.c:__lxc_start:1530 - Failed to spawn container "autopkgtest-sid-amd64".
      lxc-start 20171101123914.701 ERROR    lxc_start_ui - tools/lxc_start.c:main:368 - The container failed to start.
      lxc-start 20171101123914.701 ERROR    lxc_start_ui - tools/lxc_start.c:main:370 - To get more details, run the container in foreground mode.
      lxc-start 20171101123914.701 ERROR    lxc_start_ui - tools/lxc_start.c:main:372 - Additional information can be obtained by setting the --logfile and --logpriority options.
      lxc-start 20171101132533.307 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:220 - If you really want to start this container, set
      lxc-start 20171101132533.307 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:221 - lxc.aa_allow_incomplete = 1
      lxc-start 20171101132533.307 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:222 - in your container configuration file
      lxc-start 20171101132533.307 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
      lxc-start 20171101132533.373 ERROR    lxc_container - lxccontainer.c:wait_on_daemonized_start:754 - Received container state "ABORTING" instead of "RUNNING"
      lxc-start 20171101132533.374 ERROR    lxc_start_ui - tools/lxc_start.c:main:368 - The container failed to start.
      lxc-start 20171101132533.374 ERROR    lxc_start - start.c:__lxc_start:1530 - Failed to spawn container "autopkgtest-sid-amd64".
      lxc-start 20171101132533.374 ERROR    lxc_start_ui - tools/lxc_start.c:main:370 - To get more details, run the container in foreground mode.
      lxc-start 20171101132533.374 ERROR    lxc_start_ui - tools/lxc_start.c:main:372 - Additional information can be obtained by setting the --logfile and --logpriority options.


This is not happening on testing yet. When I upgrade a testing VM to
unstable, I can still start the container before a reboot. After a
reboot, I cannot start a container anymore. Maybe it's related to some
kernel change?

I'm copying debian-kernel in case someone there can provide some insight.

-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'unstable'), (500, 'testing'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.13.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=pt_BR.UTF-8, LC_CTYPE=pt_BR.UTF-8 (charmap=UTF-8), LANGUAGE=pt_BR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages lxc depends on:
ii  libapparmor1  2.11.1-2
ii  libc6         2.24-17
ii  libcap2       1:2.25-1.1
ii  libgnutls30   3.5.16-1
ii  liblxc1       1:2.0.9-3
ii  libseccomp2   2.3.1-2.1
ii  libselinux1   2.7-2
ii  lsb-base      9.20170808
ii  python3       3.6.3-2
ii  python3-lxc   1:2.0.9-3

Versions of packages lxc recommends:
ii  bridge-utils  1.5-14
ii  debootstrap   1.0.92
ii  dirmngr       2.2.1-5
ii  dnsmasq-base  2.78-1
ii  gnupg         2.2.1-5
ii  iptables      1.6.1-2+b1
ii  libpam-cgfs   2.0.8-1
ii  lxcfs         2.0.8-1
ii  openssl       1.1.0f-5
ii  rsync         3.1.2-2
ii  uidmap        1:4.5-1

Versions of packages lxc suggests:
pn  apparmor     <none>
ii  btrfs-progs  4.13.3-1
ii  lvm2         2.02.173-1

-- Configuration Files:
/etc/lxc/default.conf changed [not included]

-- no debconf information

[toc] | [next] | [standalone]

#59322 — Bug#880502: [pkg-lxc-devel] Bug#880502: lxc: cannot start container with kernel 4.13.10

Ohai,

On Wed, Nov 01, 2017 at 12:00:12PM -0200, Antonio Terceiro wrote:
> >       lxc-start 20171101123914.655 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:220 - If you really want to start this container, set
> >       lxc-start 20171101123914.655 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:221 - lxc.aa_allow_incomplete = 1
> >       lxc-start 20171101123914.655 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:222 - in your container configuration file
> So, I tried downgrading the kernel to the one in testing, rebooted, and
> now I can start containers again, So this is being caused by a change in
> the kernel between 4.13.4-2 and 4.13.10-1
> 
> I still need to study the lxc code path that is being triggered to be
> able to provide more useful information. Since the issue is definitively
> related to apparmor, I am also copying the apparmor team in case they
> have any input to provide.

Can you try to set "lxc.aa_allow_incomplete = 1" in your config?
LXC expects Ubuntus patched kernels when it comes to AppArmor, not the
upstream ones :(

And I think Debian enabled AppArmor by default in the latest kernels.

Evgeni

[toc] | [prev] | [next] | [standalone]

#59325 — Bug#880502: [pkg-lxc-devel] Bug#880502: lxc: cannot start container with kernel 4.13.10

[Multipart message — attachments visible in raw view] — view raw

On Wed, 2017-11-01 at 15:38 +0100, Evgeni Golov wrote:
> Ohai,
> 
> On Wed, Nov 01, 2017 at 12:00:12PM -0200, Antonio Terceiro wrote:
> > >       lxc-start 20171101123914.655 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:220 - If you really want to start this container, set
> > >       lxc-start 20171101123914.655 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:221 - lxc.aa_allow_incomplete = 1
> > >       lxc-start 20171101123914.655 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:222 - in your container configuration file
> > 
> > So, I tried downgrading the kernel to the one in testing, rebooted, and
> > now I can start containers again, So this is being caused by a change in
> > the kernel between 4.13.4-2 and 4.13.10-1
> > 
> > I still need to study the lxc code path that is being triggered to be
> > able to provide more useful information. Since the issue is definitively
> > related to apparmor, I am also copying the apparmor team in case they
> > have any input to provide.
> 
> Can you try to set "lxc.aa_allow_incomplete = 1" in your config?
> LXC expects Ubuntus patched kernels when it comes to AppArmor, not the
> upstream ones :(
> 
> And I think Debian enabled AppArmor by default in the latest kernels.

Yes, that's the change made in 4.13.10-1.

Ben.

-- 
Ben Hutchings
Make three consecutive correct guesses and you will be considered an
expert.

[toc] | [prev] | [next] | [standalone]

#59333 — Bug#880502: [pkg-apparmor] Bug#880502: [pkg-lxc-devel] Bug#880502: lxc: cannot start container with kernel 4.13.10

[Multipart message — attachments visible in raw view] — view raw

Hello,

seeing the AppArmor denials would be helpful to get this fixed ;-)

Please either
    grep -i apparmor /var/log/syslog 
or, if you have auditd installed, check
    /var/log/audit/audit.log

For more details, see https://wiki.debian.org/AppArmor/Debug


Regards,

Christian Boltz
-- 
> Anyway, what does our mission statement say?                                                                                                                                    
"Have a lot of fun..."                                                                                                                                                            
[> Per Jessen and Greg KH in opensuse-factory]                                                                                                                                    

[toc] | [prev] | [next] | [standalone]

#59334 — Bug#880502: lxc: cannot start container with kernel 4.13.10

Hi,

On Thu, Nov 02, 2017 at 07:09:10PM +0100, Christian Boltz wrote:
> seeing the AppArmor denials would be helpful to get this fixed ;-)

I think the issue is different.

Looking at the LXC log, we see the following:
lxc-start 20171102130036.516 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:234 - No such file or directory - failed to change apparmor profile to lxc-container-default-cgns

And indeed, we see no profiles:
# aa-status
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

I think the issue is that when LXC is installed *before* AppArmor is
enabled, the postinst snippet generated by dh_apparmor [1] is not
registering any profiles. And now that AppArmor is enabled, the profile
is missing and cannot be applied.

This is just a theory, I did not have time to actually reproduce and try
it.

Evgeni

[1]
# Automatically added by dh_apparmor/2.11.1-2
aa_is_enabled() {
   if command aa-enabled >/dev/null 2>&1; then
      # apparmor >= 2.10.95-2
      aa-enabled --quiet 2>/dev/null
   else
      # apparmor << 2.10.95-2
      # (This should be removed once Debian Stretch and Ubuntu 18.04 are out.)
      rc=0
      aa-status --enabled 2>/dev/null || rc=$?
      [ "$rc" = 0 ] || [ "$rc" = 2 ]
   fi
}

if [ "$1" = "configure" ]; then
    APP_PROFILE="/etc/apparmor.d/usr.bin.lxc-start"
    if [ -f "$APP_PROFILE" ]; then
        # Add the local/ include
        LOCAL_APP_PROFILE="/etc/apparmor.d/local/usr.bin.lxc-start"

        test -e "$LOCAL_APP_PROFILE" || {
            tmp=`mktemp`
        cat <<EOM > "$tmp"
# Site-specific additions and overrides for usr.bin.lxc-start.
# For more details, please see /etc/apparmor.d/local/README.
EOM
            mkdir `dirname "$LOCAL_APP_PROFILE"` 2>/dev/null || true
            mv -f "$tmp" "$LOCAL_APP_PROFILE"
            chmod 644 "$LOCAL_APP_PROFILE"
        }

        # Reload the profile, including any abstraction updates
        if aa_is_enabled; then
            apparmor_parser -r -T -W "$APP_PROFILE" || true
        fi
    fi
fi
# End automatically added section

[toc] | [prev] | [next] | [standalone]

#59343 — Bug#880502: [pkg-apparmor] Bug#880502: lxc: cannot start container with kernel 4.13.10

Hi,

On 02.11.2017 20:09, Evgeni Golov wrote:
> Hi,
> 
> On Thu, Nov 02, 2017 at 07:09:10PM +0100, Christian Boltz wrote:
>> seeing the AppArmor denials would be helpful to get this fixed ;-)
> 
> I think the issue is different.
> 
> Looking at the LXC log, we see the following:
> lxc-start 20171102130036.516 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:234 - No such file or directory - failed to change apparmor profile to lxc-container-default-cgns
> 
> And indeed, we see no profiles:
> # aa-status
> apparmor module is loaded.
> 0 profiles are loaded.
> 0 profiles are in enforce mode.
> 0 profiles are in complain mode.
> 0 processes have profiles defined.
> 0 processes are in enforce mode.
> 0 processes are in complain mode.
> 0 processes are unconfined but have a profile defined.
> 
> I think the issue is that when LXC is installed *before* AppArmor is
> enabled, the postinst snippet generated by dh_apparmor [1] is not
> registering any profiles. And now that AppArmor is enabled, the profile
> is missing and cannot be applied.

There are two issues:

lxc expects mount mediation to be present in AppArmor. This isn't upstream (yet) so it's missing
from the Debian kernel too.
As already mentioned there is a lxc.aa_allow_incomplete setting to ignore this check.
However lxc-apparmor-load doesn't honor this setting and still skips loading profiles.


More fundamentally lxc makes the assumption that the AppArmor userspace tools are available if
AppArmor is active in the kernel.
When starting a container lxc detects that AppArmor is active and tries to transition to a
profile. This fails if the apparmor package hasn't been installed as lxc has no way to load profiles.


To fix this:
- lxc needs to stop checking for AppArmor mount mediation. This might makes sense for distros that
ship a kernel with the AppArmor patchset but not for everyone else.
- lxc must allow for the AppArmor userspace tools to be absent. This could be done by checking if
the binaries are present on the system or by checking for ENOENT after aa_change_profile() calls.

Felix

[toc] | [prev] | [next] | [standalone]

#59356 — Bug#880502: [pkg-apparmor] Bug#880502: lxc: cannot start container with kernel 4.13.10

Hi!

Sorry for the delay, I didn't expect AppArmor to be enabled in the
kernel a week ago (I thought I would coordinate this with Ben)
and I was busy with the Reproducible Builds summit this week.

Thanks Felix & Antonio for being on top of things. I'm glad the
immediate RC issue was fixed.

Felix Geyer:
> There are two issues:

> lxc expects mount mediation to be present in AppArmor. This isn't upstream (yet) so it's missing
> from the Debian kernel too.

FYI mount mediation is upstream since some time in the 4.14 cycle.
We have it in Debian experimental (Linux 4.14.0-rc7).

But for now I've disabled it on Debian even when running Linux 4.14.
It'll be enabled at some point in the future, not sure when exactly
(#880078).

> More fundamentally lxc makes the assumption that the AppArmor userspace tools are available if
> AppArmor is active in the kernel.
> When starting a container lxc detects that AppArmor is active and tries to transition to a
> profile. This fails if the apparmor package hasn't been installed as lxc has no way to load profiles.

I believe libvirt implements the exact same logic… minus the bug.
This might provide inspiration to whoever wants to fix this bug in
LXC :)

If these bugs are not tracked upstream yet: Felix, you seem to be the
one of us with the best understanding of the problem and you know
AppArmor pretty well, so perhaps you would be the best person to
report them?

Cheers,
-- 
intrigeri

[toc] | [prev] | [next] | [standalone]

#59355 — Bug#880502: [pkg-apparmor] Bug#880502: [pkg-lxc-devel] Bug#880502: lxc: cannot start container with kernel 4.13.10

Hi,

Antonio Terceiro:
> The workaround that works is using the setting in the container
> configuration:

> lxc.aa_profile = unconfined

> with disables apparmor entirely.

> I have just uploaded lxc 1:2.0.9-4 setting this for all containers. This
> is not the greatest solution, but it's also not worse than the state of
> affairs before apparmor was enabled by default in the Debian kernel: it
> was already not possible to use lxc with apparmor in Debian.

Fully agreed: top priority is to ensure AppArmor doesn't break things,
so let's disable any profile that is not ready for prime time.

Adding AppArmor confinement where we had none previously can
come later.

Cheers,
-- 
intrigeri

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web