Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #272424 > unrolled thread

QEMU: Run a container of a different architecture

Back to article view | Back to linux.debian.user

Contents

  QEMU: Run a container of a different architecture Steve Keller <keller.steve@gmx.de> - 2024-08-21 17:20 +0200
    Re: QEMU: Run a container of a different architecture Markus Schönhaber <debian-user@list-post.mks-mail.de> - 2024-08-21 19:10 +0200
      Re: QEMU: Run a container of a different architecture Steve Keller <keller.steve@gmx.de> - 2024-08-21 22:30 +0200
    Re: QEMU: Run a container of a different architecture Tim Woodall <debianuser@woodall.me.uk> - 2024-08-22 19:30 +0200
      Re: QEMU: Run a container of a different architecture Steve Keller <keller.steve@gmx.de> - 2024-08-23 16:30 +0200

#272424 — QEMU: Run a container of a different architecture

Can I run a container for a different CPU architecture using
systemd-nspawn?  I can easily install on my amd64 host a Debian
container of the same architecture and run that:

  # debootstrap stable deb12-amd64
  # systemd-nspawn -D deb12-amd64

and get a shell running in that container.  I can also install a
Debian system of a different architecture and run binaries from it
like this (using qemu-user-binfmt):

  # debootstrap --arch=arm64 --foreign stable deb-arm64
  # QEMU_LD_PREFIX=/usr/aarch64-linux-gnu deb-arm64/bin/date
  Wed Aug 21 16:43:40 CEST 2024

But the following doesn't work

  # QEMU_LD_PREFIX=/usr/aarch64-linux-gnu systemd-nspawn -D deb-arm64
  Spawning container deb-arm64 on /usr/local/deb-arm64.
  Press ^] three times within 1s to kill container.
  execv(/bin/bash, /bin/bash, /bin/sh) failed: No such file or directory
  Container deb-arm64 failed with error code 1.

I'd like to know if it's also possible to run the whole container in
arm64 architecture using systemd-nspawn like above for amd64.

Steve

[toc] | [next] | [standalone]

#272432

21.08.24, 16:56 +0200, Steve Keller:

> Can I run a container for a different CPU architecture using
> systemd-nspawn?

No. systemd-nspawn does indeed simply run a container. You can think of
that as a chroot on steroids. This means, everything inside the
container is run using the host kernel. Or to put it the other way
round: what the host kernel can't execute won't run.

If you want to run stuff for an architecture the host kernel cannot cope
with, you need a virtual machine (manager like QEMU). You can run one
using something like
qemu-system-aarch64 [...]
but it's probably much easier to use some helper like libvirt.

>  I can easily install on my amd64 host a Debian
> container of the same architecture and run that:
> 
>   # debootstrap stable deb12-amd64
>   # systemd-nspawn -D deb12-amd64
> 
> and get a shell running in that container.  I can also install a
> Debian system of a different architecture and run binaries from it
> like this (using qemu-user-binfmt):
> 
>   # debootstrap --arch=arm64 --foreign stable deb-arm64
>   # QEMU_LD_PREFIX=/usr/aarch64-linux-gnu deb-arm64/bin/date
>   Wed Aug 21 16:43:40 CEST 2024
> 
> But the following doesn't work
> 
>   # QEMU_LD_PREFIX=/usr/aarch64-linux-gnu systemd-nspawn -D deb-arm64
>   Spawning container deb-arm64 on /usr/local/deb-arm64.
>   Press ^] three times within 1s to kill container.
>   execv(/bin/bash, /bin/bash, /bin/sh) failed: No such file or directory
>   Container deb-arm64 failed with error code 1.
> 
> I'd like to know if it's also possible to run the whole container in
> arm64 architecture using systemd-nspawn like above for amd64.
As said above: no.

-- 
Regards
  mks

[toc] | [prev] | [next] | [standalone]

#272440

"Markus Schönhaber" writes:

> No. systemd-nspawn does indeed simply run a container. You can think of
> that as a chroot on steroids. This means, everything inside the
> container is run using the host kernel. Or to put it the other way
> round: what the host kernel can't execute won't run.

Actually, the kernel can execute such foreign ELF binaries.  This
works with the binfmt_misc file system and registering the appropriate
interpreter which is then automaically executed by the kernel.  This is
similar to the she-bang (!#) mechanism where the kernel also loads the
needed interpreter.

I've shown that in my mail with executing the aarch64 date binary
directly on the amd64 kernel.  I didn't see any reason why this
shouldn't work with all binaries in a container.  Now, when looking at
it again, I think the problem is the missing interpreter in the
container's file system.  Quite obvious.  OK, I'll try copying the
QEMU interpreter qemu-aarch64 and all needed libs and other files into
the appropriate place...

Steve

[toc] | [prev] | [next] | [standalone]

#272465

On Wed, 21 Aug 2024, Steve Keller wrote:

> Can I run a container for a different CPU architecture using
> systemd-nspawn?  I can easily install on my amd64 host a Debian
> container of the same architecture and run that:
>

Don't know about systemd-nspawn but I do something like this using
unshare, binfmt-support and qemu-user-static.

I don't have to do anything at all other than create the file system
with the emulated architecture and then chroot into it with those
packages installed.

$ apt-mark showmanual | grep binf
binfmt-support
$ apt-mark showmanual | grep qemu
qemu-user-static


I don't think I installed anything else but I might have missed
something.

Perhaps systemd-nspawn would similarly work with those packages
installed.

Tim.

[toc] | [prev] | [next] | [standalone]

#272488

"Tim Woodall" <debianuser@woodall.me.uk> writes:

> Don't know about systemd-nspawn but I do something like this using
> unshare, binfmt-support and qemu-user-static.
>
> I don't have to do anything at all other than create the file system
> with the emulated architecture and then chroot into it with those
> packages installed.

Thanks, the hint to qemu-user-static solved the issue very easily.
Nothing to install inside the container or other things to setup.  I
just installed qemu-user-static, which removed qemu-user-binfmt
because qemu-user-static brings its own binfmt support.

After installing I could simply run systemd-nspawn -D deb-arm64 and it
worked immediately, where deb-arm64 is the directory to which I have
installed the Debian/arm64 system using debootstrap.

I think systemd-nspawn does something very similar to what you do with
unshare and chroot.

> Perhaps systemd-nspawn would similarly work with those packages
> installed.

Yes, out of the box.  I find this a litte bit surprising since

    # head -n2 /proc/sys/fs/binfmt_misc/qemu-aarch64
    enabled
    interpreter /usr/libexec/qemu-binfmt/aarch64-binfmt-P
    # ls -l /usr/libexec/qemu-binfmt/aarch64-binfmt-P
    lrwxrwxrwx 1 root root 29 May  9 07:44 /usr/libexec/qemu-binfmt/aarch64-binfmt-P -> ../../bin/qemu-aarch64-static

these paths are not visible in the chroot environment or the mount
namespace, respectively.  That means the kernel probably does the
interpreter lookup earlier, not at execution time in the chrooted
environment.  Besides that, qemu-aarch64-static doesn't need any other
files from the host environment.  In contrast, qemu-aarch64 which I had
used instead, needs libraries which it cannot find in the chrooted
environment and therefore fails.

Nevertheless, it would be nice to be able to run qemu-aarch64
dynamically linked which finds its libs in the host environment but
runs the command in the current inherited environment.  But I guess
that's impossible.

Steve

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web