Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1182480

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR?

Show all headers | View raw

Hi,

thank you Helmut for your (again) very thorough reply. Just as with your 
last large mail to this bug, I can only second the things you said and 
have very little to add.

On 2024-01-16 07:29, Helmut Grohne wrote:
> On Tue, Jan 16, 2024 at 12:21:46AM +0100, Francesco Poli wrote:
>> Well, performance was not an issue here.
>> The copy of
>> 
>>   $ ls -lFs --si sid_amd64.img
>>   690M -rw-r-xrwx 1 $USER $USER 2.3G Jan 15 23:26 sid_amd64.img*
>> 
>> was really quick.
> 
> Probably. Copying huge chunks of data probably just feels wrong and
> technically isn't. We could also generate the file in $TMPDIR in 
> general
> (hoping it is backed by tmpfs and that the system performing the image
> build has at least 4GB ram) and then copy it to the final destination 
> in
> general. That's usability beating performance.

the patch I posted in an earlier mail to this bug also "solves" the 
problem of missing permission via copying things. Instead of letting 
/sbin/mkfs.ext4 (which runs inside the unshared user namespace) work on 
a file outside of it, the patch runs genext2fs on the outside (thus it 
will happily write to files inside your $HOME) and receives a tarball of 
the rootfs via a pipe from inside the unshared namespace. Then another 
copy is done to shift the image that genext2fs does by the correct 
offset to make room for the first partition.

>>   d.  configure sbuild-qemu and learn to use it to build Debian 
>> packages
> 
> Does this provide significant advantages over sbuild in unshare mode?
> 
>>   e.  report to you and/or Christian Kastner (and suggest possible
>>       improvements for mmdebstrap-autopkgtest-build-qemu and/or
>>       sbuild-qemu)
> 
> I suspect mmdebstrap-autopkgtest-build-qemu very much is unfinished. We
> got it into a barely-works.

Yes, but I think there are some immediate things that can be done to 
make the experience a bit better already. For starters, the error 
message that Francesco got initially is really bad and this thing should 
be caught much earlier and give the user an adequate explanation about 
what went wrong and what workaround exists.

I want to do another mmdebstrap release soon and will integrate such a 
check in it. I'll send a patch soon if Francesco wants to try it out 
first.

> I think making this all more useful eventually requires finding a
> solution for crossing uid boundaries in a sane way. At this time, I see
> two fairly generic ways to do this:
> 
> a) Have a very simple proxy fuse driver in the initial namespace and
>    mount it inside the target namespace. That way an individual
>    directory from a different user can be exposed to the namespace
>    without granting full access. (Good containment, non-trivial setup,
>    slight performance degradation.)
> 
> b) Map the calling user in the user namespace. We don't have to map the
>    user to root. We can map it to some high uid > 65535. Any process
>    that has CAP_DAC_OVERRIDE (and that capability is initially 
> available
>    and typically available to any process running as uid 0 inside) can
>    also access files by your user. (Bad containment, easy setup, no
>    performance impact.)

I also think that getting this merged will improve the situation:

https://github.com/tytso/e2fsprogs/pull/118

With that feature, /sbin/mkfs.ext4 can be run on the outside with the 
desired offset for the partition location on the disk image and receive 
a tarball from mmdebstrap via a pipe. Yes, this again means to copy some 
stuff but while the disk itself is 25G, the actual rootfs is not and 
even on my ARM Cortex A53 this is not a noticeable performance impact.

Theodore Ts'o indicated that this pull request will be "shortly 
integrated" here: 
https://lists.debian.org/debian-devel/2023/11/msg00090.html So maybe 
this is another option.

Thanks!

cheers, josch

Back to linux.debian.bugs.dist | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? "Francesco Poli (wintermute)" <invernomuto@paranoici.org> - 2024-01-11 00:00 +0100
  Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-01-11 00:50 +0100
    Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Francesco Poli <invernomuto@paranoici.org> - 2024-01-11 09:20 +0100
      Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-01-11 21:10 +0100
        Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Francesco Poli <invernomuto@paranoici.org> - 2024-01-12 20:10 +0100
          Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-01-12 23:50 +0100
            Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Francesco Poli <invernomuto@paranoici.org> - 2024-01-14 18:40 +0100
              Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-01-15 17:50 +0100
          Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Helmut Grohne <helmut@subdivi.de> - 2024-01-13 08:30 +0100
            Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-01-13 20:30 +0100
              Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-01-13 22:20 +0100
                Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Francesco Poli <invernomuto@paranoici.org> - 2024-01-14 18:40 +0100
                Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Helmut Grohne <helmut@subdivi.de> - 2024-01-15 07:40 +0100
                Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Francesco Poli <invernomuto@paranoici.org> - 2024-01-16 00:30 +0100
                Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Helmut Grohne <helmut@subdivi.de> - 2024-01-16 07:40 +0100
                Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-01-16 11:10 +0100
                Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-01-23 09:10 +0100
                Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-01-25 13:40 +0100
                Bug#1060416: mmdebstrap-autopkgtest-build-qemu fails on mkfs.ext4; does it require a 25 GiB TMPDIR? Francesco Poli <invernomuto@paranoici.org> - 2024-01-27 19:00 +0100

csiph-web