Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #66670 > unrolled thread

Bug#955469: initramfs-tools-core: Enable zstandard support

Back to article view | Back to linux.debian.kernel

Contents

  Bug#955469: initramfs-tools-core: Enable zstandard support Norbert Lange <nolange79@gmail.com> - 2020-04-01 09:10 +0200
    Bug#955469: initramfs-tools-core: Enable zstandard support Sedat Dilek <sedat.dilek@gmail.com> - 2020-04-01 12:50 +0200
      Bug#955469: initramfs-tools-core: Enable zstandard support Sedat Dilek <sedat.dilek@gmail.com> - 2020-07-31 17:00 +0200
        Bug#955469: initramfs-tools-core: Enable zstandard support Norbert Lange <nolange79@gmail.com> - 2020-08-01 00:40 +0200
          Bug#955469: initramfs-tools-core: Enable zstandard support Sedat Dilek <sedat.dilek@gmail.com> - 2020-08-01 14:30 +0200
            Bug#955469: initramfs-tools-core: Enable zstandard support Sedat Dilek <sedat.dilek@gmail.com> - 2020-08-04 09:40 +0200
    Bug#955469: initramfs-tools-core: Enable zstandard support Ben Hutchings <ben@decadent.org.uk> - 2020-04-28 05:50 +0200
      Bug#955469: initramfs-tools-core: Enable zstandard support Ben Hutchings <ben@decadent.org.uk> - 2020-04-29 19:00 +0200
        Bug#955469: initramfs-tools-core: Enable zstandard support Norbert Lange <nolange79@gmail.com> - 2020-04-29 21:40 +0200
          Bug#955469: initramfs-tools-core: Enable zstandard support Ben Hutchings <ben@decadent.org.uk> - 2020-05-16 02:00 +0200
    Bug#955469: marked as done (initramfs-tools-core: Enable  zstandard support) "Debian Bug Tracking System" <owner@bugs.debian.org> - 2020-09-11 22:40 +0200

#66670 — Bug#955469: initramfs-tools-core: Enable zstandard support

[Multipart message — attachments visible in raw view] — view raw

Package: initramfs-tools-core
Version: 0.136
Severity: wishlist
Tags: patch

Hello,

there are Kernelpatches for zstandard initramfs support
available for several years, and will hopefully accepted
upstream soon.

Please enable support for this compression.

The patch should be simple enough, I chosen to try zstd
as first decompressor because it supports a whole
range of formats on debian.

-- System Information:
Debian Release: bullseye/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.4.0-4-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages initramfs-tools-core depends on:
ii  coreutils    8.30-3+b1
ii  cpio         2.13+dfsg-2
ii  e2fsprogs    1.45.6-1
ii  klibc-utils  2.0.7-1
ii  kmod         27-2
ii  logsave      1.45.6-1
ii  udev         244.3-1

Versions of packages initramfs-tools-core recommends:
ii  busybox  1:1.30.1-4
ii  pigz     2.4-1+b1

Versions of packages initramfs-tools-core suggests:
ii  bash-completion  1:2.10-1

-- Configuration Files:
/etc/initramfs-tools/initramfs.conf changed [not included]

-- no debconf information

[toc] | [next] | [standalone]

#66671

Hi,

thanks Norbert for taking care.

I had a similar approach for Debian's userland tools when testing
version 3 of "Add support for ZSTD-compressed kernel and initramfs"
against Linux v5.6.

Norbert's patch seems to include what I have in [1] (includes my
README_zstd-v3.txt).

I am running both ZSTD support on the kernel and userland side using a
ZSTD compressed initramfs image.

As a side-note: zstd-v4 is out for testing see [2] and [3].

Thanks.

Regards,
- Sedat -

[1] LINK: https://lore.kernel.org/lkml/CA+icZUXCn2an9aNDrm+-eneSAOyGibz0W1xYhwkA5k3B3U-5vQ@mail.gmail.com/
[2] Message ID: <20200401053913.216783-1-nickrterrell@gmail.com>
[3] https://github.com/terrelln/linux/commits/zstd-v4

[toc] | [prev] | [next] | [standalone]

#67651

Just FYI:

Version 10 was now accepted in <tip.git#x86/boot>.

Let's hope this will get into upcoming Linux v5.9.

- Sedat -

[1] https://github.com/terrelln/linux/commits/zstd-v10
[2] https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/boot

[toc] | [prev] | [next] | [standalone]

#67652

Am Fr., 31. Juli 2020 um 16:48 Uhr schrieb Sedat Dilek <sedat.dilek@gmail.com>:
>
> Just FYI:
>
> Version 10 was now accepted in <tip.git#x86/boot>.
>
> Let's hope this will get into upcoming Linux v5.9.
>
> - Sedat -
>
> [1] https://github.com/terrelln/linux/commits/zstd-v10
> [2] https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/boot

Thanks,

I updated the patch, and made a merge request there:
https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests

Norbert

[toc] | [prev] | [next] | [standalone]

#67653

On Sat, Aug 1, 2020 at 12:27 AM Norbert Lange <nolange79@gmail.com> wrote:
>
> Am Fr., 31. Juli 2020 um 16:48 Uhr schrieb Sedat Dilek <sedat.dilek@gmail.com>:
> >
> > Just FYI:
> >
> > Version 10 was now accepted in <tip.git#x86/boot>.
> >
> > Let's hope this will get into upcoming Linux v5.9.
> >
> > - Sedat -
> >
> > [1] https://github.com/terrelln/linux/commits/zstd-v10
> > [2] https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/boot
>
> Thanks,
>
> I updated the patch, and made a merge request there:
> https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests
>

Hi Norbert,

Thank you, too.

You mean [1]?

[ mkinitramfs ]

+zstd) compress="zstd -q -19 -T0" ;;

From my kernel build-log:

{ cat arch/x86/boot/compressed/vmlinux.bin
arch/x86/boot/compressed/vmlinux.relocs | zstd -22 --ultra; printf
\014\015\315\001; } > arch/x86/boot/compressed/vmlinux.bin.zst

Thus I have analog here:

root# diff -uprN /usr/sbin/mkinitramfs.orig /usr/sbin/mkinitramfs
--- /usr/sbin/mkinitramfs.orig  2020-04-28 05:56:17.000000000 +0200
+++ /usr/sbin/mkinitramfs       2020-07-09 10:35:35.119280519 +0200
@@ -189,6 +189,7 @@ xz) compress="xz --check=crc32"
        # If we're not doing a reproducible build, enable multithreading
        test -z "${SOURCE_DATE_EPOCH}" && compress="$compress --threads=0"
        ;;
+zstd)  compress="zstd -22 --ultra -v" ;;
 bzip2|lzma|lzop)
        # no parameters needed
        ;;

Regards,
- Sedat -


[1] https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/33/diffs?commit_id=e94f410c71e90598b4fabca3970c7f282b5bd0a0

[toc] | [prev] | [next] | [standalone]

#67677

Now in Linus Git tree.

[1] says:

Merge tag 'x86-boot-2020-08-03' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 boot updates from Ingo Molnar:
"The main change in this cycle was to add support for ZSTD-compressed
kernel and initrd images. ZSTD has a very fast decompressor, yet it
compresses better than gzip"

- Sedat -

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c0dfadfed87489fa6126ece161a14c2d15dbdc79

[toc] | [prev] | [next] | [standalone]

#66913

[Multipart message — attachments visible in raw view] — view raw

On Wed, 01 Apr 2020 09:05:22 +0200 Norbert Lange <nolange79@gmail.com> wrote:
> Package: initramfs-tools-core
> Version: 0.136
> Severity: wishlist
> Tags: patch
> 
> Hello,
> 
> there are Kernelpatches for zstandard initramfs support
> available for several years, and will hopefully accepted
> upstream soon.

If the kernel doesn't yet support it, I'm not sure what the point of
supporting it in initramfs-tools is.

> Please enable support for this compression.
> 
> The patch should be simple enough, I chosen to try zstd
> as first decompressor because it supports a whole
> range of formats on debian.

> diff -burN initramfs-tools.org/conf/initramfs.conf initramfs-tools/conf/initramfs.conf
> --- initramfs-tools.org/conf/initramfs.conf	2019-02-06 00:55:47.000000000 +0100
> +++ initramfs-tools/conf/initramfs.conf	2020-03-31 18:38:48.558687968 +0200
> @@ -38,7 +38,7 @@
>  KEYMAP=n
> 
>  #
> -# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz ]
> +# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | zstd | xz ]

Aside from gzip (the default), this list is sorted alphabetically.  So
zstd belongs at the end.

>  #
> 
>  COMPRESS=gzip
> diff -burN initramfs-tools.org/mkinitramfs initramfs-tools/mkinitramfs
> --- initramfs-tools.org/mkinitramfs	2020-01-18 19:36:00.000000000 +0100
> +++ initramfs-tools/mkinitramfs	2020-03-31 18:31:31.046107765 +0200
> @@ -185,6 +185,7 @@
>  	fi
>  	;;
>  lz4)	compress="lz4 -9 -l" ;;
> +zstd)	compress="zstd -19" ;;
>  xz)	compress="xz --check=crc32"
>  	# If we're not doing a reproducible build, enable multithreading
>  	test -z "${SOURCE_DATE_EPOCH}" && compress="$compress --threads=0"
> diff -burN initramfs-tools.org/unmkinitramfs initramfs-tools/unmkinitramfs
> --- initramfs-tools.org/unmkinitramfs	2019-07-31 16:25:58.000000000 +0200
> +++ initramfs-tools/unmkinitramfs	2020-03-31 18:36:36.821308275 +0200
> @@ -29,7 +29,9 @@
>  	dir="$2"
>  	shift 2
> 
> -	if gzip -t "$archive" >/dev/null 2>&1 ; then
> +	if zstd -q -c -t "$archive" >/dev/null 2>&1 ; then
> +		zstd -q -c -d "$archive"
> +	elif gzip -t "$archive" >/dev/null 2>&1 ; then
>  		gzip -c -d "$archive"
>  	elif xzcat -t "$archive" >/dev/null 2>&1 ; then
>  		xzcat "$archive"

gzip should be checked first as it's most likely to be the actual
compression used.  Aside from that, I don't think the order matters
here.

Ben.

-- 
Ben Hutchings
Larkinson's Law: All laws are basically false.

[toc] | [prev] | [next] | [standalone]

#66944

[Multipart message — attachments visible in raw view] — view raw

On Tue, 2020-04-28 at 04:43 +0100, Ben Hutchings wrote:
> On Wed, 01 Apr 2020 09:05:22 +0200 Norbert Lange <nolange79@gmail.com> wrote:
> > Package: initramfs-tools-core
> > Version: 0.136
> > Severity: wishlist
> > Tags: patch
> > 
> > Hello,
> > 
> > there are Kernelpatches for zstandard initramfs support
> > available for several years, and will hopefully accepted
> > upstream soon.
> 
> If the kernel doesn't yet support it, I'm not sure what the point of
> supporting it in initramfs-tools is.
[...]

I see that the kernel support for this has been submitted recently,
though it's not clear which maintainer is being asked to apply the
patches.

Ben.

-- 
Ben Hutchings
Horngren's Observation:
              Among economists, the real world is often a special case.

[toc] | [prev] | [next] | [standalone]

#66952

[Multipart message — attachments visible in raw view] — view raw

Am Mittwoch, 29. April 2020 schrieb Ben Hutchings <ben@decadent.org.uk>:

> On Tue, 2020-04-28 at 04:43 +0100, Ben Hutchings wrote:
> > On Wed, 01 Apr 2020 09:05:22 +0200 Norbert Lange <nolange79@gmail.com>
> wrote:
> > > Package: initramfs-tools-core
> > > Version: 0.136
> > > Severity: wishlist
> > > Tags: patch
> > >
> > > Hello,
> > >
> > > there are Kernelpatches for zstandard initramfs support
> > > available for several years, and will hopefully accepted
> > > upstream soon.
> >
> > If the kernel doesn't yet support it, I'm not sure what the point of
> > supporting it in initramfs-tools is.
> [...]
>
> I see that the kernel support for this has been submitted recently,
> though it's not clear which maintainer is being asked to apply the
> patches.


It's been submitted multiple times over the years, I don't know the lkml
workings, and apparently the author of these changes is missing some detail
too.

Could you kindky point to what's missing? Sending the pull request directly
(not cc) to a maintainer?

I use initramfs-tools (unmkinitramfs in particular) for zstd compressed
initrds since years. The point would be that Debian is a nice distro for
any kind of development, including homebrewn kernels for pet project.
And that zstd is hopefully getting enabled for initrd/kernel in the near
future. It's not like this is an entirely new format.

Norbert

[toc] | [prev] | [next] | [standalone]

#67102

[Multipart message — attachments visible in raw view] — view raw

On Wed, 2020-04-29 at 21:36 +0200, Norbert Lange wrote:
> Am Mittwoch, 29. April 2020 schrieb Ben Hutchings <ben@decadent.org.uk>:
> 
> > On Tue, 2020-04-28 at 04:43 +0100, Ben Hutchings wrote:
> > > On Wed, 01 Apr 2020 09:05:22 +0200 Norbert Lange <nolange79@gmail.com>
> > wrote:
> > > > Package: initramfs-tools-core
> > > > Version: 0.136
> > > > Severity: wishlist
> > > > Tags: patch
> > > > 
> > > > Hello,
> > > > 
> > > > there are Kernelpatches for zstandard initramfs support
> > > > available for several years, and will hopefully accepted
> > > > upstream soon.
> > > 
> > > If the kernel doesn't yet support it, I'm not sure what the point of
> > > supporting it in initramfs-tools is.
> > [...]
> > 
> > I see that the kernel support for this has been submitted recently,
> > though it's not clear which maintainer is being asked to apply the
> > patches.
> 
> It's been submitted multiple times over the years, I don't know the lkml
> workings, and apparently the author of these changes is missing some detail
> too.
> 
> Could you kindky point to what's missing? Sending the pull request directly
> (not cc) to a maintainer?
[...]

There doesn't seem to be a nominated maintainer for lib/decompress*.c,
but Andrew Morton <akpm@linux-foundation.org> has signed-off most of
the changes there in the past few years.

So I would suggest bringing the kernel patch series to his attention. 
(He only works with patches, not pull requests.)

Ben.

-- 
Ben Hutchings
It's easier to fight for one's principles than to live up to them.

[toc] | [prev] | [next] | [standalone]

#68104 — Bug#955469: marked as done (initramfs-tools-core: Enable zstandard support)

[Multipart message — attachments visible in raw view] — view raw

Your message dated Fri, 11 Sep 2020 20:36:04 +0000
with message-id <E1kGpm8-000Io7-13@fasolo.debian.org>
and subject line Bug#955469: fixed in initramfs-tools 0.138
has caused the Debian Bug report #955469,
regarding initramfs-tools-core: Enable zstandard support
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
955469: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955469
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web