Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #68725 > unrolled thread

Bug#976054: initramfs-tools: [RFC] Compress initramfs file with zstd

Back to article view | Back to linux.debian.kernel

Contents

  Bug#976054: initramfs-tools: [RFC] Compress initramfs file with zstd Hideki Yamane <henrich@debian.org> - 2020-11-29 02:10 +0100
    Bug#976054: initramfs-tools: [RFC] Compress initramfs file with zstd Michael Prokop <mika@debian.org> - 2020-12-02 11:40 +0100
      Bug#976054: initramfs-tools: [RFC] Compress initramfs file with  zstd Hideki Yamane <henrich@iijmio-mail.jp> - 2021-01-31 11:50 +0100

#68725 — Bug#976054: initramfs-tools: [RFC] Compress initramfs file with zstd

[Multipart message — attachments visible in raw view] — view raw

Package: initramfs-tools
Version: 0.139
Severity: normal
Tags: patch

Dear Maintainer,

Now initramfs files are compressed with gzip
But we can use other compression, i.e. LZO, LZ4 and ZSTD.
Then which one is the best?

* compression ratio
   ZSTD > GZIP > LZO > LZ4

* decompression speed
   LZ4 > LZO = ZSTD > GZIP

I suggest to choose ZSTD, instead of GZIP since
 - better compression ratio than current GZIP
 - also better decompresion speed than current GZIP


------------------------------------------------------------------------------
With gzip
------------------------------------------------------------------------------

$ file /boot/initrd.img-5.9.0-*
/boot/initrd.img-5.9.0-3-amd64: gzip compressed data, from Unix, original size modulo 2^32 197680640
/boot/initrd.img-5.9.0-4-amd64: gzip compressed data, from Unix, original size modulo 2^32 197594112

$ du -h /boot/initrd.img-5.9.0-*
59M     /boot/initrd.img-5.9.0-3-amd64
59M     /boot/initrd.img-5.9.0-4-amd64

------------------------------------------------------------------------------
With zstd
------------------------------------------------------------------------------

$ file /boot/initrd.img-5.9.0-*
/boot/initrd.img-5.9.0-3-amd64: Zstandard compressed data (v0.8+), Dictionary ID: None
/boot/initrd.img-5.9.0-4-amd64: Zstandard compressed data (v0.8+), Dictionary ID: None

$ du -h /boot/initrd.img-5.9.0-*
40M     /boot/initrd.img-5.9.0-3-amd64
40M     /boot/initrd.img-5.9.0-4-amd64
------------------------------------------------------------------------------

Yes, 59MB initramfs file becomes 40MB (2/3)! plus bonus, better decompression speed.


However, there's a problem to do so - zstd package's Priority is not standard one 
- it's "optional". 

> Package: gzip
> Version: 1.10-2
> Priority: required

> Package: zstd
> Version: 1.4.5+dfsg-4
> Priority: optional

It means there is not zstd package in every environment. And, just raise its Priority
means bloat minimal system size that is a problem we want to avoid.

Now initramfs-tools has a hack to avoid zstd (and other compression tools)
absence in mkinitramfs command.

>　if ! command -v "${compress}" >/dev/null 2>&1; then
>         compress=gzip
>         echo "No ${compress} in ${PATH}, using gzip"
> fi

Set COMPRESS=zstd in /etc/initramfs/initramfs.conf and no zstd installed,
there's no problem.

> $ sudo update-initramfs -u
> update-initramfs: Generating /boot/initrd.img-5.9.0-4-amd64
> No zstd in /usr/bin:/sbin:/bin, using gzip

 (abobe message is patched with #971270)


So, just set default compression as zstd can brings better result for many users,
IMO (Not sure raising zstd package Priority (to standard) is required or not).

[toc] | [next] | [standalone]

#68741

[Multipart message — attachments visible in raw view] — view raw

* Hideki Yamane [Sun Nov 29, 2020 at 10:06:21AM +0900]:

> Now initramfs files are compressed with gzip
> But we can use other compression, i.e. LZO, LZ4 and ZSTD.
> Then which one is the best?

> * compression ratio
>    ZSTD > GZIP > LZO > LZ4

> * decompression speed
>    LZ4 > LZO = ZSTD > GZIP

> I suggest to choose ZSTD, instead of GZIP since
>  - better compression ratio than current GZIP
>  - also better decompresion speed than current GZIP

[...]

> Yes, 59MB initramfs file becomes 40MB (2/3)! plus bonus, better decompression speed.

Thanks for bringing this up.

Do we have any numbers for where xz resides in terms of
decompression speed? Is it enough to just decompress the
files in userspace for getting numbers, or is the kernel doing
something special™ to be kept in mind?

JFTR, some further numbers regarding file sizes:

| # file netboot.initrd*
| netboot.initrd.gzip: gzip compressed data, last modified: Wed Dec  2 10:13:21 2020, from Unix, original size 135559168
| netboot.initrd.lz4:  LZ4 compressed data (v0.1-v0.9)
| netboot.initrd.lzop: lzop compressed data - version 1.030, LZO1X-1, os: Unix
| netboot.initrd.xz:   XZ compressed data
| netboot.initrd.zstd: Zstandard compressed data (v0.8+), Dictionary ID: None
| # ls -lah netboot.initrd*
| -rw-r--r-- 1 root root 43M Dec  2 10:13 netboot.initrd.gzip
| -rw-r--r-- 1 root root 51M Dec  2 10:25 netboot.initrd.lz4
| -rw-r--r-- 1 root root 60M Dec  2 10:24 netboot.initrd.lzop
| -rw-r--r-- 1 root root 29M Dec  2 10:17 netboot.initrd.xz
| -rw-r--r-- 1 root root 32M Dec  2 10:16 netboot.initrd.zstd

regards
-mika-

[toc] | [prev] | [next] | [standalone]

#69374 — Bug#976054: initramfs-tools: [RFC] Compress initramfs file with zstd

Hi,

On Wed, 2 Dec 2020 11:28:06 +0100 Michael Prokop <mika@debian.org> wrote:
> Do we have any numbers for where xz resides in terms of
> decompression speed?

 As I've posted at salsa, its decompression is so slow.
 Its CPU is AMD Ryzen 5 PRO 3400GE (3.3GHz) but it takes 3 seconds.
 Older i686 CPUs would take more... but xz shrinks just 3 or 4MB.
 https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/37#note_211939


-- 
Regards,

 Hideki Yamane     henrich @ debian.org/iijmio-mail.jp

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web