Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > muc.lists.netbsd.tech.userlevel > #11518 > unrolled thread

Supporting compression-less release builds

Started byJulio Merino <jmmv@NetBSD.org>
First post2025-01-12 14:49 +0000
Last post2025-01-18 16:40 +0000
Articles 11 — 8 participants

Back to article view | Back to muc.lists.netbsd.tech.userlevel


Contents

  Supporting compression-less release builds Julio Merino <jmmv@NetBSD.org> - 2025-01-12 14:49 +0000
    Re: Supporting compression-less release builds Alistair Crooks <alistaircrooks@gmail.com> - 2025-01-12 09:59 -0800
    Re: Supporting compression-less release builds Simon Burge <simonb@NetBSD.org> - 2025-01-13 12:34 +1100
      Re: Supporting compression-less release builds Martin Husemann <martin@duskware.de> - 2025-01-13 11:10 +0100
        Re: Supporting compression-less release builds Greg Troxel <gdt@lexort.com> - 2025-01-13 08:27 -0500
        Re: Supporting compression-less release builds "Greg A. Woods" <woods@planix.ca> - 2025-01-13 14:01 -0800
      Re: Supporting compression-less release builds Julio Merino <jmmv@NetBSD.org> - 2025-01-18 16:33 +0000
        Re: Supporting compression-less release builds Ignatios Souvatzis <is@netbsd.org> - 2025-01-18 20:47 +0100
          Re: Supporting compression-less release builds Julio Merino <jmmv@NetBSD.org> - 2025-01-18 22:38 +0000
        Re: Supporting compression-less release builds Ignatios Souvatzis <ignatios@cs.uni-bonn.de> - 2025-01-18 18:12 +0100
    Re: Supporting compression-less release builds Julio Merino <jmmv@NetBSD.org> - 2025-01-18 16:40 +0000

#11518 — Supporting compression-less release builds

FromJulio Merino <jmmv@NetBSD.org>
Date2025-01-12 14:49 +0000
SubjectSupporting compression-less release builds
Message-ID<Z4PWhfkYlwJByv8K@NetBSD.org>
Hello,

I'm currently working on a NetBSD-based embedded disk image and, while
working on this project, the compression of various artifacts during the
build gets in the way.  Compressing the sets and kernels is very visible
the choke point of the build even on a fast machine, and is specially so
when xz is in use.  But the thing is: compression is completely useless
(for me) during the development cycle.

I started with a simple idea to preserve current behavior yet allow me
to disable sets compression:

1. Add a new COMPRESS_SETS tunable that takes no/gzip/xz as values.
   The "no" value is what allows a user to disable compression.
2. Deduce the default value of COMPRESS_SETS with the same logic we have
   today based on USE_XZ_SETS/USE_PIGZGZIP.  COMPRESS_SETS=no would never
   be set by default today.
3. Retire USE_XZ_SETS.

This is simple enough to do and I have the change ready to go, but
working on it opened up more questions: why focus on the release sets
only?  There are other artifacts of the build that are compressed such
as disk images, and those are also huge choke points of the build process.
Plus we have inconsistencies regarding compression flags throughout the
tree to indicate which compression levels to use.

So I'm thinking to evolve this change towards something more comprehensive,
and would like to think through it from the beginning to minimize code
change churn.

The general idea would be to replace COMPRESS_SETS (not yet even committed)
with a more generic COMPRESS_RELEASE (or similar name) and use that value
wherever the release build wants to produce compressed artifacts.  Then,
unify the many -0, -9, -n... flags that appear throughout the tree for gzip
and xz in one single place so that, when compression is indeed enabled, we
apply it consistently.

The results of this work should be mostly invisible.  Only those who
care to set COMPRESS_RELEASE=no explicitly would see a change.  There
might be minor size changes in the default release builds because of the
unification of flags (e.g. applying -9 in places where it isn't applied
today).

Does anyone see a problem with this plan?

I'd like to have someone help me review the intermediate changes
because, while simple, I'm sure I may be missing something due to my
long break from this codebase.  I expect 3-5 separate commits to get to
the end state.  Anyone?

Thanks

-- 
Julio Merino

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [next] | [standalone]


#11519

FromAlistair Crooks <alistaircrooks@gmail.com>
Date2025-01-12 09:59 -0800
Message-ID<CAN5gJXojuJM9VOx3BuVdfzT_Oajr4kXds6p2ZjPTzk=r8T-0ZQ@mail.gmail.com>
In reply to#11518

[Multipart message — attachments visible in raw view] — view raw

On Sun, 12 Jan 2025 at 06:49, Julio Merino <jmmv@netbsd.org> wrote:

> Hello,
>
> I'm currently working on a NetBSD-based embedded disk image and, while
> working on this project, the compression of various artifacts during the
> build gets in the way.  Compressing the sets and kernels is very visible
> the choke point of the build even on a fast machine, and is specially so
> when xz is in use.  But the thing is: compression is completely useless
> (for me) during the development cycle.
>
> I started with a simple idea to preserve current behavior yet allow me
> to disable sets compression:
>
> 1. Add a new COMPRESS_SETS tunable that takes no/gzip/xz as values.
>    The "no" value is what allows a user to disable compression.
> 2. Deduce the default value of COMPRESS_SETS with the same logic we have
>    today based on USE_XZ_SETS/USE_PIGZGZIP.  COMPRESS_SETS=no would never
>    be set by default today.
> 3. Retire USE_XZ_SETS.
>
> This is simple enough to do and I have the change ready to go, but
> working on it opened up more questions: why focus on the release sets
> only?  There are other artifacts of the build that are compressed such
> as disk images, and those are also huge choke points of the build process.
> Plus we have inconsistencies regarding compression flags throughout the
> tree to indicate which compression levels to use.
>
> So I'm thinking to evolve this change towards something more comprehensive,
> and would like to think through it from the beginning to minimize code
> change churn.
>
> The general idea would be to replace COMPRESS_SETS (not yet even committed)
> with a more generic COMPRESS_RELEASE (or similar name) and use that value
> wherever the release build wants to produce compressed artifacts.  Then,
> unify the many -0, -9, -n... flags that appear throughout the tree for gzip
> and xz in one single place so that, when compression is indeed enabled, we
> apply it consistently.
>
> The results of this work should be mostly invisible.  Only those who
> care to set COMPRESS_RELEASE=no explicitly would see a change.  There
> might be minor size changes in the default release builds because of the
> unification of flags (e.g. applying -9 in places where it isn't applied
> today).
>
> Does anyone see a problem with this plan?
>
> I'd like to have someone help me review the intermediate changes
> because, while simple, I'm sure I may be missing something due to my
> long break from this codebase.  I expect 3-5 separate commits to get to
> the end state.  Anyone?
>
> Thanks
>
> --
> Julio Merino
>

Is there any reason that you can't just use cat(1) as a compressor?

[toc] | [prev] | [next] | [standalone]


#11520

FromSimon Burge <simonb@NetBSD.org>
Date2025-01-13 12:34 +1100
Message-ID<20250113013414.EEE52758F@thoreau.thistledown.com.au>
In reply to#11518
Hi Julio,

Julio Merino wrote:

> I'm currently working on a NetBSD-based embedded disk image and, while
> working on this project, the compression of various artifacts during the
> build gets in the way.  Compressing the sets and kernels is very visible
> the choke point of the build even on a fast machine, and is specially so
> when xz is in use.  But the thing is: compression is completely useless
> (for me) during the development cycle.

If you don't want the compressed sets in your development cycle can you
just use "./build.sh ... distribution" to skip generating the sets?

> I started with a simple idea to preserve current behavior yet allow me
> to disable sets compression:
>
> 1. Add a new COMPRESS_SETS tunable that takes no/gzip/xz as values.
>    The "no" value is what allows a user to disable compression.
> 2. Deduce the default value of COMPRESS_SETS with the same logic we have
>    today based on USE_XZ_SETS/USE_PIGZGZIP.  COMPRESS_SETS=no would never
>    be set by default today.
> 3. Retire USE_XZ_SETS.

A better control knob for sets wouldn't be unwelcome (IMO) :).

> The general idea would be to replace COMPRESS_SETS (not yet even committed)
> with a more generic COMPRESS_RELEASE (or similar name) and use that value
> wherever the release build wants to produce compressed artifacts.  Then,
> unify the many -0, -9, -n... flags that appear throughout the tree for gzip
> and xz in one single place so that, when compression is indeed enabled, we
> apply it consistently.

Note that we sometimes use an explicit compression method elsewhere
in the build (eg compressing kernels) that might differ from the
compression method for sets.

The way I read your proposal you seem to be suggesting "everything
in the build that needs to be compressed will be compressed with the
thing expressed by COMPRESS_RELEASE"; I'm pointing out that this might
break some things.  Now, whether or not the things that are compressed
differently are for a valid reason is a different question!  I can
think, for example, that an image that is designed to be dd'd to the
start of a memory stick might explicitly be compressed with gzip because
it might be easier to decompress a .gz file on a Windows box (I don't
know this to be the case, just using as a possible reason for preferring
a different compressor).

Cheers,
Simon.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]


#11521

FromMartin Husemann <martin@duskware.de>
Date2025-01-13 11:10 +0100
Message-ID<20250113101028.GA8314@mail.duskware.de>
In reply to#11520
On Mon, Jan 13, 2025 at 12:34:14PM +1100, Simon Burge wrote:
> If you don't want the compressed sets in your development cycle can you
> just use "./build.sh ... distribution" to skip generating the sets?

This likely will not do as the image creation needs the set files.

> A better control knob for sets wouldn't be unwelcome (IMO) :).

Yes, especially an override for the compression level, so a simple solution
would be USE_XZ_SETS=no GZIP_LEVEL=0 or something like that (where GZIP_LEVEL
does not exist yet).

Making this all slightly more generic sounds good too, but it is pretty
intrusive - the file name is used in the release docs as well as in sysinst,
and external tools like Anita depend on a hard code list of potential file
names for the sets.

> Note that we sometimes use an explicit compression method elsewhere
> in the build (eg compressing kernels) that might differ from the
> compression method for sets.

Indeed, and also some of the compression levels must be high (or used to
be, maybe the split of DVD images fixed this properly already) to make
the resulting images fit into certain size restraints.

For the original problem, I would do something else: hack the makefiles
localy to not update the compressed aritfacts at all when some make variable
is set. If that turns out to be usefull and not totally awful, it could
even be commited with some warnings in the documentation.

Martin

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]


#11522

FromGreg Troxel <gdt@lexort.com>
Date2025-01-13 08:27 -0500
Message-ID<rmiplkqg7yd.fsf@s1.lexort.com>
In reply to#11521
Martin Husemann <martin@duskware.de> writes:

>> Note that we sometimes use an explicit compression method elsewhere
>> in the build (eg compressing kernels) that might differ from the
>> compression method for sets.
>
> Indeed, and also some of the compression levels must be high (or used to
> be, maybe the split of DVD images fixed this properly already) to make
> the resulting images fit into certain size restraints.

There is "must be" and there is "if you turn compression down to save
time, then your iso images might be bigger than will fit".   I think the
latter is fine.

> For the original problem, I would do something else: hack the makefiles
> localy to not update the compressed aritfacts at all when some make variable
> is set. If that turns out to be usefull and not totally awful, it could
> even be commited with some warnings in the documentation.

I build releases, rsync them to other machines - often on GbE LAN, and
then update those machines (with scripts in etcmanage).  Yes, the
extensions are baked in, but it might be useful to do this uncompressed,
depending on if gzip beats GbE.

jmmv@'s proposal sounds entirely reasonable to me.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]


#11523

From"Greg A. Woods" <woods@planix.ca>
Date2025-01-13 14:01 -0800
Message-ID<m1tXSVF-00Mo5zC@more.local>
In reply to#11521

[Multipart message — attachments visible in raw view] — view raw

At Mon, 13 Jan 2025 11:10:28 +0100, Martin Husemann <martin@duskware.de> wrote:
Subject: Re: Supporting compression-less release builds
>
> Yes, especially an override for the compression level, so a simple solution
> would be USE_XZ_SETS=no GZIP_LEVEL=0 or something like that (where GZIP_LEVEL
> does not exist yet).

Note that the sets Makefile sets a default for the make variable XZ_OPT,
and then passes it as XZ_OPT in the environment.

If I remember right I tried XZ_OPT=-0 but it was still painfully slow.
Instead I use USE_XZ_SETS=no and USE_PIGZGZIP=yes.

It does also passes the make variable GZIP_N_FLAG as GZIP in the
environment.  One could add a "-0" to GZIP_N_FLAG, but note that the
default value varies slightly based on the setting of ${USE_PIGZGZIP}.

> For the original problem, I would do something else: hack the makefiles
> localy to not update the compressed aritfacts at all when some make variable
> is set. If that turns out to be usefull and not totally awful, it could
> even be commited with some warnings in the documentation.

Indeed -- this could prove useful for those of us who never care about
writing installation images to limited physical media any more.

--
					Greg A. Woods <gwoods@acm.org>

Kelowna, BC     +1 250 762-7675           RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>     Avoncote Farms <woods@avoncote.ca>

[toc] | [prev] | [next] | [standalone]


#11524

FromJulio Merino <jmmv@NetBSD.org>
Date2025-01-18 16:33 +0000
Message-ID<Z4vXz-KinaFTBfQ0@NetBSD.org>
In reply to#11520
On Mon, Jan 13, 2025 at 12:34:14PM +1100, Simon Burge wrote:
> Note that we sometimes use an explicit compression method elsewhere
> in the build (eg compressing kernels) that might differ from the
> compression method for sets.
> 
> The way I read your proposal you seem to be suggesting "everything
> in the build that needs to be compressed will be compressed with the
> thing expressed by COMPRESS_RELEASE"; I'm pointing out that this might
> break some things.  Now, whether or not the things that are compressed
> differently are for a valid reason is a different question!  I can
> think, for example, that an image that is designed to be dd'd to the
> start of a memory stick might explicitly be compressed with gzip because
> it might be easier to decompress a .gz file on a Windows box (I don't
> know this to be the case, just using as a possible reason for preferring
> a different compressor).

Yes, I was thinking of unifying everything into just one method, but
that would result in differences from the current situation indeed, hence
my hesitation.

In particular, there seem to be at least three things at play:

* Release sets. These already have knobs for configuration.
* Disk images. Compressed with gzip unconditionally.
* Kernels. Compressed with gzip unconditionally.

I do not know if there are good reasons for these differences, although
I sense there aren't, and the reason sets support xz is because that's
what made the most difference in final artifact sizes.

We can take these one at a time and support differences among them if
necessary.  COMPRESS_RELEASE might not be a good idea at this point.
But I'd still like to unify compression levels (to avoid the mess that
are the -0 .. -9 flags throughout).

-- 
Julio Merino

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]


#11526

FromIgnatios Souvatzis <is@netbsd.org>
Date2025-01-18 20:47 +0100
Message-ID<Z4wFU--9z1zqBgv0@bnhb484.de>
In reply to#11524
hi,

On Sat, Jan 18, 2025 at 04:33:19PM +0000, Julio Merino wrote:

> In particular, there seem to be at least three things at play:
> 
> * Release sets. These already have knobs for configuration.
> * Disk images. Compressed with gzip unconditionally.
> * Kernels. Compressed with gzip unconditionally.
> 
> I do not know if there are good reasons for these differences, although
> I sense there aren't, and the reason sets support xz is because that's
> what made the most difference in final artifact sizes.

(At least some) boot-loaders can boot gzip-compressed kernels
directly.  This is convenient for checking whether hardware Y would
survive a kernel upgrade.... I think those loaders do not know how
to handle xz and might not have the resources doing that.

So while there's an argument for xz for the sets, I argue for
the simpler compression method for kernels. 

Regards,
	-is

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]


#11527

FromJulio Merino <jmmv@NetBSD.org>
Date2025-01-18 22:38 +0000
Message-ID<Z4wtbHmlI4lOdcLa@NetBSD.org>
In reply to#11526
On Sat, Jan 18, 2025 at 08:47:31PM +0100, Ignatios Souvatzis wrote:
> hi,
> 
> On Sat, Jan 18, 2025 at 04:33:19PM +0000, Julio Merino wrote:
> 
> > In particular, there seem to be at least three things at play:
> > 
> > * Release sets. These already have knobs for configuration.
> > * Disk images. Compressed with gzip unconditionally.
> > * Kernels. Compressed with gzip unconditionally.
> > 
> > I do not know if there are good reasons for these differences, although
> > I sense there aren't, and the reason sets support xz is because that's
> > what made the most difference in final artifact sizes.
> 
> (At least some) boot-loaders can boot gzip-compressed kernels
> directly.  This is convenient for checking whether hardware Y would
> survive a kernel upgrade.... I think those loaders do not know how
> to handle xz and might not have the resources doing that.
> 
> So while there's an argument for xz for the sets, I argue for
> the simpler compression method for kernels. 

Ah, thanks for confirming. So kernels need to be able to remain gzipped.

-- 
Julio Merino

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]


#11528

FromIgnatios Souvatzis <ignatios@cs.uni-bonn.de>
Date2025-01-18 18:12 +0100
Message-ID<8C883F4B-4327-4C19-8BC4-A6E4299033F8@cs.uni-bonn.de>
In reply to#11524

[Multipart message — attachments visible in raw view] — view raw

Julio wrote:

> * Release sets. These already have knobs for > configuration.
> * Disk images. Compressed with gzip >unconditionally.
> * Kernels. Compressed with gzip unconditionally.
> 
> I do not know if there are good reasons for these differences

Bootloaders understand gzip, but don't (and I think, don't have the resources to) understand xz. 

      -is

[toc] | [prev] | [next] | [standalone]


#11525

FromJulio Merino <jmmv@NetBSD.org>
Date2025-01-18 16:40 +0000
Message-ID<Z4vZakJwaphOnn9I@NetBSD.org>
In reply to#11518

[Multipart message — attachments visible in raw view] — view raw

On Sun, Jan 12, 2025 at 02:49:41PM +0000, Julio Merino wrote:
>
> Does anyone see a problem with this plan?

OK, I read through the comments in the sibling messages.

Unifying everything into one compression method might not be the best
idea, at least for now, because there might be reasons for kernels to be
gzipped only and because changing this would be too intrusive (requiring
changes to the docs and the like in some parameterized manner).

So let's take changes one at a time, and possibly keep the tunables for
sets / images / kernels separate. That said, I still think doing a code
cleanup to unify where flags are specified (so that compression levels
are consistently defined) is worthwhile.

I'm attaching just the first patch that I had in mind, which implements
the COMPRESS_SETS change only.  It works, but I haven't done sufficient
validation yet.

What do you think?

-- 
Julio Merino

[toc] | [prev] | [standalone]


Back to top | Article view | muc.lists.netbsd.tech.userlevel


csiph-web