Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > muc.lists.netbsd.tech.userlevel > #11518 > unrolled thread
| Started by | Julio Merino <jmmv@NetBSD.org> |
|---|---|
| First post | 2025-01-12 14:49 +0000 |
| Last post | 2025-01-18 16:40 +0000 |
| Articles | 11 — 8 participants |
Back to article view | Back to muc.lists.netbsd.tech.userlevel
Supporting compression-less release builds Julio Merino <jmmv@NetBSD.org> - 2025-01-12 14:49 +0000
Re: Supporting compression-less release builds Alistair Crooks <alistaircrooks@gmail.com> - 2025-01-12 09:59 -0800
Re: Supporting compression-less release builds Simon Burge <simonb@NetBSD.org> - 2025-01-13 12:34 +1100
Re: Supporting compression-less release builds Martin Husemann <martin@duskware.de> - 2025-01-13 11:10 +0100
Re: Supporting compression-less release builds Greg Troxel <gdt@lexort.com> - 2025-01-13 08:27 -0500
Re: Supporting compression-less release builds "Greg A. Woods" <woods@planix.ca> - 2025-01-13 14:01 -0800
Re: Supporting compression-less release builds Julio Merino <jmmv@NetBSD.org> - 2025-01-18 16:33 +0000
Re: Supporting compression-less release builds Ignatios Souvatzis <is@netbsd.org> - 2025-01-18 20:47 +0100
Re: Supporting compression-less release builds Julio Merino <jmmv@NetBSD.org> - 2025-01-18 22:38 +0000
Re: Supporting compression-less release builds Ignatios Souvatzis <ignatios@cs.uni-bonn.de> - 2025-01-18 18:12 +0100
Re: Supporting compression-less release builds Julio Merino <jmmv@NetBSD.org> - 2025-01-18 16:40 +0000
| From | Julio Merino <jmmv@NetBSD.org> |
|---|---|
| Date | 2025-01-12 14:49 +0000 |
| Subject | Supporting compression-less release builds |
| Message-ID | <Z4PWhfkYlwJByv8K@NetBSD.org> |
Hello, I'm currently working on a NetBSD-based embedded disk image and, while working on this project, the compression of various artifacts during the build gets in the way. Compressing the sets and kernels is very visible the choke point of the build even on a fast machine, and is specially so when xz is in use. But the thing is: compression is completely useless (for me) during the development cycle. I started with a simple idea to preserve current behavior yet allow me to disable sets compression: 1. Add a new COMPRESS_SETS tunable that takes no/gzip/xz as values. The "no" value is what allows a user to disable compression. 2. Deduce the default value of COMPRESS_SETS with the same logic we have today based on USE_XZ_SETS/USE_PIGZGZIP. COMPRESS_SETS=no would never be set by default today. 3. Retire USE_XZ_SETS. This is simple enough to do and I have the change ready to go, but working on it opened up more questions: why focus on the release sets only? There are other artifacts of the build that are compressed such as disk images, and those are also huge choke points of the build process. Plus we have inconsistencies regarding compression flags throughout the tree to indicate which compression levels to use. So I'm thinking to evolve this change towards something more comprehensive, and would like to think through it from the beginning to minimize code change churn. The general idea would be to replace COMPRESS_SETS (not yet even committed) with a more generic COMPRESS_RELEASE (or similar name) and use that value wherever the release build wants to produce compressed artifacts. Then, unify the many -0, -9, -n... flags that appear throughout the tree for gzip and xz in one single place so that, when compression is indeed enabled, we apply it consistently. The results of this work should be mostly invisible. Only those who care to set COMPRESS_RELEASE=no explicitly would see a change. There might be minor size changes in the default release builds because of the unification of flags (e.g. applying -9 in places where it isn't applied today). Does anyone see a problem with this plan? I'd like to have someone help me review the intermediate changes because, while simple, I'm sure I may be missing something due to my long break from this codebase. I expect 3-5 separate commits to get to the end state. Anyone? Thanks -- Julio Merino -- Posted automagically by a mail2news gateway at muc.de e.V. Please direct questions, flames, donations, etc. to news-admin@muc.de
[toc] | [next] | [standalone]
| From | Alistair Crooks <alistaircrooks@gmail.com> |
|---|---|
| Date | 2025-01-12 09:59 -0800 |
| Message-ID | <CAN5gJXojuJM9VOx3BuVdfzT_Oajr4kXds6p2ZjPTzk=r8T-0ZQ@mail.gmail.com> |
| In reply to | #11518 |
[Multipart message — attachments visible in raw view] — view raw
On Sun, 12 Jan 2025 at 06:49, Julio Merino <jmmv@netbsd.org> wrote: > Hello, > > I'm currently working on a NetBSD-based embedded disk image and, while > working on this project, the compression of various artifacts during the > build gets in the way. Compressing the sets and kernels is very visible > the choke point of the build even on a fast machine, and is specially so > when xz is in use. But the thing is: compression is completely useless > (for me) during the development cycle. > > I started with a simple idea to preserve current behavior yet allow me > to disable sets compression: > > 1. Add a new COMPRESS_SETS tunable that takes no/gzip/xz as values. > The "no" value is what allows a user to disable compression. > 2. Deduce the default value of COMPRESS_SETS with the same logic we have > today based on USE_XZ_SETS/USE_PIGZGZIP. COMPRESS_SETS=no would never > be set by default today. > 3. Retire USE_XZ_SETS. > > This is simple enough to do and I have the change ready to go, but > working on it opened up more questions: why focus on the release sets > only? There are other artifacts of the build that are compressed such > as disk images, and those are also huge choke points of the build process. > Plus we have inconsistencies regarding compression flags throughout the > tree to indicate which compression levels to use. > > So I'm thinking to evolve this change towards something more comprehensive, > and would like to think through it from the beginning to minimize code > change churn. > > The general idea would be to replace COMPRESS_SETS (not yet even committed) > with a more generic COMPRESS_RELEASE (or similar name) and use that value > wherever the release build wants to produce compressed artifacts. Then, > unify the many -0, -9, -n... flags that appear throughout the tree for gzip > and xz in one single place so that, when compression is indeed enabled, we > apply it consistently. > > The results of this work should be mostly invisible. Only those who > care to set COMPRESS_RELEASE=no explicitly would see a change. There > might be minor size changes in the default release builds because of the > unification of flags (e.g. applying -9 in places where it isn't applied > today). > > Does anyone see a problem with this plan? > > I'd like to have someone help me review the intermediate changes > because, while simple, I'm sure I may be missing something due to my > long break from this codebase. I expect 3-5 separate commits to get to > the end state. Anyone? > > Thanks > > -- > Julio Merino > Is there any reason that you can't just use cat(1) as a compressor?
[toc] | [prev] | [next] | [standalone]
| From | Simon Burge <simonb@NetBSD.org> |
|---|---|
| Date | 2025-01-13 12:34 +1100 |
| Message-ID | <20250113013414.EEE52758F@thoreau.thistledown.com.au> |
| In reply to | #11518 |
Hi Julio, Julio Merino wrote: > I'm currently working on a NetBSD-based embedded disk image and, while > working on this project, the compression of various artifacts during the > build gets in the way. Compressing the sets and kernels is very visible > the choke point of the build even on a fast machine, and is specially so > when xz is in use. But the thing is: compression is completely useless > (for me) during the development cycle. If you don't want the compressed sets in your development cycle can you just use "./build.sh ... distribution" to skip generating the sets? > I started with a simple idea to preserve current behavior yet allow me > to disable sets compression: > > 1. Add a new COMPRESS_SETS tunable that takes no/gzip/xz as values. > The "no" value is what allows a user to disable compression. > 2. Deduce the default value of COMPRESS_SETS with the same logic we have > today based on USE_XZ_SETS/USE_PIGZGZIP. COMPRESS_SETS=no would never > be set by default today. > 3. Retire USE_XZ_SETS. A better control knob for sets wouldn't be unwelcome (IMO) :). > The general idea would be to replace COMPRESS_SETS (not yet even committed) > with a more generic COMPRESS_RELEASE (or similar name) and use that value > wherever the release build wants to produce compressed artifacts. Then, > unify the many -0, -9, -n... flags that appear throughout the tree for gzip > and xz in one single place so that, when compression is indeed enabled, we > apply it consistently. Note that we sometimes use an explicit compression method elsewhere in the build (eg compressing kernels) that might differ from the compression method for sets. The way I read your proposal you seem to be suggesting "everything in the build that needs to be compressed will be compressed with the thing expressed by COMPRESS_RELEASE"; I'm pointing out that this might break some things. Now, whether or not the things that are compressed differently are for a valid reason is a different question! I can think, for example, that an image that is designed to be dd'd to the start of a memory stick might explicitly be compressed with gzip because it might be easier to decompress a .gz file on a Windows box (I don't know this to be the case, just using as a possible reason for preferring a different compressor). Cheers, Simon. -- Posted automagically by a mail2news gateway at muc.de e.V. Please direct questions, flames, donations, etc. to news-admin@muc.de
[toc] | [prev] | [next] | [standalone]
| From | Martin Husemann <martin@duskware.de> |
|---|---|
| Date | 2025-01-13 11:10 +0100 |
| Message-ID | <20250113101028.GA8314@mail.duskware.de> |
| In reply to | #11520 |
On Mon, Jan 13, 2025 at 12:34:14PM +1100, Simon Burge wrote: > If you don't want the compressed sets in your development cycle can you > just use "./build.sh ... distribution" to skip generating the sets? This likely will not do as the image creation needs the set files. > A better control knob for sets wouldn't be unwelcome (IMO) :). Yes, especially an override for the compression level, so a simple solution would be USE_XZ_SETS=no GZIP_LEVEL=0 or something like that (where GZIP_LEVEL does not exist yet). Making this all slightly more generic sounds good too, but it is pretty intrusive - the file name is used in the release docs as well as in sysinst, and external tools like Anita depend on a hard code list of potential file names for the sets. > Note that we sometimes use an explicit compression method elsewhere > in the build (eg compressing kernels) that might differ from the > compression method for sets. Indeed, and also some of the compression levels must be high (or used to be, maybe the split of DVD images fixed this properly already) to make the resulting images fit into certain size restraints. For the original problem, I would do something else: hack the makefiles localy to not update the compressed aritfacts at all when some make variable is set. If that turns out to be usefull and not totally awful, it could even be commited with some warnings in the documentation. Martin -- Posted automagically by a mail2news gateway at muc.de e.V. Please direct questions, flames, donations, etc. to news-admin@muc.de
[toc] | [prev] | [next] | [standalone]
| From | Greg Troxel <gdt@lexort.com> |
|---|---|
| Date | 2025-01-13 08:27 -0500 |
| Message-ID | <rmiplkqg7yd.fsf@s1.lexort.com> |
| In reply to | #11521 |
Martin Husemann <martin@duskware.de> writes: >> Note that we sometimes use an explicit compression method elsewhere >> in the build (eg compressing kernels) that might differ from the >> compression method for sets. > > Indeed, and also some of the compression levels must be high (or used to > be, maybe the split of DVD images fixed this properly already) to make > the resulting images fit into certain size restraints. There is "must be" and there is "if you turn compression down to save time, then your iso images might be bigger than will fit". I think the latter is fine. > For the original problem, I would do something else: hack the makefiles > localy to not update the compressed aritfacts at all when some make variable > is set. If that turns out to be usefull and not totally awful, it could > even be commited with some warnings in the documentation. I build releases, rsync them to other machines - often on GbE LAN, and then update those machines (with scripts in etcmanage). Yes, the extensions are baked in, but it might be useful to do this uncompressed, depending on if gzip beats GbE. jmmv@'s proposal sounds entirely reasonable to me. -- Posted automagically by a mail2news gateway at muc.de e.V. Please direct questions, flames, donations, etc. to news-admin@muc.de
[toc] | [prev] | [next] | [standalone]
| From | "Greg A. Woods" <woods@planix.ca> |
|---|---|
| Date | 2025-01-13 14:01 -0800 |
| Message-ID | <m1tXSVF-00Mo5zC@more.local> |
| In reply to | #11521 |
[Multipart message — attachments visible in raw view] — view raw
At Mon, 13 Jan 2025 11:10:28 +0100, Martin Husemann <martin@duskware.de> wrote:
Subject: Re: Supporting compression-less release builds
>
> Yes, especially an override for the compression level, so a simple solution
> would be USE_XZ_SETS=no GZIP_LEVEL=0 or something like that (where GZIP_LEVEL
> does not exist yet).
Note that the sets Makefile sets a default for the make variable XZ_OPT,
and then passes it as XZ_OPT in the environment.
If I remember right I tried XZ_OPT=-0 but it was still painfully slow.
Instead I use USE_XZ_SETS=no and USE_PIGZGZIP=yes.
It does also passes the make variable GZIP_N_FLAG as GZIP in the
environment. One could add a "-0" to GZIP_N_FLAG, but note that the
default value varies slightly based on the setting of ${USE_PIGZGZIP}.
> For the original problem, I would do something else: hack the makefiles
> localy to not update the compressed aritfacts at all when some make variable
> is set. If that turns out to be usefull and not totally awful, it could
> even be commited with some warnings in the documentation.
Indeed -- this could prove useful for those of us who never care about
writing installation images to limited physical media any more.
--
Greg A. Woods <gwoods@acm.org>
Kelowna, BC +1 250 762-7675 RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com> Avoncote Farms <woods@avoncote.ca>
[toc] | [prev] | [next] | [standalone]
| From | Julio Merino <jmmv@NetBSD.org> |
|---|---|
| Date | 2025-01-18 16:33 +0000 |
| Message-ID | <Z4vXz-KinaFTBfQ0@NetBSD.org> |
| In reply to | #11520 |
On Mon, Jan 13, 2025 at 12:34:14PM +1100, Simon Burge wrote: > Note that we sometimes use an explicit compression method elsewhere > in the build (eg compressing kernels) that might differ from the > compression method for sets. > > The way I read your proposal you seem to be suggesting "everything > in the build that needs to be compressed will be compressed with the > thing expressed by COMPRESS_RELEASE"; I'm pointing out that this might > break some things. Now, whether or not the things that are compressed > differently are for a valid reason is a different question! I can > think, for example, that an image that is designed to be dd'd to the > start of a memory stick might explicitly be compressed with gzip because > it might be easier to decompress a .gz file on a Windows box (I don't > know this to be the case, just using as a possible reason for preferring > a different compressor). Yes, I was thinking of unifying everything into just one method, but that would result in differences from the current situation indeed, hence my hesitation. In particular, there seem to be at least three things at play: * Release sets. These already have knobs for configuration. * Disk images. Compressed with gzip unconditionally. * Kernels. Compressed with gzip unconditionally. I do not know if there are good reasons for these differences, although I sense there aren't, and the reason sets support xz is because that's what made the most difference in final artifact sizes. We can take these one at a time and support differences among them if necessary. COMPRESS_RELEASE might not be a good idea at this point. But I'd still like to unify compression levels (to avoid the mess that are the -0 .. -9 flags throughout). -- Julio Merino -- Posted automagically by a mail2news gateway at muc.de e.V. Please direct questions, flames, donations, etc. to news-admin@muc.de
[toc] | [prev] | [next] | [standalone]
| From | Ignatios Souvatzis <is@netbsd.org> |
|---|---|
| Date | 2025-01-18 20:47 +0100 |
| Message-ID | <Z4wFU--9z1zqBgv0@bnhb484.de> |
| In reply to | #11524 |
hi, On Sat, Jan 18, 2025 at 04:33:19PM +0000, Julio Merino wrote: > In particular, there seem to be at least three things at play: > > * Release sets. These already have knobs for configuration. > * Disk images. Compressed with gzip unconditionally. > * Kernels. Compressed with gzip unconditionally. > > I do not know if there are good reasons for these differences, although > I sense there aren't, and the reason sets support xz is because that's > what made the most difference in final artifact sizes. (At least some) boot-loaders can boot gzip-compressed kernels directly. This is convenient for checking whether hardware Y would survive a kernel upgrade.... I think those loaders do not know how to handle xz and might not have the resources doing that. So while there's an argument for xz for the sets, I argue for the simpler compression method for kernels. Regards, -is -- Posted automagically by a mail2news gateway at muc.de e.V. Please direct questions, flames, donations, etc. to news-admin@muc.de
[toc] | [prev] | [next] | [standalone]
| From | Julio Merino <jmmv@NetBSD.org> |
|---|---|
| Date | 2025-01-18 22:38 +0000 |
| Message-ID | <Z4wtbHmlI4lOdcLa@NetBSD.org> |
| In reply to | #11526 |
On Sat, Jan 18, 2025 at 08:47:31PM +0100, Ignatios Souvatzis wrote: > hi, > > On Sat, Jan 18, 2025 at 04:33:19PM +0000, Julio Merino wrote: > > > In particular, there seem to be at least three things at play: > > > > * Release sets. These already have knobs for configuration. > > * Disk images. Compressed with gzip unconditionally. > > * Kernels. Compressed with gzip unconditionally. > > > > I do not know if there are good reasons for these differences, although > > I sense there aren't, and the reason sets support xz is because that's > > what made the most difference in final artifact sizes. > > (At least some) boot-loaders can boot gzip-compressed kernels > directly. This is convenient for checking whether hardware Y would > survive a kernel upgrade.... I think those loaders do not know how > to handle xz and might not have the resources doing that. > > So while there's an argument for xz for the sets, I argue for > the simpler compression method for kernels. Ah, thanks for confirming. So kernels need to be able to remain gzipped. -- Julio Merino -- Posted automagically by a mail2news gateway at muc.de e.V. Please direct questions, flames, donations, etc. to news-admin@muc.de
[toc] | [prev] | [next] | [standalone]
| From | Ignatios Souvatzis <ignatios@cs.uni-bonn.de> |
|---|---|
| Date | 2025-01-18 18:12 +0100 |
| Message-ID | <8C883F4B-4327-4C19-8BC4-A6E4299033F8@cs.uni-bonn.de> |
| In reply to | #11524 |
[Multipart message — attachments visible in raw view] — view raw
Julio wrote:
> * Release sets. These already have knobs for > configuration.
> * Disk images. Compressed with gzip >unconditionally.
> * Kernels. Compressed with gzip unconditionally.
>
> I do not know if there are good reasons for these differences
Bootloaders understand gzip, but don't (and I think, don't have the resources to) understand xz.
-is
[toc] | [prev] | [next] | [standalone]
| From | Julio Merino <jmmv@NetBSD.org> |
|---|---|
| Date | 2025-01-18 16:40 +0000 |
| Message-ID | <Z4vZakJwaphOnn9I@NetBSD.org> |
| In reply to | #11518 |
[Multipart message — attachments visible in raw view] — view raw
On Sun, Jan 12, 2025 at 02:49:41PM +0000, Julio Merino wrote: > > Does anyone see a problem with this plan? OK, I read through the comments in the sibling messages. Unifying everything into one compression method might not be the best idea, at least for now, because there might be reasons for kernels to be gzipped only and because changing this would be too intrusive (requiring changes to the docs and the like in some parameterized manner). So let's take changes one at a time, and possibly keep the tunables for sets / images / kernels separate. That said, I still think doing a code cleanup to unify where flags are specified (so that compression levels are consistently defined) is worthwhile. I'm attaching just the first patch that I had in mind, which implements the COMPRESS_SETS change only. It works, but I haven't done sufficient validation yet. What do you think? -- Julio Merino
[toc] | [prev] | [standalone]
Back to top | Article view | muc.lists.netbsd.tech.userlevel
csiph-web