Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.devel > #111829

gzip --rsyncable (was: Re: De-vendoring gnulib in Debian packages)

Show all headers | View raw

[Multipart message — attachments visible in raw view] - view raw

"Theodore Ts'o" <tytso@mit.edu> writes:

> On Sun, May 12, 2024 at 04:27:06PM +0200, Simon Josefsson wrote:
>> Going into detail, you use 'gzip -9n' but I use git-archive defaults
>> which is the same as -n aka --no-name.  I agree adding -9 aka --best is
>> an improvement.  Gnulib's maint.mk also add --rsyncable, would you agree
>> that this is also an improvement?
>
> I'm not convinced --rsyncable is an improvement.  It makes the
> compressed object slightly larger, and in exchange, if the compressed
> object changes slightly, it's possible that when you rsync the changed
> file, it might be more efficient.  But in the case of PGP signed
> release tarballs, the file is constant; it's never going to change,
> and even if there are slight changes between say, e2fsprogs v1.47.0
> and e2fsprogs v1.47.1, in practice, this is not something --rsyncable
> can take advantage of, unless you manually copy
> e2fsprogs-v1.47.0.tar.gz to e2fsprogs-v1.47.1.tar.bz, and then rsync
> e2fsprogs-v1.471.tar.g.... and I don't think anyone is doing this,
> either automatically or manually.
>
> That being said, --rsyncable is mostly harmless, so I don't have
> strong feelings about changing it to add or remove in someone's
> release workflow.

Your example had me convinced, and I thought some more about why we
really should keep using it as it consumes a small percentage more CPU
and disk space.  I have realized that another common operation is
storing and transfering _several_ different releases of e2fsprogs.  I
would suspect that most releases of software is fairly similar to the
previous release when uncompressed.  With gzip --rsyncable, the tarballs
should then be mostly similar.  Without --rsyncable, they will largely
be different if I understand correctly.  This affects dedup-able storage
and transfer methods, and some anecdotical evidence suggests this
improvement is significant - going from 215GB to 176GB vs 13GB:

https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/429

Maybe someone could do some experiment to see if there is substance to
this argument, its not clear to me that the example is comparable.
Storing/transferring several releases for the same software could add
significant savings for larger set of archives.

As the downside seems fairly small, and the potential upside may be
significant, I will use and recommend --rsyncable for git-archive
release tarballs:

   git archive --format=tar --prefix=$PACKAGE-$VERSION/ HEAD | \
      env GZIP= gzip --no-name --best --rsyncable \
      > $PACKAGE-$VERSION-src.tar.gz

/Simon

Back to linux.debian.devel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

De-vendoring gnulib in Debian packages Simon Josefsson <simon@josefsson.org> - 2024-05-11 16:10 +0200
  Re: De-vendoring gnulib in Debian packages Bruno Haible <bruno@clisp.org> - 2024-05-11 18:00 +0200
    Re: De-vendoring gnulib in Debian packages Simon Josefsson <simon@josefsson.org> - 2024-05-11 18:40 +0200
  Re: De-vendoring gnulib in Debian packages Paul Eggert <eggert@cs.ucla.edu> - 2024-05-11 18:50 +0200
  Re: De-vendoring gnulib in Debian packages "Theodore Ts'o" <tytso@mit.edu> - 2024-05-12 14:10 +0200
    Re: De-vendoring gnulib in Debian packages Simon Josefsson <simon@josefsson.org> - 2024-05-12 16:30 +0200
      Re: De-vendoring gnulib in Debian packages "Theodore Ts'o" <tytso@mit.edu> - 2024-05-12 21:00 +0200
        gzip --rsyncable (was: Re: De-vendoring gnulib in Debian packages) Simon Josefsson <simon@josefsson.org> - 2024-05-20 10:30 +0200
    Re: De-vendoring gnulib in Debian packages Russ Allbery <rra@debian.org> - 2024-05-12 17:50 +0200
      Re: De-vendoring gnulib in Debian packages Ansgar 🙀 <ansgar@43-1.org> - 2024-05-12 18:30 +0200
        Re: De-vendoring gnulib in Debian packages Russ Allbery <rra@debian.org> - 2024-05-12 19:50 +0200

csiph-web