Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #62119 > unrolled thread

Limiting the size of installed changelogs

Back to article view | Back to linux.debian.kernel

Contents

  Limiting the size of installed changelogs Ben Hutchings <ben@decadent.org.uk> - 2018-09-13 12:30 +0200
    Re: Limiting the size of installed changelogs Lars Wirzenius <liw@liw.fi> - 2018-09-13 12:50 +0200
      Re: Limiting the size of installed changelogs Ben Hutchings <ben@decadent.org.uk> - 2018-09-13 14:50 +0200
        Re: Limiting the size of installed changelogs Mattia Rizzolo <mattia@debian.org> - 2018-09-13 18:10 +0200
    Re: Limiting the size of installed changelogs "W. Martin Borgert" <debacle@debian.org> - 2018-09-13 14:00 +0200
    Re: Limiting the size of installed changelogs Roger Shimizu <rogershimizu@gmail.com> - 2018-09-13 14:30 +0200
    Re: Limiting the size of installed changelogs Boyuan Yang <byang@debian.org> - 2018-09-13 15:00 +0200
      Re: Limiting the size of installed changelogs Jeremy Bicha <jbicha@debian.org> - 2018-09-13 15:10 +0200
        Re: Limiting the size of installed changelogs Ben Hutchings <ben@decadent.org.uk> - 2018-10-13 17:40 +0200
    Re: Limiting the size of installed changelogs Mattia Rizzolo <mattia@debian.org> - 2018-09-13 18:20 +0200
      Re: Limiting the size of installed changelogs Ruben Undheim <lists@beebeetle.com> - 2018-09-14 00:40 +0200
    Re: Limiting the size of installed changelogs Ben Hutchings <ben@decadent.org.uk> - 2018-10-13 18:10 +0200

#62119 — Limiting the size of installed changelogs

[Multipart message — attachments visible in raw view] — view raw

The src:linux package has a very big changelog (about 1700 kiB
uncompressed, 600 kiB gzipped).  On my system the largest installed
changelogs, by some way, are all versions of this.  (The next largest
changelogs come from src:glibc, at about 200 kiB gzipped.)

I recently had to introduce yet more installed copies of this changelog
because the case where we used linked doc directories is no longer
valid (arch-dependent package became arch-independent).

The older history is unlikely to be of any use to users.  So on smaller
systems this could be a significant waste of space.  (I know it's
possible to filter out the installation of docs entirely, but I don't
think this option is well known.)

- A large part of the changelog is listing the changes in upstream
stable updates.  These are mostly important changes, and we already try
to leave out those that are clearly irrelevant to Debian.  Should we
continue to include these, or limit to those that address CVEs or
Debian bug reports?

- Would it make sense to split the changelog, leaving older entries
only in the source package?  If so, should this be done manually, or
would it make sense to have dh_installchangelogs split at some age or
size limit?

- Does it make sense to compress changelogs with xz?  For src:linux,
this achieves about a 20-25% reduction over gzip.

Ben.

-- 
Ben Hutchings
Computers are not intelligent.	They only think they are.

[toc] | [next] | [standalone]

#62120

[Multipart message — attachments visible in raw view] — view raw

On Thu, Sep 13, 2018 at 11:22:37AM +0100, Ben Hutchings wrote:
> The src:linux package has a very big changelog (about 1700 kiB
> uncompressed, 600 kiB gzipped).  On my system the largest installed
> changelogs, by some way, are all versions of this.  (The next largest
> changelogs come from src:glibc, at about 200 kiB gzipped.)

I was curious, so I ran the following on my laptop:

$ find /usr/share/doc -name '*changelog*' -printf '%s %p\n' | sort -n | tail -n10 | while read size path; do echo $(humanify "$size") "$path"; done
2 MB /usr/share/doc/libgstreamer-plugins-bad1.0-0/changelog.gz
2 MB /usr/share/doc/xserver-common/changelog.gz
2 MB /usr/share/doc/xserver-xephyr/changelog.gz
2 MB /usr/share/doc/xserver-xorg-core/changelog.gz
2 MB /usr/share/doc/xserver-xorg-legacy/changelog.gz
2 MB /usr/share/doc/xwayland/changelog.gz
2 MB /usr/share/doc/gimp/changelog.gz
2 MB /usr/share/doc/gimp-data/changelog.gz
2 MB /usr/share/doc/libgimp2.0/changelog.gz
4 MB /usr/share/doc/libvirt0/changelog.gz

While changelogs are sometimes really useful, that's a fair bit of
diskspace on my laptop, and a fair bit of bandwidth to transfer, for
fairly little gain.

> The older history is unlikely to be of any use to users.  So on smaller
> systems this could be a significant waste of space.  (I know it's
> possible to filter out the installation of docs entirely, but I don't
> think this option is well known.)

I agree that we can drop older changelog entries. I suggest truncating
changelogs, at least large ones, by dropping any entries from before
stable, when uploading to unstable. So uploading to unstable today,
any entries from before stretch can be dropped. When a changelog is
truncated, there should be a URL to the full changelog, so those who
really are curious can find it. (Possibly oldstable would be a better
cut-off point.)

Alternatively, we could change things so that the changelog for each
Debian release (major part of version) is put in its own file.

With this scheme, uploads to unstable today would have
/usr/share/doc/*/changelog.debian10.gz. Once buster is released,
uploads would include changelog.debian11.gz instead. This would allow
a fairly natural rotation.

> - Does it make sense to compress changelogs with xz?  For src:linux,
> this achieves about a 20-25% reduction over gzip.

I'd be OK with using xz -9 to compress changelogs. This would also be
easily doable in debhelper. From the ten files I showed at the
beginning, I get about a 29% savings recompressing as xz.

-- 
I want to build worthwhile things that might last. --joeyh

[toc] | [prev] | [next] | [standalone]

#62126

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2018-09-13 at 13:46 +0300, Lars Wirzenius wrote:
> On Thu, Sep 13, 2018 at 11:22:37AM +0100, Ben Hutchings wrote:
> > The src:linux package has a very big changelog (about 1700 kiB
> > uncompressed, 600 kiB gzipped).  On my system the largest installed
> > changelogs, by some way, are all versions of this.  (The next largest
> > changelogs come from src:glibc, at about 200 kiB gzipped.)
> 
> I was curious, so I ran the following on my laptop:
> 
> $ find /usr/share/doc -name '*changelog*' -printf '%s %p\n' | sort -n | tail -n10 | while read size path; do echo $(humanify "$size") "$path"; done
> 2 MB /usr/share/doc/libgstreamer-plugins-bad1.0-0/changelog.gz
> 2 MB /usr/share/doc/xserver-common/changelog.gz
> 2 MB /usr/share/doc/xserver-xephyr/changelog.gz
[...]

I should have said I was only considering Debian changelogs here.  It's
not too surprising that upstream changelogs can be even larger.  The
inclusion of upstream changelogs (vs release notes) was discussed in #
459427.

Ben.

-- 
Ben Hutchings
Computers are not intelligent.	They only think they are.

[toc] | [prev] | [next] | [standalone]

#62130

[Multipart message — attachments visible in raw view] — view raw

On Thu, Sep 13, 2018 at 01:45:00PM +0100, Ben Hutchings wrote:
> It's
> not too surprising that upstream changelogs can be even larger.  The
> inclusion of upstream changelogs (vs release notes) was discussed in #
> 459427.

Not everybody may know it, but ubuntu has been patching debhelper for
many years already to not automatically install upstream changelogs like
dh_installchangelogs does starting from compat 7.

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[toc] | [prev] | [next] | [standalone]

#62124

Quoting Ben Hutchings <ben@decadent.org.uk>:
> - Would it make sense to split the changelog, leaving older entries
> only in the source package?

As long as the full d/ch is in the source package, I'm fine.

On one of my embedded machines changelogs are 2/3 of /usr/share/doc.
E.g. multiple copies of the same changelog for the various binary
packages build from bind9.
The custom kernel package on my device doesn't install docs :~)

[toc] | [prev] | [next] | [standalone]

#62125

On Thu, Sep 13, 2018 at 7:22 PM, Ben Hutchings <ben@decadent.org.uk> wrote:
> The src:linux package has a very big changelog (about 1700 kiB
> uncompressed, 600 kiB gzipped).  On my system the largest installed
> changelogs, by some way, are all versions of this.  (The next largest
> changelogs come from src:glibc, at about 200 kiB gzipped.)

Makes full sense.
Especially sometimes you need to have at least two kernels on system:
the one is running and another security updated new one.

While SSD is still expensive, saving storage is always welcome.

> I recently had to introduce yet more installed copies of this changelog
> because the case where we used linked doc directories is no longer
> valid (arch-dependent package became arch-independent).
>
> The older history is unlikely to be of any use to users.  So on smaller
> systems this could be a significant waste of space.  (I know it's
> possible to filter out the installation of docs entirely, but I don't
> think this option is well known.)
>
> - A large part of the changelog is listing the changes in upstream
> stable updates.  These are mostly important changes, and we already try
> to leave out those that are clearly irrelevant to Debian.  Should we
> continue to include these, or limit to those that address CVEs or
> Debian bug reports?
>
> - Would it make sense to split the changelog, leaving older entries
> only in the source package?  If so, should this be done manually, or
> would it make sense to have dh_installchangelogs split at some age or
> size limit?

I think it should at least contain the content of one release cycle. (or two?)
E.g. When Buster get released, we sometimes need to check what's been
changed since Stretch.

> - Does it make sense to compress changelogs with xz?  For src:linux,
> this achieves about a 20-25% reduction over gzip.

If there's not much difference when uncompressing xz, I guess it's
good idea to switch to xz for Buster.

Thanks for your idea!

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

[toc] | [prev] | [next] | [standalone]

#62127

[Multipart message — attachments visible in raw view] — view raw

As for limiting / truncating Debian changelogs, there is an existing 
implementation that we may refer to, which is Ubuntu's approach: they keep 
certain amount of changelog in the .deb package (I'm not sure about the 
detailed algorithm but that is available somewhere) while adding some 
explanation words at the bottom providing a URL that can access full changelog 
online.

If Debian wants to limit changelog size as well, it may be better if we keep 
similar / identical behaviour / implementation as Ubuntu.

--
Regards,
Boyuan Yang

在 2018年9月13日星期四 EDT 上午6:22:37，Ben Hutchings 写道：
> The src:linux package has a very big changelog (about 1700 kiB
> uncompressed, 600 kiB gzipped).  On my system the largest installed
> changelogs, by some way, are all versions of this.  (The next largest
> changelogs come from src:glibc, at about 200 kiB gzipped.)
> 
> I recently had to introduce yet more installed copies of this changelog
> because the case where we used linked doc directories is no longer
> valid (arch-dependent package became arch-independent).
> 
> The older history is unlikely to be of any use to users.  So on smaller
> systems this could be a significant waste of space.  (I know it's
> possible to filter out the installation of docs entirely, but I don't
> think this option is well known.)
> 
> - A large part of the changelog is listing the changes in upstream
> stable updates.  These are mostly important changes, and we already try
> to leave out those that are clearly irrelevant to Debian.  Should we
> continue to include these, or limit to those that address CVEs or
> Debian bug reports?
> 
> - Would it make sense to split the changelog, leaving older entries
> only in the source package?  If so, should this be done manually, or
> would it make sense to have dh_installchangelogs split at some age or
> size limit?
> 
> - Does it make sense to compress changelogs with xz?  For src:linux,
> this achieves about a 20-25% reduction over gzip.
> 
> Ben.

[toc] | [prev] | [next] | [standalone]

#62128

On Thu, Sep 13, 2018 at 8:53 AM Boyuan Yang <byang@debian.org> wrote:
> If Debian wants to limit changelog size as well, it may be better if we keep
> similar / identical behaviour / implementation as Ubuntu.

More specifically, Ubuntu packages keep changelog entries for 10
Debian/Ubuntu package versions.

At the bottom of the changelog, there is a comment like:

# For older changelog entries, run 'apt-get changelog libnautilus-extension1a'

apt-get changelog works on Debian systems too; it requires Internet
access to fetch the complete changelog.

The changelog stripping is done by pkgstripfiles, a bash script in the
pkgbinarymanagler package. That package is not available in Debian.

Thanks,
Jeremy Bicha

[toc] | [prev] | [next] | [standalone]

#62352

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2018-09-13 at 09:06 -0400, Jeremy Bicha wrote:
> On Thu, Sep 13, 2018 at 8:53 AM Boyuan Yang <byang@debian.org> wrote:
> > If Debian wants to limit changelog size as well, it may be better if we keep
> > similar / identical behaviour / implementation as Ubuntu.
> 
> More specifically, Ubuntu packages keep changelog entries for 10
> Debian/Ubuntu package versions.
> 
> At the bottom of the changelog, there is a comment like:
> 
> # For older changelog entries, run 'apt-get changelog libnautilus-extension1a'
> 
> apt-get changelog works on Debian systems too; it requires Internet
> access to fetch the complete changelog.

It worked on 0 out of the 2 systems I tried it on.

One is running jessie (in LTS), so none of the changelogs for security
updates are available.  (This problem doesn't occur for stretch because
security updates get copied into proposed-updates.)

The other is running unstable, and the package I tried was slightly
older than the version in the archive.

I think we would need to fix these back-end problems before relying on
apt-get changelog.

Ben.

> The changelog stripping is done by pkgstripfiles, a bash script in the
> pkgbinarymanagler package. That package is not available in Debian.

-- 
Ben Hutchings
All the simple programs have been written, and all the good names taken

[toc] | [prev] | [next] | [standalone]

#62131

[Multipart message — attachments visible in raw view] — view raw

On Thu, Sep 13, 2018 at 11:22:37AM +0100, Ben Hutchings wrote:
> - Would it make sense to split the changelog, leaving older entries
> only in the source package?  If so, should this be done manually, or
> would it make sense to have dh_installchangelogs split at some age or
> size limit?

Yes, this would be very sensible IMHO.

Having debhelper cut off the changelogs from 4 or 6 years before (and
inserting a pointer to the source package for the rest) sounds like
a good idea to me.

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[toc] | [prev] | [next] | [standalone]

#62137

> Yes, this would be very sensible IMHO.
> 
> Having debhelper cut off the changelogs from 4 or 6 years before (and
> inserting a pointer to the source package for the rest) sounds like
> a good idea to me.

It would be nice if X number of the oldest entries are kept - particularly the
"initial release" one. "X" can be 1 also. It is useful to know when the package
entered Debian initially. Between the newer versions and the very old one(s),
there can be a comment (with some nice dots and lines) pointing to where to
find the missing ones. I remember I have been annoyed on Ubuntu machines when
just seeing the "head" of the changelog file..

I also really hope that the full changelog always remains in the source
package, and that all cutting is handled automatically by dh_installchangelogs.


Regards
Ruben

[toc] | [prev] | [next] | [standalone]

#62353

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2018-09-13 at 11:22 +0100, Ben Hutchings wrote:
[...]
> - A large part of the changelog is listing the changes in upstream
> stable updates.  These are mostly important changes, and we already try
> to leave out those that are clearly irrelevant to Debian.  Should we
> continue to include these, or limit to those that address CVEs or
> Debian bug reports?

There was one comment in favour of keeping these lists, and none for
excluding them.  I intend to continue including them.

> - Would it make sense to split the changelog, leaving older entries
> only in the source package?  If so, should this be done manually, or
> would it make sense to have dh_installchangelogs split at some age or
> size limit?

There seemed to be consensus that this would be reasonable, but
differing opinions about how to do it.

Several people mentioned Ubuntu's automatically truncated changelogs. 
These would require packaging a new tool and improving the handling of
source changelogs on metadata.ftp-master.debian.org.  I don't have time
to work on this but someone else could.

I've committed a manual change in the linux source package, moving all
entries up to the stretch branch-point into changelog.old.  This will
mean that in a buster installation the changelog will include all
changes since stretch.

> - Does it make sense to compress changelogs with xz?  For src:linux,
> this achieves about a 20-25% reduction over gzip.

There was some support for this, but differing results for compression.
It wasn't clear that the gain would be worth it.  I don't have time to
work on investigating this or proposing a policy change, but someone
else could.

Ben.

-- 
Ben Hutchings
All the simple programs have been written, and all the good names taken

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web