Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.security > #6510

Re: Size of the security-tracker repository

Show all headers | View raw

Hi,

On 04/06/2026 19:25, Christoph Biedl wrote:
> Quite frankly, I doubt I'm telling anything new here ...

Indeed:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908678

This was somewhat resolved at salsa/gitlab and continued to work fairly 
well, until a few weeks ago where I believe data/CVE/list reached a 
git/gitlab limit, where delta compression was not done as well as before.

I believe this got better though (bandwidth now <1MB/commit, was 
10M/commit at a point), maybe something was tweaked again at Salsa.

> .git/objects/pack has 28 Gbytes, and given the current rate this will
> grow beyond 100 by the end of the year.

Local delta compression (.pack files) sizes this down drastically, 
around 4-5GB. It's done regularly in the background by default by Git, 
or through 'git gc'.

The higher bandwidth requirements, and regular gitlab timeouts for 
non-daily pulls, are now a problem though.

I was suggested to use '--filter=tree:0' as a work-around, unless you 
need to inspect the full history.

> What did I miss?

See the BTS entry above for a discussion.
Among obstacles: rewriting the Git history (now requires around 2-3 days 
of computing), getting an agreement and official review, adapting the tools.

I had suggested (last BTS comment) going with the by-year rewrite, and 
add Git hooks to rebuild a single data/CVE/list, so the tooling would 
still work as-is, at least in a first step.
(This is best discussed in the BTS entry.)

Independently, there's also a partial per-CVE rewrite for git-log purposes:
https://salsa.debian.org/lts-team/cvehist
(which is another thing to adapt if the tracker repo changes ;))

Cheers!
Sylvain Beucler
Debian LTS Team

Back to linux.debian.security | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Size of the security-tracker repository Christoph Biedl <debian.axhn@manchmal.in-ulm.de> - 2026-06-04 19:40 +0200
  Re: Size of the security-tracker repository Sylvain Beucler <beuc@beuc.net> - 2026-06-06 09:10 +0200
    Re: Size of the security-tracker repository Christoph Biedl <debian.axhn@manchmal.in-ulm.de> - 2026-06-07 21:30 +0200
      Re: Size of the security-tracker repository Sylvain Beucler <beuc@beuc.net> - 2026-06-11 07:30 +0200
        Re: Size of the security-tracker repository Sylvain Beucler <beuc@beuc.net> - 2026-06-12 07:40 +0200

csiph-web