Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.security > #6509 > unrolled thread

Size of the security-tracker repository

Back to article view | Back to linux.debian.security

Contents

  Size of the security-tracker repository Christoph Biedl <debian.axhn@manchmal.in-ulm.de> - 2026-06-04 19:40 +0200
    Re: Size of the security-tracker repository Sylvain Beucler <beuc@beuc.net> - 2026-06-06 09:10 +0200
      Re: Size of the security-tracker repository Christoph Biedl <debian.axhn@manchmal.in-ulm.de> - 2026-06-07 21:30 +0200
        Re: Size of the security-tracker repository Sylvain Beucler <beuc@beuc.net> - 2026-06-11 07:30 +0200
          Re: Size of the security-tracker repository Sylvain Beucler <beuc@beuc.net> - 2026-06-12 07:40 +0200

#6509 — Size of the security-tracker repository

[Multipart message — attachments visible in raw view] — view raw

Quite frankly, I doubt I'm telling anything new here ...

Hello,

as likely many other people, I follow the changes in the
security-tracker repository¹, and I noticed it structure is making
things inconventient.

While the size of a checkout is already somewhat big (around 60 Mbyte),
there is one file data/CVE/list with a size of about 55 Mbyte, and it
gets changed several times a day. Now the way git works, each state is
stored in full. So it's little surprise git pull adds 300 to 900 Mbyte
to .git/objects/pack every(!) day.

Perhaps I'm the last person not having grown into the habit of "Why
bother, bandwidth is cheap, storage is cheap" - in my opinion however
this reached a point that is way beyond acceptable. In my checkout,
.git/objects/pack has 28 Gbytes, and given the current rate this will
grow beyond 100 by the end of the year. If you don't follow, try an
initial git clone on a box with 4 Gbyte RAM, connected with 10 Mbit or
less.

Looking into the repository, the solution is so obvious I was wondering
why it hasn't been done yet: Split data/CVE/list by year: All CVE
numbers from 1999 would be in a file data/CVE/list.1999 and so on.
Occasionally you'd have to edit two files, still you'd have to push like
5 MByte, not 55.

What did I miss?

    Christoph

¹ https://salsa.debian.org/security-tracker-team/security-tracker

[toc] | [next] | [standalone]

#6510

Hi,

On 04/06/2026 19:25, Christoph Biedl wrote:
> Quite frankly, I doubt I'm telling anything new here ...

Indeed:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908678

This was somewhat resolved at salsa/gitlab and continued to work fairly 
well, until a few weeks ago where I believe data/CVE/list reached a 
git/gitlab limit, where delta compression was not done as well as before.

I believe this got better though (bandwidth now <1MB/commit, was 
10M/commit at a point), maybe something was tweaked again at Salsa.

> .git/objects/pack has 28 Gbytes, and given the current rate this will
> grow beyond 100 by the end of the year.

Local delta compression (.pack files) sizes this down drastically, 
around 4-5GB. It's done regularly in the background by default by Git, 
or through 'git gc'.

The higher bandwidth requirements, and regular gitlab timeouts for 
non-daily pulls, are now a problem though.

I was suggested to use '--filter=tree:0' as a work-around, unless you 
need to inspect the full history.

> What did I miss?

See the BTS entry above for a discussion.
Among obstacles: rewriting the Git history (now requires around 2-3 days 
of computing), getting an agreement and official review, adapting the tools.

I had suggested (last BTS comment) going with the by-year rewrite, and 
add Git hooks to rebuild a single data/CVE/list, so the tooling would 
still work as-is, at least in a first step.
(This is best discussed in the BTS entry.)

Independently, there's also a partial per-CVE rewrite for git-log purposes:
https://salsa.debian.org/lts-team/cvehist
(which is another thing to adapt if the tracker repo changes ;))

Cheers!
Sylvain Beucler
Debian LTS Team

[toc] | [prev] | [next] | [standalone]

#6511

[Multipart message — attachments visible in raw view] — view raw

Sylvain Beucler wrote...

> On 04/06/2026 19:25, Christoph Biedl wrote:
> > Quite frankly, I doubt I'm telling anything new here ...
> 
> Indeed:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908678

Oh boy. I could make some extra waves there if anyone considers this
necessary.

> > .git/objects/pack has 28 Gbytes, and given the current rate this will
> > grow beyond 100 by the end of the year.
> 
> Local delta compression (.pack files) sizes this down drastically, around
> 4-5GB. It's done regularly in the background by default by Git, or through
> 'git gc'.

This I can confirm, the auto-gc brought this down to 5.3G last night.
Aside, that process was also blatantly ignoring the gc.bigPackThreshold
settings, but that's another story.

Still, this is a lot worse than ealier: On April 7th the total size of
.git/objects/ was just 1571M here - a day later things started to go
downhill.

> > What did I miss?
> 
> See the BTS entry above for a discussion.
> Among obstacles: rewriting the Git history (now requires around 2-3 days of
> computing), getting an agreement and official review, adapting the tools.

It's a bit saddening nothing has happened about this while everybody
should have been aware this will not resolve by itself - but rather get
worse over time :(

Data point about history rewriting, there are now 149k commits. If
rewriting a single commit takes just one second, the above number is
sensible.

> Independently, there's also a partial per-CVE rewrite for git-log purposes:
> https://salsa.debian.org/lts-team/cvehist
> (which is another thing to adapt if the tracker repo changes ;))

Whatever improves the situation and assert it's not going to hit again.

    Christoph

[toc] | [prev] | [next] | [standalone]

#6512

Hi,

On 07/06/2026 21:24, Christoph Biedl wrote:
> This I can confirm, the auto-gc brought this down to 5.3G last night.
[...]
> Still, this is a lot worse than ealier: On April 7th the total size of
> .git/objects/ was just 1571M here - a day later things started to go
> downhill.

Interesting. Do you graph the .git usage somehow, and do you have more data?


I believe that Salsa is still sending very suboptimal packs over the 
network, which are then reused by the local Git.

'git gc --aggressive --keep-largest-pack' repacked the most recents 
packs from last week MUCH more efficiently (1.6GB -> 12MB), which tends 
to confirm this hypothesis.

'git gc --aggressive' is supposed to repack everything, but I'm not able 
to run it on my computer anymore at this currently OOMs at >20GB RAM, 
which means previous packs may stay fat and non-optimized forever.


> It's a bit saddening nothing has happened about this while everybody
> should have been aware this will not resolve by itself - but rather get
> worse over time :(

I also see that files >50MB now get a warning at GitHub (before getting 
rejected at >100MB), so with our data/CVE/list at 53MB we're starting to 
hit all kinds of hard limits, confirming we have to do something sooner 
than later.

Maybe this is something where the LTS Team can help with, let's wait for 
the bookworm LTS handover to take place this week, and I'll try and 
reach out to involved teams.

Cheers!
Sylvain Beucler
Debian LTS Team

[toc] | [prev] | [next] | [standalone]

#6513

Hi,

(moving the discussion to debian-security-tracker, last post to 
debian-security :))

On 11/06/2026 07:21, Sylvain Beucler wrote:
> On 07/06/2026 21:24, Christoph Biedl wrote:
>> This I can confirm, the auto-gc brought this down to 5.3G last night.
> [...]
>> Still, this is a lot worse than ealier: On April 7th the total size of
>> .git/objects/ was just 1571M here - a day later things started to go
>> downhill.
> 
> Interesting. Do you graph the .git usage somehow, and do you have more 
> data?
> 
> 
> I believe that Salsa is still sending very suboptimal packs over the 
> network, which are then reused by the local Git.
> 
> 'git gc --aggressive --keep-largest-pack' repacked the most recents 
> packs from last week MUCH more efficiently (1.6GB -> 12MB), which tends 
> to confirm this hypothesis.
> 
> 'git gc --aggressive' is supposed to repack everything, but I'm not able 
> to run it on my computer anymore at this currently OOMs at >20GB RAM, 
> which means previous packs may stay fat and non-optimized forever.

FTR I ran:
   $ git -c pack.windowMemory=2G gc --aggressive
which took 15h and peeked at 30GB RAM (over the nproc*2GB limit).

Result:
   $ du -sh .git
   1,8G	.git
(from 7GB)

Not very practical of course, but mainly this fits my above hypothesis.

>> It's a bit saddening nothing has happened about this while everybody
>> should have been aware this will not resolve by itself - but rather get
>> worse over time :(
> 
> I also see that files >50MB now get a warning at GitHub (before getting 
> rejected at >100MB), so with our data/CVE/list at 53MB we're starting to 
> hit all kinds of hard limits, confirming we have to do something sooner 
> than later.
> 
> Maybe this is something where the LTS Team can help with, let's wait for 
> the bookworm LTS handover to take place this week, and I'll try and 
> reach out to involved teams.

Cheers!
Sylvain Beucler
Debian LTS Team

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.security

csiph-web