Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #264822 > unrolled thread

Why is /var/lib/apt/lists not in /var/cache?

Back to article view | Back to linux.debian.user

Contents

  Why is /var/lib/apt/lists not in /var/cache? Stefan Monnier <monnier@iro.umontreal.ca> - 2023-12-16 17:10 +0100
    Re: Why is /var/lib/apt/lists not in /var/cache? David Wright <deblis@lionunicorn.co.uk> - 2023-12-16 18:40 +0100
      Re: Why is /var/lib/apt/lists not in /var/cache? Stefan Monnier <monnier@iro.umontreal.ca> - 2023-12-16 19:00 +0100
        Re: Why is /var/lib/apt/lists not in /var/cache? David Wright <deblis@lionunicorn.co.uk> - 2023-12-17 06:20 +0100
          Re: Why is /var/lib/apt/lists not in /var/cache? Stefan Monnier <monnier@iro.umontreal.ca> - 2023-12-17 07:10 +0100
            Re: Why is /var/lib/apt/lists not in /var/cache? David Wright <deblis@lionunicorn.co.uk> - 2023-12-17 19:40 +0100
              Re: Why is /var/lib/apt/lists not in /var/cache? Stefan Monnier <monnier@iro.umontreal.ca> - 2023-12-17 21:40 +0100
                Re: Why is /var/lib/apt/lists not in /var/cache? David Wright <deblis@lionunicorn.co.uk> - 2023-12-18 01:50 +0100
                  Re: Why is /var/lib/apt/lists not in /var/cache? Stefan Monnier <monnier@iro.umontreal.ca> - 2023-12-18 05:10 +0100
    Re: Why is /var/lib/apt/lists not in /var/cache? Max Nikulin <manikulin@gmail.com> - 2023-12-17 03:20 +0100
      Re: Why is /var/lib/apt/lists not in /var/cache? Stefan Monnier <monnier@iro.umontreal.ca> - 2023-12-17 06:10 +0100
    Re: Why is /var/lib/apt/lists not in /var/cache? Stefan Monnier <monnier@iro.umontreal.ca> - 2023-12-18 15:10 +0100
      Re: Why is /var/lib/apt/lists not in /var/cache? <tomas@tuxteam.de> - 2023-12-18 15:40 +0100

#264822 — Why is /var/lib/apt/lists not in /var/cache?

AFAICT, all of `/var/lib/apt/lists` is made of files fetched from
repositories, which APT will re-fetch if missing.
So, it sounds to me like it belongs in `/var/cache/apt/lists`, really.
What am I missing?  Or is it just a historical accident?


        Stefan "whose `/var/lib/apt/lists` is a symlink into /`var/cache`"

[toc] | [next] | [standalone]

#264826

On Sat 16 Dec 2023 at 10:59:48 (-0500), Stefan Monnier wrote:
> AFAICT, all of `/var/lib/apt/lists` is made of files fetched from
> repositories, which APT will re-fetch if missing.
> So, it sounds to me like it belongs in `/var/cache/apt/lists`, really.
> What am I missing?  Or is it just a historical accident?

This may not answer your question, but when you fetch new lists,
they aren't the same as what was there before, and don't actually
match your system until you upgrade it.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#264827

David Wright [2023-12-16 11:30:01] wrote:
> On Sat 16 Dec 2023 at 10:59:48 (-0500), Stefan Monnier wrote:
>> AFAICT, all of `/var/lib/apt/lists` is made of files fetched from
>> repositories, which APT will re-fetch if missing.
>> So, it sounds to me like it belongs in `/var/cache/apt/lists`, really.
>> What am I missing?  Or is it just a historical accident?
>
> This may not answer your question, but when you fetch new lists,
> they aren't the same as what was there before, and don't actually
> match your system until you upgrade it.

AFAIK those lists aren't supposed to "match your system", are they?
IIUC they just keep a local copy of the repository's list.

Admittedly if I `apt update` today, and then a week from now I delete
the lists and do `apt update` again I won't get the same files.
But similar, if I delete one of the files in
`/var/cache/apt/archive`, there's no guarantee that I'll be able to
re-download this exact file next week (tho I guess
https://snapshot.debian.org/ might still make it possible if you want it
badly enough).


        Stefan

[toc] | [prev] | [next] | [standalone]

#264846

On Sat 16 Dec 2023 at 12:50:51 (-0500), Stefan Monnier wrote:
> David Wright [2023-12-16 11:30:01] wrote:
> > On Sat 16 Dec 2023 at 10:59:48 (-0500), Stefan Monnier wrote:
> >> AFAICT, all of `/var/lib/apt/lists` is made of files fetched from
> >> repositories, which APT will re-fetch if missing.
> >> So, it sounds to me like it belongs in `/var/cache/apt/lists`, really.
> >> What am I missing?  Or is it just a historical accident?
> >
> > This may not answer your question, but when you fetch new lists,
> > they aren't the same as what was there before, and don't actually
> > match your system until you upgrade it.
> 
> AFAIK those lists aren't supposed to "match your system", are they?
> IIUC they just keep a local copy of the repository's list.

Whether it's considered necessary or desirable depends on the
user and what they use the lists for (apart from the obvious).

> Admittedly if I `apt update` today, and then a week from now I delete
> the lists and do `apt update` again I won't get the same files.
> But similar, if I delete one of the files in
> `/var/cache/apt/archive`, there's no guarantee that I'll be able to
> re-download this exact file next week (tho I guess
> https://snapshot.debian.org/ might still make it possible if you want it
> badly enough).

Some packages will stay the same for years, but in the past week
I can see four occasions when changes in list contents have occurred
on oldstable. So there's little similarity.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#264849

> Some packages will stay the same for years, but in the past week
> I can see four occasions when changes in list contents have occurred
> on oldstable. So there's little similarity.

The question is not really whether "apt/lists" is similar to
"apt/archives", but whether the content of "apt/lists" is appropriate
for `/var/cache`.

AFAIK nothing bad will happen to your system if you delete
`/var/lib/apt/lists`.  It will still run as before and you can still
install packages as before.  The only thing that seems to be impacted is
that some APT operations will temporarily work in a kind of "degraded"
mode until the next `apt update`.

That seems similar to things like `locate` failing if you remove
`/var/log/plocate/plocate.db` (until that DB is rebuilt).


        Stefan

[toc] | [prev] | [next] | [standalone]

#264868

On Sun 17 Dec 2023 at 01:06:28 (-0500), Stefan Monnier wrote:
> > Some packages will stay the same for years, but in the past week
> > I can see four occasions when changes in list contents have occurred
> > on oldstable. So there's little similarity.
> 
> The question is not really whether "apt/lists" is similar to
> "apt/archives", but whether the content of "apt/lists" is appropriate
> for `/var/cache`.

I didn't bring up apt/archives, but my first words posted were:
"This may not answer your question".

> AFAIK nothing bad will happen to your system if you delete
> `/var/lib/apt/lists`.  It will still run as before and you can still
> install packages as before.

As you said.

> The only thing that seems to be impacted is
> that some APT operations will temporarily work in a kind of "degraded"
> mode until the next `apt update`.

As I said.

On how significant that difference is, we beg to differ.

> That seems similar to things like `locate` failing if you remove
> `/var/log/plocate/plocate.db` (until that DB is rebuilt).

It's tricky to discern your point as /var/log/ is not involved.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#264869

>> That seems similar to things like `locate` failing if you remove
>> `/var/log/plocate/plocate.db` (until that DB is rebuilt).
>
> It's tricky to discern your point as /var/log/ is not involved.

That was a typo.  it's `/var/cache/plocate/plocate.db`, sorry.


        Stefan

[toc] | [prev] | [next] | [standalone]

#264873

On Sun 17 Dec 2023 at 15:33:30 (-0500), Stefan Monnier wrote:
> >> That seems similar to things like `locate` failing if you remove
> >> `/var/log/plocate/plocate.db` (until that DB is rebuilt).
> >
> > It's tricky to discern your point as /var/log/ is not involved.
> 
> That was a typo.  it's `/var/cache/plocate/plocate.db`, sorry.

My plocate.db is in /var/lib/plocate/, as is bookworm's.
Is that changing in the future?

  $ ls -l /var/lib/*locate/
  /var/lib/mlocate/:
  total 19344
  -rw-r----- 1 root mlocate 19807421 Aug 28 07:43 mlocate.db

  /var/lib/plocate/:
  total 20632
  -rw-r--r-- 1 root root         183 Nov 17  2021 CACHEDIR.TAG
  -rw-r----- 1 root plocate 21121629 Dec 16 16:06 plocate.db
  $ 

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#264878

>> That was a typo.  it's `/var/cache/plocate/plocate.db`, sorry.
> My plocate.db is in /var/lib/plocate/, as is bookworm's.
> Is that changing in the future?

Hmm... I could swear that I saw it in /var/cache but every machine
I look at has it in /var/lib, indeed.
[ GNU locate puts its DB in `/var/cache/locate/`, apparently, OTOH.  ]

Anyway, thanks for all the suggestions.  From what I can tell, there is
no strong reason to put the lists in `/var/lib` rather than
`/var/cache`.  I suspect this is mostly a historical accident (at some
point, erasing `/var/lib/apt/lists` caused problems because that dir
wasn't automatically recreated if missing, so you'd have to `mkdir`
manually as well as create the lock inside of it).


        Stefan

[toc] | [prev] | [next] | [standalone]

#264842

On 16/12/2023 22:59, Stefan Monnier wrote:
> AFAICT, all of `/var/lib/apt/lists` is made of files fetched from
> repositories, which APT will re-fetch if missing.
> So, it sounds to me like it belongs in `/var/cache/apt/lists`, really.

APT running by a regular user is unable to write to /var/cache/apt. E.g. 
DNF creates cache in the user home directory in such cases. I like APT 
approach since (at least with default settings) search queries work 
without delay while repository data is being updated. (I am realizing 
that I may get stale results.)

So for APT lists are a bit more than just cache.

[toc] | [prev] | [next] | [standalone]

#264844

Max Nikulin [2023-12-17 09:10:29] wrote:
> On 16/12/2023 22:59, Stefan Monnier wrote:
>> AFAICT, all of `/var/lib/apt/lists` is made of files fetched from
>> repositories, which APT will re-fetch if missing.
>> So, it sounds to me like it belongs in `/var/cache/apt/lists`, really.
> APT running by a regular user is unable to write to /var/cache/apt.

But neither is it able to write to `/var/lib/apt/lists`, AFAICT.


        Stefan

[toc] | [prev] | [next] | [standalone]

#264886

> I would imagine that it's due to the FHS (Filesystem Hierarchy Standard)
> which defines what the various directories on a "typical Linux system" are
> for. "man hier", for example, tells me that:
>
> * /var/cache - Data cached for programs.
>
> * /var/lib - Variable state information for programs.

These leave open the question of what "data cached for the program" mean
and what "Variable state" means.

Another way to look at it is in terms of what it implies.
For me, `/var/lib` data is data that is important to keep, e.g. data you
definitely don't want to delete/lose without a very good reason, whereas
`/var/cache` is data that's handy to have but not indispensable because
we can reconstruct it (or something close enough) from other sources,
which means that it doesn't have to be included in backups and it can
be deleted if it strikes your fancy (typically when you're in urgent need
of a tiny bit more disk space).

> system, so apt's state is more like the state of the repositories.

But both the "files that contain the list of .deb files" and "the .deb
files" are part of the state of the repositories :-)

Or, stated in another way, this way to look at the problem degenerates
into philosophical questions.  So I think the question "does it have
to be included in the backup" is a better guide.


        Stefan

[toc] | [prev] | [next] | [standalone]

#264887

[Multipart message — attachments visible in raw view] — view raw

On Mon, Dec 18, 2023 at 09:05:22AM -0500, Stefan Monnier wrote:
> > I would imagine that it's due to the FHS (Filesystem Hierarchy Standard)
> > which defines what the various directories on a "typical Linux system" are
> > for. "man hier", for example, tells me that:
> >
> > * /var/cache - Data cached for programs.
> >
> > * /var/lib - Variable state information for programs.
> 
> These leave open the question of what "data cached for the program" mean
> and what "Variable state" means.
> 
> Another way to look at it is in terms of what it implies.
> For me, `/var/lib` data is data that is important to keep, e.g. data you
> definitely don't want to delete/lose without a very good reason, whereas
> `/var/cache` is data that's handy to have but not indispensable because
> we can reconstruct it (or something close enough) from other sources,
> which means that it doesn't have to be included in backups and it can
> be deleted if it strikes your fancy (typically when you're in urgent need
> of a tiny bit more disk space).

[...]

Most probably you are right, especially since TLDP only says you can
delete /var/cache's content safely across reboots [1]. The apt lists
will be regenerated after doing an apt-get update, so...

OTOH it'd be only a true-real-honestly-promised cache if every apt
command did an apt-get update whenever it found that file missing.

Cheers

[1] https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/var.html
-- 
t

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web