Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.rc > #390072 > unrolled thread

Bug#1078608: apt update silently leaves old index data

Back to article view | Back to linux.debian.bugs.rc

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1078608: apt update silently leaves old index data David Kalnischkies <david@kalnischkies.de> - 2025-04-17 23:10 +0200
    Bug#1078608: apt update silently leaves old index data Laurent Bigonville <bigon@debian.org> - 2025-04-18 11:40 +0200
      Bug#1078608: apt update silently leaves old index data David Kalnischkies <david@kalnischkies.de> - 2025-04-18 13:30 +0200
        Bug#1078608: apt update silently leaves old index data Ben Hutchings <ben@decadent.org.uk> - 2025-05-18 00:40 +0200
          Bug#1078608: apt update silently leaves old index data Julian Andres Klode <jak@jak-linux.org> - 2025-05-18 09:50 +0200
            Processed: Re: Bug#1078608: apt update silently leaves old index data "Debian Bug Tracking System" <owner@bugs.debian.org> - 2025-05-18 09:50 +0200

#390072 — Bug#1078608: apt update silently leaves old index data

[Multipart message — attachments visible in raw view] — view raw

Am Wed, Apr 16, 2025 at 11:23:03AM +0200, schrieb Laurent Bigonville:
> On Tue, 13 Aug 2024 19:57:58 +0200 Sven Bartscher <kritzefitz@debian.org>
> wrote:
> > for a while I've been having the problem with `apt update` that it
> > pretends to complete successfully, but it didn't actually pull updated
> > index data. I run it like this:
> >
> > $ sudo apt-get update
> > [sudo] Passwort für sven:
> > OK:1 https://deb.debian.org/debian testing InRelease
> > OK:2 https://deb.debian.org/debian unstable InRelease
> > OK:3 https://deb.debian.org/debian experimental InRelease
> > OK:4 https://deb.debian.org/debian-debug testing-debug InRelease
> > OK:5 https://deb.debian.org/debian-debug unstable-debug InRelease
> > OK:6 https://deb.debian.org/debian-debug experimental-debug InRelease
> > Paketlisten werden gelesen… Fertig
> 
> I do have the same issue

You have what issue?

Some people have that "same issue" by incorrectly copying/restoring the
lists/ directory. Is that your issue?

On the surface, and with the output you present/quoted this is perfectly
normal and expected. apt will ask for a InRelease file first and if that
didn't change there is no point in asking for the other files: In the
best case the server will just say the files didn't change.
Alternatively, the server replies with newer/older files we have no
checksums for and would need to refuse anyhow.


> I enabled debugging in apt (Debug::Acquire::http=true) and I see the
> following:
>
> > GET /debian/dists/unstable/InRelease HTTP/1.1
[…]
> > If-Modified-Since: Wed, 16 Apr 2025 08:18:27 GMT
>
> > Answer for: http://ftp.be.debian.org/debian/dists/unstable/InRelease
> > HTTP/1.1 304 Not Modified
> > Date: Wed, 16 Apr 2025 09:07:23 GMT
> > Server: Apache/2.4.62 (Debian)
> > Last-Modified: Wed, 16 Apr 2025 08:18:27 GMT
[…]
> > -rw-r--r-- 1 root root   302323 15 avr 04:01
> > ftp.be.debian.org_debian_dists_unstable_contrib_binary-amd64_Packages

So this contrib is listed in
https://snapshot.debian.org/file/6c3f5892d4c6363f4b4f6ed076fcc8fd7cd120cd/Release
which is dated Tue, 15 Apr 2025 02:19:53 UTC.

Some of the other index files are dated later than that Release file
through, so probably more like this one:
https://snapshot.debian.org/file/c95522b0bfabf1e1f2277d36f12b3861ae9c1b0c/Release
which is dated Tue, 15 Apr 2025 14:13:09 UTC and also lists that file
unchanged (which btw also causes apt to not download that index
"again" if you update from the first to the second Release file).

The Release file you currently have could be
https://snapshot.debian.org/file/d47c8fc56da7f58c6ab18309c26b8337b76bfc4c/Release
dated Wed, 16 Apr 2025 08:17:02 UTC, that has a different size for that
file – the moment that Release file got downloaded apt should have also
downloaded the new version of that contrib file and "installed" them
together.

I picked that file at semi-random as contrib changes less often than
main stuff & I can look for the size as your config happens to have that
file being uncompressed. You also have Contents files, which are useless
in this lookup as they are lz4 compressed – good for usage, but that
compression only exists in storage, not in the Release file (which also
explains why apt can't just "recover" from this problem by checking
hashsums of the files it has in storage as in general it doesn't have
the hashes and just has to assume that what it has in storage is what
is listed in the old Release file it has in storage, too).


The interesting thing to discover now is what happened in these 24 hours
on your system that lists/ got a new Release file (or, well InRelease),
but not new indexes… unsurprisingly that shouldn't happen, but so far
nobody has provided any leads as people notice only after the fact and
at that point any debugging is pointless.


The only scenario I know of is disabling APT::Get::List-Cleanup and
running apt with a (slightly) different sources.list – like without
contrib. I am not quite sure how we could solve that scenario given
in some way its what the user requested; but not what they wanted.
I know some front ends might use that option. I suppose some users
could produce with interesting read permissions on configuration
combined with those front ends such a case by "accident".
In any case, I doubt that is what is happening here for "most" people
with the "same" issue, so I haven't thought about that too much yet.


Now, given you are in that situation, apt will at some point "recover"
from it, given that eventually InRelease will be updated and the files
it contains will eventually change, too. You 'only' forced that by
removing the InRelease file. You also could have removed the index…
(apt would have noticed that the file isn't there and downloads it
 even if you got a hit on the Release given we have to deal with config
 changes, like you adding another architecture since the last update
 but before the next mirror change).


> severity 1078608 serious

Not that it makes any practical difference in the apt team if you tag
it wishlist or critical, but I am curious: Which section in the Debian
policy is apt violating here? Or have I just missed you joining the
apt team and/or Debian Release team?
(See https://www.debian.org/Bugs/Developer#severities)
Some maintainers can get really angry if you use the wrong severities,
so ideally, next time, you should give a justification at least.


Best regards

David Kalnischkies

[toc] | [next] | [standalone]

#390102

[Multipart message — attachments visible in raw view] — view raw

Le 17/04/25 à 22:59, David Kalnischkies a écrit :
> Am Wed, Apr 16, 2025 at 11:23:03AM +0200, schrieb Laurent Bigonville:
>> On Tue, 13 Aug 2024 19:57:58 +0200 Sven Bartscher<kritzefitz@debian.org>
>> wrote:
>>> for a while I've been having the problem with `apt update` that it
>>> pretends to complete successfully, but it didn't actually pull updated
>>> index data. I run it like this:
>>>
>>> $ sudo apt-get update
>>> [sudo] Passwort für sven:
>>> OK:1https://deb.debian.org/debian testing InRelease
>>> OK:2https://deb.debian.org/debian unstable InRelease
>>> OK:3https://deb.debian.org/debian experimental InRelease
>>> OK:4https://deb.debian.org/debian-debug testing-debug InRelease
>>> OK:5https://deb.debian.org/debian-debug unstable-debug InRelease
>>> OK:6https://deb.debian.org/debian-debug experimental-debug InRelease
>>> Paketlisten werden gelesen… Fertig
>> I do have the same issue
> You have what issue?
>
> Some people have that "same issue" by incorrectly copying/restoring the
> lists/ directory. Is that your issue?

The fact that the indexes are not updated when running "apt-get update" 
meaning that apt is not seeing the updates leaving my machine not 
up-to-date.

I didn't manipulate the indexes by hand in any ways before to that.

> [...]
>
> The interesting thing to discover now is what happened in these 24 hours
> on your system that lists/ got a new Release file (or, well InRelease),
> but not new indexes… unsurprisingly that shouldn't happen, but so far
> nobody has provided any leads as people notice only after the fact and
> at that point any debugging is pointless.

It's a laptop with a desktop environment and PackageKit is installed, 
not sure whether PK could impact this. Also the laptop has been probably 
restarted in between so that means that PK has restarted and probably 
tried to update the indexes.


> [...]
>
>
>> severity 1078608 serious
> Not that it makes any practical difference in the apt team if you tag
> it wishlist or critical, but I am curious: Which section in the Debian
> policy is apt violating here? Or have I just missed you joining the
> apt team and/or Debian Release team?
> (Seehttps://www.debian.org/Bugs/Developer#severities)
> Some maintainers can get really angry if you use the wrong severities,
> so ideally, next time, you should give a justification at least.

Serious severity is:

>     is a severe violation of Debian policy (roughly, it violates a
>     "must" or "required" directive), or, in the package maintainer's
>     or release manager's opinion, makes the package unsuitable for
>     release. 

In my opinion, having trouble to update a system makes apt "unsuitable 
for release". And marking it "serious" makes this visible to the release 
manager team. But if you, as a maintainer, believe that apt can be 
shipped in the next stable version of debian with that bug open, please 
tell me and I'll reduce the severity.

I personally get "really angry" if I have packages (with potential 
security issues) not being timely updated, everybody has their quirk I 
guess...

[toc] | [prev] | [next] | [standalone]

#390119

[Multipart message — attachments visible in raw view] — view raw

Am Fri, Apr 18, 2025 at 11:30:12AM +0200, schrieb Laurent Bigonville:
> > The interesting thing to discover now is what happened in these 24 hours
> > on your system that lists/ got a new Release file (or, well InRelease),
> > but not new indexes… unsurprisingly that shouldn't happen, but so far
> > nobody has provided any leads as people notice only after the fact and
> > at that point any debugging is pointless.
> 
> It's a laptop with a desktop environment and PackageKit is installed, not
> sure whether PK could impact this. Also the laptop has been probably
> restarted in between so that means that PK has restarted and probably tried
> to update the indexes.

Well, while all front ends do share code and logic via libapt, there is
always the possibility of the front end holding it wrong. And there is
never an easy tell which config they are using. (At least I don't know…)

Anecdotal evidence suggests its some front end as I don't use any and
don't have that problem, while initial reporter here claim they have it
only on one system – while all likely have apt installed, so a general
problem would appear in general… of course, that is a rather weak
approach especially as this is both usually a silent issue and
a self-healing one, but its all we have so far…
Or if you will: +moreinfo +unreproducible.


> > > severity 1078608 serious
> > Not that it makes any practical difference in the apt team if you tag
> > it wishlist or critical, but I am curious: Which section in the Debian
> > policy is apt violating here? Or have I just missed you joining the
> > apt team and/or Debian Release team?
> > (Seehttps://www.debian.org/Bugs/Developer#severities)
> > Some maintainers can get really angry if you use the wrong severities,
> > so ideally, next time, you should give a justification at least.
> 
> Serious severity is:
> 
> >     is a severe violation of Debian policy (roughly, it violates a
> >     "must" or "required" directive), or, in the package maintainer's
> >     or release manager's opinion, makes the package unsuitable for
> >     release.
> 
> In my opinion, having trouble to update a system makes apt "unsuitable for
> release". And marking it "serious" makes this visible to the release manager
> team.

If I tagged every bug serious I thought makes something unsuitable for
release or that I want other people to look at… but as you quoted
my own opinion is irrelevant for the label 'serious' by design and
it shouldn't be misused as "summoning magic" either…

Tag it 'grave' if that is your reasoning (and it makes you feel better),
but as already said at least here it doesn't really matter as we have
a 1000+ open bugs against apt at least someone believes is serious in
their opinion. Does it change anything? Nope.

All it really does is that testers upgrading from stable will be scared
by apt-listbugs in a sense having probably more negative impact on the
release than this bug seems to be able to. In exchange, no magic army of
coders appears that fixes bugs nor does anyone provide any meaningful
details based on severity.
For all we know, as we know nothing, apt/oldstable has that problem, too,
(assuming of course its an apt problem to begin with) making that even
more ironic.

But it makes you feel better and I don't really care, so its fine.
I was indeed just giving you a hint for next time on another package
to include a justification rather than treat it as "obviously so".


Best regards

David Kalnischkies

[toc] | [prev] | [next] | [standalone]

#392023

[Multipart message — attachments visible in raw view] — view raw

On Sun, 2025-04-27 at 16:18 +0200, Ben Hutchings wrote:
[...]
> - If I mask packagekit.service before booting (using a rescue shell), 
>   then none of the files are updated automatically.  Running 'apt
>   update' updates them all as expected.
> 
> So it seems like this problem may be specific to PackageKit.
> 
> Sven and Laurent, do your affected systems have PackageKit installed?

OK, so I think this is confirmed as somehow PackageKit-related.

I had a look through the code for "apt update" and the PackageKit APT
back-end to see what might be different.  I think this has something to
do with the different pkgAcquireStatus subclasses they use, but I
couldn't identify a specific bug in PackageKit.

I experimented with writing a test program that implements its own
pkgAcquireStatus, and I I'm attaching the source for that.  It needs to
be run on a system where a local InRelease file is out of date.  If you
answer "no" to the Pulse() after the *second* time the InRelease file is
reported done, that should reproduce the broken state.

(The default answers are set for my VM snapshot which now has very
outdated InRelease files, so I know that the Pulse() after a
ReleaseInfoChanges() is the place to stop.)

Ben.

-- 
Ben Hutchings
If at first you don't succeed, you're doing about average.

[toc] | [prev] | [next] | [standalone]

#392027

Control: tag -1 confirmed

On 18 May 2025 00:37:44 CEST, Ben Hutchings <ben@decadent.org.uk> wrote:
>On Sun, 2025-04-27 at 16:18 +0200, Ben Hutchings wrote:
>[...]
>> - If I mask packagekit.service before booting (using a rescue shell), 
>>   then none of the files are updated automatically.  Running 'apt
>>   update' updates them all as expected.
>> 
>> So it seems like this problem may be specific to PackageKit.
>> 
>> Sven and Laurent, do your affected systems have PackageKit installed?
>
>OK, so I think this is confirmed as somehow PackageKit-related.
>
>I had a look through the code for "apt update" and the PackageKit APT
>back-end to see what might be different.  I think this has something to
>do with the different pkgAcquireStatus subclasses they use, but I
>couldn't identify a specific bug in PackageKit.
>
>I experimented with writing a test program that implements its own
>pkgAcquireStatus, and I I'm attaching the source for that.  It needs to
>be run on a system where a local InRelease file is out of date.  If you
>answer "no" to the Pulse() after the *second* time the InRelease file is
>reported done, that should reproduce the broken state.

Thanks, yes. I had a quick look at the code on Salsa from my phone:

So when the fetcher stops (Pulse()=false cancels the update run), it runs the Finished method on each item.

When the Finished() method of an InRelease file is called, it commits the transaction if it has started and there were no errors; 
that is, it does not check the transaction was actually complete.

-- 
sent from my phone, excuse the brevity, if any

[toc] | [prev] | [next] | [standalone]

#392028 — Processed: Re: Bug#1078608: apt update silently leaves old index data

Processing control commands:

> tag -1 confirmed
Bug #1078608 [apt] apt update silently leaves old index data
Added tag(s) confirmed.

-- 
1078608: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1078608
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.rc

csiph-web