Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1048646 > unrolled thread

Bug#982984: Mirror blocked due to repeated errors

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#982984: Mirror blocked due to repeated errors Russ Allbery <rra@debian.org> - 2021-02-17 19:30 +0100
    Bug#982984:  Marcos Raúl Carot <marcos.carot@gmail.com> - 2021-02-19 05:10 +0100
      Bug#982984: Mirror blocked due to repeated errors Michael Tokarev <mjt@tls.msk.ru> - 2021-02-20 22:30 +0100
        Bug#982984: Mirror blocked due to repeated errors Eduard Bloch <edi@gmx.de> - 2021-02-20 23:00 +0100
          Bug#982984: Mirror blocked due to repeated errors Michael Tokarev <mjt@tls.msk.ru> - 2021-02-21 10:40 +0100
    Bug#982984: Mirror blocked due to repeated errors Sven Geuer <debmaint@g-e-u-e-r.de> - 2021-02-21 00:00 +0100
      Bug#982984: Mirror blocked due to repeated errors Eduard Bloch <edi@gmx.de> - 2021-02-21 11:10 +0100
        Bug#982984: Mirror blocked due to repeated errors Eduard Bloch <edi@gmx.de> - 2021-02-22 00:50 +0100
          Bug#982984: Mirror blocked due to repeated errors Michael Tokarev <mjt@tls.msk.ru> - 2021-02-22 07:30 +0100
          Bug#982984: Mirror blocked due to repeated errors Sven Geuer <debmaint@g-e-u-e-r.de> - 2021-02-22 19:40 +0100
    Bug#982984: strace sent Steven De Herdt <steven.deherdt@gmail.com> - 2021-02-21 12:30 +0100
      Bug#982984: strace sent Sven Geuer <debmaint@g-e-u-e-r.de> - 2021-02-21 14:40 +0100

#1048646 — Bug#982984: Mirror blocked due to repeated errors

Package: apt-cacher-ng
Version: 3.6-1
Severity: normal

I have a fairly generic apt-cacher-ng configuration that I use for
sbuild.  (The only configuration I've changed is to add BindAddress:
localhost and change the Debian backend to https://deb.debian.org/debian/)
Relatively recently I've noticed sbuild update failing with some
regularity with the following error:

source:buster-i386-sbuild: Performing update.
Err:1 http://localhost:3142/debian buster InRelease
  502  Mirror blocked due to repeated errors [IP: ::1 3142]

Restarting apt-cacher-ng clears up this problem and then sbuild update
works as normal.  This might (or might not; it happens a few minutes
earlier than this error) be correlated with the following error message:

Feb 17 09:17:16 gwaihir apt-cacher-ng[833]: [warn] Call to getaddrinfo_async with no evdns_base configured.
Feb 17 09:17:19 gwaihir apt-cacher-ng[833]: [warn] Call to getaddrinfo_async with no evdns_base configured.
Feb 17 09:17:21 gwaihir apt-cacher-ng[833]: [warn] Call to getaddrinfo_async with no evdns_base configured.

It's more strongly correlated with this error that's logged in
/var/log/apt-cacher-ng/apt-cacher.err:

Mon Feb 15 09:19:50 2021|Failure to create replacement of debrep/dists/stretch/InRelease - CHECK FOLDER PERMISSIONS!
Mon Feb 15 09:19:50 2021|Failure to create replacement of debrep/dists/stretch/Release - CHECK FOLDER PERMISSIONS!

There doesn't seem to be anything odd about the permissions of those
directories under /var/cache/apt-cacher-ng.  They're all owned by
apt-cacher-ng:apt-cacher-ng and are mode 2755.

Probably related, temporary working files seem to be left behind in
/var/cache/apt-cacher-ng/debrep/dists sometimes (but not for every
occurence of this error):

buster:
InRelease  InRelease.head  InRelease1613585303  contrib/  main/  non-free/

sid:
InRelease  InRelease.head  contrib/  main/  non-free/

These files, when they appear, are zero-length.

-- Package-specific info:

-- System Information:
Debian Release: bullseye/sid
  APT prefers unstable
  APT policy: (990, 'unstable'), (500, 'unstable-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-3-amd64 (SMP w/12 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages apt-cacher-ng depends on:
ii  adduser                  3.118
ii  debconf [debconf-2.0]    1.5.74
ii  dpkg                     1.20.7.1
ii  libbz2-1.0               1.0.8-4
ii  libc6                    2.31-9
ii  libevent-2.1-7           2.1.12-stable-1
ii  libevent-pthreads-2.1-7  2.1.12-stable-1
ii  libgcc-s1                10.2.1-6
ii  liblzma5                 5.2.5-1.0
ii  libssl1.1                1.1.1i-3
ii  libstdc++6               10.2.1-6
ii  libsystemd0              247.3-1
ii  libwrap0                 7.6.q-31
ii  lsb-base                 11.1.0
ii  zlib1g                   1:1.2.11.dfsg-2

Versions of packages apt-cacher-ng recommends:
ii  ca-certificates  20210119

Versions of packages apt-cacher-ng suggests:
pn  avahi-daemon  <none>
pn  doc-base      <none>
ii  libfuse2      2.9.9-3

-- Configuration Files:
/etc/apt-cacher-ng/security.conf [Errno 13] Permission denied: '/etc/apt-cacher-ng/security.conf'

-- debconf information:
  apt-cacher-ng/cachedir: keep
  apt-cacher-ng/proxy: keep
  apt-cacher-ng/port: keep
  apt-cacher-ng/bindaddress: keep
* apt-cacher-ng/tunnelenable: false
  apt-cacher-ng/gentargetmode: No automated setup

[toc] | [next] | [standalone]

#1048864 — Bug#982984:

[Multipart message — attachments visible in raw view] — view raw

I am seeing the same error since the upgrade to apt-cacher-ng 3.6-1
yesterday.

Also a very basic setup, my only change is the location of the cache, which
has been working fine for years.

Happy to provide more info, just tell me what you need.

Cheers,

-- 
Marcos R Carot

[toc] | [prev] | [next] | [standalone]

#1049130

Getting these errors here too, restart of apt-cacher-ng
does NOT help in my case (so I'm basically unable to use apt
anymore, will have to drop usage of the cacher).

The log contains this message for every access to the cache:

  apt-cacher-ng[xxx]: [warn] Call to getaddrinfo_async with no evdns_base configured.

/mjt

[toc] | [prev] | [next] | [standalone]

#1049135

Hallo,
* Michael Tokarev [Sun, Feb 21 2021, 12:16:06AM]:
> Getting these errors here too, restart of apt-cacher-ng
> does NOT help in my case (so I'm basically unable to use apt
> anymore, will have to drop usage of the cacher).
>
> The log contains this message for every access to the cache:
>
>  apt-cacher-ng[xxx]: [warn] Call to getaddrinfo_async with no evdns_base configured.

I tried to reproduce it but couldn't so far. I admit that there is no
error handling for the failing creation of evdns_base, though. And I have a
strange feeling that it might be connected to
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983006 .

Do you all have apparmor-profiles-extra installed?

Best regards,
Eduard.

[toc] | [prev] | [next] | [standalone]

#1049172

21.02.2021 00:46, Eduard Bloch wrote:
[..]
>>   apt-cacher-ng[xxx]: [warn] Call to getaddrinfo_async with no evdns_base configured.
> 
> I tried to reproduce it but couldn't so far. I admit that there is no
> error handling for the failing creation of evdns_base, though. And I have a
> strange feeling that it might be connected to
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983006 .
> 
> Do you all have apparmor-profiles-extra installed?

No, I don't have any additional restrictions besides regular unix controls -
neither selinux or apparmor nor anything else. I'd notice if it were logged
around too.

Thanks,

/mjt

[toc] | [prev] | [next] | [standalone]

#1049139

Package: apt-cacher-ng
Version: 3.6-1
Followup-For: Bug #982984

same issue here since upgrade to apt-cacher-ng 3.6-1. Packages apparmor-
profiles or apparmor-profiles-extra are not installed.

Sven



-- Package-specific info:

-- System Information:
Debian Release: bullseye/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-3-amd64 (SMP w/8 CPU threads)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages apt-cacher-ng depends on:
ii  adduser                  3.118
ii  debconf [debconf-2.0]    1.5.74
ii  dpkg                     1.20.7.1
ii  libbz2-1.0               1.0.8-4
ii  libc6                    2.31-9
ii  libevent-2.1-7           2.1.12-stable-1
ii  libevent-pthreads-2.1-7  2.1.12-stable-1
ii  libgcc-s1                10.2.1-6
ii  liblzma5                 5.2.5-1.0
ii  libssl1.1                1.1.1i-3
ii  libstdc++6               10.2.1-6
ii  libsystemd0              247.3-1
ii  libwrap0                 7.6.q-31
ii  lsb-base                 11.1.0
ii  zlib1g                   1:1.2.11.dfsg-2

Versions of packages apt-cacher-ng recommends:
ii  ca-certificates  20210119

Versions of packages apt-cacher-ng suggests:
ii  avahi-daemon  0.8-5
pn  doc-base      <none>
ii  libfuse2      2.9.9-3

-- Configuration Files:
/etc/apt-cacher-ng/security.conf [Errno 13] Keine Berechtigung: '/etc/apt-cacher-ng/security.conf'

-- debconf information:
* apt-cacher-ng/cachedir: keep
* apt-cacher-ng/proxy: keep
* apt-cacher-ng/gentargetmode: Set up now and update later
* apt-cacher-ng/port: keep
* apt-cacher-ng/bindaddress: keep
* apt-cacher-ng/tunnelenable: true

[toc] | [prev] | [next] | [standalone]

#1049176

Hallo,
* Sven Geuer [Sat, Feb 20 2021, 11:43:19PM]:
> Package: apt-cacher-ng
> Version: 3.6-1
> Followup-For: Bug #982984
>
> same issue here since upgrade to apt-cacher-ng 3.6-1. Packages apparmor-
> profiles or apparmor-profiles-extra are not installed.

Ok, so I think we have some kind of strange libevent bug here. Could
someone please run:

strace -s 4000 -o log.strace -f /usr/lib/apt-cacher-ng/acngtool curl http://www.debian.org

And send me that log.strace file in private. Maybe also your
resolv.conf file, thanks.

Best regards,
Eduard.

[toc] | [prev] | [next] | [standalone]

#1049285

Hallo,
* Eduard Bloch [Sun, Feb 21 2021, 10:59:14AM]:

> > same issue here since upgrade to apt-cacher-ng 3.6-1. Packages apparmor-
> > profiles or apparmor-profiles-extra are not installed.
>
> Ok, so I think we have some kind of strange libevent bug here. Could
> someone please run:
>
> strace -s 4000 -o log.strace -f /usr/lib/apt-cacher-ng/acngtool curl http://www.debian.org

Okay guys, thank you very much for the contributed traces and other
remarks, this helped to track down about two and half real or potential
bugs.

I hope I have identified all root causes for the problems in Russel's
original report, some were specific to evdns (best reproduced in
Michael's setup) which are IMHO bugs in libevent (worth another
bug report there when I have more evidence) and some could be attributed
to a file descriptor introduced in version 3.6 and some are caused by
incorrect (and actually obsolete) algorithms on cache item handling.
I might also have found a reason for hanging acngtool cronjobs along the
way (another bug report). Check
https://salsa.debian.org/blade/apt-cacher-ng/-/commits/upstream/sid if
you want to see the longer story.

If you don't mind, I would kindly ask you to test a snapshot of that,
built locally from debian/sid branch ("fakeroot debian/rules binary") or
check the snapshot builds for amd64 at
https://www.unix-ag.uni-kl.de/~bloch/acng/snap3.6.1/ . If this is still
failing, it would be good to have a similar strace log but attaching to
the daemon process, like:
strace -s 4000 -o log.strace -f -p $(pidof apt-cacher-ng)
and then rerunning the failing operation. It might also make sense to
make another run after taking a break on client activity for 30s or more
in order to trigger the specific behavior leading to this bug.

It would be good to receive some feedback by Wednesday, then I am
planing to approach the release team (expecting the "usual" messy
discussion).

Best regards,
Eduard.

[toc] | [prev] | [next] | [standalone]

#1049302

22.02.2021 02:43, Eduard Bloch wrote:

> I hope I have identified all root causes for the problems in Russel's
> original report, some were specific to evdns (best reproduced in
> Michael's setup) which are IMHO bugs in libevent (worth another
> bug report there when I have more evidence) and some could be attributed
> to a file descriptor introduced in version 3.6 and some are caused by
> incorrect (and actually obsolete) algorithms on cache item handling.

btw, did you try my (trivial) event lib and udns? :) I wrote the event
lib after trying to use libevent many years ago and finding it is somewhat
huge and illogical.. :)

> https://www.unix-ag.uni-kl.de/~bloch/acng/snap3.6.1/ . If this is still

Nope, this one does not fail anymore, it works as it should.

Thanks,

/mjt

[toc] | [prev] | [next] | [standalone]

#1049434

[Multipart message — attachments visible in raw view] — view raw

Hey Eduard,

On Mon, 2021-02-22 at 00:43 +0100, Eduard Bloch wrote:
[...]
> It would be good to receive some feedback by Wednesday, then I am
> planing to approach the release team (expecting the "usual" messy
> discussion).
> 
> Best regards,
> Eduard.
> 

Building apt-cacher-ng cloned from
ssh://git@salsa.debian.org/blade/apt-cacher-ng failed in my environment
(gbp-buildpackage) due to the upstream/3.6.1 tag being missing. But
this is a different story.

Snapshot build downloaded from
https://www.unix-ag.uni-kl.de/~bloch/acng/snap3.6.1/
works for me.

Cheers,
Sven

-- 
GPG Fingerprint
3DF5 E8AA 43FC 9FDF D086 F195 ADF5 0EDA F8AD D585

[toc] | [prev] | [next] | [standalone]

#1049189 — Bug#982984: strace sent

Hello folks

I just sent Eduard the requested strace file.  Hope that helps...

Kind regards
-Steven

[toc] | [prev] | [next] | [standalone]

#1049213 — Bug#982984: strace sent

[Multipart message — attachments visible in raw view] — view raw

Hi all,

On Sun, 2021-02-21 at 12:24 +0100, Steven De Herdt wrote:
> Hello folks
> 
> I just sent Eduard the requested strace file.  Hope that helps...
> 
> Kind regards
> -Steven
> 

Also sent my strace.

Sven

-- 
GPG Fingerprint
3DF5 E8AA 43FC 9FDF D086 F195 ADF5 0EDA F8AD D585

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web