Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #247233 > unrolled thread

What do folks use to mirror repositories

Back to article view | Back to linux.debian.user

Contents

  What do folks use to mirror repositories Sam <sam@email-scan.com> - 2022-04-15 13:20 +0200
    Re: What do folks use to mirror repositories <tomas@tuxteam.de> - 2022-04-15 13:40 +0200
      Re: What do folks use to mirror repositories Roberto C. Sánchez <roberto@debian.org> - 2022-04-15 14:30 +0200
      Re: What do folks use to mirror repositories David Christensen <dpchrist@holgerdanske.com> - 2022-04-16 04:10 +0200
        Re: What do folks use to mirror repositories Charles Curley <charlescurley@charlescurley.com> - 2022-04-16 06:20 +0200
          Re: What do folks use to mirror repositories Celejar <celejar@gmail.com> - 2022-04-19 03:10 +0200
            Re: What do folks use to mirror repositories Charles Curley <charlescurley@charlescurley.com> - 2022-04-19 05:10 +0200
            Re: What do folks use to mirror repositories <tomas@tuxteam.de> - 2022-04-19 07:30 +0200
              Re: What do folks use to mirror repositories David Wright <deblis@lionunicorn.co.uk> - 2022-04-19 18:10 +0200
                Re: What do folks use to mirror repositories Charles Curley <charlescurley@charlescurley.com> - 2022-04-19 19:00 +0200
                  Re: What do folks use to mirror repositories Polyna-Maude Racicot-Summerside <debian@polynamaude.com> - 2022-04-19 19:10 +0200
                  Re: What do folks use to mirror repositories Celejar <celejar@gmail.com> - 2022-04-20 01:50 +0200
            Re: What do folks use to mirror repositories Tim Woodall <debianuser@woodall.me.uk> - 2022-04-19 20:20 +0200
    Re: What do folks use to mirror repositories "Andrew M.A. Cater" <amacater@einval.com> - 2022-04-15 14:30 +0200
    Re: What do folks use to mirror repositories Nicholas Geovanis <nickgeovanis@gmail.com> - 2022-04-15 17:20 +0200
    Re: What do folks use to mirror repositories David Wright <deblis@lionunicorn.co.uk> - 2022-04-15 18:40 +0200
    Re: What do folks use to mirror repositories Dieter Rohlfing <dr-erc@gmx.net> - 2022-04-16 11:30 +0200

#247233 — What do folks use to mirror repositories

[Multipart message — attachments visible in raw view] — view raw

That's pretty much it. I want to mirror all my updates to a single server on  
my LAN and have everything on my LAN apt update from it. This seems more  
efficient than having everyone download their own copies.

Google told me to use apt-mirror. My updates repository turned out to have  
some metadata filename with an @ in it, apt-mirror refuses to do anything  
with this state of affairs. I learned (after poking apt-mirror's page( that  
this bug was two years old, but the project is not maintained any more.  
Digging through aptly-mirror's bug tracker I caught a reference to aptly.

I got aptly to work, after applying a reasonable amount of elbow grease. The  
only unsolvable problem was that after taking a few seconds to download a  
handful of updates it takes aptly well over an hour to republish the local  
repository snapshot, to incorporate those updates. The full repository adds  
up to about 18000 packages, and looks like aptly recompresses each one of  
them, every time. I don't know why but that's what it does. I see it  
launching xz over and over again, while aptly keeps me in the loop of how  
many of those 18000 packages are left to process.

Although I don't need to worry about wearing any SSDs (since it's spinning  
rust), this is still somewhat suboptimal. So, that brings me to: what do  
folks use to mirror repositories?

[toc] | [next] | [standalone]

#247234

[Multipart message — attachments visible in raw view] — view raw

On Fri, Apr 15, 2022 at 07:03:14AM -0400, Sam wrote:
> That's pretty much it. I want to mirror all my updates to a single server on
> my LAN and have everything on my LAN apt update from it. This seems more
> efficient than having everyone download their own copies.

If all you are looking for is efficiency, you might also consider an
apt cache (I'm pretty happy with apt-cacher-ng). It is pretty low
maintenance, as it decides itself when to throw out older entries.

The big plus for me is that it can cache across multiple repos (I
sometimes build "old" images from archives.d.o for some legacy
hardware hidden away in some customer's closets, don't ask ;-)
It just does so silently. Whenever the package isn't there, it
fetches it, next time it's served from the cache.

Another nice point is that you can just keep your /etc/apt/sources.list
as it should be (with the "real" repo addresses out there) and the
apt cache works as a proxy. An entry in (e.g.) /etc/apt/apt.conf.d/02cache
like so:

  # /etc/apt/apt.conf.d/02cache
  #
  Acquire::http::proxy "http://localhost:3142"

(or whatever your apt cache's URL is) suffices -- something you can
easily disable or remove should you ever "release" your pet machine
"into the wild" :-)

> Google told me to use apt-mirror [...]

A mirror has different characteristics from a cache. You decide
yourself which packages go in. This may be an advantage, depending
on your use case -- or a disadvantage.

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#247240

On Fri, Apr 15, 2022 at 01:36:43PM +0200, tomas@tuxteam.de wrote:
> On Fri, Apr 15, 2022 at 07:03:14AM -0400, Sam wrote:
> > That's pretty much it. I want to mirror all my updates to a single server on
> > my LAN and have everything on my LAN apt update from it. This seems more
> > efficient than having everyone download their own copies.
> 
> If all you are looking for is efficiency, you might also consider an
> apt cache (I'm pretty happy with apt-cacher-ng). It is pretty low
> maintenance, as it decides itself when to throw out older entries.
> 
> The big plus for me is that it can cache across multiple repos (I
> sometimes build "old" images from archives.d.o for some legacy
> hardware hidden away in some customer's closets, don't ask ;-)
> It just does so silently. Whenever the package isn't there, it
> fetches it, next time it's served from the cache.
> 
> Another nice point is that you can just keep your /etc/apt/sources.list
> as it should be (with the "real" repo addresses out there) and the
> apt cache works as a proxy. An entry in (e.g.) /etc/apt/apt.conf.d/02cache
> like so:
> 
>   # /etc/apt/apt.conf.d/02cache
>   #
>   Acquire::http::proxy "http://localhost:3142"
> 
> (or whatever your apt cache's URL is) suffices -- something you can
> easily disable or remove should you ever "release" your pet machine
> "into the wild" :-)
> 
I very much agree with Tomas.  After having tried a few different
solutions years ago, I have settled on this one and I am convinced it is
still the best available solution to this particular problem.

Regards,

-Roberto

-- 
Roberto C. Sánchez

[toc] | [prev] | [next] | [standalone]

#247271

On 4/15/22 04:36, tomas@tuxteam.de wrote:
> On Fri, Apr 15, 2022 at 07:03:14AM -0400, Sam wrote:
>> That's pretty much it. I want to mirror all my updates to a single server on
>> my LAN and have everything on my LAN apt update from it. This seems more
>> efficient than having everyone download their own copies.
> 
> If all you are looking for is efficiency, you might also consider an
> apt cache (I'm pretty happy with apt-cacher-ng). It is pretty low
> maintenance, as it decides itself when to throw out older entries.
> 
> The big plus for me is that it can cache across multiple repos (I
> sometimes build "old" images from archives.d.o for some legacy
> hardware hidden away in some customer's closets, don't ask ;-)
> It just does so silently. Whenever the package isn't there, it
> fetches it, next time it's served from the cache.
> 
> Another nice point is that you can just keep your /etc/apt/sources.list
> as it should be (with the "real" repo addresses out there) and the
> apt cache works as a proxy. An entry in (e.g.) /etc/apt/apt.conf.d/02cache
> like so:
> 
>    # /etc/apt/apt.conf.d/02cache
>    #
>    Acquire::http::proxy "http://localhost:3142"
> 
> (or whatever your apt cache's URL is) suffices -- something you can
> easily disable or remove should you ever "release" your pet machine
> "into the wild" :-)
> 
>> Google told me to use apt-mirror [...]
> 
> A mirror has different characteristics from a cache. You decide
> yourself which packages go in. This may be an advantage, depending
> on your use case -- or a disadvantage.


When I had several Debian machines in the past, I used approx(8) [2]. 
It was good software -- KISS, properly documented, easy to set up, 
worked efficiently, and never failed.


But approx(8) had some operational features that apt-cacher-ng(8) [3] 
seems to improve upon:

1.  Each host needed its apt(8) [1] configuration to point at the 
approx(8) server.

2.  The approx(8) server needed its configuration to point at the remote 
repositories.

3.  I seem to recall that I was able to use approx(8) with non-Debian 
repositories via creative configuration.

4.  The approx(8) cache needed to be thinned periodically by running 
approx-gc(8) [2].


David


References:

[1] https://manpages.debian.org/buster/apt/apt.8.en.html

[2]  https://manpages.debian.org/stretch/approx/approx.8.en.html

[3] 
https://manpages.debian.org/unstable/apt-cacher-ng/apt-cacher-ng.8.en.html

[toc] | [prev] | [next] | [standalone]

#247277

On Fri, 15 Apr 2022 19:06:35 -0700
David Christensen <dpchrist@holgerdanske.com> wrote:

> When I had several Debian machines in the past, I used approx(8) [2]. 
> It was good software -- KISS, properly documented, easy to set up, 
> worked efficiently, and never failed.
> 
> 
> But approx(8) had some operational features that apt-cacher-ng(8) [3] 
> seems to improve upon:
> 
> 1.  Each host needed its apt(8) [1] configuration to point at the 
> approx(8) server.

Apt-cacher-ng (hereafter acng) also requires a change in client apt
configurations. Put one line into apt.conf or a one-liner in
apt.conf.d. I use the latter, 02proxy:

Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";

There are further wrinkles for laptops and other traveling computers,
and for https only repos, which I will leave as an exercise for the
student.

> 
> 3.  I seem to recall that I was able to use approx(8) with non-Debian 
> repositories via creative configuration.

acng does support some non-deb repos. For more details, see
/etc/apt-cacher-ng/acng.conf.

I think I'll stay with acng.


-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[toc] | [prev] | [next] | [standalone]

#247352

On Fri, 15 Apr 2022 22:15:36 -0600
Charles Curley <charlescurley@charlescurley.com> wrote:

...

> Apt-cacher-ng (hereafter acng) also requires a change in client apt
> configurations. Put one line into apt.conf or a one-liner in
> apt.conf.d. I use the latter, 02proxy:
> 
> Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";
> 
> There are further wrinkles for laptops and other traveling computers,
> and for https only repos, which I will leave as an exercise for the
> student.

Yes. I use apt-cacher-ng, but having to manually add a workaround for
every SSL-only repository I use is getting rather annoying:

https://wiki.debian.org/AptCacherNg#HTTPS_repositories
https://blog.packagecloud.io/using-apt-cacher-ng-with-ssl-tls/

One ends up with an ugly and not-very readable line like this:

PassThroughPattern: (winehq\.org|xpra\.org|vscodium\.com|packagecloud\.io):443$

-- 
Celejar

[toc] | [prev] | [next] | [standalone]

#247353

On Mon, 18 Apr 2022 21:08:08 -0400
Celejar <celejar@gmail.com> wrote:

> Yes. I use apt-cacher-ng, but having to manually add a workaround for
> every SSL-only repository I use is getting rather annoying:
> 
> https://wiki.debian.org/AptCacherNg#HTTPS_repositories
> https://blog.packagecloud.io/using-apt-cacher-ng-with-ssl-tls/

Yup. Or something like:

Acquire::https::Proxy::updates.example.com "DIRECT";

to bypass acng entirely.

Either one is kludgey.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[toc] | [prev] | [next] | [standalone]

#247355

[Multipart message — attachments visible in raw view] — view raw

On Mon, Apr 18, 2022 at 09:08:08PM -0400, Celejar wrote:
> On Fri, 15 Apr 2022 22:15:36 -0600
> Charles Curley <charlescurley@charlescurley.com> wrote:
> 
> ...
> 
> > Apt-cacher-ng (hereafter acng) also requires a change in client apt
> > configurations. Put one line into apt.conf or a one-liner in
> > apt.conf.d. I use the latter, 02proxy:
> > 
> > Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";
> > 
> > There are further wrinkles for laptops and other traveling computers,
> > and for https only repos, which I will leave as an exercise for the
> > student.
> 
> Yes. I use apt-cacher-ng, but having to manually add a workaround for
> every SSL-only repository I use is getting rather annoying:

That's true. Caching and HTTPS don't play well with each other. Why
a package repository has to be https-only is anyone's guess, but I
think apt caches should learn to do "man in the middle".

Cheers
-- 
t

[toc] | [prev] | [next] | [standalone]

#247360

On Tue 19 Apr 2022 at 07:24:53 (+0200), tomas@tuxteam.de wrote:
> On Mon, Apr 18, 2022 at 09:08:08PM -0400, Celejar wrote:
> > On Fri, 15 Apr 2022 22:15:36 -0600
> > Charles Curley <charlescurley@charlescurley.com> wrote:
> > 
> > ...
> > 
> > > Apt-cacher-ng (hereafter acng) also requires a change in client apt
> > > configurations. Put one line into apt.conf or a one-liner in
> > > apt.conf.d. I use the latter, 02proxy:
> > > 
> > > Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";
> > > 
> > > There are further wrinkles for laptops and other traveling computers,
> > > and for https only repos, which I will leave as an exercise for the
> > > student.
> > 
> > Yes. I use apt-cacher-ng, but having to manually add a workaround for
> > every SSL-only repository I use is getting rather annoying:
> 
> That's true. Caching and HTTPS don't play well with each other. Why
> a package repository has to be https-only is anyone's guess, but I
> think apt caches should learn to do "man in the middle".

As for laptops, I iron out one wrinkle by not setting my proxy under
/etc/apt/, but in the command lines instead. So the cron command has:
apt-get -qq -o Acquire::http::Proxy="…" update && apt-get -qq -d -o Acquire::http::Proxy="…" dist-upgrade && find /var/cache/apt/archives/ -name '*deb'
which generates an email whenever there are packages in the cache,
and I upgrade the system with:
apt-get -o Acquire::http::Proxy="…" update && apt-get -d -o Acquire::http::Proxy="…" dist-upgrade; apt-get upgrade; read -p 'Ctrl-C to avoid clean' _; apt-get clean

(I avoid cleaning the cache when packages are held back so that an
immediate   apt-get dist-upgrade   doesn't have to refetch them.)

Apart from making it easy to run the root command when not at home
(just rubout the -o option), this command line also works around
apt-cacher-ng's occasional inability to cope with wrinkles such
as expired certificates: when apt-listbugs runs during the upgrade,
it avoids going through the cache.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#247364

On Tue, 19 Apr 2022 11:02:26 -0500
David Wright <deblis@lionunicorn.co.uk> wrote:

> As for laptops,

I handle laptops a bit differently (that's Linux for you). My proxy
statement is in its own file in /etc/apt. When the laptop is home,
there's a symlink in /etc/apt/apt.conf.d. The symlink gets removed or
made by a script NetworkManager executes at various times:
/etc/NetworkManager/dispatcher.d/50ifupdown

Non-laptops get the same file and symlink. The just don't get
NetworkManager or that script.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[toc] | [prev] | [next] | [standalone]

#247365

I've tried many...
apt-cache
debmirror
ftpsync

I prefer debmirror

You can also use aptly but it will be a non original mirror.

On 2022-04-19 12:57, Charles Curley wrote:
> On Tue, 19 Apr 2022 11:02:26 -0500
> David Wright <deblis@lionunicorn.co.uk> wrote:
>
>> As for laptops,
>
> I handle laptops a bit differently (that's Linux for you). My proxy
> statement is in its own file in /etc/apt. When the laptop is home,
> there's a symlink in /etc/apt/apt.conf.d. The symlink gets removed or
> made by a script NetworkManager executes at various times:
> /etc/NetworkManager/dispatcher.d/50ifupdown
>
> Non-laptops get the same file and symlink. The just don't get
> NetworkManager or that script.
>

-- 
Polyna-Maude R.-Summerside
-Be smart, Be wise, Support opensource development

[toc] | [prev] | [next] | [standalone]

#247368

On Tue, 19 Apr 2022 10:57:38 -0600
Charles Curley <charlescurley@charlescurley.com> wrote:

> On Tue, 19 Apr 2022 11:02:26 -0500
> David Wright <deblis@lionunicorn.co.uk> wrote:
> 
> > As for laptops,
> 
> I handle laptops a bit differently (that's Linux for you). My proxy
> statement is in its own file in /etc/apt. When the laptop is home,
> there's a symlink in /etc/apt/apt.conf.d. The symlink gets removed or
> made by a script NetworkManager executes at various times:
> /etc/NetworkManager/dispatcher.d/50ifupdown
> 
> Non-laptops get the same file and symlink. The just don't get
> NetworkManager or that script.

I just bring up a wireguard tunnel to my LAN before updating /
upgrading.

-- 
Celejar

[toc] | [prev] | [next] | [standalone]

#247366

On Mon, 18 Apr 2022, Celejar wrote:

> On Fri, 15 Apr 2022 22:15:36 -0600
> Charles Curley <charlescurley@charlescurley.com> wrote:
>
> ...
>
>> Apt-cacher-ng (hereafter acng) also requires a change in client apt
>> configurations. Put one line into apt.conf or a one-liner in
>> apt.conf.d. I use the latter, 02proxy:
>>
>> Acquire::http::Proxy "http://aptcacherdeb.localdomain:3142";
>>
>> There are further wrinkles for laptops and other traveling computers,
>> and for https only repos, which I will leave as an exercise for the
>> student.
>
> Yes. I use apt-cacher-ng, but having to manually add a workaround for
> every SSL-only repository I use is getting rather annoying:
>
> https://wiki.debian.org/AptCacherNg#HTTPS_repositories
> https://blog.packagecloud.io/using-apt-cacher-ng-with-ssl-tls/
>
> One ends up with an ugly and not-very readable line like this:
>
> PassThroughPattern: (winehq\.org|xpra\.org|vscodium\.com|packagecloud\.io):443$
>
>

I haven't really thought about it but could you point at a 'fake' url
that then gets proxied to the real one.

My thought would be to use: http://http.ssl.only.repo/ in sources.list,
point that name at a local apache server and then forward to
https://ssl.only.repo/

I only suggest apache because I've done something like this before - I'd
guess there are much leaner solutions, even stunnel might work.

depending on name resolution it might have to be a suffix rather than a
prefix.

Tim.

[toc] | [prev] | [next] | [standalone]

#247238

On Fri, Apr 15, 2022 at 07:03:14AM -0400, Sam wrote:
> That's pretty much it. I want to mirror all my updates to a single server on
> my LAN and have everything on my LAN apt update from it. This seems more
> efficient than having everyone download their own copies.
> 
> Google told me to use apt-mirror. My updates repository turned out to have
> some metadata filename with an @ in it, apt-mirror refuses to do anything
> with this state of affairs. I learned (after poking apt-mirror's page( that
> this bug was two years old, but the project is not maintained any more.
> Digging through aptly-mirror's bug tracker I caught a reference to aptly.
> 

It depends: if you've got a lot of machines, you could do worse than ftpsync
 - which, despite it's name, uses mostly rsync - and Apache.

https://www.debian.org/mirror/ftpmirror.el.html tells you to use ftpsync or
debmirror

See, for example, my blog on Planet Debian from a while ago for fairly 
detailed instructions:

http://flosslinuxblog.blogspot.com/2020/02/rebuilding-mirror-software-mirroring-of.html

[ftpsync setup]

and 

http://flosslinuxblog.blogspot.com/2020/02/rebuilding-mirror-fedora-epel-mirroring.html

[which details the Apache setup]

I don't exclude architectures or distributions - so I have a full Debian
mirror with everything on (apart from anything served only by security.d.o).

If you _only_ need amd64 and _only_ need the current distribution, for
example, your mirror will be very significantly smaller.

> So, that brings me to: what do
> folks use to mirror repositories?
> 

Hope this helps.

With every good wish, as ever,

Andy Cater

[toc] | [prev] | [next] | [standalone]

#247249

[Multipart message — attachments visible in raw view] — view raw

On Fri, Apr 15, 2022 at 6:10 AM Sam <sam@email-scan.com> wrote:

> That's pretty much it. I want to mirror all my updates to a single server
> on
> my LAN and have everything on my LAN apt update from it. This seems more
> efficient than having everyone download their own copies.
>
> Google told me to use apt-mirror.

....

As Andrew mentions in his reply, rsync is a good choice for this. Or
something based on it. It's smarter about what doesn't need to be copied so
uses less aggregate bandwidth. Lots of use-cases around the data center :-)

[toc] | [prev] | [next] | [standalone]

#247254

On Fri 15 Apr 2022 at 07:03:14 (-0400), Sam wrote:
> That's pretty much it. I want to mirror all my updates to a single
> server on my LAN and have everything on my LAN apt update from it.
> This seems more efficient than having everyone download their own
> copies.

Same here, except that the "server" is just my main workstation,
so it tends to be first on in the morning and last off at night.

> I got aptly to work, after applying a reasonable amount of elbow
> grease. The only unsolvable problem was that after taking a few
> seconds to download a handful of updates it takes aptly well over an
> hour to republish the local repository snapshot, to incorporate those
> updates. The full repository adds up to about 18000 packages, and
> looks like aptly recompresses each one of them, every time. I don't
> know why but that's what it does. I see it launching xz over and over
> again, while aptly keeps me in the loop of how many of those 18000
> packages are left to process.
> 
> Although I don't need to worry about wearing any SSDs (since it's
> spinning rust), this is still somewhat suboptimal. So, that brings me
> to: what do folks use to mirror repositories?

Like some others, I've run apt-cacher-ng for nearly 10 years. It
is caching two Debian versions for two architectures at any one time,
except when Debian releases a new version, when a third version will
be added before the oldest starts expiring, so it swells and then
shrinks again. It pretty much runs itself, apart from a couple of odd
wrinkles that are of my own making.

Pros: it builds up the cache itself, as packages get installed. It
also expires packages itself, so you don't have to maintain it.
(The expiry parameters can be configured if required.)

At the moment, its size is 12GB, and it contains 6906 .deb files.
I see that I posted it occupied 13GB in 2018. (I don't run any
Desktop Environments.)

Cons: I've only met these when I've continued running apt-cacher-ng
on an oldstable machine for a lengthy period, so there's no need for
you to hit these snags at all.

1. Back when I was running apt-cacher-ng on wheezy, changes to
the Debian repository formats/methods in jessie prevented it from
expiring packages, so the cache kept growing. Answer: I installed
a wheezy-backports version of apt-cacher-ng until I upgraded the
machine to jessie.

2. Currently, if you run apt-cacher-ng on buster, /and/ you have
installed apt-listbugs on your machines, you may have to say no
when it asks whether to check for listed bugs. Or, you can update
the lists and upgrade download-only (which gets the package(s)
through the cache as desired), but perform the actual upgrade
without going through apt-cacher-ng (which costs no bandwidth
as the .deb files are on the local machine now). But, as I said,
there's no need to hit this snag if you run apt-cacher-ng on a
machine running stable (or testing/unstable etc, I guess).

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#247285

Am Fri, 15 Apr 2022 07:03:14 -0400
schrieb Sam <sam@email-scan.com>:

>That's pretty much it. I want to mirror all my updates to a single server on
>my LAN and have everything on my LAN apt update from it. This seems more
>efficient than having everyone download their own copies.

For mirroring a repository I recommend the package 'debmirror'.

Instead of mirroring a whole repository (e.g. for mirroring Debian11 you
need about 70 GB) you can select a subset of the repository with the
include and exclude options.

Dieter

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web