Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #182185 > unrolled thread

apt-cacher-ng related program and package suggestion

Back to article view | Back to linux.debian.user

Contents

  apt-cacher-ng related program and package suggestion "tuxderlinuxfuchs77@gmail.com" <tuxderlinuxfuchs77@gmail.com> - 2017-06-14 22:00 +0200
    Re: apt-cacher-ng related program and package suggestion Andy Smith <andy@strugglers.net> - 2017-06-15 05:00 +0200
      Re: apt-cacher-ng related program and package suggestion Yvan Masson <yvan.masson@openmailbox.org> - 2017-06-15 08:50 +0200
        Re: apt-cacher-ng related program and package suggestion Yvan Masson <yvan.masson@openmailbox.org> - 2017-06-16 09:40 +0200
    Re: apt-cacher-ng related program and package suggestion Dan Ritter <dsr@randomstring.org> - 2017-06-15 16:20 +0200
      Re: apt-cacher-ng related program and package suggestion Darac Marjal <mailinglist@darac.org.uk> - 2017-06-15 16:30 +0200
      Re: apt-cacher-ng related program and package suggestion David Wright <deblis@lionunicorn.co.uk> - 2017-06-15 16:40 +0200
        Re: apt-cacher-ng related program and package suggestion "tuxderlinuxfuchs77@gmail.com" <tuxderlinuxfuchs77@gmail.com> - 2017-06-15 16:50 +0200
          Re: apt-cacher-ng related program and package suggestion David Wright <deblis@lionunicorn.co.uk> - 2017-06-15 19:00 +0200
      Re: apt-cacher-ng related program and package suggestion Bob Weber <bobrweber@gmail.com> - 2017-06-15 16:50 +0200
        Re: apt-cacher-ng related program and package suggestion Dan Ritter <dsr@randomstring.org> - 2017-06-16 17:50 +0200
    Re: apt-cacher-ng related program and package suggestion Eduard Bloch <edi@gmx.de> - 2017-06-16 17:40 +0200

#182185 — apt-cacher-ng related program and package suggestion

[Multipart message — attachments visible in raw view] — view raw

Hello,

I have a suggestion for a client program/package in combination with
apt-cacher-ng

Consider following situation:

User U_A has a laptop which he uses at home and at university

He has a poor internet connection in both places.

So he set up apt-cacher-ng services in both places but they are on
different networks (he does not have permission to change the network
settings at university to make the IP addresses of the apt-cacher-ng
servers math)

Lets say at home the server HOST_1 is http://192.168.1.100:3142
At university: server: HOST_2: http://192.168.76.56:3142

He uses Ubuntu (so this should work for Debian in a similar way).

He makes a backup "/etc/apt/sources.list.bak" of /etc/apt/sources.list

At university he sets up the cache repo as follows (after overwriting
the /etc/apt/sources.list with a copy of the backup if it was changed to
use the home apt-cacher-ng)

sed -i 's/de.archive.ubuntu.com/192.168.76.56:3142/g' /etc/apt/sources.list

The same procedure at home.

The shown way is not very comfortable.

1. it is annoying having to remember IP addresses
2. sed command is error prone (consider typos, etc.)
3. copying around backup files is time consuming having to type in all
the paths
4. typing in sed command or searching it in bash history for re-using is
time consuming, too.

What I would like is to setup all cachers one use a program (lets call
it apt-cacher-switch for now)

apt-cacher-switch add <Host1_Name> <http://<IP1>:<PORT1>>
apt-cacher-switch add <Host2_Name> <http://<IP2>:<PORT2>>

Then at university:
apt-cacher-switch enable <Host1_Name>

At home:
apt-cacher-switch enable <Host2_Name>

In an environment where no cacher is on the network
apt-cacher-switch disable

The apt-cacher switch should manage creating and copying around the
required /etc/apt/sources.list files.

As I am not very talented in C/C++ programming under Linux, I am asking
the community to make it happen.
I would be very happy if this could be implemented and the package be
added to the official Debian and Ubuntu (and other derivates') repositories.

Best regards

Chris

[toc] | [next] | [standalone]

#182190

Hello,

On Wed, Jun 14, 2017 at 09:57:50PM +0200, tuxderlinuxfuchs77@gmail.com wrote:
> apt-cacher-switch add <Host1_Name> <http://<IP1>:<PORT1>>
> apt-cacher-switch add <Host2_Name> <http://<IP2>:<PORT2>>
> 
> Then at university:
> apt-cacher-switch enable <Host1_Name>
> 
> At home:
> apt-cacher-switch enable <Host2_Name>
> 
> In an environment where no cacher is on the network
> apt-cacher-switch disable

Seems like it would be easier to install squid-deb-proxy-client on
each client machine. This listens for Avahi announcements of
_apt_proxy._tcp to choose a proxy.

Normally your server(s) would run package squid-deb-proxy (an
install of squid customised for caching Debian packages) which takes
care of announcing its presence over Avahi, but you can make your
own announcements that point at your own installs of apt-cacher-ng
if you like.

If nothing is being announced then squid-deb-proxy-client goes
direct to Debian mirrors.

That way this all works without having to run any commands on the
client depending on where the client machine is located.

One downside is that if a broken proxy announces itself then your
clients will use it, i.e. you don't have any control over whether
the proxy is used or not.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#182192

[Multipart message — attachments visible in raw view] — view raw

Le 15/06/2017 à 04:51, Andy Smith a écrit :
> Hello,
> 
> On Wed, Jun 14, 2017 at 09:57:50PM +0200, tuxderlinuxfuchs77@gmail.com wrote:
>> apt-cacher-switch add <Host1_Name> <http://<IP1>:<PORT1>>
>> apt-cacher-switch add <Host2_Name> <http://<IP2>:<PORT2>>
>>
>> Then at university:
>> apt-cacher-switch enable <Host1_Name>
>>
>> At home:
>> apt-cacher-switch enable <Host2_Name>
>>
>> In an environment where no cacher is on the network
>> apt-cacher-switch disable
> 
> Seems like it would be easier to install squid-deb-proxy-client on
> each client machine. This listens for Avahi announcements of
> _apt_proxy._tcp to choose a proxy.
> 
> Normally your server(s) would run package squid-deb-proxy (an
> install of squid customised for caching Debian packages) which takes
> care of announcing its presence over Avahi, but you can make your
> own announcements that point at your own installs of apt-cacher-ng
> if you like.
> 
> If nothing is being announced then squid-deb-proxy-client goes
> direct to Debian mirrors.
> 
> That way this all works without having to run any commands on the
> client depending on where the client machine is located.
> 
> One downside is that if a broken proxy announces itself then your
> clients will use it, i.e. you don't have any control over whether
> the proxy is used or not.
> 
> Cheers,
> Andy
> 

Another solution whichi does not depend of the server software (squid or
apt-cacher-ng) is to use this directive on the client:
Acquire::http::Proxy-Auto-Detect

I could not find a good example in English, but here is one in French.
Obviously, man 5 apt.conf also explains it.

Regards,
Yvan

[toc] | [prev] | [next] | [standalone]

#182225

[Multipart message — attachments visible in raw view] — view raw

> Another solution whichi does not depend of the server software (squid or
> apt-cacher-ng) is to use this directive on the client:
> Acquire::http::Proxy-Auto-Detect
> 
> I could not find a good example in English, but here is one in French.
> Obviously, man 5 apt.conf also explains it.

Oups, forgot the link:
https://www.blog-libre.org/2016/01/09/configuration-automatique-du-proxy-pour-apt/

The script used to detect the proxy is only two lines…

Regards,
Yvan

[toc] | [prev] | [next] | [standalone]

#182195

On Wed, Jun 14, 2017 at 09:57:50PM +0200, tuxderlinuxfuchs77@gmail.com wrote:
> The shown way is not very comfortable.
> 
> 1. it is annoying having to remember IP addresses
> 2. sed command is error prone (consider typos, etc.)
> 3. copying around backup files is time consuming having to type in all
> the paths
> 4. typing in sed command or searching it in bash history for re-using is
> time consuming, too.
> 

Simple workaround for now:

/etc/apt/sources.list:
----
deb http://site1.ip.address/debian jessie-backports main
#deb http://site2.ip.address/debian jessie-backports main
----

Now just change the comment from one to the other, run apt-get
update and go.

Second workaround, if you have lots of entries to be switched:
create two sources.list files, named home.list and uni.list.

When you move from place to place, copy the appropriate one 
in to /etc/apt/sources.list and run apt-get update.

-dsr-

[toc] | [prev] | [next] | [standalone]

#182199

[Multipart message — attachments visible in raw view] — view raw

On Thu, Jun 15, 2017 at 10:16:45AM -0400, Dan Ritter wrote:
>On Wed, Jun 14, 2017 at 09:57:50PM +0200, tuxderlinuxfuchs77@gmail.com wrote:
>> The shown way is not very comfortable.
>>
>> 1. it is annoying having to remember IP addresses
>> 2. sed command is error prone (consider typos, etc.)
>> 3. copying around backup files is time consuming having to type in all
>> the paths
>> 4. typing in sed command or searching it in bash history for re-using is
>> time consuming, too.
>>
>
>Simple workaround for now:
>
>/etc/apt/sources.list:
>----
>deb http://site1.ip.address/debian jessie-backports main
>#deb http://site2.ip.address/debian jessie-backports main
>----
>
>Now just change the comment from one to the other, run apt-get
>update and go.
>
>Second workaround, if you have lots of entries to be switched:
>create two sources.list files, named home.list and uni.list.
>
>When you move from place to place, copy the appropriate one
>in to /etc/apt/sources.list and run apt-get update.

Of course, if site1 and site2 are not accessible from each other AND
you're willing to accept errors from apt, you can keep both enabled. apt
will complain that it can't update some repositories, but as both
provide the same packages, that's not a problem.

>
>-dsr-
>

-- 
For more information, please reread.

[toc] | [prev] | [next] | [standalone]

#182200

On Thu 15 Jun 2017 at 10:16:45 (-0400), Dan Ritter wrote:
> On Wed, Jun 14, 2017 at 09:57:50PM +0200, tuxderlinuxfuchs77@gmail.com wrote:
> > The shown way is not very comfortable.
> > 
> > 1. it is annoying having to remember IP addresses
> > 2. sed command is error prone (consider typos, etc.)
> > 3. copying around backup files is time consuming having to type in all
> > the paths
> > 4. typing in sed command or searching it in bash history for re-using is
> > time consuming, too.
> > 
> 
> Simple workaround for now:
> 
> /etc/apt/sources.list:
> ----
> deb http://site1.ip.address/debian jessie-backports main
> #deb http://site2.ip.address/debian jessie-backports main
> ----
> 
> Now just change the comment from one to the other, run apt-get
> update and go.
> 
> Second workaround, if you have lots of entries to be switched:
> create two sources.list files, named home.list and uni.list.
> 
> When you move from place to place, copy the appropriate one 
> in to /etc/apt/sources.list and run apt-get update.

Another workaround is to use bash functions, say upgrade-work and
upgrade-home, to wrap some rather long command lines like:

apt-get -o Acquire::http::Proxy="http://192.168.1.19:3142/" update && apt-get -d -o Acquire::http::Proxy="http://192.168.1.19:3142/" upgrade ; apt-get upgrade

Now you may well ask why I use such a crazy command line.
Answer, I run apt-cacher-ng on my wheezy server, BUT
proxying the actual upgrade step fails because wheezy can't
handle apt-listbugs, so I do that step directly. The long
command line saves messing with apt.conf or sources.list.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#182201

Thanks for the tips. Like I said, I'd prefer a program that does all
this in the background by calling one simple command


On 15-Jun-17 4:38 PM, David Wright wrote:
> On Thu 15 Jun 2017 at 10:16:45 (-0400), Dan Ritter wrote:
>> On Wed, Jun 14, 2017 at 09:57:50PM +0200, tuxderlinuxfuchs77@gmail.com wrote:
>>> The shown way is not very comfortable.
>>>
>>> 1. it is annoying having to remember IP addresses
>>> 2. sed command is error prone (consider typos, etc.)
>>> 3. copying around backup files is time consuming having to type in all
>>> the paths
>>> 4. typing in sed command or searching it in bash history for re-using is
>>> time consuming, too.
>>>
>> Simple workaround for now:
>>
>> /etc/apt/sources.list:
>> ----
>> deb http://site1.ip.address/debian jessie-backports main
>> #deb http://site2.ip.address/debian jessie-backports main
>> ----
>>
>> Now just change the comment from one to the other, run apt-get
>> update and go.
>>
>> Second workaround, if you have lots of entries to be switched:
>> create two sources.list files, named home.list and uni.list.
>>
>> When you move from place to place, copy the appropriate one 
>> in to /etc/apt/sources.list and run apt-get update.
> Another workaround is to use bash functions, say upgrade-work and
> upgrade-home, to wrap some rather long command lines like:
>
> apt-get -o Acquire::http::Proxy="http://192.168.1.19:3142/" update && apt-get -d -o Acquire::http::Proxy="http://192.168.1.19:3142/" upgrade ; apt-get upgrade
>
> Now you may well ask why I use such a crazy command line.
> Answer, I run apt-cacher-ng on my wheezy server, BUT
> proxying the actual upgrade step fails because wheezy can't
> handle apt-listbugs, so I do that step directly. The long
> command line saves messing with apt.conf or sources.list.
>
> Cheers,
> David.
>

[toc] | [prev] | [next] | [standalone]

#182206

On Thu 15 Jun 2017 at 16:44:27 (+0200), tuxderlinuxfuchs77@gmail.com wrote:
> Thanks for the tips. Like I said, I'd prefer a program that does all
> this in the background by calling one simple command

Sure. You're free to commission one from a C/C++ programmer.
We're just illustrating that the community doesn't see any
need (IMO) to "make it happen". The tools are already around
for scripting it.

I can see a reason to run apt-cacher-ng at home if you watch
broadcast TV and your ISP makes that assumption for their
customers.

But I can't see the point in your running apt-cacher-ng at a
university. Surely they run a university-wide caching system.
Or are you saying that you have a poor link to the university's
central servers—wireless perhaps? If so, the same applies: that
poor link is far more influential on interactive web browsing,
for example, than downloading debs, which occurs mainly in
the background. So a squid-style solution would be more beneficial.

What might be useful for package administration at work is to
put a line like:

apt-get -qq update && apt-get -qq -d upgrade && find /var/cache/apt/archives/ -name '*deb'

in root's or the system's crontab (that's my way), or
configure one of the unattended upgrade packages to
download debs periodically. The line above will send
an email whenever there are packages for you to upgrade
(after inspection; I don't trust automating that step).
No more watching stuttering downloads.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#182202

> Simple workaround for now:
>
> /etc/apt/sources.list:
> ----
> deb http://site1.ip.address/debian jessie-backports main
> #deb http://site2.ip.address/debian jessie-backports main
> ----
>
> Now just change the comment from one to the other, run apt-get
> update and go.
>
> Second workaround, if you have lots of entries to be switched:
> create two sources.list files, named home.list and uni.list.
>
> When you move from place to place, copy the appropriate one 
> in to /etc/apt/sources.list and run apt-get update.
>
> -dsr-
>
Its even easier than that.  Don't mess with the .list files just change  the file

/etc/apt/apt.conf.d/000apt-cacher-ng-proxy

according to the login.

The contents of that file is just the address/port of the apt-cacher-ng server.

Acquire::http::Proxy "http://172.16.0.1:3142/";

Just write a script to check what network you are on.  Like ping the
apt-cacher-ng server(s) until a response is sent and set 
/etc/apt/apt.conf.d/000apt-cacher-ng-proxy accordingly.

Try the following line with your own existing and non existing ips.

for f in 172.16.0.1 172.16.0.2 172.16.0.33; do if ping -c3 $f; then echo $f
yes;else echo $f no;fi; done

bob

[toc] | [prev] | [next] | [standalone]

#182234

On Thu, Jun 15, 2017 at 10:44:01AM -0400, Bob Weber wrote:
> 
> 
> 
> > Simple workaround for now:
> >
> > /etc/apt/sources.list:
> > ----
> > deb http://site1.ip.address/debian jessie-backports main
> > #deb http://site2.ip.address/debian jessie-backports main
> > ----
> >
> > Now just change the comment from one to the other, run apt-get
> > update and go.
> >
> > Second workaround, if you have lots of entries to be switched:
> > create two sources.list files, named home.list and uni.list.
> >
> > When you move from place to place, copy the appropriate one 
> > in to /etc/apt/sources.list and run apt-get update.
> >
> > -dsr-
> >
> Its even easier than that.  Don't mess with the .list files just change  the file
> 
> /etc/apt/apt.conf.d/000apt-cacher-ng-proxy
> 
> according to the login.
> 
> The contents of that file is just the address/port of the apt-cacher-ng server.
> 
> Acquire::http::Proxy "http://172.16.0.1:3142/";
> 
> Just write a script to check what network you are on.  Like ping the
> apt-cacher-ng server(s) until a response is sent and set 
> /etc/apt/apt.conf.d/000apt-cacher-ng-proxy accordingly.
> 
> Try the following line with your own existing and non existing ips.
> 
> for f in 172.16.0.1 172.16.0.2 172.16.0.33; do if ping -c3 $f; then echo $f
> yes;else echo $f no;fi; done
> 

There's a formalized version of this which might be useful:

$ apt-cache show ifscheme
Package: ifscheme
Version: 1.7-3
Maintainer: Guus Sliepen <guus@debian.org>
Architecture: all
Depends: ifupdown (>= 0.6.5)
Description-en: scheme control for network interfaces ifscheme allows you
to change network configuration schemes or query the current scheme. It
integrates with the ifup(8) command and interfaces(5). For example, you
might use this program to configure a "home" scheme and a "work" scheme
for a network device on a laptop. When you move between home and work,
a simple command can reconfigure your networking.

This could be rigged to change files when the network is
changed.

-dsr-

[toc] | [prev] | [next] | [standalone]

#182233

Hallo,
* tuxderlinuxfuchs77@gmail.com [Wed, Jun 14 2017, 09:57:50PM]:
>    Hello,
> 
>    I have a suggestion for a client program/package in combination with
>    apt-cacher-ng
> 
>    Consider following situation:
> 
>    User U_A has a laptop which he uses at home and at university
> 
>    He has a poor internet connection in both places.
> 
>    So he set up apt-cacher-ng services in both places but they are on
>    different networks (he does not have permission to change the network
>    settings at university to make the IP addresses of the apt-cacher-ng
>    servers math)
> 
>    Lets say at home the server HOST_1 is [1]http://192.168.1.100:3142
>    At university: server: HOST_2: [2]http://192.168.76.56:3142
> 
>    He uses Ubuntu (so this should work for Debian in a similar way).
> 
>    He makes a backup "/etc/apt/sources.list.bak" of /etc/apt/sources.list
> 
>    At university he sets up the cache repo as follows (after overwriting the
>    /etc/apt/sources.list with a copy of the backup if it was changed to use
>    the home apt-cacher-ng)
> 
>    sed -i 's/de.archive.ubuntu.com/192.168.76.56:3142/g'
>    /etc/apt/sources.list
> 
>    The same procedure at home.
> 
>    The shown way is not very comfortable.

You can automate this easily with a one-liner in /etc/network/ip-up.d/ ,
for example.

Actually, apt-cacher-ng provides a similar kind of automation but it can
only decide between "use a proxy" and "don't use a proxy" and the need
to switch by checking whether that proxy server is broken/unreachable by
timeout or a user hook script. This can be used well in a proxy cascade,
i.e. one apt-cacher-ng running on the mobile device itself in
non-storing mode, and local applications use that proxy on localhost.

There is, however, no support for multiple proxies that would be
switchable upon the mentioned conditions check. This could be added,
maybe in future. Please open a feature requeston Alioth (apt-cacher-ng
space) or in the BTS.

Regards,
Eduard.

-- 
Every great idea is worthless without someone to do the work. --Neil Williams

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web