Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.project > #8594 > unrolled thread

shutting down httpredir.debian.org?

Back to article view | Back to linux.debian.project

Contents

  shutting down httpredir.debian.org? Peter Palfrader <weasel@debian.org> - 2016-04-12 09:10 +0200
    Re: shutting down httpredir.debian.org? Raphael Hertzog <hertzog@debian.org> - 2016-04-12 09:40 +0200
      Re: shutting down httpredir.debian.org? Peter Palfrader <weasel@debian.org> - 2016-04-12 10:00 +0200
        Re: shutting down httpredir.debian.org? Cord Beermann <cord@debian.org> - 2016-04-12 12:30 +0200
        Re: shutting down httpredir.debian.org? Christian Rohmann <crohmann@netcologne.de> - 2016-04-12 13:00 +0200
          Re: shutting down httpredir.debian.org? Raphael Hertzog <hertzog@debian.org> - 2016-04-13 10:30 +0200
            Re: shutting down httpredir.debian.org? Christian Rohmann <crohmann@netcologne.de> - 2016-04-14 10:00 +0200
              Re: shutting down httpredir.debian.org? Peter Palfrader <weasel@debian.org> - 2016-04-14 11:10 +0200
                Re: shutting down httpredir.debian.org? Bastian Blank <waldi@debian.org> - 2016-04-14 12:10 +0200
                  Re: shutting down httpredir.debian.org? David Kalnischkies <david@kalnischkies.de> - 2016-04-14 15:40 +0200
                  Re: shutting down httpredir.debian.org? Tollef Fog Heen <tfheen@err.no> - 2016-04-15 06:50 +0200
                    Re: shutting down httpredir.debian.org? Ondřej Surý <ondrej@sury.org> - 2016-04-25 14:30 +0200
                      Re: shutting down httpredir.debian.org? Tollef Fog Heen <tfheen@err.no> - 2016-04-25 14:50 +0200
                        Re: shutting down httpredir.debian.org? Ondřej Surý <ondrej@sury.org> - 2016-04-25 15:10 +0200
                          Re: shutting down httpredir.debian.org? David Kalnischkies <david@kalnischkies.de> - 2016-04-25 15:50 +0200
                            Re: shutting down httpredir.debian.org? Peter Palfrader <weasel@debian.org> - 2016-04-25 17:40 +0200
                Re: shutting down httpredir.debian.org? anarcat <anarcat@orangeseeds.org> - 2016-04-14 17:00 +0200
                  Re: shutting down httpredir.debian.org? Charles Plessy <plessy@debian.org> - 2016-04-15 03:20 +0200
                    Re: shutting down httpredir.debian.org? Paul Wise <pabs@debian.org> - 2016-04-15 04:00 +0200
                  Re: shutting down httpredir.debian.org? Tollef Fog Heen <tfheen@err.no> - 2016-04-15 07:00 +0200
            Re: shutting down httpredir.debian.org? Christian Rohmann <crohmann@netcologne.de> - 2016-04-14 14:00 +0200
              Re: shutting down httpredir.debian.org? Raphael Hertzog <hertzog@debian.org> - 2016-04-14 17:00 +0200
                Re: shutting down httpredir.debian.org? Christian Rohmann <crohmann@netcologne.de> - 2016-04-19 13:50 +0200
      Re: shutting down httpredir.debian.org? Paul Wise <pabs@debian.org> - 2016-04-12 18:30 +0200
    Re: shutting down httpredir.debian.org? Lucas Nussbaum <lucas@debian.org> - 2016-04-12 10:50 +0200
    Re: shutting down httpredir.debian.org? Raphael Geissert <geissert@debian.org> - 2016-04-12 14:00 +0200
      Re: shutting down httpredir.debian.org? Ben Hutchings <ben@decadent.org.uk> - 2016-04-12 15:10 +0200
      Re: shutting down httpredir.debian.org? Peter Palfrader <weasel@debian.org> - 2016-04-12 15:30 +0200
        Re: shutting down httpredir.debian.org? Raphael Geissert <geissert@debian.org> - 2016-04-13 14:30 +0200
      Re: shutting down httpredir.debian.org? Donald Norwood <dnorwood@portalus.com> - 2016-04-12 19:20 +0200
        Re: shutting down httpredir.debian.org? Donald Norwood <dnorwood@portalus.com> - 2016-04-12 22:50 +0200

Page 1 of 2  [1] 2  Next page →

#8594 — shutting down httpredir.debian.org?

Hi,

we keep getting reports of httpredir.debian.org not working correctly,
such as intermittently just sending errors or redirecting to mirrors
that are out of date.

Only a few of those make it to the BTS, some make it to
mirrors@debian.org, and there are several on various IRC channels.  I
suspect quite a few make it to Raphael, since that's still the contact
point listed on the website (not an email address there, either - just a
link to blogspot).

When there is a response - and there isn't always - it's usually "nobody
currently maintains httpredir, sorry".

So, it appears as if currently nobody has time or the energy to take
care of httpredir.debian.org properly.

I suggest we shut down the service for now.  If, at some future point,
somebody wants to maintain again we can always start it up again.

Opinions?
-- 
                            |  .''`.       ** Debian **
      Peter Palfrader       | : :' :      The  universal
 https://www.palfrader.org/ | `. `'      Operating System
                            |   `-    https://www.debian.org/

[toc] | [next] | [standalone]

#8595

Hi,

On Tue, 12 Apr 2016, Peter Palfrader wrote:
> So, it appears as if currently nobody has time or the energy to take
> care of httpredir.debian.org properly.
> 
> I suggest we shut down the service for now.  If, at some future point,
> somebody wants to maintain again we can always start it up again.

Will you make httpredir point to a normal mirror so as not to break
systems relying on it? (Or even to the geolocalized DNS entries if we
still have that)

If yes, then it's certainly a sensible thing to do.

I'd like also to note that once we have proper by-hash package indices in
Debian too, it's entirely reasonable to rely on MirrorBrain as HTTP
redirector. I use it for Kali for more than 3 years already.

http://mirrorbrain.org

But I still haven't gotten round to package it properly for Debian... but
I will do that at some point.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

[toc] | [prev] | [next] | [standalone]

#8596

On Tue, 12 Apr 2016, Raphael Hertzog wrote:

> On Tue, 12 Apr 2016, Peter Palfrader wrote:
> > So, it appears as if currently nobody has time or the energy to take
> > care of httpredir.debian.org properly.
> > 
> > I suggest we shut down the service for now.  If, at some future point,
> > somebody wants to maintain again we can always start it up again.
> 
> Will you make httpredir point to a normal mirror so as not to break
> systems relying on it? (Or even to the geolocalized DNS entries if we
> still have that)
> 
> If yes, then it's certainly a sensible thing to do.

I agree that breaking existing uses (of at least /debian) should be
avoided, and that, therefore, pointing it to some working system would
be the way to go.

> I'd like also to note that once we have proper by-hash package indices in
> Debian too, it's entirely reasonable to rely on MirrorBrain as HTTP
> redirector. I use it for Kali for more than 3 years already.
> 
> http://mirrorbrain.org

Looks exciting at first glance.  Need to look at it in more detail.

Cheers,
-- 
                            |  .''`.       ** Debian **
      Peter Palfrader       | : :' :      The  universal
 https://www.palfrader.org/ | `. `'      Operating System
                            |   `-    https://www.debian.org/

[toc] | [prev] | [next] | [standalone]

#8598

Hallo! Du (Peter Palfrader) hast geschrieben:

>> Will you make httpredir point to a normal mirror so as not to break
>> systems relying on it? (Or even to the geolocalized DNS entries if we
>> still have that)
>> 
>> If yes, then it's certainly a sensible thing to do.
>
>I agree that breaking existing uses (of at least /debian) should be
>avoided, and that, therefore, pointing it to some working system would
>be the way to go.

Many of our pages mention httpredir as prefered service

https://wiki.debian.org/SourcesList
https://www.debian.org/mirror/list
https://www.debian.org/mirror/
https://wiki.debian.org/DebianGeoMirror

so if this announcement
https://lists.debian.org/debian-devel-announce/2015/05/msg00003.html
isn't correct anymore, there should be made a new one.

Cord

[toc] | [prev] | [next] | [standalone]

#8599

Hey all,

On 04/12/2016 09:53 AM, Peter Palfrader wrote:
>> > I'd like also to note that once we have proper by-hash package indices in
>> > Debian too, it's entirely reasonable to rely on MirrorBrain as HTTP
>> > redirector. I use it for Kali for more than 3 years already.
>> > 
>> > http://mirrorbrain.org
> Looks exciting at first glance.  Need to look at it in more detail.

YES! Please please please do use mirrorbrain, OpenSuse, TDF
(LibreOffice), VLC, XMBC and others (see: http://mirrorbrain.org/users/)
use it. Actually most of the traffic mirror.netcologne.de receives comes
from various mirrorbrain instances.

It does wonderful things (http://mirrorbrain.org/features/):

 * Load-Balance by GeoIP / AS matching (traffic stays very local)
 * Intensive checks (even via rsync) regarding mirror consistency
 ** It can serve things down to the file level to a mirror that
still/already has a certain file
 * automatic hashing to check for
 * meta links
 * torrent support (so mirrors could function as HTTP seeds)
 * ...



Regards

Christian

[toc] | [prev] | [next] | [standalone]

#8606

Hi,

On Tue, 12 Apr 2016, Christian Rohmann wrote:
> It does wonderful things (http://mirrorbrain.org/features/):

Nice to see some much support but I would like to point out that not
everything is perfect either...

>  * Load-Balance by GeoIP / AS matching (traffic stays very local)

That's clearly the main selling point.

>  * Intensive checks (even via rsync) regarding mirror consistency

That's good too but the downside is that the mirrors must
offer rsync service, either public or at least for the mirror redirector.
It's not possible to use a mirror that cannot be scanned via rsync.

>  ** It can serve things down to the file level to a mirror that
> still/already has a certain file

The "still" is wrong. It's actually an annoying bug that needs to be
fixed for our usage... Kali wants to sponsor this bugfix but the
upstream developer is currently busy with other stuff so it has not
happened yet.

https://github.com/poeml/mirrorbrain/issues/21

Which also shows another downside: this is really a one-person
project currently (not different from httpredir though, and
this one has the advantage of being more generic so can and already serves
more projects in the free software world).

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

[toc] | [prev] | [next] | [standalone]

#8608

On 04/13/2016 10:23 AM, Raphael Hertzog wrote:
>>  ** It can serve things down to the file level to a mirror that
>> > still/already has a certain file
> The "still" is wrong. It's actually an annoying bug that needs to be
> fixed for our usage... Kali wants to sponsor this bugfix but the
> upstream developer is currently busy with other stuff so it has not
> happened yet.
> 
> https://github.com/poeml/mirrorbrain/issues/21
> 
> Which also shows another downside: this is really a one-person
> project currently (not different from httpredir though, and
> this one has the advantage of being more generic so can and already serves
> more projects in the free software world).


True. I know Peter is very busy. I opened and issue and sent in a patch
myself, to allow IPv6 AS look-up
(https://github.com/poeml/mirrorbrain/issues/154) which has not yet made
it into mod_as.

I suppose the move to GitHub is a sign that he wants to allow easier
cooperation with others and is open for patches. which can be integrated
easy.



It might sound like a very non technical argument: But what apart from
mirrorbrain which is that powerful, free and field-proven is there as
alternative? I would rather work on getting pull requests ready
resolving the various little bugs and annoyances than to discuss
something completely different once again. It's not like mirrorbrain is
fundamentally unfit to work as good mirror redirector for Debian.


Regards

Christian

[toc] | [prev] | [next] | [standalone]

#8609

Christian Rohmann schrieb am Donnerstag, dem 14. April 2016:

> It might sound like a very non technical argument: But what apart from
> mirrorbrain which is that powerful, free and field-proven is there as
> alternative? I would rather work on getting pull requests ready
> resolving the various little bugs and annoyances than to discuss
> something completely different once again. It's not like mirrorbrain is
> fundamentally unfit to work as good mirror redirector for Debian.

You could argue (and I have), that that file-based redirects are not
ideal if your update is downloading lots of little files.  The latency
hit of many redirects is non-trivial.


Regardless, currently httpredir.debian.org is in a bad shape, and users
get errors when they are using it.  This is unacceptable and it needs
fixing.

Even pointing the name to a single server that works, such as
ftp.debian.org, would be better than the status quo.

If we want to maintain some form of geographic closeness for it, then
pointing it to deb.debian.org seems like something we could try.

Raphael indicated that he plans to fix httpredir and keep maintaining
it.  If that actually works out, maybe we don't need to change anything.
We will see.

-- 
                            |  .''`.       ** Debian **
      Peter Palfrader       | : :' :      The  universal
 https://www.palfrader.org/ | `. `'      Operating System
                            |   `-    https://www.debian.org/

[toc] | [prev] | [next] | [standalone]

#8610

On Thu, Apr 14, 2016 at 09:02:18AM +0000, Peter Palfrader wrote:
> You could argue (and I have), that that file-based redirects are not
> ideal if your update is downloading lots of little files.  The latency
> hit of many redirects is non-trivial.

My last test showed problems with httpredir returning different mirrors
with each request, which made keep-alive completely impossible.
Downloading all the stuff during a cdebootstrap run from a real mirror
took 5s[1], from httpredir at least 20s[2].

> Regardless, currently httpredir.debian.org is in a bad shape, and users
> get errors when they are using it.  This is unacceptable and it needs
> fixing.

Ack.

> If we want to maintain some form of geographic closeness for it, then
> pointing it to deb.debian.org seems like something we could try.

What kind of setup is deb.debian.org?

I would prefer to have a DNS name that refers to close mirrors.  This
minimizes impact of slightly deviating mirrors, as the DNS timeout is
not that small.

And maybe we can fit the possibility to force some ip blocks to given
mirrors.  Currently we hard-code the Azure supplied mirrors into the
images[3].  However I would like to move this decision into more global
infrastructure, without paying the overhead of httpredir in the current
form.

Maybe it would be possible to use geoip2, as current bind9 uses, and add
own lists with such provider overrides.  I think I have to play with
this a bit.

Regards,
Bastian

[1]: One connection is re-used for every file and gets a large TCP
windows in the process.
[2]: One connection to httpredir and several to other locations that
never got re-used as the number was too large.
[3]: Microsoft wants to reduce the impact a large amount of Debian
systems could have on there input data pipe.
-- 
One does not thank logic.
		-- Sarek, "Journey to Babel", stardate 3842.4

[toc] | [prev] | [next] | [standalone]

#8612

[Multipart message — attachments visible in raw view] — view raw

On Thu, Apr 14, 2016 at 11:49:12AM +0200, Bastian Blank wrote:
> On Thu, Apr 14, 2016 at 09:02:18AM +0000, Peter Palfrader wrote:
> > You could argue (and I have), that that file-based redirects are not
> > ideal if your update is downloading lots of little files.  The latency
> > hit of many redirects is non-trivial.
>
> My last test showed problems with httpredir returning different mirrors
> with each request, which made keep-alive completely impossible.
> Downloading all the stuff during a cdebootstrap run from a real mirror
> took 5s[1], from httpredir at least 20s[2].

Well, its kinda the point of httpredir to give you different mirrors[1]
(which ideally all have the same state – if not you get the hashsum
mismatches). That allows "long living" clients like apt to talk to the
different mirrors at the same time (proper keep-alive and pipelining to
them all). One-shot clients like wget which debootstrap is using are
obviously not doing either (with the same argument its also unwise to
use a https mirror for those clients).

It is frequently said that apt should "just" open multiple connections
to the same mirror and stuff like apt-fast sells bigtime on this, but as
a categorical imperative[2] it makes no sense: Of course you are
speeding up your throughput if you add more cars to a traffic jam – but
only as long as other people don't use the same strategy. If they all do
it the traffic jam is even bigger and its actually slower for everyone.

[1] Which is not to say that it must be httpredir which gets us such
a list, but it was possible and someone implemented it.
[2] I know, browsers do it all the time – but most websites are build
dynamic requiring non-trivial time on the server to process so it kinda
makes sense as a workaround; repositories are pretty static) and the
HTTP1.1 spec actually allows it. It gets slightly more sensible with
HTTP2 now – at least server/proxy implementors can't say pipelining is
too much state keeping to implement properly anymore. ;)


Best regards

David Kalnischkies

[toc] | [prev] | [next] | [standalone]

#8618

]] Bastian Blank 

> On Thu, Apr 14, 2016 at 09:02:18AM +0000, Peter Palfrader wrote:
> > If we want to maintain some form of geographic closeness for it, then
> > pointing it to deb.debian.org seems like something we could try.
> 
> What kind of setup is deb.debian.org?

It's ftp.debian.org fronted by (currently at least), Fastly.  (Not sure
if that answers your question?  Happy to elaborate if you explain what
you're after.)

> And maybe we can fit the possibility to force some ip blocks to given
> mirrors.  Currently we hard-code the Azure supplied mirrors into the
> images[3].  However I would like to move this decision into more global
> infrastructure, without paying the overhead of httpredir in the current
> form.

(Taking a slight guess at what you actually mean here.)  Anycast for
HTTP directly is somewhat icky, when done over the global internet,
since you often (relatively) end up with BGP reconvergences leading you
to hit a different server, which causes the current problems.  For this,
just doing anycast for the DNS servers is better (since it's mostly UDP
(and the rest is short-lived TCP) and therefore not subject to the same
problems.

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

[toc] | [prev] | [next] | [standalone]

#8625

Tollef,

since you work at Fastly, could you make the deb.debian.org to have IP
address? :) Currently it's accessible only via Legacy IP although
static.debian.org has AAAA records, it get's redirected to
global-nossl.fastly.net that's accessible only via Legacy IP which makes
me very sad. 

Cheers,
-- 
Ondřej Surý <ondrej@sury.org>
Knot DNS (https://www.knot-dns.cz/) – a high-performance DNS server

On Fri, Apr 15, 2016, at 06:48, Tollef Fog Heen wrote:
> > What kind of setup is deb.debian.org?
> 
> It's ftp.debian.org fronted by (currently at least), Fastly.  (Not sure
> if that answers your question?  Happy to elaborate if you explain what
> you're after.)

[toc] | [prev] | [next] | [standalone]

#8626

]] Ondřej Surý 

Hi,

> since you work at Fastly, could you make the deb.debian.org to have IP
> address? :) Currently it's accessible only via Legacy IP although
> static.debian.org has AAAA records, it get's redirected to
> global-nossl.fastly.net that's accessible only via Legacy IP which makes
> me very sad. 

deb.d.o has a lower-priority ipv6 fallback, so you should file a bug on
apt asking it to use the lower-priority alternative that has the
protocol you need, since apparently that doesn't work today.

Until that gets fixed, you can use cdn-fastly-v6.deb.d.o for v6 support.
It's not the default since too many users ended up in suboptimal places.
Once that's fixed, I'm going to flip the switch back again.

Cheers,
-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

[toc] | [prev] | [next] | [standalone]

#8627

Hi,

On Mon, Apr 25, 2016, at 14:41, Tollef Fog Heen wrote:
> ]] Ondřej Surý 
> 
> Hi,
> 
> > since you work at Fastly, could you make the deb.debian.org to have IP
> > address? :) Currently it's accessible only via Legacy IP although
> > static.debian.org has AAAA records, it get's redirected to
> > global-nossl.fastly.net that's accessible only via Legacy IP which makes
> > me very sad. 
> 
> deb.d.o has a lower-priority ipv6 fallback, so you should file a bug on
> apt asking it to use the lower-priority alternative that has the
> protocol you need, since apparently that doesn't work today.

I'll try in unstable first as this was a LXC guest with jessie.

> Until that gets fixed, you can use cdn-fastly-v6.deb.d.o for v6 support.
> It's not the default since too many users ended up in suboptimal places.
> Once that's fixed, I'm going to flip the switch back again.

I am fine just with the knowledge that this is being worked on.

Cheers,
-- 
Ondřej Surý <ondrej@sury.org>
Knot DNS (https://www.knot-dns.cz/) – a high-performance DNS server

[toc] | [prev] | [next] | [standalone]

#8628

[Multipart message — attachments visible in raw view] — view raw

On Mon, Apr 25, 2016 at 03:07:40PM +0200, Ondřej Surý wrote:
> On Mon, Apr 25, 2016, at 14:41, Tollef Fog Heen wrote:
> > > since you work at Fastly, could you make the deb.debian.org to have IP
> > > address? :) Currently it's accessible only via Legacy IP although
> > > static.debian.org has AAAA records, it get's redirected to
> > > global-nossl.fastly.net that's accessible only via Legacy IP which makes
> > > me very sad. 
> > 
> > deb.d.o has a lower-priority ipv6 fallback, so you should file a bug on
> > apt asking it to use the lower-priority alternative that has the
> > protocol you need, since apparently that doesn't work today.

Assuming I read the code right apt picks a random SRV entry from the set
of records with the same priority (aka: ignoring weights in this random
pick).


> I'll try in unstable first as this was a LXC guest with jessie.

Be careful that this isn't behind a cache/proxy or such as these usually
don't bother with SRV and instead rely on the (permanent?) HTTP redirect
recently added.


> > Until that gets fixed, you can use cdn-fastly-v6.deb.d.o for v6 support.
> > It's not the default since too many users ended up in suboptimal places.
> > Once that's fixed, I'm going to flip the switch back again.
> 
> I am fine just with the knowledge that this is being worked on.

From the apt side it isn't being worked on.
Would be happy to see patches for it…


Best regards

David Kalnischkies

[toc] | [prev] | [next] | [standalone]

#8629

On Mon, 25 Apr 2016, David Kalnischkies wrote:

> > I'll try in unstable first as this was a LXC guest with jessie.
> 
> Be careful that this isn't behind a cache/proxy or such as these usually
> don't bother with SRV and instead rely on the (permanent?) HTTP redirect
                                                 ^^^^^^^^^^
> recently added.

permanent seems like a bad idea.  fixed in dsa-puppet git.

Cheers,
-- 
                            |  .''`.       ** Debian **
      Peter Palfrader       | : :' :      The  universal
 https://www.palfrader.org/ | `. `'      Operating System
                            |   `-    https://www.debian.org/

[toc] | [prev] | [next] | [standalone]

#8613

On 2016-04-14 05:02:18, Peter Palfrader wrote:
> Christian Rohmann schrieb am Donnerstag, dem 14. April 2016:
>
>> It might sound like a very non technical argument: But what apart from
>> mirrorbrain which is that powerful, free and field-proven is there as
>> alternative? I would rather work on getting pull requests ready
>> resolving the various little bugs and annoyances than to discuss
>> something completely different once again. It's not like mirrorbrain is
>> fundamentally unfit to work as good mirror redirector for Debian.
>
> You could argue (and I have), that that file-based redirects are not
> ideal if your update is downloading lots of little files.  The latency
> hit of many redirects is non-trivial.
>
> Regardless, currently httpredir.debian.org is in a bad shape, and users
> get errors when they are using it.  This is unacceptable and it needs
> fixing.
>
> Even pointing the name to a single server that works, such as
> ftp.debian.org, would be better than the status quo.

Personnally, I don't see many problems with the redirector at all. In
Canada/Montreal it works fine and I have not heard reports of problems
with it since I switched. Maybe the problems are specific to certain
regions?

Can people show specific bug reports they have filed that describe
problematic behavior?

> If we want to maintain some form of geographic closeness for it, then
> pointing it to deb.debian.org seems like something we could try.

Note sure what that is. http://deb.debian.org/ seems to say it uses the
Fastly CDN, at least from here.

A fundamental issue of all this is who we give our users to. Sending
Debian users to a commercial CDN is a political decision with huge
privacy implications for our users. I do not think we should redirect
httpredir like this without at least first informing our users, in
advance, so they can make an informed decision on which mirror they
trust with their metadata.

There were numerous discussions about CDNs here, and I am not sure I
would encourage the use of those in Debian right now.

> Raphael indicated that he plans to fix httpredir and keep maintaining
> it.  If that actually works out, maybe we don't need to change anything.
> We will see.

I would very much like Debian to continue hosting its own redirector,
somehow. I think it makes sense to share the work of maintaining that
software, but if possible, mirrorbrain, the *software*, not the
*service*, would replace the current HTTP redirector. In other words,
httpredir, if we take that route, should not be shutdown but
reconfigured with mirrorbrain instead of relying on a separate distinct
infrastructure.

Does that make sense?

I'd be curious to see how I can contribute to that effort. I have a good
grasp of Python (so Mirrorbrain seems familiar) and, to a lesser extent
Perl (so I could also handle some http-redirector) and some free time,
although I would prefer to work on this in my consulting hours of
course.

It would also be nice to see exactly what the requirements for this
project are, and what is currently missing (say priority bug reports to
fix). I feel that http-redir is leagues ahead of whatever we had before
(which is basically cdn.debian.net, and before that, nothing), so just
ditching it because it doesn't work perfectly seems like a rather
extreme measure.

A.

-- 
One of the things the Web teaches us is that everything is connected
(hyperlinks) and we all should work together (standards). Too often
school teaches us that everything is separate (many different
'subjects') and that we should all work alone.        - Aaron Swartz

[toc] | [prev] | [next] | [standalone]

#8616

Le Thu, Apr 14, 2016 at 10:53:10AM -0400, anarcat a écrit :
> 
> Can people show specific bug reports they have filed that describe
> problematic behavior?

Hi all,

On my side of the World, httpredir never worked correctly.  Here is an example
showing that despite it detects correclty that I am based in Yokohama, Japan,
I am redirected in Turkey !

------------------------------------------------------------------------------
IP: 134.160.83.73
AS: 18128
Continent: AS
Country: JP
Region: 19
City: Yokohama

Had you requested a file to /debian/, you would have been sent to one of the following mirrors:

    http://debian.gnu.gen.tr/debian/

Out of a population of: 40 mirrors
Matched by: nearby-continent
------------------------------------------------------------------------------

In the past, I have been frequently redirected to Korea or Taiwan.  I have let
Raphael know by email but did not open a bug since at that time it was still a
debian.net service.

Also, http.debian.net (at that time) was supposed to redirect Amazon cloud
users to the CloudFront CDN (https://lists.debian.org/debian-cloud/2013/05/msg00066.html),
but in practice it did not work (https://lists.debian.org/debian-cloud/2013/10/msg00044.html)
and it still does not.  Here is what I get from whithin the Japan-based Amazon cloud:

------------------------------------------------------------------------------
$ GET -e http://http.debian.net/demo/debian/
200 OK
Cache-Control: no-cache
Connection: close
Date: Fri, 15 Apr 2016 01:03:44 GMT
Pragma: no-cache
Server: Apache
Client-Date: Fri, 15 Apr 2016 01:03:43 GMT
Client-Peer: 128.31.0.66:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
Link: <http://debian.lagis.at/debian/>; rel=duplicate; pri=13590; depth=0
X-Arch: 
X-AS: 16509
X-City: Tokyo
X-Clacks-Overhead: GNU Terry Pratchett
X-Closest-Distance: 135.8997
X-Continent: AS
X-Country: JP
X-Distance: 135.8997
X-IP: 54.95.145.228
X-Match-Type: nearby-continent
X-Population: 2
X-Region: 40
X-Std-Dev: 8.30850467894193
X-URL: 
------------------------------------------------------------------------------

The reason why I refrained from reporting these bugs further is that I have do
not have time to help solve them, and I was seeing http.debian.net as a "best
effort" service.

Have a nice day,

-- 
Charles Plessy
Tsurumi, Kanagawa, Japan

[toc] | [prev] | [next] | [standalone]

#8617

On Fri, Apr 15, 2016 at 9:14 AM, Charles Plessy wrote:

> The reason why I refrained from reporting these bugs further is that I have do
> not have time to help solve them, and I was seeing http.debian.net as a "best
> effort" service.

If you have time to report bugs about the issues you found it would be
great to have a public list of httpredir deficiencies so people who
have time can work on them. Probably the best way to report them is
bugs with these pseudo-headers:

Package: mirrors
Severity: normal
User: mirrors@packages.debian.org
Usertags: httpredir

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[toc] | [prev] | [next] | [standalone]

#8619

]] anarcat 

(In the interest of full disclosure: I work for Fastly.)

> On 2016-04-14 05:02:18, Peter Palfrader wrote:
> > If we want to maintain some form of geographic closeness for it, then
> > pointing it to deb.debian.org seems like something we could try.
> 
> Note sure what that is. http://deb.debian.org/ seems to say it uses the
> Fastly CDN, at least from here.

That is correct.

> A fundamental issue of all this is who we give our users to. Sending
> Debian users to a commercial CDN is a political decision with huge
> privacy implications for our users. I do not think we should redirect
> httpredir like this without at least first informing our users, in
> advance, so they can make an informed decision on which mirror they
> trust with their metadata.

They're already being redirected to random mirrors by using httpredir,
where we have absolutely no control over their logging policy and
practices.  With Fastly, we control the logging policy and can stream
logs if we want to (or we can decide not to, which is the current
setup).  Fastly's own policy is documented on
https://docs.fastly.com/guides/compliance/security-and-technology-compliance#cache-data-and-end-user-information-management

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.project

csiph-web