Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder4.news.weretis.net!storethat.news.telefonica.de!telefonica.de!news-1.dfn.de!news.dfn.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: Sandman <mr@sandman.net>
Newsgroups: comp.os.linux.networking,comp.os.linux.security,comp.infosystems.www.servers.unix
Subject: wpad.dat attack on Linux Apache server
Date: Fri, 24 May 2013 11:22:15 +0200
Lines: 88
Message-ID: <mr-983E88.11221424052013@News.Individual.NET>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Trace: individual.net DL8Pzwu8OiUlXPPg/O/+hAERdB0pQCJO0wqaiiaHBu+3lhlRI=
X-Orig-Path: mr
Cancel-Lock: sha1:w0R0cX271XPzZvQ8/Hlh9Xtz2t0=
User-Agent: MT-NewsWatcher/3.5.2 (Intel Mac OS X)
X-Face: $@,Vfa$,)%=Qa7L]y)&oZj_\EiHc}}A<Y3TvbI2&|e"bE9zc[o<ThMgB4%*L$b1YsNl!/ <lHO$>f0Bei"4a_%)"c6TQ+P/:53>;PNGuWUmkqyeN-qM65foJ[;T_(k;>]&G\T4Lhm:2 ujye2_,iUJFE;NZn>y;.|-hl7g~bIOF1qG\o<?]4mXkW*mT3]{Bn&VwP7(M0uYnGA!V!? {"y?BkBDW6e-.=I5
X-Killfiled: yttrx, gallopinginsanity.com, Mark Kent, Maverick, NRen2, weedhopper, PC Guy, nospam@nospam.com, Oxford, Jim Lee Jr., Mocassin Joe, Chance Furlong, XX, Uncle Max, Joe Pain, Redjak, isquat@gmail.com, Robert Whelan, jt2002a@hotmail.com
Xref: csiph.com comp.os.linux.networking:2129 comp.os.linux.security:283 comp.infosystems.www.servers.unix:166

I don't actually know if it's an attack.

My Apache server was slow - symptoms being slow responses (after PHP had 
processed and sent to client) and dropped connections, looked in the log 
files, found 12GB of accesses to /wpad.dat on my catch-all vhost. 

I got thousands of requests per minute from hundreds of different hosts, 
and a sample of these showed that they all seemed like legit end-user 
hosts, not a tor proxy botnet at least.

I googled some, and found that wpad is some form of auto-discover proxy 
settings. Problem is, it should be done to the local network. So if my 
machine is on the "example.com" network, my browser will send a request 
to "wpad.example.com" to find proxy settings. not 
"wpad.remoteinternetsite.com".

So, maybe it is an attack after all?

I added a wpad.dat file to the server, with this content:

    function FindProxyForURL(url, host) { return "PROXY 127.0.0.1:445"; }

Which just tells these clients to look for a proxy on localhost. Nothing 
changed. Thousands of thousands of requests.

In /server-status for Apache, my queue is filled with /wpad.dat requests 
with the "K" status (Keep-Alive), so that sounds like why it's slow.

Ok, so I'll block it. Blocking it in apache seemd stupid, it would still 
process the requests, so to iptables:

    iptables -I INPUT -p tcp --dport 80 -m string --to 70 --algo bm \
      --string "GET /wpad.dat" -j REJECT --reject-with tcp-reset

Yes, I know this analyzes *every* request, and wastes CPU cycles, I may 
get around to chain this into a seperate iptables chain and only act on 
correct parts. In about an hour, this has blocked 45k requests, or about 
750 per minute

Either way, the requests are gone, the log file is clean(er) but the 
server is still slow and still drops connections.

So, troubleshooting some more. CPU is at 0.8% usage, Memory is 80% free. 
Asking my hosting ISP, my bandwidth capacity is at 0.62%

ethtool -S eth0 shows no errors (but a shitload of packets, of course)

Someone suggested that it could have something to do with my using a 
wildcard DNS. So, my server hosts some 100+ virtual hosts. All my 
clients are told to use a CNAME pointer for their servers. So:

 www.client.com -> CNAME -> cluster.mydomain.com -> A -> 123.123.123.123

Which means that every visitor to my sites has their web browser first 
look up www.client.com to find cluster.mydomain.com which in turn points 
to my IP. 

No, the mydomain.com had a wildcard setting, so if and when they would 
access "wpad.mydomain.com" my DNS would point that to 
cluster.mydomain.com and then that wold point to the IP. So supposedly, 
all the request could channel to my server this way.

I have now removed wildcard for mydomain.com, and also added a wpad host 
for all my domains that points to 127.0.0.1. I'm waiting to see that 
propagate and see if it makes any difference. It hasn't so far.


Do any of you guys have any ideas what this might be?  Or rather - how 
do I trouble shoot this some more?

I have:

    Slow transfer speeds on apache
    Super fast on other ports (SFTP for instance)
    Thousands of requests per minute that are now being blocked
    Super low CPU usage
    Super low RAM usage
    No reported ethernet errors








-- 
Sandman[.net]