Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.networking > #1383 > unrolled thread

apache swapping the system to death

Back to article view | Back to comp.os.linux.networking

Contents

  apache swapping the system to death unruh <unruh@invalid.ca> - 2012-05-22 01:35 +0000
    Re: apache swapping the system to death "J.O. Aho" <user@example.net> - 2012-05-22 06:57 +0200
    Re: apache swapping the system to death Wolfram Gloger <wmglo@dent.med.uni-muenchen.de> - 2012-05-22 11:24 +0200
      Re: apache swapping the system to death unruh <unruh@invalid.ca> - 2012-05-22 15:21 +0000

#1383 — apache swapping the system to death

I am running a web server on a Mandriva 2010.2 system. Today the system
suddenly became increadibly non-responsive. When I finally logged on as
root, the swap file was up around 3GB, (no wonder response was horrible)
and there were 160 instances of httpd running. After I finally managed
to shut them down, (killall -9 httpd) response was restored. I looked in
the /var/log/httpd/access_log there did not seem to be much unusual
there. There were some google.com with weird addreses, and some internal
connections which were the only things that looked out of the oridinary.

Eg
66.249.68.198 - - [21/May/2012:18:19:23 -0700] "GET /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/bit.ly/www.guardian.co.uk/business/2012/may/04/pay-vince-cable HTTP/1.1" 200 72658 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.68.73 - - [21/May/2012:18:19:26 -0700] "GET /aggregator/www.nytimes.com/2012/04/05/opinion/node/node/www.bbc.co.uk/news/uk-17769717?page=226 HTTP/1.1" 200 38984 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

and

::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"
::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"
::1 - - [21/May/2012:17:32:05 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"
::1 - - [21/May/2012:17:32:07 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"

....


But could any of these be respoinsible for 160 connections? 

The other suspicious thing is that there was 6 hour gap in
the logs

112.111.174.175 - - [21/May/2012:11:34:11 -0700] "GET /user/register
HTTP/1.0" 200 29860 "http://emergentgravity.org/user/register"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) Opera 6.01  [en]"
::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
"Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
connection)"

[toc] | [next] | [standalone]

#1384

On 22/05/12 03:35, unruh wrote:
> I am running a web server on a Mandriva 2010.2 system. Today the system
> suddenly became increadibly non-responsive. When I finally logged on as
> root, the swap file was up around 3GB, (no wonder response was horrible)
> and there were 160 instances of httpd running. After I finally managed
> to shut them down, (killall -9 httpd) response was restored. I looked in
> the /var/log/httpd/access_log there did not seem to be much unusual
> there. There were some google.com with weird addreses, and some internal
> connections which were the only things that looked out of the oridinary.

Tweak your apache settings, so that no 160 instances will be started, 
with some google you should be able to find some setting recommendation 
formulas.

I think I had some issue with an earlier wordpress version (pre 3.3), 
which got really strange, specially when it tried to access a page which 
didn't exist and it came into a horrible loop (yes, wordpress may 
request pages itself).

> Eg
> 66.249.68.198 - - [21/May/2012:18:19:23 -0700] "GET /aggregator/www.umsl.edu/~keelr/010/www.twitter.com/www.iaea.org/Publications/Documents/Board/2008/bit.ly/www.guardian.co.uk/business/2012/may/04/pay-vince-cable HTTP/1.1" 200 72658 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
> 66.249.68.73 - - [21/May/2012:18:19:26 -0700] "GET /aggregator/www.nytimes.com/2012/04/05/opinion/node/node/www.bbc.co.uk/news/uk-17769717?page=226 HTTP/1.1" 200 38984 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
>
> and
>
> ::1 - - [21/May/2012:17:32:03 -0700] "OPTIONS * HTTP/1.0" 200 - "-"
> "Apache/2.2.15 (Mandriva Linux/PREFORK-3.2mdv2010.2) (internal dummy
> connection)"

This is all okey, nothing to worry about, if you don't want to see them, 
there is an instruction how to filter those at apache.org.
Has nothing to do with the increased number of services loaded.


> But could any of these be respoinsible for 160 connections?
>
> The other suspicious thing is that there was 6 hour gap in
> the logs

That don't sound good at all, could have been caused by the swapped out 
syslog, but I would think it's wise to take a look at your system with 
rkhunter and/or chkrootkit. Also check your other logs and see if you 
miss 6 hours.


-- 

  //Aho

[toc] | [prev] | [next] | [standalone]

#1385

unruh <unruh@invalid.ca> writes:

> I am running a web server on a Mandriva 2010.2 system. Today the system
> suddenly became increadibly non-responsive. When I finally logged on as
> root, the swap file was up around 3GB, (no wonder response was horrible)
> and there were 160 instances of httpd running.

If you are running apache versions from 2010, this sounds suspiciously
like you have been a victim of

http://httpd.apache.org/security/CVE-2011-3192.txt

It would be best to upgrade to newer binaries then.

Regards,
Wolfram.

[toc] | [prev] | [next] | [standalone]

#1386

On 2012-05-22, Wolfram Gloger <wmglo@dent.med.uni-muenchen.de> wrote:
> unruh <unruh@invalid.ca> writes:
>
>> I am running a web server on a Mandriva 2010.2 system. Today the system
>> suddenly became increadibly non-responsive. When I finally logged on as
>> root, the swap file was up around 3GB, (no wonder response was horrible)
>> and there were 160 instances of httpd running.
>
> If you are running apache versions from 2010, this sounds suspiciously
> like you have been a victim of
>
> http://httpd.apache.org/security/CVE-2011-3192.txt
>
> It would be best to upgrade to newer binaries then.

Thanks. I had partially upgraded to 2.2.22, but not all of the installed
apache stuff had been upgraded. I have now done so and will watch to see
if I get further problems. So far so good.

It seems that the attack vector is via those weird google searches. Ie,
they also seem to be spoofing google bot requests as part of the attack.


>
> Regards,
> Wolfram.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.networking

csiph-web