How do I find out which user is crashing apache?

[Amin Taheri]

Hello,

One of my servers has lots of users, and one of them is repetedly crashing httpd by using all available memory on the box, whether its malicious or just out of bad code.

Since lsof, apachetop, fuser, top, etc will not show this, how can I track down which user is using which files, and which files are using the memory/cpu on the system?

 

[64bithost.com]

problem -

I had the same problem once. I allowed Tomcat to run on the server w/Redhat 5.0 and Java. The customer in question had a log file that was huge because of the errors in his code. look through the log files on your server and see who has the largest error log file and then temporarily shut them down.

regards

Chris

 

[Amin Taheri]

unfortunately that doesnt work for me in this case, no one has more then a few mb.

 

[Linulex]

Look at these directives to put in your httpd.conf mainfile:

RLimitCPU
RLimitMEM
RLimitNPROC

I find it helps a lot.

We also have webmin on our server and this has a very handy module called "system and server status". We use that to auto restart apache when a serverload is over 15 or there are more then 150 apache servers running.

regards
Jan

 

[Amin Taheri]

Thanks, I have already set those globally, but unfortunately The RLimit directives only apply to processes which are forked from Apache, such as CGI scripts. Things like PHP and mod_perl are not forked (they are executed within httpd) so RLimit will have no effect on them.

I can easily write a cron or a script that runs when the load is high, but that doesnt tell me who is causing the problems - I would like to know that so we can contact them or take action rather then restarting apache and just ignoring the underlying problem.

 

[Linulex]

we also use mod_cband with a general "googlebot_class" to stop searche engines from taking down a server in a few minutes. A poor programmed webshop with a few thousand items, that does 5 queries per page is a fieldday for google. im not naming names and sertenly not oscommerce. We dont use it to limit bandwith or so, just a single class config that holds all the ip's of knows search engines. Sites will still get spidered, but a spider cant bring a complete server down anymore. We compiled the list of ip addresses ourself from http://iplists.com
The mod_cband website is offline for a while now but there are plenty of other sites you can download it from.
http://www.montanalinux.org/mod_cband.html is one of them

atomicrocketturtle had a src rpm of it on
http://3es.atomicrocketturtle.com/packages/mod_cband/

Another option is a simple cron script that checks the load every x minutes and if the load is over 15 or so have it write http://localhost/server-status to a file. even is you have to restart the server trough the apc you still can see in the file what virtual host created the load.

regards
Jan

 

[Amin Taheri]

Hi,

Thanks for that, we actually already have mod_cband installed from ARTs repository - its nice for finding traffic hogs and seeing who is peering the last episode of lost but I havent found it usefull for finding some one consuming too many resources - for example a user who has a never ending loop will consume the entire servers resources - and since the traffic connection would be one mod cband wouldnt see it. Since its not forked apaches limit directives wouldnt kill it.

It would just kill the server, and in which case you could write a script that auto restarted apache but that doesnt solve the problem since the user could just hit that page every few minutes and tank your server for ever.

 

[Linulex]

did you think of spam? A non-protected, poorly programmed mailform can bring a server down very fast to onces spammers discover it. We use a script in our qmail that writes a log entry every mail that has been send with the server. If qmail is invoked by apache it logs the directory the script that invokted it in is in. here it is:

#!/usr/bin/perl -w

# use strict;
use Env;
my $date = `date`;
chomp $date;
open (INFO, ">>/var/log/formmail") || die "Failed to open file ::$!";
my $uid = $>;
my @info = getpwuid($uid);
if($REMOTE_ADDR) {
print INFO "$date - $REMOTE_ADDR ran $SCRIPT_NAME at $SERVER_NAME \n";
}
else {
print INFO "$date - $PWD - @info\n";
}
my $mailprog = '/var/qmail/bin/sendmail.real';
foreach (@ARGV) {
$arg="$arg" . " $_";
}
open (MAIL,"|$mailprog $arg") || die "cannot open $mailprog: $!\n";
while (<STDIN> ) {
print MAIL;
}
close (INFO);
close (MAIL);

usage:

- rename /var/qmail/bin/sendmail to /var/qmail/bin/sendmail.real (preserve user/group and rights)
- name this script sendmail and place it in /var/qmail/bin (same user/group and rights as the original)
- create /var/log/formmail or whatever you want to call it, but adjust it in the script if you do rename it.

make sure the log is writeble by everyoner who needs it, qmail, apache, root, etc. everyone who can send mail from your server. I just make mine 666. And dont forget log rotation. I find it usefull to rotate it daily and keep it 30 days.

regards
Jan

PS: not my copyright, found it on this forum some years ago

 

[Amin Taheri]

It is possible sure, in my specific example our email is done on a central server and not local to any that host web sites so we have some appliances in line that we can use to detect mail queue thresholds and hourly/daily outbound qouta enforcement per vhost but for single server aproaches that could be helpful.

 

[postman]

Just pitching in here in case it might help, as I had recent experiences of trying to track stuff like this down.
The apache log format can be amended to log the process id of the httpd process (I think it is %P or %p). If you have the pid from 'top' or 'ps -elf' of the errand httpd process you could try grepping out the pid from the access logs to narrow down the problem.
Just an idea, hope it helps,
postman

 

[Amin Taheri]

Thats a good idea - I tried using the %p and %P and I didn't see the process listed on the access_logs though.

I adeded To the log format section

LogFormat "%P" process

I also tried
LogFormat "%p %P %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

and restarted httpd each time but I don't see any change in the access logs for the vhosts in what it logs. Did I not do it correctly?