How to stop high number of Httpd processes crashing server ?

[Paul_DY]

Hope this is the right place to post this question.

Our server is running; Plesk 11.0.9 and CentOS 5.7 it has a Q8200 CPU @ 2.33GHz and 2GB of RAM. Now there are just two websites on the server plus a couple of redirects/forwarding domains, although lots of domains are still on the server but turned off in Plesk. Both websites are OSCommerce sites and I just need to keep these sites going until the end of the year when we will switch to our new Joomla based website.

Sadly our web designer who was the guy who last maintained the server, passed away at the beginning of the year and I have had to step in and from knowing nothing about LINUX servers have been frantically learning as much as I could but now I seem to have reached the end of my abilities !!

We have seen an increasing number of server crashes and after various checks of the logs, fitting a new BIOS battery, check of the hardware by EasySpace who host the server, installation of ClamAV, LMD and RKHunter (which did find some Trojans and Suspect software), I have traced it down to some external Http activity that is taking all of my CPU time and RAM. Here is a screen capture of the Htop listing and when I killed these processes the CPU and RAM went back to normal. The problem is that I usually have to restart the HTTPD service and sometimes things get so bad that the server crashes and I have to request a power cycle.

Frankly it is driving me crazy and I just do not know what to do next - any ideas ? Are there any extensions for Plesk that might help ?

 

[InderS]

Hello,

I will suggest you try to install nginx on your server so that you will not face any server crash issues. OR try to optimize your current Apache services on your serve.

 

[Linulex]

Our server is running .... CentOS 5.7

Centos 5 is on 5.10 atm. so the first thing you should do is update the OS.

Both websites are OSCommerce sites and I just need to keep these sites going until the end of the year when we will switch to our new Joomla based website.

What version of OSCommerce, The latest i hope? An old version that is on there for months and months ... "till the new site is ready" is a wet dream of every hacker/spammer.

installation of ClamAV, LMD and RKHunter (which did find some Trojans and Suspect software)

suspect software in rkhunter is normal. rkhunter is something you need to install right after the cd install and you must keep the config up to date. A new version of cp or mv by redhat/centos and will start giving errors.

It depends where the trojans are on what action to take:
in the website? ==> see comment earlier about old oscommerce
at the server? ==> enter cd, format hdd.

This may all sound hard, but im afraid that as long hackers/spammers/scriptkidies see that you run old (read insecure) scripts and/or OS they will keep trying. No matter how many modules you install. nginx might work for a while (10 minutes or so) but oscommerce is mysql heavy and 250 mysql connections are 250 mysql connections, mysql doesnt use less memory when the connections are started via nginx or via apache.

regards
Jan

 

[Paul_DY]

Hi Jan,

Thank you for your very useful reply. I will start following some of your suggestions.

In the meantime, I have some more information. It was clear that the processes that were taking all of the CPU and Memory were Apache processes and as the processes in the Htop listing that I previously posted were appearing to run some a shell script in the /TMP folder. On checking this folder, I found a number of hidden folders and directories.

After deleting these hidden folders and files (only the Apache User ones), the server was still up and running the following morning (which was a good sign), except that one CPU core had 100% utililisation.

I checked the /TMP folder again and there was one hidden folder ".jagosoxixs" which contained a folder called 'st' and a file 'st.tgz'. Digging down in the the 'st' sub-folder this had some *.txt (certificate request and private key files) an eb.php file and an index.html file for some Spanish website offering 500 euros !!

It all looked very suspicious but it is not clear as to what type of malicious activity this was. Is my server being used for a DDoS attack or is this some other hack. How are they getting these files on to my server and how do I seal up the hole ?

I will now delete this new hidden folder and keep monitoring but if you can give me any advice now that I have narrowed the symptoms down a bit it would be great.

Best regards
PAUL

PS Update - RKHunter also found some suspicious files in /var/tmp and they were the same sort of hidden folders and st.tgz files, so have got rid of these and killed the MySQL and Perl processes that were using the deleted files

PPS There are a lot of Passenger folders in /tmp and these are also being reported as suspicious by RKHunter but I am not sure if our OSCommerce site even uses Ruby or Passenger. Will I do any harm if I delete these Passenger folders ?

 

[Linulex]

Paul,

You can start with mounting /tmp as no-exec. That would help until you have a new server online later this week.

The uploading of "stuff" into /tmp is done via a website and my bet is on the ancient oscommerse you talked about. Until they are removed/fixed they will upload it again and again.

As you do not have the knowledge to fix this i would recommend hiring someone to check your server indept. If i am in over my head i always use the services of http://bobcares.com. Here is what i would do:

- disable the oscommerse
- reinstall the OS


even if you turn every stone (bit), you can never be 100% sure everything is gone from the server, hence the reinstall.
They uploaded the stuff trough a website and if the only 2 websites on it are 2 old oscommerces then its pretty clear what way they used.

look up "oscommerce security" and you will get About 2.990.000 results

But that is true for every open source script out there: if you dont have the latest version you are open to hackers, and the more popular a script is, the more hackers will attack it. Simply because they will get more results if 1% of the wordpress sites isnt updates then when they attack some script that is only used on 100 sites.

regards
Jan

 

[Paul_DY]

Hi Jan,

Thanks very much for taking the time and effort to reply to my plea.

I need to give this some serious thought as I am paranoid about breaking something on the current server and so perhaps an expert like bobcares.com is the answer - I think I have taken this as far as I can on my IT abilities.

Can all of these updates to PHP, Apache, OpenSSL, OpenSSH, OSCommerce, CentOS, Plesk, etc. be done without rebuilding the server or breaking anything ? Is there a recommended order that they need to be updated in ?

Regards
PAUL

 

[Linulex]

Paul,

no problem.

I can not answer what can be updated and what not as this depends on a zillion factors.

- what version of oscommerse is it now?
- what version of php does this need/run on?
- are there add-ons/modules?
- are they compatible with the latest oscommerse version?
- are there users in oscommerse that can upload anything?
- what version of php does the latest oscommerse version needs?
- what version is mysql?
- what version of mysql does the latest oscommerse version needs?
- is mysql using myisam or innodb?
- if you are using myisam, does the latest version of oscommerse run on myisam or does that need innodb?
- ....
- ....
- i am sure i missed a few things

Thats for oscommerse alone.

Plus, if the server was rooted, it will not go away with updating. /tmp is used to upload stuff, this can be a anything from a spam script to a rootkit and if it was a rootkit they will have replaced cp, top, etc to hide the stuff they uploaded.

If there are malicious users in oscommerce that can upload things and you dont clean them then they will still be pressent in an updated oscommerce version.

They only advise i can give in all honesty is what i sayd before

- reinstall the server

My way of handling this would be to place a new server next to it and migratie the sites with the migration manager. This way you have the sites and not any possible OS infection. After that the sites need to be checked, users checked, updated to the latest version, etc...

You can try to mount /tmp as noexec but if there is a rootkit on your server and they start using it, chances are big that your provider will take down the server.

regards
Jan

 

[Paul_DY]

Hi Jan,

These are all the questions that I started to ask myself and found I did not have the answer to !!

I had a vision of me in 2-3 wks time still struggling to learn all of these new skills, with even more grey hair and all of my normal development work completely stalled.

Time to find someone who knows how to do this sort of thing - thanks again for your advice.

Best regards
PAUL

 

[HMoellendorf87]

Dear Paul,

maybe I can assist to find the problem. Please write a MN to me with all current knowledge of the situation and I'll try to assist you being get rid of this issue asap.