Sudden raise in Panel memory usage

[200002]

Hi there,

I have a dedicated server running plesk 10.3.1 (Centos).

This night my Server Health moitoring showing a sudden raise in Panel memory usage

Panel memory usage 57.3% used (4.39 GB of 7.65 GB) as well as Swap usage 99% used (1.98 GB of 2.00 GB).

Check the attachments to see how the graphs spikes up.

This morning I couldn't reach Plesk admin panel at port :xxxx but I noticed that

Psa (plesk something) had died - so I ran /etc/init.d/psa stop and then /etc/init.d/psa start

Now the admin panel works fine again. But the high memory usage are giving me warnings in the admin interface. I only run my own sites on the server so I have no hosting customers. The websites are still working ok but I like to get the memory usage down.

In my server settings I had marked for Home> Server and Panel Settings >
"Automatically download and install updates"

I therefore figure that something has been automatically installed to the server recently. I also have this message at the plesk interface

"Information: Customer and Business Manager was successfully installed but it requires additional configuration. Click here to complete the installation."

... when I click on that link it loads https://myserver:xxxx/plesk-billing/admin/index.php and the interface gets white for a second and the interface turns up again.

The above message might have been there for a longer time, not sure. I also have a lot of sleeping processes 770.97 that occured at the same time (see second attachment)

How do I check what is causing the problem? If it is the Customer and Business Manager program can I safely remove it with command 'yum remove plesk-billing'

Is there anywhere in Plesk to check if there was anything installed at the time when the sudden raise occured or perhaps in Putty.

I would appreciate any help you can give me. Thanks.

 

[200002]

I checked the error log for Control Panel at /var/log/sw-cp-server/error_log

There is a ton of error in it. They appear a thousand times

2012-09-18 21:01:18: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:19: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:20: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:21: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:21: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 213
2012-09-18 21:01:21: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 1 load: 213
2012-09-18 21:01:21: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 2 load: 213
2012-09-18 21:01:21: (connections.c.1737) SSL (error): 5 -1 0 Success
2012-09-18 21:01:21: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:21: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:22: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:22: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:22: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:23: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:23: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:23: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:23: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe
2012-09-18 21:01:23: (connections.c.1737) SSL (error): 5 -1 0 Success
2012-09-18 21:01:23: (connections.c.1737) SSL (error): 5 -1 0 Success
2012-09-18 21:01:23: (connections.c.1737) SSL (error): 5 -1 0 Success
2012-09-18 21:01:24: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2012-09-18 21:01:24: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2012-09-18 21:01:24: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2012-09-18 21:01:24: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe

The below goes on for like 5000 rows or so

2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 1 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 2 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 3 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 4 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 5 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 1 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 2 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 3 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 4 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 5 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 1 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 2 load: 148
2012-09-20 02:42:25: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 3 load: 148
2012-09-20 02:42:27: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2012-09-20 02:42:27: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2012-09-20 02:42:27: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2012-09-20 02:42:27: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2012-09-20 02:42:27: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock
2012-09-20 02:42:27: (mod_fastcgi.c.2651) fcgi-server re-enabled: 0 /usr/local/psa/tmp/sw-engine.sock

Then this comes

2012-09-20 04:34:49: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2012-09-20 04:34:49: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2012-09-20 04:34:49: (connections.c.299) SSL: 1 error:140780E5:SSL routines:SSL23_READ:ssl handshake failure
2012-09-20 04:34:49: (mod_fastcgi.c.2588) FastCGI-stderr: PHP Warning: Cannot modify header information - headers already sent by (output started at /usr/local/psa/admin/plib/PleskException.php:44) in /usr/local/psa/admin/plib/PleskException.php on line 27

2012-09-20 04:34:49: (mod_fastcgi.c.2588) FastCGI-stderr: PHP Fatal error: Uncaught exception 'Zend_Exception' with message 'No entry is registered for key 'config'' in /usr/local/psa/admin/plib/Zend/Registry.php:145
Stack trace:
#0 /usr/local/psa/admin/plib/CommonPanel/Exception.php(84): Zend_Registry::get('config')
#1 /usr/local/psa/admin/plib/CommonPanel/Exception.php(36): CommonPanel_Exception::_sendRuntimeReportXML(Object(PleskFatalException))
#2 /usr/local/psa/admin/plib/PleskException.php(49): CommonPanel_Exception::sendNotification(Object(PleskFatalException))
#3 /usr/local/psa/admin/plib/PleskException.php(10): report_crash('Unable to conne...', Array, 'PleskFatalExcep...', 500, Object(PleskFatalException))
#4 [internal function]: plesk_exception_handler(Object(PleskFatalException))
#5 {main}
thrown in /usr/local/psa/admin/plib/Zend/Registry.php on line 145

and then this

2012-09-20 04:35:30: (mod_fastcgi.c.2588) FastCGI-stderr: PHP Warning: mysql_query(): 53 is not a valid MySQL-Link resource; File: /usr/local/psa/admin/plib/common_func.php3, Line: 146

PHP Warning: mysql_error(): 53 is not a valid MySQL-Link resource; File: /usr/local/psa/admin/plib/common_func.php3, Line: 156

PHP Warning: mysql_errno(): 53 is not a valid MySQL-Link resource; File: /usr/local/psa/admin/plib/common_func.php3, Line: 156


2012-09-20 04:35:30: (connections.c.1737) SSL (error): 5 -1 0 Success
2012-09-20 04:35:30: (connections.c.1737) SSL (error): 5 -1 0 Success
2012-09-20 04:35:30: (connections.c.1737) SSL (error): 5 -1 32 Broken pipe

2012-09-20 04:37:55: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 408
2012-09-20 04:37:55: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 408
2012-09-20 04:37:55: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 408
2012-09-20 04:37:55: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 408
2012-09-20 04:37:55: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 408
2012-09-20 04:37:55: (mod_fastcgi.c.2873) backend is overloaded, we disable it for a 2 seconds and send the request to another backend instead: reconnects: 0 load: 408

Pleask admin died but after a restart (command: /etc/init.d/psa stop and then /etc/init.d/psa start) this is in the log

2012-09-20 04:38:34: (mod_fastcgi.c.3265) response not received, request sent: 652 on socket: unix:/usr/local/psa/tmp/sw-engine.sock-0 for /login_up.php3 , closing connection
2012-09-20 04:38:34: (mod_fastcgi.c.2482) unexpected end-of-file (perhaps the fastcgi process died): pid: 20508 socket: unix:/usr/local/psa/tmp/sw-engine.sock-0
2012-09-20 04:38:34: (mod_fastcgi.c.3222) child signaled: 9
2012-09-20 11:09:18: (log.c.75) server started
2012-09-20 11:09:37: (connections.c.299) SSL: 1 error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure
2012-09-20 11:09:37: (connections.c.1737) SSL (error): 5 -1 0 Success
2012-09-20 11:09:41: (connections.c.1737) SSL (error): 5 -1 0 Success
2012-09-21 11:09:45: (connections.c.1737) SSL (error): 5 -1 0 Success

Websites works fine but Panel memory uses 56.8% of total memory
4.35 GB of 7.65 GB memory.

 

[200002]

Brute force attack on plesk

I read here http://forum.parallels.com/pda/index.php/t-208198.html about
brute force attack on plesk so I checked my file /var/log/secure

How do I block these ip:s? Will it help to "Disallow root authentication through SSH" and where do I do that?

Sep 21 05:44:46 myservername sshd[1107]: Received disconnect from xxx.xx.xx.xx: 11: Bye Bye
Sep 21 05:44:49 myservername sshd[1117]: Invalid user allsunday from xxx.xx.xx.xx
Sep 21 05:44:49 myservername sshd[1118]: input_userauth_request: invalid user allsunday
Sep 21 05:44:49 myservername sshd[1117]: pam_unix(sshd:auth): check pass; user unknown
Sep 21 05:44:49 myservername sshd[1117]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.xx.xx.xx
Sep 21 05:44:49 myservername sshd[1117]: pam_succeed_if(sshd:auth): error retrieving information about user allsunday
Sep 21 05:44:51 myservername sshd[1117]: Failed password for invalid user allsunday from xxx.xx.xx.xx port 56985 ssh2
Sep 21 05:44:52 myservername sshd[1118]: Received disconnect from xxx.xx.xx.xx: 11: Bye Bye
Sep 21 05:44:54 myservername sshd[1123]: Invalid user alonso from xxx.xx.xx.xx
Sep 21 05:44:54 myservername sshd[1124]: input_userauth_request: invalid user alonso
Sep 21 05:44:54 myservername sshd[1123]: pam_unix(sshd:auth): check pass; user unknown
Sep 21 05:44:54 myservername sshd[1123]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.xx.xx.xx
Sep 21 05:44:54 myservername sshd[1123]: pam_succeed_if(sshd:auth): error retrieving information about user alonso
Sep 21 05:44:56 myservername sshd[1123]: Failed password for invalid user alonso from xxx.xx.xx.xx port 57306 ssh2
Sep 21 05:44:57 myservername sshd[1124]: Received disconnect from xxx.xx.xx.xx: 11: Bye Bye
Sep 21 05:45:03 myservername sshd[1132]: Invalid user alsumic from xxx.xx.xx.xx
Sep 21 05:45:03 myservername sshd[1138]: input_userauth_request: invalid user alsumic
Sep 21 05:45:03 myservername sshd[1132]: pam_unix(sshd:auth): check pass; user unknown
Sep 21 05:45:03 myservername sshd[1132]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.xx.xx.xx

 

[200002]

Here's some additional info in the security file:

Sep 20 14:46:43 myservername sshd[62337]: reverse mapping checking getaddrinfo for hosted-by.altushost.com [xx.xx.xxx.xx] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 20 14:46:43 myservername sshd[62337]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xx.xx.xxx.xx user=root
Sep 20 14:46:43 myservername sshd[62338]: reverse mapping checking getaddrinfo for hosted-by.altushost.com [xxx.xx.xxx.xx] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 20 14:46:43 myservername sshd[62339]: reverse mapping checking getaddrinfo for hosted-by.altushost.com [xx.xx.xxx.xx] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 20 14:46:43 myservernamesshd[62340]: reverse mapping checking getaddrinfo for hosted-by.altushost.com [xx.xx.xxx.xx] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 20 14:46:43 myservername sshd[62338]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.xx.xxx.xx user=root
Sep 20 14:46:43 lmyservername sshd[62340]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.xx.xxx.xx user=root
Sep 20 14:46:43 lmyservername sshd[62339]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.xx.xxx.xx user=root
Sep 20 14:46:43 myservername sshd[62324]: Failed password for root from x.xx.xxx.xx port 32889 ssh2
Sep 20 14:46:43 myservername sshd[62325]: Received disconnect from x.xx.xxx.xx: 11: Bye Bye

Are these the steps that I should follow http://kb.parallels.com/en/8119

Or is there something else that can be done?

 

[TobyP]

Did you ever sort this out?

I had a similar issue and after lots of searching and googling did something with fastcgi (can't remember what) and it solved the memory issue instantly.

Now I have a 99.9% cpu usage and 200+ sleeping processes.