Out of Memory Crashes - Daily at 4 AM

[petroski2409]

Hello,

My server, which has been stable for months, has recently started having meltdowns at 4 AM every morning due to an Out of Memory event. The server shuts down and restarts all processes and carries on until the next event the next day.

The current environment particulars of this server are CentOS 5.9 (x86_64), Plesk 9.5.4, mySQL Ver 14.14 Distrib 5.5.32, PHP 5.3.25, Zend 2.3.0, ionCube 4.4.1 with all CentOS yum updates applied. I also subscribe to AtomiCorp's security product, Atomic Secured Linux version 3.2.14-31. My Linux kernel version is 3.2.46-53.art.x86_64

This all began on June 18th due to an update (from what I can tell) of mySQL and PHP. The reason that I note this is because the configuration files - my.cnf and php.ini, were replaced with new versions on that date. The reason that I suspect rather than know is that ASL uses yum to apply security updates and I am assuming that the mySQL, PHP and Linux kernels were applied at that time.

See link to messages-log.txt

(NOTE: this is a snippet, event shown repeats over and over ballooning the message log to plus/minus 20GB in a very short period of time ...)

Oh yes, IP address 77.10.98.79 is now blacklisted ...

See link to mysqld-log.txt

Anyone seem this behaviour before? Any ideas at all?

Thanks in advance for your assistance!

Peter

 

[petroski2409]

OOM Perl event caused by Logwatch

Hello,

Further to above issue, I have been able to track the OOM perl event to the logwatch application that runs every morning at 4 AM. With that information, I am able to, by running logwatch from the command line, to replicate the same issues consistently. FRom that, I was able to trace the PID and then to run strace on the execution of logwatch.pl

In looking through the log generated by strace, I'm seeing a lot of errors such as "No such file or directory", "Inappropriate ioctl for device", "Invalid Argument" and "Illegal Seek"

As I am not familiar with what the expected output is, I am hoping that someone might be able to shed some light on my problem. I am attaching the strace output file.

Thanks,
Peter