Server being used to send spam - can't find maillog

[TobyJP]

Hi all, we have a server running Plesk 11.0.9 with around 90 client domains on it. A number of our clients have experienced difficulties sending/receiving mails, and we've been added to a spam blacklist in the past.

The mail queue often holds upwards of 1,500 spam emails, which I have to remove by hand. I've tried following a few other forum posts here, including this article in the knowledgebase. I always get stuck when trying to access the log at /usr/local/psa/var/log/maillog and just get a "No such file or directory" message. I've tried changing directory command one by one until I get to /usr/local/psa/var/ and then there's no log directory.

The Plesk version was upgraded recently from 9.5 and the OS was upgraded to Debian 6 shortly before that. Any assistance gratefully received. Thanks.

 

[Faris Raouf]

I'm not very familiar with where Debian puts things, but if the maillog isn't where it should be, try searching for it:

locate maillog

If it doesn't exist then something more serious is wrong. Do you have ANY logs with recent data in them? most logs are in /var/log for most distributions.

Again I emphasise that I am not familiar with Debian. So all I can do is give you a hint of what to look for. Basically, logs are created by some sort of syslog daemon. In the old days it was "syslogd". But there's also "rsyslog" and "syslog-ng" and also "sysklogd" or "klogd".

If the appropriate service used by Debian 6 is not running then you won't have any logs.

A quick google search makes me think rsyslog is the most likely syslog server for Debian 6 but I could be wrong.
Check to see if that's installed and if it is meant to start automatically. It will write to /var/log/messages (I think) if there's an error on startup due to a configuration issue. But if it isn't /var/log/messages try /var/log/rsyslog or something along those lines.


Please do not be alarmed - this sort of thing can easily happen. The service may stop, be incorrectly configured, or ...there are lots of reasons why it might not be running. But it is also possible that your box may have been compromised, and the bad guys have deliberately disabled logging to prevent records of what they are doing. This is unlikely to have happened - it is more likely that something went wrong during an upgrade.

I hope someone more familiar with Debian will post here so that we can get more accurate information for you.

 

[TobyJP]

Thanks for your reply @Faris. I've done the locate command and got the following:

/opt/psa/var/log/maillog
/opt/psa/var/log/maillog.processed.1.gz
/opt/psa/var/log/maillog.processed.2
/opt/psa/var/log/maillog.processed.2.gz
/opt/psa/var/log/maillog.processed.3.gz
/usr/share/doc/awstats/examples/maillogconvert.pl
/var/lib/mysql/support_whmcs/tblticketmaillog.MYD
/var/lib/mysql/support_whmcs/tblticketmaillog.MYI
/var/lib/mysql/support_whmcs/tblticketmaillog.frm

So I looked in /opt/psa/var/log/maillog and found what I was looking for. Still not sure who's sending all this spam though!

 

[Faris Raouf]

Excellent. I'm glad you found it. Yup, looks like /opt/psa/var/log/maillog is the one you need.

It can be very hard to locate the sender, expecially since the maillog is difficult to interpret when there's a huge number of entries to go through.

Who are the emails "from"? Typically (but not universally), if they are "from" something@your-server-hostname then it will be a php (or possibly perl) script running on the server. Depending on your version of php, the name of the script will be in the email's headers. If not, take a look at this: http://kb.parallels.com/en/1711 (qmail) or this more modern one http://kb.parallels.com/en/114845 (postfix).

*** do make backups of any files you might change, and make a note of who owns them and what read/write/execute permissions they have, so that you can put things back the way they were is something goes wrong.

If it isn't a script, the KB you linked to in your first post will be particularly useful. That KB also gives you a hint about a different way to tell if a script is sending the messages as opposed to a compromised email account.

/var/qmail/queue/mess/ is where outgoing messages are queued before they get sent. Take a look at one of them that's a spam message. The header in that will probably give you more clues than anything about the origin.

Then some more detective work using the original KB you linked to will hopefully help you locate either the script or the compromised email account.

 

[mrtripps]

In the plesk Tools & Settings take a look at the Traffic Usage by Domain. Sort on the 'used' column. Click through the links for the domains near the top of the table and check the pop3/imap usage. If it is high then you can narrow it down to email accounts on certain domains. I would then take a look at the message log file to look for any entries related to those domains:

tail -n 500 /var/log/messages <-- this will look at the last 500 lines of the file.

If you find any entries for the affected domains paste the output here. They may record IP's that you can block to clear this up. Also once you locate affected domains, you may want to update the email password for any email accounts on those domains to secure them.

 

[TobyJP]

@mrtripps - below is just a tiny example of some of the suspicious activity on our mailserver:

Feb 12 09:06:06 host18 postfix/qmgr[2603]: D771834A2E96: from=<user1@domain.com>, size=741, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: D044234A2EA4: from=<user1@domain.com>, size=705, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: BFD5A34A2EB1: from=<user1@domain.com>, size=729, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: BA7CC34A2AF1: from=<user1@domain.com>, size=748, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: B358334A2E78: from=<user1@domain.com>, size=718, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: BD1EA34A2F3E: from=<user1@domain.com>, size=733, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 8426A34A2E0A: from=<user1@domain.com>, size=721, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 5F0B134A2F9D: from=<info@domain.com>, size=716, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 5B86934A2CD1: from=<user1@domain.com>, size=723, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 580BA34A2ADD: from=<user1@domain.com>, size=754, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: E273834A2AED: from=<user1@domain.com>, size=764, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: E168834A2F74: from=<info@domain.com>, size=717, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: E124E34A2E11: from=<user1@domain.com>, size=708, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: ED41D34A2C18: from=<user1@domain.com>, size=756, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 72AF034A2F8A: from=<info@domain.com>, size=731, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 7DB4134A2F4B: from=<user1@domain.com>, size=731, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 7D84134A2ADC: from=<user1@domain.com>, size=740, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 7CDB234A2E91: from=<user1@domain.com>, size=711, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 0949B34A2F4D: from=<user1@domain.com>, size=721, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 092BE34A2C12: from=<user1@domain.com>, size=725, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 0765E34A2E7A: from=<user1@domain.com>, size=729, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 01FE634A2C1C: from=<user1@domain.com>, size=755, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 07EE134A2C48: from=<user1@domain.com>, size=788, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 3535134A2E79: from=<user1@domain.com>, size=708, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 3262334A2C0B: from=<user1@domain.com>, size=746, nrcpt=1 (queue active)
Feb 12 09:06:06 host18 postfix/qmgr[2603]: 3F2B334A2F6C: from=<user1@domain.com>, size=760, nrcpt=1 (queue active)

I've changed the domain name to "domain.com" and changed the username to "user1". Each morning user1 gets nearly 3,000 undelivered mail messages, and the 'info' account gets around 1,000.

We're also seeing lots of entries like this:

Feb 12 07:53:17 host18 postfix/smtpd[26805]: BB69434A2D9D: client=host-78-146-241-171.as13285.net[78.146.241.171], sasl_method=LOGIN, sasl_username=user1@domain.com
Feb 12 07:57:08 host18 postfix/smtpd[27243]: C04EE34A2BAF: client=host-78-146-241-171.as13285.net[78.146.241.171], sasl_method=LOGIN, sasl_username=user1@domain.com

...suggesting that someone is logging into the mailserver and successfully authenticating using user1's email password. Weird thing is, I've already changed their password to an 8-digit randomly generated one - definitely not a dictionary attack.

Any suggestions?

 

[Faris Raouf]

Firewall that IP to start
Then restart postfix.

In the past I *think* I've seen a similar situation where changing the password for a mailbox being used for an in-progress spam run didn't seem to stop things. But this was with qmail. Restarting it seemed to resolve the problem.

I can't honestly say if this is just my memory playing tricks on me, misunderstanding what was going on due to the panic at the time or if there's some sort of caching going on -- or none of the above

But a restart won't do any harm.

However, keep in mind that messages already in the queue will still get sent. You'll need to purge the spam messages from the queue, of which there may be lots and lots.

 

[EgidijusS]

To prevent of spamming you can try using of csf. And there you can find guide, how to install on plesk server. I'm using that for long time.