Suggestions for Expansion, Please

[amandler]

I am running Plesk 8.6 on Debian Linux with a dual 3.6 Xeon server, 4 GB RAM. My server is beginning to strain a bit, frequently showing load averages of 2.5 +, and occasionally getting as high at 5 or 6. Most of the effort is running spamd to manage mail. We're handling about 150,000 messages a day.

So, time to upgrade. What's the best way to go? Should I simply add another Plesk server and move some customers over there? Or is it wiser to get more power on a single server and migrate? Do I need Plesk Expand, or just another server license? I'd love to hear what's worked well for others.

BTW, I am pre-filtering about half my email (largest domains) on another server and skimming about 30% of the spam and all the viruses off before sending it on to my Plesk box.

Thanks.

 

[Amin Taheri]

I would consider moving mail to a dedicated machine, and to do spam/virus filtering with an inline appliance instead (assuming you can afford that)

 

[amandler]

That's a possible idea. Would you suggest running a second copy of Plesk on the dedicated mail server, or just using standard tools (qmail, postfix, or exim)? The pre-filter machine I already have working amounts to an "inline appliance" already.

 

[Amin Taheri]

Plesk itself doesnt really offer you any good mail management on the server level so it would be a waste of money in my opinion.

 

[Felix Buenemann]

Or simply install spamdyke and get your load down to near zero. 150,000 mails should be peanuts. My server handles 35,000 mails a day without any significant load with about 98,5% of the mails being blocked by spamdyke before they hit spamd. Spamdyke can be used as a replacement for rblsmtpd and does a lot more, google for more info.

Filtering mail through a second server should not be needed. A single spamdyke process needs about 0.7% cpu and very little memory. But make sure to limit processes in /etc/init.d/smtp(s)_psa to void DoS by creating to many mail processes (the plesk default is UNLIMITED which is totally unsane). I'd suggest to limit to max 100 smtp and 50 smtps connections, should be plenty to handle your load. Be sure to enable the idle timeout with about 180 sec, to avoid idle connections clogging up spamdyke processes.

You can further optimize spamassasin processing time by enabling the shortcircuit plugin and enabling shortcircuit RULE_NAME spam, which should bring down spamassassin processing time for spam mails that pass spamdyke to under a second per mail instead of several seconds.

These are the rules that is shortcircuited and are also the rules with the highest spam and lowest ham hit rate (as indicated by running a sa-stats-sare over long term mail logs – don't use any rules for shortcircuiting that are possible ham):

# adjust for high efficiency rules
score URIBL_BLACK 50
score URIBL_JP_SURBL 50
score RCVD_IN_BL_SPAMCOP_NET 50
score RAZOR2_CHECK 50
score BAYES_99 50
# short circuit high efficiency rules
shortcircuit URIBL_BLACK spam
shortcircuit URIBL_JP_SURBL spam
shortcircuit RCVD_IN_BL_SPAMCOP_NET spam
shortcircuit RAZOR2_CHECK spam
shortcircuit BAYES_99 spam

Also try disabling useless network checks like DCC or Pyzor which never hit any SPAM that gets past spamdyke or DNSBL and dialup host checks for that matter.


Here is a one month statistic that clearly shows that 98% of apsm is originated by dialup hosts, badly configured (compromised) servers and those already listed in DNSBLs. Most of which can be blocked without even querying a DNSBL.

Generated by my enhanced spamdyke-stats script:

353244 DENIED_RDNS_MISSING (no reverse DNS record)
275326 DENIED_IP_IN_CC_RDNS (reverse DNS record contain IP -> dialup host)
133658 DENIED_RBL_MATCH (blocked by one of the DNSBLs below)
77.75% zen.spamhaus.org
14.42% ix.dnsbl.manitu.net
7.83% bl.spamcop.net
81010 DENIED_RDNS_RESOLVE (missing reverse DNS record)
9309 ALLOWED (OK, passed on to spamassassin)
7195 TIMEOUT (timeuts by spambots not correctly closing SMTP connection)
5754 DENIED_IP_IN_RDNS (matched more DNS blacklist keywords, .static. .ppp. etc. in my case)
3862 DENIED_BLACKLIST_NAME (matches DIALUP hostnames in my case, eg. .t-dialin.net)
1276 DENIED_OTHER
565 ERROR
213 DENIED_SENDER_NO_MX (sender domain has no MX record)
11 DENIED_RHSBL_MATCH (matched a RHSBL, be careful with RHSBL, better don't use it)
100.00% dsn.rfc-ignorant.org

Allowed: 9309
Denied : 854354
Errors : 7760
Total : 871423
% Valid: 1.07%
% Spam : 98.04%
% Error: 0.89%


-- Felix

 

[Ragefast]

I agree with Felix, I recently installed spamdyke (and installing it is really a picnic in the park on a sunny day), and after that I'm getting about 80% of spam down.

Just upgrading servers aren't enough, you're just giving more resources to spammers, and will sooner than you think end up needing another upgrade. Filtering only 30% of spams is way too low in my opinion. Letting spamassassin take care of all the mail is a shot in your foot.

Investing in appliances and external filters may be a good solution but you must study them very well, always taking notice that they have all the features you need and could have and that they have room for expansions and custom rules.

BTW Felix, I'd love to have that enhanced statistic script if you wouldn't mind sharing it

 

[Felix Buenemann]

Here are both my modified sa-stat-sare and spamdyke-stats perl scripts.

– The sa-stats-sare is modified to handle bzip2 and gzip compressed logs aswell as showing all available parameters in the usage.
– The spamdyke-stats was modified to show statistics for info loglevel, listing how the hitrates in different blacklists distribute, which can help to decide the order of the BLs and which BLs to keep and which to remove (you should keep blacklist network requests to a minimum, as it slows down processing). You can pipe compressed logs into spamdyke-stats by using sth. like: bzcat /var/log/mail.info-20080[89]* | cat - /var/log/mail.info | spamdyke-stats.pl

sa-stats-sare helps tuning your spamassassin rulesets by displaying highscores for spam and ham rules hit.

Sample output:
sa-stats-sare -n 5 -l /var/log -f 'mail\.info-200809.*'


Email: 2739 Autolearn: 1886 AvgScore: 5.91 AvgScanTime: 4.18 sec
Spam: 516 Autolearn: 172 AvgScore: 37.44 AvgScanTime: 5.44 sec
Ham: 2223 Autolearn: 1714 AvgScore: -1.41 AvgScanTime: 3.88 sec

Time Spent Running SA: 3.18 hours
Time Spent Processing Spam: 0.78 hours
Time Spent Processing Ham: 2.40 hours

TOP SPAM RULES FIRED
----------------------------------------------------------------------
RANK RULE NAME COUNT %OFMAIL %OFSPAM %OFHAM
----------------------------------------------------------------------
1 BAYES_99 332 12.34 64.34 0.27
2 URIBL_BLACK 248 9.68 48.06 0.76
3 HTML_MESSAGE 239 24.06 46.32 18.89
4 URIBL_JP_SURBL 217 8.07 42.05 0.18
5 RAZOR2_CHECK 199 8.51 38.57 1.53
----------------------------------------------------------------------

TOP HAM RULES FIRED
----------------------------------------------------------------------
RANK RULE NAME COUNT %OFMAIL %OFSPAM %OFHAM
----------------------------------------------------------------------
1 BAYES_00 2015 73.93 1.94 90.64
2 AWL 1992 73.42 3.68 89.61
3 NO_RELAYS 1637 60.09 1.74 73.64
4 HTML_MESSAGE 420 24.06 46.32 18.89
5 MIME_HTML_ONLY 64 3.40 5.62 2.88
----------------------------------------------------------------------

 

[Ragefast]

Thank you very much!

 

[amandler]

Great Suggestions

That sounds like a great way to go. I plan to install Spamdykes immediately. Felix and Ragefast, I would love to see your spamdyke.conf files as a jumping off point. Many thanks for your responses.

 

[amandler]

One more question

Felix, you mention in your first post editing smtp(s)_psa to limit the number of mail processes. However, as you may be aware, with 8.6, Plesk weirdly chose to move the functionality that was in smtp_psa to the /etc/inetd.conf file. Do you know how to set those variables in that environment (and why did they do that!!!??).

 

[amandler]

Spamdyke

One more issue. I got spamdyke compiled and installed. It works fine, but interferes with qmail when running. I'm probably missing something. Here's the edited line from /etc/inetd.conf:

smtp stream tcp nowait.1000 root /var/qmail/bin/tcp-env tcp-env -Rt0 /usr/bin/env SMTPAUTH=1 POPLOCK_TIME=20 SHORTNAMES=1 END=1 /usr/local/bin/spamdyke -f /etc/spamdyke.conf /var/qmail/bin/relaylock /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true

As I said, spamdyke starts working and blocking emails immediately, but I can't restart the qmail smtp service in the Plesk control panel. Any thoughts?

 

[amandler]

addendum

In regards to my last post, in fact, it appears qmail smtp is running correctly. However, the Plesk control panel icon under "Service Management" says it isn't.

Art

 

[Felix Buenemann]

On my system with OpenSuSE 10.3 x64 qmail is still running through xinet.d, though on my debian etch plesk VM I use for testing it runs through inet.d. You could probably just reinstall/reactivate xinet.d.

The only way to throttle connections through classic inetd it to lower nowait.1000 to nowait.300, which would allow for 300 connections in 60 seconds, or in average 5 connections per second, which should suffice.

I can see from your inetd entry that spamdyke is running before relaylock. This order should be reversed.
The only thin the init-script for qmail is restarting is qmail-send, but make sure you don't have permission problems with spamdyke. On my host I am running spamdyke from /var/qmail/bin with user qmaild and group qmail, but I guess spamdyke:qmail would be OK aswell.

Here is a good spamdyke config to start with:
greeting-delay-secs=0
max-recipients=50
reject-empty-rdns
reject-ip-in-cc-rdns
reject-missing-sender-mx
reject-unresolvable-rdns
log-level=verbose
idle-timeout-secs=60
#[.. list of blacklist/whitelist files omitted ..]
dns-blacklist-entry=zen.spamhaus.org
dns-blacklist-entry=ix.dnsbl.manitu.net
dns-blacklist-entry=bl.spamcop.net
graylist-level=none
graylist-dir=/var/qmail/spamdyke/greylist
graylist-max-secs=1814400
graylist-min-secs=300
smtp-auth-level=ondemand-encrypted
smtp-auth-command=/var/qmail/bin/smtp_auth /var/qmail/bin/true
smtp-auth-command=/var/qmail/bin/cmd5checkpw /var/qmail/bin/true
tls-certificate-file=/var/qmail/control/servercert.pem
local-domains-file=/var/qmail/control/rcpthosts

 

[amandler]

Thanks so much for your help. Everything is working great. I still have the bogus "not running" icon for smtp in the Plesk interface, but I can live with that. Since yesterday at 6 p.m. spamdyke has blocked 98,000 emails, or about 95% of incoming. Server is purring. Lovely.

 

[Ragefast]

My configuration is basically the same as Felix's, with the exception I let qmail handle authentication instead of spamdyke, and I dont use graylist, mainly because of the delay. I also keep my black/whitelists all in files.

Just a warning about the 'reject-ip-in-cc-rdns' option, it will reject legitimate emails which have it's IP and a second-level ccSLD in rDNS, meaning any host called 11-22-33-44.sharedhosting.com.br for example will also be blocked. You should initially use the 'ip-in-rdns-keyword-blacklist-entry' or the 'ip-in-rdns-keyword-blacklist-file' option instead.

 

[amandler]

More spamdyke issues

I noticed today that my smtp service had been hacked. Someone was spamming using my server. They were using the "backup" user to log in, so I removed that user. I also put their IP in the blacklist_ip file, and the email address they were sending from in the blacklist_sender file. Restarted xinetd.

And yet they continued to be able to inject messages, which spamdyke allowed!

Sep 15 12:39:34 host0 spamdyke[3778]: ALLOWED from: replyback@wachovia.com to: *******@aol.com origin_ip: 75.147.110.137 origin_rdns: 75-147-110-137-philadelphia.hfc.comcastbusiness.net auth: backup

The above showing up in syslog (and the messages in my queue) after I had done everything noted above. What, please, am I missing!!

 

[Felix Buenemann]

@amandler: well you should use safe passwords of course. If the user is really deleted it shouldn't work for any new processes started. For the remaining ones a killall spamdyke would help (but the processes should be shortlived anyways).

The blacklist_sender file is irrelevant, as the user gets whitelisted by smtp auth. As a quickfix you can block the host by iptables or adding a route like route add -host 1.2.3.4 gw 127.0.0.1 which would route all packages for the bad host with src ip 1.2.3.4 to localhost.

Also currently you should disable pop-before-smtp (it's insecure by design) and short login names, because of this security hole in plesk: http://seclists.org/bugtraq/2008/Sep/0001.html (this will probably be fixed soon by swsoft)
Note that when you change the mailserver settings from the plesk control panel, it's rewrite the inetd/xnetd config files, so you'll have to add back in spamdyke (place it after relaylock in the chain).

Maybe plesk isn't running the mailsrvmng commands because it thinks qmail isn't running and so doesn't remove the user ...

@Ragefast: I'm also doing smtp auth through qmail, the parameter is set to ondemand, which only uses spamdykes auth if qmail fails to provide it.
If a server has an RDNS of 11-22-33-44.sharedhosting.com.br it means it basically has no valid RDNS so the admin should rather fix his own DNS and create a proper reverse DNS entry which matches the forward DNS lookup for the mailserver.

Anyways you could go without this onton and add the following entries to the file specified by ip-in-rdns-keyword-blacklist-file:
cat <<EOF >/var/qmail/spamdyke/blacklist_keywords
.t-dialin.net
.t-ipconnect.de
.in-addr.arpa
.dhcp
in-addr.arpa
dhcp
dialup
cable
isdn
ppp
dsl
adsl
sdsl
vdsl
dial
dynamic
host
pool
port
ippool
static
user
pc
pop
client
dialup
dialin
dip
pppool
broadband
ip
pppoe
modem
pptp
nat
internet
EOF

[Edit: Changed to correct syntax without leading and following dot. A leading dot instructs spamdyke to match this line against the end of the hostname, so .net would block everything with an ip in the hostname and the hostname ending in the .net tld.]
-- Felix

 

[amandler]

Spamdyke blacklist and whitelist issues

Hey Folks -- this thread has been incredibly helpful. Spamdyke is working great on my system, for the most part, and I'm very pleased. I still have an issue using the blacklists and whitelists. They are not working. Here are a few lines from /etc/spamdyke.conf:

sender-blacklist-file=/var/spamdyke/blacklist_senders
sender-whitelist-file=/var/spamdyke/whitelist_senders

The blacklist_senders file contains:
.t-dialin.net
fnbderidder.com
replyback@wachovia.com
hunt4loot.com

However, spamdyke continues to allow emails from hunt4loot.com, as seen in this log example:
Sep 18 23:24:48 host0 spamdyke[4399]: ALLOWED from: info@hunt4loot.com to: [email address del] origin_ip: 209.222.78.13 origin_rdns: hunt4loot.com auth: (unknown)

Similarly, I have this in my whitelist_senders file:
askitp.com
askhg.com
york-claims.com
itpros.com

and spamdyke is still denying messages from york-claims.com, as exemplified here:

Sep 19 05:42:47 host0 spamdyke[28312]: DENIED_RDNS_MISSING from: ******@york-claims.com to: [email deleted] origin_ip: 216.83.173.208 origin_rdns: (unknown) auth: (unknown)

Now, it's true that york-claims.com has a missing RDNS, but shouldn't whitelisting trump that?

I ran spamdyke -f /etc/spamdyke.conf --config-test /var/qmail/bin/qmail-smtpd and everything looks fine.

What am I missing?

Many thanks!

 

[Ragefast]

Heya

If you want to block/whitelist entire domains, you need to put them like '@domain.com' in your config file

In your blacklist case it would be something like:

@t-dialin.net
@fnbderidder.com
replyback@wachovia.com
@hunt4loot.com

Get it?

 

[amandler]

That was the problem. Thanks. All good now (except for the mysterious icon problem in Plesk). Many thanks to all of you!