Reducing server load due to spam fighting, Heres by idea what yous yours ?

Ok as we all know and I've said before mail servers biggest task is usually spam filtering. And well it should be it's the most labor intensive filtering there is. I mean after all it has to look for : Multi word pairs, Bad grammar, Bad Spelling, Illegal char sets, forged headers, plus what ever rules you have it use. All this adds up to some pretty heavily loaded server during peak times.

But there are things you can do to help with this issue. My personal favorite and the one I'll talk about here is *TRUE* white listing or as I call it whitelightlisting. Which has nothing to do with the way SA or XSPCDA or even ASSP white list. Whitelightlisting fallows this simple idea. If you email some one you'll probably want to hear back from them. And although they may send you a virus (due to infection only you hope) or FWD's you don't find funny. They'll never really SPAM you.

So Heres how it works every time you email some one it's recorded some where (like in mysql or sqlite or even just a tabbed flat file). And this entry tell you 4 things.

#1. Who's this message going to
#2. Who you are
#3. When you last sent this person a email

Now having that all recorded some where gives you a lot of power. Because with that you can now prefilter all email before they hit your spam filter(s). And only have those emails email that are NOT from people you have sent to get scanned by the spam filter. And lets be real I know DB connection can be resource expensive, but you tell me is 1 simple connection per email more to less expensive then Spam filtering ever incoming message ?


Now for the catches:

#1. Some one could forge the �FROM� and unless you do some kinda validation and forged FROM ends up being some one on your whitelightlistin. It will not be filtered for spam. But this is really far and few between.

#2. The storage for this kinda thing can get kinda BIG and slow. So depending on your user count & feqancy of new people your users send to will dictate what back end you'll need to use (I'd just start and stay with mysql, it's GPL and works on any OS). Also you'll need to purge old records from time to time (thats why there the last time sent field)

wow, and I had thought this would have started people brain storming about other ways to help "ease" heavly loaded servers.

I have to say that I have never really thought about NOT filtering some messages to lower CPU-load, but it does start to sound like a good idea. Most of the time, we are looking at EXTRA filters (like greylisting) to lower the amount of spam.

I'm currently hoping that using Java instead of Perl will lower CPU-load. But haven't been able to test-drive it...

Very simular to your idea: we're currently seeing large amounts of "Delivery notifications" telling us some server was unable to deliver spam of which our address was the FROM-address, but that we obviously didn't send. With your database, you might also be able to figure out which messages delivery notification are about messages that really where sent by the person to which the error message is going...

Thats a good and somewhat easy problem to fix, as I've had to deal with many times now. Basicly are strategy in the fight against "unable to deliver spam which we did not send". Is to use are whitelightlisting to see what domains / users we've tried to send to if we get a bounce message from a user &/or domain we never sent to then we just end the end the transaction.

Now this method does require a little tuning from time to time as each server send a different kind of bounce message. So we need to update are "Look for this string in subject file. but it works very well for us.

Oh and as a side note, Java will end up being slower...alot slower sorry.

What perl scripts are running slow for you, I might be able to help.

Looking at this from an admin point of view, I think something like an XQM style interface would be the selling point of it.

On a side note, this sounds interesting, even more so if it could be off-loaded onto another server dedicated to mysql. But, isn't this function something you stated to not do in your antispam/assp article?

Doesn't ASSP provide the same function or am I reading that wrong? Either way, I like this better as I think it could off-load some of the functions and cpu time to another machine.

Not being a programmer this may sound novice, but wouldn't PHP be better tying into the database?

I think I'm getting alittle confused between the merging of the topics ASSP & This one. So I'll be answering this like I think I understand which topic where on.

I think your referring to the whitelightlisting so inreguards to that. There really is no need for a admin interface. it's pretty just self contained. As users send emails it records where to whom there send it to. and after X time of not sending to that user. That record is deleted. (But I guess I could see the need to delete a record sooner then X time... So ok I could see the need for an admin interface for that. Or for getting up global record that don't expire and users can't change.)

I'll try make up a simple php admin interface.

Any time your dealing with a DB thats not a flat (example would be sqlite) file you can always house the DB some where else.

Nope, this is something very different. With ASSP I recommoneded not using it's local validation. And that you should let XMail decide if it should allow replaying for not. This just decides who XMail should run filter against and who it shouldn't.

Also note that this method + ASSP don't combin overly well. As ASSP runs outside and before XMail. So there is no way for XMail to tell ASSP not to stop this message...

That being side, you can setup the whitelightlisting filter to update a ASSP file that lists all the email address you don't want it to stop / scan reguardless of that tests it fails.
(These email address also won't count against or for your spam corpes though)

Kinda, But ASSPs way there is still a chance and email could be blocked by failing another of it's tests. Also remember I never mentioned this being a addon to ASSP until this "post".

If you really want to off load some of the load from your mail server you can also move ASSP to a differnt server infront of your mail server (aka in series, or from your mail server use iptables to reroute the traffic to the ASSP server.). Becuase ASSP works as a proxy it doesn't require being on the same box.

Side Note #1 : Things that can be run on a differnt server that XMail might use. with needing to readdress or reoute anything dealing with your mail

1) SA
2) mysql
3) clam


Side Note #2 : Talking about the off loading stuff brings up another "topic" I'll be starting. It's more of a step by step on how to install a transparent mail filtering system. Basicly the idea works like this, You or I can go to any site and without readdressing, retouting or changing anything that on there network. You or I can put inplace a mail filtering system that no one knows is there. (aka antivirus,antispam,etc)

And in some cases I end up putting in a temp mail server this way just to see if they like the "new" system (in many cases xmail) over there current system or becuase I need to take down there current system and they can't afford OR don't want the down time that comes with some upgrades / changes to there servers.

The problem basicly was that our current Perl script needed some work, and... we prefer working with Java :-)
I'm not too sure if Java would be slower. Java has got the benefit of being pre-compiled. And it's been some time since the old days, when java was indeed really sluggish.
We're currently already using Java for our greylisting. Not too many clients have activated greylisting, but a small Java program is now being launched at every SMTP-connection made to our server. Most if the time the only thing it does is see in the users profile that it doesn't have to do anything. But it does give some idea.

One other benefit would be that, withing Java, we would talk directly to SpamD and ClamD via a socket. This could probably also be done in Perl, but our current Perl script saves the e-mail to yet an other temporary file for almost every check it does. So if Java is slower, then maybe that can make up (a bit) for this.

Once we have a more-or-less working version, I'll try to benchmark it and let you know my findings.

I look forward to seeing the results of that test.

And as far as perl talking with sockets, it could do so and without creating a temp file.... on the other hand so could php, which is a "lang" i preferr. (For if nothing else. It's what I've had to work with more and more)

And again if you'd like any help with the perl script let me know. And I'll see what I can do