Log in

No account? Create an account
FF Sparks (Casual)

Connecticut post-mortem, and Mail Woes

The Connecticut trip was good overall. I think I probably didn't get enough sleep, but it was really productive overall; seeing everything everyone else was doing gave me a lot clearer vision of where my own code needs to go, and I feel very revitalized with regards to knowing where my own designs need to go. Definitely looking forward to doing it again sometime!

We even had one evening of relaxing, when we went out on Andy's boat (an aging Bayliner) with Naomi and a local friend of Andy and Scott's, and motored along to a lakeside restaurant. Then we parked in the middle of the lake and sang songs at the top of our lungs -- badly -- along to Andy's iPod, which was hooked into the boat's radio. Was a lot of fun. :)

Getting back, however, I discovered that the spam load had killed my antispam solution once more; mail was being dropped, lost, spam was getting through when it shouldn't, generally it was just a horrible mess. So I ripped things apart -- and frustrated myself immensely in the process -- and rebuilt the antispam/antivirus for Noderunner and Legendary /again/. This time, I'm fairly certain it'll work reliably, so here's hoping. It's even more complicated, involving sender reverse validation and all kinds of wacky stuff to detect forged e-mail addresses and whatnot. However, so far, it's already stopped a good several thousand(!) spam messages.

I am, however, immensely disgusted by the entire spam thing. The fact that a dedicated server dies under the load, that I have had to literally stress myself out over trying to find a workable solution so my e-mail still WORKS... it's insane. It's not right. Considering spammers have all this talk about 'we don't send mail where we're not wanted' or whatnot, it infuriates me that I have to go to these lengths to protect my systems from what is, in effect, destructive mail, since it can constitute a Denial of Service attack. It is just plain not right. At all.

Grumph. Anyway, immensely tired.


I had a machine that died today because of a spammer too, they're getting more and more... insane about volume and speed of transfer.

I actually had to put limits on load average and queueing and such. (which I can do in sendmail, not sure about postfix, I know you can in exim though)

I also started using about 5 different dnsbls, and that seems to be working better...

If you have any suggestions, I'd love to hear them.

My current system goes something like this.

Primary MX system (segfault) accepts incoming mail.

At the initial accept level, I do a couple things.

If the sender's domain is a commonly-forged address (aol.com, hotmail.com, etc.) I do a reverse address check. If the VRFY comes back okay, I allow the mail. If the VRFY fails (or times out), I reject the message with a 450 (and mark the sender address in a database); real mailservers will queue the message on a 450 rejection and try again in a few hours (when I will let it through), while spammers will move on.

Assuming we either aren't a 'risky' domain or we get a VRFY that's successful, we move on to the next step. Next we hit about six different RBLs, which is fairly self-explanatory.

I run a VRFY against the domains I relay for (across a private, IP-bound SMTP channel on a non-standard port), to determine if the RCPT commands are for real folks. If the VRFY times out (which, since it's local, doesn't generally happen), the message is rejected on a 450 (and the VRFY is queued for background resolution) on the same logic as the previous verification.

Assuming we've gotten this far, the message is snagged by a ruleset and moved out of the queue, into a holding space. The scanner daemon watches for mail to be moved into the holding directory, and when some is, it runs a few (slower) RBL checks I didn't want in the Postfix checks, as well as running it through two antivirus programs /and/ SpamAssassin, with Razor and DCC. If it fails, MailScanner will move it to the Postfix bounce directory with an appropriate error message (unless the spam score was REALLY high, in which case it just eats it), and if there's a virus, it will clean the file and send it on. Once the message is certified clean, it marks it up with a few status headers, and tosses it into the Postfix outgoing queue.

Postfix then spots the message again, looks up the appropriate end-destination site (mine, or a friend's which I filter for) and passes the message off over the appropriate private SMTP channel.

Voila! Delivery.
I was tired last night, so the explanation wasn't quite accurate. How it truly works is:

1) Remote site connects to me.

2) I check at EHLO if it's a one-word sitename, if so, I reject; segfault only accepts mail from other MTA's, so should never see one-word EHLO lines.

3) At MAIL FROM, if the domain matches one of a number of 'risky' sites (hotmail.com, aol.com, yahoo.com, etc.), I check if the sender is a valid recipient there, using Postfix's 'verify' service, which connects and does an RCPT TO as if it were going to send, and caches the result. Postfix keeps a record of all this, so I only need to check once. I /also/ check against my own domains if the sender has an address for one of mine, thus catching a lot of the falsified 'randomname@yourdomain' addresses, as well as most of those stupid 'support@yourdomain' viruses. If I can't get a reply fast enough, I toss a 4xx result, temporary failure, making the remote site re-queue the message to try later. Then I probe again when I have a little more time, and cache the result for the next time I get sent a message.

4) By RCPT TO, I have had enough time to check several RBLs. If they're in one, I reject right there. I also check if the recipient is valid, before I accept (again, verify service); in this case, local verification is done over the private SMTP channel to the destination hosts.

5) If EVERYTHING else passes, the mail is accepted, but moved into a holding queue. In the holding queue, another daemon picks it up, runs clamav and panda av against it, runs SpamAssassin against it (with a few more RBL checks, Razor, and a few other things), and so on. When it's marked clean, the mail is tossed back into the outgoing queue.

6) Outgoing queue mail is picked up and handed off where it needs to go, voila, delivery.

The system is also set up so that it will accept non-local mail from any machine it relays to (i.e. filters for), allowing you to set the machine to toss all mail sent to it (i.e. spam done by A record rather than MX) for filtering, and have it tossed back on the private SMTP channel. Any server which sets up a private SMTP channel IP-bound to segfault's address and configured to return appropriate existence information on RCPT TO can be set up to use segfault as the filter in about five minutes, now.

Yaaay for Rachel! Boo for spammers.

About a gig a day though, yeesh. You'd almost start to think it looks like unless you're some big organizational setup with the resources to just pile escalating levels of hardware to absorb storage/transfer, domains might have a sort of inherent TTL now before they croak from being distributed around the spam lists.

It's not like SMTP2 is coming anytime soon to save us. :P

Remind me, someday, to show you the traffic stats for segfault. Considering it ONLY does mail and DNS, they're kind of scary.
The aforementioned machine, that was taken down is doing secondary DNS for TES.org (which is farly high traffic, but almost none of it is legitimate).

Its colo'd at CCCP, which is free colo space, and now that I've started rate limiting connects on both primary and secondary MX, I wonder if the spammers will stop. (the secondary MX gets taken down because the primary refuses so much spam now.)
That pretty much explains why I will never be a competent sysadmin. (My solution would probably be "Hey, look! Yahoo now offers 100 MB of mail storage for free! Let's give the whole company up for Yahoo accounts!")

(In case you care about such things, I vectored in from DC Simpson's LJ. Just ignore me. It's easy and fun.)
Always nice to meet another fan of good, lightly cynical humor. :)

And believe me, occasionally it's been tempting to just throw in the towel, lately, under the spam loads. I think I FINALLY got a solution that'll completely work, though.
Always nice to meet another fan of good, lightly cynical humor. :)


And believe me, occasionally it's been tempting to just throw in the towel, lately, under the spam loads. I think I FINALLY got a solution that'll completely work, though.

Guess you'll find out. Seriously: good luck on that. It can't be easy damming a flood of garbage.