Anti-spam trick: Grey listing
There is an anti-spam technique called "Grey Listing" which has almost completely eliminated spam from my main server. What's left still goes through my SpamAssassin and Amavis-new filters, but they have considerably much less work to do.
The technique is more than a year old but I've only installed a greylist plug in recently and I'm impressed at how well it works. I hope by writing this article other people that have procrastinated will decide to install a greylist system.
(for those that want technical specifics, I'm using Postfix plus Postgrey. If you use FreeBSD, just do "portinstall mail/postgrey" assuming you are already using Postfix. Sendmail users, please post some comments directing people to the Milter equivalent!)
So how does grey listing work?
Well, you know that a "black list" is a list of sites you block, and a "white list" is a list of sites that you always permit. A grey list is somewhere in between.
The basic principle is that spammers don't retry an email that couldn't be delivered. There are two kinds of "can't be delivered" (actually, more than that but two are important here). One is a "hard failure"... the email can't be delivered and nothing is going to fix it. For example, you are trying to send email to an account that doesn't exist. The second type is a "soft failure", which is a problem that is temporary. In other words, a disk is full, or there is some kind of system problem that will be fixed soon. If you get a "hard error" the email is bounced. If you get a "soft failure" the sending server is supposed to wait a bit of time and retry. That's why when you run out of disk space email stops flowing, but when you fix the problem (delete that out-of-control log file or whatever) you suddenly get a flood of backlogged email.
Spammers don't retry sending email whether it is a hard or soft failure. When you are sending email to tens of millions of addresses, its too difficult to keep track of failure codes. Besides, even if they don't get their spam sent to 20% of their list, they're still sending it to millions of addresses. Good enough, eh?
So here's how grey listing works. The first time someone tries to send you email, send a "soft error" result code. If they reply more than 5 minutes later, then actually accept it. If they are a spammer you'll never get a retry. If they are legitimate then you'll get a retry.
Implementing this is extremely simple. When someone tries to send email, gather 3 other item of information: the source IP address, the From:, the To:. Maintain a database of these 3-tuples. If you haven't seen that 3-tuple before, send the "soft failure" code. If you have seen that 3-tuple already and it was more than 5 minutes ago, accept the message.
It's amazingly simple yet it seems to be blocking about 80% of my spam right now.
Now, you may be thinking, "I can't have a 5-minute delay on all my email! That's crazy!" Well don't worry. Systems like Postgrey take this all one step further. For example, if 5 emails get through in the last month, Postgrey decides this IP address must be ok and adds it to a list that is "white listed".
Thus, the system tunes itself. Common senders immediately get into the whitelist (Yahoo, gmail, and so on). Site that disappear eventually get expired from the list because you don't hear from them in 30 days. That makes the database self-cleaning. All maintenance is automatic.
I can't believe I didn't install this years ago!
--Tom
P.S. I've also added "reject_non_fqdn_hostname" to the Postfix variable "smtpd_helo_restrictions". That means that when an STMP server issues a "HELO hostname" the email is rejected if "hostname" isn't a FQDN. This rejects about 80% of the spam I'm getting... most of which just sends "HELO friend". I haven't had any complaints from users about false-positives since I implemented this a month ago. This technique reduced spam by 80% and Postgrey reduced spam by a different-but-overlaping 80%. When both are enabled, I receive very little spam. Enough for Amavis-new and SpamAssassin to take care of easily.
LISA '06 Call For Papers
Usenix/SAGE has announced their call-for-papers for LISA 2006.
This is the 20th Large Installation System Administration Conference. Has it really been 20 years? Wow, how time flies. This year's conference will be December 3-8, 2006 in Washington, D.C. (The deadline for paper submission is May 23, 2006).
I wrote Bill LeFebvre and told him, "Ah heck, don't do the whole 'call for papers' thing this year! You're a smart guy! Write all the papers yourself! You gots tons of ideas."
But no, he wouldn't listen to me.
He said, "Tom, that's not how the conference works. We collect papers from all over the world that real sysadmins submit. These papers describe solutions and innovations from everything from video processing to new security issues to better ways to run helpdesks. So we collect them and read them all and pick the absolute very best. Just those papers are accepted and presented at the conference." (Footnote: By now I hope you realize this conversation is fictional.)
"Bah!" I replied. "That sounds like a lot of work! The conference has been around for 20 years! Why not just pick 20 past papers and reprint them. Nobody will know the difference."
"Tom, that'd be looking at the past." Bill rebuffed. "While it's always good to remember our history, LISA is about innovation. We look for papers that are forward-looking. Attendees come back from our conference saying, 'Wow! I just saw the future! I'm going to look like a freakin' genius with the new 'vision' I have for our little IT group." (Footnote: Bill would never use the term "freakin'")
"Oh well, I guess you're right." I said. That's a much better idea!
(insert "ABC After School Special" theme song)
And that's how I learned all about the importance of submitting papers to LISA.
(Disclaimer: Bill has no idea I wrote this. Click below to read the actual "Call for Papers")
--------------------------------------------------------------------
Call for Papers
LISA '06: 20th Large Installation System Administration Conference
December 3-8, 2006, Washington, D.C., USA
http://www.usenix.org/lisa06/cfpa
Extended Abstract and Paper Submissions Deadline: May 23, 2006
Sponsored by USENIX and SAGE
--------------------------------------------------------------------
Dear Colleague
The LISA '06 organizers invite you to contribute proposals for refereed papers, invited talks, and workshops, plus any ideas you have for Guru Is In sessions, Work-in-Progress reports, and training sessions.
The Call for Participation with submission guidelines and sample topics can be found on the USENIX Web site at http://www.usenix.org/lisa06/cfpa
The annual LISA conference is the meeting place of choice for system, network, security, and other computing administrators. Administrators of all specialties and levels of expertise meet at LISA to exchange ideas, sharpen skills, learn new techniques, debate current issues, and meet colleagues and friends.
People representing every work assignment from the full-time position at a large site to the part-time one at a small shop come to LISA from over 30 countries, bringing a variety of backgrounds and experience levels to the conference dedicated to them. System and network administrators from environments as diverse as academia, large corporations, small businesses, government organizations, and research sites find LISA to be the place to go for training, education, networking, and interacting with their peers.
The conference's diverse group of participants is matched by an equally broad spectrum of activities:
* Training sessions for both beginners and experienced attendees cover many administrative topics ranging from basic administrative procedures to using cutting-edge technologies.
* Refereed papers present the latest developments and ideas related to system and network administration.
* Invited talks and panels discuss important and timely topics and often spark lively debates and conversation.
* Work-in-progress reports (WiPs) provide brief peeks at next year's innovations.
GET INVOLVED!
* Submit a draft paper or extended abstract proposal for a refereed paper.
* Suggest an invited talk speaker.
* Share your experience by leading a Guru Is In session.
* Propose a training session topic.
* Organize or suggest a Birds-of-a-Feather (BoF) session.
* Email an idea to the chair: lisa06ideas@usenix.org
------------------------------------------------------------
IMPORTANT DATES
Extended Abstract and Paper Submissions Deadline: May 23, 2006
Invited Talks proposals due: June 1, 2006
Notification to authors: July 12 2006
Final papers due: September 12, 2006
Submission guidelines and more information can be found at
http://www.usenix.org/lisa06/cfpa
Sponsored by USENIX and SAGE
-------------------------------------------------------------
We look forward to hearing from you!
On behalf of the LISA '06 Program Committee,
William LeFebvre
lisa06chair@usenix.org
