Reduce your intake of spam

The war on junk email is never-ending, but our anti-spam tips may at least stem the deluge

At the start of 2004, Bill Gates made a rash promise: with Microsoft’s might, the problem of spam could be elminated in two years.

We’re all used to Microsoft’s deadlines slipping, but a solution to the problem of spam seems further off than ever.

The number of junk messages has increased, filtering techniques that seemed to catch most of the rubbish a couple of years ago no longer work, and many of us spend more time than ever filtering the junk from our mailboxes.

Are there any new technologies we could adopt to help us win back control of our inboxes? Is the solution to take on the spammers in the courtroom? Or should we bite the bullet and admit that internet email just wasn’t designed to protect us from the chancers, scammers and crooks?

Some even say the open standards that underpin the internet’s email system should be replaced with a next-generation alternative that can keep the spammers at bay.

We’ll take a look at why there’s so much junk, the best ways you can keep on top of it, and explain some of the tactics you can use to help play a part in the battle against spam.

Why spam?
While you may never have met anyone who has been taken in by the claims in junk email, they do exist. And with millions of copies of each mail sent at near zero cost, it only needs a few people to fall for a scam to make it profitable.

Even legitimate companies sometimes believe bulk email is a reasonable marketing tactic: often ending up themselves the unwitting dupes of less scrupulous mailing companies. If you receive unsolicited email at a personal address from a company in the UK, call them on the phone and politely tell them you won’t be buying or recommending their products – as we’ve found out, it can make a difference.

Zombie hordes
Spam hasn’t suddenly become more profitable, however; one of the reasons there’s so much more spam around now is the sheer number of users with broadband connections and, on top of that, the number who are running systems that don’t have adequate protection against security threats.

When spam first appeared, most mail servers on the internet would happily accept and pass on just about any email. SMTP (Simple Mail Transfer Protocol) was orginally designed for this specific purpose, to enable you to easily send and relay email via any convenient SMTP server – there is no built-in authentication mechanism in SMTP. As this convenience started to be abused, the majority of such ‘open relays’ were eventually closed.

So, nowadays a mail server typically receives email only for the addresses it actually manages, or for which it is a backup. And servers have other protection too, such as a maximum number of recipients or connections from one source. So, a spammer can no longer find an open SMTP relay and just fire millions of messages at it, leaving it to deliver them in its own time.

Instead, a lot of spam is sent directly to the receiving mail server for each address; that means connecting to many different servers around the world. If it all had to be done from one place, it would take time to do, not to mention eat up a fair bit of bandwidth – the sort of activity that can be easily spotted by an ISP monitoring its network.

Thanks to broadband, and the ropey security on many home computers, this problem is solved by so-called ‘bot nets’ – networks of compromised PCs (known as ‘zombie’ PCs) that can be remotely controlled, and used to send spam without the owner even realising. With thousands of machines sending messages, from all over the world, it’s harder to spot a pattern to the traffic, and the spammers can reach many recipients in a very short time.

In the picture
That may explain how so much spam is being sent, when there are so few open relays.

But it still doesn’t account for how much of it ends up in your inbox. When we looked at anti-spam tools in PCW a couple of years ago, some of them were remarkably effective, removing or flagging a high percentage of the junk.

Now, though, it’s a different story. A typical inbox will be full of not just the old-fashioned ads for assorted erotic enhancements, but offers of exclusive watches and – most recently – tips for shares to buy.

Needless to say, the pills don’t work, the watches are likely fake and you’ll lose money on the shares. But how does all this evade your spam filtering?

The spammers are cunning; a couple of years ago, Bayesian filtering was touted as one of the great tools for spotting spam. Instead of merely just looking for key words and phrases in a message, and filtering on those, it analysed spam for all the words, and allowed you to ‘train’ the system with good and bad messages.

The idea is that some combinations of words are typical of spam, but the same words in other contexts might not be. By weighing the probabilities, a Bayesian filter can capture a lot of spam.

However, it didn’t take long for spammers to cotton on to it, and that’s why much junk mail contains seemingly random portions of text – you might not see it if it’s in white on a white background, but it’s there – and Bayesian filters read it alongside the spam. Put enough random ordinary words and phrases in and there’s a good chance the message will sail through your filter.

There are other techniques too; systems such as Cloudmark and DCC calculate check sums or fingerprints of spam messages and share them on servers. Mail servers can use the fingerprints to identify junk, and the systems are designed with a certain amount of fuzziness, to allow for common customisations in messages.

But once again, while it might take a lot of effort for a single system to customise millions of junk emails, slowing down the sending, due to the zombie networks putting massive computing power in the hands of the senders, enough random junk can be included in messages to make fingerprinting less reliable.

And then, there’s the latest technique: image spam. While spammers have often used images to show their products, there’s a recent marked increase in their use: according to McAfee, it’s now 40 per cent of spam and rising, four times the level last year. This is evident for ‘pump-and-dump’ share scams (where the object of the spam is to inflate the value of a company’s shares by conning you into buying them).

With no real text available for scanning, and messages arriving from a huge number of different IP addresses, it’s very hard to spot these messages when they arrive at a mail server.

In theory, you could run character recognition on the messages, but that would use an awful lot of processing time. You can’t simply block messages with images either, since so many people share photographs, or attach their company logo as a signature to emails.

Fighting back
With so much spam around, it’s no wonder that some people are wondering if the internet needs a new mail system. However, this is not likely to happen. For one thing, there are simply too many people using existing standards to make migration easy. For another, there are those zombie hordes; with so many systems compromised, it surely wouldn’t be long before spammers found a way to hijack them.

Solutions are likely to be technical – including new techniques to verify email, which can be piggybacked on to the existing systems – and legal, fighting spammers in court, where they breach laws on junk mail.

Meanwhile, what can you do to keep your inbox clear? It’s obvious that, especially with image spam, it can be very hard for desktop anti-spam software to do an effective job. It will still, of course, work on many of the other types of junk mail, by simply looking for keywords, blocking known spam domains or simply using a whitelist of your known contacts. But it seems that to manage the problem effectively, spam really needs to be fought on the world’s email servers.

One of the traditional tools to do that was the blacklist – a list of IP addresses of known spammers, which enables mail servers to reject messages as soon as the spammer tries to connect. While such lists have their place, and can prevent commercial email marketing firms from pestering you, as organisations like Spamhaus have found out, business spammers tend to fight in the courts when their ‘right’ to pollute our inboxes is infringed.

There are other techniques to determine spammers too; one is to try and send a mail back to each machine that sends you a message; but this can be resource-intensive, and not always reliable.

More practical are two new technologies. One is greylisting (see box on the next page), which can help stop one-shot mailing attempts by spammers, and the other is the Sender Policy Framework (SPF). Both can be added to your own mail server, and are being used by a growing number of ISPs, hosting companies and businesses.

SPF spots the forgers
SPF is a solution designed to solve one particular spam problem – forged senders.

If you’ve ever had your email address used to send spam, you’ll know how many bounces and angry messages you can receive; the problem can be especially bad if you have a domain of your own with a catch-all email address, where messages to anything at that domain end up in your inbox.

SPF works by adding extra information to the domain system, saying which IP addresses are allowed to send mail for a particular domain.

A receiving mail server can then check and see if the computer claiming to send from, for example, nigelwhitfield.com, is allowed to by SPF and depending on the SPF settings, can either issue a warning, or reject the email.

It’s not perfect though; if you automatically forward mail from one address to another via an alias or mailing list service, SPF breaks, because the forwarding computer won’t be authorised for the domain of the original sender. Instead, if mail is forwarded, servers need to rewrite the message envelope – which can mean that bounces no longer reach the original sender if the message fails after being forwarded. And because SPF is still fairly uncommon, not everyone adds SPF information to their domains, or checks it on their mail servers.

However, if you want to protect your own domain, and only send through your own or your ISP’s email servers, you can set up SPF information easily, using the wizard at www.openspf.org; you can see exactly how it’s done in the step-by-step workshop on the previous page.

To check other people’s SPF information, you may need to update your mail server software, and that’s easier than you may think, since recent versions of systems such as Postfix and Sendmail allow plug-in filters. Adding SPF filtering to the PCW Mail Server project we built two years ago (www.nigelwhitfield.com/ v2/work.php?c=3) was simple, using the Perl policy daemon from OpenSPF.

SPF isn’t the only way to protect your domain from being used to send spam, Domain Keys is an alternative, which uses digital signatures on email headers and can even verify that a message body hasn’t been tampered with.

More information about Domain Keys is at http://antispam.yahoo.com/domainkeys. It’s not as straightforward to implement as SPF, but similarly relies on additional information being inserted into the DNS, and then checked by mail servers. And there are open-source implementations for many of the main mail server packages.

How effective?
We added both greylisting and SPF-checking to our mail server. Although it’s hard to get a conclusive figure in the relatively short time since the system’s been in use, we saw a massive drop in the amount of spam that ended up in our inboxes. Greylisting proved an effective tactic; in one day, the number of one-off attempts to deliver messages was more than 14,000.

Looking at the database that the greylisting software compiled, it was clear that many of those messages would be bounced anyway, as they were to made up addresses, eg nigelozhk@ rather than nigel@. But with the sender forged, the bounces would bounce, leaving the mail administrator with a large number of error messages to wade through.

SPF filtering stopped fewer messages – about 300 in the first 24 hours – but that’s still a welcome improvement.

In total, in the period before the new filters were added, our test server would typically receive 21,000 email messages per day; of those, almost 15,000 would be rejected, either by the existing spam filtering, or because they were to invalid addresses – some c reating bounces. A further 6,000 messages would find their way to users’ mai lboxes, many of those being image spam, or bounces caused by forged sender addresses.

With the new techniques, many messages never made it as far as the spam filter, being rejected instead by greylisting or SPF. The number of messages actually delivered to users was about 1,000; a massive drop that’s broadly in line with estimates of how much junk there is on the internet.

Fight the good fight
If you are suffering from too much junk, and your current filters aren’t working, it’s worth looking at adding greylisting to your mail server, or asking your ISP to implement it for you.

Though it’s not quite as effective, if you have your own domain, you should consider using SPF as well – even if you don’t run a mail server to check incoming messages, you can protect yourself from others who pick your domain as the sender of their junk.

Users who don’t control their own email are best off ensuring their ISP uses greylisting and SPF or Domain Keys – and then filtering the rest of their email using a good desktop anti-spam tool.

There’s no doubt that fighting junk is an arms race, and constantly updating your software can be annoying. But with email such an important tool for many, it’s a race that you can’t afford to pull out of.

Email marketing? Just say no
Among the many spams received in the mailboxes of PCW staff have been some promoting what looks like reputable software products or well-known mail-order companies. The small print claims that the messages are sent in accordance with Data Protection legislation, using lists from a third-party list supplier.

There’s a problem, though. No matter what the message says, when these unsolicited messages arrive in personal email inboxes, they are still in breach of the UK regulations. PCW put this point to two of the companies concerned – which we can’t name – who told us they were assured that the addresses in the lists were legitimate, laying the blame solely on the firms that claim to provide marketing services.

One of the firms told us that after its experience of advertising its software product – and the number of complaints it received – it wouldn’t be using email marketing again and was currently in a legal dispute with the mailing firm.

So, while your company may be offered a ‘compliant’ list of email addresses, or a service that says it will s tay within the law, the only list you can really trust is a list of people you know for sure have opted in. Bulk emailing may look like an attractive strategy, but even if you try to deal with reputable companies, it can still backfire and lose you sales, rather than gain them.

Implementing SPF on your server, step by step
Adding an SPF record to your own domain is very simple. Here’s how to do it. Note, that for the final step, the exact details will depend on the software you use to manage your own domain; if you can’t add ‘TXT’ records to it, you may need to ask your domain hosting company or ISP for help.

Step 1 - Start at openspf.org and enter your domain name in the box on the front page. Then click Go to start the wizard, which will show you this page. The wizard will work out most of the settings for you, based on which computers are known to receive email for your domain, but you should check them carefully.

Step 2 - Scroll down. We’ve added the address for our home broadband network and the wizard has created our SPF details in the text box, so you can copy and paste them. But remember to change the ~all to –all, so that mail will be rejected if it doesn’t match your settings; with ~all, warnings will be generated instead.

Step 3 - Now you can add the information to your domain details. Here, we’re using the Plesk web admin server, which allows us to manage our DNS information easily. Click on Domains, choose the domain you’re updating, then click the DNS button.

Step 4 - Click the Add New Record button. Choose TXT as the record type, and in the bottom box, paste the code from OpenSPF’s wizard, without the enclosing quotes. Click OK.

How greylisting can reduce the spam
One technique that many mail administrators are using is greylisting. So far, it’s proving surprisingly effective at rejecting a lot of spam and, unlike other solutions, there’s no risk of false positives and false negatives.

Greylisting doesn’t worry about the content of an email. Instead, when an email server receives a message, it simply rejects it with a temporary failure code. A standard mail server will see the response, and simply queue the message to deliver later. Most spam software doesn’t do that; it just sees anything other than an ‘OK’ response as a problem, and goes on to the next address on the list.

Genuine messages will be retried by the sending server – how often depends on the way they’re configured, but half an hour or an hour is common. The next time the greylisting system sees the same combination of sending server, address and recipient, it passes the mail through straight away – perhaps performing other anti-spam and anti-virus checks too.

There are refinements possible; some greylisting software will automatically add a sending mail server to an approved list, if it sends a certain number of mails, for example, so regular contacts will find that their messages are no longer greylisted.

Best of all, you can easily add greylisting to a mail server; it took less than 15 minutes, for example, to add the gld tool to our OpenBSD mail server, plugging it into the existing Postfix mail system alongside a check for SPF.

But won’t the spammers just try resending their messages? Possibly; but anything that slows down the time it takes them to send their junk means there’s less of it, and more opportunities for systems such as DCC or Cloudmark to gather fingerprints that will spot the spam.