FreeYourNet.com
The Best IT Can Be

[Home] [Contact Us]  

Home
Business Services
Residential Services
ConstantCare
Remote Support

This is a reproduction of an article written for, and commissioned by, the Shropshire Chamber of Commerce concerning a major email problem that affected businesses in Shropshire during the months of January and February 2006.

E-Mail Replication - Root Causes & Lessons To Be Learned

Manic Monday   Eliminating the Chaos   Another Manic Monday   Lessons to be Learned   The Chamber's Role

Many of us Chamber members have suffered from the effects of the now infamous email replication that caused chaos throughout Shropshire during January and February of this year. For many people the events have been a source of extreme frustration, and some have suffered not inconsiderable interference to their business operations. In recognising the severity of the situation, and following my efforts to halt the chaos, the Shropshire Chamber commissioned me to deal with the issues arising and to consider ways of reducing the chances of such an event happening again.

Having already spoken with a number of you most affected by the replication, I would like to explain here the multiple events of January that combined to create a mushroom effect, how they were handled and why, though apparently resolved, the chaos returned again in February. Finally, I would like to summarise the lessons that have been learned and consider actions that we can all take to lessen the impact on ourselves should such an event happen again.

Manic Monday

Around close of business on Monday 23rd January, 2006 a Shropshire-based company sent out a promotional email to 630 or so addresses, placing each of those addresses into the 'To' field of the email. What happened from there is that company's ISP mail server created the necessary copies of the original email and relayed them out to each of the domains specified by the 'To' addresses.

Consider now the instance of the email that is routed to the address joe@bloggs.com, say. As it arrives at the mail server which serves the domain bloggs.com, that server becomes responsible for relaying the email to its user 'joe'. Even so, the email copy it handles still has the original 'To' line (with all 630 addresses) fully intact. In relaying the email to joe (and any other of its own internal users also addressed by the email), the bloggs.com mail server does not - indeed, must not - relay the email for any of its external addresses (i.e., addresses for which it is not directly responsible).

Let us now suppose the bloggs.com email server is faulty (let's call this Server1), and it decides not only to pass the message onto 'joe', but also to relay it out again to every address it finds on the 'To' line. Now we have a problem, but so far not a big one. Server1 will cause everyone to receive two copies of the original mail instead of one. Now, add a second faulty email server (Server2) into the mix which is also addressed by the same email. What happens now is disastrous. Server1 incorrectly relays the email to all the addressees, including one or more served by Server2. Server2 does exactly the same. Each time it receives the email from Server1 it relays this message back out to all concerned, including Server1. There is nothing to stop this process, as Server1 and Server2 mutually replicate the email, relaying it to all 630 of us every time.

In fact, there were three company email servers that started to replicate the email in exactly this way.

All this mushroomed out of control during the evening of Monday 23rd January, and most people were back at home with their families blissfully unaware of events until returning to work on the Tuesday. However, some of us were affected early on by the replication and one person, in his frustration at events, responded to all addressees asking for them to stop. All this achieved, however, was to add another email into the mix, effectively doubling the number of replications that we all were to receive.

Eliminating the Chaos

It was late on the Monday that I started to look into what was happening, and at first I was as frustrated as everyone else. It seemed at first that we were being subjected to a rather large 'spam' attack and I set about attempting to track down the location of the perpetrators. In analysing the headers of the many messages arriving, though, what I found suggested that this wasn't a spam attack of the traditional kind, where the perpetrators put efforts into hiding their tracks. Instead, there had been no attempt at obfuscation, and it became clear that all of the emails were originating from three distinct sources. Each source was a Microsoft Small Business Server (which runs a cut-down version of the Microsoft Exchange Server), which I further determined to be operated by legitimate companies based in Shropshire.

Over the following day or two, I liaised with the ISP's serving those companies, the companies themselves, and those companies' outsourced IT support personnel, and it became clear what was actually happening. The fault in the email servers was a known one, and the manufacturer (Microsoft) had issued a patch to fix the problem some two years ago. In the case of the three servers involved, it turned out that the necessary updates had not been made.

In most cases those involved were very cooperative, and worked to resolve the problem as quickly as they could. As a result the replication stopped, but it took some time for the large numbers of emails to filter through recipients' mail queues. While it is difficult to estimate exactly how many replications were made in January, talking with a number of people affected it seems the replication may have created well over 1,000,000 separate copies in all. Staggering.

Another Manic Monday

Monday 20th February. Aren't Mondays fun? I was as surprised as all of you when the emails suddenly started up again in earnest. This time, another hitherto unknown email server started to send out large numbers of the same emails. Why, you may ask? It turns out that this server was actually involved back in January, only it wasn't visible because after creating the replicated the emails, they became stuck in the server's own outgoing queue and were not transmitted at the time. The company operating the server recently changed the outsourced IT support they were using, and the first thing the new personnel did (unaware of the January event) was to open up the outgoing queue...

... and now, one of the servers involved back in January got involved again. It turned out that this particular server had not in fact been patched properly. Oh, and just to make matters worse, another (i.e., fifth) server also got involved this time round.

Given our experiences of January, I was able to react more quickly to the situation and liaised with those responsible once more to have the situation resolved. This time, it seems, there were around 500,000 more replications in all.

Lessons to be Learned

Will all this happen again? Probably not, but there are no guarantees. The chaos that has happened was a combination of events, triggered mostly by ignorance and poor IT support policies. The best we can do is to learn from the events and act to prevent them from happening again. A number of lessons can be learned from the root causes:

  1. Be careful about allowing people access to mailing lists including large numbers of email addresses. When you do, be sure the users of the list understand how to use the list responsibly. The Chamber withdrew their mailing list quickly after the first outbreak in January (more about this below).

  2. Do not expose multiple email addresses in the 'To' or 'cc' fields of an email. While the original email was a catalyst for all that has happened, the sender could not have predicted the consequences that would ensue. That said, exposing email addresses in such a way is a violation of privacy, and can make those email addresses subject to abuse by others. Further, rather than use the 'bcc' field, use a 'kosher' e-mailing service instead.

  3. If you operate an email server, make sure you (or your IT support personnel) verify it is configured to relay emails properly. Ensure SMTP authentication is turned on (to prevent the server from being used to relay spam on behalf of others), and ensure that all available manufacturer-supplied patches are applied in a timely manner. It is the failure of IT administrators to patch the systems they are responsible for that was the primary cause of the chaos. While I do understand that IT administrators often work under pressure to keep company facilities running 24/7, if you fail to take systems down for the short time it takes to apply patches you will get bitten sooner or later. The companies operating servers (email, or otherwise) should also appreciate that short downtimes are a necessary part of maintenance, and allow their IT administrators the freedom to judge when this is necessary.

For those of us who are the victims of recent events, there is little we can do stop the problem, but there are actions we can take to considerably lessen the effects:

  1. When you subscribe to emailing lists of any kind, don't use your primary email address. Instead, use an address created especially for the purpose. Then, retrieve emails arriving at that address on a less regular basis, rather than as a routine part of your emailing send/receive. You will then always be able to access your primary emails without having to review or download list-based email. Most email client software offers you the facility to manage multiple addresses (each with different send/receive schedules), and your company mail server and/or ISP will mostly likely offer a facility for you to set up multiple addresses.

  2. If you receive what you believe to be spam emails, do not respond to them. If the emails are spam, all you achieve is to confirm your email address to the spammer and almost guarantee you will receive more. In this case, it wasn't spam emails (in the traditional sense) that were the problem, but this advice would have prevented the replication from mushrooming quite so fast.

  3. Implement effective spam filters. This is best done at the server level - either on your company server, or your ISP's - as it stops you having to download much spam to your computer or mobile device.

  4. If you still find that you are being bombarded by spam, especially from specific senders, use your spam filters to (at least temporarily) block emails from those defined senders. Most spam filters will allow you to do this, and again try to do this as far 'upstream' as possible: if your ISP or company provides you a facility to do this on their servers (most will), do it there, as this will stop your download link becoming overburdened and help prevent the 'quota exceeded' problem some people have experienced with the replication.

The Chamber's Role

The Shropshire Chamber is a victim in all this, just like the rest of us. While it was their email list which was used to send the original email, they did provide that list to its members in good faith, and as a service to assist its members in creating new business. The Chamber did stop offering the list soon after the events of January, and they did move to commission someone (me, in this case) to address the situation once it was apparent that further action was necessary.

In withdrawing the email list, though, it does remove a facility that some members have expressed they would find useful, and that they would like to see continue in some form or other. The Chamber are, therefore, looking to set up an opt-in email list server to be hosted on the Chamber's web facilities, whereby members can subscribe confidentially. A member wishing to promote their business will then be able to send a single email to the list (and using a password to be supplied by the Chamber), and the list server will then take care of relaying to the subscribers on a confidential basis.

I and the Chamber sincerely hope that this article will help you understand the issues involved, and assist you in dealing with any subsequent repetitions that may (but hopefully not!) happen again.

Steve Moss,
IT Consultant,
FreeYourNet.


Home | Business Services | Residential Services | ConstantCare | Remote Support


FreeYourNet is a trading name of CoCo Systems Ltd.
Copyright 2005-2017 CoCo Systems Ltd. All rights reserved.

Last updated: 13 March, 2017
webmaster@freeyournet.com