Earlier today our mail servers prevented some email from reaching some customers. The messages that were impacted had .co.uk in their domain name. Other messages were unaffected.
What happened
One of the message rules in our anti-spam/anti-malware engine misclassified the entire top level domain .co.uk as being spam and it rejected all the emails to and from that domain. All other messages to/from other domains continued to flow through the system normally.
How we responded
We were able to correctly identify the issue within 8 minutes of the problem starting, and our lead engineer was working to remediate the problem within 15 minutes of escalation.
It took us almost 3 hours to fix the problem, which is longer than we’d like to implement a fix of this magnitude. It took us that long because we wanted to ensure that the scanning engine was still working for other top level domains. In essence, we wanted to make sure as we didn’t break anything.
What we did right
We did not abandon our customers. Our service desk accommodated chat inquires, 60 emails, 15 phone calls and opened over 75 service tickets. We continue to work to close each and every one of these service tickets.
We quickly and accurately diagnosed the problem and started on a resolution.
What we did wrong
We did not update our status page in real time. And our business continuity plan is not up to date which means things like emergency messages at our call center were not changed to convey the proper information.
Perspective
This was our first message-impacting event in three years. Our uptime during that period has been greater than 99.98%.
Going forward
We are in the process of fine tuning our internal processes to ensure this never happens again.
Our commitment to you
We understand the serious nature of not being able to receive your emails. This was our first outage in over three years. We intend for it to be our last.
We are committed to excellent service and excellent customer service. Still, we recognize there are things we could have done better. We will get better.
Thank you for being a loyal customer.