---
title: "Nettica Postmortem May 17 | DuoCircle"
description: "Due to a system outage on Wednesday May 17 20:08 UTC, 30 minutes of email needed to be re-delivered to your mail server."
image: "https://www.duocircle.com/images/og-default.png"
canonical: "https://www.duocircle.com/blog/support-issues/nettica-postmortem-may-17/"
---

Quick Answer

On Wednesday May 17, 2017 at 20:08 UTC, the Nettica hosted email system (mailhostingservice.com) went down for 40 minutes after a hard drive consistency check failed on an attached storage node and forced a reboot. MXToolbox alerts fired within two minutes, the status page was updated within 12 minutes, and mail was queued on a separate cluster during the outage. When Nettica returned to service, a configuration error created roughly a 30-minute gap of undelivered messages. Using Message Replay, an internal mail-rewind capability, DuoCircle restored 100% of messages between 19:00 and 21:00 UTC. Customers may see duplicate or backdated messages as a result. Corrective actions: verify customers are using the correct MX records and escalate the storage failure with the vendor.

Share 

[ ](https://www.linkedin.com/sharing/share-offsite/?url=undefined%2Fblog%2Fsupport-issues%2Fnettica-postmortem-may-17%2F "Share on LinkedIn") [ ](https://twitter.com/intent/tweet?text=Nettica%20Postmortem%20May%2017&url=undefined%2Fblog%2Fsupport-issues%2Fnettica-postmortem-may-17%2F "Share on X/Twitter") [ ](https://www.facebook.com/sharer/sharer.php?u=undefined%2Fblog%2Fsupport-issues%2Fnettica-postmortem-may-17%2F "Share on Facebook") [ ](https://reddit.com/submit?url=undefined%2Fblog%2Fsupport-issues%2Fnettica-postmortem-may-17%2F&title=Nettica%20Postmortem%20May%2017 "Share on Reddit") [ ](mailto:?subject=Nettica%20Postmortem%20May%2017&body=Check out this article: undefined%2Fblog%2Fsupport-issues%2Fnettica-postmortem-may-17%2F "Share via Email") 

![Nettica Postmortem](https://media.mailhop.org/duocircle/images/2017/05/spf-permerror-6482.jpg) 

\*\*Incident Summary  
\*\*Due to a system outage on Wednesday May 17 20:08 UTC, 30 minutes of email needed to be re-delivered to your mail server.

**Timeline**

We became aware of a system delivery issue because our delivery checks from MXToolbox failed and then alerted us within two minutes of the issue starting. Engineering was staff responded immediately, there was no delay between notification and the time we started investigating.

We posted to the status page within 12 minutes of the outage. Our goal is to be extremely transparent during all incidents. 

**Root Cause**

Hard drive consistency checks failed on one of the attached storage nodes and required a reboot. When the system was rebooted mail was delivered but not correctly written to disk.

**Resolution and recovery**

Starting on Monday May 15 2017 DuoCircle has experienced three service interruptions with the Nettica hosted email system (mailhostingservice.com). We were able to address the first two issues for the previous with a reboot, and subsequently updating an SSL key. However today May 17th we had a hard drive issue with one of the connected node and Nettica was down for 40 minutes from 20:08 UTC (1:08PM PST).

During this outage mail was successfully queued on a separate cluster, however, when Nettica was returned to production there was a gap of about 30 minutes of emails because of a misconfiguration when the server was restored to service.

[![Message Replay](https://media.mailhop.org/duocircle/images/2017/05/windows-smtp-service-3698.jpg)](https://media.mailhop.org/duocircle/images/2017/05/windows-smtp-service-3698.jpg)

Normally in an instance like this the messages for the 30 minutes that the queuing cluster delivered to the main server would be lost however we have had a 100% recovery of the messages because of a proprietary system that we introduced a few weeks ago called [Message Replay](/email/message-replay/). It’s sort of a DVR for email, we can rewind and play messages again at a later time.

As of midnight UTC we have restored all messages for our customers using mx1.mailhostingservice.com. Rather than lose 30 minutes of messages we made the determination to restore all messages from 19:00 to 21:00 UTC to ensure that your mailbox is intact.

You may see new emails in your inbox with a timestamp from earlier today or duplicate messages that you have already received.

**Corrective and Preventative Measures**

We are in the process of ensuring that all customers are using the correct MX records. We have escalated to the storage vendors for options and actions that we can do differently.

**Personal Notes**

I am personally very impressed with the speed and professionalism of the teams responded to the issue and that our alerting system worked as designed. Thank you to our customers for your understanding, and we will be migrating mailhostingservice.com over to a new system as soon as possible., Brad Slavin

![Brad Slavin](https://media.mailhop.org/dmarcreport/images/team/brad-slavin.jpg) 

Brad Slavin 

General Manager

General Manager at DuoCircle. Product strategy and commercial lead across the email security portfolio.

## Secure your email infrastructure

Protect, authenticate, and deliver. Contact our team to find the right solution.

[Contact Sales](/contact/) [Explore Products](/products/) 

## Related Articles

[  support-issues 2m  Announcing 24/7 Phone Support  Aug 15, 2015 ](/blog/support-issues/announcing-247-phone-support/)[  support-issues 1m  DuoCircle Increases Support Coverage  Jun 1, 2015 ](/blog/support-issues/duocircle-increases-support-coverage/)[  support-issues 4m  DuoCircle Infrastructure Overview  Feb 18, 2015 ](/blog/support-issues/duocircle-infrastructure-overview-2/)[  support-issues 1m  The DuoCircle Services Explained  Mar 2, 2016 ](/blog/support-issues/duocircle-service-need/)

```json
{"@context":"https://schema.org","@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com","logo":{"@type":"ImageObject","url":"https://www.duocircle.com/images/duocircle-logo.png"},"description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","subOrganization":[{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138898167","name":"DMARC Report","url":"https://dmarcreport.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897474","name":"AutoSPF","url":"https://autospf.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897912","name":"Phish Protection","url":"https://www.phishprotection.com"}],"sameAs":["https://www.linkedin.com/company/duocircle","https://x.com/duocirclellc","https://www.facebook.com/duocirclellc","https://www.g2.com/products/phish-protection-by-duocircle/reviews","https://github.com/duocircle","https://www.crunchbase.com/organization/duocircle-llc"],"contactPoint":{"@type":"ContactPoint","contactType":"customer support","url":"https://support.duocircle.com"},"knowsAbout":["Email Security","Email Authentication","SPF","DKIM","DMARC","Phishing Protection","Spam Filtering","SMTP Relay","Email Deliverability","Email Forwarding"]}
```

```json
{"@context":"https://schema.org","@type":"WebSite","name":"DuoCircle LLC","url":"https://www.duocircle.com","description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","publisher":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com","logo":{"@type":"ImageObject","url":"https://www.duocircle.com/images/duocircle-logo.png"},"description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","subOrganization":[{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138898167","name":"DMARC Report","url":"https://dmarcreport.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897474","name":"AutoSPF","url":"https://autospf.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897912","name":"Phish Protection","url":"https://www.phishprotection.com"}]}}
```

```json
[{"@context":"https://schema.org","@type":"BlogPosting","headline":"Nettica Postmortem May 17","description":"Due to a system outage on Wednesday May 17 20:08 UTC, 30 minutes of email needed to be re-delivered to your mail server.","url":"https://www.duocircle.com/blog/support-issues/nettica-postmortem-may-17/","datePublished":"2017-05-18T08:59:29.000Z","dateModified":"2025-06-16T14:08:22.000Z","dateCreated":"2017-05-18T08:59:29.000Z","author":{"@type":"Person","@id":"https://www.duocircle.com/authors/brad-slavin/#person","name":"Brad Slavin","url":"https://www.duocircle.com/authors/brad-slavin/","jobTitle":"General Manager","description":"Brad Slavin runs DuoCircle, the company behind DMARC Report, AutoSPF, Phish Protection, and Mailhop. His focus is product strategy, customer relationships, and the commercial and compliance side of email authentication (DPAs, SLAs, enterprise procurement).","image":"https://media.mailhop.org/dmarcreport/images/team/brad-slavin.jpg","knowsAbout":["Email Security Strategy","SaaS Product Management","Enterprise Compliance","Customer Success","Email Deliverability Business"],"worksFor":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com"},"sameAs":["https://www.linkedin.com/in/bradslavin"]},"publisher":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com","logo":{"@type":"ImageObject","url":"https://www.duocircle.com/images/duocircle-logo.png"},"description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","subOrganization":[{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138898167","name":"DMARC Report","url":"https://dmarcreport.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897474","name":"AutoSPF","url":"https://autospf.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897912","name":"Phish Protection","url":"https://www.phishprotection.com"}]},"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.duocircle.com/blog/support-issues/nettica-postmortem-may-17/"},"articleSection":"support-issues","keywords":"","wordCount":452,"image":{"@type":"ImageObject","url":"https://media.mailhop.org/duocircle/images/2017/05/spf-permerror-6482.jpg","caption":"Nettica Postmortem","width":900,"height":600},"speakable":{"@type":"SpeakableSpecification","cssSelector":[".answer-block","h1"]}},{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog","item":"https://www.duocircle.com/blog/"},{"@type":"ListItem","position":2,"name":"support-issues"},{"@type":"ListItem","position":3,"name":"Nettica Postmortem May 17","item":"https://www.duocircle.com/blog/support-issues/nettica-postmortem-may-17/"}]}]
```

```json
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.duocircle.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://www.duocircle.com/blog/"},{"@type":"ListItem","position":3,"name":"support-issues","item":"https://www.duocircle.comundefined"},{"@type":"ListItem","position":4,"name":"Nettica Postmortem May 17","item":"https://www.duocircle.com/blog/support-issues/nettica-postmortem-may-17/"}]}
```

```json
{"@context":"https://schema.org","@type":"BlogPosting","headline":"Nettica Postmortem May 17","description":"Due to a system outage on Wednesday May 17 20:08 UTC, 30 minutes of email needed to be re-delivered to your mail server.","url":"https://www.duocircle.com/blog/support-issues/nettica-postmortem-may-17/","datePublished":"2017-05-18T08:59:29.000Z","dateModified":"2025-06-16T14:08:22.000Z","dateCreated":"2017-05-18T08:59:29.000Z","author":{"@type":"Person","@id":"https://www.duocircle.com/authors/brad-slavin/#person","name":"Brad Slavin","url":"https://www.duocircle.com/authors/brad-slavin/","jobTitle":"General Manager","description":"Brad Slavin runs DuoCircle, the company behind DMARC Report, AutoSPF, Phish Protection, and Mailhop. His focus is product strategy, customer relationships, and the commercial and compliance side of email authentication (DPAs, SLAs, enterprise procurement).","image":"https://media.mailhop.org/dmarcreport/images/team/brad-slavin.jpg","knowsAbout":["Email Security Strategy","SaaS Product Management","Enterprise Compliance","Customer Success","Email Deliverability Business"],"worksFor":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com"},"sameAs":["https://www.linkedin.com/in/bradslavin"]},"publisher":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com","logo":{"@type":"ImageObject","url":"https://www.duocircle.com/images/duocircle-logo.png"},"description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","subOrganization":[{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138898167","name":"DMARC Report","url":"https://dmarcreport.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897474","name":"AutoSPF","url":"https://autospf.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897912","name":"Phish Protection","url":"https://www.phishprotection.com"}]},"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.duocircle.com/blog/support-issues/nettica-postmortem-may-17/"},"articleSection":"support-issues","keywords":"","wordCount":452,"image":{"@type":"ImageObject","url":"https://media.mailhop.org/duocircle/images/2017/05/spf-permerror-6482.jpg","caption":"Nettica Postmortem","width":900,"height":600},"speakable":{"@type":"SpeakableSpecification","cssSelector":[".answer-block","h1"]}}
```