Resolved after 21 hours 49 minutes, on April 5, 2018.
**Overview** On April 6th, at around 12:33 pm PST, we noticed an increase in authentication failures and DNS resolution failures. **What went wrong** We had a dramatic increase of queries being sent to our authentication database which resulted in an automated failover process to start. During the time it took to failover and promote, some API endpoints and our UI were affected. **What we're doing to prevent this from happening again** We're currently adding tests to ensure we're able to handle such authentication traffic increases moving forward. We're also looking at solutions to prevent any disruption of service from happening when database failovers are in progress. **Resolution time** The incident was resolved at approximately 12:49 pm PST.
Update timeline
- investigating Apr 05, 2018, 06:45 PM UTC
We are currently investigating this issue.
- monitoring Apr 05, 2018, 07:20 PM UTC
We have resolved the issue and all systems are operational. We will continue to monitor status.
- resolved Apr 06, 2018, 04:34 PM UTC
Resolved. 16 minutes total time of disruption. Post-mortem following.
- postmortem Jul 30, 2018, 07:29 PM UTC
**Overview** On April 6th, at around 12:33 pm PST, we noticed an increase in authentication failures and DNS resolution failures. **What went wrong** We had a dramatic increase of queries being sent to our authentication database which resulted in an automated failover process to start. During the time it took to failover and promote, some API endpoints and our UI were affected. **What we're doing to prevent this from happening again** We're currently adding tests to ensure we're able to handle such authentication traffic increases moving forward. We're also looking at solutions to prevent any disruption of service from happening when database failovers are in progress. **Resolution time** The incident was resolved at approximately 12:49 pm PST.
Your product runs on more than Iron.io
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services