Robly incident
Technical Incident - Mail Delays, v2/v3 API Calls
Robly experienced a critical incident on July 28, 2020 affecting Email Delivery and API, lasting 12h 51m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 28, 2020, 03:19 PM UTC
Starting around 6AM MT, users may be experiencing issues regarding mail delays and delays surrounding v2/v3 API calls. We are currently investigating and will provide updates as they become available. We sincerely apologize for the any inconvenience this may cause. Please reach out to our Support team at [email protected] with any questions.
- investigating Jul 28, 2020, 04:27 PM UTC
We are still investigating issues around mail and stat delays, and issues with v2/v3 API calls. We will continue to provide updates as they become available. We thank you for your patience and sincerely apologize for any trouble this is causing. Please reach out to our Support team at [email protected] with any questions.
- investigating Jul 28, 2020, 04:53 PM UTC
We are still seeing mail and stat delays. Mail is being accepted but could be stuck in queue. Users may be experiencing a higher volume of v2/v3 API call errors, especially 5xx errors around 9:45 AM MT. We sincerely apologize for the inconvenience and thank you for your patience. Please reach out to our Support team at [email protected] with any questions.
- investigating Jul 28, 2020, 06:28 PM UTC
Due to an issue in one of our data centers this morning, mail is still stuck in delivery. We are working to get those delayed messages delivered as quickly as possible. Customers may still see errors with non-mail send related API calls. We sincerely apologize for the inconvenience and thank you for your patience. We will provide another update in 30 minutes or sooner as we get updates. Please reach out to our Support team at [email protected] with any questions.
- identified Jul 28, 2020, 06:52 PM UTC
We believe we have identified the source of the issue which has been affecting our services. Our team has tested the remediation steps and are now working toward recovering all remaining impacted services. We will provide another update in 30 minutes or sooner as they become available. We sincerely apologize for the inconvenience this has caused and thank you for your patience.
- identified Jul 28, 2020, 07:14 PM UTC
We are confident we have identified the source of the issue and are continuing work to recover the remaining impacted services. We will provide another update in 30 minutes or sooner. We sincerely apologize for the inconvenience this has caused and thank you for your patience.
- monitoring Jul 28, 2020, 07:44 PM UTC
We have recovered the remaining impacted services and are currently monitoring system performances. We will provide another update in 30 minutes or sooner. We sincerely apologize for the inconvenience this has caused and thank you for your patience.
- monitoring Jul 28, 2020, 09:16 PM UTC
We are continuing to monitor system performances. We will provide another update in 30 minutes or sooner. We sincerely apologize for the inconvenience this has caused and thank you for your patience.
- resolved Jul 29, 2020, 04:10 AM UTC
Our team has monitored our system and confirmed that the issue has been resolved. Delayed mail has been dequeued and all impacted services are up and running again. We sincerely apologize for the inconvenience this has caused and thank you for your patience.