DatoCMS incident
High response times on CMA. Admin interface not loading. CDA partially impacted.
DatoCMS experienced a minor incident on October 8, 2024 affecting Asia and Europe and 1 more component, lasting 15d 14h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 08, 2024, 12:00 AM UTC
We are currently investigating the issue
- investigating Oct 08, 2024, 12:00 AM UTC
We are still investigating the issue, and have contacted an upstream provider. We will update again once we have more information.
- resolved Oct 08, 2024, 12:00 AM UTC
Heroku resolved the issue: https://status.heroku.com/incidents/2685 We are still waiting for their post-mortem message about these two incidents.
- identified Oct 08, 2024, 12:00 AM UTC
The upstream provider identified an issue on their infrastructure: https://status.heroku.com/incidents/2684
- identified Oct 08, 2024, 12:00 AM UTC
The upstream provider is still investigating the issue. We will continue to post updates here, or you can see their status directly at https://status.heroku.com/incidents/2684 Informational note: Already-cached Content Delivery API (GraphQL) requests do not seem to be affected. Uncached CDA queries may or may not be impacted, depending on the specific query and region. We are still awaiting a resolution and will continue to monitor the situation.
- monitoring Oct 08, 2024, 12:00 AM UTC
Services (CMA, UI, CDA) appear to be coming back up and are accessible at normal performance again. However, we still have not yet received an official confirmation of resolution from the upstream provider. We will continue to monitor until an official resolution is announced.
- resolved Oct 08, 2024, 12:00 AM UTC
The upstream issue has been resolved and services are fully restored. Their last update: Starting at 4:27 PM UTC on October 8th , 2024, customers experienced increased error rates and latencies on customer applications hosted on common runtime in EU Region. Heroku engineers investigated and mitigated the impact at 9:52 PM UTC on October 8th , 2024. All application have fully recovered by 10:10 PM UTC.
- resolved Oct 08, 2024, 12:00 AM UTC
We are waiting for Heroku's post mortem in order to collect more information about the downtime
- investigating Oct 08, 2024, 12:00 AM UTC
It looks like Heroku is having the same problem again. We have already contacted them.
- investigating Oct 08, 2024, 12:00 AM UTC
The upstream provider is currently investigating the issue. This is their new status page issue: https://status.heroku.com/incidents/2685
- identified Oct 08, 2024, 12:00 AM UTC
The upstream provider is working on a fix
- resolved Oct 23, 2024, 02:22 PM UTC
Resolved — Our upstream partner published their post-mortem (Root Cause Analysis (RCA)) about this downtime. You can read the full post here: https://status.heroku.com/incidents/2684 Summary: On October 8, 2024 from 16:27 Coordinated Universal Time (UTC) to 22:10 UTC, and again on October 9, 2024 from 13:06 UTC to 16:36 UTC, a Distributed Denial of Service (DDoS) caused routing failures in a single partition of Heroku’s Common Runtime infrastructure in the EU Region. This resulted in increased error rates and latencies for some clients connecting to some customer applications hosted in that partition. The Salesforce Technology team worked with our upstream infrastructure provider to mitigate the immediate impact of this event, and to put additional network-level protections in place to improve resilience. Resolved — Heroku resolved the issue: https://status.heroku.com/incidents/2685 We are still waiting for their post-mortem message about these two incidents. Identified — The upstream provider is working on a fix Investigating — The upstream provider is currently investigating the issue. This is their new status page issue: https://status.heroku.com/incidents/2685 Investigating — It looks like Heroku is having the same problem again. We have already contacted them. Resolved — We are waiting for Heroku's post mortem in order to collect more information about the downtime Resolved — The upstream issue has been resolved and services are fully restored. Their last update: Starting at 4:27 PM UTC on October 8th , 2024, customers experienced increased error rates and latencies on customer applications hosted on common runtime in EU Region. Heroku engineers investigated and mitigated the impact at 9:52 PM UTC on October 8th , 2024. All application have fully recovered by 10:10 PM UTC. Monitoring — Services (CMA, UI, CDA) appear to be coming back up and are accessible at normal performance again. However, we still have not yet received an official confirmation of resolution from the upstream provider. We will continue to monitor until an official resolution is announced. Identified — The upstream provider is still investigating the issue. We will continue to post updates here, or you can see their status directly at https://status.heroku.com/incidents/2684 Informational note: Already-cached Content Delivery API (GraphQL) requests do not seem to be affected. Uncached CDA queries may or may not be impacted, depending on the specific query and region. We are still awaiting a resolution and will continue to monitor the situation. Identified — The upstream provider identified an issue on their infrastructure: https://status.heroku.com/incidents/2684 Investigating — We are still investigating the issue, and have contacted an upstream provider. We will update again once we have more information. Investigating — We are currently investigating the issue
- resolved Oct 23, 2024, 02:22 PM UTC
Our upstream partner published their post-mortem (Root Cause Analysis (RCA)) about this downtime. You can read the full post here: https://status.heroku.com/incidents/2684 Summary: On October 8, 2024 from 16:27 Coordinated Universal Time (UTC) to 22:10 UTC, and again on October 9, 2024 from 13:06 UTC to 16:36 UTC, a Distributed Denial of Service (DDoS) caused routing failures in a single partition of Heroku’s Common Runtime infrastructure in the EU Region. This resulted in increased error rates and latencies for some clients connecting to some customer applications hosted in that partition. The Salesforce Technology team worked with our upstream infrastructure provider to mitigate the immediate impact of this event, and to put additional network-level protections in place to improve resilience.