Grafana Outage History

Started Mar 10, 2026, 03:26 PM UTC· resolved Mar 10, 2026, 08:39 PM UTC

1. identified Mar 10, 2026, 03:26 PM UTC

   There has been a reoccurrence o the issues on the Read path of Loki services on AWS US West since yesterday 9th around ~17:15UTC. The issue has been identified, and resolutions steps has been taken to restore full service. We are currently monitoring the service status. The impact of this includes timeouts and 5xx errors when query logs for customers on this cluster.

2. resolved Mar 10, 2026, 08:39 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 09, 2026, 06:03 PM UTC· resolved Mar 10, 2026, 09:17 PM UTC

1. monitoring Mar 09, 2026, 06:03 PM UTC

   From 15:30 to 15:45 UTC and from 16:53 to 17:03 UTC, the prod-us-central-0 and prod-us-central-5 regions saw elevated latency and error rates on the write path. We're monitoring now.

2. resolved Mar 10, 2026, 09:17 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 09, 2026, 02:20 PM UTC· resolved Mar 10, 2026, 08:54 PM UTC

1. investigating Mar 09, 2026, 02:20 PM UTC

   Some users in prod-us-central-0 may be seeing elevated rate of errors when fetching configurations. Our engineers are currently investigating this issue.

2. investigating Mar 10, 2026, 06:11 PM UTC

   Our engineering team continues to work towards a resolution for this issue.

3. resolved Mar 10, 2026, 08:54 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 07, 2026, 08:07 PM UTC· resolved Mar 09, 2026, 08:59 AM UTC

1. investigating Mar 07, 2026, 08:07 PM UTC

   We are seeing elevated errors rate and outages across many of our services in prod-eu-central-0, due to an on-going AWS S3 outage in that region.

2. investigating Mar 07, 2026, 08:10 PM UTC

   We are continuing to investigate this issue.

3. investigating Mar 07, 2026, 08:10 PM UTC

   Since about 20:03 UTC we have seen AWS S3 recover and also our services are recovering, we are monitoring.

4. monitoring Mar 08, 2026, 11:30 AM UTC

   Since about 20:03 UTC we have seen AWS S3 recover and also our services are recovering, we are monitoring.

5. resolved Mar 09, 2026, 08:59 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 05, 2026, 10:27 PM UTC· resolved Mar 05, 2026, 11:36 PM UTC

1. investigating Mar 05, 2026, 10:27 PM UTC

   A recent incident affecting the data read path and rule execution within prod-eu-west-0 began at ~21:05 UTC on March 5, 2026. Customers with instances in this region may experience write failures and delays in rule evaluation. Engineering is actively engaged and assessing the issue. We will provide updates accordingly.

2. monitoring Mar 05, 2026, 10:41 PM UTC

   Engineering has released a fix and as of 22:00 UTC, customers should no longer experience write failures and delays in rule evaluation. We will continue to monitor for recurrence and provide updates accordingly.

3. resolved Mar 05, 2026, 11:36 PM UTC

   We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

Read the full incident report →

Started Mar 04, 2026, 07:47 AM UTC· resolved Mar 04, 2026, 09:29 AM UTC

1. investigating Mar 04, 2026, 07:47 AM UTC

   We are currently experiencing an issue with Fleet Management in prod-us-central-0. Users in prod-us-central-0 may observe elevated rate of errors when fetching configurations.

2. monitoring Mar 04, 2026, 08:46 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 04, 2026, 09:29 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 03, 2026, 01:00 PM UTC· resolved Mar 03, 2026, 01:00 PM UTC

1. resolved Mar 03, 2026, 06:35 PM UTC

   Test run browser screenshot upload experienced failures from 13:12 to 14:51 UTC. The issue has been resolved

Read the full incident report →

Started Mar 02, 2026, 07:37 AM UTC· resolved Mar 02, 2026, 03:48 PM UTC

1. investigating Mar 02, 2026, 07:37 AM UTC

   We are experiencing increased write latency for logs in prod-eu-west-3. Our Engineering team is aware and currently investigating this. We will provide further updates accordingly.

2. investigating Mar 02, 2026, 08:08 AM UTC

   We are now experiencing write outage for logs in prod-eu-west-3. Our Engineering team is aware and currently investigating this. We will provide further updates accordingly.

3. resolved Mar 02, 2026, 03:48 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 02, 2026, 06:43 AM UTC· resolved Jun 17, 2026, 09:58 AM UTC

1. investigating Mar 02, 2026, 06:43 AM UTC

   We are seeing elevated write and read path errors in prod-me-central-1, due to an on-going AWS UAE data center issue. We will provide further updates accordingly.

2. investigating Mar 02, 2026, 08:14 AM UTC

   We are observing write and read outage errors across all databases (metrics, logs, traces) in prod-me-central-1, due to an on-going AWS UAE data center issue. We will provide further updates accordingly.

3. investigating Mar 02, 2026, 08:21 AM UTC

   We are observing write and read outage errors across all databases (metrics, logs, traces) in prod-me-central-1, due to an on-going AWS UAE data center issue. We will provide further updates accordingly.

4. investigating Mar 02, 2026, 08:36 AM UTC

   We are updating this incident to reflect a complete outage in prod-me-central-1, due to an on-going AWS UAE data center issue. We will provide further updates accordingly.

5. investigating Mar 02, 2026, 10:04 AM UTC

   Customers are recommended to configure a new blank stack in an alternative Grafana Cloud region and to reconfigure their clients (such as Grafana Alloy) to send telemetry to that region, Fleet Management can be used for this purpose https://grafana.com/docs/grafana-cloud/send-data/fleet-management/introduction/

6. investigating Mar 02, 2026, 10:31 AM UTC

   AWS are recommending that affected customers move workloads to alternate regions https://health.aws.amazon.com/health/status and we are recommending the same. Customers who are impacted and who cannot wait for a restoration of service are asked to: 1. Create a Grafana Cloud stack in an alternate region 2. Update clients to send telemetry to the new region, if using Grafana Alloy then you can use Fleet Management https://grafana.com/docs/grafana-cloud/send-data/fleet-management/introduction/ 3. If your instance remains available and you have not configured your dashboards as code, then you may be able to use `grafanactl` to migrate dashboards https://grafana.com/docs/grafana/latest/as-code/observability-as-code/grafana-cli/grafanacli-workflows/ https://grafana.github.io/grafanactl/ We will provide updates when we have them, but we do not have an expected resolution time at this point.

7. investigating Mar 02, 2026, 10:18 PM UTC

   Please continue to refer to the AWS status page for more detailed updates specific to AWS. https://health.aws.amazon.com/health/status AWS are recommending that affected customers move workloads to alternate regions, and we are recommending the same. Customers who are impacted and who cannot wait for a restoration of service are asked to: 1. Create a Grafana Cloud stack in an alternate region 2. Update clients to send telemetry to the new region, if using Grafana Alloy then you can use Fleet Management https://grafana.com/docs/grafana-cloud/send-data/fleet-management/introduction/ 3. If your instance remains available and you have not configured your dashboards as code, then you may be able to use `grafanactl` to migrate dashboards https://grafana.com/docs/grafana/latest/as-code/observability-as-code/grafana-cli/grafanacli-workflows/ https://grafana.github.io/grafanactl/ We are continuing to work with our CSP at this time, and will provide updates as they are available.

8. investigating Mar 04, 2026, 10:28 AM UTC

   We are continuing to investigate this issue.

9. investigating Mar 04, 2026, 10:22 PM UTC

   We are actively monitoring the situation, but at this time there are no new updates to share. The next update will be provided once we have more information to share. Please reach out to our Support team if you have any questions.

10. investigating Mar 19, 2026, 12:13 PM UTC

    We have not received any further updates from AWS at this time. However, we are actively monitoring the outage and will provide additional information as it becomes available. Also, please continue to refer to the AWS status page for more detailed updates. https://health.aws.amazon.com/health/status All the guidance previously included about stack migration is still relevant. Please reach out to our Support team if you have any questions.

11. investigating Apr 20, 2026, 03:11 PM UTC

    We are continuing to investigate this issue.

12. investigating May 13, 2026, 09:59 PM UTC

    We do not have any additional updates to share at this time. Our team is actively monitoring the situation and will provide further information as it becomes available. In the meantime, please continue to refer to the AWS Status Page for the most detailed and up-to-date information.

13. investigating May 21, 2026, 11:41 AM UTC

    AWS UAE - prod-me-central-1: Public Probe checks might suffer degraded experience. We recommend migrating checks from the UAE probe to the next nearest probe suitable for your use case.

14. investigating May 27, 2026, 05:10 PM UTC

    The TLS certificates serving prod-me-central-1 endpoints expire on May 30, 2026. Replacement certificates have been imported, but the ongoing AWS regional incident is preventing them from propagating to all load balancer nodes, so customers may see certificate errors after that date until AWS restores normal operation. We do not have any additional updates to share at this time. Our team is actively monitoring the situation and will provide further information as it becomes available. In the meantime, please continue to refer to the AWS Status Page for the most detailed and up-to-date information.

15. resolved Jun 17, 2026, 09:58 AM UTC

    Following our ongoing communications regarding the complete outage in prod-me-central-1, we are now closing this incident. As noted in the latest AWS update, the Middle East (UAE) region (ME-CENTRAL-1) has suffered significant damage and restoration is expected to take several months. We strongly recommend all affected customers migrate workloads to an alternate Grafana Cloud region as soon as possible. If you have not already done so, please follow the migration steps outlined in our previous updates: 1. Create a Grafana Cloud stack in an alternate region 2. Update clients to send telemetry to the new region, if using Grafana Alloy then you can use Fleet Management https://grafana.com/docs/grafana-cloud/send-data/fleet-management/introduction/ 3. If your instance remains available and you have not configured your dashboards as code, then you may be able to use `grafanactl` to migrate dashboards https://grafana.com/docs/grafana/latest/as-code/observability-as-code/grafana-cli/grafanacli-workflows/ https://grafana.github.io/grafanactl/ For further details, please refer to the AWS incident communication directly: https://health.aws.amazon.com/health/status Please reach out to our Support team if you need any assistance with the above - https://grafana.com/profile/org#support We will continue to monitor the situation and update the incident once circumstances change.

Read the full incident report →

Started Feb 27, 2026, 04:25 PM UTC· resolved Feb 27, 2026, 04:25 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Feb 27, 2026, 01:46 PM UTC· resolved Feb 27, 2026, 11:38 PM UTC

1. investigating Feb 27, 2026, 01:46 PM UTC

   We're currently working on an issue where portions of data may be temporarily unretrievable, affecting a small percentage of tenants in all Tempo clusters.

2. identified Feb 27, 2026, 07:27 PM UTC

   Our team has identified the issue, and are in the process of testing a fix.

3. resolved Feb 27, 2026, 11:38 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 25, 2026, 07:54 PM UTC· resolved Mar 17, 2026, 06:22 PM UTC

1. investigating Feb 25, 2026, 07:54 PM UTC

   Since February 19, we have been investigating an intermittent issue causing increased write latency in the prod-us-central-0 and prod-us-central-5 regions. The issue does not affect all traffic but may result in delayed write operations for some customers. Our engineering team is actively working to identify the root cause and stabilize performance. We will share additional updates as progress is made.

2. investigating Feb 26, 2026, 04:23 PM UTC

   We have begun rolling out mitigation steps to reduce write latency in the prod-us-central-0 and prod-us-central-5 regions. While these measures are expected to improve performance, we are continuing to investigate the underlying root cause of the issue. We will provide additional updates as more information becomes available.

3. monitoring Feb 27, 2026, 02:55 PM UTC

   We've implemented mitigation in place and are continuing to monitoring and investigating this issue.

4. monitoring Feb 27, 2026, 10:05 PM UTC

   We are continuing to investigate this issue alongside the CSP. Any notable updates will continue to be shared here for tracking.

5. monitoring Mar 05, 2026, 10:22 PM UTC

   We are continuing to investigate this issue alongside the CSP, and have taken steps to escalate through the appropriate channels. The mitigation in place continues to work as expected, and any notable updates will continue to be shared here for tracking.

6. monitoring Mar 06, 2026, 08:53 PM UTC

   We have seen an increase in latency in our cloud providers services, and are rolling out a change to mitigate the issue. We are monitoring.

7. monitoring Mar 06, 2026, 09:44 PM UTC

   We are rolling out a mitigation across the environments in these regions, and preemptively where possible to ensure it doesn’t spread elsewhere.

8. resolved Mar 17, 2026, 06:22 PM UTC

   This incident is now resolved. During the incident the Cloud Metrics platform experienced intermittent latency spikes communicating with a backend cloud service in the prod-us-central-0 and prod-us-central-5 regions. During the incident the internal CSP-facing issue was escalated to a P1. After determining the scope of the latency spikes was limited to only one availability zone, the team mitigated the situation by migrating all write traffic from to the single nearly unaffected availability zone. As the CSP service team attempted to remedy the situation, the situation became worse and began affecting the previously unaffected zone. Given this, another mitigation path was needed. Changing the connection strategy employed by Cloud Metrics to a different method was deployed to all environments, stabilizing the write path once again as we found the different connection method was more reliable and not affected by these increases in latency. We have migrated all tenants back to multi-zone write paths and are happy with and confident in the current method of connectivity to the backend cloud service, which is the one we migrated to during the course of the incident. We have no immediate plans to use the previous problematic connectivity method for the foreseeable future.

Read the full incident report →