Prometheus writes in prod-eu-west-3 are degraded

Update timeline

1. investigating Mar 25, 2026, 02:11 PM UTC

   The metric writes issue reported in https://status.grafana.com/incidents/gfshj17lxj5z is still ongoing. Our Engineering team is actively investigating this and we will provide further updates as our investigation progresses.

2. investigating Mar 25, 2026, 09:35 PM UTC

   We are continuing to investigate this issue.

3. monitoring Mar 26, 2026, 12:04 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Mar 26, 2026, 05:45 PM UTC

   We are continuing to monitor the previously impacted environments.

5. monitoring Mar 27, 2026, 09:05 PM UTC

   We are continuing to monitor this through the weekend.

6. monitoring Apr 02, 2026, 09:38 PM UTC

   We are continuing to monitor for any further issues.

7. monitoring Apr 08, 2026, 08:32 PM UTC

   We are still seeing intermittent issues and continue to seek a resolution

8. monitoring Apr 14, 2026, 08:11 PM UTC

   We have deployed mitigation and seen improvement in write failures over the past week. We are still seeing intermittent spikes in latency and continue to monitor.

9. monitoring Apr 20, 2026, 03:08 PM UTC

   We are continuing to monitor for any further issues.

10. resolved Apr 23, 2026, 08:07 PM UTC

    This incident has been resolved. Thank you for your patience.