Grafana incident

Grafana rulers crash-looping on prometheus

Resolved after 5 hours 6 minutes, affecting AWS US East (OH) - prod-us-east-0: Rule Evaluation and AWS US West - prod-us-west-0: Rule Evaluation, on July 8, 2026.

monitoring Jul 08, 2026, 02:23 PM UTC

We are in the process of rolling out the fix. Our engineers are monitoring the progress.

Started
Jul 08, 2026, 12:17 PM UTC
Resolved
Jul 08, 2026, 05:23 PM UTC
Duration
5h 6m
Detected by Pingoru
Jul 08, 2026, 12:17 PM UTC

Affected components

AWS US East (OH) - prod-us-east-0: Rule EvaluationAWS US West - prod-us-west-0: Rule Evaluation

Update timeline

  1. investigating Jul 08, 2026, 12:17 PM UTC

    We are investigating issues with the grafana-ruler service on prod-us-east-2 and prod-us-west-0 which are causing periodic crash conditions. A code fix is currently being deployed to mitigate this

  2. monitoring Jul 08, 2026, 02:23 PM UTC

    We are in the process of rolling out the fix. Our engineers are monitoring the progress.

  3. resolved Jul 08, 2026, 05:23 PM UTC

    This incident has been resolved.

Your product runs on more than Grafana

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →