Grafana incident

Some Queries Failing

Resolved after 3 hours 25 minutes, affecting AWS Australia - prod-au-southeast-1 and AWS Brazil - prod-sa-east-1 and 1 more, on July 2, 2026.

identified Jul 02, 2026, 08:16 PM UTC

We are continuing to work on rolling back a PR responsible for this behavior. Once we have more information, we will share it here. Thank you for your patience.

Started: Jul 02, 2026, 06:10 PM UTC
Resolved: Jul 02, 2026, 09:35 PM UTC
Duration: 3h 25m
Detected by Pingoru: Jul 02, 2026, 06:10 PM UTC

Affected components

AWS Australia - prod-au-southeast-1AWS Brazil - prod-sa-east-1AWS Canada - prod-ca-east-0AWS Germany - prod-eu-west-2AWS UAE - prod-me-central-1AWS India - prod-ap-south-1AWS Japan - prod-ap-northeast-0AWS Singapore - prod-ap-southeast-1AWS Sweden - prod-eu-north-0AWS US West - prod-us-west-0

Update timeline

identified Jul 02, 2026, 06:10 PM UTC

Queries with drop __error__, including log volume histogram queries (the queries that generate the histogram visualization in Grafana), are failing due to a bug with series limit checks. The Root cause has been identified, and we are working on a fix.
identified Jul 02, 2026, 08:16 PM UTC

We are continuing to work on rolling back a PR responsible for this behavior. Once we have more information, we will share it here. Thank you for your patience.
resolved Jul 02, 2026, 09:35 PM UTC

This incident has been resolved. Thank you for your patience.

Your product runs on more than Grafana

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.