DoIT experienced a major incident on August 3, 2025 affecting Cloud Cost Analytics and Dashboards and 1 more component, lasting 3d 18h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 03, 2025, 03:39 PM UTC
We are currently investigating a possible outage from Google resulting in missing Google Cloud billing data
- identified Aug 03, 2025, 03:44 PM UTC
The issue is identified as a Google billing export to BQ. Google team is working on fixing the issue. Google incident: https://console.cloud.google.com/servicehealth/incidents?pageState=(%22psh_incidents_list_v2%22:(%22p%22:0,%22service%22:%5B%22Cloud%20Billing%22%5D))
- identified Aug 05, 2025, 07:09 AM UTC
This is the latest update from Google Cloud team: "Our engineering team continues to address the data delays for the billing data exports to BigQuery. As a result of the mitigation implemented by the engineering team, the data delays continue to reduce in the billing exports to BigQuery. The system is now processing the data backlog and we are actively monitoring the progress of full recovery of billing data exports to BigQuery." Google incident: https://console.cloud.google.com/servicehealth/incidents?pageState=(%22psh_incidents_list_v2%22:(%22p%22:0,%22service%22:%5B%22Cloud%20Billing%22%5D))
- monitoring Aug 06, 2025, 07:48 AM UTC
We are still seeing delays and missing data from the past few days. We are communicating with Google engineering team and this is handled with the highest priority. The Google incident is still open and you can follow it here: https://console.cloud.google.com/servicehealth/incidents?pageState=(%22psh_incidents_list_v2%22:(%22p%22:0,%22service%22:%5B%22Cloud%20Billing%22%5D))
- resolved Aug 07, 2025, 09:47 AM UTC
The incident has been resolved and we are now seeing normal incoming billing data from GCP