Brokered service logs and metrics not visible to customers

Update timeline

1. resolved Nov 20, 2025, 06:38 PM UTC

   On November 13, 2025 around 1:40 PM ET, a change was deployed to the log ingestion filters which caused brokered service logs and metrics to be parsed incorrectly. As a result of this failed log parsing, brokered database logs were not visible to customers. On November 19, 2025 around 7:30 PM ET, the Cloud.gov team fixed the bug in the log ingestion filter code, so brokered service logs and metrics are now ingesting correctly and should be visible to customers. To restore the missing logs, the Cloud.gov team plans to re-ingest the brokered service logs and metrics that were parsed incorrectly from November 13 to November 19. To prevent a similar issue from occurring in the future, the Cloud.gov team plans to improve our automated tests to validate that all log types are ingested and parsed successfully after deploying changes.

2. postmortem Nov 20, 2025, 07:52 PM UTC

   ## Summary From November 13, 2025 around 1:40 PM ET to November 19, 2025 around 7:30 PM ET, brokered service logs and metrics were ingesting, but not visible to customers. ## Timeline * November 5 - A change to the parsing logic for brokered service logs and metrics is made * November 13, ~1:40 PM ET - Change to parsing logic is deployed into production, causing the logs to become invisible to customers * November 19, ~7:30 PM ET - A [Cloud.gov](http://Cloud.gov) engineer notices the ingestion failures and makes a temporary fix for the bug, making brokered service logs and metrics visible to customers again * November 20 - A permanent fix for the bug is prepared and merged ## Impact * Brokered service logs \(currently only available for RDS databases\) and brokered service metrics \(available for RDS, S3, and Elasticsearch/OpenSearch\) were not visible to customers during the incident ## Resolution The bug in the parsing logic was fixed, allowing brokered service logs and metrics to be ingested and restoring visibility to customers. ## Next Steps * Re-ingest the brokered service logs and metrics affected by the incident so they are visible to customers * Improve automated testing for the logs system to ensure that all log types ingest successfully and and are searchable, which will ensure that there is no regression in log visibility from future changes Thank you for your patience. If you have any questions, please contact us at [[email protected]](mailto:[email protected]).