resolved Mar 19, 2026, 09:11 PM UTC
Summary From March 17, 2026 around 9:30 AM ET to March 19, 2026 around 3:10 PM ET, some application JSON logs were not ingested successfully to Cloud.gov Logs. Timeline - March 6, 2026, 10:25 AM ET - Changes were merged to the logging system configuration to make some subfields of application JSON logs searchable and aggregatable - March 17, 2026, 9:30 AM ET - Changes to JSON log field parsing begin deploying to Cloud.gov Logs system. Some JSON application logs began to fail ingestion at this point - March 19, 2026, 2:39 PM ET - Automated testing alerts Cloud.gov engineers to ingestion failures in Cloud.gov Logs. Engineers begin to investigate. - March 19, 2026, 2:59 PM ET - A Cloud.gov engineer determines that updated JSON application log parsing of timestamp fields is causing some logs to fail ingestion and slowing the overall log ingestion rate - March 19, 2026, 3:06 PM ET - A fix is deployed to change the field type to “string” for JSON log timestamp fields March 19, 2026, 3:10 PM ET - JSON log ingestion errors are resolved and logs are ingesting successfully Impact Only JSON application logs that included a “ts” field or a “timestamp” field which could not be properly parsed as a date (e.g. “1.8543923523”) failed to ingest to Cloud.gov Logs during the incident. While these logs were not ingested to Cloud.gov Logs successfully, they were ingested to offline storage that is not accessible to customers, but can be accessed by the Cloud.gov engineers if necessary. Resolution The type of the timestamp field for JSON application logs was changed to “string”, which allows even values that aren’t valid timestamps or dates to be ingested. Next Steps The Cloud.gov team will hold a retrospective to further analyze the causes of this incident and how to improve our operations. We will post our findings as a post-mortem to this incident. Thank you for your patience. If you have any questions, please contact us at [email protected].
