Adnuntius experienced a notice incident on July 15, 2024 affecting Administration Backend Server, lasting 1h 43m. The incident has been resolved; the full update timeline is below.
Affected components
Administration Backend Server
Update timeline
- investigating Jul 15, 2024, 06:20 AM UTC
Reports are currently stuck in processing or not being executed.
- identified Jul 15, 2024, 06:30 AM UTC
This issue has been identified and a fix is being prepared.
- monitoring Jul 15, 2024, 07:04 AM UTC
A fix has been implemented and and the backlog of reports has now finished processing.
- resolved Jul 15, 2024, 08:03 AM UTC
The cause was an update to our report execution system to make the report status more legible, this caused an issue with retrying reports which caused many unnecessary report executions which blocked the queue of reports. We will implement better testing for the report retry logic.