GitGuardian experienced a notice incident on October 18, 2022 affecting Webhooks, lasting 16s. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 18, 2022, 05:29 PM UTC
On October 17th 2022, at around 5pm GMT, The queuing system in charge of handling Github Check Runs and Real Time Monitoring accumulated tasks that were slow to process and impacted the whole system. The size of the task queue started increasing for several hours straight, delaying processing for all customers. until it started decreasing again and went back to normal the day after at around 1am GMT, where processing time was back to normal. This event occurred again on October 18th 2022, at around 4pm GMT, but has been detected earlier and have been solved at 5pm GMT. Actions were taken to avoid this issue to happen again: It includes a better monitoring of those queues to improve time to detect an issue. The montoring of processing latencies on a end-user point of view will be improved too. More long terms actions are also planned, including implementing mechanisms to avoid accumulating latency in case of slow tasks accumulations.
- identified Oct 18, 2022, 05:29 PM UTC
We are continuing to work on a fix for this issue.
- resolved Oct 18, 2022, 05:30 PM UTC
This incident has been resolved.