Delivery Solutions incident
System Outage Due to Spike in Webhook Traffic
Delivery Solutions experienced a major incident on February 9, 2023 affecting Place Order and Labels and 1 more component, lasting 13h 24m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 09, 2023, 05:26 PM UTC
We are currently investigating this issue.
- identified Feb 09, 2023, 05:53 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Feb 09, 2023, 06:52 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Feb 09, 2023, 08:38 PM UTC
We observed an abnormal spike in incoming web-hook traffic which affected our database and platform services. We’ve rate limited the offending provider’s web-hooks and stopped certain fetch log entries. We will continue to monitor the traffic till 6 PM CST. While we’re in this phase, certain fetch log entries may contain discrepancies.
- resolved Feb 10, 2023, 06:51 AM UTC
As no escalated incidents have been observed, moving this incident to Resolved. The rate limit introduced will stay in place. A solution to handle fetch log discrepancies is being worked upon.