Runpanther incident
Slack Alert Delivery Issues November 17, 2022
Runpanther experienced a major incident on November 17, 2022 affecting Alert Delivery, lasting 6h 57m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 17, 2022, 02:22 PM UTC
We have received additional reports of alert delivery failures for customers using our Slack webhook integration after deployment of the patch yesterday (https://status.panther.com/incidents/v4bthbs6sbx4). Our engineering team is working on an additional patch to resolve these new issues. We will update this incident with more information shortly.
- identified Nov 17, 2022, 04:04 PM UTC
Our team has begun deploying the patch to customer accounts. We will update this incident again after the deployment is complete and our monitoring indicates that the issues have been resolved.
- resolved Nov 17, 2022, 09:19 PM UTC
The patch containing the fix for this issue has been deployed to all impacted customer accounts. Alert delivery to Slack destinations using webhooks are now functioning as intended. To check for alerts that failed during this time and manually resend them, please follow the instructions below: 1. Open the Panther Console and navigate to Alerts. 2. Click the name of an alert that shows "Destination Delivery Failed". 3. Next to "Alert delivery failed", click "Show History". 4. Click the retry delivery button.