SearchStax incident
Increased Error rate observed with Azure Deployments
SearchStax experienced a major incident on March 15, 2021 affecting Admin App and APIs, lasting 6h 40m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 15, 2021, 07:42 PM UTC
Our team has observed an increased error rate with Azure Management APIs. This affects provisioning new deployments in Azure, taking backups of Azure deployments and Total Requests metrics, and alert on Total Request for Azure deployments. Our team is investigating the issue.
- identified Mar 15, 2021, 07:50 PM UTC
Azure Status page is now reflecting the errors - https://status.azure.com/en-us/status "WarningAuthentication errors across multiple Microsoft services - Investigating SUMMARY OF IMPACT: Starting at approximately 19:15 UTC on 15 Mar 2021, a subset of customers may experience issues authenticating into Microsoft services, including the Azure Portal."
- identified Mar 15, 2021, 09:19 PM UTC
Updated status from Azure: "CURRENT STATUS: Engineering teams are currently rolling out mitigation worldwide. Customers should begin seeing recovery at this time, with full mitigation expected within 60 minutes." You can also check the latest status on the Azure site here: https://status.azure.com/en-us/status
- monitoring Mar 15, 2021, 11:17 PM UTC
Azure has rolled out a fix and all services are functional again. We will be closely monitoring the services for the next hour before we close the incident.
- resolved Mar 16, 2021, 02:23 AM UTC
This incident has been resolved.