Tyk incident
Control Plane Redis storage instability in aws-ap-southeast-1
Tyk experienced a minor incident on August 29, 2023 affecting Controller at aws-ap-southeast-1, lasting 4h 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 29, 2023, 01:54 PM UTC
Our monitoring has alerted about an increase in storage-related errors in the Redis Clusters across some of the Control Plane deployments in the aws-ap-southeast-1 zone. The root cause is currently unknown, but the mitigation procedure has been identified and put in action. There is no direct impact from the issue, however the mitigation might cause a short (several minutes) downtime in the Control Plane deployments of the zone.
- identified Aug 29, 2023, 03:01 PM UTC
The cause of the storage instability is identified and there's no risk of re-occurrence currently. Meanwhile, mitigation procedures for the remaining deployments to prevent future data loss are in progress.
- identified Aug 29, 2023, 05:11 PM UTC
Majority of the deployments have successfully undergone the mitigation procedure. We're working to apply this to the rest of the affected deployments.
- monitoring Aug 29, 2023, 05:46 PM UTC
The mitigation procedures have been successfully applied to all the deployments.
- resolved Aug 29, 2023, 06:27 PM UTC
This incident has been resolved.