Magic Labs experienced a minor incident on July 25, 2022 affecting Authentication and API and 1 more component, lasting 51m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 25, 2022, 02:25 AM UTC
You may temporarily experience increased latency and error rates. We are actively investigating the degraded performance issue.
- monitoring Jul 25, 2022, 02:25 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 25, 2022, 03:16 AM UTC
This incident has been resolved.
- postmortem Aug 11, 2022, 06:54 PM UTC
# **What happened** A spike in traffic occurred and our API pods did not scale out in enough time. The following were not possible or experienced partly degraded service: * API Login * Logging into the Magic Dashboard # **How we responded** During the outage, our engineers rapidly joined a virtual war-room to triage the situation and to find the fastest, most impactful step forward. After investigating the issue, we began to increase our overall server fleet size, which stabilized the traffic spike. We also found that our scale-up policies had room for improvement. Our team immediately modified our scale-up policies to better accommodate for high traffic spikes in the future, which we were able to observe during later traffic spikes.