SEON experienced a major incident on November 24, 2021 affecting REST API (REST Service - EU Region) and REST API (REST Service - US Region), lasting —. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- resolved Nov 24, 2021, 04:30 PM UTC
At 16:26 UTC our monitoring system indicated elevated load on our API and database infrastructure. At 16:28 UTC this issue escalated further and caused an outage in our API services. As the alarm went out, our 24/7 operations team started the recovery and all systems were operational by 16:39 UTC. We had two brief performance degradations during the recovery process (16:48-16:50 and 17:06-17:08) that affected a small number of our customers. The issue was caused by a misconfiguration in our Sense Platform that led to increased database load which caused transactions to time out in our API layer. Our engineering team is working on performance improvements to prevent such issues from happening again.