Cleeng experienced a notice incident on April 27, 2024 affecting API, lasting 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 27, 2024, 10:28 AM UTC
At 00:45 CET on 27/04/2024, we saw a substantial growth in the traffic coming to our platform. Our Auto Scaling (ASG) kicked off and started adding more instances to the infrastructure. At 01:00, the spike was massive and the ASG could not keep up the rate of growth. This affected about 50% of traffic between 01:00 to 01:10. Between 1:10 to 1:40, the unhandled traffic reduced to 30%. We took further manual actions at 1:44 CET to scale to a larger number of instances. From 1:53 CET, the unhandled rate further improved to 5-10% and by 2:10, the infrastructure managed to handle all the incoming traffic as expected.
- resolved Apr 27, 2024, 10:29 AM UTC
The infrastructure works as expected from 02:10 CET on 27/04/2024
- postmortem Apr 29, 2024, 10:34 AM UTC
We have completed the RCA for this incident. If not received please request it via [[email protected]](mailto:[email protected])