Singa incident
Some of the users started the experience issues with Singa & Singa Pro around 5.30pm (UCT +2). We are investigating the issues at the moment.
Singa experienced a major incident on June 17, 2022 affecting API and Web and 1 more component, lasting 4h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 17, 2022, 03:00 PM UTC
We have detected issues and we are currently investigating, we’ll update you soon.
- monitoring Jun 17, 2022, 03:44 PM UTC
We are investing and fixing the issue. Multiple different web pages and services went down at the same time. Some of the users are already able to use Singa normally.
- resolved Jun 17, 2022, 04:09 PM UTC
Issue fixed. Happy singing!
- investigating Jun 17, 2022, 06:57 PM UTC
Our backend utilizes caching to improve the performance. When the cache is used the backend connects to cluster of cache nodes (i.e. servers). As soon as the incident began we saw that the cluster reached outbound bandwidth limit and dropped request from our backend apps. To increase available bandwidth we tried to add more nodes to the cluster but the cluster did not response to the request. The hosting service we use did not show any errors and no new nodes were ever created. Finally we initiated cluster reboot which restarted all the nodes and the cluster began responding to requests from our backend apps.