Sisense incident
Sisense for Cloud Data Teams Cache Clusters returning ST003 Errors
Sisense experienced a major incident on December 22, 2021 affecting Sisense for Cloud Data Teams, lasting 20h 17m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 22, 2021, 02:21 PM UTC
Certain queries, caching, view materialization processes are running into ST003 errors. While the number of customers impacted is low, customers with the Redshift caching infrastructure are most affected. Engineers are actively working with AWS to resolve the issue. The Support Team can be reached at [email protected]
- investigating Dec 22, 2021, 02:38 PM UTC
We are continuing to investigate this issue.
- investigating Dec 22, 2021, 02:46 PM UTC
We are continuing to investigate this issue.
- investigating Dec 22, 2021, 03:12 PM UTC
Sisense for Cloud Data Teams Engineers have diagnosed the issue to be related to the AWS outage (https://status.aws.amazon.com). Engineering are actively working to resolve the issues. The Sisense Cloud Support Team can be reached at [email protected] or via live chat.
- investigating Dec 22, 2021, 03:20 PM UTC
Sisense for Cloud Data Teams Engineers have diagnosed the issue to be related to the AWS outage (https://status.aws.amazon.com). Engineering are actively working to resolve the issues. The Sisense Cloud Support Team can be reached at [email protected] or via live chat.
- investigating Dec 22, 2021, 04:45 PM UTC
The engineering team is actively monitoring the issues while AWS services are being restored
- identified Dec 22, 2021, 05:14 PM UTC
The engineering team is actively monitoring the issues while AWS services are being restored
- identified Dec 22, 2021, 06:05 PM UTC
The engineering team is actively monitoring the issues while AWS services are being restored
- identified Dec 22, 2021, 07:03 PM UTC
The engineering team is actively monitoring the issues while AWS services are being restored
- identified Dec 22, 2021, 08:06 PM UTC
The number of affected customers is decreased. The engineering team is actively monitoring the issues while AWS services are being restored
- identified Dec 22, 2021, 09:06 PM UTC
The engineering team is actively monitoring the issues while AWS services are being restored
- identified Dec 22, 2021, 09:58 PM UTC
Most of the customers should be able to start using cache. If the problem persists, The Sisense Cloud Support Team can be reached at [email protected] or via live chat. Meanwhile our Engineering team is working with AWS to restore the operations for all the customers.
- identified Dec 22, 2021, 11:12 PM UTC
The engineering team is actively monitoring the issues while AWS services are being restored
- identified Dec 23, 2021, 12:13 AM UTC
The engineering team is actively monitoring the issues while AWS services are being restored
- monitoring Dec 23, 2021, 12:14 AM UTC
We are continuing to monitor the issue as AWS works towards full recovery of Redshift clusters in the the US-East-1 region. Complete recovery will likely be reliant on the full recovery for the EC2 / EBS issue being tracked on the Service Health Dashboard (https://status.aws.amazon.com/ ). In some cases our Engineering team was able to recover the Redshift cluster on our own and are actively working through those instances of impacted clusters The Sisense Cloud Support Team can be reached at [email protected] or via live chat.
- monitoring Dec 23, 2021, 10:49 AM UTC
All Redshift Operations should be restored. In case of opposite contact Sisense, Cloud Support Team can be reached at [email protected] or via live chat.
- resolved Dec 23, 2021, 11:04 AM UTC
All Redshift Operations should be restored. In case of opposite contact Sisense, Cloud Support Team can be reached at [email protected] or via live chat.