Sigma Computing incident
Materialization & Scheduled Exports Degraded Performance
Sigma Computing experienced a minor incident on April 20, 2023 affecting Sigma - GCP (US) and Sigma - AWS (US), lasting 12h 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 20, 2023, 01:56 PM UTC
Current Status: We are investigating technical issues leading to degraded performance for both materializations and scheduled exports. This is currently our highest priority. Apologies for any inconvenience caused. Impacted Cloud Regions & Services: AWS & GCP User Experience: Users may experience sporadic failures for materializations and scheduled exports now and in the past Incident Start Time: Approximately 11:50am UTC April 20, 2023
- monitoring Apr 20, 2023, 03:29 PM UTC
Current Status: We have implemented mitigation measures and are monitoring their effect closely. The occurrence of errors has declined substantially. Impacted Cloud Regions & Services: AWS & GCP User Experience: Users would have experienced sporadic failures for materializations and scheduled exports. Incident Start Time: Approximately 11:50am UTC April 20, 2023
- monitoring Apr 20, 2023, 05:40 PM UTC
Current Status: We have implemented mitigation measures and are monitoring their effect closely. The occurrence of errors has declined substantially. As a consequence of this mitigation, users may need to manually sync their databases/schemas/tables if changes done in the warehouse are not showing in Sigma. Additionally, navigating newly-created connections may be slow and their search functionality will not work. Apologies for any inconvenience caused. Impacted Cloud Regions & Services: AWS & GCP User Experience: Users would have experienced sporadic failures for materializations and scheduled exports. Users may need to manually sync their databases/schemas/tables if changes done in the warehouse are not showing in Sigma. Additionally, navigating newly-created connections may be slow and their search functionality will not work. Incident Start Time: Approximately 11:50am UTC April 20, 2023
- resolved Apr 21, 2023, 02:15 AM UTC
Current Status: We have implemented mitigation measures and are monitoring their effect closely. The occurrence of errors has declined substantially. Apologies for any inconvenience caused. Impacted Cloud Regions & Services: AWS & GCP User Experience: We expect that the rate of failures for materializations and scheduled exports to be substantially decreased. Users should not need to manually sync their databases/schemas/tables anymore at this time, changes done in the warehouse should automatically show in Sigma again. Navigating newly-created connections should also be functional. Incident Start Time: Approximately 11:50 UTC April 20, 2023 Incident End Time: Approximately 12:30am UTC April 21, 2023 We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days. You will receive a notification when that occurs.
- postmortem Apr 30, 2023, 12:19 AM UTC
Sigma Engineering has completed the postmortem of this incident. A Root Cause Analysis \(RCA\) is available on the Sigma Community site: [Postmortem For April 20, 2023 Materialization and Scheduled Export Incident - Sigma AWS & Sigma GCP](https://community.sigmacomputing.com/t/postmortem-for-april-20-2023-materialization-and-scheduled-export-incident-sigma-aws-sigma-gcp/2297)