Sigma Computing incident
Materialization and CSV upload failures
Sigma Computing experienced a critical incident on April 19, 2023 affecting Sigma - GCP (US) and Sigma - AWS (US), lasting 3h 15m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 19, 2023, 05:16 PM UTC
Current Status: We are investigating an issue causing failures on write-back queries for materializations and CSV uploads. We are treating this as our highest priority and we expect this issue to be transitory. Our apologies for any inconvenience caused. Impacted Cloud Regions & Services: All Connection types, AWS and GCP deployments User Experience: Users are experiencing failures on materializations and CSV uploads Incident Start Time: Approximately 13:00 UTC April 17, 2023
- monitoring Apr 19, 2023, 06:59 PM UTC
Current Status: Our engineers have implemented mitigation measures to circumvent the issue and we are monitoring their effect closely. Impacted Cloud Regions & Services: All Connection types, AWS and GCP deployments User Experience: We expect the rate of errors to decrease for all materializations and CSV uploads Incident Start Time: Approximately 13:00 UTC April 17, 2023 Incident End Time: Approximately 17:30 UTC April 19, 2023
- monitoring Apr 19, 2023, 07:41 PM UTC
Current Status: Our engineers have implemented mitigation measures to circumvent the issue and we are monitoring their effect closely. Impacted Cloud Regions & Services: All Connection types, AWS and GCP deployments. User Experience: We expect the rate of errors to decrease for all materializations and CSV uploads. Incident Start Time: Approximately 13:00 UTC April 17, 2023 Incident End Time: Approximately 17:30 UTC April 19, 2023 We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days. You will receive a notification when that occurs.
- resolved Apr 19, 2023, 08:32 PM UTC
Current Status: Our engineers have implemented mitigation measures to circumvent the issue and we are seeing materializations and CSV uploads are operational now. Impacted Cloud Regions & Services: All Connection types, AWS and GCP deployments. User Experience: We expect users should no longer see materialization and CSV upload failures due to this incident. Incident Start Time: Approximately 13:00 UTC April 17, 2023 Incident End Time: Approximately 17:30 UTC April 19, 2023 We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days. You will receive a notification when that occurs.
- postmortem Apr 26, 2023, 09:08 PM UTC
Sigma Engineering has completed the postmortem of this incident. A Root Cause Analysis \(RCA\) is available on the Sigma Community site: [Postmortem for April 17, 2023 Materialization and CSV Upload incident - Sigma AWS & Sigma GCP](https://community.sigmacomputing.com/t/postmortem-for-april-17-2023-materialization-and-csv-upload-incident-sigma-aws-sigma-gcp/2296)