Lightdash incident

Degraded performance and Snowflake Query Failures

Lightdash experienced a major incident on October 3, 2023 affecting Lightdash Cloud (US) and Lightdash Cloud (EU), lasting —. The incident has been resolved; the full update timeline is below.

Started: Oct 03, 2023, 03:39 PM UTC
Resolved: Oct 03, 2023, 03:39 PM UTC
Duration: —
Detected by Pingoru: Oct 03, 2023, 03:39 PM UTC

Affected components

Lightdash Cloud (US)Lightdash Cloud (EU)

Update timeline

resolved Oct 03, 2023, 03:39 PM UTC

We've successfully identified the root cause and have taken measures to resolve the issue. We apologize for any inconvenience this may have caused and appreciate your understanding and patience during this time.
postmortem Oct 03, 2023, 03:43 PM UTC

#### 2:11 PM * **Issue Reported**: Users report that dashboards aren’t loading any charts. #### 3:41 PM * **Additional Reports**: Other users report that app is crashing #### 3:14 PM - 4:30 PM * **Observation**: High backend latency was detected. * **Collaboration**: Engaged in a chat with other engineers to identify potential causes. * **Root Cause Analysis**: * Reviewed changes released that morning. * Suspected the Node upgrade might be the cause. * Discovered a related GitHub issue with Snowflake and Node 20: [GitHub Issue](https://github.com/snowflakedb/snowflake-connector-nodejs/issues/588) * **Decision**: * Immediate rollback. * Plan to create a PR to update the `snowflake-sdk` package. #### 4:25 PM * **Rollback**: Rolled back from release 0.797.0 to 0.795.0. #### 4:47 PM * **Feedback**: Users report that app is functioning correctly #### 5:10 PM * **Follow-up Action**: Began working on the PR to upgrade Snowflake's SDK.