Allium incident

Stale snowflake data on non-uscentral regions

Minor Resolved View vendor source →

Allium experienced a minor incident on May 27, 2025 affecting Snowflake Snowflake Data Warehouse (Database), lasting 19h 52m. The incident has been resolved; the full update timeline is below.

Started
May 27, 2025, 01:45 AM UTC
Resolved
May 27, 2025, 09:38 PM UTC
Duration
19h 52m
Detected by Pingoru
May 27, 2025, 01:45 AM UTC

Affected components

Snowflake Snowflake Data Warehouse (Database)

Update timeline

  1. investigating May 27, 2025, 01:45 AM UTC

    We are currently investigating an issue with our data infra provider. Snowflake data on non-uscentral locations will be stale

  2. investigating May 27, 2025, 04:08 AM UTC

    We have escalated the issue to snowflake, will update promptly once we have an ETA to resolution. Status: - Data freshness on snowflake shares in non-uscentral regions is still degraded - Any existing shared data is still available for query - Allium app is available for queries

  3. investigating May 27, 2025, 08:25 AM UTC

    We are still working with snowflake, providing details to help them investigate the issue. Still pending resolution on their end.

  4. identified May 27, 2025, 05:14 PM UTC

    Snowflake engineering identified a fix that is undergoing testing. In the meantime, we have identified mitigations that we are actively implementing in parallel. The allium app remains available for queries. Thank you for your patience.

  5. identified May 27, 2025, 08:30 PM UTC

    The following chains/assets are now fully recovered in all regions with updates: - Base - Arbitrum - Avalanche - Celo - Common - Berachain - Solana balances

  6. resolved May 27, 2025, 09:38 PM UTC

    All tables have been restored and freshness is back to normal. Thank you for your patience! Quick summary on what happened: - Early today at approximately 2025-05-27 0000UTC, Snowflake rolled out a change that caused a regression in a critical component of our data replication pipeline. - This caused new data updates to not be propagated to non-uscentral regions. - Snowflake has since rollback the change and the data replication pipeline is now functional. More details on the outage by end of week.