Allium incident

Degraded Solana Performance

Major Resolved View vendor source →

Allium experienced a major incident on September 4, 2025, lasting 3d 13h. The incident has been resolved; the full update timeline is below.

Started
Sep 04, 2025, 06:02 AM UTC
Resolved
Sep 07, 2025, 07:35 PM UTC
Duration
3d 13h
Detected by Pingoru
Sep 04, 2025, 06:02 AM UTC

Update timeline

  1. investigating Sep 04, 2025, 06:02 AM UTC

    We are currently investigating this issue.

  2. investigating Sep 04, 2025, 12:32 PM UTC

    We are continuing to investigate the issues. Preliminary investigations suggests that the getBlock RPC call is facing intermittent errors. We will mitigate the issue by refetching the blocks, but freshness may be degraded.

  3. identified Sep 05, 2025, 01:21 AM UTC

    Data fixes has been rolled out to Solana tables in Snowflake GCP us-central1 and realtime APIs. We are actively working on propagating all data fixes to all other Snowflake, Databricks, and GCP BigQuery destinations.

  4. monitoring Sep 05, 2025, 08:51 PM UTC

    The data fix is propagated to all downstream destinations. We are continuing to monitor effects. Note: There are reports of Kafka freshness issues - this might be due to backfilling older events into the topic. You may see occasional spikes.

  5. resolved Sep 07, 2025, 07:35 PM UTC

    This incident has been resolved.