Siena incident

Backfill Service Unavailability

Minor Resolved View vendor source →

Siena experienced a minor incident on January 1, 2025 affecting Backfill Service, lasting —. The incident has been resolved; the full update timeline is below.

Started
Jan 01, 2025, 12:00 AM UTC
Resolved
Jan 01, 2025, 12:00 AM UTC
Duration
Detected by Pingoru
Jan 01, 2025, 12:00 AM UTC

Affected components

Backfill Service

Update timeline

  1. resolved Jan 14, 2025, 10:26 AM UTC

    We experienced an issue with the Backfill Service, which ensures synchronisation of Shopify data in cases where webhooks are missing. During this time, orders for Shopify integrations were not available in our system, specifically affecting integrations where webhooks were disabled due to repeated non-200 responses.

  2. postmortem Jan 14, 2025, 10:26 AM UTC

    # Impact: Shopify webhooks were disabled due to repeated failures in returning 200-status responses during downtime, affecting 11 Shopify integrations. The backfill service, which compensates for missing data, was also down, causing major disruptions in data synchronization for orders, customers, and updates. # Timeline: * Dec 17: Backfill service stopped working * Dec 23 - 31: Multiple tickets raised for missing data. * Dec 31: Webhooks restored Jan 1: Backfill service restored & executed