Loops incident

Primary database outage

Major Resolved View vendor source →

Loops experienced a major incident on September 16, 2024 affecting Loops and API and 1 more component, lasting 38m. The incident has been resolved; the full update timeline is below.

Started
Sep 16, 2024, 02:16 PM UTC
Resolved
Sep 16, 2024, 02:54 PM UTC
Duration
38m
Detected by Pingoru
Sep 16, 2024, 02:16 PM UTC

Affected components

LoopsAPICampaignsAppTransactionalSMTP Relay

Update timeline

  1. investigating Sep 16, 2024, 02:19 PM UTC

    We are currently investigating this issue.

  2. monitoring Sep 16, 2024, 02:36 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. monitoring Sep 16, 2024, 02:37 PM UTC

    We are continuing to monitor for any further issues.

  4. monitoring Sep 16, 2024, 02:54 PM UTC

    We are continuing to monitor for any further issues.

  5. resolved Sep 16, 2024, 02:54 PM UTC

    This incident has been resolved.

  6. postmortem Sep 16, 2024, 02:55 PM UTC

    A misconfiguration by our database vendor prevented our database from restarting successfully. The misconfiguration was identified and remedied. Campaigns scheduled to send during the outage are now sent. Loop emails scheduled to send during the outage are now sent. Transactional emails or API calls made during the outage window need to be retried.