Hightouch incident

Syncs on us-east-1 may be delayed

Minor Resolved View vendor source →

Hightouch experienced a minor incident on July 22, 2024 affecting AWS us-east-1 (N. Virginia), lasting 4h 24m. The incident has been resolved; the full update timeline is below.

Started
Jul 22, 2024, 02:16 PM UTC
Resolved
Jul 22, 2024, 06:41 PM UTC
Duration
4h 24m
Detected by Pingoru
Jul 22, 2024, 02:16 PM UTC

Affected components

AWS us-east-1 (N. Virginia)

Update timeline

  1. investigating Jul 22, 2024, 02:16 PM UTC

    Reverse ETL syncs on AWS us-east-1 are currently delayed, expect some syncs to be in "Queued" status for a while.

  2. investigating Jul 22, 2024, 03:54 PM UTC

    We are continuing to investigate this issue. In the meantime, syncs are continuing to run but some syncs are experiencing a delay of ~60-90 minutes.

  3. identified Jul 22, 2024, 04:55 PM UTC

    We have identified the issue and are working to resolve this as quickly as we can. Syncs are continuing to run but some seeing delays now ~2-3 hrs. We do apologize for this inconvenience and will keep you posted as things progress.

  4. monitoring Jul 22, 2024, 05:21 PM UTC

    We have added a performance optimization to more efficiently queue syncs. All queued syncs should be running within the next 20-30 mins.

  5. monitoring Jul 22, 2024, 05:44 PM UTC

    All syncs should be run as scheduled. We're continuing to monitor the system performance.

  6. resolved Jul 22, 2024, 06:41 PM UTC

    This incident has now been resolved. Syncs are running on schedule as expected.