Togetherwork incident

[SEV-3] Togetherpay Sandbox and Integration Unavailable

Critical Resolved View vendor source →

Togetherwork experienced a critical incident on December 15, 2023 affecting Integration Test Environment and Sandbox Test Environment, lasting 25d. The incident has been resolved; the full update timeline is below.

Started
Dec 15, 2023, 03:51 PM UTC
Resolved
Jan 09, 2024, 04:21 PM UTC
Duration
25d
Detected by Pingoru
Dec 15, 2023, 03:51 PM UTC

Affected components

Integration Test EnvironmentSandbox Test Environment

Update timeline

  1. investigating Dec 15, 2023, 03:51 PM UTC

    Togetherpay sandbox and integration are both down - 503 Service Temporarily Unavailable. Teams are investigating. An update will be posted as soon as we have more information.

  2. investigating Dec 15, 2023, 08:47 PM UTC

    Teams continue to investigate this issue. As details emerge, we will let you know.

  3. investigating Dec 16, 2023, 05:58 PM UTC

    Togetherpay sandbox and integration are still unavailable. Teams are still working to understand the issue. We will provide updates as more is known.

  4. investigating Dec 18, 2023, 09:05 PM UTC

    Our teams are still working to resolve the issue with sandbox and integration. We will continue to provide updates as more is known.

  5. investigating Dec 19, 2023, 03:50 PM UTC

    Togetherpay sandbox and integration are still unavailable. Teams continue to work to resolve the issue. We will continue to provide updates.

  6. investigating Dec 21, 2023, 05:59 PM UTC

    Teams continue to investigate this issue. We will continue to provide updates as more is known.

  7. investigating Dec 26, 2023, 04:14 PM UTC

    Togetherpay sandbox and integration are still down. Teams are working to restore both environments. We will provide updates as more is known.

  8. identified Dec 27, 2023, 05:39 PM UTC

    Teams identified the issue and are working on a fix which will take some time to implement. Completion of the fix is targeted for mid next week. We will keep you posted as we learn more.

  9. identified Jan 03, 2024, 08:00 PM UTC

    Teams continue to implement a fix for Sandbox and Integration. We will provide updates as we learn more.

  10. identified Jan 05, 2024, 05:07 PM UTC

    Teams encountered an unforeseen issue resulting in further delays. At this time there is no ETA on the fix. As progress is made, we will let you know.

  11. identified Jan 08, 2024, 05:35 PM UTC

    APIs to both sandbox and integration are fully functioning. Front end access to the Togetherpay application for both environments is still down. Teams continue work to restore application access. Updates will be posted as progress is made.

  12. monitoring Jan 08, 2024, 07:35 PM UTC

    A fix has been implemented and we are monitoring the results.

  13. resolved Jan 09, 2024, 04:21 PM UTC

    Togetherpay sandbox and integration environments are both fully functioning. This incident is resolved.