Truework incident

Elevated Synchronous API Response Times and Time to Verify due to Partner and Cloud Services Outage

Major Resolved View vendor source →

Truework experienced a major incident on December 7, 2021 affecting Truework API, lasting 1d. The incident has been resolved; the full update timeline is below.

Started
Dec 07, 2021, 04:20 PM UTC
Resolved
Dec 08, 2021, 05:07 PM UTC
Duration
1d
Detected by Pingoru
Dec 07, 2021, 04:20 PM UTC

Affected components

Truework API

Update timeline

  1. monitoring Dec 07, 2021, 04:20 PM UTC

    We have an issue with one of our payroll partners, causing synchronous requests to have elevated response times. We are working with the partner and monitoring this issue.

  2. monitoring Dec 07, 2021, 04:57 PM UTC

    We have temporarily blocked requests to this partner, returning response times to normal, at the cost of reducing hit rate while the partner is down.

  3. monitoring Dec 07, 2021, 11:19 PM UTC

    Our payroll partner is continuing to have issues, as are a number of other services we rely upon. The root cause of all issues is an outage in a major cloud services provider. We're continuing to monitor the issue and will announce when we have a resolution.

  4. monitoring Dec 08, 2021, 01:01 AM UTC

    We have resolved issues with our payroll partner and are now waiting for further recovery from other service providers.

  5. resolved Dec 08, 2021, 05:07 PM UTC

    All components are functioning normally, and we are now operating at full capacity.