Tenfold incident

[Sev 1] Possible intermittent latency on tenfold API

Minor Resolved View vendor source →

Tenfold experienced a minor incident on July 1, 2020 affecting API and Dashboard, lasting 3h 41m. The incident has been resolved; the full update timeline is below.

Started
Jul 01, 2020, 07:18 PM UTC
Resolved
Jul 01, 2020, 11:00 PM UTC
Duration
3h 41m
Detected by Pingoru
Jul 01, 2020, 07:18 PM UTC

Affected components

APIDashboard

Update timeline

  1. investigating Jul 01, 2020, 07:18 PM UTC

    Tenfold automated monitoring systems have identified an elevated level of errors with our API. Users may experience intermittent latency in the application or dashboard. Our team is currently investigating and will provide updates as they are made available.

  2. investigating Jul 01, 2020, 07:30 PM UTC

    We are continuing to investigate this issue.

  3. investigating Jul 01, 2020, 07:31 PM UTC

    We are continuing to investigate this issue.

  4. resolved Jul 01, 2020, 11:00 PM UTC

    This incident has been resolved.

  5. postmortem Jul 02, 2020, 05:46 PM UTC

    Tenfold observed increases in API latency due to caching within the database, which was an artifact of the earlier outage with Google. Tenfold monitoring successfully alerted our operations team who able to implement new infrastructure improvements to alleviate the caching. Tenfold end users may have experienced brief periods of latency. There was not data loss as a result of this event.