Simwood incident

Intermittent API/portal timeout

Minor Resolved View vendor source →

Simwood experienced a minor incident on November 29, 2021 affecting API and Portal, lasting 3h 44m. The incident has been resolved; the full update timeline is below.

Started
Nov 29, 2021, 12:54 PM UTC
Resolved
Nov 29, 2021, 04:39 PM UTC
Duration
3h 44m
Detected by Pingoru
Nov 29, 2021, 12:54 PM UTC

Affected components

API and Portal

Update timeline

  1. investigating Nov 29, 2021, 12:54 PM UTC

    We are currently investigating this issue.

  2. identified Nov 29, 2021, 01:04 PM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Nov 29, 2021, 01:23 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Nov 29, 2021, 04:39 PM UTC

    Reports are processed sequentially and a single customer was effectively causing a Denial of Service with thousands of requests for a heavy report, which was in turn delaying others beyond the timeout the API has. The API, portal and the platform in general were otherwise fine and this functioned as expected.