Patchman incident

Degraded performance on data processing infrastructure

Resolved after 8 hours 20 minutes, affecting Portal and Customer API and 1 more, on June 12, 2022.

monitoring Jun 12, 2022, 10:25 PM UTC

Performance is back to normal. We will continue to monitor the situation to ensure no further regressions.

Started
Jun 12, 2022, 10:09 PM UTC
Resolved
Jun 13, 2022, 06:30 AM UTC
Duration
8h 20m
Detected by Pingoru
Jun 12, 2022, 10:09 PM UTC

Affected components

PortalCustomer APIData Processing Infrastructure

Update timeline

  1. identified Jun 12, 2022, 10:09 PM UTC

    Our backend workers are suffering from degraded performance, which we have traced to our database cluster. We are currently preparing actions to redirect the load to other cluster workers as a means of remediating the performance problems.

  2. identified Jun 12, 2022, 10:15 PM UTC

    As part of the remediation steps, there may be a temporary loss of availability of the Portal and Customer API, expected to be limited to a maximum of 1 minute.

  3. identified Jun 12, 2022, 10:21 PM UTC

    Portal and Customer API are no longer affected, remediating actions for the backend workers are still ongoing.

  4. monitoring Jun 12, 2022, 10:25 PM UTC

    Performance is back to normal. We will continue to monitor the situation to ensure no further regressions.

  5. resolved Jun 13, 2022, 06:30 AM UTC

    This incident has been resolved.

Your product runs on more than Patchman

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →