Patchman incident

Intermittent failure on data processing backend

Major Resolved View vendor source →

Patchman experienced a major incident on April 2, 2020 affecting Data Processing Infrastructure, lasting 6h 8m. The incident has been resolved; the full update timeline is below.

Started
Apr 02, 2020, 06:48 AM UTC
Resolved
Apr 02, 2020, 12:57 PM UTC
Duration
6h 8m
Detected by Pingoru
Apr 02, 2020, 06:48 AM UTC

Affected components

Data Processing Infrastructure

Update timeline

  1. investigating Apr 02, 2020, 06:48 AM UTC

    We are currently investigating an issue with our data processing backend, preventing synchronization between agents and the Patchman Portal from being executed correctly.

  2. monitoring Apr 02, 2020, 06:55 AM UTC

    We have identified the problem and implemented a solution. All systems are resuming normal operation. We continue to keep a close watch on the situation, while we roll out additional failsafes.

  3. monitoring Apr 02, 2020, 07:26 AM UTC

    Systems are stable. We are currently working through the backlog of missed messages of several hours. Current expectations are this will be done within a maximum of 30 minutes. The Portal may show slightly out-of-date information until this is complete.

  4. monitoring Apr 02, 2020, 08:10 AM UTC

    All systems are stable. We will continue to monitor the situation for regressions.

  5. resolved Apr 02, 2020, 12:57 PM UTC

    This incident has been resolved.