Patchman experienced a major incident on April 8, 2019 affecting Agent API and Data Processing Infrastructure, lasting 2d 3h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 08, 2019, 08:57 AM UTC
Our data processing cluster is currently experiencing performance issues, causing a lag in processing of incoming messages from agents. This is resulting in various pieces of information in the Portal being out-of-date, including scan times and detection data. We are currently working to identify the root cause of this problem, and will provide periodic updates here regarding our progress.
- investigating Apr 08, 2019, 10:49 AM UTC
We are still in the process of identifying the root cause of this slowdown. If, in the meantime, you have any questions about whether something you're seeing in the Portal is related to this incident, feel free to reach out to us at [email protected].
- investigating Apr 08, 2019, 03:09 PM UTC
We are currently performing emergency maintenance on the cluster, in order to resolve at least part of the performance problems. You may temporarily see Agent API connections cycle as a result of this.
- investigating Apr 08, 2019, 03:20 PM UTC
The Agent API is fully functional again, and performance is increasing on the data processing cluster. We are investigating whether this has indeed resolved the issues in full.
- monitoring Apr 08, 2019, 03:29 PM UTC
All systems in our processing cluster are currently performing as intended. We will continue to monitor the situation for any regressions.
- resolved Apr 10, 2019, 12:21 PM UTC
This incident has been resolved.