Peakon experienced a minor incident on September 2, 2019 affecting Dashboard and API, lasting 1h 55m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 02, 2019, 08:41 AM UTC
We are seeing errors performing searches in Peakon. We are investigating the issue.
- identified Sep 02, 2019, 09:39 AM UTC
We have identified the issue as being a heavily loaded search cluster. We are working to determine the root cause and add additional capacity.
- identified Sep 02, 2019, 09:57 AM UTC
We have disabled all search for now, as we recover the search cluster. This is in progress and we expect that to be completed soon.
- monitoring Sep 02, 2019, 10:13 AM UTC
We believe we have solved the underlying issue and have reenabled search. It might be slower than usual while things stabilize.
- resolved Sep 02, 2019, 10:37 AM UTC
This issue is now resolved. Performance is back to normal. While we have fixed the cause of the issue, we will be working thorugh a post mortem process. This will help us prevent similar incidents in the future, allow us to quicker diagnose and fix similar and related issues as well as improve processes.