ProcedureFlow experienced a notice incident on April 28, 2020, lasting 24m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Apr 28, 2020, 08:37 PM UTC
We are currently investigating this issue.
- identified Apr 28, 2020, 08:40 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 28, 2020, 08:44 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 28, 2020, 09:02 PM UTC
This incident has been resolved. Once we understand the full scope of this issue, we'll provide a postmortem.
- postmortem May 04, 2020, 05:54 PM UTC
On April 28, 2020 between 4:33 PM and 4:39 PM Eastern, we experienced slow requests, connectivity problems, and a sporadic outage lasting ~5 minutes affecting all customers. We’re very sorry this occurred and would like to share with you what happened, what we learned, and what we're doing to prevent incidents like this from happening in the future. This incident was caused by cascading problems: reduced capacity during maintenance, normal daily mid-peak load, poor failure modes for certain low-priority requests, and a number of slow/unoptimized requests which blocked other requests. We were making maintenance-related changes in response to [an incident from the previous week](https://status.procedureflow.com/incidents/f2k55tftxcjp). At the time of the previous incident, we thought that the root cause was different based on the deployment we were making at the time. This assumption was wrong and caused the same issue to happen again when we made these changes. We've written [a postmortem for the previous incident](https://status.procedureflow.com/incidents/f2k55tftxcjp) which describes what we've learned and what we're doing to prevent incidents like this from happening in the future. We know how much you rely on ProcedureFlow to help your business succeed. Having 2 incidents in one week is not something we're proud of, but we will continue to analyze this event for opportunities to serve you better and earn the trust you place in us.