ProcedureFlow experienced a major incident on April 22, 2025 affecting Application (https://app.procedureflow.com), lasting 21h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 22, 2025, 04:40 PM UTC
We are currently investigating an issue where change requests are sometimes not opening/saving.
- identified Apr 22, 2025, 05:13 PM UTC
The issue has been identified and a fix is being implemented. We will continue to share updates as they become available.
- identified Apr 22, 2025, 07:55 PM UTC
We are currently experiencing an issue where customers are unable to create change requests (or save drafts via the "old" UI). This is affecting all customers. The issue is related to the database limit for storing changes in change requests. We’ve hit a threshold that prevents us from adding more records, which is causing an error in the user interface. There is no data loss of existing change requests. Our team is actively working on fixing this. We are running a data migration in the background to increase the limit. We expect this data migration to be completed around 11 PM Eastern today and then will perform additional steps in our resolution plan. Unfortunately, until this issue is resolved, you will not be able to create new change requests/save new drafts. We recognize that this situation has impacted your workflow, and we're taking steps to avoid similar issues in the future. We'll continue to provide updates here and share a postmortem once the issue is resolved. If you have any questions, please contact us, and we can schedule a call to discuss further.
- identified Apr 23, 2025, 09:23 AM UTC
The data migration has completed. We are working through additional steps to use the new migrated data.
- monitoring Apr 23, 2025, 09:56 AM UTC
We have completed all steps to fix this issue and are monitoring the results.
- monitoring Apr 23, 2025, 09:57 AM UTC
We are continuing to monitor for any further issues.
- resolved Apr 23, 2025, 01:42 PM UTC
This incident has been resolved.
- postmortem May 02, 2025, 01:56 PM UTC
From approximately 12:15 PM ET on April 22 to 6:00 AM ET on April 23, all customers were unable to create change requests due to a database issue. Editing existing change requests was mostly unaffected. ## Root cause One of our database tables reached the maximum number of allowed entries \(~2.1 billion\), preventing change requests from being created. The maximum number was reached due to growth of ProcedureFlow usage. Although we have monitoring in place for database limits, we underestimated the rate of growth and were only monitoring these types of limits infrequently. ## Resolution The issue required an urgent database migration to expand the ID capacity and restore normal functionality. The migration was completed successfully without any data loss, and system availability for viewing and editing existing data was maintained throughout. The database migration and related mitigation steps took approximately 18 hours to complete due to the large volume of data involved. Under normal circumstances, we would perform this type of migration gradually and without any customer impact. However, because this issue affected all customers and prevented critical functionality, we prioritized speed and executed the migration as quickly as the system would allow, balancing urgency with data integrity and stability. ## Improvements We've completed the necessary upgrades to prevent this issue from happening again and have made improvements to our monitoring and operational processes to enable faster detection and recovery in the future. For example: * We've implemented more frequent manual and automated checks for database table entry limits. * We're monitoring and planning upgrades for any legacy tables that have the ~2.1 billion entry limit. All newer tables in ProcedureFlow do not have this restriction. --- We sincerely apologize for the disruption caused by this issue and understand the impact it had on your ability to make changes in ProcedureFlow. We are committed to ensuring this doesn't happen again, and we are continuously working to make ProcedureFlow more reliable and resilient. We value your trust and appreciate your patience as we continue to enhance the reliability of ProcedureFlow.