Pipefy Outage History

**Root Cause** The incident was caused by a cloud provider's routine maintenance which led to a problem with the network control system. This issue prevented the network from updating its configuration due to a slowdown in the internal database. Despite this, the Pipefy platform's performance remained unaffected. **Resolution** The incident was resolved after a series of actions were taken by a cloud provider’s team, which closed the issue.

**Root Cause** The issue was caused by exceeding the execution limit in integration queues, leading to a loop in automation that triggered numerous webhooks. This resulted in an overload of a specific flow, creating a large processing queue. Consequently, clients experienced delays in their process executions. **Resolution** To resolve the issue, the customer service account was deactivated, which helped alleviate the problem. **Action Plan** To prevent future occurrences, a rate limit will be activated on the system, and the timeout for execution flows will be reduced.

**Root Cause** We encountered a significant delay due to a high volume of card deletions, which affected the processing queues. **Resolution** The issue with the queues resolved itself without any intervention from the team. The system returned to normal operations on its own with the team supervision. **Action Plan**

**Root Cause** A sudden surge in login requests led to a failure due to the limited number of available connections for authentication. This caused users to be unable to log in, although those already logged in were not affected. **Resolution** The issue was resolved by rolling out the application after detecting a database failover. This action restored the system's functionality and allowed users to log in again. **Action Plan** We plan to set up alerts for database connections. Monitoring of connection pools will be enhanced. The security team is keeping an eye on suspicious login IP addresses to block them if necessary.

**Root Cause** We discovered an issue impacting filters and reports due to high memory usage in the database. The exact cause of the memory spike remains unidentified, prompting further investigation. **Resolution** The problem resolved itself without any manual intervention, and systems returned to normal operation. **Action Plan** We plan to enhance the reporting system to better monitor database performance. The team analyzed the situation and decided to optimize sorting methods to prevent future occurrences.

**Root Cause** The issue was caused by a recent third party software update, which led to the save button being disabled even after all authentication fields were completed. We identified the problem and addressed it with a new version that includes the necessary bug fix. **Resolution** We resolved the issue by implementing a new software update that contains the required bug fix, ensuring that the save button functions correctly after all fields are filled. **Action Plan** We plan to enhance our testing pipeline to ensure it is fully updated and consider involving experienced users to test in a staging environment for better reliability.

**Root Cause** The admin panel was unavailable for all organizations due to an error during the deployment process. A manual step in the code migration was missed, leading to missing data and a front-end failure. **Resolution** The issue was resolved by rolling back the changes and re-executing the migrations with the help of the support team. A fix was implemented to address the problem. **Action Plan** Improvements will be made to the migration pipeline to prevent similar issues in the future.

**Root Cause** The issue was caused by a non-optimized query that struggled when handling more than 50 records, leading to problems in database searches. **Resolution** The problem was resolved by applying a revert to address the issue. **Action Plan** To prevent similar issues in the future, unit tests will be created to avoid this type of failure scenario.

**Root Cause** A critical incident occurred due to a strategic update aimed at resolving security issues related to email feature. This incident affected key processes, including Integrations and AI automations, leading to failures in customer workflows. **Resolution** To resolve the issue, the team had to reverse the changes and execute a script to restore user accounts. This action was necessary to ensure the stability and functionality of the affected users. **Action Plan** * Future actions include setting up alerts for agent requests and specific error codes * Understanding the limitations of current alerts to be more proactive on this scenarios * Improve tests scenarios for cacth error previously

**Root Cause** An issue was detected with the system responsible for transferring data from our main database to another system. This was due to a temporary loss of connection, which caused delays in data processing. The system kept trying to reconnect until the issue was resolved, but this caused some delays in data. **Resolution** Fortunately, no data was lost during this incident. Once the connection was restored, all delayed data was successfully processed and updated in the system. **Action Plan**

**Root Cause** The issue was caused by users attempting to sign up with emails that are blacklisted, which led to failures during the account creation process. This behavior is expected in such scenarios. **Resolution** There were no specific resolution actions taken during the incident. **Action Plan** - We plan to create a fallback for customers in the user signup flow to prevent this scenario. - Improve internal communication to prevent similar tests issues in the future. - To create a mobile app deprecating communication to the customers

**Root Cause** The issue occurred because a change was made to only one part of a system element, but it needed to be applied to another part as well. This oversight led to a failure in loading the necessary data. **Resolution** The problem was resolved by reversing the previous change, which restored the system's ability to display the correct information. **Action Plan**

**Root Cause** The issue was caused by a failure in the system due to implementation in the project. This led to data fields displaying incorrect formats, affecting forms for users who were logged in. **Resolution** The problem was resolved by applying a fix that reverted the previous changes, which corrected the data format issues for logged users. **Action Plan** Moving forward, the system will be updated to preventing similar issues from occurring in the future.

**Root Cause** The AI Analytics feature experienced a complete failure, causing the system to hang in a 'processing' state without returning results. This issue arose after a deployment implemented **Resolution** The problem was resolved by implementing a fix to address the system's failure. This action restored the functionality of the AI Analytics feature, allowing it to process queries and return results as expected. **Action Plan** To prevent future incidents, we will enhance tests to better identify failures, Conduct post-implementation tests and improve the telemetry monitoring for the AI analytics feature.