Pipefy Outage History

**Root Cause** The problem was cause by new implementation using transaction finance API, which affected the purchasing service. The issue was identified and resolved by reverting the change. **Resolution** The problem was resolved by reverting the implementation, restoring the service to normal. **Action Plan**

**Root Cause** The exact cause of the issue is still under investigation. The team responsible for the dashboards is exploring a new strategy involving cache tecnologies to determine its impact on the problem. **Resolution** The issue was resolved by restarting the system responsible for data caching, which restored normal functionality. **Action Plan** To prevent future occurrences, the team will enhance the monitoring system for the dashboard application to better detect and track failures. Additionally, they are considering replacing the current data caching system with a more efficient solution.

**Root Cause** The exact cause of the issue is still being investigated. The team responsible for the dashboard is exploring a new method involving the cache technologie to determine if it will effectively address the problem. **Resolution** The issue was resolved by changing the system from using new cache data, which restored the functionality of the dashboard. **Action Plan**

**Root Cause** Some organizations faced email delivery issues due to their emails being blacklisted. This was caused by certain IPs being blocked, which affected the ability to send emails. **Resolution** The issue was mitigated by removing the problematic IP from the range, allowing emails to be sent through other available IPs. Additionally, a support ticket was opened to resolve the IP block.

**Root Cause** The issue was caused by a database instance being unavailable due to a node failure. This led to a large number of open transactions, resulting in errors and preventing users from signing in. **Resolution** The affected node was recycled and restored, allowing users to log in again without further issues. **Action Plan** New alerts have been set up to monitor errors related to open transactions. Further investigation will be conducted to understand why these transactions were left open, and regular maintenance will be performed to prevent them from remaining unresolved.

**Root Cause** The issue arose because the chat records were not linked to the channel object as expected. This lack of connection led to the problems experienced. **Resolution** To resolve the issue, the project was rolled back to a previous version, and changes were reversed. A fix was then applied to ensure that the chat and channel relationship is established, preventing further errors. **Action Plan** During the incident, changes were reversed. Afterward, a fix was implemented to establish the chat-channel relationship, preventing recurrence. Additionally, alerts were set up to notify about any future issues.

**Root Cause** The issue arose because the subscription plan records in production were missing a crucial field, leading to a failure in the system. This caused the interfaces to become inaccessible as they couldn't retrieve the necessary information. **Resolution** The solution involved reverting the changes that introduced the bug in the production environment, thereby restoring the system's functionality. **Action Plan** To prevent future occurrences, improvements were made to the error handling screen, a deployment was executed to enhance query resilience, and additional validation was implemented in testing to ensure subscription functionality.

**Root Cause** The email service account was temporarily suspended due to the detection of suspicious email activity. This led to issues with sending emails. **Resolution** A new account was created to handle future email dispatches, ensuring continuity in email services. **Action Plan** We are exploring the option of having an alternative provider as a backup. Additionally, we will disable messaging capabilities for free organizations, curate emails before processing, and collect metrics to monitor performance.

**Root Cause** The issue was caused by an error in the command that initiates the data processing system, leading to difficulties in using filters and reports for newly created entries. **Resolution** The problematic update that led to the incident has been undone to restore normal functionality. **Action Plan** We have corrected the command error and will implement a monitoring system to detect similar issues in the future. Additionally, we will update our deployment process to prevent rollbacks and establish guidelines to ensure critical systems are used appropriately.

**Root Cause** The issue was caused by a high volume of data that led to a slowdown in our system's performance. **Resolution** The problem was resolved by restarting the system, which restored normal functionality. **Action Plan** We have implemented measures to quickly address similar issues in the future. Alerts will be refined for better detection, and development efforts are underway to enhance system performance and functionality.