Pipefy Outage History

**Root Cause** Recently, efforts were made to separate the infrastructure for better handling of increased demand. This led to the failure of previously set exception rules. An update was applied to a specific host or domain without verifying its use in the integration, causing issues. **Resolution** The issue was resolved by modifying the configuration rules on Ipass app. **Action Plan** - An anomaly alert will be created for the Ipass to avoid this scenario happend again in the future

**Root Cause** The issue was caused by a failure in the integration feature, which led to an overload in the system. An unoptimized job created thousands of subflows, quickly filling up memory and affecting system availability. The misuse of the integration was identified as the root cause, prompting the disabling of the problematic flow as an initial step to mitigate the issue. **Resolution** To resolve the issue, the team deactivated the problematic flow and partially cleared the queues. This initial solution aimed to prevent new calls and address the items stuck in the queue. **Action Plan** **We will continue to monitor the system closely to ensure stability and prevent future issues. Our focus remains on continuously improving the customer experience by optimizing our processes and enhancing communication between teams.**

**Root Cause** The issue was caused by a failure in the integration feature, which led to an overload in the system. An unoptimized job created thousands of subflows, quickly filling up memory and affecting system availability. The misuse of the integration was identified as the root cause, prompting the disabling of the problematic flow as an initial step to mitigate the issue. **Resolution** To resolve the issue, the team deactivated the problematic flow and partially cleared the queues. This initial solution aimed to prevent new calls and address the items stuck in the queue. **Action Plan** **We will continue to monitor the system closely to ensure stability and prevent future issues. Our focus remains on continuously improving the customer experience by optimizing our processes and enhancing communication between teams.**

**Summary** On March 4, 2026, an incident occurred affecting multiple services in Oracle Cloud Infrastructure \(OCI\). Pipefy identified the disruption at 13:20 UTC, which impacted connectivity and service management operations for resources. As the Pipefy infrastructure relies on OCI, which affects the availability and normal behavior of certain platform functionalities for some customers. Oracle engineers investigated the issue, identified its source, and implemented mitigation actions. Full service functionality was confirmed as restored on March 4, 2026 at 19:40 UTC. Although Pipefy was back to normal at 19:40 UTC, we chose to wait for the provider to resolve the incident, which occurred on March 5, 2026 04:00am UTC . **Impact** During the incident window, customers have experienced intermittent latency or connection failures due to an underlying issue within the infrastructure. Because certain Pipefy components rely on OCI networking, services were temporarily degraded. We would like to emphasize that there was no loss or exposure of data. Full functionality was restored once the infrastructure issues were resolved. **Root Cause** The root cause of the incident was identified as an important and massive incident on Oracle's infrastructure that impacted the Pipefy platform. The disruption was caused by an issue affecting multiple Oracle Cloud Infrastructure \(OCI\) services, particularly involving components related to the Virtual Cloud Network \(VCN\). This issue affected the ability of services and resources to communicate reliably. The incident was triggered by OCI routine maintenance, the issue occurred because the Virtual Cloud Network Control Plane \(VCNCP\) was unable to propagate network configuration to the network dataplane services due to performance degradation of VCNCP’s internal database. At Pipefy, we identified the disruption at 13:20 UTC, this resulted in an inability to perform lifecycle actions, such as provisioning or scaling compute resources. Following that, OCI service teams were informed and the escalation process started. Oracle engineering teams identified the source of the disruption and implemented corrections to stabilize the affected infrastructure and restore normal connectivity and service operations for Pipefy on March 04 at 16:18 UTC, which resulted in full control plane recovery at OCI on March 05 at 04:40 UTC **Preventive Measures Performed** * To prevent this scenario from happening again, OCI implemented rigorous improvements to its infrastructure, such as migrating VCNCP services to hosts with greater memory capacity. \[Done\] * It also implemented performance optimizations in the VCNCP internal database, in addition to improvements in service monitoring. \[Done\] Next steps * Improvements in the monitoring and observability process of the Pipefy platform. **Additional informations** We would like to point out that all information regarding incidents will be available on our support tool, but we reiterate that official communication is done through our status page [https://status.pipefy.com](https://status.pipefy.com). We will continue to monitor the system closely to ensure stability and prevent future issues, while continuously working to enhance the customer experience.

**Root Cause** The incident was caused by a cloud provider's routine maintenance which led to a problem with the network control system. This issue prevented the network from updating its configuration due to a slowdown in the internal database. Despite this, the Pipefy platform's performance remained unaffected. **Resolution** The incident was resolved after a series of actions were taken by a cloud provider’s team, which closed the issue.

**Root Cause** The issue was caused by exceeding the execution limit in integration queues, leading to a loop in automation that triggered numerous webhooks. This resulted in an overload of a specific flow, creating a large processing queue. Consequently, clients experienced delays in their process executions. **Resolution** To resolve the issue, the customer service account was deactivated, which helped alleviate the problem. **Action Plan** To prevent future occurrences, a rate limit will be activated on the system, and the timeout for execution flows will be reduced.

**Root Cause** We encountered a significant delay due to a high volume of card deletions, which affected the processing queues. **Resolution** The issue with the queues resolved itself without any intervention from the team. The system returned to normal operations on its own with the team supervision. **Action Plan**

**Summary:** **The issue at hand involved the inability to create cards via the pipe email in Pipefy. This was due to a rule on firewall provider that was inadvertently blocking incoming emails, which are essential for the automatic creation of cards.** **Impact:** **The impact of this issue was significant as it disrupted the workflow of users relying on email-triggered card creation. This could have led to delays in task management and project tracking, affecting overall productivity and efficiency for users who depend on this feature for seamless operations.** **Root Cause:** **The root cause of the problem was identified as a specific rule on Cloudflare that was blocking incoming emails. This blockage prevented the emails from reaching the system, thereby halting the automatic card creation process.** **Corrective Actions:** * **The team disabled the problematic rule on Cloudflare, which restored the normal flow of incoming emails and resumed the card creation process. - A ticket was opened with SendGrid to request a range of IP addresses to ensure smoother operations in the future. - An alert system is to be created to monitor the incoming emails from the SendGrid endpoint to prevent similar issues.** **Next Steps/Conclusion:** **We will continue to monitor the system closely to ensure stability and prevent future occurrences of similar issues. Our ongoing efforts are focused on enhancing the customer experience by ensuring that all features function seamlessly and efficiently.**