Pipefy incident
default queue latency above 30s SLO
Resolved after 1 hour 14 minutes, on October 22, 2026.
We’ve recovered the default Sidekiq queue and it has drained. The overload was driven by repeated field-condition save mutations (`Resolvers::FieldConditions::Save`) on a single customer pipe (`307245277`, Copa Energia). Each condition created enqueued a `Cache::PipeInvalidatorWorker` job on the `default` queue, and for this pipe (≈25k cards) each job took ~80–110s, dominating the queue. As mitigation, we scaled up the Core workers around 5:53pm -03 (America/Buenos_Aires). By ~6:17pm -03 (America/Buenos_Aires), Grafana showed default queue latency dropping back to ~0 and the backlog clearing. The queue has drained, and high latency is no longer occurring. As aligned with the stakeholders, the incident is being closed.
Update timeline
- investigating Oct 22, 2026, 08:21 PM UTC
We are investigating the default queue latency above 30s.
- resolved Oct 22, 2026, 09:35 PM UTC
We’ve recovered the default Sidekiq queue and it has drained. The overload was driven by repeated field-condition save mutations (`Resolvers::FieldConditions::Save`) on a single customer pipe (`307245277`, Copa Energia). Each condition created enqueued a `Cache::PipeInvalidatorWorker` job on the `default` queue, and for this pipe (≈25k cards) each job took ~80–110s, dominating the queue. As mitigation, we scaled up the Core workers around 5:53pm -03 (America/Buenos_Aires). By ~6:17pm -03 (America/Buenos_Aires), Grafana showed default queue latency dropping back to ~0 and the backlog clearing. The queue has drained, and high latency is no longer occurring. As aligned with the stakeholders, the incident is being closed.
Your product runs on more than Pipefy
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services