Nango Outage History

Started Oct 02, 2026, 05:33 PM UTC· resolved Oct 06, 2026, 09:07 AM UTC

1. investigating Oct 02, 2026, 05:33 PM UTC

   Actions and syncs are failing to execute. We are investigating and will update here as soon as we have more information.

2. investigating Oct 02, 2026, 05:54 PM UTC

   We are still investigating. All nango tasks (actions, syncs, and webhooks) are currently not running.

3. investigating Oct 02, 2026, 05:58 PM UTC

   We are seeing some recovery in task processing.

4. investigating Oct 02, 2026, 06:00 PM UTC

   This issue has been mitigated. We are now investigating the root cause and will update here when complete.

5. resolved Oct 06, 2026, 09:07 AM UTC

   POSTMORTEM: A recent change caused the execution of all functions (actions, webhooks, syncs) to stop on Friday October 2nd between 17:22 and 17:54 UTC. We identified a performance regression introduced by a recent query refactor in our task scheduler. The change caused one database query to scale poorly when the number of fresh execution increased significantly. Under normal load, the query remained fast, which is why the issue was not immediately visible after deployment. 8 hours later, a spike in queued tasks triggered the inefficient execution path, increasing database work and causing the dequeuing query to time out. We have since corrected the query and are adding additional safeguards, including additional monitoring to reduce the likelihood of similar regressions in the future.

6. investigating Oct 06, 2026, 09:07 AM UTC

   POSTMORTEM: A recent change caused the execution of all functions (actions, webhooks, syncs) to stop on Friday October 2nd between 17:22 and 17:54 UTC. We identified a performance regression introduced by a recent query refactor in our task scheduler. The change caused one database query to scale poorly when the number of fresh execution increased significantly. Under normal load, the query remained fast, which is why the issue was not immediately visible after deployment. 8 hours later, a spike in queued tasks triggered the inefficient execution path, increasing database work and causing the dequeuing query to time out. We have since corrected the query and are adding additional safeguards, including additional monitoring to reduce the likelihood of similar regressions in the future.

Read the full incident report →

Started Sep 13, 2026, 11:00 PM UTC· resolved Sep 23, 2026, 07:48 PM UTC

1. investigating Sep 13, 2026, 11:00 PM UTC

   Some syncs may be failing to persist records. We are investigating the root cause.

2. investigating Sep 14, 2026, 02:35 AM UTC

   We have mitigated the issue and will continue observing the service behavior.

3. investigating Sep 14, 2026, 03:02 AM UTC

   This issue has been resolved

4. resolved Sep 23, 2026, 07:48 PM UTC

   Degraded record sync performance post-mortem Post-incident summary Date: 12–14 September 2026 Impact: Slower record syncs, with >1% failing on timeout and retrying. Most syncs completed successfully. Status: Resolved Summary Our records database experienced progressive performance degradation over a period of roughly two days. Customers experienced slower sync runs, and a subset (>1%) of syncs failed with timeout errors and had to retry. Most syncs continued to complete successfully throughout. Customers syncing large record volumes were significantly more likely to see failures, and some experienced repeated failures on specific syncs over a period of hours. Other parts of the system were not affected. No data was lost. Timeline Issue began: 12 September, 16:00 UTC Issue detected: 13 September, 23:05 UTC Mitigated: 14 September, 02:30 UTC Fixed: 17 September, 16:49 UTC Root cause A background job removes stored records for connections that have gone inactive. Because of a change made earlier this year, it was performing all of its work as a single, long-running database operation rather than in small batches. While it ran, the database could not perform its routine internal cleanup, which reclaims obsolete data and can only run once no active operation still depends on it. Obsolete rows accumulated, so every query had to read progressively more to return the same results. As read costs rose, write operations began exceeding their time limit and failing — and because failed syncs retry, the degradation compounded. Resolution We restored the batching behavior to the cleanup job, so it now processes a bounded amount of data at a time and releases the database between batches. The database's internal cleanup resumed immediately. Read performance returned to its normal baseline the same day and has remained stable since. Syncs that failed during the incident recovered automatically on their next run; no customer action was required. Next steps Monitoring: The condition behind this incident was visible on our health dashboards throughout, but we had no monitor configured to alert on it. We are closing this gap with new monitors.

5. resolved Sep 23, 2026, 07:48 PM UTC

   ## Degraded record sync performance post-mortem ### Post-incident summary **Date:** 12–14 September 2026 **Impact:** Slower record syncs, with >1% failing on timeout and retrying. Most syncs completed successfully. **Status:** Resolved #### Summary Our records database experienced progressive performance degradation over a period of roughly two days. Customers experienced slower sync runs, and a subset (>1%) of syncs failed with timeout errors and had to retry. Most syncs continued to complete successfully throughout. Customers syncing large record volumes were significantly more likely to see failures, and some experienced repeated failures on specific syncs over a period of hours. Other parts of the system were not affected. No data was lost. #### Timeline Issue began: 12 September, 16:00 UTC Issue detected: 13 September, 23:05 UTC Mitigated: 14 September, 02:30 UTC Fixed: 17 September, 16:49 UTC #### Root cause A background job removes stored records for connections that have gone inactive. Because of a change made earlier this year, it was performing all of its work as a single, long-running database operation rather than in small batches. While it ran, the database could not perform its routine internal cleanup, which reclaims obsolete data and can only run once no active operation still depends on it. Obsolete rows accumulated, so every query had to read progressively more to return the same results. As read costs rose, write operations began exceeding their time limit and failing — and because failed syncs retry, the degradation compounded. #### Resolution We restored the batching behavior to the cleanup job, so it now processes a bounded amount of data at a time and releases the database between batches. The database's internal cleanup resumed immediately. Read performance returned to its normal baseline the same day and has remained stable since. Syncs that failed during the incident recovered automatically on their next run; no customer action was required. #### Next steps * Monitoring: The condition behind this incident was visible on our health dashboards throughout, but we had no monitor configured to alert on it. We are closing this gap with new monitors.

Read the full incident report →

Started Aug 03, 2026, 12:00 AM UTC· resolved Aug 03, 2026, 01:26 AM UTC

1. investigating Aug 03, 2026, 12:00 AM UTC

   Nango logs are not being ingested

2. resolved Aug 03, 2026, 01:26 AM UTC

   2026-08-03 Missing logs Summary On August 3, 2026, Nango's log ingestion pipeline began rejecting writes because the index for that day had already transitioned to a read-only state. Our investigation indicates that several Connect UI events were received from a browser whose system clock was approximately six days ahead. We accepted the client-provided timestamp without validation and used it to select the storage index, causing the August 3 index to be created prematurely. After 48 hours, our lifecycle policy moved the index to the warm tier, shrank it, and made it read-only—before August 3 had arrived. Customer-facing function executions were not affected. We mitigated the incident by re-enabling writes to the affected index. Customer impact What: Nango logs could not be written while the affected index was read-only. Function executions and other customer workloads were not affected. When: August 3, 2026, from approximately 00:00 UTC to 01:26 UTC. Data integrity: All log events received during this period were lost. What happened Log indices are selected using the createdAt timestamp attached to each event. On July 28, a Connect UI session from a Windows/Chrome client emitted several events. The client’s clock appears to have been approximately six days ahead, so these events carried August 3 timestamps. The telemetry endpoint accepted the client-supplied timestamp and used it directly as createdAt. The first event therefore created the August 3 index on July 28. Our index lifecycle policy moves indices to the warm tier after 48 hours, where they are shrunk and made read-only. Around July 30, the policy applied this transition based on the index’s actual age. When normal August 3 traffic began writing to the same index, those writes were rejected because the index had already been shrunk and locked. The lifecycle policy behaved as configured. The underlying issue was that an unvalidated client timestamp could control index selection and create a future-dated index. Resolution We restored log ingestion by re-enabling writes to the shrunk index. What we’re doing to prevent recurrence Validate clock skew: We will normalize client-provided timestamps that fall unreasonably far in the past or future. We apologize for the disruption. If you have questions about how this affected you, please reach out.

3. investigating Aug 03, 2026, 01:26 AM UTC

   # 2026-08-03 Missing logs ## Summary On August 3, 2026, Nango's log ingestion pipeline began rejecting writes because the index for that day had already transitioned to a read-only state. Our investigation indicates that several Connect UI events were received from a browser whose system clock was approximately six days ahead. We accepted the client-provided timestamp without validation and used it to select the storage index, causing the August 3 index to be created prematurely. After 48 hours, our lifecycle policy moved the index to the warm tier, shrank it, and made it read-only—before August 3 had arrived. Customer-facing function executions were not affected. We mitigated the incident by re-enabling writes to the affected index. ## Customer impact - **What:** Nango logs could not be written while the affected index was read-only. Function executions and other customer workloads were not affected. - **When:** August 3, 2026, from approximately 00:00 UTC to 01:26 UTC. - **Data integrity:** All log events received during this period were lost. ## What happened Log indices are selected using the `createdAt` timestamp attached to each event. On July 28, a Connect UI session from a Windows/Chrome client emitted several events. The client’s clock appears to have been approximately six days ahead, so these events carried August 3 timestamps. The telemetry endpoint accepted the client-supplied timestamp and used it directly as `createdAt`. The first event therefore created the August 3 index on July 28. Our index lifecycle policy moves indices to the warm tier after 48 hours, where they are shrunk and made read-only. Around July 30, the policy applied this transition based on the index’s actual age. When normal August 3 traffic began writing to the same index, those writes were rejected because the index had already been shrunk and locked. The lifecycle policy behaved as configured. The underlying issue was that an unvalidated client timestamp could control index selection and create a future-dated index. ## Resolution We restored log ingestion by re-enabling writes to the shrunk index. ## What we’re doing to prevent recurrence - **Validate clock skew:** We will normalize client-provided timestamps that fall unreasonably far in the past or future. We apologize for the disruption. If you have questions about how this affected you, please reach out.

Read the full incident report →

Started Jul 16, 2026, 08:30 AM UTC· resolved Jul 16, 2026, 10:45 AM UTC

1. investigating Jul 16, 2026, 08:30 AM UTC

   SSO login is currently failing due to an outage at a third-party authentication provider. Existing sessions and email/password login are unaffected. We're monitoring and will update here with more information.

2. investigating Jul 16, 2026, 09:43 AM UTC

   We continue to see problems with SSO logins due to a third-party issue.

3. resolved Jul 16, 2026, 10:45 AM UTC

   Issues which were impacting SSO login have been addressed by the third-party integration. We have seen full recovery of the SSO login flow for the last 10–20 minutes.

Read the full incident report →

Started Jun 16, 2026, 08:42 PM UTC· resolved Jun 16, 2026, 09:21 PM UTC

1. investigating Jun 16, 2026, 08:42 PM UTC

   We are experiencing an issue executing functions on Nango (actions, syncs, and webhooks). We are currently investigating and will provide updates here.

2. resolved Jun 16, 2026, 09:21 PM UTC

   The issue was an internal telemetry ingestion delay. Function executions were not affected at all. No customers were impacted.

Read the full incident report →

Started Jun 16, 2026, 06:10 PM UTC· resolved Jun 16, 2026, 06:10 PM UTC

1. resolved Jun 16, 2026, 06:10 PM UTC

   2026-06-11 Incident post-mortem Summary On June 11, 2026, our jobs service (which runs syncs, actions, and webhooks) became progressively overloaded and then unavailable for a period of roughly five hours, with customer impact concentrated in the final ~80 minutes. During this window, actions and syncs were slow, some timed out (failed), and queued work built up before being processed. No data was lost, delayed tasks were retried and worked off once the service recovered. We have fully mitigated the incident and have a concrete plan to prevent recurrence. Customer impact What: Elevated latency on actions and syncs, some actions and syncs timed out, webhook and sync processing was delayed as a backlog formed. When: June 11, 2026, from approximately 21:20 UTC (gradual degradation) to 02:03 UTC on June 12 (mitigated). The heaviest impact was between ~00:40 and ~02:03 UTC. Data integrity: No data was lost. Delayed work was queued and processed once the service recovered. What happened Our processing service was sized with very little spare capacity relative to its normal load. At about 20:00 UTC a first instance stopped responding due to being overloaded. This triggered escalating crash-and-recover waves: when the service came back, each instance immediately picked up the full incoming workload plus the work that had queued during the restart. Because there was little headroom, the instances ran out of CPU faster than they could work through the backlog. Overloaded instances failed their health checks and were automatically restarted by our infrastructure, which returned them to the same overloaded starting point. This created a self-reinforcing loop: each restart left more work queued, which made the next start-up harder, until the service could no longer keep up. Resolution We broke the loop by significantly increasing the processing capacity available to the service (raising its scaling ceiling). With more instances sharing the load, each one had enough CPU headroom to pass its health checks, stay running, and work through the backlog. The service returned to normal latency and the queued work was fully processed. What we're doing to prevent recurrence Right-sizing the service: increasing its CPU allocation and lowering the threshold at which it scales out, so it always runs with meaningful headroom. We are reviewing our other services for the same pattern. Earlier detection: adding monitoring on the leading indicators (runtime/CPU health, unavailable instances, and scaling-at-maximum) so we catch this class of degradation well before it affects customers. We apologize for the disruption. If you have questions about how this affected you, please reach out.

2. investigating Jun 16, 2026, 06:10 PM UTC

   # 2026-06-11 Incident post-mortem ## Summary On **June 11, 2026**, our jobs service (which runs syncs, actions, and webhooks) became progressively overloaded and then unavailable for a period of roughly **five hours**, with customer impact concentrated in the final ~80 minutes. During this window, actions and syncs were slow, some timed out (failed), and queued work built up before being processed. No data was lost, delayed tasks were retried and worked off once the service recovered. We have fully mitigated the incident and have a concrete plan to prevent recurrence. ## Customer impact - **What:** Elevated latency on actions and syncs, some actions and syncs timed out, webhook and sync processing was delayed as a backlog formed. - **When:** June 11, 2026, from approximately **21:20 UTC** (gradual degradation) to **02:03 UTC** on June 12 (mitigated). The heaviest impact was between ~00:40 and ~02:03 UTC. - **Data integrity:** No data was lost. Delayed work was queued and processed once the service recovered. ## What happened Our processing service was sized with very little spare capacity relative to its normal load. At about 20:00 UTC a first instance stopped responding due to being overloaded. This triggered escalating crash-and-recover waves: when the service came back, each instance immediately picked up the full incoming workload plus the work that had queued during the restart. Because there was little headroom, the instances ran out of CPU faster than they could work through the backlog. Overloaded instances failed their health checks and were automatically restarted by our infrastructure, which returned them to the same overloaded starting point. This created a self-reinforcing loop: each restart left more work queued, which made the next start-up harder, until the service could no longer keep up. ## Resolution We broke the loop by significantly increasing the processing capacity available to the service (raising its scaling ceiling). With more instances sharing the load, each one had enough CPU headroom to pass its health checks, stay running, and work through the backlog. The service returned to normal latency and the queued work was fully processed. ## What we're doing to prevent recurrence - **Right-sizing the service**: increasing its CPU allocation and lowering the threshold at which it scales out, so it always runs with meaningful headroom. We are reviewing our other services for the same pattern. - **Earlier detection**: adding monitoring on the leading indicators (runtime/CPU health, unavailable instances, and scaling-at-maximum) so we catch this class of degradation well before it affects customers. We apologize for the disruption. If you have questions about how this affected you, please reach out.

Read the full incident report →

Started Jun 15, 2026, 08:25 PM UTC· resolved Jun 15, 2026, 09:28 PM UTC

1. investigating Jun 15, 2026, 08:25 PM UTC

   Nango Connect UI isn't closing when a connection is successfully created.

2. investigating Jun 15, 2026, 09:27 PM UTC

   We have identified the root cause of the issue affecting the Connect UI popup and have restored the feature. The popup now closes successfully once a connection is created. We'll continue to monitor closely and will publish a full post-mortem later this week.

3. resolved Jun 15, 2026, 09:28 PM UTC

   This issue has bee resolved.

Read the full incident report →

Started Jun 12, 2026, 12:40 AM UTC· resolved Jun 12, 2026, 02:04 AM UTC

1. investigating Jun 12, 2026, 12:40 AM UTC

   We are investigating a degradation on our action executions

2. investigating Jun 12, 2026, 01:10 AM UTC

   We have identified and mitigated the root cause of the issue, action and sync run should recover now

3. resolved Jun 12, 2026, 02:04 AM UTC

   Systems fully recovered

Read the full incident report →

Started Jun 10, 2026, 01:24 PM UTC· resolved Jun 15, 2026, 08:09 AM UTC

1. investigating Jun 10, 2026, 01:24 PM UTC

   We are currently experiencing degraded performance in Nango syncs. We are currently investigating and will update here.

2. investigating Jun 10, 2026, 01:32 PM UTC

   We have identified the issue and rolled out the fix. We will continue monitoring.

3. investigating Jun 10, 2026, 01:42 PM UTC

   We have confirmed the incident is now resolved.

4. resolved Jun 15, 2026, 08:09 AM UTC

   2026-06-11 Incident post-mortem Summary On June 11, 2026, part of our platform that stores the data produced by syncs began rejecting writes for a subset of syncs over a window of roughly 15 minutes. During this period, affected syncs were unable to save their fetched data and reported failures. The issue was caused by an internal change that was rolled out ahead of the component that still depended on the previous behavior. We have fully mitigated the incident and have a concrete plan to prevent recurrence. Customer impact What: Saving of synced data failed for a portion of syncs. Affected sync runs reported errors and did not store their data during the window. When: June 11, 2026, from approximately 13:13 UTC (incident start) to 13:30 UTC (mitigated), with full resolution shortly after. Data integrity: No data was lost. Once the fix was in place, affected syncs resumed saving data normally on their subsequent runs. What happened As part of ongoing maintenance, we were retiring something internal that was no longer needed. This was planned as a multi-step rollout: first update everything that relied on it, then remove it. During the rollout, one of those updates was missed. As a result, when the cleanup step was completed, one part of the system was still relying on what had just been removed. This caused the affected writes to fail, and the impacted syncs to error. Resolution We applied the missing update, after which the errors stopped immediately. Affected syncs recovered on their next run. What we're doing to prevent recurrence Ensuring changes reach every impacted component: improving our process so that a change in our codebase reliably triggers an update and rollout of all the components it affects, removing the chance of one being left running an older, incompatible version. We apologize for the disruption. If you have questions about how this affected you, please reach out.

5. investigating Jun 15, 2026, 08:09 AM UTC

   # 2026-06-11 Incident post-mortem ## Summary On June 11, 2026, part of our platform that stores the data produced by syncs began rejecting writes for a subset of syncs over a window of roughly 15 minutes. During this period, affected syncs were unable to save their fetched data and reported failures. The issue was caused by an internal change that was rolled out ahead of the component that still depended on the previous behavior. We have fully mitigated the incident and have a concrete plan to prevent recurrence. ## Customer impact - What: Saving of synced data failed for a portion of syncs. Affected sync runs reported errors and did not store their data during the window. - When: June 11, 2026, from approximately 13:13 UTC (incident start) to 13:30 UTC (mitigated), with full resolution shortly after. - Data integrity: No data was lost. Once the fix was in place, affected syncs resumed saving data normally on their subsequent runs. ## What happened As part of ongoing maintenance, we were retiring something internal that was no longer needed. This was planned as a multi-step rollout: first update everything that relied on it, then remove it. During the rollout, one of those updates was missed. As a result, when the cleanup step was completed, one part of the system was still relying on what had just been removed. This caused the affected writes to fail, and the impacted syncs to error. ## Resolution We applied the missing update, after which the errors stopped immediately. Affected syncs recovered on their next run. ## What we're doing to prevent recurrence - Ensuring changes reach every impacted component: improving our process so that a change in our codebase reliably triggers an update and rollout of all the components it affects, removing the chance of one being left running an older, incompatible version. We apologize for the disruption. If you have questions about how this affected you, please reach out.

Read the full incident report →

Started Apr 27, 2026, 12:16 PM UTC· resolved Apr 29, 2026, 08:25 AM UTC

1. investigating Apr 27, 2026, 12:16 PM UTC

   We are experiencing an increase in 502s on our public API. We believe we have found the cause and are preparing a release to resolve it.

2. investigating Apr 27, 2026, 12:16 PM UTC

   We are experiencing an increase in 502s on our public API. We believe we have found the cause and are preparing a release to resolve it.

3. investigating Apr 27, 2026, 02:59 PM UTC

   This has now been resolved.

4. resolved Apr 29, 2026, 08:25 AM UTC

   Post-Incident Summary Date: 29 April 2026 Summary A gradual increase in Records API response payload size eventually exceeded the memory available to the pods serving those requests, causing them to be terminated. As pods were terminated and replaced, the load balancer returned 502 errors for requests routed to them. The growth had been building over several days and was not caught by internal monitoring before a customer reported the issue. Timeline (UTC) Issue began: 24 April, ~12:00 Reported by customer: 25 April, ~00:00 Investigation started: 27 April, ~07:00 Mitigated: 27 April, 15:00 Resolved: 27 April, 15:00 Root Cause The Records API response payload size had been growing gradually for several days. Once the payloads crossed the available pod memory headroom, the pods serving those requests began being terminated for exceeding memory limits. As terminated pods were replaced, the load balancer returned 502 errors for requests routed to pods that were shutting down or starting up. The growth was gradual rather than sudden, which meant our internal monitoring did not flag it before user-visible errors occurred. Diagnosis was also slowed because the default per-pod memory view smoothed out the underlying spikes, masking the memory-pressure pattern until the smoothing was removed and pod termination state was inspected directly. Resolution Per-pod memory limits on the affected service were increased, restoring headroom for the larger payloads and stopping the terminations. Once the new limits were in place, the load balancer stopped returning 502s and the service returned to normal. Follow-Up Actions Detection A monitor now pages the on-call team when load-balancer 5XX rates rise, so future user-impacting issues are caught immediately rather than relying on customer reports. System safeguards Bound peak memory per Records API request so response sizes cannot pressure pod memory. Improve telemetry around per-request response size so unusual growth surfaces in monitoring before it impacts users.

5. investigating Apr 29, 2026, 08:25 AM UTC

   **Post-Incident Summary** Date: 29 April 2026 **Summary** A gradual increase in Records API response payload size eventually exceeded the memory available to the pods serving those requests, causing them to be terminated. As pods were terminated and replaced, the load balancer returned 502 errors for requests routed to them. The growth had been building over several days and was not caught by internal monitoring before a customer reported the issue. **Timeline (UTC)** * Issue began: 24 April, ~12:00 * Reported by customer: 25 April, ~00:00 * Investigation started: 27 April, ~07:00 * Mitigated: 27 April, 15:00 * Resolved: 27 April, 15:00 **Root Cause** The Records API response payload size had been growing gradually for several days. Once the payloads crossed the available pod memory headroom, the pods serving those requests began being terminated for exceeding memory limits. As terminated pods were replaced, the load balancer returned 502 errors for requests routed to pods that were shutting down or starting up. The growth was gradual rather than sudden, which meant our internal monitoring did not flag it before user-visible errors occurred. Diagnosis was also slowed because the default per-pod memory view smoothed out the underlying spikes, masking the memory-pressure pattern until the smoothing was removed and pod termination state was inspected directly. **Resolution** Per-pod memory limits on the affected service were increased, restoring headroom for the larger payloads and stopping the terminations. Once the new limits were in place, the load balancer stopped returning 502s and the service returned to normal. **Follow-Up Actions** Detection A monitor now pages the on-call team when load-balancer 5XX rates rise, so future user-impacting issues are caught immediately rather than relying on customer reports. System safeguards Bound peak memory per Records API request so response sizes cannot pressure pod memory. Improve telemetry around per-request response size so unusual growth surfaces in monitoring before it impacts users.

Read the full incident report →

Started Apr 12, 2026, 08:00 AM UTC· resolved Apr 14, 2026, 08:00 AM UTC

1. investigating Apr 12, 2026, 08:00 AM UTC

   Syncs are still delayed, while actions appear unaffected. The issue seems to be related to how the database is handling sync schedules. Once sync processing recovers, synced data will catch up automatically.

2. investigating Apr 12, 2026, 09:05 AM UTC

   We are still investigating the issues impacting syncs and actions, and working on early mitigation steps.

3. investigating Apr 12, 2026, 09:49 AM UTC

   We have deployed a first step to mitigate the issue but are not seeing recovery yet; we continue investigating how to speed up recovery.

4. investigating Apr 12, 2026, 10:50 AM UTC

   Syncs and actions remain impacted. We've identified a load-related issue in our infrastructure and are actively working on mitigation.

5. investigating Apr 12, 2026, 12:01 PM UTC

   We are seeing some partial recovery for actions but syncs remain fully impacted. We are working on reducing the load in one of our infrastructure components.

6. investigating Apr 12, 2026, 01:13 PM UTC

   We are still seeing partial recovery for actions and syncs but are still working towards a full recovery.

7. investigating Apr 12, 2026, 02:45 PM UTC

   We have not yet fully recovered as some underlying infrastructure issues caused by high load have not been fully addressed. We are working towards a solution to fully reduce the load.

8. investigating Apr 12, 2026, 03:37 PM UTC

   We have found an infrastructure issue that may have been preventing recovery and have already fixed it. We are seeing partial recovery on syncs and actions and continue moving traffic over to bring the system to full recovery.

9. investigating Apr 12, 2026, 03:55 PM UTC

   We are seeing full recovery over the last 10 minutes and will continue monitoring our metrics.

10. resolved Apr 14, 2026, 08:00 AM UTC

    Post-Incident Summary Date: 12 April 2026 Impact: Degraded sync execution and delayed actions and webhook processing Status: Resolved Summary A webhook flood originating from a single customer environment caused one of our databases to saturate, resulting in broad degradation of asynchronous job processing. Sync execution dropped to near zero, and a large portion of actions and webhook-driven work were delayed or unable to run. A secondary bug in the scheduling system amplified the incident and blocked two consecutive recovery attempts before a fix was deployed. Timeline (UTC) Issue began: 07:00 Detected by monitoring: 07:00 Status page updated: 07:00 Mitigated: 15:30 Resolved: 15:55 Root Cause A single customer environment generated a sustained webhook flood, well above the typical baseline. Each incoming webhook triggered a database query to check the current queue depth for that customer's group before deciding whether to admit a new task. Under flood conditions, this query saturated one of our databases' CPU, preventing other work — including syncs and actions from all customers — from being scheduled or processed. Once the per-group queue cap was reached, new work could no longer be enqueued, and the system remained effectively stalled. Recovery was complicated by a separate bug in the recurring schedule path. When the scheduler encountered a group that had already hit the queue cap, an error in the code caused the exception to be swallowed silently. As a result, affected schedules were never marked as processed and were repeatedly retried on each scheduler tick, adding further load to an already saturated database. This caused two consecutive recovery attempts to fail. Resolution A fix was deployed to correct the scheduling bug, ensuring that capped groups are handled correctly and schedules are properly advanced after each pass. Task execution times were shifted forward in bulk to drain pressure from the database, then restored in batches. Once the backlog cleared, the system returned to a healthy state and full processing resumed by 15:55 UTC. Follow-Up Actions System safeguards Improve the current per-enqueue queue-depth admission-control mechanism to reduce database load under flood conditions. Define a rate limiting and load shedding strategy for webhook ingestion to protect the platform when a single customer generates sustained enqueue pressure. Fix the scheduling bug to correctly handle capped groups without silent failures (completed).

11. resolved Apr 14, 2026, 08:00 AM UTC

    **Post-Incident Summary** Date: 12 April 2026 Impact: Degraded sync execution and delayed actions and webhook processing Status: Resolved **Summary** A webhook flood originating from a single customer environment caused one of our databases to saturate, resulting in broad degradation of asynchronous job processing. Sync execution dropped to near zero, and a large portion of actions and webhook-driven work were delayed or unable to run. A secondary bug in the scheduling system amplified the incident and blocked two consecutive recovery attempts before a fix was deployed. **Timeline (UTC)** * Issue began: 07:00 * Detected by monitoring: 07:00 * Status page updated: 07:00 * Mitigated: 15:30 * Resolved: 15:55 **Root Cause** A single customer environment generated a sustained webhook flood, well above the typical baseline. Each incoming webhook triggered a database query to check the current queue depth for that customer's group before deciding whether to admit a new task. Under flood conditions, this query saturated one of our databases' CPU, preventing other work — including syncs and actions from all customers — from being scheduled or processed. Once the per-group queue cap was reached, new work could no longer be enqueued, and the system remained effectively stalled. Recovery was complicated by a separate bug in the recurring schedule path. When the scheduler encountered a group that had already hit the queue cap, an error in the code caused the exception to be swallowed silently. As a result, affected schedules were never marked as processed and were repeatedly retried on each scheduler tick, adding further load to an already saturated database. This caused two consecutive recovery attempts to fail. **Resolution** * A fix was deployed to correct the scheduling bug, ensuring that capped groups are handled correctly and schedules are properly advanced after each pass. * Task execution times were shifted forward in bulk to drain pressure from the database, then restored in batches. * Once the backlog cleared, the system returned to a healthy state and full processing resumed by 15:55 UTC. **Follow-Up Actions** *System safeguards* * Improve the current per-enqueue queue-depth admission-control mechanism to reduce database load under flood conditions. * Define a rate limiting and load shedding strategy for webhook ingestion to protect the platform when a single customer generates sustained enqueue pressure. * Fix the scheduling bug to correctly handle capped groups without silent failures (completed).

Read the full incident report →

Started Mar 05, 2026, 03:05 PM UTC· resolved Mar 05, 2026, 04:15 PM UTC

1. investigating Mar 05, 2026, 03:05 PM UTC

   We are experiencing an issue executing functions on Nango (actions, syncs, and webhooks). We are currently investigating and will provide updates here.

2. investigating Mar 05, 2026, 04:13 PM UTC

   We're implementing a fix that will take effect shortly. Still investigating the root cause.

3. resolved Mar 05, 2026, 04:15 PM UTC

   Functions recovered. Post mortem: Post-Incident Summary Date: 6 March 2026 Impact: Degraded function execution for actions and webhooks Status: Resolved Summary A bug in the task scheduling system disabled per-environment concurrency limits, allowing a single tenant to generate an unbounded burst of invocations. At the same time, the tenant’s functions were significantly longer running than typical workloads (120–150 seconds), which caused execution environments to remain occupied for extended periods. As the burst of tasks exceeded the rate at which execution environments became available, a backlog formed in the asynchronous invocation queue. This backlog increased the age of queued events and introduced elevated action latency. Eventually, queued tasks began exceeding the expiration limits enforced by the task system and expired before execution. Monitoring detected the issue through elevated latency metrics, after which the workload was identified and mitigated. Timeline (CET) Issue began: 16:00 Detected by monitoring / on-call paged: 16:20 Mitigated: 17:15 Root Cause A bug in the task system disabled per-environment concurrency limits, allowing a single tenant environment to generate an unbounded burst of invocations. The tenant’s functions were also significantly longer running than typical workloads (120–150 seconds), which meant execution environments remained occupied for extended periods and did not recycle quickly. Because provisioned concurrency was configured with a low maximum, most of the burst traffic was handled by on-demand capacity. While the function runtime continued scaling additional execution environments, the combination of burst traffic and long-running executions caused capacity to ramp more slowly than the incoming workload required. This created a backlog in the asynchronous invocation queue, which increased async event age and action latency. As the backlog grew, queued tasks eventually exceeded the expiration limits enforced by the task system and expired before they could be executed. Detection was delayed because alerting relies on latency averaged over 15-minute windows. Resolution The source tenant generating the burst workload was identified and incoming traffic was halted. Task system throttling logic was fixed to restore per-environment concurrency limits. After the workload was stopped and backlog drained, function processing returned to normal. All systems were fully operational by 17:15 CET. Follow-Up Actions System safeguards Fix the task system throttling bug to ensure per-environment concurrency limits are always enforced (completed). Capacity & scaling Increase provisioned concurrency autoscaling limits to better absorb bursts and reduce cold-start spillover. Monitoring & alerting Add alerts for AsyncEventAge to detect queue backlogs earlier. Add alerts for ProvisionedConcurrencySpilloverInvocations. Add anomaly detection on actions executed per minute / throughput drops.

4. resolved Mar 05, 2026, 04:15 PM UTC

   Functions recovered. Post mortem: # Post-Incident Summary **Date:** 6 March 2026 **Impact:** Degraded function execution for actions and webhooks **Status:** Resolved --- ## Summary A bug in the task scheduling system disabled per-environment concurrency limits, allowing a single tenant to generate an unbounded burst of invocations. At the same time, the tenant’s functions were significantly longer running than typical workloads (120–150 seconds), which caused execution environments to remain occupied for extended periods. As the burst of tasks exceeded the rate at which execution environments became available, a backlog formed in the asynchronous invocation queue. This backlog increased the age of queued events and introduced elevated action latency. Eventually, queued tasks began exceeding the expiration limits enforced by the task system and expired before execution. Monitoring detected the issue through elevated latency metrics, after which the workload was identified and mitigated. --- ## Timeline (CET) - **Issue began:** 16:00 - **Detected by monitoring / on-call paged:** 16:20 - **Mitigated:** 17:15 --- ## Root Cause A bug in the task system disabled per-environment concurrency limits, allowing a single tenant environment to generate an unbounded burst of invocations. The tenant’s functions were also significantly longer running than typical workloads (120–150 seconds), which meant execution environments remained occupied for extended periods and did not recycle quickly. Because provisioned concurrency was configured with a low maximum, most of the burst traffic was handled by on-demand capacity. While the function runtime continued scaling additional execution environments, the combination of burst traffic and long-running executions caused capacity to ramp more slowly than the incoming workload required. This created a backlog in the asynchronous invocation queue, which increased async event age and action latency. As the backlog grew, queued tasks eventually exceeded the expiration limits enforced by the task system and expired before they could be executed. Detection was delayed because alerting relies on latency averaged over 15-minute windows. --- ## Resolution - The source tenant generating the burst workload was identified and incoming traffic was halted. - Task system throttling logic was fixed to restore per-environment concurrency limits. - After the workload was stopped and backlog drained, function processing returned to normal. All systems were fully operational by **17:15 CET**. --- ## Follow-Up Actions ### System safeguards - Fix the task system throttling bug to ensure per-environment concurrency limits are always enforced (**completed**). ### Capacity & scaling - Increase provisioned concurrency autoscaling limits to better absorb bursts and reduce cold-start spillover. ### Monitoring & alerting - Add alerts for **AsyncEventAge** to detect queue backlogs earlier. - Add alerts for **ProvisionedConcurrencySpilloverInvocations**. - Add anomaly detection on **actions executed per minute / throughput drops**.

Read the full incident report →

Started Dec 05, 2025, 09:10 AM UTC· resolved Dec 05, 2025, 09:16 AM UTC

1. investigating Dec 05, 2025, 09:10 AM UTC

   We're seeing elevated error rates for function executions, both actions & syncs. Investigating.

2. investigating Dec 05, 2025, 09:14 AM UTC

   We suspect errors are due to external APIs being impacted by the Cloudflare outage, in particular token refreshing. Still investigating.

3. resolved Dec 05, 2025, 09:16 AM UTC

   Metrics have recovered. Likely due to the Cloudflare incident mitigation.

Read the full incident report →

Started Oct 12, 2025, 08:53 AM UTC· resolved Oct 12, 2025, 09:42 AM UTC

1. investigating Oct 12, 2025, 08:53 AM UTC

   The Nango syncs are delayed as we catchup with the processing that was missed during the maintenance period. Please be patient, we will provide updates here.

2. resolved Oct 12, 2025, 09:42 AM UTC

   All syncs have caught up.

Read the full incident report →

Started Oct 06, 2025, 05:22 PM UTC· resolved Oct 07, 2025, 09:02 AM UTC

1. investigating Oct 06, 2025, 05:22 PM UTC

   Nango is experiencing problems and we are investigating.

2. investigating Oct 06, 2025, 06:19 PM UTC

   We have a lead on the route cause and working on restoring services asap.

3. investigating Oct 06, 2025, 06:35 PM UTC

   It looks like we are back but still monitoring the situation and will provide further updates.

4. investigating Oct 06, 2025, 07:13 PM UTC

   Nango services are up and running again. We will follow up with a postmortem here.

5. resolved Oct 07, 2025, 09:02 AM UTC

   Post-Incident Summary — Database Lock Contention and Service Outage Date: 6 October 2025 Impact: Temporary outage of Nango services Status: Resolved Summary A sustained, high-volume workload from a downstream integration generated a large number of webhook events over an extended period. The follow-on fetches (getRecords) executed concurrently at scale, and a small update in that code path (persisting a “last fetched” timestamp) created heavy database contention. Lock saturation and connection exhaustion led to elevated latencies and 499 responses across public APIs until mitigations were applied. Timeline (CET) - Issue began: 19:21 - Detected by monitoring: 19:37 - Mitigated: 20:35 - Fully resolved: 23:17 Root Cause The getRecords flow includes a database write to persist a “last fetched” timestamp. Under extreme concurrency, these updates serialized and waited on one another, creating widespread lock contention. As locks piled up, available connections were exhausted, which introduced long response times and impacted all Nango services. Resolution - Logs and metrics showed elevated lock waits and deadlock messages (over 18,000 waiting operations at peak). - High-load webhook sources were temporarily disabled to shed database work, after which services recovered to normal. - The “last fetched” update in getRecords was determined to be unnecessary and removed to eliminate this contention pattern. All systems were fully operational by 20:35 CET. Follow-Up Actions - Rate limiting on webhook endpoints. - More controls to limit the blast-radius of isolated single-tenant spikes. - Enhanced observability for lock growth and connection-pool saturation to enable earlier, automated mitigation.

6. resolved Oct 07, 2025, 09:02 AM UTC

   **Post-Incident Summary — Database Lock Contention and Service Outage** **Date:** 6 October 2025 **Impact:** Temporary outage of Nango services **Status:** Resolved **Summary** A **sustained, high-volume workload** from a downstream integration generated a large number of webhook events over an extended period. The follow-on fetches (getRecords) executed concurrently at scale, and a small update in that code path (persisting a “last fetched” timestamp) created heavy database contention. Lock saturation and connection exhaustion led to elevated latencies and 499 responses across public APIs until mitigations were applied. **Timeline (CET)** - **Issue began:** 19:21 - **Detected by monitoring:** 19:37 - **Mitigated:** 20:35 - **Fully resolved:** 23:17 **Root Cause** The getRecords flow includes a database write to persist a “last fetched” timestamp. Under extreme concurrency, these updates serialized and waited on one another, creating widespread lock contention. As locks piled up, available connections were exhausted, which introduced long response times and impacted all Nango services. **Resolution** - Logs and metrics showed elevated lock waits and deadlock messages (over 18,000 waiting operations at peak). - High-load webhook sources were temporarily disabled to shed database work, after which services recovered to normal. - The “last fetched” update in getRecords was determined to be unnecessary and **removed** to eliminate this contention pattern. All systems were fully operational by **20:35 CET**. **Follow-Up Actions** - Rate limiting on webhook endpoints. - More controls to limit the blast-radius of isolated single-tenant spikes. - Enhanced observability for lock growth and connection-pool saturation to enable earlier, automated mitigation.

Read the full incident report →

Started Sep 23, 2025, 08:52 PM UTC· resolved Sep 24, 2025, 07:55 PM UTC

1. investigating Sep 23, 2025, 08:52 PM UTC

   Syncs are currently running slower than usual. We are investigating.

2. investigating Sep 23, 2025, 09:08 PM UTC

   Services are back to normal

3. investigating Sep 24, 2025, 09:09 AM UTC

   Syncs are periodically degraded. We are currently investigating.

4. resolved Sep 24, 2025, 07:55 PM UTC

   We have identified and fixed the root cause of this issue and syncs are now operating as normal.

Read the full incident report →

Started Sep 09, 2025, 04:44 PM UTC· resolved Sep 09, 2025, 04:51 PM UTC

1. investigating Sep 09, 2025, 04:44 PM UTC

   We are investigating a wave of actions and syncs failures.

2. resolved Sep 09, 2025, 04:51 PM UTC

   We have identified the root cause and deployed the fix. Operation back to normal.

Read the full incident report →