Zuper incident

Workflow Builder API errors in us-east-1

Minor Resolved View vendor source →

Zuper experienced a minor incident on September 16, 2026, lasting –. The incident has been resolved; the full update timeline is below.

Started
Sep 16, 2026, 01:50 AM UTC
Resolved
Sep 16, 2026, 01:50 AM UTC
Duration
–
Detected by Pingoru
Sep 16, 2026, 01:50 AM UTC

Update timeline

  1. resolved Sep 16, 2026, 01:50 AM UTC

    Status: Resolved SUMMARY On September 16, 2026, between 01:50 UTC and 03:50 UTC (7:20 AM – 9:20 AM IST), customers in the us-east-1 region experienced intermittent 502 errors while using Workflow Builder. Timeline September 16, 2026, 01:50 AM UTC | 7:20 AM IST We are aware of an issue affecting Workflow Builder in the us-east-1 region, where API requests are intermittently returning 502 errors. Our engineering team is investigating and will provide an update shortly. Thank you for your patience. September 16, 2026, 02:15 AM UTC | 7:45 AM IST This has been classified as an incident. Our team has identified a sudden increase in MySQL database connections in the us-east-1 region, exceeding the configured maximum connection limit, which is causing the intermittent 502 errors. We are working on a fix. September 16, 2026, 03:00 AM UTC | 8:30 AM IST The maximum connection limit on the MySQL database has been increased to accommodate the higher connection demand. We are monitoring database connections and Workflow Builder API performance to confirm recovery. September 16, 2026, 03:30 AM UTC | 9:00 AM IST The Engineering team identified the root cause and implemented a hotfix to address the issue September 16, 2026, 03:50 AM UTC | 9:20 AM IST We are pleased to confirm that Workflow Builder API performance has returned to normal and intermittent 502 errors have been resolved. Our team will continue monitoring to ensure service stability. Root Cause Analysis An event trigger configured in a table within the Workflow Builder database encountered an edge case that caused a table lock. Subsequent queries were left in a waiting state, leading to database connection thread exhaustion. This resulted in database connection failures and intermittent 502 errors on Workflow Builder API requests. Resolution The Engineering team identified the root cause and implemented a hotfix to address the issue. Additionally, the maximum connection limit on the MySQL database was increased to accommodate the elevated connection demand and help prevent similar issues. Remediation Items Review and adjust the MySQL maximum connection limit to better accommodate peak request volume in us-east-1. Set up proactive alerting on MySQL active connection count to notify the team before connections approach exhaustion.