Beefree SDK unavailable (502 or 504 errors)

Update timeline

1. investigating Oct 14, 2026, 12:24 PM UTC

   We’re currently experiencing an issue that makes the Beefree SDK unreachable. Requests to the SDK return a 502 Bad Gateway or 504 Gateway Timeout error, so the editor may fail to load or respond. Our team is already investigating the cause and working to restore service as quickly as possible. We’ll post an update here as soon as we have more information.

2. resolved Oct 14, 2026, 12:30 PM UTC

   The issue has been resolved. The 502 and 504 errors were caused by a configuration issue in our gateway, which our team has fixed. The Beefree SDK is back online, and all services are operating normally.

3. postmortem Oct 14, 2026, 03:09 PM UTC

   # **Summary** On Oct 14, between 12:12 UTC and 12:26 UTC, a service disruption impacted the availability of the Editor component for end-users. The incident occurred during a planned configuration update aimed at decommissioning a legacy service. During execution via Terraform, an undetected state drift between the remote infrastructure and the defined configuration caused an unexpected destruction and recreation of the API Gateway, followed by a listener misconfiguration on the Application Load Balancer \(ALB\). Total service degradation lasted approximately 14 minutes. Full functionality was restored at 12:26 UTC following targeted manual remediation. # **Status** **Solved** # **Timeline \(UTC\)** Oct 14, 2026 * 12:10 UTC – A routine infrastructure update was initiated via Terraform to remove resources associated with a decommissioned service. * 12:12 UTC – Upon execution, Terraform identified an implicit dependency drift, triggering an unintended `destroy-and-recreate` cycle for the API Gateway instance. * 12:18 UTC – The API Gateway replacement was provisioned. However, traffic routing failed due to a mismatched configuration state on the Application Load Balancer \(ALB\) listener bindings. * 12:20 UTC – Engineering teams identified the routing anomaly on the ALB listener and initiated corrective configuration procedures. * 12:26 UTC – Listener configuration was fully restored and verified. Traffic flow to the EVC cluster normalized, resolving customer impact. **Incident window**: ~14m, all EVC services not reachable # **Customer impact** Duration: 14 minutes \(12:12 UTC – 12:26 UTC\) Affected Components: EVC Cluster underlying services, Editor interface. User Experience: End-users attempting to access or perform operations within the Editor experienced temporary service unavailability and request timeouts. No data loss or corruption occurred during this period. # **Root Cause** The root cause was an unexpected dynamic resource replacement triggered by Terraform execution, resulting from a latent configuration drift between the active cloud resources and the declarative state file. During the removal of legacy service declarations, an implicit resource attribute dependency within the Terraform provider schema evaluated the update as a breaking change requiring resource replacement rather than an in-place update. Because the environment had experienced minor manual or unindexed attribute shifts over time \(configuration drift\), the plan preview did not clearly highlight the cascading effect on the API Gateway's dependent ALB listener configuration. Consequently, when the gateway was recreated, the associated ALB listener bindings failed to map correctly to the target group, cutting off ingress traffic to the EVC cluster. # **Lessons Learned & Action Items** To prevent similar issues during future infrastructure changes, the following systemic improvements will be implemented: 1. Enhanced Pre-Apply Plan Inspection & Safety Gates: * Implement automated CI/CD checks to block Terraform execution if a plan contains destructive actions \(`destroy`\) on critical network infrastructure \(gateways, ALBs\) without explicit secondary authorization. 2. Drift Detection & Reconciliation Automation: * Introduce periodic scheduled drift-detection pipelines \(e.g., automated `terraform plan` audits\) to identify and reconcile discrepancies between real-world infrastructure and state files prior to active maintenance windows. 3. Decoupling Dependency Graphs: * Refactor infrastructure-as-code modules to isolate API Gateway definitions from underlying ALB target group lifecycle management, ensuring gateway updates cannot implicitly impact ingress routing listeners.