Swapcard Outage History

Status: Resolved This incident has been resolved. It lasted from 09:00 UTC to 09:30 UTC and is now resolved.

Status: Resolved We are providing a detailed post-mortem report regarding the service disruption that affected Swapcard customers on February 12th, 2026, from 09:00 UTC to 09:30 UTC. This issue was caused by an unprecedented traffic spike that exceeded the capacity of one of our internal services, leading to temporary degraded performance across badge printing \(SwapAccess\) and Studio access. The goal of this post-mortem is to share insights from our assessment and the steps taken to resolve the issue while providing transparency to our customers. Summary On February 12th, 2026, Swapcard experienced a 30-minute service disruption affecting badge printing via the Check-in App \(SwapAccess\) and access to the Studio interface. Customers with live events during this window experienced printing delays, degraded Studio access, and intermittent error messages. The incident was caused by an unusual and unprecedented volume of concurrent traffic hitting one of our internal services. This traffic pattern had not been observed before and exceeded the scaling thresholds configured at the time. The sudden load caused elevated response times that cascaded to downstream services, including badge generation and Studio access. The platform self-recovered as traffic levels normalized around 09:30 UTC. Timeline 09:00 UTC | An unprecedented spike in concurrent traffic began hitting one of our core internal services, exceeding previously observed traffic patterns. 09:00–09:15 UTC | The service could not scale fast enough to absorb the sudden load. Elevated latency cascaded to dependent services, causing badge generation timeouts and degraded Studio access. 09:15–09:30 UTC | Traffic levels began to normalize. The platform progressively recovered as request queues cleared. 09:30 UTC | Full service restoration. Badge printing and Studio access returned to normal operation. Onsite report | Our infrastructure team conducted a thorough investigation and immediately applied improvements to prevent recurrence. Root Cause Analysis The root cause of this incident was an unprecedented and sudden spike in concurrent traffic that exceeded the scaling capacity of one of our internal services. This traffic pattern had not been encountered before in production, and the service's auto-scaling configuration was not tuned to react quickly enough to absorb such a rapid increase. As response times on this service climbed, the impact cascaded to dependent features — including badge generation and Studio, which rely on it for real-time operations. This cascading effect amplified the user-facing impact beyond what the initial traffic surge alone would have caused. Remediation Following this incident, our infrastructure team immediately took action to strengthen the resilience of the affected services: Scaling improvements: We have significantly increased the resource capacity and improved the scaling configuration of the affected internal service to handle traffic spikes well beyond the levels observed during this incident. The service can now absorb sudden surges much more effectively. Resource optimization: We have optimized the resource usage of the service to ensure it operates more efficiently under load, reducing the likelihood of capacity issues even during unexpected traffic peaks. We have deployed additional monitoring and alerting specifically targeting the failure patterns observed during this incident. This ensures that if a similar traffic surge were to occur, our team would be automatically notified within seconds and could intervene proactively before any customer-facing impact. Conclusion This incident was caused by an unpredictable traffic pattern that had not been previously observed on our platform. While the disruption was brief, we understand the impact it had on customers running live events during that window, and we take that seriously. The scaling, optimization, and alerting improvements we have put in place significantly reduce the risk of a similar incident occurring in the future. Our infrastructure team continues to monitor the situation closely. If you have any questions or concerns regarding this incident, please don't hesitate to reach out to our support team.

Status: Resolved This incident has been resolved. Affected components Event App (Operational)

Status: Resolved We are providing a detailed post-mortem report regarding a production change that caused continuous emails to be sent again \(duplicates\) to some attendees. The impacted scope was limited to events created before February 4, 2025 that still had continuous emails enabled at the time of the incident. The goal of this post-mortem is to share what happened, why it happened, how it was resolved, and what we are doing to prevent a recurrence. Summary A background task runs every 30 minutes to process continuous email campaigns: • Retrieve all enabled continuous email campaigns • For each campaign, verify whether each targeted attendee already received the corresponding email • Send the email only to attendees who have not received it yet The incident was caused by the "already received" check relying on a database that only started being populated after February 4, 2025. For events created before that date, the delivery history existed in a different \(legacy\) datastore. As a result, the system incorrectly concluded that certain attendees had not received the email yet, and re-sent it. Timeline • 2026-02-04 07:56 UTC | A change was released that caused continuous emails to be incorrectly sent again \(duplicates\) to some attendees. • 2026-02-04 11:10 UTC | The change was reverted, stopping further duplicate sends. Resolution We resolved the incident by reverting the change introduced at 2026-02-04 07:56 UTC, restoring the previous behavior and preventing additional duplicate emails from being sent. Root Cause Analysis The root cause was an incorrect assumption in the delivery-history lookup logic: • The deduplication check queried a datastore that only contained delivery records populated starting 2025-02-04. • For events created before 2025-02-04, delivery records were stored in a legacy datastore. • Because the check did not account for the legacy source, it returned false negatives \("not sent yet"\), resulting in duplicate emails being sent on subsequent background-task executions. Immediate Actions Completed • Reverted the problematic change to stop duplicate email sends. • Identified and confirmed the gap between legacy and current delivery-history datastores for pre-2025-02-04 events. Short-Term Improvements • Update the "already received" check to consult the correct source for both legacy and current events. • Add safeguards to prevent re-sending when delivery history is missing/ambiguous \(fail-safe behavior\). • Add targeted tests covering pre-2025-02-04 events with continuous emails enabled. Long-Term Improvements • Consolidate delivery-history storage to a single source of truth \(including backfill/migration strategy for legacy data\). • Introduce idempotency guarantees at send-time \(e.g., campaign\+attendee idempotency keys\) to prevent duplicates even if checks regress. • Add monitoring and alerting on abnormal duplicate-send rates for continuous email campaigns. Conclusion We recognize the impact that duplicate emails can have on attendee experience and event organizer trust. We are implementing the preventative measures above to ensure legacy data paths are consistently handled and duplicates are prevented by design. If you have any questions or concerns regarding this incident, please reach out to our support team.