Leap Event Technology Outage History

This incident has been resolved.

Activations platform (APIs, Public Web Activations, Admin CMS) suffered from degraded availability between 8:08 - 8:52 AM EDT related to an internal DNS issue with our services. The availability issue was resolved, and investigation for complete root cause and additional mitigation is under way.

On September 16, Leap Ticketing experienced a service disruption for 11 minutes caused by a production configuration change related to our search infrastructure. The change resulted in an unexpected increase in database connections, which exhausted the available connection capacity and temporarily impacted application availability. Once the issue was identified, our engineering team disabled the affected configuration and restored normal service. We are reviewing the incident and will implement appropriate safeguards around connection limits, capacity monitoring, and production changes to prevent a similar issue from occurring in the future. We apologize for the disruption and appreciate your patience while our team worked to restore service.

This incident has been resolved.

Activations experienced a brief outage from 9:09 AM EDT to 9:36 AM EDT. This was caused by a network configuration mismatch during a security patch. Recovery was delayed due to a secondary technical block in our deployment pipeline. All systems are now stable

The Mobile CMS experienced brief outages at approximately 10:21 AM EST and 10:29 AM EST, each lasting a few seconds. The issue was due to a performance bottleneck in our user-state data model for ticket-based audiences. During these windows, CMS users may have encountered slowness or temporary downtime. We have identified and planned improvements to help prevent future occurrences.

The SSL certificate for several activation domains was removed from CloudFlare without notice and before its expiration date. We are working with CloudFlare support to determine the reason why the certificate was removed. This issue has been resolved.

An outage with our cloud delivery network (CDN) provider caused clients and customers to not be able to visit our pages. Issue was from 2:56AM EST to 3:12AM EST.

This incident has been resolved.

ShowClix had an 18 minute outage this morning. Between 11:24 and 11:42 AM EST engineering noted elevated error rates while processing credit card payments due to an underlying network issue.

As of 6:53 PM EDT last night we reported full recovery.

At 8:20 PM EDT replication stopped on a read-only replica database which caused 404 responses when accessing data created or updated after that time. Engineering attempted to restart the database at 10:05 PM. However the process did not restart on its own, leading to connection failures to the replica database for a period of around 6 minutes. At 10:11 PM DEVOPS pulled the affected replica out of rotation which resolved all issues. Practically the effects were that customers and clients experienced “not found” errors when viewing some data (viewing orders, event configurations, etc.) between 8:20PM and 10:05 PM and widespread 500 errors (which render as the broken ticket page) when accessing the application (purchasing tickets, logging in, accessing the admin, etc). between 10:05 and 10:11 PM. This affected all ShowClix, Ticketbooth and white label clients.