Ticketmatic incident

Service problems due to high load

Minor Resolved View vendor source →

Ticketmatic experienced a minor incident on December 13, 2022 affecting Websales and Backoffice and 1 more component, lasting 2h 26m. The incident has been resolved; the full update timeline is below.

Started
Dec 13, 2022, 12:19 PM UTC
Resolved
Dec 13, 2022, 02:45 PM UTC
Duration
2h 26m
Detected by Pingoru
Dec 13, 2022, 12:19 PM UTC

Affected components

WebsalesBackofficeAPI

Update timeline

  1. investigating Dec 13, 2022, 12:19 PM UTC

    We are experiencing service problems due to very high load. We are investigating the issue.

  2. investigating Dec 13, 2022, 12:19 PM UTC

    We are continuing to investigate this issue.

  3. investigating Dec 13, 2022, 12:45 PM UTC

    We managed to bring the load on the system under control. We will now gradually increase the rate limiters to normal levels.

  4. monitoring Dec 13, 2022, 01:33 PM UTC

    The load is under control and we are monitoring the situation.

  5. resolved Dec 13, 2022, 02:45 PM UTC

    This incident has been resolved.

  6. postmortem Dec 14, 2022, 09:19 AM UTC

    Our apologies for the inconvenience that was caused by this disruption. We understand that this can have serious impact on your organisation and we take this very seriously. Due to an exceptional large peak as a consequence of 14 simultaneous peak sales at different Ticketmatic customers, an excessive load has occurred on the Ticketmatic systems. This overload has triggered safety measures which have in turn lead to Ticketmatic being temporarily unavailable or only after long response times. No data has been impacted. The Ticketmatic engineers have immediately taken action and have brought the situation back under control after 26 minutes. In order to avoid a similar scenario in the future, structural measures will be taken to improve defences against new occurrences of this scenario.