TicketNetwork experienced a major incident on December 8, 2023 affecting Data Platform / Reporting and Broker Software and 1 more component, lasting 1d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 08, 2023, 05:13 PM UTC
Our data processing infrastructure is having connection issues due to issues between AWS and our data center which are causing issues in our reporting and reporting tools. We have a critical ticket in with our data center and they are currently reviewing.
- identified Dec 08, 2023, 09:15 PM UTC
The issue has been identified and is being worked on by our data center with our tech leads being kept informed on their progress. TN has this escalated to their top support to resolve.
- monitoring Dec 08, 2023, 10:22 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Dec 09, 2023, 04:49 PM UTC
Our data center provider's fixes have not fully resolved the issues we have been experiencing, troubleshooting, and monitoring their fixes for. Status is downgraded due to field work we are doing today with another provider to stabilize systems until our data center has a complete resolution in place. TN leads are working with field techs as we speak to mitigate. Updates will be posted as soon as we get them.
- resolved Dec 09, 2023, 07:54 PM UTC
We have bypassed the trouble circuit and are currently routing around the issue. Replacement hardware has been ordered to restore redundancy, but at this point we have been fully online and stabilized through an alternate circuit since ~11AM. Our network team will continue to monitor all systems closely through the rest of the weekend and should have redundancy fully restored by next week.