SPARC Media Hub experienced a critical incident on January 10, 2023 affecting Main Application, lasting 2h 57m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 10, 2023, 12:51 PM UTC
We are currently investigating a Platform outage, we will provide an update as soon as one becomes available. We apologize for the inconvience.
- monitoring Jan 10, 2023, 01:28 PM UTC
The team have implemented a fix for the platform outage. We will continue to monitor the situation and provide updates as they become available.
- resolved Jan 10, 2023, 03:44 PM UTC
This issue has been determined to be resolved.
- postmortem Jan 11, 2023, 07:21 PM UTC
On January 10th, 2023 at approximately 7:45 AM EST, SPARC Media Hub experienced a complete Platform outage which prevented customers from accessing the workflow system. At 7:46 AM EST, SPARC Media Hub deployed a team to immediately start investigating the issue. When customers would visit the platform, customers would see “No route to database”. Which prompted engineers to investigate the connection to our database servers. SPARC Media Hub utilizes an internal private network which technology that significantly improves performance between servers without sacrificing security. At 7:52 AM EST, the team identified the connection to the client application servers and the database servers were not communicating through the private network which caused the errors to appear. The team began troubleshooting steps such as restarting the servers and ensuring all software versions were up-to-date. Upon updating the private network software to the latest version, connection resumed and the platform server was restored at approximately 8:04 AM EST. The total downtown was 19 minutes.