SPARC Media Hub incident

Platform Outage

Resolved after 2 hours 57 minutes, affecting Main Application, on January 10, 2023.

postmortem Jan 11, 2023, 07:21 PM UTC

On January 10th, 2023 at approximately 7:45 AM EST, SPARC Media Hub experienced a complete Platform outage which prevented customers from accessing the workflow system. At 7:46 AM EST, SPARC Media Hub deployed a team to immediately start investigating the issue. When customers would visit the platform, customers would see “No route to database”. Which prompted engineers to investigate the connection to our database servers. SPARC Media Hub utilizes an internal private network which technology that significantly improves performance between servers without sacrificing security. At 7:52 AM EST, the team identified the connection to the client application servers and the database servers were not communicating through the private network which caused the errors to appear. The team began troubleshooting steps such as restarting the servers and ensuring all software versions were up-to-date. Upon updating the private network software to the latest version, connection resumed and the platform server was restored at approximately 8:04 AM EST. The total downtown was 19 minutes.

Started
Jan 10, 2023, 12:46 PM UTC
Resolved
Jan 10, 2023, 03:44 PM UTC
Duration
2h 57m
Detected by Pingoru
Jan 10, 2023, 12:46 PM UTC

Affected components

Main Application

Update timeline

  1. investigating Jan 10, 2023, 12:51 PM UTC

    We are currently investigating a Platform outage, we will provide an update as soon as one becomes available. We apologize for the inconvience.

  2. monitoring Jan 10, 2023, 01:28 PM UTC

    The team have implemented a fix for the platform outage. We will continue to monitor the situation and provide updates as they become available.

  3. resolved Jan 10, 2023, 03:44 PM UTC

    This issue has been determined to be resolved.

  4. postmortem Jan 11, 2023, 07:21 PM UTC

    On January 10th, 2023 at approximately 7:45 AM EST, SPARC Media Hub experienced a complete Platform outage which prevented customers from accessing the workflow system. At 7:46 AM EST, SPARC Media Hub deployed a team to immediately start investigating the issue. When customers would visit the platform, customers would see “No route to database”. Which prompted engineers to investigate the connection to our database servers. SPARC Media Hub utilizes an internal private network which technology that significantly improves performance between servers without sacrificing security. At 7:52 AM EST, the team identified the connection to the client application servers and the database servers were not communicating through the private network which caused the errors to appear. The team began troubleshooting steps such as restarting the servers and ensuring all software versions were up-to-date. Upon updating the private network software to the latest version, connection resumed and the platform server was restored at approximately 8:04 AM EST. The total downtown was 19 minutes.

Your product runs on more than SPARC Media Hub

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →