SalesScreen experienced a major incident on March 13, 2024 affecting Web, lasting 58m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 13, 2024, 10:52 AM UTC
We are currently investigating an issue with connecting to TV screens
- monitoring Mar 13, 2024, 11:18 AM UTC
A fix has been implemented and we are monitoring the results. Users should be able to connect to tv.salesscreen.com without issues now.
- resolved Mar 13, 2024, 11:50 AM UTC
This incident has been resolved.
- postmortem Mar 13, 2024, 02:54 PM UTC
At 10:39 UTC on March 13th 2024 the SalesScreen Engineering team deployed an update to the network configuration of the application. While the update itself was successful, the new configuration resulted in some unintended side effects for an existing network rule used for the [tv.salesscreen.com](http://tv.salesscreen.com) domain. This resulted in users being redirected to the SalesScreen login page when trying to reload or connect screens using the [tv.salesscreen.com](http://tv.salesscreen.com) domain. The issue was immediately noticed by members of the Engineering team and a fix for the issue was quickly developed. Network configuration updates are by nature slow to propagate, so a full resolution of the issue was delayed until 11:18 UTC. Once the fix was fully deployed, users could once again connect to TV screens using the [tv.salesscreen.com](http://tv.salesscreen.com) domain. SalesScreen has robust procedures for the deployment of changes to our production environment, which was followed in this case. The network change had been tested successfully in our development environment, however this testing was limited to the added network rule. As a result we did not pick up on the issue with the TV endpoint during this testing. Going forward we will update our internal routines for making network configuration updates to include testing of all existing network rules.