Teem incident
Multiple service disruptions across Teem services
Teem experienced a critical incident on November 22, 2019 affecting Web Interface and EventBoard, lasting 3d 21h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 22, 2019, 05:49 PM UTC
We are receiving reports that several services are failing to load, or taking longer than normal. These services include but are not limited to: app.teem.com, activating devices, and LobbyConnect. We're currently investigating the cause. Next update will be at Noon MT/2pm ET
- investigating Nov 22, 2019, 06:54 PM UTC
We are continuing to investigate the root cause of the disruptions. We will update again at the latest by 2PM MT/4PM ET.
- identified Nov 22, 2019, 08:49 PM UTC
We believe we have identified the root cause. We are implementing a hot fix, and will release it in the next 1-2 hours. Next update will be by 4pm MT/6pm ET
- monitoring Nov 22, 2019, 10:23 PM UTC
The hot fix has been implemented, and we are currently monitoring the results. Next update: 5pm MT/7pm ET
- monitoring Nov 22, 2019, 11:47 PM UTC
We are seeing improvements in performance, and are continuing to monitor. Next update: 7pm MT/9pm ET
- identified Nov 23, 2019, 01:11 AM UTC
We are seeing improved load times on app.teem.com and devices are able to activate intermittently. We have identified a hardware related issue that appears to be contributing to the remaining latency. A maintenance window has been scheduled for Sunday, 8pm-12am MT, as shown on the status page, to address this. Next update will be Monday by 12:30am MT/2:30am ET
- monitoring Nov 25, 2019, 06:53 AM UTC
The maintenance has been completed, and all systems are normal. We will continue monitoring through the day. Next update: Monday by 5pm MT/7pm ET
- resolved Nov 26, 2019, 03:27 PM UTC
This incident has been resolved. All systems continue to work as expected.