Teem experienced a critical incident on April 14, 2020 affecting Web Interface and Mobile Data and 1 more component, lasting 11h 48m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 14, 2020, 06:30 AM UTC
We are currently investigating this issue.
- monitoring Apr 14, 2020, 09:41 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 14, 2020, 06:18 PM UTC
This incident has been resolved. Teem has been monitoring performance for several hours and will now close this incident. At approximately 10:45 MDT internal systems began detecting an anomaly affecting the API. Triage began and the issue escalated over the next hour. Customers may have experienced intermittent access to the platform (devices, mobile and plugins) until 2:30 MDT when a temporary patch was deployed. Work continued to correct the root cause (a caching system that became unavailable cascaded to the web server pools causing them to drop offline, until the system was restored/rebuilt fully early this morning) through the monitoring period. All systems have been operational since approximately 2:30 MDT and full remediation work has been completed. Additional information on this incident will be provided at the completion of Teem's internal review process.