Teem experienced a critical incident on February 19, 2020 affecting Web Interface and Mobile Data and 1 more component, lasting 8h 42m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 19, 2020, 08:44 PM UTC
We are currently investigating this issue.
- monitoring Feb 19, 2020, 08:52 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 20, 2020, 05:26 AM UTC
After a period of system monitoring to ensure proper function Teem has resolved this incident. At approximately 1310 MT a disk storage limit failure occurred on the backup system for the primary database systems. Once identified additional capacity was added and and affected components restarted returning systems to full service by 1345 MT. Additional monitoring and alerting for the offending system has been added to help remediate such failures moving forward.