Teem incident
Degraded performance: app.teem.com and other services
Teem experienced a minor incident on January 6, 2020 affecting Web Interface and Mobile Data, lasting 3h 15m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 06, 2020, 07:45 PM UTC
We’re currently seeing loading issues with app.teem.com and related services such as activating devices and signing into the mobile app. Our team is working to identify the root cause and implement a solution. Next update will be by 3:00pm MT/6:00pm ET
- monitoring Jan 06, 2020, 08:52 PM UTC
A fix has been implemented, and all services are working as normal. We will monitor through the rest of the day. Next update will be by 5:00pm MT/7:00pm ET
- resolved Jan 06, 2020, 11:01 PM UTC
Performance continues to be normal. This incident is now resolved.
- postmortem Jan 28, 2020, 09:25 PM UTC
On 6 January, as usage of Teem services increased for the day, there was an increase of duplication errors on our database. This increase caused a delay in replication, which in turn caused a greater number of duplication errors. This negative feed forward mechanism caused a degraded site performance. Reduced load on the database by scaling down our service instances allowed the replication to catch up, and remain health. Going forward, there will be database maintenance on the specific tables that exhibit these issues to guard against this in the future.