Teem experienced a critical incident on April 18, 2024 affecting Web Interface, lasting 2h 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 18, 2024, 12:24 AM UTC
We are currently experiencing an issue with the Teem application resulting in a 404 when trying to access the site.
- monitoring Apr 18, 2024, 01:27 AM UTC
A fix has been implemented. We are moving into the Monitoring Phase for the next 30 minutes.
- resolved Apr 18, 2024, 02:57 AM UTC
As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase.
- postmortem May 09, 2024, 12:33 PM UTC
**Teem Detailed Root Cause Analysis | 4.05.2024** **404 Error Site Wide Outage** We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Description:** On April 5th, 2024, internal teams and customer support noticed issues happening with logging and being able to access sites. Reports of some Teem instances experiencing inability to login with a forward of a 404 error also were presented. This issue impacted a large scale of Teem Customers. This cause of this site outage was due to a migration to a new server that would allow easier rollbacks and data integrity, that unfortunately had a performance error and needed to be done early. Upon receiving notice of the site access issues, our dedicated team promptly took action to restart the migration process, ensuring its successful completion and resolving the issue. **Type of Event:** Site Access **Remediation:** Upon immediate notice of the login issues, our dedicated Teem team was able to restart the migration process to successful completion, resolving the issue. **Root Cause Analysis:** There was a production outage caused by a database that we were hosting on. We needed to then move the data base to our standby server and set up a new one. We had planned the migration for a few weeks later and were preparing for it but the server encountered errors before the planned maintenance happened forcing us to start the migration early. We started the migration server backed up the DB and re-established connection with services. **Preventative Action:** The Teem team was able to start a migration process and monitor it through successful completion and review with internal testing to ensure the issue was resolved for customers. This incident has been closed, but our team is dedicated to closely monitoring future updates as they are released to ensure the best customer experience.