Lokalise main application degraded performance

Update timeline

1. investigating Feb 05, 2025, 01:05 PM UTC

   We are seeing issues with import and export functionality of the Lokalise app. We are investigating the issue.

2. identified Feb 05, 2025, 01:18 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Feb 05, 2025, 01:28 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Feb 05, 2025, 02:44 PM UTC

   This incident has been resolved.

5. postmortem Feb 10, 2025, 08:19 AM UTC

   On February 5, 2025, we experienced a major service degradation affecting our Import functionality. The incident began at 11:00 UTC and during this time, customers were unable to perform imports in the UI or utilize Git integrations, which may have disrupted their workflows. Services have been fully restored by 13:26 UTC. The outage was caused by a misconfiguration during our migration to a new infrastructure system. To resolve the issue, our team rolled back to the previous system. In response to this incident, we are implementing several measures to prevent similar occurrences in the future: * We will enhance our deployment health checks to cover a previously missing set of dependencies, so that we can catch misconfiguration before newly deployed instances start to be used. * Additional monitoring and alerting systems will be established to quickly identify and address issues as they arise. * We are increasing coverage of our automated tests to ensure all the critical functionality is covered. We sincerely apologize for the inconvenience this incident caused to our customers. We understand how crucial these services are for your operations, and we appreciate your patience as we work to improve our systems and ensure reliable service in the future. Your trust is important to us, and we are committed to enhancing our processes to prevent such occurrences. Thank you for your understanding.