ftrack incident
Degraded performance for clients in the Asia Pacific region
ftrack experienced a major incident on March 7, 2023 affecting Hosted applications, lasting 45m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 07, 2023, 10:57 PM UTC
Due to a configuration change in the Asia Pacific region, ftrack workspaces hosted in the region are experiencing a partial outage. We have identified the issue and are implementing a fix.
- monitoring Mar 07, 2023, 11:14 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 07, 2023, 11:15 PM UTC
This incident has been resolved.
- postmortem Mar 08, 2023, 08:39 AM UTC
**Root Cause:** During routine maintenance, a configuration change was made to the Kubernetes clusters in Google Cloud, which inadvertently caused an internal service to stop responding. This caused a partial outage and affected the ability of users to access certain features. **Resolution:** Our team immediately identified the issue and worked to resolve it as quickly as possible. The configuration change was rolled back, which restored the service. **Impact:** During the partial outage, users may have experienced difficulty accessing certain features. We apologize for any inconvenience this may have caused. **Next Steps:** We will continue working closely with our cloud service providers to ensure our services remain reliable and available to our users. We will also thoroughly review our change management policies and procedures to prevent similar incidents in the future.