Kobiton incident
Issues with Launching Devices at Atlanta Data Center
Kobiton experienced a minor incident on July 24, 2024 affecting Public Cloud Devices and Private Cloud Devices, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 24, 2024, 10:18 PM UTC
We are currently experiencing issues with launching some devices at our Atlanta Data Center. Our team is investigating the cause and working to resolve the issue as quickly as possible.
- investigating Jul 25, 2024, 01:33 AM UTC
We are continuing to investigate this issue.
- investigating Jul 25, 2024, 03:31 AM UTC
We found an Android device issue that might be related to this problem. We are in the process of validating the information.
- investigating Jul 25, 2024, 05:08 AM UTC
We are still in the process of validating the bug with android devices
- investigating Jul 25, 2024, 07:40 AM UTC
The issue is intermittent and we are actively looking into this issue further.
- investigating Jul 25, 2024, 08:51 AM UTC
We have found the root cause of this issue. Per our checking with one of the machines having the problem, its CPU reached 900% for several days. After restarting the Macmini, the CPU drops to ~70% and we can launch devices normally. We are planning to apply the fixes to all machines having this high CPU problem.
- investigating Jul 25, 2024, 07:58 PM UTC
We have restarted all the impacted Mac minis and the issue has been resolved.
- investigating Jul 25, 2024, 07:58 PM UTC
We are continuing to investigate this issue.
- resolved Jul 25, 2024, 10:47 PM UTC
This incident has been resolved.