Grid AI experienced a major incident on August 18, 2021 affecting Grid Service, lasting 10h 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 18, 2021, 04:38 PM UTC
We have identified that CUDA drivers are not available in GPU-enabled environments. We are actively investigating the source of this issue.
- investigating Aug 18, 2021, 06:39 PM UTC
We have identified that the issue happens with the mechanism that mounts drivers into Sessions and Experiments. We are investigating the root cause.
- identified Aug 18, 2021, 06:53 PM UTC
We have identified an issue with the underlying AMIs. We are working on a solution.
- identified Aug 18, 2021, 09:20 PM UTC
We have identified a solution and will be releasing a fix soon.
- monitoring Aug 19, 2021, 01:17 AM UTC
We made a patch release and the issue has been resolved on production. We'll be monitoring the situation and issue a full resolution tomorrow if no further incidents are identified.
- resolved Aug 19, 2021, 03:20 AM UTC
All systems have been updated and the incident is now considered resolved.