Elevated GIT/Application Builder Errors

Update timeline

1. identified Mar 31, 2026, 12:55 PM UTC

   We're experiencing an elevated level of errors in our application builder infrastructure and are currently looking into the issue.

2. monitoring Apr 08, 2026, 07:51 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Apr 08, 2026, 07:51 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Apr 21, 2026, 04:30 PM UTC

   This incident has been resolved.

5. postmortem Apr 21, 2026, 05:06 PM UTC

   Starting around March 11, some cloud builds began failing intermittently with no such image errors. The failures were non-deterministic and affected all architectures. At peak, some users saw around 50% failure rates. We identified and fixed several bugs in the builder's image garbage collector that caused it to over-count freed disk space and run too aggressively, eventually deleting images that in-progress builds still needed. Fixes were deployed between March 19 and April 14, with build failure rates dropping to near-zero after the final deploy. We're continuing to monitor and working on additional safeguards to prevent the garbage collector from targeting images that active builds depend on.