Code services are degraded, git clone is failing

Update timeline

1. investigating Jul 16, 2026, 06:59 AM UTC

   We are currently investigating this issue.

2. resolved Jul 16, 2026, 07:01 AM UTC

   This incident has been resolved.

3. postmortem Jul 22, 2026, 04:52 PM UTC

   # Summary Between June 19 and July 17, 2026, the built-in Git Clone step — and any pipeline step using the drone-git clone plugin — failed on ARM64 Kubernetes build infrastructure with the error exec /usr/local/bin/clone: exec format error. AMD64 \(Intel/AMD\) builds, Windows builds, and the VM containerless binary path were not affected. The root cause was a defect in our internal image publishing process that caused ARM64-tagged drone-git images to actually contain AMD64 binaries. We identified and mitigated the issue the same day it was reported by reverting the drone-git image to the last known-good version. No customer action or configuration change was required. # Root Cause On June 19, 2026, a security remediation restructured how the drone-git image is built. AMD64 builds were updated correctly, but the ARM64 build pipeline didn't build ARM64 files directly — it adapted the AMD64 build file via text substitution and compiled it on ARM64 infrastructure. The June 19 change altered the AMD64 file so that substitution silently no-op'd instead of failing, so the pipeline published an image tagged ARM64 whose Git Clone and Git LFS binaries were still compiled for AMD64. # Impact * Affected: The built-in Git Clone step, and any pipeline step using the drone-git clone plugin, running on ARM64 Kubernetes build infrastructure, across all accounts, between July 6 and July 17, 2026. * Symptom: Builds failed at the Git Clone step with exec /usr/local/bin/clone: exec format error. * Not affected: AMD64 \(Intel/AMD\) Kubernetes and VM builds, Windows builds, the VM containerless execution path, and our hardened image variant. # Mitigation We reverted the drone-git image version used across all affected services to the last known-good release. This fully resolved the ARM64 execution failures; no customer configuration changes were required. # Next Steps To prevent such issues from happening again. * Rebuild the ARM64 image publishing pipeline to build our dedicated ARM64 build files directly, rather than adapting the AMD64 build files. * Enhance automated post-publish validation to every image release: verify binary architecture matches the image tag, and run a functional smoke test before an image is considered releasable. * Expand automated test coverage to include ARM64 Kubernetes build scenarios. * Remove the affected intermediate image versions from circulation once the corrected release is validated.