We are noticing CI hosted build failures

Update timeline

1. investigating Apr 16, 2026, 10:06 AM UTC

   We are currently investigating this issue.

2. identified Apr 16, 2026, 10:14 AM UTC

   The issue has been identified and a fix is being implemented.

3. identified Apr 16, 2026, 11:08 AM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Apr 16, 2026, 11:41 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Apr 16, 2026, 12:02 PM UTC

   This incident has been resolved.

6. postmortem Apr 23, 2026, 03:34 PM UTC

   On April 16, 2026, Hosted CI Linux pipelines experienced intermittent initialization failures due to an upstream outage affecting package repositories used during environment setup. ### **Impact** A subset of customers running Hosted CI pipelines encountered failures during the initialization phase, preventing jobs from starting successfully. * Affected: 24 accounts * Failed executions: 129 pipelines * Impact duration: ~5 hours 49 minutes ### **Root Cause** The issue was caused by a service disruption in an external package repository provider. During CI environment provisioning, dependency installation requests to this upstream service timed out, causing initialization failures. ### **Remediation** **Immediate Mitigation** We updated runner configurations to bypass dependency installation during initialization and rolled out updated environments across affected clusters. **Permanent Fix** We have improved resilience in our CI infrastructure by: * Using pre-configured environments with required dependencies pre-installed * Eliminating runtime dependency on external package repositories during initialization * Enhancing failure handling for external dependency timeouts ### **Action Items / Next Steps** * Continue improving isolation from external dependencies during environment startup * Strengthen monitoring and alerting for upstream service degradation * Optimize rollout speed for infrastructure changes to reduce mitigation time