Logs not visible in CI Executions post completion

Update timeline

1. investigating Jan 20, 2026, 11:00 AM UTC

   We are currently investigating an issue affecting log visibility for a few CI steps after execution completion. Our team is actively working on identifying the root cause.

2. identified Jan 20, 2026, 02:21 PM UTC

   The issue was identified in the latest log-service deployment; as a mitigation, the deployment was rolled back.

3. monitoring Jan 20, 2026, 02:21 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jan 20, 2026, 02:24 PM UTC

   This incident has been resolved.

5. postmortem Jan 28, 2026, 07:33 PM UTC

   ## **Summary** On January 19, 2026, some customers experienced missing logs for certain CI pipeline steps with very short execution times. The issue was related to increased latency in the log streaming process, which caused log data for affected steps to be unavailable after execution completed. The issue was identified and mitigated promptly. ## **Impact** During the incident window: * CI pipeline steps with very short execution durations may have shown incomplete or missing logs after completion. * The issue primarily affected steps that completed execution before log streaming was fully established. * Pipeline execution itself was **not impacted**, and builds continued to run successfully. There was **no impact to pipeline correctness, execution results, or customer data**, aside from log visibility for the affected steps. ## **Root Cause** The issue occurred due to increased latency in the log streaming initialization process. For very short-running steps, execution completed before the log stream was fully established. As a result, the system attempted to finalize logging before the stream was ready, leading to an empty log snapshot being recorded. This behavior caused valid log data to be unavailable for a subset of short-duration steps under specific timing conditions. ## **Mitigation** As an immediate mitigation: * The recent log service change contributing to increased stream latency was rolled back. * This action restored normal log streaming behavior and prevented further log loss. Service stability and log visibility were fully restored following the mitigation. ## **Action Items** To prevent recurrence and strengthen resiliency, the following improvements are being implemented: ‌ * Add safeguards to prevent log streams from being finalized before initialization is complete. * Improve validation logic to ensure valid log data is not overwritten by empty or incomplete snapshots. * Introduce additional protections for short-duration steps to ensure consistent log availability. * Enhance monitoring around log streaming latency to enable earlier detection of similar conditions.