GitLab incident

Elevated Errors Affecting AI Gateway Health Checks

Minor Resolved View vendor source →

GitLab experienced a minor incident on July 14, 2026 affecting GitLab Duo and GitLab agent server for Kubernetes (Google Compute Engine), lasting 1h 59m. The incident has been resolved; the full update timeline is below.

Started
Jul 14, 2026, 10:47 PM UTC
Resolved
Jul 15, 2026, 12:47 AM UTC
Duration
1h 59m
Detected by Pingoru
Jul 14, 2026, 10:47 PM UTC

Affected components

GitLab DuoGitLab agent server for Kubernetes (Google Compute Engine)

Update timeline

  1. investigating Jul 14, 2026, 10:47 PM UTC

    We're aware of elevated errors affecting AI Gateway health checks. We've identified a likely cause and are actively working on a fix. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519 Back to current status Status History Subscribe to receive status updates by email Close Subscribe Manage Subscription Manage Existing Subscription Create New Subscription Subscribe to receive status updates by webhook Each status update will POST a JSON payload to this URL Email address for managing webhook Close Subscribe Manage Subscription Manage Existing Subscription Create New Subscription Subscribe to receive status updates in Microsoft Teams Enter your Microsoft Teams webhook. View Instructions Email address for managing subscription Close Subscribe Manage Subscription Manage Existing Subscription Create New Subscription Subscribe to receive status updates via RSS RSS Feed Close Follow our status updates on Twitter/X @gitlabstatus Close Subscribe to receive status updates in Slack Slack channel ID Find the channel ID: Select the channel in your Slack workspace. The channel ID is displayed in the browser URL. Example: https://app.slack.com/client/T04SJBK1C/ C03SKGJ1P Email address Close Manage Subscription Manage Existing Subscription Create New Subscription Subscribe to receive status updates via iCalendar iCalendar Feed Close

  2. investigating Jul 14, 2026, 10:47 PM UTC

    We're aware of elevated errors affecting AI Gateway health checks. We've identified a likely cause and are actively working on a fix. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  3. identified Jul 14, 2026, 11:02 PM UTC

    We have identified the problematic change causing this behavior and are working to revert it. Details will be shared in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  4. identified Jul 14, 2026, 11:02 PM UTC

    We have identified the problematic change causing this behavior and are working to revert it. Details will be shared in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  5. identified Jul 14, 2026, 11:37 PM UTC

    We're continuing to work on reverting the change causing this issue. Details will be shared in: gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  6. identified Jul 14, 2026, 11:37 PM UTC

    We're continuing to work on reverting the change causing this issue. Details will be shared in: gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  7. identified Jul 15, 2026, 12:00 AM UTC

    The revert remains in progress. Additional updates will be shared in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  8. identified Jul 15, 2026, 12:00 AM UTC

    The revert remains in progress. Additional updates will be shared in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  9. monitoring Jul 15, 2026, 12:24 AM UTC

    Deployment for the revert has reached production. Manual verification is successful with error rates trending down. We are continuing to monitor. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  10. monitoring Jul 15, 2026, 12:24 AM UTC

    Deployment for the revert has reached production. Manual verification is successful with error rates trending down. We are continuing to monitor. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  11. monitoring Jul 15, 2026, 12:37 AM UTC

    We are monitoring the situation to ensure that services have returned to normal. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  12. monitoring Jul 15, 2026, 12:37 AM UTC

    We are monitoring the situation to ensure that services have returned to normal. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  13. resolved Jul 15, 2026, 12:47 AM UTC

    We are seeing error rates return to normal. This issue has been resolved. Details at https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519