Pipeline executions failing for only users whose names contain non-ASCII characters

Update timeline

1. identified Jan 26, 2026, 01:10 PM UTC

   The issue has been identified and a fix is being implemented.

2. identified Jan 26, 2026, 01:16 PM UTC

   All users having no ascii characters failed to execute pipelines.

3. identified Jan 26, 2026, 02:17 PM UTC

   We are continuing to work on a fix for this issue.

4. resolved Jan 27, 2026, 01:25 AM UTC

   This incident has been resolved.

5. postmortem Jan 29, 2026, 09:52 PM UTC

   ## **Summary** On January 26, 2026, a subset of pipeline executions in the production environment failed when triggered by users whose names contained non-ASCII characters. The issue was caused by an encoding error introduced as part of a recent platform change related to propagating user context information between services. The issue was identified, mitigated, and fully resolved through a corrective deployment. ## **Impact** During the incident window: * Pipeline executions triggered by users with non-ASCII characters in their usernames failed to start. * Pipeline executions triggered by users without special characters were not impacted. * Pipeline definitions, execution results, and stored artifacts were not affected. There was **no data loss**, and the impact was limited to execution initiation for a specific subset of users. ## **Root Cause** A recent platform enhancement introduced logic to pass user context information between internal services using HTTP headers. This implementation did not correctly encode header values containing non-ASCII characters. When a pipeline execution was triggered by a user whose name contained special or international characters, the improperly encoded header value caused request validation to fail, resulting in execution failures. ‌ ## **Mitigation** As an immediate mitigation: * Customers were advised to temporarily trigger pipelines using a user account without non-ASCII characters in the username. * A corrective fix was deployed to properly encode and decode user context data passed between services. Following these actions, pipeline executions resumed normal operation for all users. ## **Action Items** To prevent recurrence and improve overall platform robustness, the following actions are being taken: * Enforce proper encoding and decoding of all user-provided data used in service-to-service communication. * Expand automated test coverage to include users with non-ASCII and international characters in usernames and related user fields. * Improve validation across additional user-facing workflows to ensure consistent handling of internationalized input. * Introduce broader internationalization-focused test scenarios to detect similar issues earlier in the development lifecycle.