Vasion experienced a critical incident on May 21, 2025 affecting Quota/Cost Management and Quota/Cost Management and 1 more component, lasting 3d 7h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 21, 2025, 08:47 PM UTC
We are currently investigating this issue.
- investigating May 22, 2025, 12:28 PM UTC
We are continuing to investigate this issue.
- identified May 23, 2025, 01:16 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring May 23, 2025, 02:10 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 25, 2025, 03:55 AM UTC
This incident has been resolved.
- postmortem May 27, 2025, 07:55 PM UTC
**Issue Summary:** On May 21, 2025, a critical issue was identified where IdP groups and users were not syncing in Quota Management \(QM\). When selecting an IdP group in QM, the expected user list was returned empty, displaying no users to the end-user. Initially considered an instance-level database issue, it was immediately established as a Critical customer bug. After determining the affected regions, it was elevated to Blocking status on May 21, 2025, at 2:43 PM MT. **Root Cause:** The issue originated from a change implemented in Production on May 2, 2025, which affected the QM endpoint. The endpoint received an incorrect parameter, causing the failure to retrieve and display user lists. **Resolution:** Vasion’s Engineering teams resolved the issue by adjusting the calling procedures to pass the correct UUIDs instead of INTs. The fix was validated on May 22, 2025. It was then deployed to Production and validated on May 23, 2025. **Mitigation:** To prevent similar issues, QM testing will be included in future changes with Identity, given QM’s dependency on Identity. Each dependent service owner, including QM, will create and provide API tests for the Identity team to run whenever changes are made to Identity endpoint inputs or outputs. **Conclusion:** The issue was successfully resolved by correcting the parameter type passed to the endpoint, with the fix validated in Production. To enhance system reliability, integrating QM-specific API tests into the Vasion’s Engineering teams change process will help detect potential issues early, ensuring smoother operation of dependent services.