Instability affecting ID CLOUD

Update timeline

1. investigating Aug 30, 2026, 02:35 AM UTC

   Dear Customer, At approximately 11:00 PM (BRT) on Friday, August 29, 2026, we identified an instability affecting process creation in IDCloud. During this period, some requests may fail or experience slower than usual response times. We recognize the inconvenience this may cause and appreciate your understanding. A new update will be provided shortly. Unico Team

2. identified Aug 30, 2026, 02:49 AM UTC

   Our engineering team has mapped the origin of the instability affecting process creation in IDCloud. The technical team has already begun applying corrective measures, including scaling resources in the affected layer and continuously monitoring key indicators. Restoring the service is being handled with top priority and full engineering mobilization. We reiterate that further updates will be sent shortly. Unico Team

3. monitoring Aug 30, 2026, 02:57 AM UTC

   Dear Customer, Our engineering team has carried out the corrective actions required to resolve the instability affecting process creation in IDCloud. The environment has returned to stability, with availability and response-time indicators restored to normal operating levels as of 11:48 PM (BRT). The service is available for normal use. At this time, our technical team remains under assisted monitoring, closely watching the environment to ensure consistent performance and to act immediately in case of any fluctuation. We appreciate your understanding and reiterate that further updates will be sent shortly. Unico Team

4. resolved Aug 30, 2026, 03:15 AM UTC

   Dear Customer, Executive Summary and Impact Between 11:00 PM and 11:48 PM (BRT) on August 29, 2026, an outage affected process creation in IDCloud. During this window, some customers using these flows experienced failures and increased response times, with propagation to dependent journeys including authentication, document capture, and notification delivery. There was no compromise to the integrity or security of the processed information, and no data was lost. Indicators returned to normal operating levels at 11:48 PM (BRT), with normalization confirmed both in our monitoring and on the customer side, covering process creation and completion. Root Cause and Resolution The origin was identified in our data layer: an internal maintenance routine became stuck, and application operations began queuing behind it until the available connection limit was exhausted, reducing processing capacity. Our engineering team surgically terminated only the stuck routine, immediately releasing the queue. Connections and processing capacity were fully restored, with no data changes. Commitment and Next Steps Our engineering team will focus on three fronts: limiting wait times for internal maintenance routines, preventing concurrent routines from running against the same structure, and strengthening service resilience to fluctuations in the data layer. A detailed postmortem, including the complete timeline and an action plan with deadlines, will be shared shortly. We sincerely apologize for the impact on your operations and remain available through our support channels. Unico Team

5. postmortem Sep 10, 2026, 07:43 PM UTC

   **Summary** In the early hours of August 29–30, 2026, between 23:00 and 23:47 \(Brasília time\), process creation on the platform became unavailable, resulting in 503 errors for all users attempting to access the service. The incident lasted approximately 47 minutes and affected 91 user groups, with a direct impact on more than 63,000 requests. **Impact** The success rate for the process creation flow dropped to 67.86% against a 99% availability target. Of the 196,884 requests during the period, 63,273 resulted in failure. The highest volume of errors was recorded among users of sports betting platforms, which operate with high traffic volumes especially during live matches. In addition, other user groups — including financial and healthcare services — were affected to a lesser extent. **Root Cause** The incident was caused by a design flaw in the interaction between two automated database maintenance processes, compounded by high traffic volume at the time of the event. The database was running a mandatory internal maintenance operation on the main process table. This type of maintenance is non-interruptible and holds an exclusive lock on the table. At the same time, a scheduled partition creation job on another table — which has a foreign key relationship with the process table — attempted to acquire a lock on the same main table. Because the database uses a lock queue, the partition job waited indefinitely \(with no timeout configured\), causing all subsequent write operations to also queue up. This accumulation exhausted the service's available connection limit, causing new connections to be rejected. Since application pods perform a database connectivity check on startup and fail immediately when unable to connect — with no retry mechanism — 99 out of 100 pods entered a continuous restart loop, rendering the service completely unavailable at the edge layer. **Resolution** The mitigation was precise and surgical: only the blocked partition job session was terminated. Immediately after, the accumulated connections were released — dropping from ~1,500 to 19 within seconds — the pods returned to normal operation, and the availability indicator reached 100% at 23:48. **Lessons Learned** * The scheduled maintenance job had no lock acquisition timeout configured. A timeout of just a few seconds would have been enough for the job to fail gracefully and be rescheduled in the next cycle, preventing the cascading failure. * The foreign key relationship between the partition table and the main process table was identified as the architectural link that connected a low-impact maintenance job to an extremely high-write table. This relationship was removed as part of the fixes applied after the incident.