GDSC Datamart Ingestion Issue

Update timeline

1. investigating Apr 27, 2026, 03:58 PM UTC

   The datamart ingestion pipeline is experiencing partial failures, causing some datasets not to be ingested successfully. We are actively investigating the issue to determine the root cause and restore normal operation.

2. identified Apr 27, 2026, 06:51 PM UTC

   The root cause has been identified, and we are currently working on implementing a potential fix for the issue.

3. identified Apr 27, 2026, 10:21 PM UTC

   We are continuing to investigate an issue affecting the datamart ingestion service. Our engineering team is actively working to identify the root cause and restore normal operations. During this time, some data may be delayed or incomplete. We will provide further updates as more information becomes available.

4. identified Apr 29, 2026, 12:45 PM UTC

   A fix has been applied and seems issue is resolved. We are monitoring the service.

5. investigating Apr 30, 2026, 04:25 PM UTC

   Status: Mitigating We are continuing to address degraded data ingestion performance, which is causing delays in downstream components that rely on ingested data. The platform remains available; however, data freshness is impacted. The root cause has not yet been confirmed. In coordination with Cloud provider Support, current findings suggest database lock contention in Data warehouse DB as a contributing factor. We are actively tuning ingestion concurrency, scaling database capacity, and working with Cloud provider to restore normal processing rates. Further updates will be provided as mitigation progresses.

6. monitoring Apr 30, 2026, 11:50 PM UTC

   A mitigation has been successfully implemented and we are monitoring the results. The data warehouse engine rollback has completed, and data ingestion throughput has recovered as expected. The processing backlog is decreasing rapidly, and downstream components dependent on ingestion are beginning to normalize. We will continue to closely monitor ingestion performance and data freshness to ensure sustained stability before marking the incident as fully resolved.