Cleeng experienced a notice incident on February 23, 2024 affecting ChurnIQ, lasting 1d 6h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 23, 2024, 01:23 PM UTC
Delay in ChurnIQ data refresh Due to a surge in data load in the last 24 hours, some of the ChurnIQ data refresh pipelines did not run as expected. As a result, your ChurnIQ dashboard and exports may not be up to date. We are working on a fix and will update the status page periodically.
- monitoring Feb 24, 2024, 12:54 AM UTC
The issue is fixed and is under monitor now.
- resolved Feb 24, 2024, 07:45 PM UTC
This incident has been resolved.
- postmortem Feb 28, 2024, 04:38 PM UTC
Dear Cleeng Broadcasters, We regret to inform you of an incident that occurred on February 23rd, 2024, at 11:00 UTC \(13:00 CET\), impacting the processing of our DBT \(Data Build Tool\) pipeline. A surge in data load within the preceding 24 hours resulted in disruptions to our ChurnIQ data refresh pipelines, specifically affecting the 6-hour and 24-hour intervals. This led to delays in our reporting data, affecting the accuracy of ChurnIQ dashboards and exports. We sincerely apologise for any inconvenience this may have caused and assure you that our team quickly responded to the incident and mitigated the impact on our services. The root cause of the incident was identified as an error indicating duplicated data in one of the DBT models. Despite initial indications, further analysis confirmed no actual duplicates were present. Additionally, over the weekend of February 25th, 2024, our service provider for data infrastructure had to roll back their recent deployment due to an unspecified bug. This unexpected rollback caused disruptions in our data processing systems, and leading to the aforementioned root cause. We are actively working to improve our procedures to prevent similar occurrences in the future. Your trust and satisfaction are of top priority to us, and we appreciate your patience and understanding in our dealing with this matter. Thank you for your continued support. Sincerely, The Cleeng Team