CommCare HQ incident

Degraded performance affecting some case updates, conditional alerts, sending reports and creating new app builds on www.commcarehq.org

Resolved after 43 minutes, affecting App Release Management and User Management and 1 more, on October 21, 2024.

resolved Oct 21, 2024, 10:53 PM UTC

Dear Users, We have observed a full recovery as all previously backlogged tasks have now been processed and new tasks are being processed at a normal rate. We sincerely apologize for any inconvenience caused.

Started
Oct 21, 2024, 10:10 PM UTC
Resolved
Oct 21, 2024, 10:53 PM UTC
Duration
43m
Detected by Pingoru
Oct 21, 2024, 10:10 PM UTC

Affected components

App Release ManagementUser ManagementProject Reporting

Update timeline

  1. identified Oct 21, 2024, 10:10 PM UTC

    Dear Users, We have identified an issue beginning at 18:37 UTC in which tasks to be processed are backed up in queue. Affected tasks include updating calculated properties, making a new build on the app builder, creating ccz, creating user cases, sending reports, resending pending invitations, running case update rules on save, and rebuilding cases. Our developers are aware of the issue and are in the process of implementing a fix. We apologize for any inconvenience caused and will have an update shortly.

  2. monitoring Oct 21, 2024, 10:13 PM UTC

    Dear Users, A fix for this issue has been implemented and we are seeing an increased rate of backlogged tasks being processed. We expect a full resolution of this issue within a few hours. We will continue to monitor the progress of this fix and check in with any further updates hourly or when this issue is resolved.

  3. resolved Oct 21, 2024, 10:53 PM UTC

    Dear Users, We have observed a full recovery as all previously backlogged tasks have now been processed and new tasks are being processed at a normal rate. We sincerely apologize for any inconvenience caused.

Your product runs on more than CommCare HQ

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →