Cumulocity incident

Cumulocity IoT DataHub (CDH) for cumulocity.com experiencing issues

Critical Resolved View vendor source →

Cumulocity experienced a critical incident on October 22, 2024 affecting cumulocity.com API Services, lasting 7d 4h. The incident has been resolved; the full update timeline is below.

Started
Oct 22, 2024, 06:26 AM UTC
Resolved
Oct 29, 2024, 11:21 AM UTC
Duration
7d 4h
Detected by Pingoru
Oct 22, 2024, 06:26 AM UTC

Affected components

cumulocity.com API Services

Update timeline

  1. identified Oct 22, 2024, 06:26 AM UTC

    Users may experience instability and unavailability of the service. We have identified the issue, a hotfix is being tested in our staging environment.

  2. identified Oct 23, 2024, 12:39 PM UTC

    After testing in staging environments, a hotfix version of DataHub is ready. We will deploy it shortly.

  3. monitoring Oct 23, 2024, 02:58 PM UTC

    The hotfix version has been deployed. We are monitoring the environment.

  4. investigating Oct 24, 2024, 09:42 AM UTC

    The hotfix did not work as planned and Dremio is still unstable. We have escalated with the third-party vendor and are following up with them.

  5. investigating Oct 28, 2024, 10:58 AM UTC

    We have applied several recommendations of the third-party vendor, however every time the system went back to unstable state. We will now implement further recommendations, which require full downtime of Dremio. The expected downtime is about one hour.

  6. investigating Oct 28, 2024, 12:22 PM UTC

    The change has been implemented successfully. We are continuing to monitor the situation.

  7. resolved Oct 29, 2024, 11:21 AM UTC

    The issue has been resolved by implementing multiple changes to hardware and software configurations. Thank you for your patience.