DCS incident

Long Running Backup Jobs

Minor Resolved View vendor source →

DCS experienced a minor incident on June 28, 2024 affecting CloudCover365 Portal and CloudCover365 Services, lasting 26d 4h. The incident has been resolved; the full update timeline is below.

Started
Jun 28, 2024, 08:08 AM UTC
Resolved
Jul 24, 2024, 12:41 PM UTC
Duration
26d 4h
Detected by Pingoru
Jun 28, 2024, 08:08 AM UTC

Affected components

CloudCover365 PortalCloudCover365 Services

Update timeline

  1. investigating Jun 28, 2024, 08:08 AM UTC

    We are aware of an ongoing issue with the backup service. We are investigating as a top priority

  2. investigating Jun 28, 2024, 11:15 AM UTC

    We are continuing to investigate an issue that some of our customers are seeing with their backup jobs. The issue seems to be affecting a small subset of our proxy servers and we are working with up most urgency to resolve.

  3. investigating Jun 28, 2024, 02:28 PM UTC

    Some customers jobs have been restored to normal functionality but there is still an issue on some of our proxy servers. We are actively engaged with our vendors R&D team seeking a resolution.

  4. investigating Jun 28, 2024, 04:49 PM UTC

    Issue remains under investigation. Engineers will be working OOH to ensure the swiftest resolution possible.

  5. investigating Jul 02, 2024, 09:03 AM UTC

    We are continuing to work with our vendors to bring this matter to a resolution. Issue looks to be linked to an issue with our object storage vendor. Their engineers are working around the clock to restore their platform to normal performance.

  6. investigating Jul 02, 2024, 10:24 AM UTC

    We are continuing to investigate this issue.

  7. monitoring Jul 03, 2024, 01:21 PM UTC

    We’re pleased to report that the platform has now stabilised. Our engineers continue to monitor the situation, liaising with the Object Storage vendor to ensure services continue to work as expected.

  8. monitoring Jul 07, 2024, 04:25 PM UTC

    We have detected a slowdown in some jobs that would appear to be be a repeat of the previous issue. We are investigating with our 3rd party storage vendor.

  9. resolved Jul 24, 2024, 12:41 PM UTC

    Speeds are now back to normal levels. We are continuing to monitor the speeds of all jobs to make sure this continues.