ScalePad incident

Backup Radar: EU Email Backup Result Loss

Minor Resolved View vendor source →

ScalePad experienced a minor incident on April 15, 2025 affecting Backup Radar, lasting 8d 3h. The incident has been resolved; the full update timeline is below.

Started
Apr 15, 2025, 02:11 PM UTC
Resolved
Apr 23, 2025, 05:42 PM UTC
Duration
8d 3h
Detected by Pingoru
Apr 15, 2025, 02:11 PM UTC

Affected components

Backup Radar

Update timeline

  1. monitoring Apr 15, 2025, 02:11 PM UTC

    We've identified an issue where incoming backup emails for our EU-based accounts were not being processed correctly. We've deployed a fix and are currently reprocessing emails that were incorrectly parsed. This affected emails that were received between April 14, 8:00 UTC and April 15, 12:00 UTC. We will update again here once we have finished reprocessing all affected messages.

  2. monitoring Apr 15, 2025, 02:11 PM UTC

    We are continuing to monitor for any further issues.

  3. monitoring Apr 16, 2025, 06:28 PM UTC

    We've made significant progress in reprocessing affected backup emails from April 14, 08:00 UTC to April 15, 12:00 UTC. The majority of impacted partners should now see their backup results restored. A small number of partners with a high volume of backups may still be processing, and we are continuing to monitor this. We'll provide a final update once reprocessing is fully complete across all accounts.

  4. monitoring Apr 18, 2025, 01:59 PM UTC

    We’re updating this incident based on new information confirming that some backup result emails were not saved and are unrecoverable. Between April 14 at 08:00 UTC and April 15 at 12:00 UTC, a deployment in our EU region was impacted by a missing backend library responsible for saving incoming emails. Although the deployment was logged as successful, this omission prevented some emails from being stored — resulting in missing backup results for affected partners. Only email-based integrations in the EU region were affected. API-based integrations and US-region partners were not impacted. We are implementing additional safeguards and monitoring to prevent similar issues in the future. We sincerely apologize for the disruption and appreciate your understanding.

  5. resolved Apr 23, 2025, 05:42 PM UTC

    This incident has been resolved.