Accidental Deletion of Paid User Accounts

Update timeline

1. resolved May 14, 2025, 02:47 PM UTC

   .

2. postmortem May 14, 2025, 02:48 PM UTC

   ## **Incident Summary** Between March 18 and April 10, 2025, Storj initiated a routine cleanup of long-frozen user accounts as part of ongoing system maintenance. Unfortunately, due to a series of procedural oversights, this process resulted in the accidental deletion of about a dozen paid user accounts on the EU1 satellite that had upgraded their frozen account during the deletion procedure. This post provides a high-level overview of what happened, how we responded, and what we’re doing to prevent similar issues in the future. ## **What Happened** * On March 18, a list of accounts was generated for deletion, targeting users whose accounts had been frozen \(due to lack of payment\) prior to January 1, 2025, and were still frozen as of March 18th, 2025. * On April 10, an internal command was run to prepare a set of frozen EU1 accounts for deletion and ultimately delete them. This internal command was not part of our usual SOP for account deletion, which is automated with robust QA and testing. * This command mistakenly included some active, paid users who had upgraded their frozen account during the two week period after the deletion lists were generated and did not include robust safeguards to catch this sort of error. The command directly set a new account status, regardless of its previous state. * The issue was discovered on April 24 after a user reported account access issues, triggering a broader investigation. ## **Impact** * 12 users total were identified as paid users who were marked for deletion while active. * 11 users were mistakenly deleted from the EU1 satellite * No users on AP1 or US1 were affected. ## **Root Cause** The deletion logic did not account for users who had recently upgraded to a paid tier after the deletion lists were generated but before the deletion process was executed \(a two week period\). We mistakenly assumed that accounts which had been frozen \(i.e. data was inaccessible\) and delinquent on payment for three months or more, were abandoned and therefore would not be upgraded. Our processes did include one or more reviews on all steps, including the list generation and the development and execution of the account deletion steps. The reviewers of the list generation and tool generation and command execution were independent and were not aware of the broader context and sequence of steps being taken in this case. **Response and Remediation** * We immediately halted further deletions once the issue was identified. * Three independent investigations and cross-references were run to definitively identify all affected accounts. * Impacted users are being contacted directly, and we are offering account restoration support and compensation where possible. * Internal tools are being updated to: * Prevent any future deletions of active paid users in the edge case that they upgrade their account while account cleanups are happening. The old internal tools are being removed and replaced with new versions that only allow explicit account status transitions from one state to another, rather than direct status update. * Add validation checks before executing bulk actions including end to end reviews of any future manual deletion processes. This will be in addition to our existing review, QA, and testing processes. * Require and include additional defensive safeguards on internal tools used for deletion including checks for other account upgrade information, including date filters beyond just status check. These checks are additional defense-in-depth measures that will catch any future mistakes on top of the above measures. ## **Our Apology** We deeply regret the disruption this has caused our affected users. Ensuring data integrity and account reliability is our top priority, and this event fell short of our standards. We appreciate the community’s patience and feedback as we work to make things right and build stronger safeguards moving forward. — The Storj Team