Storj Outage History

Started Oct 20, 2026, 04:25 PM UTC· resolved Oct 20, 2026, 07:00 PM UTC

1. identified Oct 20, 2026, 04:25 PM UTC

   The issue has been identified and a fix is being implemented.

2. resolved Oct 20, 2026, 07:00 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 04, 2026, 04:39 PM UTC· resolved Jul 04, 2026, 06:10 PM UTC

1. investigating Jul 04, 2026, 04:39 PM UTC

   We're currently investigating reports of errors that occur when attempting to access resources, including objects and buckets.

2. monitoring Jul 04, 2026, 05:54 PM UTC

   We have implemented a fix and are monitoring the results.

3. monitoring Jul 04, 2026, 05:54 PM UTC

   We have applied a fix for the previously mentioned performance impact and are continuing to monitor the system.

4. monitoring Jul 04, 2026, 05:56 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Jul 04, 2026, 06:10 PM UTC

   The previously identified issue affecting performance has been resolved.

Read the full incident report →

Started Jul 02, 2026, 08:00 PM UTC· resolved Jul 02, 2026, 08:05 PM UTC

1. investigating Jul 02, 2026, 08:00 PM UTC

   We are investigating reports of EU1 login errors.

2. investigating Jul 02, 2026, 08:01 PM UTC

   We are continuing to investigate this issue.

3. monitoring Jul 02, 2026, 08:03 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 02, 2026, 08:05 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 03, 2026, 07:21 PM UTC· resolved Jun 03, 2026, 07:53 PM UTC

1. investigating Jun 03, 2026, 07:21 PM UTC

   We are investigating reports of elevated error rates from our API services.

2. investigating Jun 03, 2026, 07:21 PM UTC

   We are continuing to investigate this issue.

3. monitoring Jun 03, 2026, 07:29 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jun 03, 2026, 07:53 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 06, 2026, 10:10 AM UTC· resolved Mar 06, 2026, 12:31 PM UTC

1. investigating Mar 06, 2026, 10:10 AM UTC

   We are currently investigating this issue.

2. monitoring Mar 06, 2026, 10:48 AM UTC

   We have applied a fix for the previously mentioned performance impact and are continuing to monitor the system.

3. resolved Mar 06, 2026, 12:31 PM UTC

   The previously identified issue affecting performance has been resolved.

Read the full incident report →

Started Feb 24, 2026, 06:26 PM UTC· resolved Feb 24, 2026, 07:32 PM UTC

1. investigating Feb 24, 2026, 06:26 PM UTC

   We are investigating the issue

2. investigating Feb 24, 2026, 06:27 PM UTC

   We are continuing to investigate this issue.

3. identified Feb 24, 2026, 06:58 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Feb 24, 2026, 06:58 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Feb 24, 2026, 07:32 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 15, 2025, 12:31 PM UTC· resolved Dec 15, 2025, 12:31 PM UTC

1. resolved Dec 15, 2025, 12:31 PM UTC

   The issue with elevated error rates for uploads has been resolved for all affected users. The incident was limited to a region-specific storage tier in the United States.

Read the full incident report →

Started Sep 18, 2025, 06:41 PM UTC· resolved Sep 18, 2025, 07:34 PM UTC

1. investigating Sep 18, 2025, 06:41 PM UTC

   The main storj landing page is returning 500 errors. All Storj services remain unaffected. Use https://www.storj.io/login to login to the web console and use your accounts until this is resolved.

2. resolved Sep 18, 2025, 07:34 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 13, 2025, 05:52 AM UTC· resolved Sep 13, 2025, 08:22 AM UTC

1. investigating Sep 13, 2025, 07:42 AM UTC

   We have been alerted to an issue causing access keys in the EU1 region to stop working.

2. investigating Sep 13, 2025, 08:14 AM UTC

   We are continuing to investigate this issue.

3. resolved Sep 13, 2025, 08:22 AM UTC

   This incident has been resolved.

4. postmortem Sep 19, 2025, 03:30 PM UTC

   On September 13th, 2025, we experienced an issue that caused access keys to stop working for some customers in the EU1 region. This problem was isolated to a small subset of our infrastructure. We identified the root cause as a database configuration mismatch that occurred during a software update performed the previous day at approximately 21:12 UTC. Our team quickly redirected traffic away from the affected server to minimize service interruption. After implementing the necessary configuration changes, we were able to restore full functionality. The service has fully recovered, and we're taking steps to prevent similar issues in the future. We apologize for any inconvenience this may have caused.

Read the full incident report →

Started Sep 03, 2025, 07:44 PM UTC· resolved Sep 03, 2025, 09:49 PM UTC

1. investigating Sep 03, 2025, 08:13 PM UTC

   We are investigating reports of elevated error rates during uploads

2. identified Sep 03, 2025, 08:14 PM UTC

   The issue has been identified and a fix is being applied

3. monitoring Sep 03, 2025, 09:09 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 03, 2025, 09:49 PM UTC

   This incident has been resolved

Read the full incident report →

Started Aug 27, 2025, 09:25 PM UTC· resolved Aug 28, 2025, 06:37 PM UTC

1. identified Aug 27, 2025, 09:25 PM UTC

   Email sending on AP1 is disabled due to a bug which caused excessive amounts of bounced emails in a short period of time. The root cause of the bounced emails has been fixed, and we are working with our mail service provider to re-enable sending from ap1.storj.io as soon as possible.

2. resolved Aug 28, 2025, 06:37 PM UTC

   This incident has been resolved.

3. postmortem Sep 19, 2025, 09:14 PM UTC

   On 2025-08-04, we enabled a “trial expiration escalation” job, which runs every day to automatically mark accounts who have completed their free trial without upgrading as “pending deletion”, 90 days after trial expiration. When accounts are marked as “pending deletion”, they can no longer be accessed, but the data still needs to be deleted. On 2025-08-26, we executed a batch of these deletions, which fully deactivated 50k\+ users, deleting their data, and moving them from a “pending deletion” to “deactivated” state. Due to a bug in the “trial expiration escalation” job, a large portion of these deleted users were moved backwards from a “deactivated” status to a “pending deletion” status, resulting in a large batch of associated notification emails being sent to the already-deactivated account emails. A fully deactivated account has an email which looks like `[email protected]`, and prior to this incident, `[email protected]` was not a registered email address. In other words, on 2025-08-25, we attempted to send 50k\+ emails that bounced, to `[email protected]`. Our email service provider temporarily disabled our [ap1.storj.io](http://ap1.storj.io) sending domain as a result, due to “poor mailing list quality”. Once we became aware of the issue, we quickly registered `[email protected]` so that future emails sent to that address would not bounce. Additionally, we disabled our “trial expiration escalation” job to give us time to investigate and identify the issue described above. Finally, we corresponded with the support team for our email service provider to get email sending for AP1 reinstated as soon as possible. The full range of time that email service for AP1 was down spanned from roughly 2025-08-27 at 18:46 UTC, to 2025-08-28 at 16:35 UTC.

Read the full incident report →

Started Aug 26, 2025, 05:32 AM UTC· resolved Aug 26, 2025, 06:41 AM UTC

1. identified Aug 26, 2025, 05:32 AM UTC

   We have identified the cause of the elevated API error rate and will be applying a fix shortly.

2. monitoring Aug 26, 2025, 06:19 AM UTC

   A fix has been applied and we will continue to monitor the system.

3. resolved Aug 26, 2025, 06:41 AM UTC

   The previously identified issue has been resolved.

Read the full incident report →

Started Aug 07, 2025, 09:25 PM UTC· resolved Aug 07, 2025, 11:05 PM UTC

1. investigating Aug 07, 2025, 09:25 PM UTC

   We are investigating upload/download failures on US1

2. identified Aug 07, 2025, 09:30 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 07, 2025, 09:55 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Aug 07, 2025, 10:31 PM UTC

   We are investigating another increase in error rates.

5. monitoring Aug 07, 2025, 10:45 PM UTC

   Error rate are normal and we are continuing to monitor for further issues.

6. resolved Aug 07, 2025, 11:05 PM UTC

   This incident has been resolved.

7. postmortem Aug 22, 2025, 07:24 PM UTC

   ### Summary of the Storj US1 Service Disruption on August 7, 2025 #### Overview: On August 7, 2025, at approximately 21:04 UTC, the Storj US1 satellite experienced a performance degradation due to an unusually high volume of concurrent uploads to the same objects. The incident affected only the US1 satellite, while the AP1 and EU1 satellites remained fully operational throughout the event. #### Root Cause: The primary root cause was an exceptionally high volume of simultaneous uploads targeting the same objects, which triggered a bottleneck in the database due to transaction contention. This led to a cascade of issues, including database locking, request timeouts, and connection pool exhaustion. Database transactions can cause issues because they hold locks on the affected data, preventing other operations from accessing or modifying that data until the transactions are committed or rolled back. The increased level of long running transactions eventually leads to request timeouts. #### Impact: The incident affected customers and applications relying on the Storj US1 satellite for storage and retrieval operations. The AP1 and EU1 satellites were not affected by this incident and continued to operate normally. #### Timeline: 21:04 UTC: Database transaction locks started to increase. 21:14 UTC: The on-call team received a page and started investigating the issue. 21:55 UTC: A fix was implemented and the on-call team started monitoring the results. 22:31 UTC: The on-call team started investigating another increase in error rates. 22:45 UTC: Error rates trended back down to normal and the on-call team continued to monitor for further issues. 23:05 UTC: Operations were fully restored. #### Remediation and Prevention: To address the issue, we periodically reset the state of connections thus avoiding a cascading growth of contention related errors. To prevent similar incidents from occurring in the future, we implemented the following measures: 1. Identified and resolved sources of contentions between database transactions by implementing appropriate code updates. 2. Implemented additional monitoring and alerting mechanisms to detect and notify the team of increased levels of database locks. 3. Conducted thorough post-mortem analysis to identify any other potential improvements to our processes and systems, including building tools that ease or eliminate the need for manual maintenance. We apologize for any impact this service disruption may have caused our customers and users. We are committed to learning from this incident and continuing to improve the reliability and resilience of our platform.

Read the full incident report →

Started Jun 18, 2025, 07:59 AM UTC· resolved Jun 18, 2025, 02:27 PM UTC

1. investigating Jun 18, 2025, 07:59 AM UTC

   Some requests to Storj US1 are failing with error 500. We are investigating the issue.

2. monitoring Jun 18, 2025, 09:55 AM UTC

   We identified high lock time in the database and we scaled it up. We are already seeing reduced error levels as we continue to monitor.

3. monitoring Jun 18, 2025, 01:58 PM UTC

   Our additional analysis spotted a problem with the Redis cache. We restarted Redis and saw the remaining errors clearing.

4. resolved Jun 18, 2025, 02:27 PM UTC

   No more errors have been observed since the last intervention.

Read the full incident report →

Started May 22, 2025, 01:53 AM UTC· resolved May 22, 2025, 05:13 AM UTC

1. investigating May 22, 2025, 02:09 AM UTC

   We are investigating reports of performance impacts to one or more components of our system.

2. investigating May 22, 2025, 02:09 AM UTC

   We are continuing to investigate this issue.

3. monitoring May 22, 2025, 02:47 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 22, 2025, 05:13 AM UTC

   This incident has been resolved.

5. postmortem May 23, 2025, 05:20 PM UTC

   On May 22, 2025 from 00:12 to 02:54 UTC Storj experienced a service disruption causing elevated error rates and increased request latency that affected the US1 API, Gateway, Linksharing, and US1 - Select services. We discovered lock contention triggered by high-load delete operations. In the short term, we are artificially slowing down some of these intensive requests to leave needed headroom. We will eliminate the lock contention going forward.

Read the full incident report →

Started May 06, 2025, 05:22 PM UTC· resolved May 06, 2025, 05:22 PM UTC

1. resolved May 06, 2025, 05:51 PM UTC

   We are investigating reports of elevated error rates from our US1 services.

Read the full incident report →

Started May 06, 2025, 04:10 PM UTC· resolved May 07, 2025, 12:25 AM UTC

1. investigating May 06, 2025, 07:59 PM UTC

   We are investigating intermittent connectivity issues affecting the Linksharing service in the AP1 region.

2. identified May 06, 2025, 08:02 PM UTC

   We have identified the issue and are implementing a fix.

3. monitoring May 06, 2025, 08:12 PM UTC

   A fix has been implemented, and we are monitoring the results.

4. resolved May 07, 2025, 12:25 AM UTC

   The incident has been resolved.

Read the full incident report →

Started Apr 25, 2025, 07:55 PM UTC· resolved Apr 25, 2025, 08:06 PM UTC

1. investigating Apr 25, 2025, 07:55 PM UTC

   Our monitoring detected a brief period of uploads failing for our US Select service only. Our engineering team is actively investigating.

2. identified Apr 25, 2025, 07:59 PM UTC

   The issue has been identified and automated fail over kept the disruption minimal.

3. resolved Apr 25, 2025, 08:06 PM UTC

   This incident has been resolved. The short-lived disruption in uploads was due to an entire zone losing connectivity for a brief period of time. US Select is designed with three zone redundancy and the service automatically excluded the affected zone. The brief upload failures were during the period during cutover. Reads were never impacted. We apologize to all who might have been affected by the disruption.

Read the full incident report →

Started Apr 24, 2025, 02:47 PM UTC· resolved Apr 24, 2025, 02:47 PM UTC

1. resolved May 14, 2025, 02:47 PM UTC

   .

2. postmortem May 14, 2025, 02:48 PM UTC

   ## **Incident Summary** Between March 18 and April 10, 2025, Storj initiated a routine cleanup of long-frozen user accounts as part of ongoing system maintenance. Unfortunately, due to a series of procedural oversights, this process resulted in the accidental deletion of about a dozen paid user accounts on the EU1 satellite that had upgraded their frozen account during the deletion procedure. This post provides a high-level overview of what happened, how we responded, and what we’re doing to prevent similar issues in the future. ## **What Happened** * On March 18, a list of accounts was generated for deletion, targeting users whose accounts had been frozen \(due to lack of payment\) prior to January 1, 2025, and were still frozen as of March 18th, 2025. * On April 10, an internal command was run to prepare a set of frozen EU1 accounts for deletion and ultimately delete them. This internal command was not part of our usual SOP for account deletion, which is automated with robust QA and testing. * This command mistakenly included some active, paid users who had upgraded their frozen account during the two week period after the deletion lists were generated and did not include robust safeguards to catch this sort of error. The command directly set a new account status, regardless of its previous state. * The issue was discovered on April 24 after a user reported account access issues, triggering a broader investigation. ## **Impact** * 12 users total were identified as paid users who were marked for deletion while active. * 11 users were mistakenly deleted from the EU1 satellite * No users on AP1 or US1 were affected. ## **Root Cause** The deletion logic did not account for users who had recently upgraded to a paid tier after the deletion lists were generated but before the deletion process was executed \(a two week period\). We mistakenly assumed that accounts which had been frozen \(i.e. data was inaccessible\) and delinquent on payment for three months or more, were abandoned and therefore would not be upgraded. Our processes did include one or more reviews on all steps, including the list generation and the development and execution of the account deletion steps. The reviewers of the list generation and tool generation and command execution were independent and were not aware of the broader context and sequence of steps being taken in this case. **Response and Remediation** * We immediately halted further deletions once the issue was identified. * Three independent investigations and cross-references were run to definitively identify all affected accounts. * Impacted users are being contacted directly, and we are offering account restoration support and compensation where possible. * Internal tools are being updated to: * Prevent any future deletions of active paid users in the edge case that they upgrade their account while account cleanups are happening. The old internal tools are being removed and replaced with new versions that only allow explicit account status transitions from one state to another, rather than direct status update. * Add validation checks before executing bulk actions including end to end reviews of any future manual deletion processes. This will be in addition to our existing review, QA, and testing processes. * Require and include additional defensive safeguards on internal tools used for deletion including checks for other account upgrade information, including date filters beyond just status check. These checks are additional defense-in-depth measures that will catch any future mistakes on top of the above measures. ## **Our Apology** We deeply regret the disruption this has caused our affected users. Ensuring data integrity and account reliability is our top priority, and this event fell short of our standards. We appreciate the community’s patience and feedback as we work to make things right and build stronger safeguards moving forward. — The Storj Team

Read the full incident report →

Started Apr 14, 2025, 03:33 PM UTC· resolved Apr 14, 2025, 04:07 PM UTC

1. investigating Apr 14, 2025, 03:33 PM UTC

   We are experiencing an issue with the US1 satellite (metadata region), which is intermittently inaccessible. Our engineering team is actively investigating. We apologize to all who are affected by the disruption.

2. monitoring Apr 14, 2025, 03:54 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 14, 2025, 04:07 PM UTC

   The issue with the US metadata region has been resolved for all affected users. We thank you for your patience while we worked on resolving the issue.

Read the full incident report →

Started Apr 04, 2025, 01:46 AM UTC· resolved Apr 04, 2025, 02:51 AM UTC

1. investigating Apr 04, 2025, 01:46 AM UTC

   We are currently experiencing issues with the Link Sharing service in the Europe region. Customers may notice increased request times and error rates. Our engineering team is actively investigating. We apologize to all who are affected by the disruption.

2. monitoring Apr 04, 2025, 02:31 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 04, 2025, 02:51 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 12, 2025, 09:10 PM UTC· resolved Mar 13, 2025, 12:18 AM UTC

1. monitoring Mar 12, 2025, 09:10 PM UTC

   We have scaled up to meet the load and are monitoring behavior.

2. monitoring Mar 12, 2025, 09:28 PM UTC

   We are continuing to monitor for any further issues.

3. resolved Mar 13, 2025, 12:18 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 06, 2025, 10:08 PM UTC· resolved Mar 06, 2025, 10:49 PM UTC

1. investigating Mar 06, 2025, 10:08 PM UTC

   We are currently investigating this issue.

2. identified Mar 06, 2025, 10:31 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 06, 2025, 10:48 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 06, 2025, 10:49 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 18, 2025, 03:57 PM UTC· resolved Feb 18, 2025, 04:17 PM UTC

1. investigating Feb 18, 2025, 03:57 PM UTC

   We are experiencing an issue with edge services connectivity affecting customers in Asia-Pacific. Our engineering team is actively investigating. We apologize to all who are affected by the disruption.

2. resolved Feb 18, 2025, 04:17 PM UTC

   This issue has been resolved for all affected users. We thank you for your patience while we worked on resolving the issue.

Read the full incident report →

Started Feb 12, 2025, 11:09 AM UTC· resolved Feb 12, 2025, 11:54 AM UTC

1. monitoring Feb 12, 2025, 11:49 AM UTC

   A fix has been implemented and we are monitoring the results.

2. resolved Feb 12, 2025, 11:54 AM UTC

   This incident has been resolved.

Read the full incident report →