InfluxData Outage History

Started Sep 29, 2026, 05:28 AM UTC· resolved Oct 02, 2026, 06:17 PM UTC

1. investigating Sep 29, 2026, 05:28 AM UTC

   We are currently investigating an issue affecting query performance in Azure West Europe.

2. investigating Sep 29, 2026, 06:38 AM UTC

   We are continuing to investigate this issue and will provide further updates as soon as available.

3. identified Sep 29, 2026, 09:09 AM UTC

   The issue has been identified and a fix is being implemented.

4. identified Sep 29, 2026, 11:34 AM UTC

   We are continuing to work on a fix for this issue.

5. identified Sep 29, 2026, 03:25 PM UTC

   We are continuing work to remediate immediate query performance degradation, and investigating actions for a permanent fix.

6. monitoring Sep 29, 2026, 04:39 PM UTC

   Query performance has returned to normal levels. Planned stabilisation work is still ongoing, and we are monitoring closely.

7. monitoring Sep 29, 2026, 06:33 PM UTC

   Query performance remains stable, and we continue to monitor closely. Additional mitigation and stabilization work is ongoing.

8. monitoring Sep 29, 2026, 11:31 PM UTC

   Query performance remains stable, and we continue to monitor closely as traffic increases. We will continue to assess performance and provide updates.

9. monitoring Sep 30, 2026, 07:34 AM UTC

   We are working on resolving the ongoing intermittent query performance. During peak usage periods, large or complex queries may run slower than normal. Writes and data durability are not affected at this time. We have deployed several improvements overnight and further changes are in progress.

10. identified Sep 30, 2026, 07:53 AM UTC

    We have identified the issue and the team is actively working on implementing a fix.

11. investigating Sep 30, 2026, 08:50 AM UTC

    We are still having issues in the region, the engineering team is working on it.

12. investigating Sep 30, 2026, 10:49 AM UTC

    We are continuing to investigate this issue.

13. investigating Sep 30, 2026, 12:08 PM UTC

    We are continuing to investigate this issue.

14. investigating Sep 30, 2026, 01:44 PM UTC

    We have made several recent changes to improve query performance and we continue to monitor the region closely.

15. monitoring Sep 30, 2026, 02:24 PM UTC

    Performance has returned to normal levels following an update. We will continue to monitor.

16. monitoring Sep 30, 2026, 05:16 PM UTC

    We have increased isolation in the storage layer for the impacted region to provide greater amounts of resilience and to allow for faster recovery. We are continuing to investigate heightened memory consumption and page faults within the storage layer. Query success rates and response durations have returned to normal levels. We will continue work to increase the resiliency of this region over the next 24 hours and will carefully monitor for any increase in query latency.

17. monitoring Oct 01, 2026, 08:13 AM UTC

    The cluster remains stable, and we are continuing to monitor it.

18. resolved Oct 02, 2026, 06:17 PM UTC

    We appreciate the continued patience while we worked through this incident. We have been monitoring the previous fixes for the last 24 hours and all changes have responded well during sustained traffic. The region is stable and we are resolving this incident. An RCA is in progress and will be added to our status page as soon as its available.

19. postmortem Oct 06, 2026, 07:00 PM UTC

    # RCA # Query Performance Degradation in Azure West Europe, September 27 to October 1, 2026 ## Summary From the evening of September 27 to the morning of September 30 \(UTC\), customers in the Azure West Europe region experienced slow queries and timeouts. Writes were accepted throughout, but on some partitions, new data was not immediately queryable. Earlier incidents on this cluster \(September 7, 11, 15 and 16\) produced the same symptom with different triggers and have their own RCAs. We believe the condition described herein was the root cause of all prior incidents. ## Cause All data in the cluster is stored on redundant pods across many physical nodes. These pods read time-series data through memory-mapped files. The host’s operating system keeps the most frequently accessed parts of those files in a page cache built from memory not consumed by the pods themselves. In this region, each pod holds about 365 GB of data. The cache on a host could not hold the data its pods were querying, so reads that would normally come from cache became page faults served from disk. Each pod read about 90 MB/s from disk, thus a host with ten pods reached roughly 990 MB/s, which is near the throughput ceiling on each host. Memory consumption looked normal on every metric we monitor. The storage process itself uses little memory, and the cache takes the remainder, so a shortage would not appear as memory pressure or shortfall. At the throughput ceiling, index lookups on the write path were delayed by disk reads, causing writes to fall behind and pods to fail their replication-lag health check. Pods in this region spent 8.9% of the day past the lag threshold, against 0.09% in a comparable region. At peak, both replicas of the busiest partitions fell behind together, creating access delays which were most acutely visible through the query path. Slow disk caused health and readiness checks to time out, so healthy pods were removed from rotation and restarted. Each restart took 30 to 85 minutes to recover, during which the pod's partner carried the load alone, with the rebuild adding even more disk load. A file-reference safeguard was also causing 50\+ pod restarts a day where a normal rate would be 0. ## Remediation While early changes helped, and resolved many trouble indicators, they did not solve the root cause. We doubled query-layer capacity, restarted pods, raised the readiness timeouts, raised cleanup timeouts to avoid safeguard restarts, and doubled recovery concurrency after a restart. We ruled out a controller lock, a per-disk limit, and a single tenant's load. The root cause fix was to raise each storage pod's resource request so that each host holds fewer pods. Each pod now has twice as much memory for page cache and disk throughput. Within 1 hour of this change, query abandonment fell from 31% to 0.1%. Page faults and disk reads dropped sharply. During the next morning's peak, abandonment was 0.06%, within normal limits. ## Timeline \(UTC\) Sep 27 22:58 - Query queue depth alert fires; incident opened. Query-layer capacity increased over the next hours without lasting effect Sep 28 07:00 - Degradation at morning peak; storage pods fall behind on writes and fail health checks; stuck pods restarted one at a time Sep 28 11:33 - Restart loop identified: slow disk causes health checks to trigger restart; each restart takes 30 to 85 minutes to recover Sep 28 13:45 - Lost file-reference cleanup identified as a second source of forced restarts Sep 29 08:58 - Readiness probe timeout raised and rolled out fleet-wide Sep 29 11:10 - Throughput appears to recover; degradation returns, with 36% of queries abandoned Sep 29 16:20 - Queue clears after recovery-concurrency and probe changes Sep 30 00:25 - Cleanup timeout raised to 24 hours; forced restarts stop Sep 30 07:00 - Degradation at peak; abandonment about 55% Sep 30 11:59 - Analysis shows cache capacity per pod is too small and disk reads are at the host ceiling Sep 30 12:43 - Rollout begins to spread pods across more nodes Sep 30 13:21 - Query performance returns to normal; abandonment down to 0.1% within the hour Sep 30 14:20 - Page faults and disk reads confirmed sharply lower after rollout Oct 1 07:00 to 09:00 - Morning peak with 0.06% abandonment versus 40% to 55% the day before Oct 1 09:48 - Incident declared resolved internally ## Future mitigation We added warning alerts for page-fault rate, cache coverage, disk reads on storage volumes, and query abandonment. Each alert is designed so that cache pressure appears before causing external visibility. Runbooks now suggest major page faults as a leading indicator of storage performance degradation, with details on “relief valves” available to tune. We are continuing to track down the storage-engine reference leak and moving the disk check out of the health path. Neither leak nor check is a risk factor for triggering this type of incident; they are conditions observed during diagnosis and remediation that warrant investigation.

Read the full incident report →

Started Sep 28, 2026, 08:33 AM UTC· resolved Sep 28, 2026, 08:19 PM UTC

1. investigating Sep 28, 2026, 08:33 AM UTC

   We are currently investigating this issue.

2. identified Sep 28, 2026, 09:54 AM UTC

   Some queries may be slower than normal and recently written data may take longer to appear in results. Data ingestion and durability are not affected. We have identified the cause and mitigation is underway

3. identified Sep 28, 2026, 11:45 AM UTC

   We are continuing to work on a fix for this issue.

4. identified Sep 28, 2026, 01:10 PM UTC

   We are continuing to work on a fix for this issue.

5. identified Sep 28, 2026, 03:08 PM UTC

   We are continuing to work on a fix for this issue.

6. monitoring Sep 28, 2026, 06:01 PM UTC

   A fix has been implemented and we are monitoring the results.

7. resolved Sep 28, 2026, 08:19 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 16, 2026, 10:08 AM UTC· resolved Sep 16, 2026, 04:57 PM UTC

1. investigating Sep 16, 2026, 10:08 AM UTC

   We are currently investigating this issue.

2. identified Sep 16, 2026, 11:26 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Sep 16, 2026, 12:56 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 16, 2026, 04:57 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 11, 2026, 04:32 PM UTC· resolved Sep 11, 2026, 07:26 PM UTC

1. investigating Sep 11, 2026, 04:32 PM UTC

   The SSO Login Service is currently experiencing downtime and the team is currently looking into it

2. monitoring Sep 11, 2026, 04:37 PM UTC

   The engineering team has resolved the issue and is continuing to monitor the health of the SSO Login Service

3. resolved Sep 11, 2026, 07:26 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 11, 2026, 08:45 AM UTC· resolved Sep 11, 2026, 09:46 AM UTC

1. identified Sep 11, 2026, 08:45 AM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring Sep 11, 2026, 09:27 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 11, 2026, 09:46 AM UTC

   This incident has been resolved.

4. postmortem Sep 28, 2026, 01:14 PM UTC

   # **Root Cause Analysis: Query Latency on September 11, 2026** Region: Azure West Europe Cluster: prod01-eu-west-1 ‌ ‌ **Incident report: query latency in Azure West Europe, September 11, 2026** On the morning of September 11, some customers in Azure West Europe saw slower than usual query responses. Read latency was elevated and there were occasional query timeouts. Performance returned to normal at 08:52 UTC. We're sorry for the disruption. **What happened** On September 10, a single workload ran a sustained volume of deletes. That activity left a large backlog of delete markers on every storage partition in the region. Delete markers are normally cleared by a scheduled delete-compaction process. Until that process ran, each hourly retention pass had to work through the full backlog. Passes that usually take about three minutes were taking 10 to 15 minutes across the cluster. While a partition was in one of these longer passes it responded more slowly, and queries that spanned it were delayed. At 07:55 UTC on September 11, one storage partition entered a self-recovery restart. Its companion replica was in the middle of a retention pass at the time, so for a short period that partition had no healthy replica available to serve queries. This was not the root cause, but the overlap added to the impact on queries touching that partition. Alerts on query queue depth fired at 08:14 and we opened an incident. An operator restarted the companion replica at 08:25. The impaired partition completed its recovery at 08:47. The query queue then drained without further intervention, and by 08:52 it was back to normal and the alerts had cleared. **Timeline** \(UTC\) Time Event Sep 10 - Sustained high delete volume from a single workload Sep 11, 07:55 - Storage partition restarted and entered extended recovery 08:14 - Query queue-depth alerts triggered; incident opened 08:25 - Companion replica restarted 08:47 - Impaired partition completed recovery 08:52 - Query queue returned to normal; alerts cleared **Changes we are making** There are four pieces of work under way. None of them is complete yet, and we won't consider them validated until we have confirmed the effect in production. * Retention enforcement is moving to an off-peak window. Longer passes should then fall outside business-hours query load. * Full compactions will run more frequently, so a delete backlog can clear between scheduled runs instead of accumulating. * Per-partition compaction throughput is being increased. * New alerting and telemetry will tell us whether the changes above are having the intended effect.

Read the full incident report →

Started Sep 07, 2026, 10:11 AM UTC· resolved Sep 07, 2026, 04:03 PM UTC

1. identified Sep 07, 2026, 10:11 AM UTC

   The issue has been identified and a fix is being implemented.

2. identified Sep 07, 2026, 11:13 AM UTC

   We are continuing to work on a fix for this issue.

3. identified Sep 07, 2026, 01:11 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Sep 07, 2026, 02:52 PM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Sep 07, 2026, 03:37 PM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Sep 07, 2026, 04:03 PM UTC

   This incident has been resolved.

7. postmortem Sep 28, 2026, 01:11 PM UTC

   # **Root Cause Analysis: Query Latency on September 7, 2026** Region: Azure West Europe Cluster: prod01-eu-west-1 ## **Summary and impact** On September 7, 2026, some customers in Azure West Europe experienced increased query latency. Query success rates and write operations were not affected. Three storage replicas fell behind after increased delete activity coincided with scheduled retention processing. Recovery required staged restarts and changes to compaction and retention processing. Query latency returned to normal at 15:50 UTC, and the incident was resolved at 16:16 UTC. ## **Cause** A sudden burst of delete requests from one organization coincided with a scheduled retention pass on three storage replicas. The combined load exhausted their compaction capacity, causing them to fall behind in processing incoming data. The replicas did not recover on their own after the delete burst stopped. Each retention pass began immediately after the preceding pass finished, leaving no window to clear the backlog. A steady stream of delete requests from a second organization also contributed to the slow recovery. ## **Recovery actions** The team contacted the organization issuing the delete burst, and the burst stopped at 10:10 UTC. The affected replicas were restarted in stages, and compaction was temporarily paused to prioritize clearing the backlog. The retention schedule was also lengthened to give replicas recovery time between passes. Backlogs subsequently decreased across all affected replicas, and query latency returned to normal. ## **Timeline** All times are UTC. Time Event 09:40 - Delete burst began; storage replicas fell behind. 10:01 - Alerts triggered; incident opened. 10:04 - Cause identified; originating organization contacted; status page published. 10:10 - Delete burst stopped. 10:15 - Staged restart of affected replicas began. 11:14 - First replica fully recovered; the other two continued clearing their backlogs. 12:11 - Compaction was paused on the most-affected replica. 14:29 - Retention schedule was lengthened to allow recovery between passes. 15:29 - Backlogs were decreasing on all affected replicas; status page moved to monitoring. 15:50 - Query latency returned to normal. 16:16 - Incident resolved. ## **Follow-up work and Mitigations** The retention scheduling change was applied during the incident response. Additional work described in the investigation includes: * Continued investigation into why the affected replicas did not self-recover after the load was removed. * Evaluation of scheduling changes to prevent consecutive retention passes while a replica is behind. * Consideration of per-organization delete-rate limits, in addition to existing circuit-breaker controls. The scheduling safeguards and rate limits remain under evaluation; they are not described as completed fixes; however, we continue to look for ways to provide a healthy and stable environment.

Read the full incident report →

Started Aug 25, 2026, 01:49 PM UTC· resolved Aug 25, 2026, 03:41 PM UTC

1. identified Aug 25, 2026, 01:49 PM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring Aug 25, 2026, 02:29 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Aug 25, 2026, 03:41 PM UTC

   This incident has been resolved.

4. postmortem Sep 28, 2026, 01:00 PM UTC

   # **Root Cause Analysis: Query Latency on August 25, 2026** Region: Azure West Europe Cluster: prod01-eu-west-1 ## **Summary and impact** On August 25, 2026, some customers in Azure West Europe experienced increased query latency and query timeouts. Additional query capacity helped reduce the initial queue, but full recovery required a manual restart of a stalled storage partition. Query latency returned to normal at 15:35 UTC. ## **Cause** A surge in query volume triggered scaling that quickly exhausted available resources. Shortly after the initial trigger, a storage partition did not complete startup successfully. Although additional capacity brought the query queue back to a nominal state, customer latency signals showed that recovery was incomplete. We discovered the stalled partition required manual intervention to complete initialization. One half of the partition remained online to serve queries and writes while its companion half was impaired. The underlying reason for the partition startup failure remains under investigation. ## **Recovery actions** The team allocated additional query resources, then manually inspected and restarted the impaired partition after identifying the incomplete recovery. The partition fully recovered at 15:32 UTC, and query latency returned to normal at 15:35 UTC. ## **Timeline** All times are UTC. Time Event 13:13 - Scaling alerts triggered; incident opened. 13:33 - Additional resources were added and consumed. 13:35 - Query queue returned to a nominal state. 14:11 - Impaired partition attempted a self-repair restart. 14:40 - Customer latency signals indicated incomplete recovery; investigation resumed. 15:24 - Impaired partition was manually inspected and restarted. 15:32 - Impaired partition fully recovered. 15:35 - Query latency returned to normal. ## **Future Mitigations** Improvements to the partition management workflow are in progress to strengthen resilience and automate recovery. Investigation into the startup failure continues. These improvements are ongoing work, rather than completed preventive fixes. We continue to strive for a healthy platform for all customers on our cloud infrastructure.

Read the full incident report →

Started May 27, 2026, 03:51 PM UTC· resolved May 27, 2026, 04:58 PM UTC

1. investigating May 27, 2026, 03:51 PM UTC

   We are currently investigating this issue.

2. monitoring May 27, 2026, 04:00 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 27, 2026, 04:58 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 17, 2026, 07:18 PM UTC· resolved Mar 17, 2026, 10:41 PM UTC

1. investigating Mar 17, 2026, 07:18 PM UTC

   We are currently investigating intermittent query failures affecting queries in AWS eu-central-1.

2. monitoring Mar 17, 2026, 08:38 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 17, 2026, 10:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 10, 2026, 05:26 PM UTC· resolved Mar 10, 2026, 08:24 PM UTC

1. investigating Mar 10, 2026, 05:26 PM UTC

   We are currently investigating this issue.

2. identified Mar 10, 2026, 06:40 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 10, 2026, 07:38 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 10, 2026, 08:24 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 06, 2026, 03:05 AM UTC· resolved Mar 06, 2026, 04:34 AM UTC

1. investigating Mar 06, 2026, 03:05 AM UTC

   We are currently investigating elevated ingest failure rates affecting Cloud Serverless in the EU-Central region.

2. resolved Mar 06, 2026, 04:34 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 18, 2026, 11:23 PM UTC· resolved Feb 19, 2026, 12:10 AM UTC

1. investigating Feb 18, 2026, 11:23 PM UTC

   This is to notify that there is an outage of the Cloud Dedicated Management API and Admin UI, but writes/queries are not affected. Our platform team is continuing to investigate the issue.

2. monitoring Feb 18, 2026, 11:49 PM UTC

   The issue has been identified and the fix has been implemented. Cloud Dedicated Management API and Admin UI should be accessible now.

3. resolved Feb 19, 2026, 12:10 AM UTC

   Cloud Dedicated Management API and Admin UI access issue has been resolved.

Read the full incident report →

Started Feb 16, 2026, 04:48 PM UTC· resolved Feb 17, 2026, 05:11 PM UTC

1. investigating Feb 16, 2026, 04:48 PM UTC

   This is to notify that there is an outage of the Cloud Dedicated Management API and Admin UI, but writes/queries are not affected. Our platform team is working to resolve this issue.

2. investigating Feb 16, 2026, 06:12 PM UTC

   This is to notify that there is an outage of the Cloud Dedicated Management API and Admin UI, but writes/queries are not affected. Our platform team is continuing to investigate the issue.

3. identified Feb 16, 2026, 07:58 PM UTC

   The issue has been identified and the fix has been implemented. Cloud Dedicated Management API and Admin UI should be accessible now.

4. identified Feb 16, 2026, 07:58 PM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Feb 16, 2026, 08:49 PM UTC

   Cloud Dedicated Management API and Admin UI are accessible now. We are continuing to monitor.

6. resolved Feb 17, 2026, 05:11 PM UTC

   Cloud Dedicated Management API and Admin UI access issue has been resolved.

Read the full incident report →

Started Feb 12, 2026, 11:26 AM UTC· resolved Feb 12, 2026, 04:31 PM UTC

1. investigating Feb 12, 2026, 11:26 AM UTC

   We are aware of the query errors in Azure West Europe and we are currently investigating it.

2. monitoring Feb 12, 2026, 12:08 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Feb 12, 2026, 02:36 PM UTC

   We are continuing to monitor and fix any sub issues related to the incident.

4. resolved Feb 12, 2026, 04:31 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 10, 2026, 05:37 PM UTC· resolved Mar 10, 2026, 01:19 AM UTC

1. investigating Feb 10, 2026, 05:37 PM UTC

   We are investigating an issue related to queries. Some recent data is not showing up on query responses

2. identified Feb 10, 2026, 06:12 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Feb 10, 2026, 07:21 PM UTC

   We are continuing to work on the fix for this issue.

4. identified Feb 10, 2026, 09:40 PM UTC

   We are continuing to work on the fix for this issue.

5. identified Feb 10, 2026, 10:54 PM UTC

   We are continuing to work on the fix for this issue.

6. identified Feb 11, 2026, 01:37 AM UTC

   Our team has identified the issue and is actively working to restore data for affected users.

7. identified Feb 11, 2026, 04:46 AM UTC

   Our team has identified the issue and is actively working to restore data for affected users.

8. identified Feb 11, 2026, 07:32 AM UTC

   Our team has identified the issue and is actively working to restore data for affected users.

9. identified Feb 11, 2026, 10:11 AM UTC

   Our team is actively working to restore data for affected users.

10. identified Feb 11, 2026, 04:36 PM UTC

    Our team continues to actively work to restore data for all affected users.

11. identified Feb 11, 2026, 06:57 PM UTC

    Our team continues to actively work to restore data for all affected users.

12. identified Feb 12, 2026, 01:29 AM UTC

    Our team continues to actively work to restore data for all affected users.

13. identified Feb 12, 2026, 07:46 AM UTC

    Our team continues to actively work to restore data for all affected users.

14. identified Feb 12, 2026, 10:49 AM UTC

    Our team continues to actively work to restore data for all affected users.

15. identified Feb 13, 2026, 01:43 AM UTC

    Our team continues to actively work to restore data for all affected users.

16. identified Feb 14, 2026, 12:08 AM UTC

    Our team has been working around the clock to restore data availability for all affected users. Work will continue through the weekend with no interruption. We do not yet have a resolution timeline to share, but we are committed to providing updates as soon as they are available until data restoration is complete. We sincerely apologize for the continued impact.

17. identified Feb 18, 2026, 04:18 PM UTC

    We apologize for the ongoing disruption to prod101-us-east-1. We are continuing to restore data from our backups. During the restore, there may occasionally be heightened TTBR.

18. identified Feb 19, 2026, 05:07 PM UTC

    Our team continues to actively work to restore data for all affected users.

19. identified Feb 20, 2026, 11:41 PM UTC

    We apologize for the length of time that it has taken to restore customer data that was inadvertently deleted. The data is all available in our backups, but it is taking us an unacceptably long time to restore it. as we have had to balance the restoration of historical data with service availability of the cluster. After we have resolved this issue, we will be revisiting our backup/restore strategy to improve the performance of cluster-wide restorations. For this particular restoration, we have been restoring the data in two phases (data from February 2-9, and data prior to February 2nd). Most customers should now be able to see all of their data from February 2nd onwards, with the last remaining data from this time range expected to be available by Sunday February 20th. Once we have restored all the data from February 2nd onwards, we will be able to give a more accurate time estimate to restore the remaining older data. Once again, we sincerely apologize for how long it is taking us to restore all of your data. After we have completed the restoration, we will provide a full RCA for this outage.

20. identified Feb 26, 2026, 07:10 PM UTC

    We sincerely apologize for the length of this incident. Restoration of data from February 2 onwards is now substantially complete, and the second phase of restoration, covering data prior to February 2, is underway, and has already been completed for many of our customers. While we continue to restore the remaining historical data, some customers may notice elevated time-to-become-readable (TTBR) on queries. We apologize in advance for the inconvenience that this may cause. We are trying to maintain the responsiveness of the service while also restoring the last of the historical data as quickly as possible. Please contact support with any questions or concerns.

21. resolved Mar 10, 2026, 01:19 AM UTC

    This incident has been resolved

22. postmortem Mar 10, 2026, 01:20 AM UTC

    ## Summary A software change was introduced that was intended to improve data compaction for very large partitions. The change had a bug that, under certain circumstances, caused some data to be dropped during the compaction process. When compaction completed, the new compacted file \(missing a portion of the source data\) replaced the original files. This made some data unavailable until it was restored from backup. ‌ Because the underlying data remained intact in backups, no permanent data loss occurred. After identifying the issue, we rolled back the software change and initiated a full data restoration from backups. While the restoration process is reliable, restoring data at cluster scale while maintaining normal system operations required an extended recovery period. During the restoration process, some customers experienced elevated TTBR and degraded query performance while restoration was underway. ## Cause Data in InfluxDB Cloud 2 is organized into 64 partitions. Each partition is regularly compacted to improve query performance since uncompacted data is slower to query. ‌ In January, we identified that one of the partitions in prod101-us-east-1 \(partition 47\) was no longer compacting successfully. As a result, partially compacted files accumulated in this partition and began impacting performance. To address this, our engineering team designed and implemented an optimization to run multiple compaction jobs to run in parallel. Each job targeted a subset of the data in the partition \(based on the org id prefix\), enabling heavily loaded partitions, such as partition 47, to be compacted more effectively. This code change was reviewed by two additional engineers as part of our standard release process. However, the optimization introduced a bug that, in certain circumstances, caused the compactor to fail when reading some of the source TSM data files. This compactor did not correctly handle this failure and instead silently skipped the TSM files, compacted the rest of the data, and then replaced the original files with the compacted data. This issue was not detected during code review or internal testing, and only surfaced after the change was applied to prod101-us-east-1. ## Recovery Once the issue was identified, we immediately rolled back the compaction change to prevent any further impact. We then began restoring the affected data. ‌ Cloud 2 maintains two separate backup sources. Each cluster is protected by weekly TSM snapshots, and all incoming writes are recorded as snapshots in Kafka logs. Together, these provide a complete record of the data written to the cluster and allow us to fully restore the affected data. ‌ Because the issue only impacted partitions that were compacted while the buggy code was running, not all data in the cluster was affected. However, as a precaution, the recovery process prioritized restoring the most recent data first as it is typically the most critical data for customer operations. Recent data is reconstructed from Kafka logs, which contain a complete record of writes into the cluster. Historical data is then restored from TSM snapshots. ‌ The cluster remained online throughout the recovery process because customers rely on it for real-time workloads. As a result, restoration had to be performed incrementally so recovery activity would not disrupt ongoing ingest and query operations. ‌ Although this approach takes longer than restoring the cluster offline from a full backup, it allows the system to remain available while the recovery progresses. During portions of the recovery, restoration activity resulted in elevated TTBR and slower query performance. ‌ ## Timeline Feb 9 - Deployed compaction change to prod101-us-east-1. Feb 9 - First ticket raised indicating potential data availability issues. Investigation began. Feb 10 - Additional tickets raised. Root cause identified. Feb 10 - Compaction change rolled back. Feb 10 - Data restoration process initiated, starting with the most recent data. Feb 12 - Recent data \(Feb 2-10\) fully restored for a test set of impacted customers. Recent data restoration began for all customers. Feb 17 - Modified restoration process to improve efficiency for historical data recovery. Feb 23 - Recent data \(Feb 2-9\) fully restored for all customers. Feb 23 - Historical data restoration resumed for all customer data from Feb 2 and earlier Feb 25 - Historical data is fully restored for initial test set of customers. Mar 8 - Historical data fully restored for all paying customers. ‌ ## Future mitigation ‌ We recognize the disruption this incident caused and sincerely apologize to customers who were impacted. During portions of the recovery process, some customers experienced degraded query performance while restoration was underway. Reliable access to your data is critical, and we take incidents like this very seriously. ‌ Following this incident, we are implementing several improvements to reduce the risk of similar issues in the future. ‌ We are expanding our testing coverage for changes that affect critical storage components such as the compactor, including additional test scenarios that better reflect production-scale workloads. ‌ We are also enhancing safeguards around the compaction process to ensure failures are detected and handled safely. ‌ In addition, we are reviewing our restoration workflows and operational procedures to reduce the potential for elevated TTBR or query performance degradation during large-scale restoration. ‌ We remain committed to strengthening the resilience and operational safety of the platform.

Read the full incident report →

Started Jan 26, 2026, 10:45 PM UTC· resolved Jan 26, 2026, 10:57 PM UTC

1. investigating Jan 26, 2026, 10:45 PM UTC

   We are experiencing intermittent UI login errors on AWS US-East-1. Queries and writes are not affected. We are addressing the issue.

2. investigating Jan 26, 2026, 10:48 PM UTC

   We are continuing to investigate this issue.

3. resolved Jan 26, 2026, 10:57 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 23, 2026, 03:19 AM UTC· resolved Jan 23, 2026, 07:42 AM UTC

1. monitoring Jan 23, 2026, 03:19 AM UTC

   Our team identified some query failures affecting our US-EAST-1 region for InfluxCloud 2.0 (Serverless/IOx). A fix has been implemented but our team is monitoring closely in case any further issues resurface.

2. investigating Jan 23, 2026, 03:32 AM UTC

   We are continuing to investigate a recurrence of query failures and errors. Our team is working toward a permanent resolution, and we will share further details as they become available.

3. identified Jan 23, 2026, 04:39 AM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Jan 23, 2026, 06:09 AM UTC

   Our team continues to actively monitor for any issues.

5. resolved Jan 23, 2026, 07:42 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 21, 2026, 04:08 PM UTC· resolved Jan 21, 2026, 04:48 PM UTC

1. investigating Jan 21, 2026, 04:08 PM UTC

   We are currently investigating an Authentication issue that may impact access to database resources

2. resolved Jan 21, 2026, 04:48 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 20, 2026, 12:31 PM UTC· resolved Jan 20, 2026, 04:41 PM UTC

1. investigating Jan 20, 2026, 12:31 PM UTC

   We are aware of higher than normal query failures in the eu-central-1 region of AWS and are investigating the issue.

2. monitoring Jan 20, 2026, 01:32 PM UTC

   Queries are back to normal levels in eu-central-1 and we are monitoring the cluster.

3. monitoring Jan 20, 2026, 01:33 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Jan 20, 2026, 04:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 15, 2026, 05:18 PM UTC· resolved Jan 16, 2026, 04:21 PM UTC

1. investigating Jan 15, 2026, 05:18 PM UTC

   We are currently investigating intermittent authentication issues (read/write token authentication is not affected)

2. monitoring Jan 15, 2026, 10:12 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Jan 16, 2026, 12:49 AM UTC

   A fix has been implemented and we are monitoring the results. Please let us know via the support portal if you see any issues logging in to your cloud account.

4. resolved Jan 16, 2026, 04:21 PM UTC

   This incident has been resolved.

5. postmortem Jan 16, 2026, 11:06 PM UTC

   **Summary** On January 15, 2026, user login for Cloud 2 was failing intermittently in the eu-central AWS cluster. The impact was that for some users, they could not login via the web UI. For others, they were able to login but then could not see their resources \(buckets, dashboards etc.\) in the web UI. This was an intermittent issue, not affecting all users, and in some cases, recoverable by retrying. It was also isolated to the web UI and did not impact API-based writes and queries. **Cause of the Incident** The incident was caused by the clock on some of the nodes being slightly behind the master clock. When a user logs in, a JWT session token is signed with a nbf \(not-before\) timestamp based on the signing server's clock. When the request hits a gateway pod, the token is checked and one of the checks is to confirm that the token time is valid. If the gateway pod is one whose node clock is behind the signing node's clock, the JWT is rejected because from that node's perspective, the token's nbf time is in the future. The reason why this problem was intermittent was that a load balancer distributed the authentication requests across pods, and not all the pods had clock-skew. The same token worked fine on some of the nodes \(where the time was correct\) and not on a few nodes \(where the time was incorrect\). **Recovery** We identified the nodes whose clock time had drifted, and drained the affected nodes. When the service was restarted on new nodes with the correct time, the problem was resolved. **Timeline** * 2026-01-15 10:12 UTC - Customers reported errors when navigating to dashboards and tasks in the UI in the eu-central AWS cluster, and engineering began to investigate the issue. * 2026-01-15 14:34 UTC - Additional customers reported the same errors. * 2026-01-15 16:28 UTC - The issue was identified as an intermittent user login issue. * 2026-01-15 21:12 UTC - Further investigation revealed clock skew in some nodes on the cluster. * 2026-01-15 21:41 UTC - The nodes that were showing signs of clock skew were drained. * 2026-01-15 21:45 UTC - Confirmed that authentications were successful. * 2026-01-16 16:11 UTC - An additional node was showing signs of clock skew, so the node was drained and the incident was closed. **Future mitigation** We are still researching why the clocks drifted on these nodes, as these nodes use the AWS-provided time synchronization service, which should have kept the clocks in sync.

Read the full incident report →

Started Jan 15, 2026, 05:15 PM UTC· resolved Jan 15, 2026, 07:29 PM UTC

1. monitoring Jan 15, 2026, 05:15 PM UTC

   We have implemented a fix and are monitoring the cluster.

2. resolved Jan 15, 2026, 07:29 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 12, 2026, 11:32 AM UTC· resolved Jan 12, 2026, 12:10 PM UTC

1. monitoring Jan 12, 2026, 11:32 AM UTC

   A fix has been implemented and we are monitoring the results.

2. resolved Jan 12, 2026, 12:10 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 09, 2026, 10:41 AM UTC· resolved Jan 13, 2026, 03:55 PM UTC

1. investigating Jan 09, 2026, 10:41 AM UTC

   We are currently investigating this issue.

2. investigating Jan 09, 2026, 12:48 PM UTC

   We are continuing to investigate this issue.

3. investigating Jan 09, 2026, 02:10 PM UTC

   We are continuing to investigate the issue

4. investigating Jan 09, 2026, 03:43 PM UTC

   We are continuing to investigate this issue.

5. investigating Jan 09, 2026, 05:16 PM UTC

   Tasks are running as expected but Task Run Reporting is still affected. The investigation continues into this and all other service issues.

6. investigating Jan 09, 2026, 06:59 PM UTC

   We have resolved the issue with regard to task execution, and tasks are running correctly. We are still working on a related issue, where the task execution state is not showing correctly in the UI or APIs. This does not impact the tasks themselves

7. investigating Jan 09, 2026, 10:21 PM UTC

   Task execution is working correctly. We are still fixing a UI/API display issue, which does not affect task behavior.

8. identified Jan 09, 2026, 11:52 PM UTC

   We’ve identified the root cause and are implementing a remediation. Tasks remain operational, and reporting functionality is in the process of recovering.

9. monitoring Jan 10, 2026, 02:43 AM UTC

   We are monitoring the recovery of task visibility in the region and will update here on any change of status.

10. monitoring Jan 10, 2026, 10:19 AM UTC

    Task visibility has been restored to the region and we are continuing to monitor.

11. resolved Jan 13, 2026, 03:55 PM UTC

    This incident has been resolved.

Read the full incident report →

Started Jan 08, 2026, 01:24 PM UTC· resolved Jan 08, 2026, 06:18 PM UTC

1. investigating Jan 08, 2026, 01:24 PM UTC

   We are currently investigating a high number of query errors reported in eu-central-1.

2. monitoring Jan 08, 2026, 01:34 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Jan 08, 2026, 04:39 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Jan 08, 2026, 06:18 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 06, 2026, 09:27 PM UTC· resolved Jan 07, 2026, 05:16 AM UTC

1. investigating Jan 06, 2026, 09:27 PM UTC

   We are currently investigating an increase in query times in the US-EAST-1 and EU-CENTRAL-1 clusters.

2. investigating Jan 06, 2026, 10:37 PM UTC

   We are currently investigating an increase in query times in the US-EAST-1 and EU-CENTRAL-1 clusters. Investigation is ongoing, and we will provide updates as more information becomes available.

3. identified Jan 06, 2026, 10:45 PM UTC

   The root cause of the incident has been identified, and we are monitoring the clusters as they continue to recover.

4. identified Jan 07, 2026, 12:16 AM UTC

   The impacted clusters continue to recover, and we are actively monitoring them.

5. identified Jan 07, 2026, 01:25 AM UTC

   The root cause of the incident has been identified, and we are monitoring the clusters as they continue to recover.

6. monitoring Jan 07, 2026, 02:17 AM UTC

   A fix has been implemented and we are monitoring the results.

7. monitoring Jan 07, 2026, 04:34 AM UTC

   The underlying issue has been resolved, and our team is continuing to monitor the situation to ensure stability.

8. resolved Jan 07, 2026, 05:16 AM UTC

   This incident has been resolved.

Read the full incident report →