Datacake Outage History

Started Jul 25, 2026, 09:21 AM UTC· resolved Jul 25, 2026, 01:51 PM UTC

1. investigating Jul 25, 2026, 09:21 AM UTC

   Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly.

2. identified Jul 25, 2026, 10:35 AM UTC

   We have identified the cause of the delays and implemented corrective measures. The fix has improved processing, but the existing queue backlog is still being cleared. Delays in data ingestion therefore continue at this time.

3. monitoring Jul 25, 2026, 01:11 PM UTC

   We have identified and resolved the root cause of the delays. An internal metrics exporter was placing additional load on the processing queue. We have implemented optimizations and are already seeing a significant performance improvement. The remaining backlog is still being processed, and we will provide another update once the queue has fully cleared. No data has been lost.

4. resolved Jul 25, 2026, 01:51 PM UTC

   The processing queue has now fully cleared, and all system metrics look healthy. No data was lost during the incident. We apologize for the inconvenience and will publish a detailed technical postmortem soon.

Read the full incident report →

Started Apr 28, 2026, 11:35 AM UTC· resolved Apr 28, 2026, 12:03 PM UTC

1. investigating Apr 28, 2026, 11:35 AM UTC

   We are currently investigating sporadic errors when fetching time series data. This may result in incomplete or delayed data being displayed in dashboards and via the API. Our team is actively looking into the root cause and will provide updates as we learn more.

2. resolved Apr 28, 2026, 12:03 PM UTC

   After investigating the issue, we worked with our database operator to implement a fix. Data fetching has returned to normal and all systems are operating as expected.

Read the full incident report →

Started Mar 17, 2026, 02:14 PM UTC· resolved Mar 17, 2026, 07:30 PM UTC

1. investigating Mar 17, 2026, 02:14 PM UTC

   We are currently investigating issues with the availability of our API..

2. identified Mar 17, 2026, 02:22 PM UTC

   We have identified the issue to be caused by an upstream database. We are continuing to work on a fix for this incident..

3. monitoring Mar 17, 2026, 02:39 PM UTC

   We identified a number of database queries that got stuck, which impacted overall performance. After clearing those, things are back to normal again. There is still a backlog of data being processed, so you might notice slight delays in data ingestion until the queue has fully caught up. We’re keeping a close eye on it and will share further updates if needed..

4. monitoring Mar 17, 2026, 05:28 PM UTC

   There is still a backlog of data being processed, which is currently causing ingestion delays of around 30–40 minutes. We’re actively working on improving queue performance to reduce this delay and will continue to keep you updated as things progress..

5. resolved Mar 17, 2026, 07:30 PM UTC

   All systems have fully recovered. The ingestion backlog has been cleared and incoming data is now being processed in real time again. Dashboards and API responses are up to date. We’ll publish a detailed postmortem shortly with more information on what happened and what we’re doing to prevent this in the future..

6. monitoring Mar 17, 2026, 07:39 PM UTC

   ### Summary During a routine deployment, we introduced a database migration that ended up degrading the performance of our primary database more than expected. This slowed down our ingestion pipeline and caused a growing backlog of incoming measurements. As the backlog increased, it exposed a weakness in how our ingestion workers handle concurrency. Instead of recovering once the database performance stabilized, the system got stuck in a state where workers were effectively slowing each other down. That meant the backlog couldn’t clear itself, and data delays persisted longer than they should have. Throughout the incident, no measurement data was lost. However, dashboards and API responses showed outdated values until the system fully recovered. --- ### Root Cause There were two things at play here. The initial trigger was the database migration, which temporarily made queries slower and reduced ingestion throughput. That part alone would have been manageable. The bigger issue was how our ingestion workers behaved under pressure. They were configured to handle many tasks concurrently using shared database and cache connections. Under normal conditions, this works well and is efficient. But once the backlog built up, tasks started competing for those shared resources. Instead of working through the queue faster, they got in each other’s way, turning a temporary slowdown into a sustained bottleneck. --- ### Resolution We first addressed the database performance issue to remove the initial trigger. After that, we changed how ingestion workers run. Instead of using a shared, thread-based model, we moved to process-based workers where each one has its own dedicated connections. This removed the contention entirely, and the system caught up within minutes. --- ### Follow-up Actions We’re adding better visibility into this kind of situation, especially around queue depth and processing times, so we can react earlier if something similar happens again. We’re also reviewing other parts of the system for similar concurrency patterns that could behave poorly under load. Finally, we’ll introduce an additional staging step for database migrations to catch performance regressions before they reach production. --- If anything was unclear during the incident or you have questions, feel free to reach out..

Read the full incident report →

Started Dec 08, 2025, 11:43 AM UTC· resolved Dec 08, 2025, 12:34 PM UTC

1. investigating Dec 08, 2025, 11:43 AM UTC

   We are investigating increased latencies with the API..

2. monitoring Dec 08, 2025, 11:58 AM UTC

   We’ve identified the root cause as an issue with one of our database providers. The affected database has recovered, and all services are fully back online. While we continue to monitor the situation, we’re working closely with the vendor to understand what happened and to prevent future occurrences. We excuse any inconvenience caused and appreciate your patience..

3. resolved Dec 08, 2025, 12:34 PM UTC

   All systems continue to be stable, so we are marking this incident as resolved. Our database vendor has acknowledged the issue as originating on their end and is continuing to investigate the root cause..

4. monitoring Dec 09, 2025, 12:51 PM UTC

   On December 8, several of our services experienced a brief period of instability due to an issue originating from our database provider. The incident began at 11:42 UTC and services recovered by 11:51 UTC. **What happened** Our vendor identified the root cause as a failure in their backend node replacement service. This failure caused delays in deploying new nodes, which led to performance issues and affected tasks related to DNS updates, including node replacements. The impact was limited to databases whose nodes had recently been replaced or newly created, for example after forking or scheduled maintenance. **Resolution** The vendor has identified and fixed the underlying issue in their node replacement service. Once the affected database recovered, all Datacake services returned to normal operation and have remained stable since. **Next steps** We are continuing to work with the vendor to ensure the issue is fully understood and that safeguards are in place to prevent a recurrence..

Read the full incident report →

Started Oct 08, 2025, 10:10 AM UTC· resolved Oct 09, 2025, 04:24 AM UTC

1. investigating Oct 08, 2025, 10:10 AM UTC

   We are currently observing a partial outage impacting the **EU1 region of The Things Industries (TTI)**. This affects both the **Datacake LoRaWAN Network Server (LNS)** and **customers who operate their own TTI instances in the EU1 region**. As a result, some LoRaWAN devices may experience delayed or missing uplinks until the issue is resolved. The incident originates from TTI’s infrastructure. For the latest updates, please refer to their official status page: . We are monitoring the situation closely and will update this post as more information becomes available..

2. monitoring Oct 08, 2025, 01:22 PM UTC

   **The Things Industries (TTI)** has implemented a fix for the issue affecting the **EU1 region**. Both TTI and Datacake are now monitoring the situation to ensure full recovery of all affected LoRaWAN data flows. We will continue to observe system performance and provide further updates if necessary..

3. resolved Oct 09, 2025, 04:24 AM UTC

   The incident affecting the EU1 region of The Things Industries (TTI) — and consequently the Datacake LoRaWAN Network Server (LNS) — has been resolved. TTI has confirmed that services in the EU1 region are fully operational again. LoRaWAN data ingestion and related functionality have returned to normal. We will continue to monitor system performance, but no further impact is expected. For reference, see TTI’s incident report: ..

Read the full incident report →

Started Oct 06, 2025, 11:56 AM UTC· resolved Oct 10, 2025, 07:43 AM UTC

1. investigating Oct 06, 2025, 11:56 AM UTC

   We are currently experiencing an issue where dashboards and other parts of the application may show incomplete historical data for certain devices in specific retention plans. Data from early to mid-September 2025 may not be visible in queries and visualizations for affected devices, though **all data remains safely stored and intact**. We have identified the root cause as a database indexing issue with our external database vendor and are working directly with their support team to resolve it..

2. resolved Oct 10, 2025, 07:43 AM UTC

   The issue causing incomplete historical data display for certain devices has been resolved. All data is now fully accessible across dashboards and the application. **What Happened:** We identified an issue where historical data from early to mid-September 2025 was not appearing in queries and visualizations for specific devices in certain retention plans. The root cause was traced to a bug in our time-series database software related to Bloom filters on compressed data chunks. These filters were incorrectly marking data as non-existent, causing the database to skip chunks that actually contained the requested data. **Resolution:** Our database vendor Timescale identified and fixed a software bug in their compression filtering logic. The fix has been applied to our database instance, and all queries now return complete and accurate results. **Data Safety:** All data remained safely stored throughout this incident. No data was lost or corrupted - it was simply not being returned in certain filtered queries due to the database software bug. We appreciate Timescale's quick response in identifying and resolving this issue. We apologize for any confusion or inconvenience this may have caused and thank you for your patience..

Read the full incident report →

Started Apr 09, 2025, 05:24 AM UTC· resolved Apr 09, 2025, 12:06 PM UTC

1. investigating Apr 09, 2025, 05:24 AM UTC

   Our MQTT broker is experiencing processing delays. No data has been lost, and the system should be caught up shortly..

2. resolved Apr 09, 2025, 12:06 PM UTC

   Earlier this morning, we identified an incident where outgoing MQTT messages began queueing up, causing delays in message delivery. Our operations team immediately implemented countermeasures by adjusting infrastructure configurations, which successfully increased queue processing rates. The backlog has now been completely cleared, and after a thorough monitoring period, we can confirm the system is operating normally. All MQTT services have returned to optimal performance levels. We apologize for any inconvenience this may have caused. Our engineering team is actively working on improvements to our MQTT infrastructure to prevent similar incidents in the future..

Read the full incident report →