Datacake Outage History
Datacake is up right nowPingoru has recorded 30 incidents for Datacake: 17 in the last 12 months and 6 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 1 | 1 of 720 (0%) |
| May 2026 | 1 | 5 of 744 (1%) |
| June 2026 | 1 | 4 of 720 (1%) |
| July 2026 | 1 | 5 of 744 (1%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 0 | 0 of 720 (0%) |
| October 2026 | 2 | 15 of 744 (2%) |
| November 2026 | 4 | 42 of 193 (22%) |
Incidents, newest first
Major November 3, 2026 Downtime affecting published sites and the editor 1d 4h Docs
Started Nov 03, 2026, 05:16 AM UTC· resolved Nov 04, 2026, 09:33 AM UTC
- investigating Nov 03, 2026, 05:16 AM UTC
We’re currently experiencing downtime affecting published documentation sites and the GitBook editor. Our team is working to restore service, and we’ll share updates here. Sorry for the disruption.
- monitoring Nov 03, 2026, 12:15 PM UTC
On November 2 we detected unauthorized access to parts of GitBook's infrastructure. We contained it quickly and took some services offline as a precaution. Our team is now working to bring GitBook back online. We'll post another update here within the next few hours. Our investigation is still in progress. Once it's complete, we'll confirm whether any customer data was affected. You don't need to do anything right now. If there are steps you need to take after services are restored, we'll share them with you. We'll also publish a full post-incident report when the investigation is complete. We're sorry for the disruption, and thank you for your patience. If you have questions, please contact your account manager.
- monitoring Nov 03, 2026, 12:32 PM UTC
Both the GitBook App and public content should be online. We are continuing to monitor the situation. Further updates as they come.
- monitoring Nov 03, 2026, 02:58 PM UTC
We are aware of some specific issues affecting some customers related to published sites. A few services may still be degraded as we finish restoring everything to full performance. We continue to monitor the situation closely, and will provide updates.
Minor November 3, 2026 RateQueue Host Down Did Not Recover 8h 47m Outbound SMS
Started Nov 03, 2026, 02:14 AM UTC· resolved Nov 03, 2026, 11:01 AM UTC
- investigating Nov 03, 2026, 02:14 AM UTC
We are investigating delays in messages presented, while no messages are lost. We expect to provide another update in 1 hour or as soon as more information becomes available
- identified Nov 03, 2026, 02:50 AM UTC
Our engineering team has identified and mitigated the issue causing delays in messages which were presented. While no messages were lost, we continue to investigate the root cause. We expect to provide another update in 1 hour or as soon as more information becomes available.
- identified Nov 03, 2026, 04:53 AM UTC
Our engineering team has identified and mitigated the issue causing delays in message delivery; while no messages were lost, we continue to investigate the root cause. We expect to provide another update in 2 hours or as soon as more information becomes available.
- identified Nov 03, 2026, 06:55 AM UTC
Our engineering team has identified and mitigated the issue causing delays in messages which were presented, while no messages were lost we continue to investigate the root cause. We expect to provide another update in 4 hour or as soon as more information becomes available.
Major November 2, 2026 GitBook services unavailable 8h 25m Docs
Started Nov 02, 2026, 08:26 PM UTC· resolved Nov 03, 2026, 04:52 AM UTC
- identified Nov 02, 2026, 08:26 PM UTC
We’re currently investigating an issue affecting logging in to the GitBook app via SSO, as well as published sites that use authenticated access. Users may be unable to sign in, and visitors to authenticated sites may be unable to load content. Public sites are not affected and are loading as normal. Our team is actively working on a fix and we’ll share another update as soon as we have more information.
- monitoring Nov 03, 2026, 03:37 AM UTC
The incident has now been mitigated and all published content, editing, and Git services are back up. We are continuing to monitor and will post a detailed post-mortem in the next 24 hours.
- monitoring Nov 03, 2026, 03:40 AM UTC
Both app and public content components are operational.
- resolved Nov 03, 2026, 04:52 AM UTC
This incident should be fully resolved. We are continuing our monitoring on the security incident more recently.
Minor November 2, 2026 Git Sync causing missing page content 5h 13m Docs
Started Nov 02, 2026, 01:56 PM UTC· resolved Nov 02, 2026, 07:09 PM UTC
- investigating Nov 02, 2026, 01:56 PM UTC
We are investigating an issue where Git Sync operations can cause page content to become empty. Our team is working urgently to resolve this.
- monitoring Nov 02, 2026, 03:32 PM UTC
Cause identified as a build tooling change that affected Git Sync. The change has been rolled back, so new Git Sync operations are no longer affected. Recovery of affected spaces started at 15:00 GMT.
- resolved Nov 02, 2026, 07:09 PM UTC
This incident has been resolved. The build change that caused the Git Sync issue was rolled back, and Git Sync operations are now working normally. Affected spaces have been restored. If you still see missing or empty content in any space, please contact our support team at [email protected]: mailto:[email protected] and we will investigate promptly.
Minor October 28, 2026 Link Shortening Redirect Errors on Branded Domains 50m Outbound SMS
Started Oct 28, 2026, 11:44 PM UTC· resolved Oct 29, 2026, 12:35 AM UTC
- investigating Oct 28, 2026, 11:44 PM UTC
We are aware of intermittent HTTP 400 errors affecting redirects for shortened links on some branded short domains. Customers may find that shortened URLs do not redirect when accessed in any browser. Our team is actively investigating and rolling out a fix. We will provide another update as soon as more information becomes available.
- investigating Oct 28, 2026, 11:45 PM UTC
We are aware of intermittent HTTP 400 errors affecting redirects for shortened links on some branded short domains. Customers may find that shortened URLs do not redirect when accessed in any browser. Our team is actively investigating and rolling out a fix. We will provide another update in one hour or as soon as more information becomes available.
- monitoring Oct 29, 2026, 12:07 AM UTC
Our engineers have implemented a fix and are monitoring system performance. We will provide another update in 30 minutes or as soon as more information becomes available.
- resolved Oct 29, 2026, 12:35 AM UTC
Our engineers have monitored the fix and confirmed the issue with shortened links on some branded short domains has been resolved. All services are now operating normally at this time.
Minor October 28, 2026 API operations are slower than usual 14h 32m Docs
Started Oct 28, 2026, 09:15 AM UTC· resolved Oct 28, 2026, 11:48 PM UTC
- identified Oct 28, 2026, 09:15 AM UTC
Our API is responding slower than usual due to a known incident on Cloudflare. Any operations going through the affected region are slower or timing out. We are monitoring the incident closely. Cloudflare incident: https://www.cloudflarestatus.com/incidents/snth5ero95rz
- monitoring Oct 28, 2026, 02:28 PM UTC
We have routed parts of our API connection to other regions to improve stability while this gets fixed by Cloudflare. We continue to monitor the situation as it progresses.
- resolved Oct 28, 2026, 11:48 PM UTC
The incident on Cloudflare is resolved and our monitoring confirms that things are back to normal.
Minor July 25, 2026 Data processing delays 4h 30m GraphQL API, MQTT Server, Outgoing Webhooks
Started Jul 25, 2026, 09:21 AM UTC· resolved Jul 25, 2026, 01:51 PM UTC
- investigating Jul 25, 2026, 09:21 AM UTC
Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly..
- investigating Jul 25, 2026, 09:21 AM UTC
Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly.
- identified Jul 25, 2026, 10:35 AM UTC
We have identified the cause of the delays and implemented corrective measures. The fix has improved processing, but the existing queue backlog is still being cleared. Delays in data ingestion therefore continue at this time..
- identified Jul 25, 2026, 10:35 AM UTC
We have identified the cause of the delays and implemented corrective measures. The fix has improved processing, but the existing queue backlog is still being cleared. Delays in data ingestion therefore continue at this time.
- monitoring Jul 25, 2026, 01:11 PM UTC
We have identified and resolved the root cause of the delays. An internal metrics exporter was placing additional load on the processing queue. We have implemented optimizations and are already seeing a significant performance improvement. The remaining backlog is still being processed, and we will provide another update once the queue has fully cleared. No data has been lost..
- monitoring Jul 25, 2026, 01:11 PM UTC
We have identified and resolved the root cause of the delays. An internal metrics exporter was placing additional load on the processing queue. We have implemented optimizations and are already seeing a significant performance improvement. The remaining backlog is still being processed, and we will provide another update once the queue has fully cleared. No data has been lost.
- resolved Jul 25, 2026, 01:51 PM UTC
The processing queue has now fully cleared, and all system metrics look healthy. No data was lost during the incident. We apologize for the inconvenience and will publish a detailed technical postmortem soon..
- resolved Jul 25, 2026, 01:51 PM UTC
The processing queue has now fully cleared, and all system metrics look healthy. No data was lost during the incident. We apologize for the inconvenience and will publish a detailed technical postmortem soon.
Minor June 15, 2026 Data processing delays 4h 22m Web Application, Outgoing Webhooks
Started Jun 15, 2026, 05:34 AM UTC· resolved Jun 15, 2026, 09:56 AM UTC
- investigating Jun 15, 2026, 05:34 AM UTC
Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly..
- investigating Jun 15, 2026, 05:34 AM UTC
Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly.
- identified Jun 15, 2026, 07:12 AM UTC
The data processing queue has processed and measurements should appear in realtime again. We are still working through a backlog of queued outgoing webhooks..
- identified Jun 15, 2026, 07:12 AM UTC
The data processing queue has processed and measurements should appear in realtime again. We are still working through a backlog of queued outgoing webhooks.
- resolved Jun 15, 2026, 09:56 AM UTC
This incident has been resolved, all pending webhooks have been processed..
- resolved Jun 15, 2026, 09:56 AM UTC
This incident has been resolved, all pending webhooks have been processed.
Minor May 13, 2026 Data processing delays 4h 31m MQTT Server, Outgoing Webhooks
Started May 13, 2026, 01:00 AM UTC· resolved May 13, 2026, 05:31 AM UTC
- investigating May 13, 2026, 01:00 AM UTC
Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly.
- investigating May 13, 2026, 01:00 AM UTC
Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly..
- resolved May 13, 2026, 05:31 AM UTC
This incident has been resolved..
- resolved May 13, 2026, 05:31 AM UTC
This incident has been resolved.
Minor April 28, 2026 Investigating intermittent time series data fetch errors 28m GraphQL API, Web Application, Reports
Started Apr 28, 2026, 11:35 AM UTC· resolved Apr 28, 2026, 12:03 PM UTC
- investigating Apr 28, 2026, 11:35 AM UTC
We are currently investigating sporadic errors when fetching time series data. This may result in incomplete or delayed data being displayed in dashboards and via the API. Our team is actively looking into the root cause and will provide updates as we learn more..
- investigating Apr 28, 2026, 11:35 AM UTC
We are currently investigating sporadic errors when fetching time series data. This may result in incomplete or delayed data being displayed in dashboards and via the API. Our team is actively looking into the root cause and will provide updates as we learn more.
- resolved Apr 28, 2026, 12:03 PM UTC
After investigating the issue, we worked with our database operator to implement a fix. Data fetching has returned to normal and all systems are operating as expected..
- resolved Apr 28, 2026, 12:03 PM UTC
After investigating the issue, we worked with our database operator to implement a fix. Data fetching has returned to normal and all systems are operating as expected.
Major March 17, 2026 API outage 5h 16m Overall
Started Mar 17, 2026, 02:14 PM UTC· resolved Mar 17, 2026, 07:30 PM UTC
- investigating Mar 17, 2026, 02:14 PM UTC
We are currently investigating issues with the availability of our API..
- identified Mar 17, 2026, 02:22 PM UTC
We have identified the issue to be caused by an upstream database. We are continuing to work on a fix for this incident..
- monitoring Mar 17, 2026, 02:39 PM UTC
We identified a number of database queries that got stuck, which impacted overall performance. After clearing those, things are back to normal again. There is still a backlog of data being processed, so you might notice slight delays in data ingestion until the queue has fully caught up. We’re keeping a close eye on it and will share further updates if needed..
- monitoring Mar 17, 2026, 05:28 PM UTC
There is still a backlog of data being processed, which is currently causing ingestion delays of around 30–40 minutes. We’re actively working on improving queue performance to reduce this delay and will continue to keep you updated as things progress..
- resolved Mar 17, 2026, 07:30 PM UTC
All systems have fully recovered. The ingestion backlog has been cleared and incoming data is now being processed in real time again. Dashboards and API responses are up to date. We’ll publish a detailed postmortem shortly with more information on what happened and what we’re doing to prevent this in the future..
- postmortem Mar 17, 2026, 07:39 PM UTC
### Summary During a routine deployment, we introduced a database migration that ended up degrading the performance of our primary database more than expected. This slowed down our ingestion pipeline and caused a growing backlog of incoming measurements. As the backlog increased, it exposed a weakness in how our ingestion workers handle concurrency. Instead of recovering once the database performance stabilized, the system got stuck in a state where workers were effectively slowing each other down. That meant the backlog couldn’t clear itself, and data delays persisted longer than they should have. Throughout the incident, no measurement data was lost. However, dashboards and API responses showed outdated values until the system fully recovered. --- ### Root Cause There were two things at play here. The initial trigger was the database migration, which temporarily made queries slower and reduced ingestion throughput. That part alone would have been manageable. The bigger issue was how our ingestion workers behaved under pressure. They were configured to handle many tasks concurrently using shared database and cache connections. Under normal conditions, this works well and is efficient. But once the backlog built up, tasks started competing for those shared resources. Instead of working through the queue faster, they got in each other’s way, turning a temporary slowdown into a sustained bottleneck. --- ### Resolution We first addressed the database performance issue to remove the initial trigger. After that, we changed how ingestion workers run. Instead of using a shared, thread-based model, we moved to process-based workers where each one has its own dedicated connections. This removed the contention entirely, and the system caught up within minutes. --- ### Follow-up Actions We’re adding better visibility into this kind of situation, especially around queue depth and processing times, so we can react earlier if something similar happens again. We’re also reviewing other parts of the system for similar concurrency patterns that could behave poorly under load. Finally, we’ll introduce an additional staging step for database migrations to catch performance regressions before they reach production. --- If anything was unclear during the incident or you have questions, feel free to reach out..
Major February 19, 2026 API outage 5m Overall
Started Feb 19, 2026, 03:23 AM UTC· resolved Feb 19, 2026, 03:28 AM UTC
- investigating Feb 19, 2026, 03:23 AM UTC
We are currently investigating issues with the availability of our API..
- resolved Feb 19, 2026, 03:28 AM UTC
We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..
Major February 3, 2026 API outage 5m Overall
Started Feb 03, 2026, 03:18 AM UTC· resolved Feb 03, 2026, 03:24 AM UTC
- investigating Feb 03, 2026, 03:18 AM UTC
We are currently investigating issues with the availability of our API..
- resolved Feb 03, 2026, 03:24 AM UTC
We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..
Major February 2, 2026 API outage 10m Overall
Started Feb 02, 2026, 03:20 AM UTC· resolved Feb 02, 2026, 03:31 AM UTC
- investigating Feb 02, 2026, 03:20 AM UTC
We are currently investigating issues with the availability of our API..
- resolved Feb 02, 2026, 03:31 AM UTC
We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..
Major January 30, 2026 API outage 5m Overall
Started Jan 30, 2026, 03:23 AM UTC· resolved Jan 30, 2026, 03:28 AM UTC
- investigating Jan 30, 2026, 03:23 AM UTC
We are currently investigating issues with the availability of our API..
- resolved Jan 30, 2026, 03:28 AM UTC
We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..
Major January 17, 2026 API outage 5m Overall
Started Jan 17, 2026, 03:18 AM UTC· resolved Jan 17, 2026, 03:24 AM UTC
- investigating Jan 17, 2026, 03:18 AM UTC
We are currently investigating issues with the availability of our API..
- resolved Jan 17, 2026, 03:24 AM UTC
We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..
Minor December 8, 2025 Performance degradation 50m Overall
Started Dec 08, 2025, 11:43 AM UTC· resolved Dec 08, 2025, 12:34 PM UTC
- investigating Dec 08, 2025, 11:43 AM UTC
We are investigating increased latencies with the API..
- monitoring Dec 08, 2025, 11:58 AM UTC
We’ve identified the root cause as an issue with one of our database providers. The affected database has recovered, and all services are fully back online. While we continue to monitor the situation, we’re working closely with the vendor to understand what happened and to prevent future occurrences. We excuse any inconvenience caused and appreciate your patience..
- resolved Dec 08, 2025, 12:34 PM UTC
All systems continue to be stable, so we are marking this incident as resolved. Our database vendor has acknowledged the issue as originating on their end and is continuing to investigate the root cause..
- postmortem Dec 09, 2025, 12:51 PM UTC
On December 8, several of our services experienced a brief period of instability due to an issue originating from our database provider. The incident began at 11:42 UTC and services recovered by 11:51 UTC. **What happened** Our vendor identified the root cause as a failure in their backend node replacement service. This failure caused delays in deploying new nodes, which led to performance issues and affected tasks related to DNS updates, including node replacements. The impact was limited to databases whose nodes had recently been replaced or newly created, for example after forking or scheduled maintenance. **Resolution** The vendor has identified and fixed the underlying issue in their node replacement service. Once the affected database recovered, all Datacake services returned to normal operation and have remained stable since. **Next steps** We are continuing to work with the vendor to ensure the issue is fully understood and that safeguards are in place to prevent a recurrence..
Major October 8, 2025 Partial Outage - EU1 Region (The Things Industries / Datacake LNS) 18h 14m
Started Oct 08, 2025, 10:10 AM UTC· resolved Oct 09, 2025, 04:24 AM UTC
- investigating Oct 08, 2025, 10:10 AM UTC
We are currently observing a partial outage impacting the **EU1 region of The Things Industries (TTI)**. This affects both the **Datacake LoRaWAN Network Server (LNS)** and **customers who operate their own TTI instances in the EU1 region**. As a result, some LoRaWAN devices may experience delayed or missing uplinks until the issue is resolved. The incident originates from TTI’s infrastructure. For the latest updates, please refer to their official status page: . We are monitoring the situation closely and will update this post as more information becomes available..
- monitoring Oct 08, 2025, 01:22 PM UTC
**The Things Industries (TTI)** has implemented a fix for the issue affecting the **EU1 region**. Both TTI and Datacake are now monitoring the situation to ensure full recovery of all affected LoRaWAN data flows. We will continue to observe system performance and provide further updates if necessary..
- resolved Oct 09, 2025, 04:24 AM UTC
The incident affecting the EU1 region of The Things Industries (TTI) — and consequently the Datacake LoRaWAN Network Server (LNS) — has been resolved. TTI has confirmed that services in the EU1 region are fully operational again. LoRaWAN data ingestion and related functionality have returned to normal. We will continue to monitor system performance, but no further impact is expected. For reference, see TTI’s incident report: ..
Minor October 6, 2025 Incomplete Data Display for Some Devices 3d 19h
Started Oct 06, 2025, 11:56 AM UTC· resolved Oct 10, 2025, 07:43 AM UTC
- investigating Oct 06, 2025, 11:56 AM UTC
We are currently experiencing an issue where dashboards and other parts of the application may show incomplete historical data for certain devices in specific retention plans. Data from early to mid-September 2025 may not be visible in queries and visualizations for affected devices, though **all data remains safely stored and intact**. We have identified the root cause as a database indexing issue with our external database vendor and are working directly with their support team to resolve it..
- resolved Oct 10, 2025, 07:43 AM UTC
The issue causing incomplete historical data display for certain devices has been resolved. All data is now fully accessible across dashboards and the application. **What Happened:** We identified an issue where historical data from early to mid-September 2025 was not appearing in queries and visualizations for specific devices in certain retention plans. The root cause was traced to a bug in our time-series database software related to Bloom filters on compressed data chunks. These filters were incorrectly marking data as non-existent, causing the database to skip chunks that actually contained the requested data. **Resolution:** Our database vendor Timescale identified and fixed a software bug in their compression filtering logic. The fix has been applied to our database instance, and all queries now return complete and accurate results. **Data Safety:** All data remained safely stored throughout this incident. No data was lost or corrupted - it was simply not being returned in certain filtered queries due to the database software bug. We appreciate Timescale's quick response in identifying and resolving this issue. We apologize for any confusion or inconvenience this may have caused and thank you for your patience..
Major May 5, 2025 API outage 10m
Started May 05, 2025, 12:59 AM UTC· resolved May 05, 2025, 01:09 AM UTC
- investigating May 05, 2025, 12:59 AM UTC
We are currently investigating issues with the availability of our API..
- resolved May 05, 2025, 01:09 AM UTC
We have identified and fixed the underlying issue, which was caused by a database having issues. We have replaced this database with another instance and the service is back online. To ensure such outages do not happen again, we will continue making our infrastructure more robust and fail-safe..
Minor April 14, 2025 Performance degradation 1h 7m
Started Apr 14, 2025, 09:55 AM UTC· resolved Apr 14, 2025, 11:02 AM UTC
- investigating Apr 14, 2025, 09:55 AM UTC
We are investigating sporadic failures when querying historic data. This can affect components such as device list, device history and dashboards..
- resolved Apr 14, 2025, 11:02 AM UTC
All systems are operational again. The issue was traced to a database managed by an external service provider. We are continuing to work with them to identify the root cause..
Minor April 9, 2025 MQTT processing delays 6h 42m
Started Apr 09, 2025, 05:24 AM UTC· resolved Apr 09, 2025, 12:06 PM UTC
- investigating Apr 09, 2025, 05:24 AM UTC
Our MQTT broker is experiencing processing delays. No data has been lost, and the system should be caught up shortly..
- resolved Apr 09, 2025, 12:06 PM UTC
Earlier this morning, we identified an incident where outgoing MQTT messages began queueing up, causing delays in message delivery. Our operations team immediately implemented countermeasures by adjusting infrastructure configurations, which successfully increased queue processing rates. The backlog has now been completely cleared, and after a thorough monitoring period, we can confirm the system is operating normally. All MQTT services have returned to optimal performance levels. We apologize for any inconvenience this may have caused. Our engineering team is actively working on improvements to our MQTT infrastructure to prevent similar incidents in the future..