The Things Industries Outage History

Started Jul 24, 2026, 11:38 AM UTC· resolved Jul 24, 2026, 02:41 PM UTC

1. investigating Jul 24, 2026, 11:38 AM UTC

   We are currently investigating an issue affecting TTIG Pro gateway connectivity in the The Things Stack Cloud NAM1 region. We sincerely apologise for any inconvenience this may cause. Our engineering team is actively investigating the issue and is closely monitoring the situation. We will provide further updates as more information becomes available.

2. identified Jul 24, 2026, 11:47 AM UTC

   About 300 The Things Indoor Gateway Pros that are connected via Cellular in the United States or Canada are offline. This is because of an outage of our Cellular data provider. We are monitoring here: https://status.soracom.io/incidents/388301#activity-id-804909

3. resolved Jul 24, 2026, 02:41 PM UTC

   All affected The Things Indoor Gateway Pros connected via Cellular in the United States and Canada have successfully reconnected and are now online.

Read the full incident report →

Started Jul 09, 2026, 11:15 AM UTC· resolved Jul 09, 2026, 01:23 PM UTC

1. investigating Jul 09, 2026, 11:15 AM UTC

   We are currently investigating an issue affecting TTIG Pro gateway connectivity in The Things Stack Cloud EU2 region due to a certificate renewal issue. As a result, a subset of gateways in EU2 may experience intermittent connectivity disruptions. We sincerely apologize for any inconvenience this may cause. Our engineering team is actively investigating the issue and is closely monitoring the situation. We will provide further updates as more information becomes available.

2. monitoring Jul 09, 2026, 11:41 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jul 09, 2026, 01:23 PM UTC

   This incident has been resolved.

4. postmortem Jul 10, 2026, 01:22 PM UTC

   ## Summary On 9 Jul 2026 , it was reported that some The Things Indoor Gateway Pro \(TTIG Pro\) gateways lost connection to the LNS for some of the tenants in EU2 region. This was triggered by certificate renewal issue. Once the remediation was in place, the affected gateways started reconnecting. ## Impact * Some gateways got disconnected for some of the tenants in EU2 region. * The number of connected gateways kept declining gradually. * Eventually, once the fix was in place, the affected gateways have reconnected and service has recovered without intervention. ## Root Cause The Let's Encrypt R12 certificate serving port `8889` expired on `Jul 8 2026 at 05:54 UTC`. This port is used exclusively by TTIG Pro gateways for their WebSocket connection to the LNS. Certbot had already renewed the certificate before expiry and stored the new YR-signed certificate in the `proxy_tls` secret in AWS Secrets Manager. However, the secret was not synced by the GatewayProxy \(Envoy\) service. As a result, Envoy continued to serve the expired R12 certificate from memory, causing all TTIG Pro TLS handshakes to fail with certificate verification error. The same expired-cert condition was present on AU1 and AS1 \(no TTIG Pro gateways in those regions, so no customer impact\) and was approaching on NAM1 \(cert expiring Jul 14\). ## Resolution The fix was applied by restarting the GatewayProxy ECS task on each affected cluster, which forced Envoy to re-read the `proxy_tls` secret from Secrets Manager on startup and load the already-renewed YR certificate. ## Prevention / Long-term improvements Investigate why the renewed certificate was not synchronized to Envoy and implement a permanent fix to ensure certificate updates are automatically reloaded before certificate expiry. Additionally, improve monitoring and alerting to detect certificate synchronization issues before they can impact customer traffic.

Read the full incident report →

Started May 06, 2026, 10:43 AM UTC· resolved May 06, 2026, 05:30 PM UTC

1. identified May 06, 2026, 10:43 AM UTC

   We have identified an issue causing increased failure rates in AWS IoT application package messages in The Things Stack Cloud NAM1 Cluster. Our engineering team is actively working on a resolution. Impact: Users may experience intermittent disruptions with AWS IoT integration during this time. We apologise for the inconvenience and will post updates here as we make progress.

2. monitoring May 06, 2026, 04:06 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 06, 2026, 05:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 23, 2026, 10:44 AM UTC· resolved Mar 23, 2026, 11:40 AM UTC

1. identified Mar 23, 2026, 10:44 AM UTC

   An update intended for other regions was inadvertently applied to the nam1 region outside of its scheduled maintenance window. As a result, some gateways in nam1 are experiencing connectivity issues. We sincerely apologize for the disruption. Our team is actively investigating and has the situation under close monitoring. We will provide further updates as the investigation progresses.

2. monitoring Mar 23, 2026, 11:10 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 23, 2026, 11:40 AM UTC

   This incident has been resolved.

4. postmortem Mar 23, 2026, 12:53 PM UTC

   ## Summary On March 23, 2026, during a scheduled maintenance window, the Gateway Server \(GS\) component in the NAM1 region was accidentally restarted, causing gateways to disconnect in a similar pattern to the March 3 incident. The development team took the opportunity to roll out a fix that had been planned for the next maintenance window. The fix resolved the reconnection issue and gateways are now reconnecting within several minutes. ## Impact Some gateways got disconnected for some of the tenants in the NAM1 region following an accidental Gateway Server restart during a maintenance window. ## Root Cause The incident was triggered by an accidental restart of the Gateway Server component in NAM1 during a scheduled maintenance window, causing gateways to disconnect in the same pattern observed during the March 3 incident — where simultaneous reconnects under high server load led to premature connection drops due to insufficient timeout and cache configurations. ## Resolution The development team used the opportunity to roll out a fix ahead of its planned release date. The deployed fix resolved the reconnection bottleneck, and affected gateways are now reconnecting within several minutes. No customer action is required. ## Prevention / Action items ### Process improvements Procedures around component restarts during maintenance windows will be reviewed to prevent accidental restarts of production-critical components such as the Gateway Server. ### Infrastructure improvements — already applied The fix rolled out during this incident addresses the reconnection performance issues identified in the March 3 post-mortem. Gateway Server instances in NAM1 are now able to handle mass reconnect scenarios significantly more efficiently, with reconnection times reduced to within several minutes.

Read the full incident report →

Started Mar 03, 2026, 08:08 PM UTC· resolved Mar 05, 2026, 10:16 AM UTC

1. investigating Mar 03, 2026, 08:08 PM UTC

   We are observing gateways disconnects for some of tenants in NAM1, this seems to be triggered by an AWS update procedure which has affected the GatewayServer component. We are currently investigating the issue

2. monitoring Mar 04, 2026, 02:28 AM UTC

   Affected gateways have reconnected and service has recovered without intervention. We are monitoring for any recurrence and continuing to investigate the root cause.

3. resolved Mar 05, 2026, 10:16 AM UTC

   This incident has been resolved.

4. postmortem Mar 05, 2026, 10:17 AM UTC

   ## Summary On Mar 3, 2026 , it was reported that some gateways lost connection to the LNS for some of the tenants in NAM1 region. This was triggered by an AWS update procedure which has affected the Gateway Server component. Although the affected gateways reconnected eventually, it took longer than expected \(8 hours for some tenants\). ## Impact * Some gateways got disconnected for some of the tenants in NAM1 region. * Three Gateway Server instances restarted during the incident, disconnecting a large number of gateways which failed to immediately reconnect to remaining active instances. * The number of connected gateways kept declining gradually. * Eventually, the affected gateways have reconnected and service has recovered without intervention. ## Root Cause The incident was triggered by an AWS infrastructure event \(task retirement\), which caused several Gateway Server instances in NAM1 to undergo a rolling restart. As instances restarted one by one, gateways began disconnecting gradually. Since the restart was rolling rather than simultaneous, some gateways maintained their connection to instances that remained active throughout the event. The root cause of the prolonged recovery, however, was a short connection timeout configured on some gateways. With a large number of gateways attempting to reconnect simultaneously, the Gateway Server was operating under unusually high load — and the short timeout was insufficient under these conditions, causing connections to close prematurely before they could be fully established. This cycle repeated until the restarted instances completed their post-restart operations — at which point server load normalised and Gateway Server caches became available, significantly speeding up the connection process for the remaining disconnected gateways until service was fully restored. In short: the AWS infrastructure event triggered the affected gateways disconnects, but the timeout misconfiguration is what made the recovery take up to 8 hours. ## Resolution There was no manual intervention to resolve this incident. The affected gateways reconnected automatically after the downtime. ## Prevention / Long-term improvements ### Proactive outreach to affected tenant owners regarding gateway connection timeout A minimum 60-second timeout is necessary for reliable connection establishment under high server load conditions. We will be reaching out to affected tenant owners, recommending that the `TC_TIMEOUT` setting in their Basic Station configuration is set to at least the default value of `60s`. This change will help prevent premature connection drops during periods of elevated reconnect activity. ### Documentation improvements Existing documentation will be improved to specifically address recommendations for longer `TC_TIMEOUT` setting of the Basic Station configuration. ### Infrastructure improvements Our Cloud infrastructure configuration will be improved to reduce and accommodate higher instance load post-restart.

Read the full incident report →

Started Jan 09, 2026, 03:16 PM UTC· resolved Jan 09, 2026, 04:13 PM UTC

1. identified Jan 09, 2026, 03:16 PM UTC

   We're conducting emergency maintenance on the infrastructure running The Things Gateway Controller. This causes connectivity issues with The Things Indoor Gateway Pro.

2. monitoring Jan 09, 2026, 04:07 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 09, 2026, 04:13 PM UTC

   The incident has been resolved. The Things Indoor Gateway Pro gateways are reconnecting. This may take some additional time due to reconnect backoffs.

Read the full incident report →

Started Dec 19, 2025, 02:50 PM UTC· resolved Dec 19, 2025, 04:53 PM UTC

1. investigating Dec 19, 2025, 02:50 PM UTC

   We are investigating reports about webhook failures in our eu1 cluster.

2. monitoring Dec 19, 2025, 03:26 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Dec 19, 2025, 04:53 PM UTC

   A significant increase in downlink queue operations within a short time window and this caused internal database connections to accumulate as each request waited for exclusive access to device resources. This connection buildup exhausted the available connection pool, resulting in degraded performance for other operations across the Application Server. Unrelated requests experienced increased latency and timeouts as they competed for the limited available connections. Measures were taken so that this incident do not happen again.

Read the full incident report →

Started Oct 16, 2025, 11:41 AM UTC· resolved Oct 16, 2025, 01:12 PM UTC

1. investigating Oct 16, 2025, 11:41 AM UTC

   We are currently investigating a connectivity issue with TTIG Pro Gateways.

2. identified Oct 16, 2025, 11:42 AM UTC

   The issue has been identified, and a fix is being implemented.

3. monitoring Oct 16, 2025, 12:44 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 16, 2025, 01:12 PM UTC

   This incident has been resolved.

5. postmortem Oct 16, 2025, 01:14 PM UTC

   The Things Gateway Controller was down intermittently for a total duration of 56 minutes between 2025/10/16 13:12 UTC and 14:42 UTC. This also caused some gateways to disconnect from The Things Stack. The issue started after deploying an update to The Things Gateway Controller. When the newly updated services started, all gateways try to reconnect. Due to the amount of The Things Indoor Gateway Pro gateways in the field, the number of database connections opened to connect the gateways simultaneously exceeded the maximum number of allowed database connections. For connecting one gateway, there are several parallel, independent tasks that execute database queries. Because not enough database connections were available, almost all gateways were suffering because one or more tasks could not complete, causing the gateway to disconnect. This happened in a loop. We temporarily increased the number of allowed database connections so that all gateways could connect. In the meantime, we work on a permanent fix to cap the maximum number of database connections and put proper wait and back-off behavior in place.

Read the full incident report →

Started Oct 08, 2025, 10:12 AM UTC· resolved Oct 09, 2025, 02:21 AM UTC

1. investigating Oct 08, 2025, 10:12 AM UTC

   We are investigating reports about webhook failures in the TTSC eu1 cluster.

2. monitoring Oct 08, 2025, 10:57 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 09, 2025, 02:21 AM UTC

   This incident has been resolved.

4. postmortem Oct 13, 2025, 01:57 PM UTC

   On October 7-8, 2025, the webhook ingestion service in our eu1 cluster experienced a degradation that led to an increase in processing errors. This resulted in delayed data delivery due to processing retries and, in a small number of cases, data loss after too many retries occurred. ‌ The issue was caused by a lock that, under certain conditions, could be held indefinitely because it was configured without an automatic expiration. This created a chain reaction where subsequent processes would wait for the lock while holding their Redis connections open. The accumulation of these waiting processes likely caused a rapid increase in Redis connections, which in turn impacted the performance of dependent services. ‌ Service was restored after deploying an update that adjusted the locking logic. To enhance future reliability, our follow-up actions include a review of similar code patterns and performing more heavy load tests on the webhook related services.

Read the full incident report →

Started Oct 07, 2025, 05:51 PM UTC· resolved Oct 07, 2025, 07:12 PM UTC

1. investigating Oct 07, 2025, 05:51 PM UTC

   We are investigating reports about webhook failures in our eu1 cluster.

2. monitoring Oct 07, 2025, 06:33 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 07, 2025, 07:12 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 19, 2025, 05:36 AM UTC· resolved Jul 21, 2025, 06:58 AM UTC

1. investigating Jul 19, 2025, 05:36 AM UTC

   We are currently investigating this issue.

2. identified Jul 19, 2025, 06:46 AM UTC

   We are deploying a fix.

3. monitoring Jul 19, 2025, 10:43 AM UTC

   A fix has been implemented and we are monitoring the results.

4. investigating Jul 19, 2025, 02:12 PM UTC

   We have increased the resources for the Proxy services to accommodate the spike in traffic. However, the issue still persists. Our team is continuing to investigate the root cause.

5. monitoring Jul 20, 2025, 01:25 PM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Jul 21, 2025, 06:58 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 19, 2025, 12:23 PM UTC· resolved Jun 20, 2025, 12:57 PM UTC

1. investigating Jun 19, 2025, 12:23 PM UTC

   We are currently investigating this issue.

2. identified Jun 20, 2025, 10:52 AM UTC

   We are deploying a fix.

3. monitoring Jun 20, 2025, 11:22 AM UTC

   We have deployed a fix and we are monitoring.

4. resolved Jun 20, 2025, 12:57 PM UTC

   We've received confirmation that the problem has been fixed. If issues persist, please reach out to [email protected]

Read the full incident report →

Started May 14, 2025, 09:05 AM UTC· resolved May 14, 2025, 01:17 PM UTC

1. investigating May 14, 2025, 09:05 AM UTC

   We are currently investigating an issue affecting a subset of customers who are seeing a blank page when attempting to log in to the Console. This appears to impact only certain clients. We will provide more information as we continue to investigate.

2. identified May 14, 2025, 10:04 AM UTC

   We have identified the cause of the issue resulting in a blank screen during login for some users. A fix is being prepared and will be deployed shortly.

3. monitoring May 14, 2025, 10:15 AM UTC

   A fix has been deployed to address the issue affecting the login screen. Affected users should now be able to access the Console normally. We are monitoring the situation to ensure full stability.

4. resolved May 14, 2025, 01:17 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 24, 2025, 08:37 AM UTC· resolved Apr 24, 2025, 08:53 AM UTC

1. identified Apr 24, 2025, 08:37 AM UTC

   The Things Indoor Gateway Pros in United States and Canada that are (only) using the cellular backhaul may suffer from cellular connectivity issues. The underlying issue started at 05:47 UTC. The cause is an incident at our MVNO Soracom. See https://status.soracom.io/incidents/287968 for more up-to-date information.

2. monitoring Apr 24, 2025, 08:52 AM UTC

   Most gateways are coming back online. We're monitoring closely. For avoid cellular connectivity issues, make sure to connect the gateway also via WiFi or ethernet. See https://www.thethingsindustries.com/docs/hardware/gateways/models/thethingsindoorgatewaypro/#connection-settings for more information.

3. resolved Apr 24, 2025, 08:53 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 23, 2025, 10:00 PM UTC· resolved Apr 24, 2025, 11:00 AM UTC

1. resolved Apr 24, 2025, 11:01 AM UTC

   We identified an issue where users were shown a notification stating that the license for The Things Stack Cloud was expiring soon. This message was incorrect and not related to any active subscriptions held by the users. The notification should not have been displayed and has since been resolved.

Read the full incident report →