The Things Industries incident

Webhook failures in TTSC eu1 cluster

Resolved after 2 hours 2 minutes, affecting Europe 1 (eu1), on December 19, 2025.

resolved Dec 19, 2025, 04:53 PM UTC

A significant increase in downlink queue operations within a short time window and this caused internal database connections to accumulate as each request waited for exclusive access to device resources. This connection buildup exhausted the available connection pool, resulting in degraded performance for other operations across the Application Server. Unrelated requests experienced increased latency and timeouts as they competed for the limited available connections. Measures were taken so that this incident do not happen again.

Started
Dec 19, 2025, 02:50 PM UTC
Resolved
Dec 19, 2025, 04:53 PM UTC
Duration
2h 2m
Detected by Pingoru
Dec 19, 2025, 02:50 PM UTC

Affected components

Europe 1 (eu1)

Update timeline

  1. investigating Dec 19, 2025, 02:50 PM UTC

    We are investigating reports about webhook failures in our eu1 cluster.

  2. monitoring Dec 19, 2025, 03:26 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Dec 19, 2025, 04:53 PM UTC

    A significant increase in downlink queue operations within a short time window and this caused internal database connections to accumulate as each request waited for exclusive access to device resources. This connection buildup exhausted the available connection pool, resulting in degraded performance for other operations across the Application Server. Unrelated requests experienced increased latency and timeouts as they competed for the limited available connections. Measures were taken so that this incident do not happen again.

Your product runs on more than The Things Industries

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →