The Things Industries incident

Degraded Downlink performance

Minor Resolved View vendor source →

The Things Industries experienced a minor incident on November 18, 2020 affecting Europe 1 (eu1) and North America 1 (nam1) and 1 more component, lasting 16h 47m. The incident has been resolved; the full update timeline is below.

Started
Nov 18, 2020, 03:42 PM UTC
Resolved
Nov 19, 2020, 08:30 AM UTC
Duration
16h 47m
Detected by Pingoru
Nov 18, 2020, 03:42 PM UTC

Affected components

Europe 1 (eu1)North America 1 (nam1)Australia 1 (au1)

Update timeline

  1. identified Nov 18, 2020, 03:42 PM UTC

    We experience degraded performance for downlink. We've identified the cause and will apply a fix shortly.

  2. monitoring Nov 18, 2020, 08:56 PM UTC

    We identified the cause of the issue to be unintended throttling of uplink messages in the Network Server which caused a queueing of uplinks and disrupting downlink flow. We've applied a fix that ensures that this throttling is mitigated and traffic reaches the application server as intended. We're continuing to monitor the fix.

  3. resolved Nov 19, 2020, 08:30 AM UTC

    We monitored the fixed infrastructure overnight and all metrics are back and healthy.