The Things Industries incident

Downlinks not being sent in eu1

Minor Resolved View vendor source →

The Things Industries experienced a minor incident on September 8, 2020 affecting Europe 1 (eu1), lasting 1h 42m. The incident has been resolved; the full update timeline is below.

Started
Sep 08, 2020, 07:21 AM UTC
Resolved
Sep 08, 2020, 09:03 AM UTC
Duration
1h 42m
Detected by Pingoru
Sep 08, 2020, 07:21 AM UTC

Affected components

Europe 1 (eu1)

Update timeline

  1. investigating Sep 08, 2020, 07:21 AM UTC

    We are investigating reports that downlinks are not being sent in our eu1 cluster.

  2. monitoring Sep 08, 2020, 08:31 AM UTC

    We've identified the cause and have applied a fix. We're monitoring traffic to confirm.

  3. resolved Sep 08, 2020, 09:03 AM UTC

    After investigation, we found that one of the databases that we use to store state unexpectedly ran out of memory. Upgrading the resources solved the issue.