Helium incident

Proof-of-Coverage and Data Transfer degraded for some Hotspots in AU/NZ

Minor Resolved View vendor source →

Helium experienced a minor incident on April 3, 2023 affecting Helium Packet Router and Configuration Service, lasting 10h 20m. The incident has been resolved; the full update timeline is below.

Started
Apr 03, 2023, 07:17 PM UTC
Resolved
Apr 04, 2023, 05:37 AM UTC
Duration
10h 20m
Detected by Pingoru
Apr 03, 2023, 07:17 PM UTC

Affected components

Helium Packet RouterConfiguration Service

Update timeline

  1. investigating Apr 03, 2023, 07:17 PM UTC

    We are investigating an issue where Hotspots in Australia appear to be listening and operating on the wrong frequencies after several Makers have updated their firmware to use the newer gateway-rs firmware. Affected Hotspots will not Beacon, Witness or transfer data correctly.

  2. identified Apr 03, 2023, 08:23 PM UTC

    The Core Developers believe that the issue stems from a configuration in the Configuration Service, which tells Hotspots which region plan to use. A fix to the Configuration Service is being tested.

  3. monitoring Apr 03, 2023, 10:50 PM UTC

    The Core Developers have applied the fix to the Configuration Service. All Hotspots in the AU/NZ region should now be receiving the correct configuration, and if the Hotspot Maker has set up the firmware correctly, the Hotspot should see the correct configuration and adopt it within 30 minutes. We are seeing evidence, however, that some Makers are not prepared to use the information correctly when it is provided. We will update the community as we learn more.

  4. identified Apr 04, 2023, 01:17 AM UTC

    Unfortunately, many Hotspots in Australia and New Zealand do not appear to have the required firmware elements to support the new "AS923_1C" band plan being delivered by the Configuration Service. We will be rolling back the Configuration Service to deliver the older AS923_1 plan over the next several hours. Once this rollback is complete, we will ask the various Hotspot Makers to ensure that they have support for the AS923_1C plan and re-enable it.

  5. monitoring Apr 04, 2023, 02:32 AM UTC

    The Configuration Service has been updated. All AU/NZ Hotspots should be receiving the AS923_1 configuration now. Hotspots make take up to 30 minutes to receive the changes. Thereafter, Proof-of-Coverage Beacons and Data Transfer should resume as normal.

  6. monitoring Apr 04, 2023, 02:32 AM UTC

    We are continuing to monitor for any further issues.

  7. resolved Apr 04, 2023, 05:37 AM UTC

    This incident has been resolved.