Helium experienced a critical incident on July 25, 2023 affecting Helium Packet Router, lasting 6h 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Jul 25, 2023, 11:52 PM UTC
The Helium Packet Routers started experiencing intermittent outage from 2:38PM PT due to misconfigured open file limit, the issue was resolved on 3:43 PT. The team is monitoring the incident as there still seem to be some loose ends. Will continue the update here.
- monitoring Jul 26, 2023, 12:31 AM UTC
Starting 5:19PM PT, the U.S. Packet Routers are offline. The core team is investigating the outage and will continue the update here.
- identified Jul 26, 2023, 12:31 AM UTC
The issue has been identified and a fix is being implemented.
- identified Jul 26, 2023, 12:35 AM UTC
The U.S. Packet Routers have been rebooted. In the meantime, the core team is investigating the outage and will continue the update here.
- monitoring Jul 26, 2023, 01:46 AM UTC
The Packet Routers have been stable for the past hour. The core team will keep monitoring the incident.
- identified Jul 26, 2023, 03:39 AM UTC
Starting 8:18PM PT, the U.S. Packet Routers are offline again. The core team hasn't identified the root cause yet as all server metrics were healthy up til the crash, we are going to reboot and remove a load optimization constraint.
- resolved Jul 26, 2023, 06:12 AM UTC
The Packet Routers have been stable for the 2.5 hours. Marking this incident resolved and will follow up with post mortem.