Helium experienced a minor incident on May 23, 2023 affecting Helium Packet Router, lasting 5d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 23, 2023, 04:26 PM UTC
Due to a high amount of packet rate, the data transfer in U.S. and Asia may be experiencing degraded service (e.g. join loop), which likely caused the issue above. The core team is investigating and we will provides updates here. Thanx.
- investigating May 23, 2023, 05:03 PM UTC
Based on preliminary investigations, the issue is related with memory usage on packet routers. The core team is working on a patch and we will provides updates here. Thanx.
- investigating May 23, 2023, 06:03 PM UTC
The core has deployed a hot fix on the packet routers, and our testing devices are now able to join successfully. We will be monitoring the situation along with the community. Thanx.
- investigating May 23, 2023, 09:16 PM UTC
The hot fix has performed well. The core team is beginning a rollout for packet routers, which should fully stabilize in 1 hr. Thanx.
- investigating May 23, 2023, 10:03 PM UTC
The rollout is complete. We will make another announcement here when the packet routers fully stabilize. Thanx.
- monitoring May 24, 2023, 12:01 AM UTC
All packet routers have stabilized except for those in Asia. We will be monitoring the situation and update once they are stabilized as well. Thanx.
- resolved May 28, 2023, 06:38 PM UTC
All packet routers have stabilized including those in Asia. Thanx.