Helium experienced a minor incident on May 5, 2023 affecting Helium Packet Router, lasting 3h 39m. The incident has been resolved; the full update timeline is below.
Affected components
Helium Packet Router
Update timeline
- monitoring May 05, 2023, 02:03 AM UTC
The core team is rolling out packet routers across all regions to optimize GRPC process leaks. Data transfer may be temporarily unstable during the rollout. Should stabilize within 30 minutes.
- monitoring May 05, 2023, 04:01 AM UTC
Based on the monitoring, the core team has begun rollback on the packet routers due to high memory usage.
- monitoring May 05, 2023, 04:02 AM UTC
We are continuing to monitor for any further issues.
- monitoring May 05, 2023, 04:33 AM UTC
The core team has been monitoring the rollback for almost 1hr and packet routers have stabilized. We will monitor them for 1 more hour before resolving the incident.
- resolved May 05, 2023, 05:42 AM UTC
Based on the monitoring, all packet routers have fully stabilized.