Helium experienced a notice incident on May 9, 2023 affecting Helium Packet Router, lasting 2h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring May 09, 2023, 04:11 PM UTC
The core team is doing a staged rollout for packet routers to optimize GRPC process leaks. We’ll start with a rollout on a few packet routers, monitor for a bit before releasing to all packet routers. The rollout is doing fine in other regions, but causing memory issues in U.S. The core team is upgrading the packet router instances in U.S., the expected downtime is 15-20 minutes.
- monitoring May 09, 2023, 04:28 PM UTC
The U.S. packet routers are back up. We are now evaluating the patch performance on GRPC connection stability in the selected few packet routers. After confirmation, we will rollout to the other packet routers.
- monitoring May 09, 2023, 04:43 PM UTC
Based on the monitoring, the core team is rolling out the GRPC connection stability update to all packet routers.
- resolved May 09, 2023, 06:34 PM UTC
Based on the monitoring, all packet routers have fully stabilized and rollout is complete.