Some wifi notecards temporarily could not connect to Notehub

Update timeline

1. investigating Jan 17, 2024, 06:40 PM UTC

   We are currently investigating an issue where some devices are unable to connect to Notehub.

2. monitoring Jan 17, 2024, 07:38 PM UTC

   We believe this issue has been addressed and Notehub is operating normally. This incident only impacted a subset of Wifi-connected Notecards and the Notehub API calls for those devices. Cellular-connected notecards were not impacted. The wifi connectivity issues occurred between approximately 1:20pm and 2:20pm US Eastern Time. We continue to monitor all Notehub traffic, but do not expect any further connectivity issues.

3. resolved Jan 18, 2024, 02:53 AM UTC

   Issue was successfully resolved as of 2:20pm Eastern

4. postmortem Jan 18, 2024, 03:42 AM UTC

   Starting at 1:18pm EST, some wifi Notecards could not connect to Notehub, causing a delay in event delivery. All cellular Notecards continued to operate normally. The issue was partially resolved starting at 2:05pm and was fully resolved at 2:23pm. An unexpected networking issue arose during a one-time maintenance procedure. We had tested this procedure in multiple pre-production environments and, the day before, had successfully performed the procedure on several production servers. An unexpected hosting provider issue, which arose during the rollout to the rest of production servers, caused a small number of servers to not respond to events from wifi notecards. Our monitoring detected this issue as soon as it occurred and we were able to prevent any further impact while we addressed the root cause. Because of each Notecard’s resilient event caching architecture, no events were lost, just delayed. We were able to safely complete the maintenance procedure later in the day without impact.