Esper.io experienced a minor incident on April 16, 2024 affecting Esper Systems, lasting 14h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 16, 2024, 07:31 PM UTC
Our team has identified a possible cause of the partial service outage which impacts device provisioning and commands. We’re working to resolve it. Which services are affected? Device Provisioning: Yes. Limited set of devices. Commands: Yes. Commands are not reaching a limted set of devices. API: All APIs are working correctly. No impact Console: Console is accessible to login and other operations. No impact. Customer Devices: Devices are working fine
- identified Apr 16, 2024, 08:34 PM UTC
The issue has been identified and we're working on a fix. Commands continue to process slowly and stay in Queued state till the fix is deployed.
- monitoring Apr 16, 2024, 11:05 PM UTC
We've resolved the issue by rolling out a fix and verified that new commands are being processed by the devices correctly. Commands that were queued during this incident will be processed in a few hours. More fixes are being rollout out immediately to speed up message processing. We'll continue to monitor the services for next few hours before closing the incident.
- monitoring Apr 17, 2024, 06:03 AM UTC
Specific commands are continuing to process slowly. We're monitoring and anticipate the lag to subside in a few hours.
- monitoring Apr 17, 2024, 08:43 AM UTC
Command processing is back to normal now and should no longer see delays. We're continuing to monitor.
- resolved Apr 17, 2024, 09:54 AM UTC
This incident has been resolved.