Expel experienced a minor incident on January 27, 2025 affecting Workbench global network connectivity and Workbench login, lasting 4h 48m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 27, 2025, 05:20 PM UTC
At 11:49am EST, we began experiencing a problem with our authorization service that is preventing users from using workbench due to added latency. We will provide an update by 1pm EST.
- monitoring Jan 27, 2025, 05:46 PM UTC
We have implemented a fix and believe the incident to be resolved. We are monitoring the health of the infrastructure and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems. Please expect another update by 1:30pm EST.
- monitoring Jan 27, 2025, 06:30 PM UTC
We have implemented a fix and believe the incident to be resolved. We are continuing to monitor the health of the infrastructure and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems. We will issue another update by 2pm EST.
- monitoring Jan 27, 2025, 07:12 PM UTC
We have identified a further issue and are rolling out an additional fix that we believe will resolve the incident. We are continuing to monitor the health of the infrastructure and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems. Please expect another update by 2:30pm EST.
- monitoring Jan 27, 2025, 07:31 PM UTC
We have identified a further issue and have rolled out an additional fix that we believe will resolve the incident. We are continuing to monitor the health of the infrastructure and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems. Please expect another update by 2:30pm EST.
- monitoring Jan 27, 2025, 08:40 PM UTC
Our change did not resolve the incident. We are rolling back to an earlier version of workbench. We are continuing to monitor the health of the infrastructure. Please let us know if you are experiencing any problems. Please expect another update by 4:00pm EST.
- monitoring Jan 27, 2025, 08:57 PM UTC
We have rolled back to an earlier version of workbench. We are continuing to monitor the health of the infrastructure. Please let us know if you are experiencing any problems. Please expect another update by 5:00pm EST.
- resolved Jan 27, 2025, 10:08 PM UTC
The incident has been resolved, and all systems are once again fully available. No customer data was lost.