Level365 experienced a major incident on May 29, 2024 affecting Fax Services, lasting 7h 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 29, 2024, 04:21 PM UTC
We are currently investigating faxes not completing. We will provide more updates soon.
- identified May 29, 2024, 04:37 PM UTC
We have identified an issue with the server not responding to fax requests. We are working on applying a fix and will provide more updates soon.
- identified May 29, 2024, 07:11 PM UTC
We are continuing to work on a fix for this issue.
- monitoring May 29, 2024, 09:41 PM UTC
A fix has been implemented and we are monitoring the results. Please note, some faxes may be delayed or require additional attempts as the system processes queued faxes.
- resolved May 29, 2024, 11:58 PM UTC
Faxing services are fully operational and all faxes in the backlog have been processed.
- postmortem Jun 04, 2024, 12:56 PM UTC
A bug was identified in the latest software release which was creating inconsistent connections. A fix was applied to resolve this issue. All fax communication processed normally once the patch was in place. We apologize for any inconvenience this issue may have caused. We are also adding additional infrastructure for increased redundancy of the faxing services to help reduce future issues.