Level365 experienced a minor incident on December 6, 2023 affecting Secondary Services, lasting 3h 36m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 06, 2023, 04:48 PM UTC
We are currently investigating reports of call connection issues when using the MS Teams Connector.
- monitoring Dec 06, 2023, 06:09 PM UTC
Impacted calls are now completing properly. We will continue to monitor these services for any further issues.
- monitoring Dec 06, 2023, 07:54 PM UTC
We are continuing to monitor for any further issues.
- resolved Dec 06, 2023, 08:25 PM UTC
A vendor Level365 utilizes for the MS Teams connector experienced a connectivity issue and has restarted a caching service to resolve the issue. They are investigating the root cause at this time. We will follow up with any post-mortem findings.
- postmortem Dec 20, 2023, 03:12 PM UTC
We have received a Reason for Outage from our 3rd party vendor which facilitates our MS Teams connection. The outage started on December 6 at 10:10am EST and lasted until 12:25pm EST. During this timeframe, our vendor's affected infrastructure was the US EAST SBC. This SBC became unreachable due to an invalid DNS entry. This caused inbound and outbound calls to fail when trying to come through the US East SBC. This DNS entry was added by mistake during a routine modification to DNS records. This DNS record was then removed to restore service. In the aftermath of this incident, our 3rd party vendor have implemented new procedures that will require review and approval of all DNS changes by a supervisor before they can be implemented. They have also put in place new monitoring and alerting for service affecting DNS records.