Nebula experienced a minor incident on October 1, 2024 affecting Core Network and Mobile Applications and 1 more component, lasting 8h 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 01, 2024, 12:06 PM UTC
Our engineering team is currently investigating reports by a small number of customers of delay or no dial tone on outbound calling, and calls still ringing after calls being answered elsewhere. We are working hard to identify the issue and will provide updates as they become available
- identified Oct 01, 2024, 02:40 PM UTC
Our engineers continue to investigate the issue and have created a fix for all related symptoms, including those affecting desk phone features such as BLF keys. We plan to roll this out during off-peak hours to avoid impacting our wider user base. We will provide our next update within the next hour.
- identified Oct 01, 2024, 03:40 PM UTC
The fix is still scheduled for implementation after working hours and we expect all services to be fully restored after successful deployment this evening.
- resolved Oct 01, 2024, 08:38 PM UTC
An update has now been implemented and after a period of successful monitoring our engineers are confirming this incident as resolved.
- postmortem Oct 04, 2024, 01:58 PM UTC
At 14:30 on 1st October 2024 we became aware of an issue with some customers experiencing a delay of up to 10secs when initiating a call. After investigation we found that it was caused by an internal process essential to outbound call authorisation, which caused a delay in connection and was exponentially proportional to the number of users held by each partner - as a result, only partners with a moderate-to-large customer base would have experienced significant delays ,and the impact was unnoticeable for most end users across our estate. Following successful short-term adjustment to these processes, we have now installed a patch which greatly decreases the complexity and increases the speed of the call initiation process so that this problem will not occur again in the future.