SuperOffice experienced a major incident on February 7, 2024 affecting Login services and API and Apps, lasting 1d 18h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 07, 2024, 06:32 AM UTC
We are currently investigating Network disruption that is causing intermittent availability issues with the SuperOffice CRM Cloud
- identified Feb 07, 2024, 07:03 AM UTC
The issue has been identified and a fix is being implemented.
- identified Feb 07, 2024, 08:55 AM UTC
We are continuing to work on a fix for this issue.
- identified Feb 07, 2024, 10:37 AM UTC
We are continuing to work on a fix for this issue.
- identified Feb 07, 2024, 01:14 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Feb 07, 2024, 06:32 PM UTC
A fix has been implemented and we are monitoring the results. More information will come after the root cause has been investigated.
- monitoring Feb 08, 2024, 08:49 AM UTC
We are continuing to monitor for any further issues.
- monitoring Feb 08, 2024, 04:34 PM UTC
We are continuing to monitor for any further issues.
- resolved Feb 09, 2024, 12:48 PM UTC
The recent incident has been resolved, and services are back to normal. Our team will conduct a postmortem analysis in the coming days to understand the root cause fully. We'll provide the postmortem report as soon as it's available. Thank you for your patience and support.
- postmortem Feb 13, 2024, 11:22 AM UTC
**Description and consequence of the incident** SuperOffice Online was hit by two different incidents on Wednesday February 7th 2024. There was one isolated incident with the SuperId login service \([id.superoffice.com](http://id.superoffice.com)\), and another incident related capacity of outbound network traffic in the Visma datacenter. The SuperId incident resulted in users struggling to log in. This incident was identified 08:44 and lasted until about 12:00. Indications of the incident related to capacity of outbound network traffic in the datacenter was first observed 05:50. The bottleneck for the of the outbound network traffic was identified at 18:15. A preliminary fix resolving the bottleneck was in place 18:45. **The Root cause** The root cause of the incident related to the Superid login service was a network degradation incident in Azure west Europe. This aligns with Microsoft Incident Tracking ID: 6V3Z-VZZ The root cause of the incident related to capacity of outbound network traffic was caused by an increased amount of open outbound network connections to queue services in Azure. The increase of the amount of connections were greater than anticipated. All outbound network traffic in SuperOffice Online pass through a proxy service, as part of the security measures. This proxy service was not capable of handling this unexpected increase of outbound network connections. **Mitigation** The incident related to SuperId was resolved by scaling out and restarting the service in Azure 9:10. The incident related to capacity of outbound network traffic was resolved using several action: 1\) The services were reconfigured to handle a greater number of connections. 2\) The service was scaled out to handle additional load. 3\) The feature in version 10.3.1 causing an unexpected number of outgoing connections where rolled back in a HotFix of the product.