Exclaimer experienced a major incident on May 1, 2025 affecting AU: Data: User Synchronization and AU: UI: New Customer Onboarding, lasting 16h 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 01, 2025, 08:01 AM UTC
We are currently investigating reports of an issue when attempting to run the Exclaimer Cloud Setup wizard for customers using Microsoft 365 subscriptions. This is not impacting already configured Exclaimer Cloud accounts and signatures are being applied as expected
- investigating May 01, 2025, 08:02 AM UTC
We are continuing to investigate this issue.
- identified May 01, 2025, 08:14 AM UTC
Engineers have identified another area of impact and have been able to identify a possible cause for the issue. The team are currently working through remediation processes
- monitoring May 01, 2025, 09:14 AM UTC
A fix has now been applied to the service to restore full functionality. Our Engineering team are continuing to monitor the services for any further issues. If you are continuing to experience issues, please look to log out and log back into the Exclaimer portal, or raise a ticket with our support team.
- resolved May 02, 2025, 12:13 AM UTC
After continued monitoring by our Engineering team, we can confirm this incident has now been resolved. If you are continuing to experience issues within Exclaimer Cloud, please reach out to our Support team who will be able to assist in investigating the issue
- postmortem May 16, 2025, 04:38 PM UTC
Issue: Customers unable to complete setup of Exclaimer or synchronize User Details manually. Incident Length: 17 hours and 21 minutes Incident Date: 30/04/2025, UTC 16:30 – 09:51 01/05/2025\] Incident Status: Resolved **Summary** A subset of customers that attempted run the setup for Exclaimer experienced an error advising that ‘An unexpected error has happened.’ within the portal causing the Configuration to fail. Existing subscriptions that were configured, and emails that were being routed would not have been impacted by this issue. **Root Cause** As part of standard maintenance of the Exclaimer Cloud service within all regions, the Secret keys used to verify the certifications used by Enterprise Applications for Exclaimer Cloud did not return the expected values to the configuration wizard. This resulted in subscriptions not being able to authenticate the application consent grant requested by Exclaimer Cloud’s setup. **Mitigation** To address the issue, the Secret Keys that are applied to the applications were reapplied to the Exclaimer Cloud applications in all regions, ensuring that the setup wizard was able to authenticate as expected. Additional reviews and investigation have been undertaken internally to determine why the cause of the incident was not detected earlier and ensure that proper alerting and monitoring has been configured to prevent similar issues with other Enterprise applications from occurring. **Incident Timeline** 01/05/2025 08:29 - Reports of configuration errors when attempting to configure Exclaimer Cloud from the Customer Support team prompted an investigation to begin with the Engineering team. 08:32 - A probable cause of the failure based on the failure reason being provided was determined and Engineers began to work on a resolution. 08:53 - A new set of Secret keys was generated and applied to the Enterprise Applications for AU region. 09:14 - Engineering confirmed the Secret keys had been applied and work began to deploy the Applications 09:22 - Engineers identified that the keys within the DE region were also impacted and began work to address this. 09:29 – Engineers confirmed that the Secret keys for AU applications had been applied successfully. 09:51 – Engineers confirmed success in resolving the fault for Germany and Australia regions through reapplying the Secret keys. 10:00 – Investigation began to determine why alerting did not advise of the error. 10:05 – Engineering was able to confirm the start of the failure rate within regional logging for Australia. 10:08 – Engineers identified that additional regions with pending Secret key changes may experience the issue; preventative work begins to address the issue within these regions. 10:16 – Engineers confirm success in reapplying the keys within Canada region 10:23 – Engineers confirm success in reapplying the keys within Europe region. 10:29 – Engineers confirm success in reapplying the keys within US region. 10:37 – Engineers confirm that keys in all regions have been reapplied successfully and the failure rate for this error has dropped to zero.