xMatters Outage History

Started Oct 20, 2026, 06:24 AM UTC· resolved Oct 20, 2026, 10:27 PM UTC

1. investigating Oct 20, 2026, 06:24 AM UTC

   xMatters monitoring tools have identified a potential issue with xMatters On-Demand for some clients located in the Europe region. We are currently investigating the issue and will update as information becomes available. Please see incident details for specific services impacted. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. identified Oct 20, 2026, 06:31 AM UTC

   The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

3. monitoring Oct 20, 2026, 06:36 AM UTC

   The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

4. resolved Oct 20, 2026, 10:27 PM UTC

   The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

Read the full incident report →

Started Oct 01, 2026, 05:22 PM UTC· resolved Oct 01, 2026, 06:56 PM UTC

1. investigating Oct 01, 2026, 05:22 PM UTC

   xMatters monitoring tools have identified a potential issue with xMatters On-Demand for some clients located in the North America region. We are currently investigating the issue and will update as information becomes available. Please see incident details for specific services impacted. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. investigating Oct 01, 2026, 05:30 PM UTC

   We are continuing to investigate this issue.

3. identified Oct 01, 2026, 05:40 PM UTC

   The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

4. monitoring Oct 01, 2026, 05:45 PM UTC

   The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

5. resolved Oct 01, 2026, 06:56 PM UTC

   The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

6. postmortem Oct 13, 2026, 08:12 PM UTC

   **What happened?** On October 1, 2026, xMatters monitoring first detected service degradation at 9:27 AM Pacific, and the first customer report was received at 10:03 AM. Customers in US-EAST1 experienced intermittent Web Interface access issues, API timeouts, and delays or failures when integrations initiated alerts and notifications. Some customers in US-CENTRAL1 may also have experienced voice notification issues during the same period. **Why did it happen?** During automated infrastructure scaling in US-EAST1, multiple application services became concentrated on the same compute resource. That resource reached its processing capacity, reducing the capacity available to serve customer requests. Service degradation continued across dependent xMatters functions while additional processing capacity was being added. **How did we respond?** As soon as the first customer reported an issue, xMatters Customer Support initiated the internal Major Incident Management process and engaged the Engineering and Operations teams. The incident response team increased service capacity, monitored the affected services, and verified recovery across the Web Interface, APIs, integrations, and notification processing. Service began stabilizing by 10:37 AM, and primary customer impact ended at 10:46 AM. The team continued monitoring before resolving the public incident at 11:56 AM. **What are we doing to prevent it from happening again?** Engineering is reviewing service capacity and workload distribution controls to reduce the risk of multiple critical services competing for the same compute resources. The team is also reviewing service health checks and alerting so that capacity issues are identified earlier to ensure required capacity is always available.

Read the full incident report →

Started Sep 21, 2026, 06:14 PM UTC· resolved Oct 16, 2026, 04:03 PM UTC

1. investigating Sep 21, 2026, 06:14 PM UTC

   xMatters has identified an issue with xMatters SMS Notification for some clients in All Regions to numbers in the United States and Canada from our short code 87961. We have failed over to our secondary short code 66936. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. monitoring Sep 21, 2026, 06:41 PM UTC

   All SMS traffic to the US and Canada is now being delivered through our secondary short code 66936. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

3. resolved Oct 16, 2026, 04:03 PM UTC

   The short code issue has been addressed, and all SMS notifications to Canada and the United States are now being delivered from our 87961 short code. Thank you for your patience while we addressed this matter.

4. postmortem Oct 16, 2026, 04:24 PM UTC

   **What happened?** On September 21st, 2026, customers reported an issue to xMatters Client Assistance where SMS messages were failing to send to end users in North America. **Why did it happen?** The SMS delivery failures occurred because the short code used to send SMS messages became unavailable. The system should have automatically routed failed messages through the secondary messaging provider. However, the specific error returned for these failures did not trigger the configured failover process, causing disrupted SMS delivery to end users in the US and Canada. **How did we respond?** Once Engineering identified the SMS delivery issue, they initiated the Major Incident Management process and began removing the affected short code from the messaging configuration. During this process, Engineering encountered additional issues that prevented the system from automatically switching SMS traffic to the secondary messaging provider. Engineering manually intervened to complete the switch to the secondary provider and restore SMS delivery. After service recovered, Engineering monitored the system to confirm that SMS delivery was operating normally. **What are we doing to prevent it from happening again?** The affected short code remains unavailable, so Engineering continues to route SMS traffic through the secondary messaging provider while they work to restore the previous primary short code. Engineering is also addressing the failover behavior that prevented the system from automatically switching providers when this specific error occurred. These improvements to the process will reduce the risk of a similar short code issue disrupting SMS delivery in the future.

Read the full incident report →

Started May 01, 2026, 08:45 AM UTC· resolved May 01, 2026, 09:19 AM UTC

1. investigating May 01, 2026, 08:45 AM UTC

   xMatters have identified a potential issue with the xMatters Web User Interface for some clients located in the Europe region. Accessing some pages including Alert details produces an error. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. identified May 01, 2026, 09:01 AM UTC

   The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

3. monitoring May 01, 2026, 09:03 AM UTC

   The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

4. resolved May 01, 2026, 09:19 AM UTC

   The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

5. postmortem May 12, 2026, 08:32 PM UTC

   **What happened?** On May 1st, 2026, some customers reported an issue to xMatters Customer Support where attempting to send a message via the web user interface or viewing an alert on the Alerts report resulted in an error being displayed. The issue only affected the Reporting functions in the EMEA region; the system continued to accept signals, generate alerts, and send notifications across all regions. **Why did it happen?** The issue occurred when, during routine database maintenance, the database called a mismatched version of the library, resulting in an internal database error. The version mismatch within the cluster was traced to a prior database engine upgrade where a subset of replica nodes did not restart into the upgraded version. At no point was there any risk to data integrity. **How did we respond?** As soon as Customer Support confirmed the issue, they engaged the Engineering teams, who were able to identify the root cause and restore version consistency across all nodes. The teams validated stability and confirmed that all services were restored. **What are we doing to prevent it from happening again?** The Engineering Team has added explicit post-upgrade verification checks and monitoring to ensure node alignment is confirmed and maintained. This will provide safeguards to ensure node version alignment after upgrades and prevent this issue from reoccurring.

Read the full incident report →

Started Apr 01, 2026, 02:06 PM UTC· resolved Apr 01, 2026, 02:36 PM UTC

1. investigating Apr 01, 2026, 02:06 PM UTC

   xMatters monitoring tools have identified a potential issue with the API for some clients located in the North America region. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. identified Apr 01, 2026, 02:30 PM UTC

   The xMatters Incident Response team has identified an issue with the API for some clients located in the North America region which is causing some intermittent API timeouts. This is also causing issues with running reports in the system. We are currently investigating the issue and will update as information becomes available.

3. monitoring Apr 01, 2026, 02:33 PM UTC

   The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

4. resolved Apr 01, 2026, 02:36 PM UTC

   The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

5. postmortem Apr 06, 2026, 11:21 PM UTC

   **What happened?** On April 1st, 2026, some customers reported an issue to xMatters Customer Support where the Alerts or Notifications reports were timing out and failing to load. **Why did it happen?** This issue occurred because a backend service was experiencing significant unexpected load that caused report processing to be delayed. The ongoing resource constraint resulted in timeouts. **How did we respond?** As soon as customers reported an issue, Customer Support launched an investigation and escalated to the Engineering teams. The teams initiated rolling restart procedures for the applicable backend services to restore functionality. Once the rolling restart was completed, service was fully restored. **What are we doing to prevent it from happening again?** The Engineering teams are currently working to identify any possible bottlenecks that may have caused performance issues. In the interim, they have increased and adjusted resource allocation for several services to handle potential processing delays and prevent potential recurrences.

Read the full incident report →

Started Mar 10, 2026, 08:36 PM UTC· resolved Mar 10, 2026, 08:54 PM UTC

1. investigating Mar 10, 2026, 08:36 PM UTC

   We have reports that some users are not able to login to xMatters using SSO to login to the xMatters Web User Interface in All Regions. We are currently investigating the issue and will update as information becomes available. Native Login still works to bypass SSO. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. identified Mar 10, 2026, 08:51 PM UTC

   The xMatters Incident Response team believes they have identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

3. resolved Mar 10, 2026, 08:54 PM UTC

   The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

4. postmortem Mar 20, 2026, 06:01 PM UTC

   **What happened?** On March 10th, 2026, some customers reported an issue to xMatters Customer Support where they were encountering a 404 error page when attempting to log in to their instances via SSO. Some users may also have encountered a “We’ve run into a problem while retrieving your data.” error message. **Why did it happen?** This issue occurred during a routine maintenance update to the xMatters platform. Although several components of the platform were updated, specific configurations of the component related to SSO-based authentication conflicted with the update and resulted in 404 errors. This issue was limited to those few customers that had specific criteria set for their SSO configuration. **How did we respond?** As soon as customers reported the issue, Customer Support verified the issue and escalated immediately to Engineering. The team traced the issue to the maintenance deployment and initiated rollback procedures to restore functionality. Once the rollback was completed, the 404 errors ceased and service was confirmed restored. **What are we doing to prevent it from happening again?** The Engineering teams have implemented additional testing for any configuration criteria related to the SSO-based authentication component. In addition, they have begun working on an improved maintenance plan to prevent further issues that could occur during similar deployments.

Read the full incident report →

Started Oct 20, 2025, 10:20 AM UTC· resolved Oct 21, 2025, 04:34 PM UTC

1. identified Oct 20, 2025, 10:20 AM UTC

   xMatters monitoring tools have identified a issue with xMatters Conferencing and Voice calls for clients in All Regions. The issue is due to downstream provider outages.

2. monitoring Oct 20, 2025, 12:23 PM UTC

   Our downstream providers have deployed a fix and testing shows conferencing and voice systems are now working as expected. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

3. resolved Oct 21, 2025, 04:34 PM UTC

   Our provider has confirmed the issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

4. postmortem Oct 29, 2025, 09:52 PM UTC

   **What happened?** On October 20th, 2025, at approximately 1:21 AM Pacific, customers began reporting an issue affecting live call routing, conferences, and voice notifications. During this issue, customers in all regions would have been affected. **Why did it happen?** This issue was caused by a global AWS outage that impacted one of our downstream providers, resulting in voice notifications, live call routing, and conferencing they were handling to fail. **How did we respond?** As soon as the first customer reported an issue, Customer Support engaged the Engineering teams and launched an investigation. Once they identified the root cause, the teams updated the primary provider for affected regions to an alternate provider that was not affected by the AWS outage. When the new provider was assigned, customers reported that all services were operating correctly. **What are we doing to prevent it from happening again?** While this issue was out of our control or that of our provider, we are working to identify potential ways to improve resilience in case of external factors such as this.

Read the full incident report →

Started Oct 10, 2025, 10:58 PM UTC· resolved Oct 14, 2025, 03:58 PM UTC

1. monitoring Oct 10, 2025, 10:58 PM UTC

   The xMatters monitoring tools have alerted Customer Support to a potential issue with the integration platform in the North America region. Our technical teams have identified intermittent request failures and occasional slowdowns, though the service appears to be responsive and recovering correctly. We are continuing to monitor the situation and mitigating where possible, though some users may encounter a 5xx response code when submitting a request. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. resolved Oct 14, 2025, 03:58 PM UTC

   The issue has been addressed, and all services are running as expected. Thank you for your patience while we addressed this matter.

3. postmortem Oct 31, 2025, 06:48 PM UTC

   **What happened?** On October 10th, 2025, at approximately 4:14 PM Pacific, the xMatters internal monitoring tools alerted Customer Support to service degradation related to an issue that was already being internally monitored with the Integration Platform in the North American region. While this issue was being investigated and mitigated, customers may have experienced intermittent request failures and occasional slowdowns. **Why did it happen?** These particular issues were caused by a security update that caused conflicts with the underlying runtime environment, specifically the memory management routine. Slow performance of the routine was triggering frequent health checks for a request processing service and causing automatic restarts. **How did we respond?** As soon as the internal monitoring tools alerted Engineering to a potential issue, they began performing manual rolling restarts and deployed more forgiving liveness checks to avoid increasing error rates and to minimize any potential impact to customers. They also increased resources for the impacted service to improve responsiveness of the underlying routine and deployed a configuration fix to the Http Client Cache to try and stabilize the system. While mitigating the potential impact, they also increased monitoring levels as they continued to investigate the root cause and were able to deploy an update to the service and environment configuration on October 13 that resolved the issue. They continued monitoring and confirmed that the system was stable and all services were operational. **What are we doing to prevent it from happening again?** Engineering has updated the backend service and deployed additional updates and monitoring to the system configuration that will improve overall stability for the environment and prevent this issue from reoccurring.

Read the full incident report →

Started Sep 24, 2025, 03:59 AM UTC· resolved Sep 24, 2025, 04:43 AM UTC

1. investigating Sep 24, 2025, 03:59 AM UTC

   xMatters identified a issue with the Apple iOS Push Notifications for clients in All Regions. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. identified Sep 24, 2025, 04:19 AM UTC

   xMatters has identified the issue with the Apple Push notifications for clients located in the in All Regions. We are currently investigating a solution and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

3. identified Sep 24, 2025, 04:32 AM UTC

   We are continuing to work on a fix for this issue.

4. investigating Sep 24, 2025, 04:34 AM UTC

   xMatters has identified the issue with the Apple Push notifications for clients located in the in All Regions. We are currently deploying a solution and will update as information becomes available. Asia Pacific customers can now receive Apple push notifications.

5. resolved Sep 24, 2025, 04:43 AM UTC

   The issue has been addressed, and Apple Push notifications have been restored in all regions. Thank you for your patience while we addressed this matter.

6. postmortem Oct 15, 2025, 06:16 AM UTC

   **What happened?** On September 23rd, 2025, at approximately 8:31 PM Pacific, the xMatters internal monitoring tools alerted the xMatters Support Team to an issue with Apple iOS notifications. Users were not receiving notifications on Apple iOS devices; other device types and alert and response processing were not impacted and continued to operate without interruption. **Why did it happen?** The issue occurred because of a backend service patch applied by the Engineering teams that affected necessary dependencies for Apple Push notification delivery. **How did we respond?** As soon as the monitoring tools alerted the Support team, they engaged the Engineering teams to launch an investigation. The teams quickly identified the source of the issue as a recent backend service patch and initiated a rollback of the patch. Once the rollback was complete, the teams confirmed that all services had been restored. **What are we doing to prevent it from happening again?** The Engineering team is working to enhance testing of future patches and improving visibility into the health of services. This will help to proactively resolve issues or, when necessary, initiate rollbacks more quickly without impacting customers.

Read the full incident report →

Started Aug 06, 2025, 03:27 AM UTC· resolved Aug 06, 2025, 05:18 AM UTC

1. investigating Aug 06, 2025, 03:27 AM UTC

   xMatters monitoring tools have identified a potential issue with xMatters On-Demand for clients in North America that are hosted in the on us-central1 region. We are currently investigating the issue and will update as information becomes available. Please see incident details for specific services impacted. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support team is waiting to help.

2. identified Aug 06, 2025, 03:44 AM UTC

   The xMatters Incident Response team has identified an issue with our message routing system and is working on a fix. We will update once a solution has been identified and implemented.

3. monitoring Aug 06, 2025, 03:59 AM UTC

   The xMatters Incident Response team has deployed a fix for the issue. We are currently monitoring the situation to ensure the implementation is stable and that all services are restored.

4. resolved Aug 06, 2025, 05:18 AM UTC

   The issue has been addressed, and all services have been restored. Thank you for your patience while we addressed this matter.

5. postmortem Aug 14, 2025, 01:16 AM UTC

   ### What happened? On August 5th, 2025, at approximately 7:53 PM Pacific, the xMatters internal monitoring tools alerted the xMatters Support team to an issue in the North America region with notification delivery. Some xMatters customers may have experienced delays when receiving notifications or noticed failed alerts in the system. ### Why did it happen? The issue occurred because of a network problem that disrupted communication between parts of the queuing system. This caused some components to become temporarily out of sync, leading to timeouts, internal connectivity failures, and a small number of messages not being processed during the disruption. As the system recovered, some performance was degraded until normal operation could be restored. ### How did we respond? As soon as the xMatters monitoring tools reported an issue, the xMatters Support Team initiated the internal Major Incident Management process and engaged the Engineering and incident response teams. The teams quickly mitigated the issue and restored performance by redirecting traffic from our message queuing system in the affected region \(us-central\) to our message queuing system in another local region \(us-east\). After the issue was resolved, the teams directed traffic back to the restored region. ### What are we doing to prevent it from happening again? The Engineering team has determined that the best approach to prevent this issue from reoccurring is to replace the current message queuing system. Work on the replacement system is well underway and the teams will retire the current system as soon as they have finished the replacement.

Read the full incident report →

Started Jun 23, 2025, 08:14 PM UTC· resolved Jun 23, 2025, 08:46 PM UTC

1. investigating Jun 23, 2025, 08:14 PM UTC

   We have identified a problem affecting the xMatters Support Center. Our team is investigating and working to resolve the issue. Details: Users will be unable to access the xMatters Support Center. Tickets can be created via email or through Support line.

2. resolved Jun 23, 2025, 08:46 PM UTC

   We have now resolved the issue with our support center.

Read the full incident report →

Started Jun 12, 2025, 06:47 PM UTC· resolved Jun 12, 2025, 10:12 PM UTC

1. investigating Jun 12, 2025, 06:47 PM UTC

   xMatters monitoring tools have identified a potential issue with the xMatters Integration Platform for some clients located in the North America region. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. identified Jun 12, 2025, 07:04 PM UTC

   The xMatters Incident Response team has identified the source of the issue with an upstream provider and is working with them on identifying a fix. We will update once a solution has been identified and implemented.

3. identified Jun 12, 2025, 07:34 PM UTC

   We are still working with our upstream providers. Their engineers are continuing to mitigate the issue and we have confirmation that the issue is recovered in some locations. We do not have an ETA on full mitigation at this point. We continue to operate in a degraded state.

4. identified Jun 12, 2025, 08:06 PM UTC

   We are continuing to work with our upstream providers. Their engineers have confirmed that the underlying dependency is recovered in most regions and all their engineering teams are actively engaged and working on service recovery. We do not have an ETA for full service recovery.

5. monitoring Jun 12, 2025, 08:53 PM UTC

   At this time, we are seeing all services recovering. We will continue to monitor until we have confirmation from our upstream provider that they have fully resolved the issue.

6. resolved Jun 12, 2025, 10:12 PM UTC

   We are currently seeing error rates down to normal levels and we are considering this issue resolved at this time

7. postmortem Jun 23, 2025, 05:40 PM UTC

   **What happened?** On June 12th, 2025, at approximately 11:29 AM Pacific, the xMatters internal monitoring tools alerted to an issue in the North America region with the xMatters integration platform. While the issue was in progress, some xMatters customers may have experienced workflows not executing correctly or signals not being accepted as expected. **Why did it happen?** The issue occurred because multiple Google Cloud and Google Workspace products experienced external API issues for about three hours between the hours of 10:49 AM to 1:49 PM Pacific. **How did we respond?** As soon as the xMatters monitoring tools reported an issue with the xMatters integration platform, the xMatters Customer Support team initiated the internal Major Incident Management process and engaged the Engineering and incident response teams. The teams quickly determined that the issues were due to problems with an upstream provider and immediately opened a support case for them to investigate. They continued to work with the provider throughout the incident and confirmed with their engineers as they recovered the underlying dependencies. **What are we doing to prevent it from happening again?** The xMatters Support and Engineering teams requested an RCA and prevention methods from the upstream provider. In their response, they confirmed that they will be modularizing their Service Control architecture to prevent failures from affecting the entire service.

Read the full incident report →

Started Apr 03, 2025, 05:15 PM UTC· resolved Apr 03, 2025, 09:45 PM UTC

1. investigating Apr 03, 2025, 05:15 PM UTC

   xMatters monitoring tools have identified a potential issue with xMatters Integration Platform for some clients in All Regions. We are currently investigating the issue and will update as information becomes available. If you are also experiencing issues, or if you're not sure whether this issue impacts your service, please contact xMatters Client Assistance at https://support.xmatters.com/hc/en-us/requests/new - our support agents are waiting to help.

2. identified Apr 03, 2025, 05:17 PM UTC

   The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

3. identified Apr 03, 2025, 05:46 PM UTC

   The xMatters Incident Response team has identified the source of the issue and is still working on a fix. We will update once a solution has been identified and implemented.

4. identified Apr 03, 2025, 06:01 PM UTC

   The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

5. identified Apr 03, 2025, 06:28 PM UTC

   The xMatters Incident Response team has identified the source of the issue and is still working on a fix. We will update once a solution has been identified and implemented.

6. identified Apr 03, 2025, 06:44 PM UTC

   The xMatters Incident Response team has identified the source of the issue and is actively working on a fix, there is no current estimate on resolution time. We will update once a solution has been implemented.

7. identified Apr 03, 2025, 07:00 PM UTC

   The xMatters Incident Response team has identified the source of the issue and is still actively working on a fix, there is no current estimate on resolution time. We will update once a solution has been implemented.

8. identified Apr 03, 2025, 07:26 PM UTC

   The xMatters Incident Response team has identified the source of the issue and is still actively working on a fix, there is no current estimate on resolution time. We will update once a solution has been implemented.

9. identified Apr 03, 2025, 07:56 PM UTC

   The xMatters Incident Response team has identified the source of the issue and is still actively working on a fix, there is no current estimate on resolution time. We will update once a solution has been implemented.

10. identified Apr 03, 2025, 08:26 PM UTC

    The xMatters Incident Response team has identified the source of the issue and is still actively working on a fix, there is no current estimate on resolution time. We will update once a solution has been implemented.

11. identified Apr 03, 2025, 08:59 PM UTC

    The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

12. identified Apr 03, 2025, 09:00 PM UTC

    The xMatters Incident Response team has identified the source of the issue and is working on a fix. We will update once a solution has been identified and implemented.

13. identified Apr 03, 2025, 09:13 PM UTC

    The xMatters Incident Response team has identified the source of the issue and are currently testing a fix. We will provide another update shortly.

14. monitoring Apr 03, 2025, 09:18 PM UTC

    A fix has been implemented and we are monitoring the results.

15. resolved Apr 03, 2025, 09:45 PM UTC

    This incident has been resolved.

16. postmortem Apr 07, 2025, 06:55 PM UTC

    **What happened?** On April 3, 2025, at approximately 9:32 AM Pacific, the xMatters internal monitoring systems identified an issue where the system was not processing events initiated via Flow Designer across multiple regions. Customers may have observed the system not processing events or creating alerts while the issue was in progress. **Why did it happen?** The issue occurred when a routine update to add new permissions to the xMatters' Google Cloud Platform \(GCP\) unexpectedly removed required permissions. When the teams performed the update, which should not have had any impact to customers, the policy used in the automation script was the authoritative resource at the GCP project level rather than authoritative at the individual resource level. Although the teams tested the change before deploying it and found no changes beyond what was included in the update, when the update was deployed to production GCP removed all permissions that were not in the policy in the background. Because the policy only included the new permissions, all other permissions were removed. **How did we respond?** As soon as the xMatters monitoring tools reported an issue with the system not processing events, the incident response teams initiated the internal Major Incident Management process and engaged the Engineering and Support teams. The teams were able to quickly identify the recent update as the root cause of the issue and reverted the change to restore the permissions that were managed by the automation process. This restored access and functionality for most of the xMatters services, but restoring permissions for Flow Designer proved to be more complicated. The teams determined that missing permissions for the xMatters infrastructure were Google-generated permissions essential for specific xMatters services and engaged GCP Support to aid in the investigation. The teams generated a list of all permissions that existed prior to the update and designed a fix to re-apply them to the development environment. Once the teams had implemented the change and validated that the missing permissions and all services had been restored in the development environment, they moved to quickly apply the fix across all staging and production environments. Monitoring tools and customers confirmed that services were fully functional, and the teams continued to monitor the system as it processed all messages queued by the Flow Designer services. The system was fully restored at 2:46 PM Pacific. **What are we doing to prevent it from happening again?** The Engineering teams were able to identify all of the permissions created in the xMatters environment, including those that are created by Google, and are ensuring they are added to the management scripts. The teams are adding additional rigor to the application of these types of infrastructure changes to run idempotence tests after the changes are applied to ensure that there are no changes pending. Should a change be applied that fails this test, it will cause failures in development environments, which would catch and prevent a similar issue from occurring.

Read the full incident report →