ServiceChannel Outage History
ServiceChannel is up right nowPingoru has recorded 50 incidents for ServiceChannel: 13 in the last 12 months and 1 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 3 | 97 of 720 (13%) |
| May 2026 | 2 | 7 of 744 (1%) |
| June 2026 | 0 | 0 of 720 (0%) |
| July 2026 | 0 | 0 of 744 (0%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 1 | 15 of 720 (2%) |
| October 2026 | 0 | 0 of 515 (0%) |
Incidents, newest first
Minor September 2, 2026 Issues with Inventory Admin 14h 42m Inventory Manager
Started Sep 02, 2026, 04:00 PM UTC· resolved Sep 03, 2026, 06:43 AM UTC
- investigating Sep 02, 2026, 04:00 PM UTC
Inventory Admin is currently experiencing some issues. Users can continue to access and use inventory through the mobile app and other non-admin inventory functions.
- identified Sep 02, 2026, 04:01 PM UTC
The issue has been identified and a fix is being implemented.
- identified Sep 02, 2026, 04:01 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Sep 03, 2026, 12:41 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 03, 2026, 06:43 AM UTC
Monitoring confirms that the incident is resolved.
Minor May 27, 2026 Increase in errors submitting proposals 1h 21m Proposal Manager
Started May 27, 2026, 08:37 PM UTC· resolved May 27, 2026, 09:58 PM UTC
- investigating May 27, 2026, 08:37 PM UTC
We are investigating an increase in errors when some Providers are submitting proposals.
- monitoring May 27, 2026, 09:01 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 27, 2026, 09:58 PM UTC
This incident has been resolved.
Minor May 8, 2026 Subset of scheduled jobs interrupted due to issues with cloud provider 5h 24m
Started May 08, 2026, 02:07 PM UTC· resolved May 08, 2026, 07:32 PM UTC
- investigating May 08, 2026, 02:07 PM UTC
A small subset of scheduled jobs have been interrupted due to a cloud provider outage.
- identified May 08, 2026, 02:09 PM UTC
The issue has been identified and a fix is being implemented.
- identified May 08, 2026, 05:13 PM UTC
The cloud provider continues to work towards the fix.
- resolved May 08, 2026, 07:32 PM UTC
This incident has been resolved.
Minor April 24, 2026 Increased Latency impacting Dashboard services 8h 40m Dashboard
Started Apr 24, 2026, 02:00 PM UTC· resolved Apr 24, 2026, 10:40 PM UTC
- investigating Apr 24, 2026, 02:00 PM UTC
We are currently investigating this issue.
- identified Apr 24, 2026, 02:58 PM UTC
There is an issue with the underlying cloud platform, and we are awaiting resolution from the provider.
- identified Apr 24, 2026, 04:44 PM UTC
We are engaged with our cloud platform provider and continue working toward resolution.
- monitoring Apr 24, 2026, 06:24 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Apr 24, 2026, 10:40 PM UTC
We are continuing to monitor for any further issues.
- resolved Apr 24, 2026, 10:40 PM UTC
This incident has been resolved.
Major April 9, 2026 Elevated API Errors 1m Work Order Manager, Proposal Manager
Started Apr 09, 2026, 06:54 PM UTC· resolved Apr 09, 2026, 06:55 PM UTC
- investigating Apr 09, 2026, 06:54 PM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
- resolved Apr 09, 2026, 06:55 PM UTC
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon. Please ignore this was posted in error due to testing our status page updates.
Minor April 4, 2026 Degraded Performance with Universal Connector 3d 15h Universal Connector
Started Apr 04, 2026, 12:13 AM UTC· resolved Apr 07, 2026, 04:00 PM UTC
- investigating Apr 04, 2026, 12:13 AM UTC
We are currently investigating an issue which is impacting Universal Connector.
- identified Apr 04, 2026, 05:28 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 04, 2026, 03:08 PM UTC
A Fix has been implemented and we are monitoring the results. New connector events are now processing and may appear out of sequence as old and new messages are now being processed in parallel.
- resolved Apr 07, 2026, 04:00 PM UTC
This incident has been resolved. All old messages have been processed.
Major February 27, 2026 Errors connecting to Service Channel Platform 6m Login
Started Feb 27, 2026, 10:16 PM UTC· resolved Feb 27, 2026, 10:22 PM UTC
- investigating Feb 27, 2026, 10:16 PM UTC
We are currently investigating this issue.
- identified Feb 27, 2026, 10:16 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Feb 27, 2026, 10:21 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 27, 2026, 10:22 PM UTC
This incident has been resolved.
- postmortem Mar 12, 2026, 12:31 AM UTC
**Incident Report: Login Failures due to port exhaustion** **Date of Incident:** 02/27/2026 **Time/Date Incident Started:** 02/27, 5:00 PM EST **Time/Date Stability Restored:** 02/27, 5:06 PM EST **Time/Date Incident Resolved:** 02/27, 5:15 PM EST **Users Impacted:** Some **Frequency:** Continuous **Impact:** Major **Incident description:** On February 27, 2026, the Service Channel platform experienced an issue with the Login and Authentication services, resulting in some users being unable to log in. Additionally, a subset of users who were already logged in encountered authentication errors, which caused their sessions to become invalid. Monitoring systems alerted the SRE team at 5:00 PM EST. The SRE team initiated incident triage and identified the underlying issue. Service restarts and connection resets restored stability by 5:06 PM EST. All servers and services were fully verified and confirmed operational by 5:15 PM EST. **Root Cause Analysis:** The incident was caused by a sudden increase in concurrent connections to the Login and Authentication services, which resulted in port exhaustion on impacted nodes. As available ports were depleted, the system was unable to establish new connections and intermittently failed to maintain existing authentication sessions. This condition led to users being unable to log in, while a subset of users who were already authenticated experienced session validation failures that caused authentication interruptions. The issue was resolved by restarting the affected nodes, which reset the exhausted ports and restored normal connection handling. **Actions Taken:** * Monitoring alerts were received by the SRE team indicating authentication service degradation. * The SRE team initiated incident triage and investigated the login and authentication failures. * Port exhaustion was identified on impacted nodes during troubleshooting. * Recovery actions, including restarting affected nodes and resetting connections, were performed. * Services returned to normal operation following these recovery actions. * Platform functionality was fully verified after stability was restored.
Major February 26, 2026 Multiple issues on Production 33m
Started Feb 26, 2026, 02:42 PM UTC· resolved Feb 26, 2026, 03:15 PM UTC
- resolved Feb 26, 2026, 03:36 PM UTC
Issues on multiple services
- postmortem Mar 12, 2026, 12:28 AM UTC
**Incident Report: Cloud Provider issue cause Service Channel Platform Interruption** **Date of Incident:** 02/26/2026 **Time/Date Incident Started:** 02/26, 9:42 AM EST **Time/Date Stability Restored:** 02/26, 10:06 AM EST **Time/Date Incident Resolved:** 02/26, 10:15 AM EST **Users Impacted:** All U.S. **Frequency:** Continuous **Impact:** Major **Incident description:** On February 26, 2026, the Service Channel platform experienced multiple service interruptions due to an outage at a cloud hosting provider. The outage impacted several platform services that rely on the provider’s managed infrastructure. Our monitoring systems alerted the SRE team at 9:42 AM EST. The cloud hosting provider identified and resolved the underlying issue, after which service stability was restored by 10:06 AM EST. All servers and services were fully verified and confirmed operational by 10:15 AM EST. **Root Cause Analysis:** The cloud hosting provider experienced an issue with their managed SQL instances, which disrupted communication between several web applications and the database layer. This resulted in partial service interruptions across the platform. The cloud provider implemented a corrective action to resolve the SQL service issue, which restored connectivity and platform functionality. **Actions Taken:** * Monitoring alerts were received by the SRE team indicating service degradation. * Investigation identified connectivity issues between certain application modules and the database servers. * During incident triage, multiple recovery actions were attempted across impacted components; the issue persisted throughout these efforts * Services returned to normal operation once the cloud provider resolved the SQL issue
Major November 20, 2025 Error while creating invoices 1h 57m Invoice Manager
Started Nov 20, 2025, 07:29 PM UTC· resolved Nov 20, 2025, 09:26 PM UTC
- investigating Nov 20, 2025, 07:29 PM UTC
We are currently investigating this issue.
- identified Nov 20, 2025, 07:36 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Nov 20, 2025, 07:48 PM UTC
A fix has been implemented and we are monitoring the results.
- identified Nov 20, 2025, 08:07 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Nov 20, 2025, 09:10 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 20, 2025, 09:26 PM UTC
This incident has been resolved.
Major November 20, 2025 WebAPI Proposal authentication 23m Authentication
Started Nov 20, 2025, 05:28 PM UTC· resolved Nov 20, 2025, 05:51 PM UTC
- investigating Nov 20, 2025, 05:28 PM UTC
We are currently investigating this issue.
- monitoring Nov 20, 2025, 05:46 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 20, 2025, 05:51 PM UTC
This incident has been resolved.
Minor November 19, 2025 provider APP unspecified error while check in / out 3h 16m SC Provider
Started Nov 19, 2025, 04:06 PM UTC· resolved Nov 19, 2025, 07:22 PM UTC
- investigating Nov 19, 2025, 04:06 PM UTC
We are currently investigating this issue.
- identified Nov 19, 2025, 05:09 PM UTC
The issue has been identified and a fix is being implemented.
- identified Nov 19, 2025, 05:10 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Nov 19, 2025, 05:10 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 19, 2025, 07:22 PM UTC
This incident has been resolved.
Major November 3, 2025 WO Search Filters issue 8m Work Order Manager
Started Nov 03, 2025, 03:30 PM UTC· resolved Nov 03, 2025, 03:38 PM UTC
- investigating Nov 03, 2025, 03:30 PM UTC
We are currently investigating this issue.
- identified Nov 03, 2025, 03:33 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Nov 03, 2025, 03:37 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 03, 2025, 03:38 PM UTC
This incident has been resolved.
Notice October 22, 2025 Issues with calling into call center 46m Phone - Inbound
Started Oct 22, 2025, 12:43 PM UTC· resolved Oct 22, 2025, 01:30 PM UTC
- investigating Oct 22, 2025, 12:43 PM UTC
We are currently investigating this issue.
- identified Oct 22, 2025, 01:29 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Oct 22, 2025, 01:30 PM UTC
This incident has been resolved.
Notice October 20, 2025 Reported Issues with Cloud Services Provider Affecting Services 1h 43m
Started Oct 20, 2025, 09:53 AM UTC· resolved Oct 20, 2025, 11:36 AM UTC
- investigating Oct 20, 2025, 09:53 AM UTC
We are currently experiencing disruptions due to reported issues with one of our cloud services providers. This may affect certain functionalities, including email delivery and related services. Our team is closely monitoring the situation and working with the provider to mitigate the impact and restore normal operations as soon as possible. We will continue to post updates here as more information becomes available.
- monitoring Oct 20, 2025, 10:37 AM UTC
Services have recovered following earlier issues with one of our cloud services providers. All affected functionalities, including email delivery, are now operating normally. We continue to closely monitor system performance to ensure full stability and resolution.
- resolved Oct 20, 2025, 11:36 AM UTC
This incident has been resolved.
Minor October 17, 2025 intermittent work order page loading slowly 38m Dashboard, Work Order Manager, Proposal Manager
Started Oct 17, 2025, 02:51 PM UTC· resolved Oct 17, 2025, 03:29 PM UTC
- investigating Oct 17, 2025, 02:51 PM UTC
we are currently investigating
- identified Oct 17, 2025, 03:26 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Oct 17, 2025, 03:29 PM UTC
This incident has been resolved
Notice September 26, 2025 Issues with work order notes 8m Work Order Manager
Started Sep 26, 2025, 09:35 PM UTC· resolved Sep 26, 2025, 09:44 PM UTC
- investigating Sep 26, 2025, 09:35 PM UTC
We are currently investigating this issue.
- investigating Sep 26, 2025, 09:43 PM UTC
We are continuing to investigate this issue.
- monitoring Sep 26, 2025, 09:43 PM UTC
We are continuing to investigate this issue.
- resolved Sep 26, 2025, 09:44 PM UTC
This incident has been resolved.
- postmortem Oct 10, 2025, 07:34 PM UTC
**Incident Report: Unstable Network component cause Work Order Notes Issues.** **Dates of Incident:** 09/24/2025, 09/26/2025 **Time/Date Incident Started:** 09/24/2025, 1:17 PM EDT **Time/Date Stability Restored:** 09/26/2025, 5:43 PM EDT **Time/Date Incident Resolved:** 09/26/2025, 5:43 PM EDT **Users Impacted:** Many **Frequency:** Intermittent **Impact:** Major **Incident description:** On September 24 and September 26, 2025, Service Channel experienced 2 brief service disruptions that impacted the Work Order Notes feature for certain customers. These incidents each lasted approximately 15 to 30 minutes, during which some users encountered difficulties loading notes or received \(NTE\) errors when attempting to create a note. Our internal monitoring systems also recorded a small rise in HTTP 4xx errors related to these events. **Root Cause Analysis:** With the assistance of our Cloud Provider, the root cause was traced to an internal network component that had entered an unstable state. While in this state, it would periodically direct some traffic for the Work Order Notes service to an incorrect internal endpoint. This misdirected traffic was subsequently blocked as per security protocols, causing connection failures. This behavior directly corresponds to the intermittent periods of disruption and resulted in the loading issues and NTE errors observed by users. **Actions Taken:** * The SRE team began investigating alerts on 09/24 at 1:17 PM EDT, but troubleshooting was challenging due to the issue's intermittent nature. * The incident was escalated to Microsoft support for further analysis. * Joint diagnostics identified a faulty internal network component. * The SRE team removed the component from production to restore stability. * After removal, service was stable again by 5:43 PM EDT on 09/26. **Mitigation Measures:** * Internal service configurations were audited and updated to eliminate dependencies on this decommissioned network component, removing this failure point from other services. * Standard work includes a monitor that now detects and reports this error for faster future identification.
Major September 25, 2025 Work Orders issue - displaying inconsistent results 4h 15m Work Order Manager
Started Sep 25, 2025, 01:07 PM UTC· resolved Sep 25, 2025, 05:22 PM UTC
- investigating Sep 25, 2025, 01:07 PM UTC
We are currently investigating this issue.
- identified Sep 25, 2025, 02:08 PM UTC
The issue has been identified and a fix is being implemented.
- identified Sep 25, 2025, 04:17 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Sep 25, 2025, 05:15 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 25, 2025, 05:22 PM UTC
This incident has been resolved.
- postmortem Oct 10, 2025, 07:23 PM UTC
**Incident Report: Database Index Degradation result in Work Order Issues** **Date of Incident:** 09/25/2025 **Time/Date Incident Started:** 09/25/2025, 9:07 AM EDT **Time/Date Stability Restored:** 09/25/2025, 1:15 pm EDT **Time/Date Incident Resolved:** 09/25/2025, 1:22 pm EDT **Users Impacted:** Some **Frequency:** Intermittent **Impact:** Major **Incident description:** On September 25, 2025, Service Channel investigated a report of Work Order notes not displaying. Some users experienced intermittent issues with note visibility, including with newly created notes; however, all data was securely saved. Partial service resumed at 12:23 PM EDT, with full functionality restored by 1:15 PM EDT. **Root Cause Analysis:** The root cause of the incident was traced to performance degradation affecting two key indexes in the database responsible for managing Work Order notes. The indexes had degraded to a state where data retrieval queries became significantly inefficient. Consequently, application requests to display notes were timed out before the necessary data could be retrieved successfully. This directly led to the intermittent symptoms experienced by some users, including the failure to load existing notes and the non-display of newly submitted notes. **Actions Taken:** * The SRE team declared an incident and additional engineering resources were engaged as the issue was confirmed to be impacting at least 2 clients. * Initial remediation efforts included restarting application services and rolling back a recent code release, neither of which resolved the issue. * The investigation found database index degradation as the root cause, and rebuilding of the two indexes began. * Concurrently, a code-based mitigation was implemented to circumvent the inefficient queries, serving as an alternative resolution method. * At 12:23 PM EDT, the code mitigation was deployed, and the first index rebuild was completed, which restored partial functionality for users. * Full-service stability was restored at 1:15 PM EDT upon the successful completion of the second index rebuild. **Mitigation Measures:** * The planned migration of the Work Order notes database to a new platform has been successfully completed. This new environment offers enhanced monitoring and management tools that will help prevent and more quickly resolve index-related performance issues in the future. * The code-based mitigation deployed during the incident has been permanently implemented. This change optimizes the logic for retrieving notes, making the application more resilient to potential database performance fluctuations.
Major September 24, 2025 Work Orders issue 28m Work Order Manager
Started Sep 24, 2025, 04:50 PM UTC· resolved Sep 24, 2025, 05:18 PM UTC
- investigating Sep 24, 2025, 04:50 PM UTC
We are currently investigating this issue.
- identified Sep 24, 2025, 05:08 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Sep 24, 2025, 05:08 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 24, 2025, 05:18 PM UTC
This incident has been resolved.
- postmortem Oct 10, 2025, 07:33 PM UTC
**Incident Report: Unstable Network component cause Work Order Notes Issues.** **Dates of Incident:** 09/24/2025, 09/26/2025 **Time/Date Incident Started:** 09/24/2025, 1:17 PM EDT **Time/Date Stability Restored:** 09/26/2025, 5:43 PM EDT **Time/Date Incident Resolved:** 09/26/2025, 5:43 PM EDT **Users Impacted:** Many **Frequency:** Intermittent **Impact:** Major **Incident description:** On September 24 and September 26, 2025, Service Channel experienced 2 brief service disruptions that impacted the Work Order Notes feature for certain customers. These incidents each lasted approximately 15 to 30 minutes, during which some users encountered difficulties loading notes or received \(NTE\) errors when attempting to create a note. Our internal monitoring systems also recorded a small rise in HTTP 4xx errors related to these events. **Root Cause Analysis:** With the assistance of our Cloud Provider, the root cause was traced to an internal network component that had entered an unstable state. While in this state, it would periodically direct some traffic for the Work Order Notes service to an incorrect internal endpoint. This misdirected traffic was subsequently blocked as per security protocols, causing connection failures. This behavior directly corresponds to the intermittent periods of disruption and resulted in the loading issues and NTE errors observed by users. **Actions Taken:** * The SRE team began investigating alerts on 09/24 at 1:17 PM EDT, but troubleshooting was challenging due to the issue's intermittent nature. * The incident was escalated to Microsoft support for further analysis. * Joint diagnostics identified a faulty internal network component. * The SRE team removed the component from production to restore stability. * After removal, service was stable again by 5:43 PM EDT on 09/26. **Mitigation Measures:** * Internal service configurations were audited and updated to eliminate dependencies on this decommissioned network component, removing this failure point from other services. * Standard work includes a monitor that now detects and reports this error for faster future identification.
Major September 19, 2025 Work Order Notes are experiencing intermittent errors 4m Work Order Manager
Started Sep 19, 2025, 03:27 PM UTC· resolved Sep 19, 2025, 03:31 PM UTC
- investigating Sep 19, 2025, 03:27 PM UTC
We are currently investigating this issue.
- monitoring Sep 19, 2025, 03:31 PM UTC
A Network issue was identified on the hosting platform which has been resolved.
- resolved Sep 19, 2025, 03:31 PM UTC
This incident has been resolved.
Minor August 28, 2025 Work Order Notes are experiencing latency 14m Work Order Manager
Started Aug 28, 2025, 01:47 PM UTC· resolved Aug 28, 2025, 02:01 PM UTC
- investigating Aug 28, 2025, 01:47 PM UTC
We are currently investigating this issue.
- monitoring Aug 28, 2025, 01:54 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Aug 28, 2025, 02:01 PM UTC
This incident has been resolved.
Major August 21, 2025 proposal screen not loading 29m Proposal Manager
Started Aug 21, 2025, 02:48 PM UTC· resolved Aug 21, 2025, 03:17 PM UTC
- investigating Aug 21, 2025, 02:48 PM UTC
We are currently investigating this issue.
- resolved Aug 21, 2025, 03:17 PM UTC
This incident has been resolved.
Major August 8, 2025 Isssues with work order notes –
Started Aug 08, 2025, 09:00 AM UTC· resolved Aug 08, 2025, 09:00 AM UTC
- resolved Aug 08, 2025, 03:38 PM UTC
SRE team was notified of issues creating work order notes and investigated the issue. A fix was implemented and the issue cleared.
Major August 7, 2025 Slowness seen in work order notes 1h 20m Work Order Manager
Started Aug 07, 2025, 11:44 AM UTC· resolved Aug 07, 2025, 01:04 PM UTC
- investigating Aug 07, 2025, 11:44 AM UTC
We are currently investigating this issue.
- monitoring Aug 07, 2025, 12:57 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Aug 07, 2025, 01:04 PM UTC
This incident has been resolved.
Minor August 6, 2025 ServiceChannel - Fixxbook Provider search 39m Fixxbook
Started Aug 06, 2025, 11:52 AM UTC· resolved Aug 06, 2025, 12:32 PM UTC
- investigating Aug 06, 2025, 11:52 AM UTC
We are currently investigating issues with Fixxbook Provider Search
- monitoring Aug 06, 2025, 12:04 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Aug 06, 2025, 12:32 PM UTC
This incident has been resolved.
Major August 5, 2025 System Outage Phone inbound / Five9 26m Phone - Inbound
Started Aug 05, 2025, 08:09 PM UTC· resolved Aug 05, 2025, 08:36 PM UTC
- investigating Aug 05, 2025, 08:09 PM UTC
The ServiceChannel SRE team is currently investigating an issue affecting the ServiceChannel platform. We will provide an ETA shortly. Thank you for your patience.
- investigating Aug 05, 2025, 08:10 PM UTC
We are continuing to investigate this issue.
- identified Aug 05, 2025, 08:11 PM UTC
We are currently working with our Carrier to resolve inbound call issues at our Service Center.
- resolved Aug 05, 2025, 08:36 PM UTC
This incident has been resolved.