ServiceChannel Outage History

Started Jul 21, 2025, 03:39 PM UTC· resolved Jul 22, 2025, 01:03 PM UTC

1. investigating Jul 21, 2025, 03:39 PM UTC

   We are currently investigating this issue.

2. investigating Jul 21, 2025, 06:11 PM UTC

   We are continuing to investigate this issue.

3. investigating Jul 21, 2025, 08:54 PM UTC

   We are actively working with Looker support to resolve the issue

4. monitoring Jul 22, 2025, 12:06 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Jul 22, 2025, 01:03 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 10, 2025, 03:13 PM UTC· resolved Jul 10, 2025, 07:09 PM UTC

1. investigating Jul 10, 2025, 03:13 PM UTC

   We are currently experiencing intermittent issues with Provider users adding Notes to work orders. Our Engineering Team is releasing a fix to resolve the issue and restore full functionality. We appreciate your patience and will provide updates as more information becomes available.

2. investigating Jul 10, 2025, 03:14 PM UTC

   We are continuing to investigate this issue.

3. monitoring Jul 10, 2025, 04:15 PM UTC

   We have released a fix that resolves the reported issues with provider user updates. The system is now functioning as expected, and we will continue to monitor performance closely. Thank you for your patience.

4. resolved Jul 10, 2025, 07:09 PM UTC

   This incident has been resolved.

5. postmortem Jul 31, 2025, 07:51 PM UTC

   **\[Service providers experiencing the error when attempting to add notes to work orders \]** **Incident Report** **Date of Incident:** 07/10/2025 **Time/Date Incident Started:** 07/10/2025, 08:55 am EDT **Time/Date Stability Restored:** 07/10/2025, 09:59 am EDT **Time/Date Incident Resolved:** 07/10/2025, 12:11 am EDT **Users Impacted:** Some users **Frequency:** Intermittent **Impact:** Minor **Incident description:** On July 10, 2025, some of our service providers experienced intermittent difficulties when attempting to add notes to work orders. We sincerely apologize for any inconvenience this may have caused to your business operations **Root Cause Analysis:** Our investigation revealed that the primary issue was an authentication system malfunction within our platform that prevented some providers from properly accessing the note-adding functionality. The situation was complicated by issues with our third-party cloud hosting provider. These external complications extended the time needed to fully restore service to all users. **Actions Taken:** Our engineering team took immediate action upon discovering the issue: * **Rapid Response** - We began investigating the problem within 5 minutes of detection * **Temporary Stabilization** - We implemented temporary measures that restored service for most users by adjusting some networking rules by 9:59 AM EDT * **Comprehensive Solution** - We developed and deployed permanent fixes to prevent similar issues in the future * **Third-Party Coordination** - We worked closely with our cloud hosting provider to resolve their configuration issues that were impacting our service. * **Thorough Testing** - We conducted extensive testing to ensure all functionality was working properly before declaring the incident resolved **Mitigation Measures:** We take the reliability of our services very seriously. To prevent similar issues in the future, we are: * **Enhancing Monitoring** - Implementing additional safeguards to detect authentication issues more quickly * **Improving Testing** - Strengthening our testing procedures to catch potential problems before they affect users * **Third-Party Reliability** - Working with our gateway provider to improve their service reliability and implement better failover procedures * **Communication Protocols** - Reviewing our incident response procedures to ensure faster resolution times

Read the full incident report →

Started Jun 12, 2025, 02:14 PM UTC· resolved Jun 12, 2025, 08:22 PM UTC

1. investigating Jun 12, 2025, 02:14 PM UTC

   We are actively investigating degraded system performance. An update will be provided shortly. Thank you for your patience.

2. investigating Jun 12, 2025, 04:24 PM UTC

   We continue to see issues with work order search the team is actively troubleshooting

3. monitoring Jun 12, 2025, 06:44 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jun 12, 2025, 08:22 PM UTC

   This incident has been resolved.

5. postmortem Jun 23, 2025, 07:40 PM UTC

   **Date of Incident:** 06/12/2025 **Time/Date Incident Started:** 06/12/2025, 9:35 am EDT **Time/Date Stability Restored:** 06/12/2025, 2:12 pm EDT **Time/Date Incident Resolved:** 06/12/2025, 2:12 pm EDT **Users Impacted:** All **Frequency:** Intermittent **Impact:** Major **Incident description:** US clients experienced intermittent issues loading their work order lists due to degraded performance in the underlying services responsible for filtering and NTE calculations. **Root Cause Analysis:** Processing certain very large provider accounts with a high number of serviceable locations triggered significant memory overconsumption, causing prolonged response times \(>60 seconds\) from an internal Elasticsearch service that affected all users. The system had been functioning normally until these large provider accounts triggered the condition causing a cascading effect on all users for work order lists. The intermittent nature of these issues extended the overall duration of user-facing problems **Actions Taken:** As soon as the issue was identified, our team initiated a series of mitigation steps to restore service as quickly as possible: * Restarted and scaled out the Elasticsearch service to address potential performance or resource bottlenecks. This had a slight positive effect but wasn't enough to stabilize the system. * Rolled back recent changes affecting client interaction with Elasticsearch, resulting in minor improvements. * Removed multiple clients from the Elasticsearch service to reduce load, which slightly decreased strain, but overall service stability was still insufficient. * Rolled back Elasticsearch itself to a previously stable version, reducing some system pressure but not fully resolving the problem. * Finally, rolled back the entire release, leading to system recovery and a return to normal service performance within approximately 30 minutes. **Mitigation Measures:** * Implemented code updates to address an edge case affecting providers with extensive serviceable location networks. * Enhanced code for better system resilience with effective Elastic fallback mechanisms. * Expanded monitoring capabilities with incident-specific metrics to increase visibility and decrease diagnostic time.

Read the full incident report →

Started May 08, 2025, 06:29 AM UTC· resolved May 08, 2025, 07:07 AM UTC

1. resolved May 15, 2025, 09:59 PM UTC

   During the scheduled US production code release on May 8, 2025, ServiceChannel encountered technical issues that impacted service availability on our platform. Users experienced login difficulties from 2:29 AM to 3:07 AM EDT, while critical dashboard functionality was unavailable from 2:29 AM to 4:12 AM EDT.

2. postmortem May 15, 2025, 10:00 PM UTC

   **Incident Report: Code Release Causes US Environment Outage** **Date of Incident:** 05/08/2025 **Time/Date Incident Started:** 05/08/2025, 2:29 am EDT **Time/Date Stability Restored:** 05/08/2025, 4:07 am EDT **Time/Date Incident Resolved:** 05/08/2025, 4:12 am EDT **Users Impacted:** Many **Frequency:** Continuous **Impact:** Major **Incident description:** During the scheduled US production code release on May 8, 2025, ServiceChannel encountered technical issues that impacted service availability on our platform. Users experienced login difficulties from 2:29 AM to 3:07 AM EDT, while critical dashboard functionality was unavailable from 2:29 AM to 4:12 AM EDT. **Root Cause Analysis:** Login Module Issue**:** As part of ongoing deployment process enhancements, a configuration adjustment was made that worked correctly in our testing environments but behaved differently in production. The issue was identified and resolved through our standard troubleshooting procedures. Dashboard Issue**:** A configuration setting that was properly configured in our development environments had not been fully synchronized to the production environment. This discrepancy wasn't detected until the new code attempted to access the setting during the deployment. Full platform functionality was confirmed restored by 4:12 AM EDT **Actions Taken:** * SRE team immediately investigated upon receiving alerts starting at 2:29am EDT indicating issues with two critical systems: dashboard and login * CICD team successfully rolled back the login module to the prior version, restoring user access by 3:07 AM EDT * Dashboard continued to experience issues, so investigation continued while login was restored * Dashboard functionality was restored by ensuring all required configuration settings were properly applied to production **Mitigation Measures:** * Reviewed existing deployment procedures to include improved configuration validation and improved rollback protocols to prevent similar configuration-related issues in the future * Implemented process improvements for immediate communication with support teams following any service disruptions to ensure proper customer follow-up and transparency

Read the full incident report →

Started Mar 31, 2025, 04:24 PM UTC· resolved Mar 31, 2025, 07:45 PM UTC

1. investigating Mar 31, 2025, 04:24 PM UTC

   The ServiceChannel SRE team is currently investigating an issue affecting the IVR. We will provide an ETA shortly. Thank you for your patience.

2. investigating Mar 31, 2025, 04:36 PM UTC

   We're seeing that that only IVR is impacted by this Twilio outage. The workaround is using the mobile app to check in/out. We're monitoring the situation closely.

3. investigating Mar 31, 2025, 04:36 PM UTC

   We are continuing to investigate this issue.

4. identified Mar 31, 2025, 04:36 PM UTC

   The issue has been identified and a fix is being implemented.

5. identified Mar 31, 2025, 04:43 PM UTC

   We are continuing to work on a fix for this issue.

6. monitoring Mar 31, 2025, 04:45 PM UTC

   A fix has been implemented, and we are monitoring the issue; the IVR should now be working

7. resolved Mar 31, 2025, 07:45 PM UTC

   This incident has been resolved.

Read the full incident report →