ServiceChannel experienced a minor incident on July 10, 2025 affecting Work Order Manager and Invoice Manager, lasting 3h 55m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 10, 2025, 03:13 PM UTC
We are currently experiencing intermittent issues with Provider users adding Notes to work orders. Our Engineering Team is releasing a fix to resolve the issue and restore full functionality. We appreciate your patience and will provide updates as more information becomes available.
- investigating Jul 10, 2025, 03:14 PM UTC
We are continuing to investigate this issue.
- monitoring Jul 10, 2025, 04:15 PM UTC
We have released a fix that resolves the reported issues with provider user updates. The system is now functioning as expected, and we will continue to monitor performance closely. Thank you for your patience.
- resolved Jul 10, 2025, 07:09 PM UTC
This incident has been resolved.
- postmortem Jul 31, 2025, 07:51 PM UTC
**\[Service providers experiencing the error when attempting to add notes to work orders \]** **Incident Report** **Date of Incident:** 07/10/2025 **Time/Date Incident Started:** 07/10/2025, 08:55 am EDT **Time/Date Stability Restored:** 07/10/2025, 09:59 am EDT **Time/Date Incident Resolved:** 07/10/2025, 12:11 am EDT **Users Impacted:** Some users **Frequency:** Intermittent **Impact:** Minor **Incident description:** On July 10, 2025, some of our service providers experienced intermittent difficulties when attempting to add notes to work orders. We sincerely apologize for any inconvenience this may have caused to your business operations **Root Cause Analysis:** Our investigation revealed that the primary issue was an authentication system malfunction within our platform that prevented some providers from properly accessing the note-adding functionality. The situation was complicated by issues with our third-party cloud hosting provider. These external complications extended the time needed to fully restore service to all users. **Actions Taken:** Our engineering team took immediate action upon discovering the issue: * **Rapid Response** - We began investigating the problem within 5 minutes of detection * **Temporary Stabilization** - We implemented temporary measures that restored service for most users by adjusting some networking rules by 9:59 AM EDT * **Comprehensive Solution** - We developed and deployed permanent fixes to prevent similar issues in the future * **Third-Party Coordination** - We worked closely with our cloud hosting provider to resolve their configuration issues that were impacting our service. * **Thorough Testing** - We conducted extensive testing to ensure all functionality was working properly before declaring the incident resolved **Mitigation Measures:** We take the reliability of our services very seriously. To prevent similar issues in the future, we are: * **Enhancing Monitoring** - Implementing additional safeguards to detect authentication issues more quickly * **Improving Testing** - Strengthening our testing procedures to catch potential problems before they affect users * **Third-Party Reliability** - Working with our gateway provider to improve their service reliability and implement better failover procedures * **Communication Protocols** - Reviewing our incident response procedures to ensure faster resolution times