ServiceChannel incident

ServiceChannel - Fixxbook Provider Profile Viewing

Minor Resolved View vendor source →

ServiceChannel experienced a minor incident on May 7, 2024 affecting Compliance Manager, lasting 1h 24m. The incident has been resolved; the full update timeline is below.

Started
May 07, 2024, 04:39 PM UTC
Resolved
May 07, 2024, 06:03 PM UTC
Duration
1h 24m
Detected by Pingoru
May 07, 2024, 04:39 PM UTC

Affected components

Compliance Manager

Update timeline

  1. investigating May 07, 2024, 04:39 PM UTC

    We are actively investigating an issue with errors occurring while viewing provider profiles on Fixxbook. An update will be provided shortly. Thank you for your patience.

  2. monitoring May 07, 2024, 05:38 PM UTC

    A fix has been implemented and you should be able to view provider profiles on Fixxbook now. We will continue to monitor to ensure stability going forward.

  3. resolved May 07, 2024, 06:03 PM UTC

    This incident has been resolved. All services are working as expected.

  4. postmortem May 17, 2024, 08:10 PM UTC

    **Compliance Manager Outage Incident Report** **Date of Incident:** 5/8/2024 **Time/Date Incident Started:** 5/8/2024, 12:01 pm EDT **Time/Date Stability Restored:** 5/8/2024, 1:30 pm EDT **Time/Date Incident Resolved:** 5/8/2024, 1:41 pm EDT **Users Impacted:** Users utilizing the Fixxbook Compliance Manager **Frequency:** Duration of the incident event **Impact:** Major ‌ **Incident description:** At 12:00 EDT on May 8th, the Site Reliability Engineering \(SRE\) team responded to increased error rates through alerts, followed by user reports of malfunctions within the ServiceChannel Fixxbook service. This issue affected Fixxbook users, particularly those using features of the Compliance Manager, leading to a less than ideal user experience. **Root Cause Analysis:** After an in-depth analysis, the Site Reliability Engineering \(SRE\) team pinpointed the errors and discovered that an incorrectly set DNS record was disrupting the functionality of the Fixxbook application. The issue was promptly resolved, and the application's performance was reinstated to full capacity. The SRE team maintained vigilant oversight to confirm the application's sustained stability. **Actions Taken:** 1. Manually tested our services to replicate the issue. 2. Investigated system-generated alerts and identified affected platform functionality. 3. Updated DNS entry to restore functionality. **Mitigation Measures:** 1. Introduced enhanced alerting mechanisms designed to rapidly and more precisely detect issues within the Fixxbook application components.