ServiceChannel incident
ServiceChannel System Performance Degradation
ServiceChannel experienced a critical incident on September 5, 2023 affecting Asset Manager, lasting 38m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 05, 2023, 02:05 PM UTC
We are actively investigating degraded system performance. An update will be provided shortly. Thank you for your patience.
- resolved Sep 05, 2023, 02:44 PM UTC
This incident has been resolved. All services are working as expected.
- postmortem Sep 19, 2023, 01:59 PM UTC
**Infrastructure/Hardware Instability** **Incident Report** **Date of Incident:**` `09/05/2023 **Time/Date Incident Started:** 09/05/2023, 09:15 am EDT **Time/Date Stability Restored:**` `09/05/2023, 10:19 am EDT **Time/Date Incident Resolved:**` `09/05/2023, 10:25 am EDT **Users Impacted:** All **Frequency:** Intermittent **Impact:** Major **Incident description:** Third party vendor infrastructure/hardware instability **Root Cause Analysis:** A third-party vendor infrastructure issue affected performance and system availability for the underlying data storage layer that services platform resources. **Actions Taken:** 1. Investigated system-generated alerts and identified affected platform functionality. 1. SRE and DBA teams initiated a platform infrastructure redeployment, forcing the new infrastructure to be spun up on unaffected infrastructure/hardware. **Mitigation Measures:** 1. Continue the ongoing investigation into root causes of infrastructure issues within our cloud hosting provider. 1. Continue to implement high availability improvements to prepare the platform to respond better to unexpected hardware issues that are beyond our control.