ServiceChannel experienced a major incident on May 2, 2023 affecting Login and Authentication and 1 more component, lasting 2h 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 02, 2023, 01:21 AM UTC
We are actively investigating degraded system performance. An update will be provided shortly. Thank you for your patience.
- monitoring May 02, 2023, 01:56 AM UTC
System stability has been restored and services are functioning normally. We will continue to monitor closely for any further issues.
- resolved May 02, 2023, 03:40 AM UTC
This incident has been resolved. All services are working as expected.
- postmortem May 16, 2023, 05:49 PM UTC
**Infrastructure/hardware instability** **Incident Report** **Date of Incident:**` `05/01/2023 **Time/Date Incident Started:** 05/01/2023, 5:00 pm EDT **Time/Date Stability Restored:**` `05/01/2023, 11:48 pm EDT **Time/Date Incident Resolved:**` `05/01/2023, 11:48pm EDT **Users Impacted:** All **Frequency:** Intermittent **Impact:** Major **Incident description:** Third party vendor infrastructure/hardware instability **Root Cause Analysis:** A third party vendor infrastructure issue affected performance and system availability for the underlying data storage layer servicing platform resources. **Actions Taken:** 1. Investigated system-generated alerts and identified affected platform functionality 2. SRE and DBA teams initiated a platform infrastructure redeployment, forcing the new infrastructure to spun up on unaffected infrastructure/hardware **Mitigation Measures:** 1. Continue the ongoing investigation into root causes of the infrastructure issue within our cloud hosting provider. 2. Continue to implement high availability improvements to prepare the platform to respond better to unexpected hardware issues that are beyond our control.