Layer Systems incident
Increased latency resulting in slow performance
Layer Systems experienced a minor incident on January 18, 2023 affecting Mail Triage and Layer App and 1 more component, lasting 5h 42m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 18, 2023, 11:47 AM UTC
We're currently investigating reports of latency affecting some requests. We'll update this page when we have further information.
- identified Jan 18, 2023, 12:02 PM UTC
Our upstream provider has provided the following statement: Impact Statement: Starting at 11:24 UTC on 18 Jan 2023, a subset of customers using storage services in Europe may experience higher than expected latency, timeouts or HTTP 500 errors when accessing data stored on accounts hosted in this region. Additional services with dependencies on storage may also experience impact. Current Status: We have determined that a subset of racks in a storage scale unit, hosted in one of the region datacenter experienced a drop in availability and may have rebooted. Our telemetry is showing that a power event impacted the said racks (impacting storage and network resources in the racks). Following the power event, most impacted resources auto-recovered through their normal recovery mechanisms. However, some resources required manual intervention to fully recover. Most customers should already be seeing signs of recovery and we are working on recovering the remaining impacted resources, while monitoring the health of previously recovered workloads. An update will be provided within 60 minutes, or as events warrant.
- resolved Jan 18, 2023, 05:30 PM UTC
This incident has been resolved.