TrackJS experienced a major incident on January 10, 2023 affecting Error Ingestion and Management UI, lasting 15h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 10, 2023, 10:08 PM UTC
We are experiencing a widespread network outage with our hosting provider. We are working to get things resolved as quickly as possible.
- identified Jan 10, 2023, 11:17 PM UTC
Our hosting provider has identified the network issue and is working on remediation now. We are seeing most services return back to normal but our hosting provider has not given the all clear.
- monitoring Jan 11, 2023, 01:57 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 11, 2023, 01:10 PM UTC
After monitoring last evening the issue appears entirely resolved.
- postmortem Jan 11, 2023, 05:05 PM UTC
Our monitoring alerted us to problems with our Redis cluster. The servers appeared to be online and functioning normally, but the network connectivity between certain servers was not great. We reached out to our infrastructure provider, OVH.experienced network disruption on the private network that connects our servers \(the VRACK\). The network disruption was intermittent, but it caused our Redis cluster to often lose connectivity with our other servers. We quickly swapped Redis master to a different server that appeared to be more stable on the network while OVH resolved the issue. However, during the outage, some customers saw an error page while interacting with the UI and there were some delays and drops in processing new errors.