Topsort incident
Partial Outage - Central services slow response times, 50Xs
Topsort experienced a major incident on February 16, 2023 affecting Management Portal - US-East-2 and Catalog API and 1 more component, lasting 1h 24m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 16, 2023, 06:48 PM UTC
The central services API is returning some 50Xs and has higher latencies than expected for all customers. Management portal has been impacted as well. Auctions and Events are not impacted. We have identified the issue and are working to repair it.
- monitoring Feb 16, 2023, 07:10 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 16, 2023, 08:12 PM UTC
This incident has been resolved.
- postmortem Feb 16, 2023, 08:30 PM UTC
**What happened?** Central services API received an increase of traffic to the catalog service endpoints. This increase of traffic caused all the Central Services pods to consume high amounts of memory causing all the Central Services pods to restart. The pods were unable to come fully back online due to being hit by the same amount of increased traffic with each restart. This caused all Central Service endpoints to have an increased latency as well return 500 type errors. The Manager was impacted due to it's reliance on Central Services and additionally was displaying 500 type errors. **What was done?** More strict rate limiting was applied to the catalog service API to prevent heavy traffic on high memory consuming endpoints. The number of pods to handle Central Service requests was increased. **What is being worked on?** Investigation into high memory usage on catalog service endpoints to remediate the issue and remove the stricter rate limiting.