Easyship incident

APIs unstable and slow issue

Critical Resolved View vendor source →

Easyship experienced a critical incident on November 12, 2020 affecting Core API, lasting 1h 13m. The incident has been resolved; the full update timeline is below.

Started
Nov 12, 2020, 09:30 PM UTC
Resolved
Nov 12, 2020, 10:43 PM UTC
Duration
1h 13m
Detected by Pingoru
Nov 12, 2020, 09:30 PM UTC

Affected components

Core API

Update timeline

  1. investigating Nov 12, 2020, 10:34 PM UTC

    Core API check failed

  2. resolved Nov 12, 2020, 10:43 PM UTC

    This incident has been resolved.

  3. postmortem Nov 12, 2020, 11:03 PM UTC

    In this incident, users encountered the API unstable and slow issue. All times are UTC. 20:44 A new deployment on the production site. 21:30 Monitoring Servers report a check failed issue to the DevOps team. 21:35 Developer investigates the issue. 21:46 DevOps found an error from Core API. 21:51 DevOps found Database performance slow and all API latency is very high. 22:39 DevOps decides to rollback to the previous stable version. 22:44 All services are back to normal. **Root cause:** The root cause is the new migration codes caused a lot of locks on tables and then slow down all API calls.