VinSolutions experienced a major incident on February 24, 2020 affecting Contacts API and Leads API, lasting 5h 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 24, 2020, 03:55 PM UTC
We are currently investigating reports of intermittent HTTP 500 / 504 responses affecting primarily the Contact endpoints. Multiple teams are engaged as this is a system-wide issue affecting the CRM.
- investigating Feb 24, 2020, 04:01 PM UTC
We are also investigating timeouts associated with requests to the Lead endpoints.
- investigating Feb 24, 2020, 05:20 PM UTC
We have placed the Contact API in maintenance mode while teams troubleshoot it. Requests to this endpoint will be responded to with HTTP 503 - Service Unavailable, with an additional header "Scheduled Maintenance."
- identified Feb 24, 2020, 06:02 PM UTC
CRM performance has improved significantly for clients; however, we will leave the Contact endpoint in maintenance mode while engineering and I/T bring online other processes which were disabled during troubleshooting. We will remove maintenance mode once we receive the all clear that supporting processes are again online.
- monitoring Feb 24, 2020, 07:50 PM UTC
After confirming CRM stability, we are now taking the Contact API out of maintenance mode. We continue to monitor the CRM's performance for slowness and will take corrective actions as necessary.
- resolved Feb 24, 2020, 09:14 PM UTC
CRM and API performance have returned to normal levels.