Open Exchange Rates incident
[Resolved] Exchange rates API experiencing intermittent failures
Open Exchange Rates experienced a minor incident on January 1, 2017, lasting 8d 8h. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Jan 01, 2017, 07:05 AM UTC
A number of clients have reported receiving timeouts while connecting to our API over the past few hours. We are investigating and will respond as soon as we have further information.
- investigating Jan 01, 2017, 10:18 AM UTC
We are continuing to investigate this issue and will provide an update as soon as we have further information.
- monitoring Jan 01, 2017, 05:29 PM UTC
This incident caused connection timeouts for a large number of API clients. We have now deployed a fix and will continue to monitor throughout the day, with a post-mortem to follow shortly.
- resolved Jan 09, 2017, 03:40 PM UTC
This platform issue caused request timeouts for many of our API clients, between approximately 00:30 UTC and 13:00 UTC on January 1st. This resulted from an unprecedented number of API requests on New Year's Day, occurring in a way that did not allow our platform to efficiently auto-scale, due to an unexpected bottleneck between our database and cache. We began investigation as soon as we became aware of this issue, and resolved it with a deployment later in the day. Although a small number of clients continued to experience occasional timeouts for up to 48 hours afterwards, we have not experienced a recurrence of this issue. We have since made further infrastructure updates to safeguard your service. We sincerely apologise for any inconvenience this may have caused you. Please rest assured that we continue to closely monitor our API platform and responses for unexpected behaviour, providing you with the consistent and stable service availability you expect. Please don't hesitate to contact us if you have any questions or concerns about this incident. Warm regards, Open Exchange Rates Team