Rhumbix incident

High API load leading to errors

Major Resolved View vendor source →

Rhumbix experienced a major incident on November 30, 2021, lasting —. The incident has been resolved; the full update timeline is below.

Started
Nov 30, 2021, 09:50 PM UTC
Resolved
Nov 30, 2021, 12:26 PM UTC
Duration
Detected by Pingoru
Nov 30, 2021, 09:50 PM UTC

Update timeline

  1. resolved Nov 30, 2021, 09:50 PM UTC

    High load on our API endpoints is causing errors for some customers.

  2. postmortem Nov 30, 2021, 09:50 PM UTC

    This morning Rhumbix experienced about an hour and a half during which our services were partially degraded. We have identified that the downtime was caused by unusual usage of our public API by multiple customers simultaneously. The downtime has been resolved after 1.5 hours and we are working with these customers to transition them to our recently released public API v3. API v3 provides a way to interact with Rhumbix without risking this kind of impact. We are also investigating changes we could make to our API to prevent similar downtimes in the future for customers using the older API.