Orb experienced a notice incident on May 16, 2025 affecting Reads and Analytics, lasting 6h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 16, 2025, 08:51 AM UTC
We are experiencing periods of elevated API response times specifically on read and analytics requests. Our oncall team is currently investigating. Event ingestion, invoice finalization, and write requests are unaffected.
- monitoring May 16, 2025, 09:42 AM UTC
We've identified a failure mode tied to a degenerate workload that degrades query performance for one of our usage datastores. A temporary resolution has been put in place to ensure system stability, our oncall team will continue to monitor the error rates for the API and adjust workloads as needed to ensure API availability.
- resolved May 16, 2025, 02:54 PM UTC
After ongoing monitoring, this issue is now fully resolved. Endpoints have been recovered for the last several hours; the last period of errors occurred before 05/16/25 08:30 UTC.