Orb had 38 outages in the last 2 years totaling 29h 42m of downtime — averaging 1.6 incidents per month.
There were 38 Orb outages since June 11, 2024 totaling 29h 42m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are currently investigating delays in our event ingestion pipeline.
Ingestion delays are recovering and we are actively monitoring the pipeline.
This incident has been resolved.
We saw a temporarily latency and error rate spike from 17:52 to 17:55 UTC.
From approximately 22:40 UTC to 22:43 UTC, our system experienced a high rate of API errors due to increase contention on a database. We have fully recovered and are continuing to monitor the affected database.
Our systems continue to remain stable and the issue affecting the affected database is fully resolved.
Between 23:09 and 23:17 UTC, an unintended process caused elevated contention in our databases, leading to elevated response times and error rates. This incident has since been resolved. Event ingestion endpoints were not affected.
We have detected and are working to resolve delays for some customers for the following webhooks: - subscription.usage_exceeded - subscription.cost_exceeded - customer.credit_balance_depleted - customer.credit_balance_dropped - customer.credit_balance_recovered The underlying issue has been resolved and our systems are working to return our webhook latency back to normal. We expect a full recover within the next few hours.
Webhooks latencies have returned to normal levels for all webhooks.
We've discovered an issue causing our data exports to be delayed for the Daily Line Item Revenue and Invoice Line Item Billing resources. We've identified the issue and are in the process of backfilling and exporting this data. We expect this to be fully resolved in the next few hours.
All data has been backfilled for the Daily Line Item Revenue and Invoice Line Item Billing resources.
A change in httpx v0.28.0 (released earlier today) affects fresh installs of Orb's Python SDK, and may cause API errors. Please either pin the httpx version to before 0.28.0 (via uv, requirements.txt, etc), or upgrade to version 2.17.0 of the Orb Python SDK.
We are currently investigating an issue with an upstream auth provider on access to the Orb admin dashboard.
The upstream provider has identified the issue; we're continuing to monitor for updates.
The upstream issue is resolved, and we're seeing recovery. We'll continue to monitor but no longer believe Orb users continue to be impacted.
This incident has been resolved, and the Orb dashboard should be functioning as normal.
A query load issue has been identified starting at 2024/10/29 8:45 UTC which was causing intermittent spikes of errors accessing internal data storage, resulting in API errors and delays in async processing. This issue has been mitigated for the majority of customers as we continue to investigate.
API services stabilized at 12:30 UTC and async processing has been restored, and we are continuing to monitor our systems.
API services and asynchronous processes are stable. The query load issue has been fully isolated and we're working on improvements to prevent this issue in the future.
This incident has been resolved.
A database migration caused a brief spike in API timeouts on read and analytics endpoints.
We have identified delays in issuing invoices since around 16:00 UTC 9/16/2024. A fix has been deployed, and we expect to fully recover within an hour as asynchronous processing catches up.
This incident has been resolved.
We're investigating an issue with our cloud provider that is causing elevated error rates on event ingestion related endpoints. File based ingestion is not affected.
As of 00:40 UTC, we've seen a significant reduction in error rates on the event ingestion related endpoint. We're continuing to monitor the situation with our cloud provider.
We've confirmed that error rates have stopped as of 07/31/24 01:12 UTC. Our on-call team will continue to monitor the service until we see recovery from our cloud provider.
The event ingestion service is fully operational, our on-call team will continue monitoring our systems.
The incident has been resolved