Orb Outage History
Orb is up right nowPingoru has recorded 50 incidents for Orb: 25 in the last 12 months and 5 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 2 | 12 of 720 (2%) |
| May 2026 | 3 | 4 of 744 (0%) |
| June 2026 | 2 | 2 of 720 (0%) |
| July 2026 | 4 | 8 of 744 (1%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 4 | 30 of 720 (4%) |
| October 2026 | 0 | 0 of 569 (0%) |
Incidents, newest first
Minor July 3, 2025 Delays in asynchronous work 26m Real-time alerting
Started Jul 03, 2025, 01:31 AM UTC· resolved Jul 03, 2025, 01:58 AM UTC
- investigating Jul 03, 2025, 01:31 AM UTC
Orb is affected by a cloud provider incident and this is affecting our asynchronous workload. Currently this is not affecting APIs or data ingestion, but we're continuing to monitor in partnership with our service provider.
- monitoring Jul 03, 2025, 01:51 AM UTC
We're seeing recovery and are continuing to monitor.
- resolved Jul 03, 2025, 01:58 AM UTC
We're not seeing any continued impact on our services.
Notice June 28, 2025 Elevated API Errors 1h Reads, Writes, Analytics
Started Jun 28, 2025, 01:13 AM UTC· resolved Jun 28, 2025, 02:14 AM UTC
- monitoring Jun 28, 2025, 01:13 AM UTC
We experienced an elevated level of API errors, which has now subsided. We are actively monitoring for further errors.
- resolved Jun 28, 2025, 02:14 AM UTC
This incident has been resolved.
Notice May 16, 2025 Increased latency on read and analytics endpoints 6h 2m Reads, Analytics
Started May 16, 2025, 08:51 AM UTC· resolved May 16, 2025, 02:54 PM UTC
- investigating May 16, 2025, 08:51 AM UTC
We are experiencing periods of elevated API response times specifically on read and analytics requests. Our oncall team is currently investigating. Event ingestion, invoice finalization, and write requests are unaffected.
- monitoring May 16, 2025, 09:42 AM UTC
We've identified a failure mode tied to a degenerate workload that degrades query performance for one of our usage datastores. A temporary resolution has been put in place to ensure system stability, our oncall team will continue to monitor the error rates for the API and adjust workloads as needed to ensure API availability.
- resolved May 16, 2025, 02:54 PM UTC
After ongoing monitoring, this issue is now fully resolved. Endpoints have been recovered for the last several hours; the last period of errors occurred before 05/16/25 08:30 UTC.
Notice April 10, 2025 Increased datastore latency resulting in timeouts 7m
Started Apr 10, 2025, 08:19 AM UTC· resolved Apr 10, 2025, 08:26 AM UTC
- resolved Apr 10, 2025, 09:06 AM UTC
Due to a large increase in query volume, analytics and read queries were degraded for a time period, resulting in a brief period of API timeouts (approx. 08:19 UTC - 08:26 UTC). We apologize for the disruption, and are working to provision more capacity as a result.
Notice April 7, 2025 Errors on invoice and customer portal usage graphs 20h 28m
Started Apr 07, 2025, 07:00 AM UTC· resolved Apr 08, 2025, 03:28 AM UTC
- resolved Apr 07, 2025, 06:32 PM UTC
From 04/07/25 07:14 to 04/07/25 15:28 UTC, customers attempting to load usage graph data for upcoming invoices in their customer portal or on the invoice portal would have encountered an error that prevented the graph from rendering. This did not impact internal views of the invoice, calculations of invoice or usage amounts, invoice issuance, or usage ingestion. The cause of this issue was a recently modified front-end component that did not properly account for data routing behavior in external portal views. The issue was resolved at 15:28am UTC.
Notice April 1, 2025 Elevated API error rates –
Started Apr 01, 2025, 08:00 PM UTC· resolved Apr 01, 2025, 08:00 PM UTC
- resolved Apr 01, 2025, 08:26 PM UTC
From approximately 12:57pm PT to 13:06pm PT on 2025-04-01, customers may have seen an elevated rate of API response errors. The cause of this issue was due to an increase is connection timeouts to a system cache for usage data. This issue has since been resolved and all connections have been restored.
Notice March 31, 2025 Elevated API response times and error rates –
Started Mar 31, 2025, 07:17 PM UTC· resolved Mar 31, 2025, 07:17 PM UTC
- resolved Mar 31, 2025, 08:01 PM UTC
From approximately 12:17pm PT to 12:22pm PT, customers may have seen an elevated rate of API response times and errors. The cause of this issue has been resolved. The root cause of the issue was due to elevated latency on our event data store.
Notice March 21, 2025 Error loading customer pages 26m
Started Mar 21, 2025, 08:40 PM UTC· resolved Mar 21, 2025, 09:06 PM UTC
- resolved Mar 21, 2025, 09:43 PM UTC
Some customers may have seen errors loading customer pages via the Orb dashboard from 20:40 UTC to 21:06 UTC. This was due to deploy skew, and is now resolved. Note that no other Orb functionality was impacted, including invoice issuance, event ingestion, and subscription management actions.
Minor February 28, 2025 Event ingestion delays 1h 19m Ingest
Started Feb 28, 2025, 07:43 PM UTC· resolved Feb 28, 2025, 09:03 PM UTC
- investigating Feb 28, 2025, 07:43 PM UTC
We are currently investigating delays in our event ingestion pipeline.
- monitoring Feb 28, 2025, 08:08 PM UTC
Ingestion delays are recovering and we are actively monitoring the pipeline.
- resolved Feb 28, 2025, 09:03 PM UTC
This incident has been resolved.
Notice February 28, 2025 Elevated API and async error rates from 17:52 - 17:55 UTC 5m
Started Feb 28, 2025, 05:50 PM UTC· resolved Feb 28, 2025, 05:55 PM UTC
- resolved Feb 28, 2025, 06:05 PM UTC
We saw a temporarily latency and error rate spike from 17:52 to 17:55 UTC.
Notice February 7, 2025 Elevated API error rate 33m Reads, Ingest, Writes, Analytics
Started Feb 07, 2025, 11:10 PM UTC· resolved Feb 07, 2025, 11:43 PM UTC
- monitoring Feb 07, 2025, 11:10 PM UTC
From approximately 22:40 UTC to 22:43 UTC, our system experienced a high rate of API errors due to increase contention on a database. We have fully recovered and are continuing to monitor the affected database.
- resolved Feb 07, 2025, 11:43 PM UTC
Our systems continue to remain stable and the issue affecting the affected database is fully resolved.
Major January 28, 2025 Elevated error rates on subscription related endpoints 17m
Started Jan 28, 2025, 11:00 PM UTC· resolved Jan 28, 2025, 11:17 PM UTC
- resolved Jan 28, 2025, 11:28 PM UTC
Between 23:09 and 23:17 UTC, an unintended process caused elevated contention in our databases, leading to elevated response times and error rates. This incident has since been resolved. Event ingestion endpoints were not affected.
Notice January 5, 2025 Potential delays in "Usage Patterns and Balance Events" webhooks 51m Webhooks
Started Jan 05, 2025, 07:09 PM UTC· resolved Jan 05, 2025, 08:01 PM UTC
- monitoring Jan 05, 2025, 07:09 PM UTC
We have detected and are working to resolve delays for some customers for the following webhooks: - subscription.usage_exceeded - subscription.cost_exceeded - customer.credit_balance_depleted - customer.credit_balance_dropped - customer.credit_balance_recovered The underlying issue has been resolved and our systems are working to return our webhook latency back to normal. We expect a full recover within the next few hours.
- resolved Jan 05, 2025, 08:01 PM UTC
Webhooks latencies have returned to normal levels for all webhooks.
Notice December 10, 2024 Data Export Delays for Daily Line Item Revenue and Invoice Line Item Billing resources 3h 7m
Started Dec 10, 2024, 07:56 PM UTC· resolved Dec 10, 2024, 11:03 PM UTC
- monitoring Dec 10, 2024, 07:56 PM UTC
We've discovered an issue causing our data exports to be delayed for the Daily Line Item Revenue and Invoice Line Item Billing resources. We've identified the issue and are in the process of backfilling and exporting this data. We expect this to be fully resolved in the next few hours.
- resolved Dec 10, 2024, 11:03 PM UTC
All data has been backfilled for the Daily Line Item Revenue and Invoice Line Item Billing resources.
Notice November 28, 2024 Python SDK Warning –
Started Nov 28, 2024, 08:45 PM UTC· resolved Nov 28, 2024, 08:45 PM UTC
- resolved Nov 28, 2024, 08:45 PM UTC
A change in httpx v0.28.0 (released earlier today) affects fresh installs of Orb's Python SDK, and may cause API errors. Please either pin the httpx version to before 0.28.0 (via uv, requirements.txt, etc), or upgrade to version 2.17.0 of the Orb Python SDK.
Notice November 26, 2024 Elevated rate of dashboard failures 7m
Started Nov 26, 2024, 07:29 PM UTC· resolved Nov 26, 2024, 07:37 PM UTC
- investigating Nov 26, 2024, 07:29 PM UTC
We are currently investigating an issue with an upstream auth provider on access to the Orb admin dashboard.
- monitoring Nov 26, 2024, 07:31 PM UTC
The upstream provider has identified the issue; we're continuing to monitor for updates.
- monitoring Nov 26, 2024, 07:35 PM UTC
The upstream issue is resolved, and we're seeing recovery. We'll continue to monitor but no longer believe Orb users continue to be impacted.
- resolved Nov 26, 2024, 07:37 PM UTC
This incident has been resolved, and the Orb dashboard should be functioning as normal.
Notice October 29, 2024 Elevated API errors and delayed async processing 16h 34m Reads, Writes, Analytics
Started Oct 29, 2024, 01:09 PM UTC· resolved Oct 30, 2024, 05:44 AM UTC
- investigating Oct 29, 2024, 01:09 PM UTC
A query load issue has been identified starting at 2024/10/29 8:45 UTC which was causing intermittent spikes of errors accessing internal data storage, resulting in API errors and delays in async processing. This issue has been mitigated for the majority of customers as we continue to investigate.
- monitoring Oct 29, 2024, 02:12 PM UTC
API services stabilized at 12:30 UTC and async processing has been restored, and we are continuing to monitor our systems.
- monitoring Oct 29, 2024, 07:39 PM UTC
API services and asynchronous processes are stable. The query load issue has been fully isolated and we're working on improvements to prevent this issue in the future.
- resolved Oct 30, 2024, 05:44 AM UTC
This incident has been resolved.