Orb Outage History

A fix has been implemented and we are monitoring the results.

Primary impact was between 19:43 and 19:50 UTC (with partial/intermittent recovery in between those times as well). We will resolve this status page incident within the next hour once we confirm there are no remaining issues.

We're seeing recovery in these workloads. Although delayed, we do not expect any lost alerts or webhooks from this incident. We'll continue to update.

We're continuing to see recovery as we drain our backlog. We apologize for the issue.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring as latency is resolved in async services

We're seeing recovery as of 12:33 AM UTC, and are continuing to monitor for impact.

We identified elevated latencies for fetching usage (and potentially some associated actions that required invoicing via manual action). The vast majority of the impact was from April 3 21:22 to 21:26 UTC. Impact was fully mitigated for a remaining small (<1%) of errors by 21:39.

Following a deployment at 5:40 UTC, the Orb API started experience elevated timeouts for applying and cancelling subscription pending changes. We have rolled back and errors have subsided fully as of 12:55 UTC, and we are continuing to monitor.

We have applied a fix and are monitoring recovery.

We observed an increase in page load failures in the Orb dashboard and a subset of APIs (5%) starting at 02:25 UTC for approximately 10 minutes. Services are now recovered, and we are continuing to monitor. We will provide more detailed updates to affected customers. Data ingestion was not impacted.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

We have applied a fix and will continue to monitor invoice issuance.

Continuing to see quick recovery on API latencies. We'll provide any further updates in 10 minutes, or close out this incident.

A recent code deploy caused a minor increase in analytics request errors. Our automated canary systems rolled this back as part of our deploy process without any operator intervention. We're continuing to monitor to ensure errors do not recur, and the offending logic is not deployed. This did not impact writes or data ingestion.

We're seeing recovery and are continuing to monitor.

We experienced an elevated level of API errors, which has now subsided. We are actively monitoring for further errors.

We've identified a failure mode tied to a degenerate workload that degrades query performance for one of our usage datastores. A temporary resolution has been put in place to ensure system stability, our oncall team will continue to monitor the error rates for the API and adjust workloads as needed to ensure API availability.