Is Bitmovin down?

We are currently investigating an issue affecting Cloud Connect customers running encodings on AWS. Encoding instances may fail to start due to image authorization errors when pulling required images in customer AWS accounts. Encodings on other cloud providers and Bitmovin-managed infrastructure are not affected. We will provide an update as soon as we have more information..

We have identified the root cause of the AWS image authorization issue affecting Cloud Connect customers and are currently rolling out a fix. We will provide a further update once the fix has been fully deployed..

This incident has been resolved. Permissions to the encoder AMIs have been restored and encodings for Cloud Connect customers on AWS have been running normally since the fix was deployed..

This incident has been resolved. Permissions to the encoder AMIs have been restored and encodings for Cloud Connect customers on AWS have been operating normally since 15:46 UTC. We will publish a detailed root cause analysis within 24 hours. We apologize for any inconvenience caused..

# Root Cause Analysis: AWS Image Authorization Issues for Cloud Connect Customers On August 11, AWS Cloud Connect customers temporarily lost access to the encoder AMIs, causing encodings to fail at instance startup. The cause was a recent change to our infrastructure management API that altered internal request routing. As a result, an AMI access synchronization step in our encoder release process received an empty customer list and revoked customer AMI permissions. A first, partially completed run affected a small number of customers starting at 14:06 UTC; a second run at 14:58 UTC revoked access for all remaining customers. Access was fully restored by 15:46 UTC. No data was lost. ## Timeline (UTC) * **14:06** \- An encoder release process runs. Its AMI access synchronization step queries the list of Cloud Connect customers, receives an empty result, and begins revoking customer access to the encoder AMIs. The release process is restarted, canceling the run partway through. A small number of customers are affected at this point. * **14:46** \- The first customer reports about failing encodings on AWS come in. Since only a small number of customers are affected at this point, the initial investigation focuses on customer-specific causes. * **14:58** \- The restarted release process completes. The same synchronization step again receives an empty customer list and revokes AMI access for all remaining customers. * **\~15:28** \- After the second synchronization run, internal monitoring alerts on failing encoding startups across all AWS Cloud Connect customers, making the wider scope of the issue visible. * **15:30** \- The revoked AMI permissions are identified as the cause and traced back to the release process. * **15:34** \- We trigger a re-run of the grant process with the correct customer list. * **15:46** \- The grant process completes. AMI access is restored and encodings start normally again. We continue monitoring to confirm stable operation. ## Root cause Our encoder release process includes a step that synchronizes customer access permissions to the encoder AMIs. It queries our infrastructure management API for the list of Cloud Connect customers and grants or revokes AMI access accordingly. In the week before the incident, we made changes to the infrastructure management API to improve resiliency and redundancy. These changes also altered how requests are routed internally. When the release process ran on August 11, its customer list query was routed to an endpoint that returned an empty list with a successful response. Because the response was technically valid, no error was raised, and the synchronization step treated the empty list as the desired state and revoked customer AMI permissions. This happened in two runs: a first run that was canceled partway through when the release process was restarted, and a second run that completed and revoked access for all remaining customers. ## Resolution We re-ran the grant process with the correct customer list, which restored AMI access for all affected customers. Encodings started normally again as of 15:46 UTC, and we confirmed stable operation over the following hours. ## Preventive measures * We are adding safeguards to the AMI access synchronization so it fails instead of acting when the customer list is empty or shrinks beyond a defined threshold. * We are removing ACL management from the encoder release process entirely. Customer AMI access is already managed by our infrastructure service, and the release process was an unintended second place where these permissions could be changed. Consolidating this into a single owner eliminates this class of failure. * We are improving our detection for this failure mode so that permission changes of this kind are alerted on directly, rather than indirectly through failing encodings. This will let us identify the scope of similar issues faster, even when only few customers are initially affected. We apologize for the disruption this caused. If you have questions about this incident, please reach out to our support team..

Starting at 14:33 CEST, our core Bitmovin Observability cluster is experiencing an outage. Ingestion is unaffected and incoming data is being buffered, so no data loss is expected. Querying (dashboards and API) and data export are currently unavailable. We are investigating and will post an update within 30 minutes..

The database cluster has recovered. Querying (dashboards and API) and data export have resumed, and incoming data is being written normally again. We are now triggering the backfill of the buffered data. Until it completes, recent data may be incomplete in queries and exports. We are monitoring the results and will confirm once the backfill has finished..

As of 15:17 CEST, the backfill has completed and all buffered data is available. All systems are operational again. No data was lost. We will provide a root cause analysis as soon as possible..

# RCA: Bitmovin Observability query and export outage (27 July 2026) On 27 July 2026, Bitmovin Observability was unable to serve queries or data exports for approximately 20 minutes, with reduced data completeness for a further 24 minutes. Data ingestion was never interrupted, incoming data was buffered throughout, and no customer data was lost. This post explains what happened, what we have done, and what we are changing. ## Impact Between 12:33 and 12:53 UTC, the Bitmovin Observability query API, dashboards and data export were unavailable. Queries and exports resumed at 12:53\. Data collected during the outage was buffered and then backfilled, completing at 13:17 UTC. Until that point, queries covering the affected period could return incomplete results. Data ingestion and collection were not affected at any time, and there was no data loss. ## What happened An internal query executed against our observability database cluster triggered a defect in the database engine. The memory used by this query was not correctly accounted for, which meant the cluster's memory safeguards never engaged. Instead of the query being limited or rejected, memory was consumed until nodes became unresponsive, and the failure propagated across the cluster rather than staying contained to the node serving the query. Automated alerting notified our on-call engineers within 30 seconds. The cluster was restarted at 12:43 and completed recovery of all shards, resuming full write acceptance, at 12:53\. Buffered data was then backfilled and fully available at 13:17. ## Timeline (UTC) | Time | Event | | ----- | ---------------------------------------------------------------------------------------- | | 12:32 | An internal query is executed against the observability database cluster | | 12:33 | First query and export errors observed | | 12:34 | The cluster becomes unavailable, automated alerts fire and engineers begin investigating | | 12:43 | Cluster restart initiated | | 12:53 | All shards recovered, writes fully accepted, queries and exports resume | | 13:17 | Backfill of buffered data complete, all systems operational | ## What we are changing * The query that triggered the defect was an ad hoc analytical query run by an operator, not a query our application issues in normal operation. We have instructed our engineering team not to run this class of query against the production cluster until the underlying defect is resolved. * We are reporting the defect upstream and validating whether a fix is available in a newer version of the database engine. Our ingestion buffering worked exactly as designed and protected customer data. We apologise for the disruption and for the incomplete data during the backfill window..