Replicate had 37 outages in the last 2 years totaling 164h 9m of downtime — averaging 1.5 incidents per month.
There were 37 Replicate outages since August 24, 2025 totaling 164h 9m of downtime. Each is summarised below — incident details, duration, and resolution information.
Status: Identified The API for creating predictions for A100s is currently degraded as a piece of backing infrastructure failed. We are working on getting it back online
Status: Resolved We have restarted the affected component and service should be back to normal. Thank you for your patience
Status: Identified Models that pull from huggingface during setup have not been succeeding, which is resulting in scale-out delays and queue backups. Affected components A100 Hardware (Degraded performance) H100 Hardware (Degraded performance) L40S Hardware (Degraded performance)
Status: Identified We have confirmed that turning off our caching layer for huggingface downloads allows models to successfully set up, so we are making that change for all models while we continue to troubleshoot. Affected components H100 Hardware (Degraded performance) L40S Hardware (Degraded performance) A100 Hardware (Degraded performance)
Status: Monitoring All affected hardware types have recovered and we are continuing to monitor and troubleshoot. Thanks for your patience! Affected components H100 Hardware (Operational) L40S Hardware (Operational) A100 Hardware (Operational)
Status: Resolved We're looking stable now, and with caching back on. Thanks again for your patience! Affected components A100 Hardware (Operational) H100 Hardware (Operational) L40S Hardware (Operational)
Status: Identified We are over provisioned on H100s currently which is causing long queue times for some models running on H100s Affected components H100 Hardware (Degraded performance) Official Models (Degraded performance)
Status: Monitoring GPU usage is back below capacity. We're continuing to monitor. Affected components H100 Hardware (Degraded performance) Official Models (Degraded performance)
Status: Monitoring This issue is now resolved Affected components H100 Hardware (Operational) Official Models (Operational)
Status: Resolved This issue is resolved Affected components H100 Hardware (Operational) Official Models (Operational)
Status: Investigating We are aware of 504s being returned by models that reach out to HuggingFace during setup. We believe this is likely related to the disruption visible at https://status.huggingface.co/. We are monitoring and will update when we can identify the HuggingFace CDN is back up/
Status: Monitoring HuggingFace is reporting recovery, which we are also seeing on our end. However, we still see minor degradation, which we believe is due to thundering herd once the main issue was resolved. We are still keeping an eye out, but believe we are recovering
Status: Monitoring We have seen most HuggingFace calls start succeeding, except for a small subset that are being downloaded through cas-bridge.xethub.hf.co "cas-bridge.xethub.hf.co": http://cas-bridge.xethub.hf.co. Most models are now succeeding to spin up
Status: Monitoring Still seeing signs of improvement, most models are able to spin up with a few still waiting their turn from HuggingFace. Models should spin up after enough tries if they are failing for HuggingFace related reasons.
Status: Resolved We have not seen elevated rates of HuggingFace/model setup errors for a couple hours now, so we believe this incident is cleared
Status: Investigating Long queue times, especially on BFL models Affected components H100 Hardware (Degraded performance)
Status: Monitoring We pushed a fix and are monitoring Affected components H100 Hardware (Degraded performance)
Status: Resolved H100 capacity has returned to normal levels Affected components H100 Hardware (Operational)
Status: Monitoring We are seeing high contention on H100 hardware which is resulting in delays on predictions and scale-out. Affected components H100 Hardware (Degraded performance)
Status: Resolved We are back under max capacity for H100 hardware. Thank you for your patience! Affected components H100 Hardware (Operational)
Status: Identified We recently received a sharp increase in demand for H100 capacity which is resulting in delayed scale-out and queue backup. Affected components H100 Hardware (Degraded performance)
Status: Resolved The H100 capacity is back in good health. Thanks for your patience! Affected components H100 Hardware (Operational)
Status: Investigating We're seeing long setup times and high contention for models on some L40S and H200 clusters. Affected components H100 Hardware (Partial outage) L40S Hardware (Partial outage)
Status: Investigating System is back to operating normally Affected components H100 Hardware (Partial outage) L40S Hardware (Partial outage)
Status: Resolved System is back to operating normally Affected components H100 Hardware (Operational) L40S Hardware (Operational)
Status: Investigating Long queue times for black-forest-labs/flux-2-klein-4b resulting in canceled predictions Affected components Official Models (Degraded performance)
Status: Resolved This issue has been resolved and queue times are back to normal Affected components Official Models (Operational)
Status: Identified Our message queues for prediction and training status updates are hitting capacity limits which are causing connection failures for queue consumers. We are in the process of bringing additional capacity online. Affected components Playground (Degraded performance) H100 Hardware (Degraded performance) HTTP API (Degraded performance) T4 Hardware (Degraded performance) CPU Hardware (Degraded performance) Home Page (Degraded performance) L40S Hardware (Degraded performance) A100 Hardware (Degraded performance) Streaming API (Degraded performance)
Status: Resolved Message flows are healthy. Affected components L40S Hardware (Operational) A100 Hardware (Operational) Streaming API (Operational) HTTP API (Operational) T4 Hardware (Operational) CPU Hardware (Operational) Home Page (Operational) Playground (Operational) H100 Hardware (Operational)
Status: Identified We are seeing heightened demand for H100 hardware which is causing severe queue delays. Affected components H100 Hardware (Partial outage)
Status: Resolved H100 hardware contention has resolved. Thank you for your patience! Affected components H100 Hardware (Operational)
Status: Identified Demand for constrained H100 hardware is causing scaling delays. This impacts queue size and inference speed for any models running on H100s. Affected components H100 Hardware (Degraded performance)
Status: Resolved There is no more contention for H100 hardware. Thank you for your patience! Affected components H100 Hardware (Operational)
Status: Monitoring All predictions and trainings targeting A100 hardware are experiencing degraded performance while control plane nodes restart. Affected components A100 Hardware (Degraded performance)
Status: Resolved All A100 capacity is back. Thanks for your patience! Affected components A100 Hardware (Operational)
Status: Investigating The persistent storage for all A100 hardware is under maintenance and is expected to be degraded until completion. Affected components A100 Hardware (Partial outage)
Status: Resolved The maintenance is complete and all systems are reporting healthy. Thank you for your patience! Affected components A100 Hardware (Operational)
Status: Identified Some Black Forest Labs models are failing due to downstream errors from BFL. We are monitoring the situation and working on work arounds. BFL Status page: https://status.bfl.ml/ Affected components Official Models (Degraded performance)
Status: Resolved Black Forest Labs has resolved the issue. Affected components Official Models (Operational)
Status: Investigating We are investigating an outage that is only affecting Flux Schnell.
Status: Monitoring A GPU provider has an outage. Traffic is being rerouted and we are processing new Flux Schnell requests.
Status: Resolved Flux Schnell requests are being served normally.
Status: Investigating We are currently investigating why a large number of models are not currently processing requests, with predictions stalled with a "starting" status. Affected components Streaming API (Partial outage) HTTP API (Partial outage) Official Models (Partial outage)
Status: Monitoring We have identified the root cause and have made an update. We are continuing to monitor as we start to see things improve. Affected components Streaming API (Degraded performance) HTTP API (Degraded performance) Official Models (Degraded performance)
Status: Resolved Models are once again operational Affected components Streaming API (Operational) HTTP API (Operational) Official Models (Operational)