Inngest Outage History

Status: Identified We are currently experiencing degraded performance affecting a subset of users utilizing our REST API for retrieving runs and events data. Responses may be delayed or temporarily return incomplete data, with recent updates not appearing immediately. We have identified the cause of the issue. We're actively working on implementing a fix to resume normal operation of the REST API. Function execution is not affected and is working as expected. Affected components API (REST and GraphQL) (Degraded performance)

Status: Identified We have scaled up several resources across the system and to handle a large increase in scale within the system. Services are scaled up and we have also added new function state shards, but rollout of those new shards can take up to ~30m. We are also working on networking improvements to improve efficiency of the system with this significantly higher load. Affected components Function execution (Degraded performance)

Status: Identified We're deploying an in-memory optimization within the the part of the system the schedules new function runs. This optimization will alleviate pressure on underlying systems and increase throughput. The change will be rolled out momentarily. We're also working in parallel on a system change to create a dedicated service for processing for event batching which is the cause of the overall backlog on the system. Affected components Function execution (Degraded performance)

Status: Identified Changes have increase throughput, but not yet to typical levels. We are actively testing the new system change to decouple batch processing before enabling it for all accounts. Affected components Function execution (Degraded performance)

Status: Identified There is an increase in throughput since 15:37 UTC (~15 min ago). We are continuing to apply changes and prepare a larger change to decouple parts of the system. Event observability: Events may be delayed when appearing in the dashboard as the database ingestion for these events is also related to this part of the system that handles function scheduling. Events continue to be ingested and the Event API remains unaffected. Affected components Function execution (Degraded performance) Observability (Degraded performance)

Status: Identified The system is consuming the event backlog as fast as possible, with an ETA of ~10-15 minutes until function scheduling is caught up. After function scheduling is caught up, function execution in your account may still be limited by your account concurrency or a given function's own flow control settings (concurrency, rate limit, etc.). We will continue to share more updates as soon as we can. Affected components Function execution (Degraded performance) Observability (Degraded performance)

Status: Identified Function scheduled delays should be caught up. With events processed and new runs scheduled, your system my still see backlogs based on your function's flow control (e.g. concurrency) config and your account's concurrency. We still see backlogs in processing step.waitForEvent, step.invoke and cancelOn event expressions. We are continuing to work on this. We also are continuing our rollout of isolated batch processing as previously mentioned to further isolate parts of our system. EDIT - This was edited to include step.invoke as well for completeness. Affected components Function execution (Degraded performance) Observability (Operational)

Status: Identified Function execution scheduling and processing throughput is at normal levels. Async "pause" operations (step.waitForEvent, step.invoke, cancelOn) are severely backlogged which may cause delays in any of these operations from completing. This may cause issues with your function execution if you rely on them. The team is working on fixes and clear this backlog and fix the key issues. We do not yet have an ETA on resolving this specific issue. Affected components Function execution (Degraded performance) Observability (Operational)

Status: Identified The Inngest dashboard is down due to an issue with our downstream provider. Vercel. We are working quickly to bring this back up Affected components Inngest Dashboard (Full outage)

Status: Identified We area pushing a hotfix to the dashboard as recommended by Vercel's incident report. The rest of the Inngest system, function execution, API, etc. all remain functional. Affected components Inngest Dashboard (Full outage)

Status: Identified We have identified the cause of the issue affecting a core system queue. We are rolling our a mitigation now and preparing follow up changes. Affected components Function execution (Degraded performance)

Status: Identified We have deployed an additional hot fix. The earlier change rolled out have addressed the core issue and the system is now processing the backlog. The backlog is decreasing. We will provide another update as we have an estimate on time to recovery. Affected components Function execution (Degraded performance)

Status: Identified We have identified the cause of the issue with networking. We are quickly working to fix the issue. Affected components Function execution (Full outage)

Status: Identified We have identified the issue with some problem servers groups that reached saturation. These servers have been isolated and removed from the usage pool. The system is stable now as we bring additional capacity online for redundancy and overhead. Affected components Function execution (Degraded performance)

Status: Identified We have identified the cause of the issue. We're actively working on implementing a fix to resume normal operation of the system. Our internal networking teams are improving the scalability of NAT64 as we scale our services. Affected components Function execution (Degraded performance)

Status: Identified Function execution delays have returned for a subset of users. As we mitigated issues from earlier, some queue related slowness has returned affecting a subset of users running on certain shards. Affected components Function execution (Degraded performance)