Is Inngest down?

Status: Investigating We are actively investigating an issue with function execution that is leading to delays starting function runs after events are received. We will provider further updates as we identify the cause and resolve the issue. Affected components Function execution (Partial outage)

Status: Identified We have identified the cause of the issue - the team has deployed a fix and is scaling to catch up on function delays. Affected components Function execution (Partial outage)

Status: Monitoring New function execution has resumed and nearly caught up from the incurred backlog. Wait for event and cancel on handlers are still in a backlog and are catching up with processing. No events were dropped during this issue, all functions will be executed, but with delays. The root cause has been confirmd and changes are underway to prevent reoccurrence. Affected components Function execution (Degraded performance)

Status: Resolved The incident is now resolved and the system is full operational. Delays began around 14:03 UTC caused by an issue publishing to a new Kafka topic. Recovery began around 14:40 UTC as the team was able to isolate this topic. The consuming services which schedules new function runs was scaled out and began to consume the backlog. No events were dropped during this incident, all functions were scheduled, but with delays during this time window. No action is needed for manual recovery. Affected components Function execution (Operational)

Status: Investigating We are actively investigating an issue with our metrics service. Metrics displayed in the Inngest UI may not be up to date, but core platform functionality is unaffected and function execution is operating normally. We will provide further updates as we identify the cause and resolve the issue. Affected components Observability (Partial outage)

Status: Monitoring A fix has been deployed and we're monitoring the system to ensure the system is fully operational. We confirmed all executions were functioning as expected, so this only impacts the metrics on the dashboard. Affected components Inngest Dashboard (Degraded performance) Observability (Partial outage)

Status: Resolved Metrics in the UI are now restored. Users may notice metrics data continue to be unavailable during the time of the incident. Affected components Observability (Operational) Inngest Dashboard (Operational)

Status: Investigating We are actively investigating an issue with latency and execution errors. We will provider further updates as we identify the cause and resolve the issue. Affected components Function execution (Degraded performance)

Status: Monitoring We identified issues in the system and have deployed changes to mitigate the issues. Throughput has increased across the system. We continue to monitor latency as well as we expect it to reduce with the throughput increases. We are continuing to investigate for any additional issues that may have occurred. Affected components Function execution (Degraded performance)

Status: Resolved Throughput and latency have recovered and the system should be fully operational again. The core issue is related to the part of the system that persists run state. Some operations experienced and increase in errors and retries, causing a backlog in some parts of the system and some failed operations like checkpoints or signals. We are continuing a thorough post-mortem investigation to prevent reoccurrance. Affected components Function execution (Operational)

Status: Investigating We are investigating a partial outage affecting function triggering from incoming events. Some events may experience delays before triggering functions, or functions may not trigger as expected. Our team is actively investigating and will provide updates as more information becomes available. Affected components Function execution (Partial outage)

Status: Monitoring We've made updates to the system, and function throughput is continuing to improve for users affected. The backlog is actively being processed, and we're closely monitoring recovery to ensure processing returns to normal. Affected components Function execution (Operational)

Status: Monitoring We are continuing monitoring the system. Users affected by this incident may need to replay impacted events to ensure their associated functions are executed. Affected components Function execution (Operational)

Status: Resolved The incident is now resolved and the system is full operational. Affected components Function execution (Operational)

Status: Investigating We are actively investigating degraded function execution performance. We will provider further updates as we identify the cause and resolve the issue. Affected components Function execution (Partial outage)

Status: Monitoring A fix has been deployed and we're monitoring the system to ensure the system is fully operational. Affected components Function execution (Partial outage)

Status: Resolved The incident is now resolved and the system is full operational. Affected components Function execution (Operational)