Is Temporal down?

Some customers in AWS us-east-1 experienced a brief period of elevated latencies and unavailable responses between 17:45-18:08 UTC, and this has been resolved. We failed namespaces with MRN out of the cell. We are closely monitoring the situation, and we will fail the namespaces back into the cell when we are confident it is healthy.

This is still mitigated, and we are monitoring. We will fail the namespaces back into the cell when we are confident it is healthy.

This is still mitigated, and we are still monitoring. We have failed the namespaces back into their original location.

We are investigating a few reports where a workflow or activity is becoming stuck and not making progress when it should. Please report any such issues you see to support.

We are continuing to investigate this issue.

We are investigating a recent update that could have caused intermittent missed task dispatches, resulting in some tasks being stuck. Impact identified as of now is limited to a handful of workflows that were caught in a rare edge case. We will continue to share more updates as investigation progresses.

We're rolling back the recent update as a possible mitigation to avoid any further occurrences of this issue. We are however still in the process of investigating root cause and will post updates as we have them.

A bug in the recent update in History pagination caused a rare edge case where an extra task could be added before the response was returned. This incorrectly advanced the read level, causing some tasks to be skipped. We’ve identified the issue, and are currently rolling back the update to stop this from happening again. We will be refreshing all executions in cells where the update was deployed and will monitor these closely. This will progress any stuck workflows, but will not impact any of the closed ones.

Rollback is still in progress and we are continuing to monitor the situation.

The rollback is making progress across impacted cells, and we are continuing to monitor the situation.

Rollbacks are still in progress; refresh of open workflows is underway and will unblock impacted workflows.

Refreshes are still in progress and we are monitoring the situation.

Workflows will continue to be refreshed overnight, and we are monitoring the situation.

Workflows will continue to be refreshed overnight, and we are monitoring the situation.

We are continuing to monitor the ongoing refresh process for all open workflows. We are also considering options for refreshing closed workflows. We will continue to share more updates as this progresses.

As refresh of closed workflows progresses, some users may see unblocked stuck parent workflows from closed child workflows; this is happening because the missed task for closed child workflows is refreshed and delivered to parent. It could also happen on other stuck workflows that are waiting for task updates which were missed due to this incident.

Refresh across open and closed workflows continues to progress smoothly. We're monitoring the situation and will post updates once complete.

Refresh across open and closed workflows continues to progress smoothly. Once this is done, the incident will be resolved.

Refresh operations across both open and closed workflows are complete in most regions; we continue to make steady progress across the remaining few namespaces in the us-east-1, us-west-2,  us-east4, us-west1, us-central1, asia-south1, europe-west3. No additional impact has been observed since mitigation was implemented, and we continue to closely monitor the environment. The next update will be provided once the refresh process has completed.