Resolved -
This incident has been resolved.
Jun 16, 21:08 UTC
Update -
Refresh across all executions is nearing completion with no additional impact observed. Incident is fully mitigated and resolved.
Jun 16, 21:07 UTC
Update -
Refresh across all executions is nearing completion with no additional impact observed. Incident is fully mitigated and resolved.
Jun 16, 21:00 UTC
Update -
The last few namespaces are still refreshing. We are starting to patch the code, and are monitoring the situation.
Jun 16, 15:56 UTC
Update -
Refresh operations across both open and closed workflows are complete in most regions; we continue to make steady progress across the remaining few namespaces in the us-east-1, us-west-2, us-east4, us-west1, us-central1, asia-south1, europe-west3. No additional impact has been observed since mitigation was implemented, and we continue to closely monitor the environment. The next update will be provided once the refresh process has completed.
Jun 11, 20:25 UTC
Update -
Refresh across open and closed workflows continues to progress smoothly. Once this is done, the incident will be resolved.
Jun 8, 19:02 UTC
Update -
Refresh across open and closed workflows continues to progress smoothly. We're monitoring the situation and will post updates once complete.
Jun 5, 22:02 UTC
Update -
As refresh of closed workflows progresses, some users may see unblocked stuck parent workflows from closed child workflows; this is happening because the missed task for closed child workflows is refreshed and delivered to parent. It could also happen on other stuck workflows that are waiting for task updates which were missed due to this incident.
Jun 4, 22:35 UTC
Update -
We are continuing to monitor the ongoing refresh process for all open workflows. We are also considering options for refreshing closed workflows. We will continue to share more updates as this progresses.
Jun 4, 17:40 UTC
Monitoring -
Workflows will continue to be refreshed overnight, and we are monitoring the situation.
Jun 4, 07:08 UTC
Update -
Workflows will continue to be refreshed overnight, and we are monitoring the situation.
Jun 4, 06:45 UTC
Update -
Refreshes are still in progress and we are monitoring the situation.
Jun 4, 06:14 UTC
Update -
Rollback is complete; refresh of open workflows is underway to unblock impacted workflows.
Jun 4, 04:27 UTC
Update -
The rollback is making progress across impacted cells, and we are continuing to monitor the situation.
Jun 4, 01:40 UTC
Update -
Rollback is still in progress and we are continuing to monitor the situation.
Jun 3, 23:36 UTC
Identified -
A bug in the recent update in History pagination caused a rare edge case where an extra task could be added before the response was returned. This incorrectly advanced the read level, causing some tasks to be skipped. We’ve identified the issue, and are currently rolling back the update to stop this from happening again.
We will be refreshing all executions in cells where the update was deployed and will monitor these closely. This will progress any stuck workflows, but will not impact any of the closed ones.
Jun 3, 21:35 UTC
Update -
We're rolling back the recent update as a possible mitigation to avoid any further occurrences of this issue. We are however still in the process of investigating root cause and will post updates as we have them.
Jun 3, 20:41 UTC
Update -
We are investigating a recent update that could have caused intermittent missed task dispatches, resulting in some tasks being stuck. Impact identified as of now is limited to a handful of workflows that were caught in a rare edge case. We will continue to share more updates as investigation progresses.
Jun 3, 19:51 UTC
Update -
We are continuing to investigate this issue. Please report any stuck workflows or activities to support so we can help unblock them as we work on root cause investigation and resolution.
Jun 3, 18:30 UTC
Investigating -
We are investigating a few reports where a workflow or activity is becoming stuck and not making progress when it should. Please report any such issues you see to support.
Jun 3, 00:45 UTC