Recover from quarantine and version state issue

[benmoriceau]

What

If a workflow is in a quarantined state, we need to allow to try to restart from a clean state. This clean state need to be reach through the cancellation of the running job and their related attempt. The only operation allowed on a quarantine workflow is to cancel the sync.

When in a quarantined state and receiving a cancel signal, we will need to "force" the restart of all the related jobs to a given job. It needs to:

• Get all the currently non terminal jobs
• Fail all those jobs and their related attempt
• Continue as new

In order to do that, if any manual operation is performed on a quarantined workflow we will:

• Terminate the current workflow
• Restart it which will trigger the cleaning of the state

Restart fail

If the update of the DB fails, we need to go back to a quarantined state.

Version issue state

if the workflow is not reachable, it can be because of a version issue. In order to handle those we will try to terminate all the workflow that are unreachable like we do for the quarantine ones.

DoD

If a workflow is in a quarantined stated, we can receive a signal to force them to be unstuck.

[davinchia]

This happens when the connection manager workflow is quarantined?