Workflow scheduling doesn't keep track of progress within a step.

[jmchilton]

From @nsoranzo on Gitter:

I have huge issues trying to schedule a workflow which executes a series of tools on ridiculously large (21989 elements) collections. If at some point a job handler is restarted it seems it restarts already (partially) completed steps from scratch, in fact a particular step/tool has already 114,557 jobs on the history (in various states).

So the problem is we flush invocation step states only after the whole step has been scheduled. With a collection that size though you could imagine scheduling a single step to take an hour - so it is very possible a hard restart of Galaxy could cause problems like this.

[nsoranzo]

For what I can see in the database, the workflow_invocation_step table is filled for a step only after all the jobs for it have been scheduled, in fact doing:

SELECT COUNT(id), workflow_step_id FROM workflow_invocation_step WHERE workflow_invocation_id=588 GROUP BY workflow_step_id;

I have a count 21989 for a completed step and nothing for currently scheduling step.

[jmchilton]

@nsoranzo Yes - as a first order approximation of how this should be working that made some sense. I used the the existence of a workflow_invocation_step to determine if the step had been scheduled or not I believe - so for efficiency and consistency it made sense to bulk commit those along with the probably the implicit output collections associated with that step.

We probably need to do a few things here to make it a bit better:

• Add an attribute called perhaps committed to workflow_invocation_step.
• Push workflow_invocation_step creation logic into the bulk tool execution/collection stuff so we get an "uncommitted" workflow_invocation_step with each job creation. Right now the only thing we do differently when executing jobs in the job execution code for workflow invocations is to add a workflow invocation UUID to the job parameters for scheduling workflows together if one wants - we can follow a similar pattern and create the workflow_invocation_step deep in the execution code.
• Rework the bulk tool execution/collection stuff to recover "uncommitted" workflow_invocation_steps and pull the job information back out if needed.
• At the end instead of creating all the workflow_invocation_steps - we just switch them to committed=True (which should be a pretty quick database operation).
• Modify the schedule tracking stuff to determine a workflow step has been "scheduled" when there are committed workflow_invocation_steps.