Skip to content

[batch] Add job groups to the hailtop.batch interface #14610

Open
@daniel-goldstein

Description

@daniel-goldstein

What happened?

Batch now supports organizing jobs into a hierarchy through job groups, but this is so far only used internally by Query on Batch. Users who want to leverage this structure cannot do so through the hailtop.batch interface. All jobs created by e.g. b.new_job() are implicitly assigned to the "root" job group. We should add the following to hailtop.batch:

  • A JobGroup class to the public interface that shares any relevant methods that currently exist on the Batch class like new_{bash|python}_job, wait, etc.
  • A method on Batch to create a new job group and a method on JobGroup to create a child job group
  • A JobGroup should not have a run method, because there is no sound way in the lower-level batch client currently to only submit a subset of staged jobs/groups and it's unclear whether that behavior would ever be desired.

Version

0.2.132

Relevant log output

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions