metadata support for aws_batch_scheduler

[clumsy]

Description

AppDef.metadata is currently not implemented for aws_batch_scheduler. This feature request suggests adding it via AWS Batch Tags.

Motivation/Background

This allows adding metadata that is not available to the training jobs itself (unlike env variables). It could be used to filter/group AWs Batch Jobs but won't be pollute env variables for the training job.

Detailed Proposal

Currently there's no way to pass/change AWS Batch job tags for aws_batch_scheduler (unlike env variables). We can add AppDef.metadata key-value pairs as tags in here.
This way a custom component can pass the required key-value pairs via AppDef.metadata and they will be set on the AWS Batch job as tags.

Alternatives

• Using env variables could result in name clash and, more importantly, leak metadata into the training process itself (e.g. the flag can be used by a dependency).
• Implementing a custom scheduler on top of the standard aws_batch_scheduler just to pass custom tags seems like an overkill

Additional context/links

• torchx/torchx/schedulers/aws_batch_scheduler.py

  Line 515 in 13a4787

  "tags": {

[kiukchung]

Seems reasonable, would you be open to contributing a PR with this change? Feel free to assign me as the reviewer. Thanks!

[clumsy]

Thanks, @kiukchung I'll contribute a PR shortly.