feat:Add dataset truncation to prevent deadlocks during training

[ved1beta]

What does this PR do?

• Add truncate_dataset and gradient_steps parameters to BatchSamplerShard
• Implement optimal dataset size calculation
• Add truncation logic to both split and non-split batch iterators
  Fixes Feature request: ability to truncate datasets on split rather than pad #2019

Before submitting

• Was this discussed/approved via a Github issue or the Please add a link
  to it if that's the case.
• Did you write any new necessary tests?

Who can review?

@SunMarc @muellerzr

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.