[REQUEST] Exposing Parallelism rank as environment Variable for integration with profiling tools like CUPTI

[WoosungMyung]

First, thank you for your amazing work on this project and for being open to community feedback.

I would like to propose a small enhancement that could significantly improve compatibility with low-level profiling tools such as NVIDIA CUPTI or Intel VTune.

Motivation)

Tools like NVIDIA CUPTI rely on environment variables by Torch like RANK, WORLD_SIZE, or LOCAL_RANK to understand the role of each process in a distributed training job. However, It is not easy to know the DP Rank
from accelerate easily.

Proposion)

After initializing the process group (e.g., via torch.distributed.init_process_group or Accelerator()),
each process can set the following environment variables (locally) just once:

import torch.distributed as dist

if dist.is_initialized():
    os.environ["DP_RANK"] = str(dist.get_rank())
    os.environ["DP_WORLD_SIZE"] = str(dist.get_world_size())

This would allow external profilers to correlate collected GPU kernel events with the Data Parallel rank of each process, which makes User understanding the bottleneck of Training.

Thank you once again for your time and for considering this suggestion.
I'd be happy to submit a PR if this is aligned with your design direction.

Best regards,
[Woosung Myung]

[S1ro1]

@WoosungMyung We can probably add a feature like that in #3651 which kind of generalizes parallelism in Accelerate, is that something you'd like? I.e. DP_RANK, PP_RANK, CP_RANK etc? I hope to have it merged by next-next release so it's not entirely close, though I'd like to avoid temporary code changes for 1 version

[WoosungMyung]

Thank you for the thoughtful response!

Yes, I agree that aligning with #3651 for generalizing parallelism roles (DP/PP/CP) makes sense from a long-term design perspective.

My intention with this proposal was to provide a simple hancement that would help low-level profilers like CUPTI or VTune correlate kernel activity with process roles today.

If a temporary solution is not desirable for the current release cycle, I’m happy to align with the broader direction in #3651. Please feel free to tag me when the discussion moves forward — I'd still be happy to contribute or help implement the env export logic once the direction is settled.

Thanks again!

[S1ro1]

@WoosungMyung I'm fine with a faster temporary solution, though it is a bit not as straightforward due to different parallelisms (FSDP, DeepSpeed, DDP, TP...) etc. present in the accelerate codebase and us not having a central place. I don't have capacity to figure out a temporary solution sadly, but if you manage to get something ready I'd love to assist you with reviews etc. If you won't, I plan to release the parallelism config hopefully by the end of the month, so it should be in pretty soon-ish.

[WoosungMyung]

@S1ro1
Thank you so much for taking the time to share your thoughts.

I’ve faced several challenges when trying to use Hugging Face with distributed training.
I’m happy to wait for your upcoming parallelism config release.
If I have additional suggestion at that time, I would be happy to open a PR at that time.

I'll wait for your work. Thanks

[S1ro1]

This should be closed by #3702. Reopen if needed but I think we don't want to change the user ENV, rather let user decide if they need it.