PreTrain-T5-[get_samples_mapping]-[ RuntimeError: Socket Timeout ]

[Hanlard]

In the megatron\data\dataset_utils.py "get_samples_mapping" function, the following line had run for 12 hours.
samples_mapping = helpers.build_mapping(..)

And this line only runs for rank=0, so the next line torch.distributed.all_reduce(counts, ... ) reports [ RuntimeError: Socket Timeout ]

[bugface]

I had a similar issue when using DDP and model parallel.
I solved the issue as
set timeout to a longer time (default is 30min) in the torch.distributed.init_process_group (megatron/initialize.py at line 183)

torch.distributed.init_process_group( 
  backend=args.distributed_backend, 
  timeout=datetime.timedelta(0, 3600),  ## notice here I change the timeout to 2 hours 
  world_size=args.world_size, 
  rank=args.rank, 
  init_method=init_method 
)

[github-actions]

Marking as stale. No activity in 60 days. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

Marking as stale. No activity in 60 days.