4-card A100 training display cuda out of memory

[julia-0105]

Hello, I used 4-card A100 to train hunyuan model but cuda out of memory was displayed. I noticed that 8 cards were used in your source code. I would like to ask you:

1. Can 4-card A100 with a total memory of 80G train hunyuan model in lora mode?
2. Is the 40GB maximum memory mentioned in GitHub a single card or 8 cards overall memory?
   Appreciate and look forward to your reply 😊

[sayakpaul]

Thanks for your interest!

Can you share your env details with diffusers-cli env? Also, please share your training command.

We have landed DeepSpeed support in #139. Do you want to check if using DeepSpeed solves your problem?

[julia-0105]

Thanks for your interest!

Can you share your env details with diffusers-cli env? Also, please share your training command.

We have landed DeepSpeed support in #139. Do you want to check if using DeepSpeed solves your problem?
Thank you for your reply, I have used the Deepspeed method, but the problem has not been solved, and an error is reported:Less than two GPU ids were configured and tried to run on on multiple GPUs.There are my process.sh and diffusers-cli env:
`#!/bin/bash
export PYTHONWARNINGS="ignore"

export TORCH_LOGS="+dynamo,recompiles,graph_breaks"

export TORCHDYNAMO_VERBOSE=1

export WANDB_MODE="offline"
export NCCL_P2P_DISABLE=1
export TORCH_NCCL_ENABLE_MONITORING=0
export FINETRAINERS_LOG_LEVEL=DEBUG
#export CUDA_VISIBLE_DEVICES=2

GPU_IDS="0，1"

DATA_ROOT="/home/azure/workspace_julia_02/finetrainers/video-dataset-disney"
CAPTION_COLUMN="/home/azure/workspace_julia/finetrainers/prompt.txt"
VIDEO_COLUMN="/home/azure/workspace_julia/finetrainers/videos.txt"
OUTPUT_DIR="/home/azure/workspace_julia/finetrainers/videos_output"

Model arguments

model_cmd="--model_name hunyuan_video
--pretrained_model_name_or_path hunyuanvideo-community/HunyuanVideo"

Dataset arguments

dataset_cmd="--data_root $DATA_ROOT
--video_column $VIDEO_COLUMN
--caption_column $CAPTION_COLUMN
--id_token afkx
--video_resolution_buckets 16x384x512
--caption_dropout_p 0.05"

Dataloader arguments

dataloader_cmd="--dataloader_num_workers 0"

Diffusion arguments

diffusion_cmd=""

Training arguments

training_cmd="--training_type lora
--seed 42
--mixed_precision bf16
--batch_size 1
--train_steps 5000
--rank 128
--lora_alpha 128
--target_modules to_q to_k to_v to_out.0
--gradient_accumulation_steps 1
--gradient_checkpointing
--checkpointing_steps 1000
--checkpointing_limit 2
--enable_slicing
--enable_tiling"

Optimizer arguments

optimizer_cmd="--optimizer adamw
--lr 2e-5
--lr_scheduler constant_with_warmup
--lr_warmup_steps 100
--lr_num_cycles 1
--beta1 0.9
--beta2 0.95
--weight_decay 1e-4
--epsilon 1e-8
--max_grad_norm 1.0"

Validation arguments

validation_cmd="--validation_prompts "afkx A baker carefully cuts a green bell pepper cake on a white plate against a bright yellow background, followed by a strawberry cake with a similar slice of cake being cut before the interior of the bell pepper cake is revealed with the surrounding cake-to-object sequence.@@@49x512x768:::afkx A cake shaped like a Nutella container is carefully sliced, revealing a light interior, amidst a Nutella-themed setup, showcasing deliberate cutting and preserved details for an appetizing dessert presentation on a white base with accompanying jello and cutlery, highlighting culinary skills and creative cake designs.@@@49x512x768:::afkx A cake shaped like a Nutella container is carefully sliced, revealing a light interior, amidst a Nutella-themed setup, showcasing deliberate cutting and preserved details for an appetizing dessert presentation on a white base with accompanying jello and cutlery, highlighting culinary skills and creative cake designs.@@@61x512x768:::afkx A vibrant orange cake disguised as a Nike packaging box sits on a dark surface, meticulous in its detail and design, complete with a white swoosh and 'NIKE' logo. A person's hands, holding a knife, hover over the cake, ready to make a precise cut, amidst a simple and clean background.@@@61x512x768:::afkx A vibrant orange cake disguised as a Nike packaging box sits on a dark surface, meticulous in its detail and design, complete with a white swoosh and 'NIKE' logo. A person's hands, holding a knife, hover over the cake, ready to make a precise cut, amidst a simple and clean background.@@@97x512x768:::afkx A vibrant orange cake disguised as a Nike packaging box sits on a dark surface, meticulous in its detail and design, complete with a white swoosh and 'NIKE' logo. A person's hands, holding a knife, hover over the cake, ready to make a precise cut, amidst a simple and clean background.@@@129x512x768:::A person with gloved hands carefully cuts a cake shaped like a Skittles bottle, beginning with a precise incision at the lid, followed by careful sequential cuts around the neck, eventually detaching the lid from the body, revealing the chocolate interior of the cake while showcasing the layered design's detail.@@@61x512x768:::afkx A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage@@@61x512x768"
--num_validation_videos 1
--validation_steps 100"

Miscellaneous arguments

miscellaneous_cmd="--tracker_name finetrainers-hunyuan-video
--output_dir $OUTPUT_DIR
--nccl_timeout 1800
--report_to wandb"

cmd="accelerate launch --config_file accelerate_configs/uncompiled_8.yaml --gpu_ids $GPU_IDS train.py --mixed_precision bf16
$model_cmd
$dataset_cmd
$dataloader_cmd
$diffusion_cmd
$training_cmd
$optimizer_cmd
$validation_cmd
$miscellaneous_cmd"

echo "Running command: $cmd"
eval $cmd
echo -ne "-------------------- Finished executing script --------------------\n\n"
`


diffusers-cli env:

• 🤗 Diffusers version: 0.33.0.dev0
• Platform: Linux-6.5.0-1022-azure-x86_64-with-glibc2.35
• Running on Google Colab?: No
• Python version: 3.10.12
• PyTorch version (GPU?): 2.4.1+cu121 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.27.0
• Transformers version: 4.47.1
• Accelerate version: 1.2.1
• PEFT version: 0.14.0
• Bitsandbytes version: 0.45.0
• Safetensors version: 0.4.5
• xFormers version: not installed
• Accelerator: NVIDIA A100 80GB PCIe, 81920 MiB
  NVIDIA A100 80GB PCIe, 81920 MiB
  NVIDIA A100 80GB PCIe, 81920 MiB
  NVIDIA A100 80GB PCIe, 81920 MiB
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[julia-0105]

Thank you for your reply, I have used the Deepspeed method, but the problem has not been solved, and an error is reported:Less than two GPU ids were configured and tried to run on on multiple GPUs.There are my diffusers-cli env:
🤗 Diffusers version: 0.33.0.dev0
Platform: Linux-6.5.0-1022-azure-x86_64-with-glibc2.35
Running on Google Colab?: No
Python version: 3.10.12
PyTorch version (GPU?): 2.4.1+cu121 (True)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Huggingface_hub version: 0.27.0
Transformers version: 4.47.1
Accelerate version: 1.2.1
PEFT version: 0.14.0
Bitsandbytes version: 0.45.0
Safetensors version: 0.4.5
xFormers version: not installed
Accelerator: NVIDIA A100 80GB PCIe, 81920 MiB
NVIDIA A100 80GB PCIe, 81920 MiB
NVIDIA A100 80GB PCIe, 81920 MiB
NVIDIA A100 80GB PCIe, 81920 MiB
Using GPU in script?:
Using distributed or parallel set-up in script?:
cuda out of memory:

After Deepspeed error:

[sayakpaul]

What is your DeepSpeed config and training command?

[a-r-r-o-w]

@julia-0105 Could you upgrade your pytorch to 2.5.1 or above and give it a try? Less than that version are known to be problematic from a few other reported issues. 2.4.1 may give you completely black videos or run into OOM

[julia-0105]

Hello, I have upgraded torch to 2.5.1, but there is no difference between the trained lora video generation and the video generated by direct use of mixed elements. Do you have similar experience for my reference? In addition, I would like to ask you about the input of 70 6s30fps video in your project, but the code input 49 frames, how is this 49 set, how to sample from 180 frames? Thank you very much! Looking forward to your reply! Aryan ***@***.***>于2024年12月30日 周一15:17写道：

…

@julia-0105 <https://github.com/julia-0105> Could you upgrade your pytorch to 2.5.1 or above and give it a try? Less than that version are known to be problematic from a few other reported issues. 2.4.1 may give you completely black videos or run into OOM — Reply to this email directly, view it on GitHub <#162 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BNPWBUXS5WSE47RWAGL6B2L2IDXPZAVCNFSM6AAAAABULKTRHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRVGEYTAMJVGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[a-r-r-o-w]

I've added reproducible examples in https://github.com/a-r-r-o-w/finetrainers/tree/main/examples/training/sft, which have been tested on 2 or more setups with DDP/FSDP and confirmed to work on my end.

See release checkpoints at https://huggingface.co/finetrainers