Slow on "Loading checkpoint shards"

[CaledoniaProject]

I'm trying to load SD3.5-Medium to a g4dn.xlarge instance (T4, 16GB vRAM)

import os
import json
import torch

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-3.5-medium", 
    torch_dtype=torch.bfloat16
)
pipe = pipe.to('cuda')
pipe.enable_xformers_memory_efficient_attention()
pipe.enable_model_cpu_offload()

It's very slow here,

Loading checkpoint shards:  50%|███████████████████████████████████████████████████████████▌                                                           | 1/2 [03:56<03:56, 236.58s/it]

But GPU usage is none

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.120                Driver Version: 550.120        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       Off |   00000000:00:1E.0 Off |                    0 |
| N/A   21C    P8              9W /   70W |       3MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

I'm wondering if it's caused by pipe.enable_model_cpu_offload()? I need to enable it due to https://huggingface.co/stabilityai/stable-diffusion-3.5-medium/discussions/18

[CaledoniaProject]

Resolved, was caused by low disk performance.

[0zxq0]

请问是怎么解决的？

[CaledoniaProject]

@0zxq0 AWS磁盘快照太慢了，你也是AWS？

[0zxq0]

@0zxq0 AWS磁盘快照太慢了，你也是AWS？

我看错了，不是一个项目，但是问题有的相似，我是用llamafactory训练模型，之前window加载很快，但是现在用ubuntu很慢

[CaledoniaProject]

先检查IO