issue with nccl

[Aveline67]

Describe the bug

When building with nccl, it get stuck in the "run dummy" step

Latest commit or version

HEAD

===== System Info =====
Linux Avlin 6.6.75.1-microsoft-standard-WSL2+ #2 SMP PREEMPT_DYNAMIC Wed Mar  5 19:10:41 CET 2025 x86_64 x86_64 x86_64 GNU/Linux

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

===== NVIDIA & CUDA =====
NVIDIA Driver: 572.61
572.61
nvcc: nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0

===== Rust =====
rustc 1.85.1 (4eb161250 2025-03-15)
cargo 1.85.1 (d73d2caf9 2024-12-31)

===== Other Tools =====
gcc: gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
python3: Python 3.12.3
make: GNU Make 4.3

/usr/lib/x86_64-linux-gnu/libnccl.so
/usr/lib/x86_64-linux-gnu/libnccl.so.2
/usr/local/lib/libnccl.so
/usr/local/lib/libnccl.so.2
/usr/local/lib/libnccl.so.2.26.2

[EricLBuehler]

@Aveline67 what model are you using?

[EricLBuehler]

@Aveline67 this is fixed in #1228.

[Aveline67]

wow ! impressive, let me test again

[Aveline67]

./mistralrs-server --isq Q4K -n52 --port 8000 vision-plain -m google/gemma-3-27b-it -a gemma3
2025-03-23T22:14:44.106126Z  INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: true
2025-03-23T22:14:44.107439Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2025-03-23T22:14:44.107444Z  INFO mistralrs_server: Using flash attention.
2025-03-23T22:14:44.110148Z  INFO mistralrs_server: Model kind is: normal (no adapters)
2025-03-23T22:14:44.113699Z  INFO hf_hub: Using token file found "/home/avlin/.cache/huggingface/token"
2025-03-23T22:14:44.118486Z  INFO mistralrs_core::pipeline::vision: Loading `tokenizer.json` at `google/gemma-3-27b-it`
2025-03-23T22:14:44.126090Z  INFO mistralrs_core::pipeline::vision: Loading `config.json` at `google/gemma-3-27b-it`
2025-03-23T22:14:45.552498Z  INFO mistralrs_core::pipeline::paths: Found model weight filenames ["model-00001-of-00012.safetensors", "model-00002-of-00012.safetensors", "model-00003-of-00012.safetensors", "model-00004-of-00012.safetensors", "model-00005-of-00012.safetensors", "model-00006-of-00012.safetensors", "model-00007-of-00012.safetensors", "model-00008-of-00012.safetensors", "model-00009-of-00012.safetensors", "model-00010-of-00012.safetensors", "model-00011-of-00012.safetensors", "model-00012-of-00012.safetensors"]
2025-03-23T22:14:45.690560Z  INFO mistralrs_core::pipeline::vision: Loading `generation_config.json` at `google/gemma-3-27b-it`
2025-03-23T22:14:45.896017Z  INFO mistralrs_core::pipeline::vision: Loading `preprocessor_config.json` at `google/gemma-3-27b-it`
2025-03-23T22:14:45.999123Z  INFO mistralrs_core::pipeline::vision: Loading `processor_config.json` at `google/gemma-3-27b-it`
2025-03-23T22:14:45.999172Z  INFO mistralrs_core::pipeline::vision: Loading `tokenizer_config.json` at `google/gemma-3-27b-it`
2025-03-23T22:14:46.265481Z  INFO mistralrs_core::pipeline::vision: Loading `chat_template.json` at `google/gemma-3-27b-it`
2025-03-23T22:14:47.312334Z  INFO mistralrs_core::utils::log: Loading model using a NCCL-parallelized pipeline.
2025-03-23T22:14:47.447441Z  INFO mistralrs_core::utils::normal: Detected minimum CUDA compute capability 8.6
2025-03-23T22:14:47.611007Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-03-23T22:14:47.611111Z  INFO mistralrs_core::pipeline::vision: Model config: WithVision { text_config: Gemma3TextConfig { attention_bias: false, head_dim: 128, hidden_activation: GeluPytorchTanh, hidden_size: 5376, intermediate_size: 21504, num_attention_heads: 32, num_hidden_layers: 62, num_key_value_heads: 16, rms_norm_eps: 1e-6, rope_theta: 1000000.0, vocab_size: 262208, sliding_window: 1024, attn_logit_softcapping: None, final_logit_softcapping: None, query_pre_attn_scalar: 168, max_position_embeddings: 131072, quantization_config: None, use_flash_attn: false, tie_word_embeddings: true, rope_local_base_freq: 10000.0, sliding_window_pattern: 6, rope_scaling: Some(Gemma3RopeScalingConfig { factor: 8.0, rope_type: Linear }) }, vision_config: SiglipVisionConfig { hidden_size: 1152, intermediate_size: 4304, num_hidden_layers: 27, num_attention_heads: 16, num_channels: 3, image_size: 896, patch_size: 14, hidden_act: GeluPytorchTanh, layer_norm_eps: 1e-6 }, image_token_index: 262144, mm_tokens_per_image: 256 }
2025-03-23T22:14:47.613293Z  INFO mistralrs_core::distributed: Local tensor parallel world size is 1
2025-03-23T22:14:47.613306Z  INFO mistralrs_core::distributed: Global tensor parallel world size is 2
2025-03-23T22:14:49.066272Z  INFO mistralrs_core::distributed: All workers have received the ids!
2025-03-23T22:14:49.353395Z  INFO mistralrs_core::distributed: Loading all ranks.
2025-03-23T22:17:37.784661Z  INFO mistralrs_core::pipeline::isq: Applying in-situ quantization into Some(Q4K) to 435 tensors.
2025-03-23T22:17:37.795100Z  INFO mistralrs_core::pipeline::isq: Applying ISQ on 52 threads.
2025-03-23T22:18:11.836945Z  INFO mistralrs_core::pipeline::isq: Applied in-situ quantization into Some(Q4K) to 435 tensors out of 435 total tensors. Took 31.12s
2025-03-23T22:18:11.837652Z  INFO mistralrs_core::paged_attention: Allocating 992 MB for PagedAttention KV cache per GPU
2025-03-23T22:18:11.837672Z  INFO mistralrs_core::paged_attention: Using PagedAttention with block size 32 and 128 GPU blocks: available context length is 4096 tokens
2025-03-23T22:18:12.177858Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<bos>", eos_toks = "<eos>", "<end_of_turn>", unk_tok = <unk>
2025-03-23T22:18:12.207661Z  INFO mistralrs_server: Model loaded.
2025-03-23T22:18:12.208005Z  INFO mistralrs_core: Beginning dummy run.

using nccl test :

~/nccl-tests$ ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 2
# nThread 1 nGpus 2 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4265 on      Avlin device  0 [0000:17:00] NVIDIA GeForce RTX 3090
#  Rank  1 Group  0 Pid   4265 on      Avlin device  1 [0000:85:00] NVIDIA GeForce RTX 3090
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    42.95    0.00    0.00      0    70.49    0.00    0.00      0
          16             4     float     sum      -1    48.95    0.00    0.00      0    69.20    0.00    0.00      0
          32             8     float     sum      -1    51.66    0.00    0.00      0    39.45    0.00    0.00      0
          64            16     float     sum      -1    42.27    0.00    0.00      0    39.49    0.00    0.00      0
         128            32     float     sum      -1    42.44    0.00    0.00      0    40.64    0.00    0.00      0
         256            64     float     sum      -1    28.09    0.01    0.01      0    37.99    0.01    0.01      0
         512           128     float     sum      -1    34.86    0.01    0.01      0    72.12    0.01    0.01      0
        1024           256     float     sum      -1    43.94    0.02    0.02      0    75.86    0.01    0.01      0
        2048           512     float     sum      -1    39.01    0.05    0.05      0    46.02    0.04    0.04      0
        4096          1024     float     sum      -1    47.03    0.09    0.09      0    45.81    0.09    0.09      0
        8192          2048     float     sum      -1    40.75    0.20    0.20      0    47.78    0.17    0.17      0
       16384          4096     float     sum      -1    106.7    0.15    0.15      0    31.02    0.53    0.53      0
       32768          8192     float     sum      -1    83.99    0.39    0.39      0    45.78    0.72    0.72      0
       65536         16384     float     sum      -1    43.01    1.52    1.52      0    43.01    1.52    1.52      0
      131072         32768     float     sum      -1    62.37    2.10    2.10      0    53.75    2.44    2.44      0
      262144         65536     float     sum      -1    83.97    3.12    3.12      0    63.04    4.16    4.16      0
      524288        131072     float     sum      -1    77.65    6.75    6.75      0    97.63    5.37    5.37      0
     1048576        262144     float     sum      -1    136.6    7.68    7.68      0    106.0    9.90    9.90      0
     2097152        524288     float     sum      -1    222.0    9.44    9.44      0    212.7    9.86    9.86      0
     4194304       1048576     float     sum      -1    364.9   11.50   11.50      0    340.2   12.33   12.33      0
     8388608       2097152     float     sum      -1    672.9   12.47   12.47      0    690.1   12.16   12.16      0
    16777216       4194304     float     sum      -1   1304.3   12.86   12.86      0   1270.6   13.20   13.20      0
    33554432       8388608     float     sum      -1   2525.8   13.28   13.28      0   2491.2   13.47   13.47      0
    67108864      16777216     float     sum      -1   4979.4   13.48   13.48      0   5153.3   13.02   13.02      0
   134217728      33554432     float     sum      -1   9994.9   13.43   13.43      0    10038   13.37   13.37      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 4.4191
#

[EricLBuehler]

@Aveline67 I found the bug! The fix is in PR #1232. If you could test it and check if it works for you that would be great.

For future reference, our Flash Attention kernels use the default cuda stream.

https://github.com/EricLBuehler/candle/blob/1f137cd4f99ee76797cc4a6969802d85e041dd54/candle-flash-attn/kernels/flash_api.cu#L131

I'm testing Llama 3.3 70B on a TP=2 system and it works with --features cuda,nccl,flash-attn.

[Aveline67]

build and test, but looks like it is still stuck in "beginning dummy run"

git fetch
git pull
cargo clean
RUSTFLAGS="-C target-cpu=native "  cargo build --release --features "cuda cudnn flash-attn nccl"
cd target/release/
./mistralrs-server --isq Q4K --port 8000 vision-plain -m google/gemma-3-27b-it -a gemma3

2025-03-24T10:12:13.245693Z  INFO mistralrs_server: avx: true, neon: false, simd128: false, f16c: true
2025-03-24T10:12:13.245724Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2025-03-24T10:12:13.245726Z  INFO mistralrs_server: Using flash attention.
2025-03-24T10:12:13.245745Z  INFO mistralrs_server: Model kind is: normal (no adapters)
2025-03-24T10:12:13.248179Z  INFO hf_hub: Using token file found "/home/avlin/.cache/huggingface/token"
2025-03-24T10:12:13.249762Z  INFO mistralrs_core::pipeline::vision: Loading `tokenizer.json` at `google/gemma-3-27b-it`
2025-03-24T10:12:13.258905Z  INFO mistralrs_core::pipeline::vision: Loading `config.json` at `google/gemma-3-27b-it`
2025-03-24T10:12:13.397034Z  INFO mistralrs_core::pipeline::paths: Found model weight filenames ["model-00001-of-00012.safetensors", "model-00002-of-00012.safetensors", "model-00003-of-00012.safetensors", "model-00004-of-00012.safetensors", "model-00005-of-00012.safetensors", "model-00006-of-00012.safetensors", "model-00007-of-00012.safetensors", "model-00008-of-00012.safetensors", "model-00009-of-00012.safetensors", "model-00010-of-00012.safetensors", "model-00011-of-00012.safetensors", "model-00012-of-00012.safetensors"]
2025-03-24T10:12:13.497880Z  INFO mistralrs_core::pipeline::vision: Loading `generation_config.json` at `google/gemma-3-27b-it`
2025-03-24T10:12:13.596237Z  INFO mistralrs_core::pipeline::vision: Loading `preprocessor_config.json` at `google/gemma-3-27b-it`
2025-03-24T10:12:13.703269Z  INFO mistralrs_core::pipeline::vision: Loading `processor_config.json` at `google/gemma-3-27b-it`
2025-03-24T10:12:13.703334Z  INFO mistralrs_core::pipeline::vision: Loading `tokenizer_config.json` at `google/gemma-3-27b-it`
2025-03-24T10:12:13.804976Z  INFO mistralrs_core::pipeline::vision: Loading `chat_template.json` at `google/gemma-3-27b-it`
2025-03-24T10:12:14.787413Z  INFO mistralrs_core::utils::log: Loading model using a NCCL-parallelized pipeline.
2025-03-24T10:12:14.885389Z  INFO mistralrs_core::utils::normal: Detected minimum CUDA compute capability 8.6
2025-03-24T10:12:15.005162Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-03-24T10:12:15.005199Z  INFO mistralrs_core::pipeline::vision: Model config: WithVision { text_config: Gemma3TextConfig { attention_bias: false, head_dim: 128, hidden_activation: GeluPytorchTanh, hidden_size: 5376, intermediate_size: 21504, num_attention_heads: 32, num_hidden_layers: 62, num_key_value_heads: 16, rms_norm_eps: 1e-6, rope_theta: 1000000.0, vocab_size: 262208, sliding_window: 1024, attn_logit_softcapping: None, final_logit_softcapping: None, query_pre_attn_scalar: 168, max_position_embeddings: 131072, quantization_config: None, use_flash_attn: false, tie_word_embeddings: true, rope_local_base_freq: 10000.0, sliding_window_pattern: 6, rope_scaling: Some(Gemma3RopeScalingConfig { factor: 8.0, rope_type: Linear }) }, vision_config: SiglipVisionConfig { hidden_size: 1152, intermediate_size: 4304, num_hidden_layers: 27, num_attention_heads: 16, num_channels: 3, image_size: 896, patch_size: 14, hidden_act: GeluPytorchTanh, layer_norm_eps: 1e-6 }, image_token_index: 262144, mm_tokens_per_image: 256 }
2025-03-24T10:12:15.005852Z  INFO mistralrs_core::distributed: Local tensor parallel world size is 1
2025-03-24T10:12:15.005876Z  INFO mistralrs_core::distributed: Global tensor parallel world size is 2
2025-03-24T10:12:15.963194Z  INFO mistralrs_core::distributed: All workers have received the ids!
2025-03-24T10:12:16.220032Z  INFO mistralrs_core::distributed: Loading all ranks.
2025-03-24T10:14:39.405691Z  INFO mistralrs_core::pipeline::isq: Applying in-situ quantization into Some(Q4K) to 435 tensors.
2025-03-24T10:14:39.419126Z  INFO mistralrs_core::pipeline::isq: Applying ISQ on 52 threads.
2025-03-24T10:15:11.580777Z  INFO mistralrs_core::pipeline::isq: Applied in-situ quantization into Some(Q4K) to 435 tensors out of 435 total tensors. Took 29.24s
2025-03-24T10:15:11.582693Z  INFO mistralrs_core::paged_attention: Allocating 992 MB for PagedAttention KV cache per GPU
2025-03-24T10:15:11.582704Z  INFO mistralrs_core::paged_attention: Using PagedAttention with block size 32 and 128 GPU blocks: available context length is 4096 tokens
2025-03-24T10:15:11.897577Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<bos>", eos_toks = "<eos>", "<end_of_turn>", unk_tok = <unk>
2025-03-24T10:15:11.921340Z  INFO mistralrs_server: Model loaded.
2025-03-24T10:15:11.932270Z  INFO mistralrs_core: Beginning dummy run.

[Aveline67]

@EricLBuehler no way to make it work, looks like it is using cpu on 1 thread for the dummy run

I will try on bare linux instead of WSL