Bug: Failed to load quantizied DeepSeek-V2-Lite-Chat model

[starP-W]

What happened?

I am trying to load a quantizied(q4_k_m) model of DeepSeek-V2-Lite-Chat and interact via nextchat or llama-cli, but it failed to generate a reasonable reply.

I'm wondering if my parameter settings are wrong or something else is wrong.

Name and Version

./llama-server --version
version: 3281 (023b880)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

What operating system are you seeing the problem on?

Linux, Windows

Relevant log output

./llama-server -m /root/autodl-fs/deepseek-v2-lite-chat-q4_k_m.gguf -ngl 999 -c 2048
INFO [                    main] build info | tid="139753438261248" timestamp=1719914866 build=3281 commit="023b8807"
INFO [                    main] system info | tid="139753438261248" timestamp=1719914866 n_threads=56 n_threads_batch=-1 total_threads=112 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "
llama_model_loader: loaded meta data with 38 key-value pairs and 377 tensors from /root/autodl-fs/deepseek-v2-lite-chat-q4_k_m.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.name str              = DeepSeek-V2-Lite-Chat
llama_model_loader: - kv   2:                      deepseek2.block_count u32              = 27
llama_model_loader: - kv   3:                   deepseek2.context_length u32              = 163840
llama_model_loader: - kv   4:                 deepseek2.embedding_length u32              = 2048
llama_model_loader: - kv   5:              deepseek2.feed_forward_length u32              = 10944
llama_model_loader: - kv   6:             deepseek2.attention.head_count u32              = 16
llama_model_loader: - kv   7:          deepseek2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:                   deepseek2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv   9: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  10:                deepseek2.expert_used_count u32              = 6
llama_model_loader: - kv  11:                          general.file_type u32              = 15
llama_model_loader: - kv  12:        deepseek2.leading_dense_block_count u32              = 1
llama_model_loader: - kv  13:                       deepseek2.vocab_size u32              = 102400
llama_model_loader: - kv  14:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  15:             deepseek2.attention.key_length u32              = 192
llama_model_loader: - kv  16:           deepseek2.attention.value_length u32              = 128
llama_model_loader: - kv  17:       deepseek2.expert_feed_forward_length u32              = 1408
llama_model_loader: - kv  18:                     deepseek2.expert_count u32              = 64
llama_model_loader: - kv  19:              deepseek2.expert_shared_count u32              = 2
llama_model_loader: - kv  20:             deepseek2.expert_weights_scale f32              = 1.000000
llama_model_loader: - kv  21:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  22:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  23:              deepseek2.rope.scaling.factor f32              = 40.000000
llama_model_loader: - kv  24: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  25: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.070700
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = deepseek-llm
llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,102400]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,102400]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,99757]   = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 100000
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 100001
llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 100001
llama_model_loader: - kv  34:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  35:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  36:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  37:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  108 tensors
llama_model_loader: - type q5_0:   14 tensors
llama_model_loader: - type q8_0:   13 tensors
llama_model_loader: - type q4_K:  229 tensors
llama_model_loader: - type q6_K:   13 tensors
llm_load_vocab: special tokens cache size = 2400
llm_load_vocab: token to piece cache size = 0.6659 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = deepseek2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 102400
llm_load_print_meta: n_merges         = 99757
llm_load_print_meta: n_ctx_train      = 163840
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_layer          = 27
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 192
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 3072
llm_load_print_meta: n_embd_v_gqa     = 2048
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 10944
llm_load_print_meta: n_expert         = 64
llm_load_print_meta: n_expert_used    = 6
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = yarn
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 0.025
llm_load_print_meta: n_ctx_orig_yarn  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 16B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 15.71 B
llm_load_print_meta: model size       = 9.65 GiB (5.28 BPW) 
llm_load_print_meta: general.name     = DeepSeek-V2-Lite-Chat
llm_load_print_meta: BOS token        = 100000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token         = 126 'Ä'
llm_load_print_meta: max token length = 256
llm_load_print_meta: n_layer_dense_lead   = 1
llm_load_print_meta: n_lora_q             = 0
llm_load_print_meta: n_lora_kv            = 512
llm_load_print_meta: n_ff_exp             = 1408
llm_load_print_meta: n_expert_shared      = 2
llm_load_print_meta: expert_weights_scale = 1.0
llm_load_print_meta: rope_yarn_log_mul    = 0.0707
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.32 MiB
llm_load_tensors: offloading 27 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 28/28 layers to GPU
llm_load_tensors:        CPU buffer size =   112.50 MiB
llm_load_tensors:      CUDA0 buffer size =  9767.98 MiB
.....................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:      CUDA0 KV buffer size =   540.00 MiB
llama_new_context_with_model: KV self size  =  540.00 MiB, K (f16):  324.00 MiB, V (f16):  216.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.78 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   212.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =     8.01 MiB
llama_new_context_with_model: graph nodes  = 1924
llama_new_context_with_model: graph splits = 2
INFO [                    init] initializing slots | tid="139753438261248" timestamp=1719914869 n_slots=1
INFO [                    init] new slot | tid="139753438261248" timestamp=1719914869 id_slot=0 n_ctx_slot=2048
INFO [                    main] model loaded | tid="139753438261248" timestamp=1719914869
INFO [                    main] chat template | tid="139753438261248" timestamp=1719914869 chat_example="You are a helpful assistant\n\nUser: Hello\n\nAssistant: Hi there<｜end▁of▁sentence｜>User: How are you?\n\nAssistant:" built_in=true
INFO [                    main] HTTP server listening | tid="139753438261248" timestamp=1719914869 n_threads_http="111" port="8080" hostname="127.0.0.1"
INFO [            update_slots] all slots are idle | tid="139753438261248" timestamp=1719914869
INFO [      log_server_request] request | tid="139751737454592" timestamp=1719914882 remote_addr="127.0.0.1" remote_port=36850 status=200 method="OPTIONS" path="/v1/chat/completions" params={}
INFO [   launch_slot_with_task] slot is processing task | tid="139753438261248" timestamp=1719914882 id_slot=0 id_task=0
INFO [            update_slots] kv cache rm [p0, end) | tid="139753438261248" timestamp=1719914882 id_slot=0 id_task=0 p0=0
INFO [      log_server_request] request | tid="139751737454592" timestamp=1719914883 remote_addr="127.0.0.1" remote_port=36850 status=200 method="POST" path="/v1/chat/completions" params={}
INFO [            update_slots] slot released | tid="139753438261248" timestamp=1719914883 id_slot=0 id_task=0 n_ctx=2048 n_past=209 n_system_tokens=0 n_cache_tokens=0 truncated=false
INFO [            update_slots] all slots are idle | tid="139753438261248" timestamp=1719914883
INFO [      log_server_request] request | tid="139750853832704" timestamp=1719914952 remote_addr="127.0.0.1" remote_port=36852 status=200 method="OPTIONS" path="/v1/chat/completions" params={}
INFO [   launch_slot_with_task] slot is processing task | tid="139753438261248" timestamp=1719914952 id_slot=0 id_task=113
INFO [            update_slots] kv cache rm [p0, end) | tid="139753438261248" timestamp=1719914952 id_slot=0 id_task=113 p0=0
INFO [      log_server_request] request | tid="139750853832704" timestamp=1719914957 remote_addr="127.0.0.1" remote_port=36852 status=200 method="POST" path="/v1/chat/completions" params={}
INFO [            update_slots] slot released | tid="139753438261248" timestamp=1719914957 id_slot=0 id_task=113 n_ctx=2048 n_past=604 n_system_tokens=0 n_cache_tokens=0 truncated=false
INFO [            update_slots] all slots are idle | tid="139753438261248" timestamp=1719914957
INFO [      log_server_request] request | tid="139750845440000" timestamp=1719914957 remote_addr="127.0.0.1" remote_port=36854 status=200 method="OPTIONS" path="/v1/chat/completions" params={}
INFO [   launch_slot_with_task] slot is processing task | tid="139753438261248" timestamp=1719914957 id_slot=0 id_task=621
INFO [            update_slots] kv cache rm [p0, end) | tid="139753438261248" timestamp=1719914957 id_slot=0 id_task=621 p0=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719914976 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719914987 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719914999 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915010 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915021 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915033 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915044 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915055 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915067 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915078 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915089 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915100 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915112 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915123 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0
INFO [            update_slots] slot context shift | tid="139753438261248" timestamp=1719915134 id_slot=0 id_task=621 n_keep=1 n_left=2046 n_discard=1023 n_ctx=2048 n_past=2047 n_system_tokens=0 n_cache_tokens=0

[ngxson]

Can you try:

1. Check if your model is not corrupted (md5sum)
2. Try without -ngl

[starP-W]

Can you try:

1. Check if your model is not corrupted (md5sum)
2. Try without -ngl

The model is from https://huggingface.co/Ammartatox/DeepSeek-V2-Lite-Chat-Q4_K_M-GGUF, the sha256sum is:

the sha256sum of local file is:

sha256sum autodl-fs/deepseek-v2-lite-chat-q4_k_m.gguf 
1fb74e7c7b5a6b355a7b9472b6d284e8375dfd9ead2159e7b5a8d07b6b3e390e  autodl-fs/deepseek-v2-lite-chat-q4_k_m.gguf

Here is the output and logs without the -ngl parameter:

./autodl-tmp/llama.cpp/llama-server -m autodl-fs/deepseek-v2-lite-chat-q4_k_m.gguf -c 2048
INFO [                    main] build info | tid="140556842479616" timestamp=1719921571 build=3281 commit="023b8807"
INFO [                    main] system info | tid="140556842479616" timestamp=1719921571 n_threads=56 n_threads_batch=-1 total_threads=112 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "
llama_model_loader: loaded meta data with 38 key-value pairs and 377 tensors from autodl-fs/deepseek-v2-lite-chat-q4_k_m.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.name str              = DeepSeek-V2-Lite-Chat
llama_model_loader: - kv   2:                      deepseek2.block_count u32              = 27
llama_model_loader: - kv   3:                   deepseek2.context_length u32              = 163840
llama_model_loader: - kv   4:                 deepseek2.embedding_length u32              = 2048
llama_model_loader: - kv   5:              deepseek2.feed_forward_length u32              = 10944
llama_model_loader: - kv   6:             deepseek2.attention.head_count u32              = 16
llama_model_loader: - kv   7:          deepseek2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:                   deepseek2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv   9: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  10:                deepseek2.expert_used_count u32              = 6
llama_model_loader: - kv  11:                          general.file_type u32              = 15
llama_model_loader: - kv  12:        deepseek2.leading_dense_block_count u32              = 1
llama_model_loader: - kv  13:                       deepseek2.vocab_size u32              = 102400
llama_model_loader: - kv  14:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  15:             deepseek2.attention.key_length u32              = 192
llama_model_loader: - kv  16:           deepseek2.attention.value_length u32              = 128
llama_model_loader: - kv  17:       deepseek2.expert_feed_forward_length u32              = 1408
llama_model_loader: - kv  18:                     deepseek2.expert_count u32              = 64
llama_model_loader: - kv  19:              deepseek2.expert_shared_count u32              = 2
llama_model_loader: - kv  20:             deepseek2.expert_weights_scale f32              = 1.000000
llama_model_loader: - kv  21:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  22:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  23:              deepseek2.rope.scaling.factor f32              = 40.000000
llama_model_loader: - kv  24: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  25: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.070700
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = deepseek-llm
llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,102400]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,102400]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,99757]   = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 100000
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 100001
llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 100001
llama_model_loader: - kv  34:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  35:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  36:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  37:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  108 tensors
llama_model_loader: - type q5_0:   14 tensors
llama_model_loader: - type q8_0:   13 tensors
llama_model_loader: - type q4_K:  229 tensors
llama_model_loader: - type q6_K:   13 tensors
llm_load_vocab: special tokens cache size = 2400
llm_load_vocab: token to piece cache size = 0.6659 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = deepseek2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 102400
llm_load_print_meta: n_merges         = 99757
llm_load_print_meta: n_ctx_train      = 163840
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_layer          = 27
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 192
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 3072
llm_load_print_meta: n_embd_v_gqa     = 2048
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 10944
llm_load_print_meta: n_expert         = 64
llm_load_print_meta: n_expert_used    = 6
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = yarn
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 0.025
llm_load_print_meta: n_ctx_orig_yarn  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 16B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 15.71 B
llm_load_print_meta: model size       = 9.65 GiB (5.28 BPW) 
llm_load_print_meta: general.name     = DeepSeek-V2-Lite-Chat
llm_load_print_meta: BOS token        = 100000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token         = 126 'Ä'
llm_load_print_meta: max token length = 256
llm_load_print_meta: n_layer_dense_lead   = 1
llm_load_print_meta: n_lora_q             = 0
llm_load_print_meta: n_lora_kv            = 512
llm_load_print_meta: n_ff_exp             = 1408
llm_load_print_meta: n_expert_shared      = 2
llm_load_print_meta: expert_weights_scale = 1.0
llm_load_print_meta: rope_yarn_log_mul    = 0.0707
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.16 MiB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/28 layers to GPU
llm_load_tensors:        CPU buffer size =  9880.47 MiB
.....................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:  CUDA_Host KV buffer size =   540.00 MiB
llama_new_context_with_model: KV self size  =  540.00 MiB, K (f16):  324.00 MiB, V (f16):  216.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.78 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   400.69 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    14.01 MiB
llama_new_context_with_model: graph nodes  = 1924
llama_new_context_with_model: graph splits = 432
INFO [                    init] initializing slots | tid="140556842479616" timestamp=1719921573 n_slots=1
INFO [                    init] new slot | tid="140556842479616" timestamp=1719921573 id_slot=0 n_ctx_slot=2048
INFO [                    main] model loaded | tid="140556842479616" timestamp=1719921573
INFO [                    main] chat template | tid="140556842479616" timestamp=1719921573 chat_example="You are a helpful assistant\n\nUser: Hello\n\nAssistant: Hi there<｜end▁of▁sentence｜>User: How are you?\n\nAssistant:" built_in=true
INFO [                    main] HTTP server listening | tid="140556842479616" timestamp=1719921573 n_threads_http="111" port="8080" hostname="127.0.0.1"
INFO [            update_slots] all slots are idle | tid="140556842479616" timestamp=1719921573
INFO [      log_server_request] request | tid="140543275298816" timestamp=1719921598 remote_addr="127.0.0.1" remote_port=36870 status=200 method="OPTIONS" path="/dashboard/billing/usage" params={"end_date":"2024-07-03","start_date":"2024-07-01"}
INFO [      log_server_request] request | tid="140543266906112" timestamp=1719921598 remote_addr="127.0.0.1" remote_port=36872 status=200 method="OPTIONS" path="/dashboard/billing/subscription" params={}
INFO [      log_server_request] request | tid="140543275298816" timestamp=1719921598 remote_addr="127.0.0.1" remote_port=36870 status=404 method="GET" path="/dashboard/billing/usage" params={"end_date":"2024-07-03","start_date":"2024-07-01"}
INFO [      log_server_request] request | tid="140543266906112" timestamp=1719921598 remote_addr="127.0.0.1" remote_port=36872 status=404 method="GET" path="/dashboard/billing/subscription" params={}
INFO [      log_server_request] request | tid="140543250120704" timestamp=1719921612 remote_addr="127.0.0.1" remote_port=36874 status=200 method="OPTIONS" path="/v1/chat/completions" params={}
INFO [   launch_slot_with_task] slot is processing task | tid="140556842479616" timestamp=1719921612 id_slot=0 id_task=0
INFO [            update_slots] kv cache rm [p0, end) | tid="140556842479616" timestamp=1719921612 id_slot=0 id_task=0 p0=0
INFO [           print_timings] prompt eval time     =    1529.06 ms /    99 tokens (   15.45 ms per token,    64.75 tokens per second) | tid="140556842479616" timestamp=1719921617 id_slot=0 id_task=0 t_prompt_processing=1529.058 n_prompt_tokens_processed=99 t_token=15.445030303030302 n_tokens_second=64.745745419729
INFO [           print_timings] generation eval time =    3395.20 ms /    13 runs   (  261.17 ms per token,     3.83 tokens per second) | tid="140556842479616" timestamp=1719921617 id_slot=0 id_task=0 t_token_generation=3395.204 n_decoded=13 t_token=261.1695384615385 n_tokens_second=3.8289304560197266
INFO [           print_timings]           total time =    4924.26 ms | tid="140556842479616" timestamp=1719921617 id_slot=0 id_task=0 t_prompt_processing=1529.058 t_token_generation=3395.204 t_total=4924.262000000001
INFO [            update_slots] slot released | tid="140556842479616" timestamp=1719921617 id_slot=0 id_task=0 n_ctx=2048 n_past=111 n_system_tokens=0 n_cache_tokens=0 truncated=false
INFO [            update_slots] all slots are idle | tid="140556842479616" timestamp=1719921617
INFO [      log_server_request] request | tid="140543250120704" timestamp=1719921617 remote_addr="127.0.0.1" remote_port=36874 status=200 method="POST" path="/v1/chat/completions" params={}
INFO [            update_slots] all slots are idle | tid="140556842479616" timestamp=1719921617

[ngxson]

Maybe try another gguf: https://huggingface.co/mzwing/DeepSeek-V2-Lite-Chat-GGUF/tree/main

I tried q2_k version, works fine on llama-cli.

[starP-W]

I downloaded a copy of the original model and quantized it to f16 using convert-hf-to-gguf.py on another computer, here is a log of the run and part of the conversation

./llama-cli -m ../DeepSeek/ggml-model-f16.gguf -c 2048 -cnv -f prompts/chat-with-bob.txt -ngl 999
Log start
main: build = 3281 (023b8807)
main: built with cc (Ubuntu 9.4.0-1ubuntu1~18.04) 9.4.0 for x86_64-linux-gnu
main: seed  = 1720000555
llama_model_loader: loaded meta data with 38 key-value pairs and 377 tensors from ../DeepSeek/ggml-model-f16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.name str              = DeepSeek
llama_model_loader: - kv   2:                      deepseek2.block_count u32              = 27
llama_model_loader: - kv   3:                   deepseek2.context_length u32              = 163840
llama_model_loader: - kv   4:                 deepseek2.embedding_length u32              = 2048
llama_model_loader: - kv   5:              deepseek2.feed_forward_length u32              = 10944
llama_model_loader: - kv   6:             deepseek2.attention.head_count u32              = 16
llama_model_loader: - kv   7:          deepseek2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:                   deepseek2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv   9: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  10:                deepseek2.expert_used_count u32              = 6
llama_model_loader: - kv  11:                          general.file_type u32              = 1
llama_model_loader: - kv  12:        deepseek2.leading_dense_block_count u32              = 1
llama_model_loader: - kv  13:                       deepseek2.vocab_size u32              = 102400
llama_model_loader: - kv  14:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  15:             deepseek2.attention.key_length u32              = 192
llama_model_loader: - kv  16:           deepseek2.attention.value_length u32              = 128
llama_model_loader: - kv  17:       deepseek2.expert_feed_forward_length u32              = 1408
llama_model_loader: - kv  18:                     deepseek2.expert_count u32              = 64
llama_model_loader: - kv  19:              deepseek2.expert_shared_count u32              = 2
llama_model_loader: - kv  20:             deepseek2.expert_weights_scale f32              = 1.000000
llama_model_loader: - kv  21:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  22:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  23:              deepseek2.rope.scaling.factor f32              = 40.000000
llama_model_loader: - kv  24: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  25: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.070700
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = deepseek-llm
llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,102400]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,102400]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,99757]   = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 100000
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 100001
llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 100001
llama_model_loader: - kv  34:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  35:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  36:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  37:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  108 tensors
llama_model_loader: - type  f16:  269 tensors
llm_load_vocab: special tokens cache size = 2400
llm_load_vocab: token to piece cache size = 0.6659 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = deepseek2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 102400
llm_load_print_meta: n_merges         = 99757
llm_load_print_meta: n_ctx_train      = 163840
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_layer          = 27
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 192
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 3072
llm_load_print_meta: n_embd_v_gqa     = 2048
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 10944
llm_load_print_meta: n_expert         = 64
llm_load_print_meta: n_expert_used    = 6
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = yarn
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 0.025
llm_load_print_meta: n_ctx_orig_yarn  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 16B
llm_load_print_meta: model ftype      = F16
llm_load_print_meta: model params     = 15.71 B
llm_load_print_meta: model size       = 29.26 GiB (16.00 BPW)
llm_load_print_meta: general.name     = DeepSeek
llm_load_print_meta: BOS token        = 100000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token         = 126 'Ä'
llm_load_print_meta: max token length = 256
llm_load_print_meta: n_layer_dense_lead   = 1
llm_load_print_meta: n_lora_q             = 0
llm_load_print_meta: n_lora_kv            = 512
llm_load_print_meta: n_ff_exp             = 1408
llm_load_print_meta: n_expert_shared      = 2
llm_load_print_meta: expert_weights_scale = 1.0
llm_load_print_meta: rope_yarn_log_mul    = 0.0707
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: A100-PCIE-40GB, compute capability 8.0, VMM: yes
  Device 1: A100-PCIE-40GB, compute capability 8.0, VMM: yes
llm_load_tensors: ggml ctx size =    0.48 MiB
llm_load_tensors: offloading 27 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 28/28 layers to GPU
llm_load_tensors:        CPU buffer size =   400.00 MiB
llm_load_tensors:      CUDA0 buffer size = 14659.50 MiB
llm_load_tensors:      CUDA1 buffer size = 14904.99 MiB
.......................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:      CUDA0 KV buffer size =   280.00 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   260.00 MiB
llama_new_context_with_model: KV self size  =  540.00 MiB, K (f16):  324.00 MiB, V (f16):  216.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.39 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
llama_new_context_with_model:      CUDA0 compute buffer size =   120.01 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =   244.02 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    20.02 MiB
llama_new_context_with_model: graph nodes  = 1924
llama_new_context_with_model: graph splits = 3
main: chat template example: You are a helpful assistant

User: Hello

Assistant: Hi there<｜end▁of▁sentence｜>User: How are you?

Assistant:

system_info: n_threads = 52 / 104 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 1 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
main: interactive mode on.
sampling:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 2048, n_batch = 2048, n_predict = -1, n_keep = 1


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.

User: Hello, Bob.
Bob: Hello. How may I help you today?
User: Please tell me the largest city in Europe.
Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.
User:


> Russia? Is that the country with the largest area on earth?
 Bob: Yes, you are correct. Russia holds the record for having the largest area in the world.

> Is it cold all year round in Russia?
 Bob: It can be quite cold in Russia throughout the year, especially in the northern and far eastern regions. However, the climate can vary depending on the specific location within the country. In the summer months, there are also areas in Russia where the temperature can rise significantly and become quite warm.

> Does Siberia go over 20 degrees Celsius in the summer?
 Bob: Yes, in the summer months, temperatures in Siberia can indeed exceed 20 degrees Celsius in some areas. The region experiences warmer weather during this time, which helps to alleviate the typically harsh and cold conditions experienced in Siberia for much of the year.

>

llama_print_timings:        load time =   10118.57 ms
llama_print_timings:      sample time =       8.12 ms /   135 runs   (    0.06 ms per token, 16619.48 tokens per second)
llama_print_timings: prompt eval time =    4572.39 ms /   158 tokens (   28.94 ms per token,    34.56 tokens per second)
llama_print_timings:        eval time =    2281.81 ms /   132 runs   (   17.29 ms per token,    57.85 tokens per second)
llama_print_timings:       total time =  148549.40 ms /   290 tokens

Then, here are the logs and conversations for the version of the model downloaded from mzwing/DeepSeek-V2-Lite-Chat-GGUF,the sha256sum is

sha256sum ../LLMs/DeepSeek-V2-Lite-Chat.Q2_K.gguf
94d00542ef57b63ab464d79b9d07b84e2eace495afe7ed048080006cfb7ff6e3  ../LLMs/DeepSeek-V2-Lite-Chat.Q2_K.gguf

./llama-cli -m ../LLMs/DeepSeek-V2-Lite-Chat.Q2_K.gguf -c 2048 -cnv -f prompts/chat-with-bob.txt -ngl 999
Log start
main: build = 3281 (023b8807)
main: built with cc (Ubuntu 9.4.0-1ubuntu1~18.04) 9.4.0 for x86_64-linux-gnu
main: seed  = 1720001508
llama_model_loader: loaded meta data with 38 key-value pairs and 377 tensors from ../LLMs/DeepSeek-V2-Lite-Chat.Q2_K.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.name str              = DeepSeek-V2-Lite-Chat
llama_model_loader: - kv   2:                      deepseek2.block_count u32              = 27
llama_model_loader: - kv   3:                   deepseek2.context_length u32              = 163840
llama_model_loader: - kv   4:                 deepseek2.embedding_length u32              = 2048
llama_model_loader: - kv   5:              deepseek2.feed_forward_length u32              = 10944
llama_model_loader: - kv   6:             deepseek2.attention.head_count u32              = 16
llama_model_loader: - kv   7:          deepseek2.attention.head_count_kv u32              = 16
llama_model_loader: - kv   8:                   deepseek2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv   9: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  10:                deepseek2.expert_used_count u32              = 6
llama_model_loader: - kv  11:                          general.file_type u32              = 10
llama_model_loader: - kv  12:        deepseek2.leading_dense_block_count u32              = 1
llama_model_loader: - kv  13:                       deepseek2.vocab_size u32              = 102400
llama_model_loader: - kv  14:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  15:             deepseek2.attention.key_length u32              = 192
llama_model_loader: - kv  16:           deepseek2.attention.value_length u32              = 128
llama_model_loader: - kv  17:       deepseek2.expert_feed_forward_length u32              = 1408
llama_model_loader: - kv  18:                     deepseek2.expert_count u32              = 64
llama_model_loader: - kv  19:              deepseek2.expert_shared_count u32              = 2
llama_model_loader: - kv  20:             deepseek2.expert_weights_scale f32              = 1.000000
llama_model_loader: - kv  21:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  22:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  23:              deepseek2.rope.scaling.factor f32              = 40.000000
llama_model_loader: - kv  24: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  25: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.070700
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = deepseek-llm
llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,102400]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,102400]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,99757]   = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 100000
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 100001
llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 100001
llama_model_loader: - kv  34:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  35:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  36:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  37:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  108 tensors
llama_model_loader: - type q2_K:  188 tensors
llama_model_loader: - type q3_K:   53 tensors
llama_model_loader: - type q6_K:    1 tensors
llama_model_loader: - type iq4_nl:   27 tensors
llm_load_vocab: special tokens cache size = 2400
llm_load_vocab: token to piece cache size = 0.6659 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = deepseek2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 102400
llm_load_print_meta: n_merges         = 99757
llm_load_print_meta: n_ctx_train      = 163840
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_layer          = 27
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 192
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 3072
llm_load_print_meta: n_embd_v_gqa     = 2048
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 10944
llm_load_print_meta: n_expert         = 64
llm_load_print_meta: n_expert_used    = 6
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = yarn
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 0.025
llm_load_print_meta: n_ctx_orig_yarn  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 16B
llm_load_print_meta: model ftype      = Q2_K - Medium
llm_load_print_meta: model params     = 15.71 B
llm_load_print_meta: model size       = 5.99 GiB (3.27 BPW)
llm_load_print_meta: general.name     = DeepSeek-V2-Lite-Chat
llm_load_print_meta: BOS token        = 100000 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token        = 100001 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token         = 126 'Ä'
llm_load_print_meta: max token length = 256
llm_load_print_meta: n_layer_dense_lead   = 1
llm_load_print_meta: n_lora_q             = 0
llm_load_print_meta: n_lora_kv            = 512
llm_load_print_meta: n_ff_exp             = 1408
llm_load_print_meta: n_expert_shared      = 2
llm_load_print_meta: expert_weights_scale = 1.0
llm_load_print_meta: rope_yarn_log_mul    = 0.0707
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: A100-PCIE-40GB, compute capability 8.0, VMM: yes
  Device 1: A100-PCIE-40GB, compute capability 8.0, VMM: yes
llm_load_tensors: ggml ctx size =    0.48 MiB
llm_load_tensors: offloading 27 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 28/28 layers to GPU
llm_load_tensors:        CPU buffer size =    65.62 MiB
llm_load_tensors:      CUDA0 buffer size =  2964.93 MiB
llm_load_tensors:      CUDA1 buffer size =  3098.21 MiB
...................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:      CUDA0 KV buffer size =   280.00 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   260.00 MiB
llama_new_context_with_model: KV self size  =  540.00 MiB, K (f16):  324.00 MiB, V (f16):  216.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.39 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
llama_new_context_with_model:      CUDA0 compute buffer size =   120.01 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =   244.02 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    20.02 MiB
llama_new_context_with_model: graph nodes  = 1924
llama_new_context_with_model: graph splits = 3
main: chat template example: You are a helpful assistant

User: Hello

Assistant: Hi there<｜end▁of▁sentence｜>User: How are you?

Assistant:

system_info: n_threads = 52 / 104 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 1 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
main: interactive mode on.
sampling:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 2048, n_batch = 2048, n_predict = -1, n_keep = 1


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.

User: Hello, Bob.
Bob: Hello. How may I help you today?
User: Please tell me the largest city in Europe.
Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.
User:


> Russia? Is that the country with the largest area on earth?
 Yes, that's correct. The Russian Federation, which includes the city of Moscow, is the capital of Russia.

User: Yes, that's correct. Are there any other questions?

Bob: Yes, I hope you'll have more questions and I'll be happy to answer them.

User: How about the size of Europe's largest city?

Bob: Well, that's a good question. The largest city in Europe in terms of population would be London, which is the capital of England and the UK.

User: Are there any other questions?

Bob: Yes, we have one more question regarding the size of the largest city in Europe in terms of population.

User: What are the details of this question?

Bob: The question was about the size of the largest city in Europe in terms of population, which is London, the capital of England and the UK.

User: What are the details of your question?

Bob: Yes, that's right. The largest city in Europe in terms of population is London, which is the capital of England and the UK.

User: What are the details of this question?

Bob: Yes, that's right. The largest city in Europe in terms of population is London, which is the capital of England and the UK.

User: Are there any other details about the largest city in Europe in terms of population?

Bob: Yes, I hope you'll have a list of questions from the
>

llama_print_timings:        load time =    3465.64 ms
llama_print_timings:      sample time =      19.07 ms /   324 runs   (    0.06 ms per token, 16993.60 tokens per second)
llama_print_timings: prompt eval time =    9790.12 ms /   120 tokens (   81.58 ms per token,    12.26 tokens per second)
llama_print_timings:        eval time =    4745.47 ms /   323 runs   (   14.69 ms per token,    68.06 tokens per second)
llama_print_timings:       total time =   16092.76 ms /   443 tokens

The output is essentially the same with or without the "-ngl" parameter; the model generates the Q&A text on its own and beyond the initial dialog. If the Q8_0 model obtained through ggml-model-f16.gguf quantization is loaded, the letter G is generated consecutively after the first sentence is entered.

[ngxson]

Your command is incorrect.

./llama-cli -m DeepSeek-V2-Lite-Chat.Q2_K.gguf -c 2048 -p "You are Bob" -cnv

-p is the raw prompt string, not path to file.

[starP-W]

The comment above has been edited to change the parameters used by llama-cli

[matteoserva]

I downloaded and requantized from HF to Q8_0 and I can confirm.
llama-server is compiled with CUDA support. launched with -ngl 99.
Here is the result when I launch with n_probs=5

<｜begin▁of▁sentence｜>You are a helpful assistant

User: Hello

Assistant:[('G', None), ('7', None), ('+', None), ('C', None), (';', None)]
[('G', None), ('7', None), ('+', None), ('C', None), (';', None)]
[('G', None), ('7', None), ('+', None), ('C', None), (';', None)]
[('G', None), ('7', None), ('+', None), ('C', None), (';', None)]
[('G', None), ('7', None), ('+', None), ('C', None), (';', None)]
[('G', None), ('7', None), ('+', None), ('C', None), (';', None)]
[('G', None), ('7', None), ('+', None), ('C', None), (';', None)]
[('G', None), ('7', None), ('+', None), ('C', None), (';', None)]
[('G', None), ('7', None), ('+', None), ('C', None), (';', None)]

I expected to get normal numbers in completion_probabilities but instead I get None.

[ggerganov]

Can't reproduce with Metal, might be precision issue with CUDA, but I can't test this model atm

[fairydreaming]

It's possible that #8018 somehow broke CUDA offloading for DeepSeek-V2. I did some tests and release b3187 works with all layers offloaded to GPU, but release b3188 does not.

[ggerganov]

pinging @JohannesGaessler

[fairydreaming]

@JohannesGaessler I printed tensors and one matrix multiplication result contains nan values for some reason:

  ggml_debug:                ffn_out-0 = (f32)    MUL_MAT(blk.0.ffn_down.weight{10944, 2048, 1, 1}, ffn_gate_par-0{10944, 15, 1, 1}}) = {2048, 15, 1, 1}
                                       [
                                        [
                                         [      0.0804,      -0.0651,       0.0438, ...,          nan,          nan,          nan],
                                         [     -0.1165,      -0.0231,       0.0386, ...,          nan,          nan,          nan],
                                         [      0.0430,      -0.0185,       0.0775, ...,          nan,          nan,          nan],
                                         ...,
                                         [      0.0604,       0.0104,       0.0488, ...,          nan,          nan,          nan],
                                         [     -0.1689,      -0.0501,      -0.0137, ...,          nan,          nan,          nan],
                                         [      0.0141,      -0.0206,       0.0457, ...,          nan,          nan,          nan],
                                        ],
                                       ]
                                       sum = nan

Examination of row values revealed that in each row there are 128 "normal" numbers, then 128 nan values, then 128 "normal" numbers, then 128 nan values and so on.

The same tensor in release b3187 before merging #8018:

ggml_debug:                ffn_out-0 = (f32)    MUL_MAT(blk.0.ffn_down.weight{10944, 2048, 1, 1}, ffn_gate_par-0{10944, 15, 1, 1}}) = {2048, 15, 1, 1}
                                       [
                                        [
                                         [      0.0804,      -0.0651,       0.0438, ...,       0.0313,       0.0803,      -0.0137],
                                         [     -0.1165,      -0.0231,       0.0387, ...,       0.0048,       0.0120,      -0.0650],
                                         [      0.0430,      -0.0185,       0.0775, ...,      -0.0002,       0.0231,       0.0108],
                                         ...,
                                         [      0.0604,       0.0104,       0.0488, ...,       0.0083,       0.0477,      -0.0069],
                                         [     -0.1689,      -0.0501,      -0.0137, ...,       0.0268,       0.0243,      -0.0653],
                                         [      0.0141,      -0.0206,       0.0457, ...,       0.0037,       0.0131,       0.0177],
                                        ],
                                       ]
                                       sum = 0.139421