Compared to SGLANG, Dynamo services have lower throughput. Is it a problem with my configuration?

[a4zhangfei]

Our test results show that the concurrency is not high. Is there a problem with my configuration?
PrefillWorker handles a single request every time. Is it the problem here? How to change it to batch?

dynamo/examples/llm/components/prefill_worker.py

Lines 103 to 106 in 457e78f

	prefill_request = await prefill_queue.dequeue_prefill_request()
	if prefill_request is not None:
	print(f"Dequeued prefill request: {prefill_request.request_id}")
	async for _ in self.generate(prefill_request):

Dyname result: Request throughput is 1.10 (TP1 BF16 3P1D)

============ Serving Benchmark Result ============
Backend:                                 vllm
Traffic request rate:                    32.0
Max reqeuest concurrency:                not set
Successful requests:                     1000
Benchmark duration (s):                  907.39
Total input tokens:                      4007607
Total generated tokens:                  56814
Total generated tokens (retokenized):    50731
Request throughput (req/s):              1.10
Input token throughput (tok/s):          4416.63
Output token throughput (tok/s):         62.61
Total token throughput (tok/s):          4479.24
Concurrency:                             481.16
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   436601.67
Median E2E Latency (ms):                 440035.82
---------------Time to First Token----------------
Mean TTFT (ms):                          433931.46
Median TTFT (ms):                        437615.93
P99 TTFT (ms):                           865498.48
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.60
Median ITL (ms):                         11.52
P95 ITL (ms):                            12.52
P99 ITL (ms):                            13.84
Max ITL (ms):                            119.50
==================================================

sglang result: Request throughput is 6.42 (TP1 BF16 3P1D)

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    32.0
Max reqeuest concurrency:                not set
Successful requests:                     1000
Benchmark duration (s):                  155.80
Total input tokens:                      4007607
Total generated tokens:                  109354
Total generated tokens (retokenized):    76166
Request throughput (req/s):              6.42
Input token throughput (tok/s):          25723.23    8574.41 tok/s/gpu
Output token throughput (tok/s):         701.90
Total token throughput (tok/s):          26425.13
Concurrency:                             398.61
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   62102.68
Median E2E Latency (ms):                 62686.63
---------------Time to First Token----------------
Mean TTFT (ms):                          60663.67
Median TTFT (ms):                        61226.33
P99 TTFT (ms):                           120785.82
---------------Inter-Token Latency----------------
Mean ITL (ms):                           13.30
Median ITL (ms):                         12.51
P95 ITL (ms):                            15.42
P99 ITL (ms):                            29.09
Max ITL (ms):                            322.48
==================================================

graph config: 3 Prefill worker and 1 decode worker

Frontend:
  served_model_name: model-thought-7b
  endpoint: dynamo.Processor.chat/completions
  port: 8000
  ServiceArgs:
    workers: 4

Processor:
  model: /mnt/yscfs/model-thought/rag_fc_7b_20250314_Qwen25_7b_instruct_yuanshi_chat_v4_bf16_thought_action_250313/
  router: round-robin
  ServiceArgs:
    workers: 4

VllmWorker:
  model: /mnt/yscfs/model-thought/rag_fc_7b_20250314_Qwen25_7b_instruct_yuanshi_chat_v4_bf16_thought_action_250313/
  kv-transfer-config: '{"kv_connector":"DynamoNixlConnector"}'
  max-model-len: 16384
  remote-prefill: true
  conditional-disagg: true
  max-local-prefill-length: 10
  tensor-parallel-size: 1
  ServiceArgs:
    workers: 1
    resources:
      gpu: 1

PrefillWorker:
  model: /mnt/yscfs/model-thought/rag_fc_7b_20250314_Qwen25_7b_instruct_yuanshi_chat_v4_bf16_thought_action_250313/
  kv-transfer-config: '{"kv_connector":"DynamoNixlConnector"}'
  max-model-len: 16384
  max-num-batched-tokens: 16384
  tensor-parallel-size: 1
  ServiceArgs:
    workers: 3
    resources:
      gpu: 1

[liao4s]

Hi, I'm sorry I don't know how to solve your problem. But I have a request. Is the benchmark you use written by yourself? Or open source code? Can you share this benchmark?

[a4zhangfei]

Hi, I'm sorry I don't know how to solve your problem. But I have a request. Is the benchmark you use written by yourself? Or open source code? Can you share this benchmark?

@liao4s we based on SGLang's bench_serving：https://github.com/sgl-project/sglang/blob/main/python/sglang/bench_serving.py

[liao4s]

Hi, I'm sorry I don't know how to solve your problem. But I have a request. Is the benchmark you use written by yourself? Or open source code? Can you share this benchmark?

@liao4s we based on SGLang's bench_serving：https://github.com/sgl-project/sglang/blob/main/python/sglang/bench_serving.py

Thank you so much！

[sitabulaixizawaluduo]

The disaggregate PD of sglang should not currently include kv cache transfer, so the throughput will be higher.

[a4zhangfei]

The disaggregate PD of sglang should not currently include kv cache transfer, so the throughput will be higher.
@sitabulaixizawaluduo

We have made modifications based on sglang and already support kv transmission. The above test feels very problematic and does not start concurrently.

[sitabulaixizawaluduo]

The disaggregate PD of sglang should not currently include kv cache transfer, so the throughput will be higher.
@sitabulaixizawaluduo
We have made modifications based on sglang and already support kv transmission. The above test feels very problematic and does not start concurrently.

The results show that the TTFT of SGLang is smaller and the ITL is similar, indicating that the efficiency of KV Cache transfer is higher?

[liao4s]

Hi, I'm sorry I don't know how to solve your problem. But I have a request. Is the benchmark you use written by yourself? Or open source code? Can you share this benchmark?

@liao4s we based on SGLang's bench_serving：https://github.com/sgl-project/sglang/blob/main/python/sglang/bench_serving.py

In this script, if vllm is backend, the API_URL used is /v1/completions. It seems that dynamo does not support this form now (someone mentioned it in the issue, as shown in the following figure). Can you tell me what changes you made?

elif args.backend in ["sglang-oai", "vllm", "lmdeploy"]:
        api_url = (
            f"{args.base_url}/v1/completions"
            if args.base_url
            else f"http://{args.host}:{args.port}/v1/completions"
        )

[a4zhangfei]

base_url

We changed some of the code to adapt to dynamo's request：

[liao4s]

base_url
We changed some of the code to adapt to dynamo's request：

Thanks! This benchmark is running successfully!

[kkranen]

Dynamo using VLLM should use inflight batching by default, so I'm not sure this is a batching issue.

As mentioned, ITLs look on the same order of magnitude, this seems like Prefill being an issue, the question is why.

Would you be able to send the logs showing the distribution of workers routed to for prefill? Should be round robin, but there could be a bug with worker stackup.

[gujingit]

@a4zhangfei Hi, how can I run SGLang with PD-disagg? It seems that SGLang does not currently support PD-disagg, as noted in sgl-project/sglang#4655.

[mjp9527]

parser.add_argument(
        "--max-prefill-queue-size",
        type=int,
        default=3,
        help="Do not send remote prefill requests (prefill locally) if the queue size is greater than this value",
    )

Maybe it's because this parameter wasn't set? All the requests are using locally prefill .

[liao4s]

Maybe you didn't send enough requests? How about a few more tests? I ran the tests on 8xH100, each time I set it to 500 requests, and found that the results of the first test were significantly worse than the results of the second and third tests, and kept the vllm api_server running for multiple tests. I guess it's because the model needs to "warm up."

[a4zhangfei]

Maybe you didn't send enough requests? How about a few more tests? I ran the tests on 8xH100, each time I set it to 500 requests, and found that the results of the first test were significantly worse than the results of the second and third tests, and kept the vllm api_server running for multiple tests. I guess it's because the model needs to "warm up."

I am the test after warmup.