Heterogeneous Inference: llm-d and round robin on Dell PowerEdge R7725

Concurrency within the latency target
p90 time to first token at or under 2 s
3.2×higher
Round robin65
llm-d208
Time to first token
p90 at the same concurrency, about 65
5.3×faster
Round robin1.70s
llm-d0.32s
Output throughput
Peak tokens per second across the pool
1.3×higher
Round robin2,502tok/s
llm-d3,249tok/s
Prefix cache reuse
Requests with their prefix cached, at the highest concurrency in target
4.0×higher
Round robin22%
llm-d89%
Energy efficiency
Output tokens per joule, all four GPUs
1.1×higher
Round robin1.23tok/J
llm-d1.35tok/J

How each router places requests

Current promptBClassify this support ticket

Prefix in cache, 0.21 s
Full prefill, 0.65 s

Fig 1Standard round robin

Prefix cached: 0%
Last TTFT: 0.65s

Kubernetes sends each request to the next GPU in turn.

Fig 2llm-d

Prefix cached: 67%
Last TTFT: 0.21s

llm-d sends each prefix to the GPU that already holds it.

System under test

Same hardware, model and prompts for both routers

2× Dell PowerEdge R7725
2× AMD EPYC 9965, 384 cores per server
2× RTX PRO 6000
NVIDIA Blackwell, 96 GB
2× Instinct MI350P
AMD, 144 GB HBM3E

Qwen3.8-27B on vLLM 0.31, one serving per GPU, prefix caching on

Supporting measurements

Medians of three runs, measured 2 Nov 2026. Each router served 26,280 requests.

Supporting measurements
Measure	Standard round robin	llm-d	Difference
Median time to first token at concurrency about 65	0.77 s	0.22 s	3.6× faster
p90 time to first token at each router's highest concurrency within the target	1.70 s at 65	1.10 s at 208	3.2× the concurrency
Concurrency at peak throughput	721	208	llm-d peaks earlier and higher
Prompt tokens computed in prefill	188.8 M	135.2 M	28% fewer
GPU energy per output token	0.816 J	0.739 J	9% less
Mean pool power over the run	1.51 kW	1.45 kW	Similar draw
Failed requests	1 of 26,280	0 of 26,280	Both runs complete

Results by concurrency

Each point is one 60-second load step, the median of three runs. Concurrency is the average number of requests sent and not yet finished during the step. Dotted lines mark each router's highest concurrency within the target.

Time to first token

p90 per load step, seconds, log scale

Round robin
llm-d

Time to first token, p90 per load step, seconds, log scale
Load step	Standard round robin	llm-d
1	0.78 s at concurrency 11	0.25 s at concurrency 10
2	1.05 s at concurrency 20	0.29 s at concurrency 18
3	1.38 s at concurrency 35	0.26 s at concurrency 30
4	1.14 s at concurrency 49	0.28 s at concurrency 41
5	1.70 s at concurrency 65	0.35 s at concurrency 53
6	2.83 s at concurrency 84	0.32 s at concurrency 63
7	3.31 s at concurrency 100	0.38 s at concurrency 82
8	5.57 s at concurrency 148	0.38 s at concurrency 94
9	10.48 s at concurrency 232	0.52 s at concurrency 124
10	68.25 s at concurrency 450	1.10 s at concurrency 208
11	185.10 s at concurrency 721	47.96 s at concurrency 297
12	293.39 s at concurrency 943	205.13 s at concurrency 454
13	502.06 s at concurrency 1,328	367.08 s at concurrency 793
14	825.32 s at concurrency 1,967	692.35 s at concurrency 1,481
15	1,272.07 s at concurrency 2,857	1,114.89 s at concurrency 2,351
16	1,804.95 s at concurrency 3,926	1,636.53 s at concurrency 3,386
17	2,539.12 s at concurrency 5,356	2,260.20 s at concurrency 4,753

Output throughput

Tokens per second across the pool

Round robin
llm-d

Output throughput, Tokens per second across the pool
Load step	Standard round robin	llm-d
1	461 tok/s at concurrency 11	468 tok/s at concurrency 10
2	803 tok/s at concurrency 20	821 tok/s at concurrency 18
3	1,216 tok/s at concurrency 35	1,266 tok/s at concurrency 30
4	1,463 tok/s at concurrency 49	1,589 tok/s at concurrency 41
5	1,765 tok/s at concurrency 65	1,955 tok/s at concurrency 53
6	1,954 tok/s at concurrency 84	2,252 tok/s at concurrency 63
7	1,688 tok/s at concurrency 100	2,511 tok/s at concurrency 82
8	1,997 tok/s at concurrency 148	2,796 tok/s at concurrency 94
9	1,842 tok/s at concurrency 232	2,826 tok/s at concurrency 124
10	2,313 tok/s at concurrency 450	3,249 tok/s at concurrency 208
11	2,502 tok/s at concurrency 721	3,178 tok/s at concurrency 297
12	2,378 tok/s at concurrency 943	3,131 tok/s at concurrency 454
13	2,261 tok/s at concurrency 1,328	2,917 tok/s at concurrency 793
14	2,326 tok/s at concurrency 1,967	3,020 tok/s at concurrency 1,481
15	2,336 tok/s at concurrency 2,857	2,679 tok/s at concurrency 2,351
16	2,322 tok/s at concurrency 3,926	2,353 tok/s at concurrency 3,386
17	2,272 tok/s at concurrency 5,356	2,352 tok/s at concurrency 4,753

Prefix cache reuse

Share of requests whose prefix was already cached

Round robin
llm-d

Prefix cache reuse, Share of requests whose prefix was already cached
Load step	Standard round robin	llm-d
1	27 % at concurrency 11	97 % at concurrency 10
2	22 % at concurrency 20	93 % at concurrency 18
3	19 % at concurrency 35	99 % at concurrency 30
4	25 % at concurrency 49	98 % at concurrency 41
5	22 % at concurrency 65	96 % at concurrency 53
6	19 % at concurrency 84	98 % at concurrency 63
7	17 % at concurrency 100	94 % at concurrency 82
8	19 % at concurrency 148	97 % at concurrency 94
9	16 % at concurrency 232	95 % at concurrency 124
10	10 % at concurrency 450	89 % at concurrency 208
11	1 % at concurrency 721	69 % at concurrency 297
12	2 % at concurrency 943	53 % at concurrency 454
13	1 % at concurrency 1,328	40 % at concurrency 793
14	2 % at concurrency 1,967	7 % at concurrency 1,481
15	2 % at concurrency 2,857	11 % at concurrency 2,351
16	2 % at concurrency 3,926	12 % at concurrency 3,386
17	2 % at concurrency 5,356	15 % at concurrency 4,753

Energy efficiency

Output tokens per joule in each load step, all four GPUs

Round robin
llm-d

Energy efficiency, Output tokens per joule in each load step, all four GPUs
Load step	Standard round robin	llm-d
1	0.40 tok/J at concurrency 11	0.40 tok/J at concurrency 10
2	0.61 tok/J at concurrency 20	0.65 tok/J at concurrency 18
3	0.90 tok/J at concurrency 35	1.00 tok/J at concurrency 30
4	1.03 tok/J at concurrency 49	1.22 tok/J at concurrency 41
5	1.18 tok/J at concurrency 65	1.44 tok/J at concurrency 53
6	1.27 tok/J at concurrency 84	1.63 tok/J at concurrency 63
7	1.08 tok/J at concurrency 100	1.73 tok/J at concurrency 82
8	1.26 tok/J at concurrency 148	1.85 tok/J at concurrency 94
9	1.13 tok/J at concurrency 232	1.88 tok/J at concurrency 124
10	1.38 tok/J at concurrency 450	2.00 tok/J at concurrency 208
11	1.46 tok/J at concurrency 721	1.93 tok/J at concurrency 297
12	1.35 tok/J at concurrency 943	1.92 tok/J at concurrency 454
13	1.32 tok/J at concurrency 1,328	1.74 tok/J at concurrency 793
14	1.40 tok/J at concurrency 1,967	1.76 tok/J at concurrency 1,481
15	1.34 tok/J at concurrency 2,857	1.53 tok/J at concurrency 2,351
16	1.32 tok/J at concurrency 3,926	1.39 tok/J at concurrency 3,386
17	1.34 tok/J at concurrency 5,356	1.38 tok/J at concurrency 4,753

Where the work ran

The pool has two GPU types. The bars show how each router spread the work across them.

KV cache in use, by GPU type

2× NVIDIA RTX PRO 6000
2× AMD Instinct MI350P

Round robin

% of pool KV in use

llm-d

% of pool KV in use

Measured at each router's highest concurrency within the target: 65 for round robin, 208 for llm-d. The AMD pair holds 62% of the pool's KV memory, and the dashed line marks that split.

Pool power

Mean GPU power across all four GPUs over the run

Round robin kW

llm-d kW

Both routers draw about the same power. llm-d turns more of it into output: 0.739 J per token against 0.816 J for round robin.

Full configuration

Both routers ran on the same servers with the same serving flags. Only the router in front of the four GPUs changed.

Servers, GPUs and serving configuration
Item	Node 1	Node 2
Server	Dell PowerEdge R7725	Dell PowerEdge R7725
GPUs	2× NVIDIA RTX PRO 6000 Blackwell, 96 GB each	2× AMD Instinct MI350P, 144 GB HBM3E each
CPUs	2× AMD EPYC 9965, 384 cores	2× AMD EPYC 9965, 384 cores
Memory	1.5 TB DDR5-6400	1.5 TB DDR5-6400
Storage	3× 3.84 TB NVMe	4× 3.84 TB NVMe
Network	1 GbE and a 400 GbE NIC	1 GbE and a 400 GbE NIC
Kubernetes role	k3s server, control plane and router	k3s agent
Checkpoint	`nvidia/Qwen3.8-27B-NVFP4`	`amd/Qwen3.8-27B-Quark-AWQ-MXFP4`
Precision	NVFP4 MLP, FP8 attention	MXFP4 weights and activations
Runtime	`vLLM 0.31, CUDA 13.2`	`vLLM 0.31, ROCm 10.1`
Context served	16,384 tokens	16,384 tokens
Tensor parallel	1 per serving, 2 servings	1 per serving, 2 servings
Prefix caching	On, identical for both routers	On, identical for both routers

GPU memory per GPU

Each vendor loads its own FP4 build of the same model. Weights take about 20 GB on both, and the rest holds KV cache.

Model weights
KV cache and headroom

NVIDIA RTX PRO 6000

AMD Instinct MI350P

llm-d holds 3.2× the concurrency of round robin within the 2 s latency target

Key results

Concurrency within the latency target