llm-d Prism¶
BLIS binaries these results describe
(devel)+9e3c509c8788fe0743605a8f06e0bf5c1c39ac1f, binary sha25629fb09054a10ee51dc104f491cb09337e0ba527055b85653d281f23230451dff
Public llm-d benchmark results (llm-d-prism and the gs://llm-d-benchmarks bucket) for models in blis-catalog, replayed by BLIS.
Signed error 100 * (BLIS / measured - 1): positive means BLIS is higher (slower latency, or more throughput). Geo-mean bias is 100 * (exp(mean log ratio) - 1).
Provenance verdict¶
No Prism run is EXACT (0 EXACT anchors). Every run on a catalog model leaves material settings unstated (vLLM version, serve args, router config, offload size) with no run artifact that resolves them, so each anchor is PARTIAL (the source analysis calls it APPROXIMATE).
A1: PARTIAL¶
Llama-3.1-70B on H100, standalone vLLM (14 replica x TP configs, concurrency 1 to 256, random 10000/1000).
Source: llm-d-prism archive at 6769be260f44; serve args from llm-d-benchmark 58b2d1b.
- vLLM image
vllm-openai:lateston the run dates (v0.10.1.1 or v0.10.2); BLIS prices v0.31.0 behaviour - serve args reconstructed from the benchmark repo plus vLLM v0.10.2 defaults; the replica x TP treatment list is in no commit
- routing is a Kubernetes Service (connection-level random); BLIS uses round-robin
- the archive keeps no p90 and no per-request data; output tok/s is derived from the request rate
A2: PARTIAL¶
Llama-3.1-70B on H100, P/D through llm-d (5 prefill/decode layouts), as recorded (disaggregated).
Source: same archive; llm-d-benchmark e98acd0.
- all of A1's differences
- llm-d image tag
auto(vLLM version unknown); gpu-memory-utilization not passed - router
plugins-v2.yamlon inference-scheduler v0.2.1, which BLIS's EPP config cannot ingest, with no P/D profile handler - NIXL over TCP; BLIS uses a nominal 100GbE fabric; BLIS pools cannot share a node
A2': PARTIAL¶
The same P/D runs modelled as colocated replicas (hypothesis: the measured runs did not disaggregate).
Source: same as A2.
- all of A2's differences
- the mixed-TP layout is omitted
B: PARTIAL¶
Llama-3.3-70B on H100, 4 x TP4, shared-prefix 30k/50k/70k at concurrency 20, with prefix-cache offloading.
Source: 7 curated reports in the public bucket gs://llm-d-benchmarks/prefix-cache-offloading/.
- model graph is Llama-3.1-70B's (architecture-identical config)
- LMCache and llm-d-fs offload are not modelled; the CPU tier size is unstated
- the EPP plugin config is unstated, so two BLIS routings bracket it
- vLLM version unknown (llm-d-cuda v0.5.1/v0.6.0); the input length is the measured mean
What would make a run EXACT: the vLLM version, serve args and router config recorded with the run, an EPP version BLIS can ingest, and (for B) an offload connector BLIS models with its CPU tier size stated.
Sets that cannot be scored at all (NOT_REPRODUCIBLE):
| set | needed |
|---|---|
| Qwen/Qwen3-32B | catalog entry models/qwen3-32b (config.json + graph.yaml) |
| facebook/opt-125m | catalog entry models/opt-125m (config.json + graph.yaml) |
| openai/gpt-oss-120b | a run with non-sanity load and a recorded deployment (plan/ |
| Qwen/Qwen3-VL-32B-Instruct | catalog entry models/qwen3-vl-32b-instruct (config.json + graph.yaml) |
| amd/Llama-3.3-70B-Instruct-FP8-KV | catalog entry models/llama-3.3-70b-instruct-fp8-kv (config.json + graph.yaml) |
| Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 | catalog entry models/qwen3-coder-480b-a35b-instruct-fp8 (config.json + graph.yaml) |
| Qwen/Qwen3-0.6B | catalog entry models/qwen3-0.6b (config.json + graph.yaml) |
| deepseek-ai/DeepSeek-V3 | a run with non-sanity load and a recorded deployment (plan/ |
| deepseek-ai/DeepSeek-R1-0528 | catalog entry models/deepseek-r1-0528 (config.json + graph.yaml) |
| ibm-granite/granite-3.3-2b-instruct | catalog entry models/granite-3.3-2b-instruct (config.json + graph.yaml) |
| meta-llama/Llama-3.1-8B-Instruct | a run with non-sanity load and a recorded deployment (plan/ |
| HuggingFaceTB/SmolLM2-360M-Instruct | catalog entry models/smollm2-360m-instruct (config.json + graph.yaml) |
| deepseek-ai/DeepSeek-V2-Lite-Chat | a run with non-sanity load and a recorded deployment (plan/ |
| unlabelled-model runs | Qwen3-32B catalog entry |
| GIQ profiles | per-profile parallelism and serve args |
Errors, output metrics first¶
| anchor | metric | points | signed mean | MAPE % | geo-mean bias | shape error % | share BLIS-high |
|---|---|---|---|---|---|---|---|
| A1 | output tok/s | 84 | +7.2% | 23.0 | +3.7% | 45.0 | 0.55 |
| A1 | TPOT mean | 84 | +30.5% | 39.7 | +25.7% | 6.3 | 0.86 |
| A1 | ITL mean | 84 | +30.5% | 39.7 | +25.7% | 6.3 | 0.86 |
| A1 | per-user tok/s | 84 | -16.3% | 31.3 | -20.5% | 5.7 | 0.14 |
| A1 | TTFT mean | 84 | -13.6% | 27.8 | -23.8% | 29.8 | 0.24 |
| A1 | TTFT p50 | 84 | +0.2% | 30.8 | -13.6% | 31.7 | 0.29 |
| A2 | output tok/s | 29 | -20.1% | 28.2 | -28.6% | 25.3 | 0.24 |
| A2 | TPOT mean | 29 | +27.7% | 53.4 | +8.6% | 40.7 | 0.55 |
| A2 | ITL mean | 29 | +27.1% | 52.7 | +8.4% | 40.0 | 0.55 |
| A2 | per-user tok/s | 29 | +8.1% | 45.1 | -7.9% | 57.2 | 0.45 |
| A2 | TTFT mean | 29 | +331.8% | 363.5 | +83.3% | 318.9 | 0.59 |
| A2 | TTFT p50 | 29 | +705.0% | 739.2 | +108.9% | 722.5 | 0.59 |
| A2' | output tok/s | 24 | -13.1% | 13.1 | -13.6% | 20.9 | 0.00 |
| A2' | TPOT mean | 24 | +26.0% | 26.0 | +25.6% | 8.9 | 1.00 |
| A2' | ITL mean | 24 | +26.0% | 26.0 | +25.6% | 8.9 | 1.00 |
| A2' | per-user tok/s | 24 | -20.2% | 20.2 | -20.4% | 10.0 | 0.00 |
| A2' | TTFT mean | 24 | -27.1% | 35.5 | -33.4% | 48.8 | 0.21 |
| A2' | TTFT p50 | 24 | -34.2% | 39.1 | -41.1% | 49.8 | 0.21 |
| B (prefix routing) | output tok/s | 7 | -12.5% | 12.5 | -13.3% | -- | 0.14 |
| B (prefix routing) | TPOT mean | 7 | +47.2% | 47.2 | +45.4% | -- | 1.00 |
| B (prefix routing) | TPOT p90 | 7 | +72.7% | 72.7 | +71.3% | -- | 1.00 |
| B (prefix routing) | ITL mean | 7 | +47.2% | 47.2 | +45.4% | -- | 1.00 |
| B (prefix routing) | ITL p90 | 7 | +19.3% | 19.3 | +19.2% | -- | 1.00 |
| B (prefix routing) | TTFT mean | 7 | -9.7% | 19.7 | -11.9% | -- | 0.29 |
| B (prefix routing) | TTFT p50 | 7 | -68.3% | 74.6 | -96.7% | -- | 0.14 |
| B (rr routing) | output tok/s | 7 | -13.5% | 16.9 | -14.9% | -- | 0.14 |
| B (rr routing) | TPOT mean | 7 | +65.1% | 65.1 | +61.3% | -- | 1.00 |
| B (rr routing) | TPOT p90 | 7 | +54.0% | 56.3 | +50.1% | -- | 0.86 |
| B (rr routing) | ITL mean | 7 | +65.1% | 65.1 | +61.3% | -- | 1.00 |
| B (rr routing) | ITL p90 | 7 | +18.1% | 18.1 | +18.0% | -- | 1.00 |
| B (rr routing) | TTFT mean | 7 | -26.1% | 31.4 | -45.6% | -- | 0.29 |
| B (rr routing) | TTFT p50 | 7 | -39.9% | 57.8 | -61.6% | -- | 0.29 |
KV pool: the Prism records carry no vLLM start-up KV figure, so no KV-pool ratio is reported.