Internal llm-d deployments (DSV4.1 H200)¶
BLIS binaries these results describe
(devel)+9e3c509c8788fe0743605a8f06e0bf5c1c39ac1f, binary sha25629fb09054a10ee51dc104f491cb09337e0ba527055b85653d281f23230451dff
DeepSeek-V4.1-Flash on H200 under vLLM, measured on internal llm-d deployments: single-replica ablations of the TP8 deployment (no DSpark, no KV offload, no Engram), a TP4 deployment, and one- and two-replica aggregated deployments behind the EPP router. The workload is an agentic multi-turn corpus replayed closed loop, 900 s per level, c8 to c512.
These runs are not public, so the pages cite no URL and carry no measured value; the source reports are pinned by sha256 in the validation inputs.
Build: 9e3c509c. Signed error 100 * (BLIS / measured - 1).
Provenance verdict¶
Agg-1r: PARTIAL¶
Scored levels: c8, c16, c32, c64, c128, c256. Excluded: c512 (measured CPU-tier hits above zero, or router behaviour unresolved above its concurrency guard).
| kind | difference |
|---|---|
| unremovable | Synthetic speculative acceptance: vLLM's per-position schedule (2 or 3 drafts accepted, mean 3.51) vs BLIS's deterministic mean acceptance length 3.51. |
| unremovable | KV CPU tier: vLLM's blocks_per_chunk 4 is not representable (BLIS uses 1), and BLIS does not price the CPU-to-GPU reload. Material only where measured CPU-tier hits are above zero. |
| unstated | Multimodal flags vs --language-model-only: which one ran is UNSTATED (memory only). |
| unstated | No raw run artifacts (benchmark-client export, server log) were kept. |
| unstated | Block size 64, fp8 KV, FULL_AND_PIECEWISE graphs and max_num_seqs 1024 are not in the run manifest; BLIS uses vLLM's defaults or backend-forced values. |
| match | Router: the EPP v0.10.0 configuration matches (MATCH). |
Agg-2r: PARTIAL¶
Scored levels: c8, c16, c32, c64, c128, c256, c512.
| kind | difference |
|---|---|
| unremovable | Synthetic speculative acceptance: vLLM's per-position schedule (2 or 3 drafts accepted, mean 3.51) vs BLIS's deterministic mean acceptance length 3.51. |
| unremovable | KV CPU tier: vLLM's blocks_per_chunk 4 is not representable (BLIS uses 1), and BLIS does not price the CPU-to-GPU reload. Material only where measured CPU-tier hits are above zero. |
| unstated | Multimodal flags vs --language-model-only: which one ran is UNSTATED (memory only). |
| unstated | No raw run artifacts (benchmark-client export, server log) were kept. |
| unstated | Block size 64, fp8 KV, FULL_AND_PIECEWISE graphs and max_num_seqs 1024 are not in the run manifest; BLIS uses vLLM's defaults or backend-forced values. |
| match | Inter-node fabric speed unrecorded; two BLIS fabrics give byte-identical results (immaterial). |
TP4-baseline: PARTIAL¶
Scored levels: c8, c16, c32, c64. Excluded: c128, c256, c512 (measured CPU-tier hits above zero, or router behaviour unresolved above its concurrency guard).
| kind | difference |
|---|---|
| unremovable | Synthetic speculative acceptance: vLLM's per-position schedule (2 or 3 drafts accepted, mean 3.51) vs BLIS's deterministic mean acceptance length 3.51. |
| unremovable | KV CPU tier: vLLM's blocks_per_chunk 4 is not representable (BLIS uses 1), and BLIS does not price the CPU-to-GPU reload. Material only where measured CPU-tier hits are above zero. |
| unstated | Endpoint: the report says direct to vLLM, the committed environment names the EPP router. UNSTATED; immaterial up to c256, where the measured router and direct arms agree. |
| unstated | Multimodal flags vs --language-model-only: which one ran is UNSTATED (memory only). |
| unstated | No raw run artifacts (benchmark-client export, server log) were kept. |
| unstated | Block size 64, fp8 KV, FULL_AND_PIECEWISE graphs and max_num_seqs 1024 are not in the run manifest; BLIS uses vLLM's defaults or backend-forced values. |
| unstated | The TP4 manifest comes from an earlier revision, and the reports disagree on the GPU count (8 vs 4), so per-GPU metrics are not compared. |
TP8-no-DSpark: PARTIAL¶
Scored levels: c8, c16, c32, c64, c128, c256. Excluded: c512 (measured CPU-tier hits above zero, or router behaviour unresolved above its concurrency guard).
| kind | difference |
|---|---|
| unremovable | KV CPU tier: vLLM's blocks_per_chunk 4 is not representable (BLIS uses 1), and BLIS does not price the CPU-to-GPU reload. Material only where measured CPU-tier hits are above zero. |
| unstated | Endpoint: the report says direct to vLLM, the committed environment names the EPP router. UNSTATED; immaterial up to c256, where the measured router and direct arms agree. |
| unstated | Multimodal flags vs --language-model-only: which one ran is UNSTATED (memory only). |
| unstated | No raw run artifacts (benchmark-client export, server log) were kept. |
| unstated | Block size 64, fp8 KV, FULL_AND_PIECEWISE graphs and max_num_seqs 1024 are not in the run manifest; BLIS uses vLLM's defaults or backend-forced values. |
| unstated | The ablation manifest was not kept; settings come from the report text and the start-up sizing log. |
TP8-no-Engram: PARTIAL¶
Scored levels: c8, c16, c32, c64, c128, c256. Excluded: c512 (measured CPU-tier hits above zero, or router behaviour unresolved above its concurrency guard).
| kind | difference |
|---|---|
| unremovable | Synthetic speculative acceptance: vLLM's per-position schedule (2 or 3 drafts accepted, mean 3.51) vs BLIS's deterministic mean acceptance length 3.51. |
| unremovable | KV CPU tier: vLLM's blocks_per_chunk 4 is not representable (BLIS uses 1), and BLIS does not price the CPU-to-GPU reload. Material only where measured CPU-tier hits are above zero. |
| unstated | Endpoint: the report says direct to vLLM, the committed environment names the EPP router. UNSTATED; immaterial up to c256, where the measured router and direct arms agree. |
| unstated | Multimodal flags vs --language-model-only: which one ran is UNSTATED (memory only). |
| unstated | No raw run artifacts (benchmark-client export, server log) were kept. |
| unstated | Block size 64, fp8 KV, FULL_AND_PIECEWISE graphs and max_num_seqs 1024 are not in the run manifest; BLIS uses vLLM's defaults or backend-forced values. |
| note | This ablation ran the baseline configuration (Engram cpu_offload defaults to true in vLLM v0.31.0), so it is a measured replicate of the baseline. |
TP8-no-KV-offload: PARTIAL¶
Scored levels: c8, c16, c32, c64, c128, c256. Excluded: c512 (measured CPU-tier hits above zero, or router behaviour unresolved above its concurrency guard).
| kind | difference |
|---|---|
| unremovable | Synthetic speculative acceptance: vLLM's per-position schedule (2 or 3 drafts accepted, mean 3.51) vs BLIS's deterministic mean acceptance length 3.51. |
| unstated | Endpoint: the report says direct to vLLM, the committed environment names the EPP router. UNSTATED; immaterial up to c256, where the measured router and direct arms agree. |
| unstated | Multimodal flags vs --language-model-only: which one ran is UNSTATED (memory only). |
| unstated | No raw run artifacts (benchmark-client export, server log) were kept. |
| unstated | Block size 64, fp8 KV, FULL_AND_PIECEWISE graphs and max_num_seqs 1024 are not in the run manifest; BLIS uses vLLM's defaults or backend-forced values. |
| unstated | The ablation manifest was not kept; settings come from the report text and the start-up sizing log. |
Gap re-checks¶
Each modelling gap claimed for these deployments, re-checked by paired BLIS probes (latest build):
| gap | empirical verdict |
|---|---|
| CPU offload tier: CPU-to-GPU reload unpriced | CONFIRMED |
| Engram host gather | CONFIRMED (partial): placement changes KV only; step-time effect <= 2.1% ITL at c32 |
| all-to-all backend | moot at DP=1 (byte-identical across 4 backends); REFUTED at DP=2: deepep_low_latency vs deepep_high_throughput differ |
| inter-node fabric | immaterial |
What would make an arm EXACT: the raw run artifacts and ablation manifest kept with the run; every UNSTATED setting stated; BLIS representing vLLM's blocks_per_chunk and pricing the CPU-to-GPU reload (or a run without CPU-tier hits); and the speculative acceptance schedule replayed per position.
Output metrics¶
Over the scored levels. MAPE is mean absolute error; shape error divides each side by its lowest-level value and excludes that level.
| arm | metric | signed mean | MAPE % | shape error % | sign |
|---|---|---|---|---|---|
| Agg-1r | output tok/s | +69.6% | 69.6 | 66.5 | BLIS high at every level |
| Agg-1r | output tok/s per user | +86.9% | 95.9 | 187.5 | mixed (5 high / 1 low) |
| Agg-1r | interactivity p90 | +174.6% | 174.6 | 104.7 | BLIS high at every level |
| Agg-1r | ITL mean | -47.7% | 47.7 | 41.1 | BLIS low at every level |
| Agg-1r | ITL p90 | -50.9% | 50.9 | 33.6 | BLIS low at every level |
| Agg-2r | output tok/s | +59.2% | 59.2 | 55.8 | BLIS high at every level |
| Agg-2r | output tok/s per user | +65.9% | 65.9 | 53.0 | BLIS high at every level |
| Agg-2r | interactivity p90 | +116.0% | 116.0 | 78.3 | BLIS high at every level |
| Agg-2r | ITL mean | -45.0% | 45.0 | 37.4 | BLIS low at every level |
| Agg-2r | ITL p90 | -44.1% | 44.1 | 32.4 | BLIS low at every level |
| TP4-baseline | output tok/s | +43.5% | 43.5 | 46.4 | BLIS high at every level |
| TP4-baseline | output tok/s per user | +11.1% | 63.6 | 92.6 | mixed (2 high / 2 low) |
| TP4-baseline | interactivity p90 | +99.2% | 99.2 | 76.5 | BLIS high at every level |
| TP4-baseline | ITL mean | -39.0% | 39.0 | 41.4 | BLIS low at every level |
| TP4-baseline | ITL p90 | -41.2% | 41.2 | 34.3 | BLIS low at every level |
| TP8-no-DSpark | output tok/s | +67.0% | 67.0 | 63.3 | BLIS high at every level |
| TP8-no-DSpark | output tok/s per user | +64.5% | 64.5 | 64.1 | BLIS high at every level |
| TP8-no-DSpark | interactivity p90 | +92.7% | 92.7 | 36.3 | BLIS high at every level |
| TP8-no-DSpark | ITL mean | -42.7% | 42.7 | 24.6 | BLIS low at every level |
| TP8-no-DSpark | ITL p90 | -45.2% | 45.2 | 23.1 | BLIS low at every level |
| TP8-no-Engram | output tok/s | +66.7% | 66.7 | 63.3 | BLIS high at every level |
| TP8-no-Engram | output tok/s per user | +59.5% | 86.3 | 852.1 | mixed (5 high / 1 low) |
| TP8-no-Engram | interactivity p90 | +162.4% | 162.4 | 99.6 | BLIS high at every level |
| TP8-no-Engram | ITL mean | -46.4% | 46.4 | 39.4 | BLIS low at every level |
| TP8-no-Engram | ITL p90 | -49.3% | 49.3 | 32.7 | BLIS low at every level |
| TP8-no-KV-offload | output tok/s | +40.6% | 46.7 | 45.5 | mixed (5 high / 1 low) |
| TP8-no-KV-offload | output tok/s per user | +68.8% | 70.8 | 66.5 | mixed (5 high / 1 low) |
| TP8-no-KV-offload | interactivity p90 | +141.8% | 141.8 | 98.0 | BLIS high at every level |
| TP8-no-KV-offload | ITL mean | -37.8% | 38.4 | 37.8 | mixed (1 high / 5 low) |
| TP8-no-KV-offload | ITL p90 | -40.5% | 40.5 | 37.8 | BLIS low at every level |
TTFT¶
| arm | metric | signed mean | MAPE % | shape error % | sign |
|---|---|---|---|---|---|
| Agg-1r | TTFT p50 | -62.7% | 62.7 | 13.1 | BLIS low at every level |
| Agg-2r | TTFT p50 | -65.6% | 65.6 | 14.9 | BLIS low at every level |
| TP4-baseline | TTFT p50 | -60.5% | 60.5 | 13.4 | BLIS low at every level |
| TP8-no-DSpark | TTFT p50 | -59.2% | 59.2 | 9.7 | BLIS low at every level |
| TP8-no-Engram | TTFT p50 | -61.1% | 61.1 | 12.4 | BLIS low at every level |
| TP8-no-KV-offload | TTFT p50 | +4.8% | 103.1 | 193.5 | mixed (1 high / 5 low) |
KV pool¶
BLIS's KV capacity per replica divided by the KV tokens vLLM logged at start-up (the report's sizing table). The aggregated arms have no sizing row.
| arm | BLIS / vLLM-logged |
|---|---|
| Agg-1r | not logged |
| Agg-2r | not logged |
| TP4-baseline | 1.57 |
| TP8-no-DSpark | 1.29 |
| TP8-no-Engram | 1.38 |
| TP8-no-KV-offload | 1.38 |