InferenceX DSV4.1 agentic (request-level pairs)¶
BLIS binaries these results describe
(devel)+9e3c509c8788fe0743605a8f06e0bf5c1c39ac1f, binary sha25629fb09054a10ee51dc104f491cb09337e0ba527055b85653d281f23230451dff
DeepSeek-V4.1-Flash on Hopper, vLLM with DSpark speculative decoding, driven by InferenceX's agentic replay (aiperf scenario inferencex-agentx-mvp). BLIS replays the same trace corpus and seed. Measured per-request records come from the runs' GitHub Actions artifacts, bound to the published database rows by recomputing every row with InferenceX's own code.
Build: 9e3c509c. Every error is 100 * (BLIS / measured - 1): negative means BLIS is faster (lower latency) or, for throughput and interactivity, lower.
Sources¶
| arm | deployment | InferenceX commit | launch script | aiperf (submodule) | measured rows |
|---|---|---|---|---|---|
h100-tp8-0918 |
H100 TP8 (run of 09-18) | 578048f90375 | benchmarks/single_node/agentic/dsv41flash_fp4_h100_vllm_mtp.sh |
754356e9a39a |
InferenceX db-dump/2026-09-28 (restored) |
h200-tp4-0921 |
H200 TP4 (run of 09-21) | bc2b235fc26a | benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh |
754356e9a39a |
InferenceX db-dump/2026-09-28 (restored) |
h200-tp8-0921 |
H200 TP8 (run of 09-21) | bc2b235fc26a | benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh |
754356e9a39a |
InferenceX db-dump/2026-09-28 (restored) |
Provenance verdict¶
Every arm is PARTIAL. The deployment matches field for field (TP, EP, max-num-seqs, capture size, max-num-batched-tokens, gpu-memory-utilization, max-model-len, block size, CUDA graph mode, Engram CPU offload, speculative settings), and the corpus, scenario, seed and window are the run's. These differences remain:
| setting | measured run | BLIS | verdict | note |
|---|---|---|---|---|
| vLLM image | vllm/vllm-openai:nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3 |
kernel engine behaviour v0.31.0 (blis-latency-kernel new.go:2517) | MISMATCH (not removable) | server log: vLLM 0.29.1rc1.dev128+gcd10ed6f9 |
| aiperf harness | aiperf profile --scenario inferencex-agentx-mvp |
aiperf release 0.12.0 transcription | MISMATCH (not removable) | fork differs from release in agentic_replay.py, weka_trace.py, branch_orchestrator.py, replay_dependencies.py; metric formulas unchanged |
| trace idle-gap cap | --trace-idle-gap-cap-seconds 300 |
unset (schema refuses) | MISMATCH (not removable) | inactive at c1 (0 'Per-trace idle cap advanced' lines), active at c>=2 |
| cache-pressure warmup | --warmup-requests-per-lane 10 |
synthesized only | MISMATCH (not removable) | |
| compilation mode | NONE |
no field (kernel prices CUDA-graph replay) | UNSTATED | torch.compile disabled in the run (mode NONE); graphs FULL_AND_PIECEWISE |
| frontend/runner env | Rust frontend launched (server log) |
not modelled | UNSTATED |
Population metrics at concurrency ≥ 2 are confounded by the idle-gap cap: it raises the offered load the run saw beyond what BLIS replays. The comparison that survives every mismatch is the request-level pairing: the same request on both sides, with the same number of requests in flight.
What would make it EXACT: a run on the vLLM release whose behaviour the kernel prices (v0.31.0), or kernel pricing for the nightly that ran; the aiperf release BLIS transcribes (0.12.0), or a transcription of the fork; blis-schemas Replay support for the per-trace idle-gap cap and per-lane warm-up requests, or a run without them; and the UNSTATED rows stated by the run.
Output metrics: ITL on paired requests¶
Median per-request error over pairs. Solo: nothing else in flight on either side. Same in-flight: in-flight counts equal ±1. Levels with fewer than 30 solo pairs are omitted.
| arm | c | solo pairs | ITL solo | same-in-flight pairs | ITL same in-flight | request latency solo |
|---|---|---|---|---|---|---|
| H100 TP8 (run of 09-18) | 1 | 235 | -13.6% | 254 | -13.6% | -17.1% |
| H100 TP8 (run of 09-18) | 2 | 123 | -30.3% | 138 | -30.6% | -32.0% |
| H100 TP8 (run of 09-18) | 4 | 207 | -12.9% | 282 | -12.7% | -17.9% |
| H100 TP8 (run of 09-18) | 8 | 94 | -12.4% | 151 | -14.1% | -18.9% |
| H200 TP4 (run of 09-21) | 1 | 233 | -10.1% | 257 | -10.0% | -12.7% |
| H200 TP4 (run of 09-21) | 2 | 174 | -10.6% | 196 | -10.5% | -13.4% |
| H200 TP4 (run of 09-21) | 4 | 214 | -9.4% | 287 | -9.4% | -13.6% |
| H200 TP4 (run of 09-21) | 8 | 96 | -9.1% | 151 | -9.6% | -13.5% |
| H200 TP8 (run of 09-21) | 1 | 239 | -19.2% | 256 | -19.1% | -21.1% |
| H200 TP8 (run of 09-21) | 2 | 173 | -19.2% | 192 | -19.1% | -20.9% |
| H200 TP8 (run of 09-21) | 4 | 250 | -18.8% | 308 | -18.8% | -22.2% |
| H200 TP8 (run of 09-21) | 8 | 103 | -18.9% | 147 | -20.4% | -24.4% |
| H200 TP8 (run of 09-21) | 16 | 31 | -18.6% | 49 | -19.3% | -21.6% |
ITL error by requests in flight (load-matched pairs)¶
Pairs pooled over levels from the replay with gaps over 300 s compressed (a sensitivity approximation of the run's idle cap), binned by in-flight count.
| arm | in flight | pairs | ITL median error |
|---|---|---|---|
| H100 TP8 (run of 09-18) | 0-0 | 866 | -13.8% |
| H100 TP8 (run of 09-18) | 1-1 | 559 | -16.5% |
| H100 TP8 (run of 09-18) | 2-3 | 460 | -17.8% |
| H100 TP8 (run of 09-18) | 4-7 | 444 | -18.6% |
| H100 TP8 (run of 09-18) | 8-15 | 497 | -20.9% |
| H100 TP8 (run of 09-18) | 16-31 | 148 | -25.5% |
| H200 TP4 (run of 09-21) | 0-0 | 887 | -9.5% |
| H200 TP4 (run of 09-21) | 1-1 | 504 | -12.1% |
| H200 TP4 (run of 09-21) | 2-3 | 389 | -16.8% |
| H200 TP4 (run of 09-21) | 4-7 | 401 | -24.0% |
| H200 TP4 (run of 09-21) | 8-15 | 369 | -23.2% |
| H200 TP4 (run of 09-21) | 16-31 | 91 | -29.7% |
| H200 TP4 (run of 09-21) | 32-63 | 51 | -42.3% |
| H200 TP4 (run of 09-21) | 64-inf | 7 | -18.1% |
| H200 TP8 (run of 09-21) | 0-0 | 980 | -18.7% |
| H200 TP8 (run of 09-21) | 1-1 | 614 | -23.8% |
| H200 TP8 (run of 09-21) | 2-3 | 541 | -27.6% |
| H200 TP8 (run of 09-21) | 4-7 | 400 | -30.5% |
| H200 TP8 (run of 09-21) | 8-15 | 228 | -37.1% |
| H200 TP8 (run of 09-21) | 16-31 | 13 | -49.9% |
| H200 TP8 (run of 09-21) | 32-63 | 171 | -40.1% |
| H200 TP8 (run of 09-21) | 64-inf | 12 | -31.4% |
BLIS's marginal ITL cost of one more in-flight request, divided by the measured one (least-squares slopes over the same pairs; 1.00 is exact):
| arm | slope ratio BLIS / measured |
|---|---|
| H100 TP8 (run of 09-18) | 0.55 |
| H200 TP4 (run of 09-21) | 0.69 |
| H200 TP8 (run of 09-21) | 0.66 |
Output metrics: population, as replayed¶
Per-level population metrics over the whole replay window. MAPE and signed mean are over all levels; shape error divides each side by its lowest-concurrency value and excludes that level. Only the c1 column is free of the offered-load confound.
| arm | metric | signed c1 | signed mean | MAPE % | shape error % | sign |
|---|---|---|---|---|---|---|
| H100 TP8 (run of 09-18) | output tok/s | +6.7% | +17.9% | 60.3 | 66.4 | mixed (3 high / 5 low) |
| H100 TP8 (run of 09-18) | interactivity p90 | +16.2% | +451.5% | 451.5 | 428.4 | BLIS high at every level |
| H100 TP8 (run of 09-18) | interactivity p50 | +16.1% | +255.1% | 255.1 | 235.4 | BLIS high at every level |
| H100 TP8 (run of 09-18) | ITL p50 | -13.9% | -47.7% | 47.7 | 45.0 | BLIS low at every level |
| H100 TP8 (run of 09-18) | ITL p90 | -13.9% | -54.8% | 54.8 | 54.3 | BLIS low at every level |
| H200 TP4 (run of 09-21) | output tok/s | +4.7% | +65.5% | 98.0 | 108.2 | mixed (3 high / 5 low) |
| H200 TP4 (run of 09-21) | interactivity p90 | +9.7% | +119.6% | 119.6 | 114.6 | BLIS high at every level |
| H200 TP4 (run of 09-21) | interactivity p50 | +11.6% | +110.6% | 110.6 | 101.5 | BLIS high at every level |
| H200 TP4 (run of 09-21) | ITL p50 | -10.4% | -37.4% | 37.4 | 34.5 | BLIS low at every level |
| H200 TP4 (run of 09-21) | ITL p90 | -8.8% | -40.8% | 40.8 | 40.0 | BLIS low at every level |
| H200 TP8 (run of 09-21) | output tok/s | +9.7% | +24.5% | 65.3 | 73.1 | mixed (2 high / 6 low) |
| H200 TP8 (run of 09-21) | interactivity p90 | +25.8% | +145.0% | 145.0 | 108.4 | BLIS high at every level |
| H200 TP8 (run of 09-21) | interactivity p50 | +24.0% | +164.1% | 164.1 | 129.1 | BLIS high at every level |
| H200 TP8 (run of 09-21) | ITL p50 | -19.4% | -48.5% | 48.5 | 41.3 | BLIS low at every level |
| H200 TP8 (run of 09-21) | ITL p90 | -20.5% | -49.2% | 49.2 | 41.3 | BLIS low at every level |
Output tok/s under the idle-cap approximation¶
Signed output tok/s error per level when corpus gaps over 300 s are compressed. This is a sensitivity run, not a like-for-like comparison.
| arm | c | output tok/s |
|---|---|---|
| H100 TP8 (run of 09-18) | 1 | +6.8% |
| H100 TP8 (run of 09-18) | 2 | +19.1% |
| H100 TP8 (run of 09-18) | 4 | +7.4% |
| H100 TP8 (run of 09-18) | 8 | -3.6% |
| H100 TP8 (run of 09-18) | 16 | -7.4% |
| H100 TP8 (run of 09-18) | 20 | -1.0% |
| H100 TP8 (run of 09-18) | 24 | +19.2% |
| H100 TP8 (run of 09-18) | 28 | +390.1% |
| H200 TP4 (run of 09-21) | 1 | +5.2% |
| H200 TP4 (run of 09-21) | 2 | +6.3% |
| H200 TP4 (run of 09-21) | 4 | +9.0% |
| H200 TP4 (run of 09-21) | 8 | +1.1% |
| H200 TP4 (run of 09-21) | 16 | +1.9% |
| H200 TP4 (run of 09-21) | 32 | +19.3% |
| H200 TP4 (run of 09-21) | 64 | +41.3% |
| H200 TP4 (run of 09-21) | 128 | +18.6% |
| H200 TP8 (run of 09-21) | 1 | +8.9% |
| H200 TP8 (run of 09-21) | 2 | +0.8% |
| H200 TP8 (run of 09-21) | 4 | +8.2% |
| H200 TP8 (run of 09-21) | 8 | +1.7% |
| H200 TP8 (run of 09-21) | 16 | +0.3% |
| H200 TP8 (run of 09-21) | 32 | +18.3% |
| H200 TP8 (run of 09-21) | 64 | +20.1% |
| H200 TP8 (run of 09-21) | 128 | +114.8% |
TTFT¶
Paired solo TTFT (median), and the cold full prefill: the c1 warm-up primer, with an empty prefix cache and nothing else running.
| arm | c | solo pairs | TTFT solo |
|---|---|---|---|
| H100 TP8 (run of 09-18) | 1 | 236 | -40.7% |
| H100 TP8 (run of 09-18) | 2 | 123 | -44.8% |
| H100 TP8 (run of 09-18) | 4 | 207 | -34.8% |
| H100 TP8 (run of 09-18) | 8 | 94 | -34.7% |
| H200 TP4 (run of 09-21) | 1 | 234 | -31.9% |
| H200 TP4 (run of 09-21) | 2 | 174 | -31.6% |
| H200 TP4 (run of 09-21) | 4 | 214 | -27.6% |
| H200 TP4 (run of 09-21) | 8 | 96 | -24.1% |
| H200 TP8 (run of 09-21) | 1 | 240 | -36.5% |
| H200 TP8 (run of 09-21) | 2 | 173 | -35.2% |
| H200 TP8 (run of 09-21) | 4 | 250 | -36.9% |
| H200 TP8 (run of 09-21) | 8 | 103 | -36.8% |
| H200 TP8 (run of 09-21) | 16 | 31 | -33.3% |
Cold full prefill (the c1 primer):
| arm | TTFT |
|---|---|
| H100 TP8 (run of 09-18) | -10.2% |
| H200 TP4 (run of 09-21) | -21.2% |
| H200 TP8 (run of 09-21) | -33.6% |
Population TTFT, as replayed:
| arm | metric | signed c1 | signed mean | MAPE % | shape error % |
|---|---|---|---|---|---|
| H100 TP8 (run of 09-18) | TTFT p50 | -41.0% | -46.6% | 46.6 | 22.2 |
| H100 TP8 (run of 09-18) | TTFT p90 | -39.8% | -61.5% | 61.5 | 42.8 |
| H100 TP8 (run of 09-18) | TTFT mean | -30.3% | -53.5% | 53.5 | 41.8 |
| H200 TP4 (run of 09-21) | TTFT p50 | -33.2% | -45.3% | 45.3 | 34.4 |
| H200 TP4 (run of 09-21) | TTFT p90 | -37.5% | -55.0% | 55.0 | 33.7 |
| H200 TP4 (run of 09-21) | TTFT mean | -29.2% | -43.2% | 43.2 | 37.2 |
| H200 TP8 (run of 09-21) | TTFT p50 | -35.0% | -44.1% | 44.1 | 27.8 |
| H200 TP8 (run of 09-21) | TTFT p90 | -40.7% | -51.9% | 51.9 | 22.7 |
| H200 TP8 (run of 09-21) | TTFT mean | -31.0% | -38.6% | 38.6 | 29.8 |
KV pool¶
BLIS's KV-cache pool in tokens divided by the pool vLLM logged at start-up for the same deployment (1.00 is exact).
| arm | c1 | top level | top level ratio |
|---|---|---|---|
| H100 TP8 (run of 09-18) | 1.30 | c28 | 1.78 |
| H200 TP4 (run of 09-21) | 1.22 | c128 | 1.51 |
| H200 TP8 (run of 09-21) | 1.38 | c128 | 1.79 |