Skip to content

Validation results

BLIS binaries these results describe

  • (devel)+9e3c509c8788fe0743605a8f06e0bf5c1c39ac1f, binary sha256 29fb09054a10ee51dc104f491cb09337e0ba527055b85653d281f23230451dff

How close BLIS's predictions are to measured serving runs, anchor by anchor. Each anchor is one measured deployment and workload that BLIS replays as it ran.

No anchor is EXACT

Every comparison below has at least one configuration or workload difference that this build cannot remove (a different vLLM version, benchmark-client fork, or unrecorded setting). The errors are reported per anchor with those differences listed; none is used to fit coefficients. An anchor becomes EXACT when its configuration and workload diff is empty and no provenance row is MISMATCH or UNVERIFIABLE: each page says what that would take.

Verdicts

  • EXACT: the diff between the run and the BLIS replay is empty, and every setting is stated by the run.
  • PARTIAL: scored, but with differences listed per anchor; some are removable (the run did not record a setting), some are not (BLIS cannot express it).
  • NOT_REPRODUCIBLE: the run cannot be replayed as it ran (no engine log, model not in the catalog, or similar); not scored.

Error definitions

  • Signed error per point: 100 * (BLIS / measured - 1). Its sign is the bias direction; the sign row says whether it holds at every level.
  • MAPE: mean absolute signed error over points.
  • Shape error: each side divided by its own lowest-load value, then the same error, excluding the lowest level. It removes a constant level offset and keeps how the metric moves with load.
  • KV-pool ratio: BLIS's KV-cache pool divided by the pool vLLM logged at start-up.

Output metrics (output tok/s, ITL/TPOT, interactivity) come first on every page, TTFT second.

Build: 9e3c509c.

Summary

anchor verdict output-metric error TTFT error KV pool BLIS/vLLM
InferenceX agentic, H100 TP8 (run of 09-18) PARTIAL -13.6% (ITL, solo c1) -40.7% (solo c1) 1.30
InferenceX agentic, H200 TP4 (run of 09-21) PARTIAL -10.1% (ITL, solo c1) -31.9% (solo c1) 1.22
InferenceX agentic, H200 TP8 (run of 09-21) PARTIAL -19.2% (ITL, solo c1) -36.5% (solo c1) 1.38
InferenceX single-turn, all families PARTIAL (0 EXACT) +1.2% (TPOT, signed mean) -6.3% (signed mean) not logged
internal llm-d DSV4.1 H200, Agg-1r PARTIAL -47.7% (ITL mean) -62.7% (p50) not logged
internal llm-d DSV4.1 H200, Agg-2r PARTIAL -45.0% (ITL mean) -65.6% (p50) not logged
internal llm-d DSV4.1 H200, TP4-baseline PARTIAL -39.0% (ITL mean) -60.5% (p50) 1.57
internal llm-d DSV4.1 H200, TP8-no-DSpark PARTIAL -42.7% (ITL mean) -59.2% (p50) 1.29
internal llm-d DSV4.1 H200, TP8-no-Engram PARTIAL -46.4% (ITL mean) -61.1% (p50) 1.38
internal llm-d DSV4.1 H200, TP8-no-KV-offload PARTIAL -37.8% (ITL mean) +4.8% (p50) 1.38
llm-d Prism A1 PARTIAL +30.5% (TPOT, signed mean) -13.6% (mean, signed mean) not logged
llm-d Prism A2 PARTIAL +27.7% (TPOT, signed mean) +331.8% (mean, signed mean) not logged
llm-d Prism A2' PARTIAL +26.0% (TPOT, signed mean) -27.1% (mean, signed mean) not logged
llm-d Prism B (round-robin) PARTIAL +65.1% (TPOT, signed mean) -26.1% (mean, signed mean) not logged

Details:

Data policy

These pages publish derived error metrics, verdicts and mismatch lists only. No measured value from InferenceX, llm-d Prism or the internal runs is copied here; sources are cited by URL and commit. The measured inputs stay outside this repository, pinned by sha256:

input sha256
results (9e3c509c) 95c2f79d2eebe585c639139fa7000cadb2b0f4ad92fcebdf8cdbde00772948f9
llm-d Prism results (9e3c509c) 4270cd786405dfb4f577ce1af38d6758eddcc45192f4ef914be7a52be1484d3f

Regenerate with tools/validation/render_docs.py (its docstring gives the arguments); --check fails when a page is out of date.