This summarises the BLIS target design for engineers who know llm-d and vLLM and are about to work in one of its repositories; the design is the authority, and each claim below links to the section that states it.
BLIS simulates an LLM serving cluster. It runs the router of llm-d unchanged, and Go ports of the vLLM and SGLang engines. A cost model built from published hardware facts and fitted coefficients prices every step (§1, §7.1). The design rests on five ideas:
| Repository | Owns | Changes when |
|---|---|---|
| blis-substrate | Contracts and validators, the catalog, the coefficient registry, the cost model, the invariant registry | a new model or chip, a data drop, an accuracy improvement, a new contract |
| blis-vllm, blis-sglang | The upstream's scheduler, KV-cache manager, connectors, defaults and lowering, in Go, for one upstream release at a time; its oracle, field list and surface list | an upstream release; a fidelity fix |
| llm-d-router fork | llm-d's router code made runnable in simulated time, and nothing that changes a routing decision | an llm-d release; a plugin a user needs |
| inference-sim | The event loop, cluster, workload, replay, engine selection, the Environment, link sharing, the command line, the Result | a pin bump; a simulator feature |
| blis-config-search | Experiments that search configurations, and their reports | a new experiment; a simulator release |
| accuracy-dashboard | Public accuracy and coverage, and the scoring harness | a simulator release; a new InferenceX dump |
To place a fact, ask what it depends on. The engine release decides it: the engine repository. The llm-d release: the router fork. The simulator's design: inference-sim. Otherwise it is true of the hardware, the model, the physics or the data, and the substrate owns it. A fact such as H200 memory bandwidth lives in the catalog; how fast one attention kernel runs on H200 lives in the coefficient registry; a user's scorer weights live in the scenario, copied verbatim (§2.2). Documents that cross a repository boundary, the scenario, the deployment, the Result and the per-point scoring record, each have one definition in the substrate; the applications validate them through inference-sim and define none of their own (§2.2, §5).
The substrate is one Go module, so a change that touches a model, its data, its coefficients and the cost model is one pull request, one review and one release (§3.1). It has two kinds of package. Those under contracts/ hold the interfaces, data types, validators, conformance suites, engine harness and invariant registry, import only the Go standard library, the YAML parser and each other, and change only compatibly. The rest hold the cost model, the validators, the loader and the embedded data, and only inference-sim imports them (§3.2). A refit therefore needs no engine or router release: Go builds a program with the newest substrate version any module requires, and the contracts have not changed incompatibly. The reverse is guarded too. An engine requires the oldest substrate whose contracts it uses, and never one newer than inference-sim requires, so an engine release cannot change the simulator's data (§3.2).
Data reaches a run in one of two ways, and each run records which. A path (--catalog, --registry) replaces the embedded copy, so a recalibration candidate is scored with the released simulator and no rebuild. With no path, the run uses the data compiled into the release. The loader refuses anything that would make the data ambiguous, and a path load and an embedded load give the same data and the same content hash (§3.3).
Only an engine repository knows its upstream, so only it works out defaults, backend choices, padding and graph-capture sizes. It hands the cost model three objects in physical units (§6.1):
PoolPlan, fixed for the life of one pool: hardware and fabric, parallel layout, the lowered layers with the kernel implementation of each, the KV groups and the KV budget, the tiers below GPU memory, the graph policy, the draft policy, host work, resident weights, buffers, and how many steps may be in flight (§6.2).StepShape, one step across the data-parallel ranks that run in lockstep, already padded as the upstream pads it, with a mode (mixed, extend-only, decode-only, verify, prebuilt or idle) (§6.2).engine.Transfer, one KV move, which the cost model turns into bytes per link (§6.2).PoolPlan (1). At each step the scheduler forms a StepShape (2) and the cost model returns a time (3). The engine reports the step done with its metrics and KV events and inference-sim advances simulated time (4), then delivers the metrics and events to the router (5). (§2.3)An engine cannot size its KV cache without the cost model's memory arithmetic, and the cost model cannot price a step without the engine's plan. The loop is broken by an Environment, supplied by inference-sim when it builds the engine. Construction has two phases. In the plan phase the engine resolves its configuration, lowers the model into a partial plan with an empty KV budget, asks the Environment for the memory the weights, activations and buffers take, and then applies its upstream's budget rule. In the run phase the engine forms a step and asks for its price (§6.3).
Five rules follow. The engine pads, and the cost model prices the padded shape. The engine names the exact kernel implementation, and a coefficient is keyed by a hash of the kernel's source, so a library version bump that leaves a kernel unchanged needs no substrate change. The engine decides which KV blocks move between tiers and the substrate prices each move. Sharing a link over time is the simulator's job, the one resource arithmetic it owns. With pipeline parallelism the engine emits one shape per stage and combines the stage times (§6.2). The one resource arithmetic the simulator keeps is stated as an exception to X3. The contract differs between engines in step modes, scheduling ahead of the running step, page size, graph capture and event emission, and each difference has a place in the contract (§6.4).
An engine port covers four parts of its upstream: scheduling; KV-cache management; KV transfer and tiering; resolution and lowering (§7.1). "Faithful" has a precise meaning. The upstream's own scheduler and the Go port receive the same inputs and make the same decisions at every step. To run the upstream without GPUs, a harness replaces only its environment (GPU, clock, transport, platform, outside services), and every replacement is listed in a manifest. A recorded trace changes only in a commit marked as reviewed (§7.2, §7.7).
The same tiers apply to every engine repository (§7.4):
| Tier | What is compared |
|---|---|
| T0, T1 | Resident weights and priced against launched work; configuration fields accepted and rejected, defaults resolved, inertness proofs |
| T2, T3 | Scheduler and KV-cache decisions; the engine loop, parallelism and launched-shape padding |
| T4, T5 | KV connectors and P/D protocols; tiering, eviction and promotion |
| T6, T7 | Metrics and KV events byte for byte with what llm-d reads; a mutation sweep that every deliberate break of a ported function is caught |
The tiers test only ported code. The surface list shows what is not ported. The generator reads the upstream's own registration tables, so a new feature appears without anyone remembering to add it. Code that the upstream does not register, such as an out-of-tree connector loaded by path, is not on the list, and a point whose run loads such code is not scored (§11.4). Each item has one status. Modelled is the goal. Refused means the engine refuses it by name until it is modelled, with an issue, and the backlog counts what it blocks. Out of scope is allowed for three reasons only: the item runs on hardware BLIS does not simulate, it does no work the simulator prices, or it is deprecated upstream. "Hard to port" is a priority, not a reason (§7.5). A configuration field has the parallel statuses modelled, proven inert with a passing proof, or not modelled with an issue (§4). Shared frameworks are ported as a unit before the features built on them (§7.6). The router fork changes upstream files only in a fixed list of ways, none of which alters a routing decision, and a classifier checks each changed hunk against that list (§5).
| DEVS term | In BLIS |
|---|---|
| component | an engine instance, the router, a link, a client |
| time advance | the time of an engine's next scheduled wake-up |
| internal / external / confluent transition | a step ends / a request arrives / both at the same instant |
| output function | the tokens, metrics and KV events an engine sends |
| experimental frame | what a run feeds in, when it measures, and what it measures |
(§8)
Components interact only through timestamped messages, and none reads another's state. A link is the one component whose state changes at a constant rate between events: its state is the transfers in progress and the bytes each has left, and every start or end of a transfer recomputes the rate of the others (§8.1). An instant is processed in three phases. Output functions run; each component receives its bag of messages sorted by (time, class, sender, the sender's own send count), with no global counter; each component makes exactly one transition. Each engine declares, for each kind of external input, the step boundary at which it takes effect; under vLLM's asynchronous scheduling, for example, a request arriving during one step usually first joins the step after next (§8.2).
Time is an integer number of microseconds, with one rounding rule. Only finitely many events may happen in finite time, so every zero-delay cycle contains a component with a positive time advance or a declared bound (§8.3). Every random draw comes from a substream named by component, purpose and entity, seeded from the scenario seed. This gives common random numbers: two runs that differ only in configuration give the same requests the same inputs. It does not remove noise, because some draws depend on how the system responds, so comparisons still need replications (§8.4). Input distributions name their family, parameters and source trace, and pass a goodness-of-fit test against held trace data (§8.5).
The experimental frame is one document recorded in every Result: the generator and seeds, the initial state, the phases, the stop rule, the acceptance conditions and the metric definitions. A run that fails its acceptance conditions produces no figure. Run, export and replay under the same configuration give an identical Result (§8.6). Every metric has one definition, with a formula, statistic, population, window and unit, in the metric registry, and is compared with another only at the same scope and aggregate. A request's fate (completed, dropped with a reason, timed out, in flight or queued at the horizon, or not started) is written once, every latency sample derives from it, and the generator's own count of arrivals must equal the fates. A latency statistic states how it treats requests unfinished at the horizon (§8.7).
Outputs are estimates. A terminating run, such as a benchmark of a set number of prompts, is estimated from the mean of replications, starting at five, with a confidence interval. A steady-state run discards a warm-up found by a stated method and estimates from replications or batch means; a percentile gets its interval from replications only. A backlog that grows means overload, and latency figures from such a run are not reported. A comparison of two configurations is better, worse or not distinguished, and a search that compares many configurations controls the chance of a false winner and re-estimates the winner on fresh seeds. The interval covers workload randomness; the coefficients' own uncertainty is separate and a figure says which it covers (§8.8). Every Result also states whether its scenario lies inside the model's validity domain, the range of models, chips, layouts and loads its held-out data cover (§8.9).
The cost model uses, for each lookup, the entry of the smallest rank: measured, calibrated, transferred, literature, vendor_spec, roofline, assumed. Some rank has a value for every primitive family and every catalog chip, so a missing measurement never causes a refusal, and only the trust in the price varies (§9.1, §9.4). Each method's typical error is measured by leaving one chip out. A fitted entry records the range it covered, and a lookup outside it ranks with transferred (§9.2, §9.3). A change to a measured value passes a landing gate: criteria committed first, then held-out error not worse on any lane and better on the target lane (§9.4).
The ledger assigns each unit one role, computed from a secret seed per group of near-copies, so near-copies cannot fit and judge. A role is computed, not stored, and a new unit's role is drawn once, before any job uses it. Only the fit, anchor and report lists are published, because InferenceX data are public and a list of held units would let a change be tuned against them. Results on select, validate and held units exist only as aggregates over at least ten units, from jobs that hold the seed and never run a pull request's code. A candidate job, which does run the pull request's code, has no secret and writes one prediction per unit. A unit from a partition made before the ledger existed keeps its old side, and is reported as not blind (§10.2, §10.4).
Scoring a point has four steps. State: copy the run's recorded configuration verbatim. Estimate: apply the engine's own rule for a value the run did not record. Decide: read each field's status from the engine's field list. Price: use the most trusted entry (§11.2). The dashboard never classifies a setting itself (§11.1). Grades run from G1, where the run's engine release is the one scoring and every condition holds, through G2 with named substitutions, G3 and G4 for another engine release, to G5 for a point not scored for a listed reason (§11.3, §11.4). One simulator release scores every point, so a substrate improvement reaches every point without a backport (§11.5). The scoring harness is a script released from dashboard tags and run on a runner that builds the simulator. Anyone may run it on any branch; what changes is which units it may read. A figure computed on a branch is labelled with its build and its data, and is experimental unless both are released, so it can be posted on a pull request but never published as BLIS's accuracy (§11.7).
Each outside event has one playbook with the same fields: detection, a preflight that computes scope, steps, checks, a definition of done, and a size. A new model on existing capabilities is one substrate pull request (§12.1). A data drop pins a new revision and refits only the families whose fit data changed (§12.2). A new chip adds its datasheet facts to the catalog, with roofline values as the floor (§12.8). An accuracy idea carries a pre-registration before any held-out data is read, one change per pull request (§12.3). An upstream release gets a surface diff, a field demand and a source diff, and only changed functions are re-ported (§12.4): the old release keeps a release branch, the oracle's pin moves, every new surface item and field is classified, each changed trace is reviewed, and the engine is tagged on the new release (§12.4, §14.4). An llm-d release is merged into the fork, never rebased, and every new clock, goroutine, random draw or map-order use on a decision path is routed through the runtime interface (§12.5, §5). A new engine is a new repository that adds to the contracts (§12.6).
Deterministic checks decide what can be decided mechanically. Agents do bounded work from a preflight. People answer the questions that need judgement, approve and set direction (§13). A rule is kept as an automated check, a maintainer practice or agent guidance, and the design says which (§4.2). The invariant registry holds every rule with its checks; it generates the design's appendix, each repository's AGENTS.md and the pull-request checklist (§13.1). Whole-run accuracy gates run the pinned harness (§13.5). The release bot merges, tags and re-pins bottom-up (§13.6).
Versions are written in one place and documents say "the pinned release" (§4.1). An engine module models exactly one upstream release, tagged v<upstream>-blis.N; a higher N on the same release is a BLIS-side change, and a plain upstream tag is never made. Each simulator release runs one upstream release of each engine and of llm-d and lists them in stack.json, and a scenario naming another is refused with the newest release that does run it (§14.1, §14.2). When BLIS moves to a new upstream release, the old one keeps a release branch for fixes (§14.4). Releases are cut bottom-up, and a manual tag by a person is an annotated tag on the checked commit (§13.6, §14.2).
Each repository's card in the design gives its purpose, what it owns and must never hold, its interfaces, what triggers a change, how its correctness is shown, and its own invariants (§5). The points a contributor meets first:
pkg/embed.New (§5).engine: {kind, version}, and unsupported or ambiguous input is refused with a message that names the field, the value and the cause, never degraded silently. A deployment the upstream would not start is refused with the upstream's own condition, so a search never ranks a layout that cannot run (§4). A fixed set of scenarios, the byte gate, must stay byte-identical across a refactor (§5).Some rules govern every change. Repositories are released bottom-up, and a change that spans repositories is one review set. Pins are exact release tags. A skipped test in a required job is a failure. Every review finding ends fixed, filed as an issue, or rejected with a reason before merge, and an issue cited as the reason for a refusal stays open until the change that removes the last citation (§4.1). Agents work the items a preflight lists, in a branch, and iterate until the checks pass. They may not approve a merge, cut a release, weaken or skip a check, or change a success criterion after reading held-out data (§13.7).
For the full text and the check behind each rule, see the invariant index in the design's appendix, and the repository card in (§5) for the repository you are changing.