Vocabularies¶

Every enumerated key in a document accepts only the values listed here; an unrecognized value is an error. The kernel's own enumerations are listed at the end. The lists are generated from the Go constants, so they are the values the code accepts.

These vocabularies are fixed by the schema and do not change with an engine release. Values that do, such as backend names and cache dtypes, are listed with the rules pack for each release.

Coefficients¶

Units¶

The unit of a coefficient's value.

dimensionless · us_per_layer · us_per_request · us_per_step · us_per_hop · bytes_per_us · bytes_per_rank · us_per_load · us_per_transfer · tokens · sm_count · us_per_token

Methods¶

How a coefficient's value was obtained. This is the field that keeps a measurement distinguishable from an estimate, so it is never inferred.

measured · literature · vendor_spec · copied · assumed · not_charged · calibrated

measured: From a measurement. The entry should cite it; the schema does not require that.
literature: From a published result that was not independently reproduced.
vendor_spec: A datasheet figure: nominal, not achieved.
copied: Carried over from another scope, which copied_from names.
assumed: A placeholder, or an estimate reached by reasoning. A careful estimate is still assumed; the reasoning goes in rationale.
not_charged: Deliberately zero, so that "no cost" can be told apart from "cost unknown".
calibrated: Back-solved through the cost model from end-to-end measured runs, and checked against held-out ones. The entry cites its runs as sources of kind measured_case and names the held-out metrics in validated. It is not derived, which is a catalog provenance: a figure computed from other catalog facts.

measured, literature, vendor_spec and calibrated count as evidenced. That is a statement about the basis of a number, not its quality: a vendor figure is evidenced but nominal.

Scope keys¶

The dimensions along which a coefficient's validity is bounded. A coefficient's scope uses these as its keys.

hardware · tp · ep · nodes_spanned · model

Source kinds¶

discussion · publication · datasheet · model · vendor_doc · measured_case

Source roles¶

primary · supporting · upper_bound

Catalog facts¶

Provenances¶

Where a chip's or fabric's figures come from. vendor_spec is a datasheet figure; derived is one computed from datasheet figures by a step the datasheet does not state.

vendor_spec · derived

Model graphs¶

Primitives¶

A node's op.

GEMM · GroupedGEMM · Attention · RecurrentUpdate · Elementwise · Gather · AllReduce · AllGather · ReduceScatter · All2All

Dtypes¶

A graph's weight_dtype, and a node's weight_dtype and state_dtype. Only formats that change a byte count or a peak rate appear. nvfp4, mxfp4 and int4 are three different 4-bit formats, not spellings of one.

bf16 · fp16 · fp8 · nvfp4 · mxfp4 · int8 · int4 · fp32

Attention kinds¶

An Attention node's kind. mla and sparse_mla store one latent vector per token, which is why their n_kv is 1. sparse_mla is attention by a sparse kernel over an explicit set of positions, bounded by a top-k (index_topk), a compressed stream (compress_ratio), a sliding window (window), or a combination: DeepSeek-V4.1's window-only layers run the same sparse FlashMLA kernel as its compressed ones.

gqa · mla · sparse_mla · swa

Recurrent kinds¶

A RecurrentUpdate node's recurrent_kind.

mamba2 · kda · gdn

Emit conditions¶

A node's emit. "", the empty value, means the node always exists, and is what an omitted emit reads as.

"" · tensor_parallel · expert_parallel · tensor_parallel_unless_sp_moe · sequence_parallel · tensor_parallel_unless_sequence_parallel

tensor_parallel: Present when the tensor-parallel width exceeds 1.
expert_parallel: Present when the expert-parallel width exceeds 1.
tensor_parallel_unless_sp_moe: Present when the tensor-parallel width exceeds 1 and the engine has not made the MoE input sequence-parallel. Under sequence-parallel MoE the reduction is replaced by a reduce-scatter and all-gather pair.
sequence_parallel: Present when the model's own sequence parallelism is on, as the rules pack decides it for global.architecture. DeepSeek-V4.1 then shards the residual stream by token and all-gathers each attention input, then reduce-scatters its output.
tensor_parallel_unless_sequence_parallel: Present when the tensor-parallel width exceeds 1 and the model's own sequence parallelism is off: the all-reduces sequence parallelism replaces.

The last two belong on collectives. Work runs in every layout, including tensor parallelism one, where neither condition holds, so a node stated twice, once under each, would vanish there. Work is stated once, with under_sequence_parallel for how the mode divides it. A non-collective node carrying either condition draws a warning.

A model that decides its own sequence parallelism never consults the engine-wide rule tensor_parallel_unless_sp_moe names, so its decoder-layer reductions, the MoE's included, use tensor_parallel_unless_sequence_parallel. DeepSeek-V4.1 passes its own flag to its MoE, and vLLM's FusedMoE then skips the final tensor-parallel all-reduce. The 0.31.0 rules pack reports the engine-wide condition on such a model as an error.

The condition holds on DeepSeek-V4's and V4.1's mega-MoE backend too. That backend requires expert parallelism, so it runs without the model's sequence parallelism only at tensor parallelism one or pipeline parallelism above one. There its routed experts combine over the expert-parallel group with no tensor-parallel all-reduce, and its shared expert, which fuses into the kernel only under sequence parallelism or tensor parallelism one, runs tensor-parallel and all-reduces its own output. So the layer still reduces one hidden-width tensor across tensor-parallel ranks, as on the default backend.

Modalities¶

A graph's modality. An omitted modality means text_only.

text_only · text_decoder_of_multimodal

Tensor-parallel layouts¶

A node's tp. An omitted tp leaves the layout to the cost model's inference. A GEMM accepts column, row and replicated; an Elementwise or Attention node accepts sharded and replicated. No other primitive accepts one.

"" · column · row · sharded · replicated

column: The output width n splits across ranks.
row: The input width k splits across ranks. The partial sums are reduced by a collective the graph states separately.
sharded: The work splits across ranks with the heads or columns it operates on.
replicated: Every rank holds the whole weight and does the whole node for every token it holds.

Sequence-parallel splits¶

A node's under_sequence_parallel: how it divides when the model's own sequence parallelism is on. An omitted value means the node runs as tp states, over every token the step schedules. It needs global.architecture, and a collective cannot carry it.

"" · token_sharded

token_sharded: The node runs on its rank's shard of the step's tokens, ceil(tokens / tp) rows, because vLLM's sp_shard pads the count to a multiple of tp. A GEMM, Elementwise or Attention node then holds its whole weight, whatever tp states: a column or row split needs every rank to hold the same rows. A GroupedGEMM's experts stay divided as expert parallelism decides, and a Gather's table as its shard states.

tp alone can't say this. Under DeepSeek-V4.1's sequence parallelism the residual stream is sharded by token, and only attention runs on the gathered stream, between its all-gather and its reduce-scatter. mHC, before the gather, and the fused q/kv A-projection, after it, are both replicated, yet the first sees 1/tp of the tokens and the second all of them. Engram's wkv is column-split without the mode and replicated with it, and runs at tensor parallelism one too. It is one node, tp: column with under_sequence_parallel: token_sharded.

Cache kinds¶

A cache's kind, which decides how it grows.

kv · index · window · state

kv: Keys and values, or a latent: one entry per tokens_per_entry context tokens.
index: A sparse-attention indexer's key cache. It grows like kv.
window: A sliding-window ring, bounded by window tokens per request.
state: Fixed per-request state, entries entries for the request's lifetime.

Gather residency¶

A Gather node's residency, and the engine setting that can change it, its residency_control. An omitted residency means device; an omitted residency_control means no setting moves the table.

device · host

"" · engram_cpu_offload

engram_cpu_offload: vLLM's Engram cpu_offload. A deployment sets it with engine.engram_config.cpu_offload. The graph states the residency the engine gives the table when the deployment leaves it alone, which is a release fact: under 0.31.0 it is host, and that pack's engram-residency-default rule checks it.

Gather shards¶

A Gather node's shard. An omitted shard means replicated.

replicated · rows · heads

rows: The rows split evenly across ranks, and each token's row is read on the rank that holds it. vLLM's vocabulary embedding works this way.
heads: The table is rows_per_token sub-tables, one per hash head, split across S shards. Each shard holds ceil(rows_per_token / S) of them and reads one row from each for every token. S is the tensor-parallel width, or that times the node's data-parallel replicas when they split the table rather than share one host copy, which the deployment's engram_config decides. DeepSeek-V4.1's Engram works this way.

Context work¶

An Elementwise node's context.per, context.phase and context.decode_extent.

query_token · request

query_token: Once per scheduled token, each over the context.
request: Once per request per step, over its context, however many of its tokens are scheduled.

"" · prefill

An omitted phase means every step; prefill means only steps that schedule the request's prompt tokens.

"" · max_model_len

An omitted decode_extent means the request's context; max_model_len means the engine's max_model_len, for a kernel launched over a buffer padded to it.

Drafting modes¶

A speculator's drafting. An omitted drafting means autoregressive.

"" · parallel_block

parallel_block: The draft stack runs once per step over num_spec_tokens positions per request, and each position predicts one draft token. DeepSeek-V4.1's DSpark drafts this way.

Deployments¶

Pool roles¶

colocated · prefill · decode

Frontends¶

A pool's engine.frontend, the API server that renders the chat template and tokenizes the prompt before the request reaches the engine core. python, the default, is the Python OpenAI server. rust is the vllm-rs frontend, which vLLM launches only when VLLM_USE_RUST_FRONTEND=1 (vllm/envs.py and vllm/entrypoints/cli/serve.py; the default is 0 in v0.29.0 and v0.31.0). It tokenizes through fastokens.

python · rust

Data-parallel load balancing¶

A pool's parallel.data_parallel_load_balancing. internal, the default, serves one endpoint and the engine balances requests across every rank. hybrid (--data-parallel-hybrid-lb) serves one endpoint per node, balancing across that node's ranks. external (--data-parallel-external-lb, or --data-parallel-multi-port-external-lb, which launches one such server per local rank) serves one endpoint per rank, and the router chooses the rank. Pack.ResolveDPLoadBalancing returns the mode that runs: hybrid with one rank per node runs external, and with every rank on one node runs internal.

internal · hybrid · external

Traces¶

Trace modes¶

A trace header's mode: how the trace was produced.

real · generated · replayed

Replay scenarios¶

A replay's scenario.

inferencex-agentx-mvp

Corpus formats¶

A replay corpus's format.

weka-trace-v7

Corpus encodings¶

A replay corpus's encoding.

jsonl

Cache-bust modes¶

A replay's cache_bust.

first_turn_prefix

Warmup modes¶

A replay's warmup.mode. synthesized sends, for every stream mid-flight at its lane's snapshot, its last earlier request with one output token, and starts profiling when all have returned. cache_pressure runs an accelerated warmup for a stated duration.

synthesized · cache_pressure

Time units¶

A trace header's time_unit. Several members name the same unit, because the tools that write traces spell it differently.

ns · nanoseconds · us · microseconds · ms · milliseconds · s · seconds

Acceptance¶

Thinking modes¶

A profile's and a workload's thinking: whether the target model ran with its reasoning mode on.

on · off

Measured runs¶

The vocabularies of a measured run and its licence registry.

Verdicts¶

EXACT · PARTIAL · NOT_REPRODUCIBLE

Resolutions¶

STRICT · RESOLVED

Measurement source kinds¶

inferencex · internal_llmd · llmd_benchmark · prism · vllm_bench · pytorch_hud · mlperf · aiconfigurator

Input roles¶

report · manifest · env · router · raw_artifact · server_log · tokenizer · dataset

Extraction methods¶

html_js_literal · html_table · yaml · json · csv · log_text · materialized

Ramp strategies¶

linear · exponential

Metric formulas¶

interactivity is 1000 over an inter-token-latency statistic, in tokens per second per user. output_throughput, total_throughput and request_throughput are rates over a window, with a denominator rather than a statistic.

ttft · tpot_excl_first · itl · e2el · output_throughput · total_throughput · request_throughput · interactivity · user_tps

Statistics¶

A percentile is written pNN or pNN.N, strictly between 0 and 100, with no leading or trailing zeros: p50, p99, p99.9.

mean · median · std · max

Populations¶

per_request · pooled_chunks · pooled_tokens

Denominators¶

main_phase_wall · profiling_window · per_gpu · per_replica

Clocks¶

client · server

Start events¶

send_after_semaphore · arrival

Stop events¶

first_content_chunk · last_chunk

Output length sources¶

usage · tokenizer · target

Percentile methods¶

numpy_linear

Metric units¶

ms · tokens_per_s · requests_per_s

Provenance row verdicts¶

MATCH · MISMATCH · UNSTATED · UNVERIFIABLE

Materiality¶

yes · no · unknown · bounded

Difference kinds¶

not_expressible · not_modelled · conflict · unresolved · approximated

Licence statuses¶

verified · unverified · refused

Permitted uses¶

score · derive_coefficients · redistribute

Router kinds¶

direct · epp

Kernel¶

Resources¶

What a StepEstimate attributes cost to.

sm · hbm · nvlink · nic · pcie · copy_engine · host

Directions¶

A tier transfer's direction.

to_tier · from_tier

Source: vocab, and the spec/ and kernel packages.