Known divergences¶

The places where the kernel knowingly prices something differently from the engine it models, or leaves part of the engine's behavior out. Each is a note in the source, beside the code it concerns, and this page is generated from those notes, so it lists exactly what the code admits to.

Each entry gives a short summary, then the note itself, which records the evidence and cites the vLLM v0.31.0 lines it rests on. A note is marked in the source by one of the phrases KNOWN DIVERGENCE, KNOWN over-charge or COVERAGE LIMIT, and internal/docgen collects every one outside the tests.

Known divergence: The engine does something the kernel prices differently on purpose.
Known over-charge: A term the kernel knows is charged too high, kept because removing it makes the measured accuracy worse. The note records the measurement.
Coverage limit: Engine behavior the kernel does not model, with what it charges instead.

Window admission under pipeline parallelism¶

Coverage limit, in (*Kernel).maxInFlightTokens in caches.go.

A window cache's admission reservation is sized for one more batch in flight than the pipeline-parallel width, as vLLM's default model runner holds under asynchronous scheduling. Its older runner holds one fewer with pipeline parallelism on, and the kernel cannot see which runs, so such a deployment's SequenceAdmissionBytes is up to one token budget a window group high.

The note in the source

COVERAGE LIMIT: the V1 model runner holds pp batches under asynchronous scheduling with pipeline parallelism, one fewer than this; the kernel cannot see which runner the engine picks, and at pp 1 both hold two.

Draft passes not padded under data-parallel lockstep¶

Coverage limit, in (*Kernel).priceDraft in draft.go.

A data-parallel rank's pad tokens run the target model's forward but not the drafter's passes. vLLM pads the draft passes across the group too, to a count set by every rank's draft batch, which one rank's batch does not state, so a padded rank's draft work is under-stated by its pad.

The note in the source

COVERAGE LIMIT: the draft passes run at the requests' own draft rows, without the data-parallel lockstep pad (Batch.PadTokens). vLLM's drafters coordinate their own padded count across the data-parallel group (coordinate_batch_across_dp, e.g. vllm/v1/spec_decode/dflash.py:230-261 at v0.31.0), which depends on every rank's draft batch, and a batch states only this rank's pad for the target's forward.

An autoregressive drafter's own graphs¶

Coverage limit, in (*Kernel).draftHasOwnGraphs in draft.go.

A parallel-block drafter such as DSpark is charged under its own CUDA graphs and capture ceiling. An autoregressive drafter (MTP, EAGLE) is still charged under the target's graph mode and ceiling; whether it captures graphs of its own is not modelled.

The note in the source

COVERAGE LIMIT: an autoregressive drafter (MTP, EAGLE) is charged under the target's graph mode and ceiling, as before; its own capture under Model Runner V2 is not modelled.

KV cache layouts not sized¶

Coverage limit, in KVBytesPerToken in internal/price/memory.go.

The kernel does not size every KV cache layout vLLM has. It refuses the per-token-head quantized formats rather than size them. Padded pages and a cache that compresses several tokens into one stored state are sized only where the model graph states its caches; a graph that does not is sized as one full-length cache per layer.

The note in the source

The layouts that state their own size are applied by the kernel where a deployment names them, or where its backend repacks the cache (the packed ds_mla caches; see Kernel.lift and sparseMLABackend in the root package). COVERAGE LIMIT: per-token-head quantization adds two fp32 scales a token (vllm/model_executor/layers/attention/mla_attention.py:1580-1593), so the kernel refuses those cache dtypes rather than size them here; padded pages are sized by their own specs (vllm/v1/kv_cache_interface.py:527-536), and a compressed cache (a compress_ratio layer) holds fewer states than tokens. Neither is modelled HERE: a graph that states its caches (blis-schemas model.Cache) is sized from them in the root package (caches.go), and this law is what a layer that states none is sized by.

Expert dispatch priced at 16 bits¶

Coverage limit, in ActivationBytes in internal/price/plan.go.

The tokens sent to the experts are priced at 16 bits per element. On a quantized deployment some expert kernels receive them at 8 or 4 bits, and the kernel cannot tell which kernel serves a layer, so an fp8 deployment whose experts take quantized input has its dispatch charged about twice its bytes.

The note in the source

The MoE combine is at this width too: the experts' output is bf16. The MoE DISPATCH is priced at it as well, and on a quantized deployment that is an upper bound, not the engine's width. The default allgather_reducescatter backend (vllm/config/parallel.py:202) quantizes the hidden state before gathering it -- moe_kernel_quantize_input in _quantize_and_setup_dispatch, vllm/model_executor/layers/fused_moe/prepare_finalize/ naive_dp_ep.py:16-51 -- and gathers per-token or per-block scales beside it (a static scalar scale is not gathered, :46-50), unless the experts kernel quantizes its own input (expects_unquantized_inputs, modular_kernel.py:505-512). Which experts kernel serves a layer is chosen per quantization method and platform, and this kernel does not model that choice, so it cannot tell a 1-byte (fp8) or half-byte (fp4) dispatch from a 2-byte one. COVERAGE LIMIT: an fp8 w8a8 deployment whose experts kernel takes quantized input has its dispatch charged about twice its bytes; its combine is charged right.

Every expert priced as gated¶

Coverage limit, in planNodes in internal/price/plan.go.

Each expert is priced as three matrices, a gated MLP. An expert with no gate, such as Nemotron-H's relu2 experts, holds two, and blis-schemas has no field saying which an expert is, so such a model's expert weights and arithmetic are over-stated by half.

The note in the source

COVERAGE LIMIT: every expert is priced as gated. A non-gated expert (relu2, as Nemotron-H's are) holds two matrices, and blis-schemas v0.3.0 has no field that says which an expert is, so such a model's experts are over-stated by half.

A young window ring's trailing page¶

Known over-charge, in (*Kernel).SequenceBoundedBytesAt in kernel.go.

Before a request has computed a window's worth of tokens, its window ring is charged one page more than vLLM holds, as the window is sized as if it might start part-way through a page. At most one page a ring, and none once the request is past its window.

The note in the source

KNOWN over-charge: a ring that has not yet outgrown its window starts on a page edge, so its trailing page is charged though vLLM holds none, at most one page a ring.

Mamba-2 groups widened past the tensor-parallel width¶

Coverage limit, in (*Kernel).recurrentRankBytes in kernel.go.

A Mamba-2 layer's recurrent state is divided by the tensor-parallel width. Where that width exceeds the layer's group count, vLLM widens the groups so every head's shard has one, which holds a little more state than this charges.

The note in the source

COVERAGE LIMIT: where tp exceeds a Mamba-2 layer's group count, vLLM widens the groups to cover every head's shard (extra_groups_for_head_shards, mamba_utils.py:206-210), which this division leaves out.

Hybrid attention under decode-context parallelism¶

Coverage limit, in (*Kernel).SequenceVariableBytes in kernel.go.

For a model that mixes sliding-window and full-attention layers, decode-context parallelism does not shard the KV cache the kernel reports, which overstates it. vLLM refuses such a deployment under its default KV cache manager, so this matters only with that manager turned off.

The note in the source

COVERAGE LIMIT, STATED RATHER THAN PAPERED OVER. kvBytesPerToken is ONE whole-model scalar, summed over the inferred layers of every attention kind, and this function has no per-kind context to spend. So the shard applies only when every KV-holding layer is a kind DCP shards. A hybrid stack -- gpt-oss-120b alternates a 128-token swa layer with a full gqa one -- is left UNSHARDED, which overstates what it needs. That is the conservative direction, and under the engine's default hybrid KV-cache manager it costs nothing real: the engine refuses the deployment, asserting dcp == 1 with "DCP not support sliding window" (kv_cache_interface.py:868-872). With the hybrid manager disabled -- explicitly, or by a KV connector that does not support it (vllm/config/vllm.py) -- the windowed layers are promoted to full attention and sharded, which this overstates. Expressing the hybrid case exactly would mean splitting kvBytesPerToken per kind -- a larger refactor, for configurations that either do not start or run only with the hybrid manager off.

Expert tokens under prefill-context parallelism¶

Coverage limit, in (*Kernel).moeDPFunnel in kernel.go.

The tokens reaching a rank's experts are multiplied by the data-parallel width, because vLLM gathers every data-parallel rank's tokens to them, but not by the prefill-context width. How the experts gather a mix of split prefill tokens and replicated decode tokens is not modeled.

The note in the source

COVERAGE LIMIT: prefill-context ranks are not counted here. A PCP rank's prefill tokens are its own share while its decode rows are replicated across the group, and how the MoE gathers that mixture is not modelled.

P/D handoffs between pools of different widths¶

Coverage limit, in (*Kernel).pdTransferBytes in kernel.go.

Each decode rank's pull is priced as one rank's transfer at one rank's link rate. That is exact when the prefill and decode pools have the same tensor-parallel width. Where the decode pool is wider, several decode ranks pull from one prefill rank, whose link then carries several payloads, and a placement that names a node rather than a rank cannot express that.

The note in the source

COVERAGE LIMIT: each pull is priced as one rank's demand at one rank's link rate. That is exact when the pools' tensor-parallel widths match. Where the decode pool is wider, several decode ranks pull from one prefill rank (tp_mapping.py:97), whose link then carries decode_tp / prefill_tp payloads, which a placement that names a node and not a rank cannot express.

The engine workspace is all outside the memory budget¶

Known divergence, in (*Kernel).PreSnapshotBytes in kernel.go.

The registry's engine workspace is one lumped allowance for the CUDA context, cuBLAS and allocator slack, and the kernel reports all of it outside the gpu_memory_utilization budget, with the communicator. vLLM charges the part that arises after its memory snapshot, library workspaces made at the first forward, against the KV cache, which the kernel leaves out.

The note in the source

KNOWN DIVERGENCE: the workspace allowance is one lumped figure, and the part of it that arises after the snapshot -- cuBLAS and other library workspaces made at the first forward -- is charged against the KV cache by vLLM and is left out here with the rest; the registry states no split.

Expert weights under prefill-context parallelism¶

Coverage limit, in (*Kernel).lift in new.go.

With expert parallelism off and prefill-context parallelism on, each rank is charged for a 1/(dp × tp) slice of each expert, where vLLM gives it 1/(dp × pcp × tp). Dividing by pcp as well would leave out a reduction the kernel does not price, and make the experts cheaper than the engine runs them.

The note in the source

COVERAGE LIMIT: the flattening runs over pcp as well, and the shard here does not. With pcp > 1 vLLM gives a rank a 1/(dp x pcp x tp) slice of each expert, and this charges 1/(dp x tp). The token gather over those ranks is priced where it runs -- with dp > 1 the dispatch spans dp x pcp x tp ranks (resolve.Layout.MoEGroupWidth); with dp = 1 there is no dispatch (all2all_utils.py:202-214) -- but the reduction a tensor-parallel MoE runs over the wider slice is not, so dividing by pcp alone would make a PCP step's experts cheaper than the engine runs them.

DeepSeek-V4.1 load-time memory under expert parallelism¶

Coverage limit, in (*Kernel).computeFixedBytes in new.go.

vLLM holds about 9 GiB a rank more than the kernel charges for DeepSeek-V4.1-Flash with expert parallelism, the same at TP 8 and TP 4, so it is not sharded weights, which agree. Its source is not yet localized, and capacity under expert parallelism is overstated by that much a rank until it is.

The note in the source

COVERAGE LIMIT: on DeepSeek-V4.1-Flash with expert parallelism, vLLM v0.31.0's "Model loading took" figure exceeds what this charges by about 9 GiB a rank, at TP 8 and TP 4 alike. our calibration runs' H200 sizing gives 45.81 and 81.49 GiB, so a fit of R + S/tp has R = 10.1 GiB replicated and S = 285.4 GiB sharded, against this kernel's 1.0 and 286.1. The sharded weights agree, and the gap does not scale with TP; about 2 GiB of it comes with DSpark (42.64 GiB without it). The parameter and buffer paths read at v0.31.0 -- routed experts, Marlin's MXFP4 repack and its round-up, the shared expert, the drafter's MoE, the embedding and LM head, Engram's host tables, the init-time buffers and the rotary caches -- do not account for it. Until a per-module memory dump on the part localizes it, capacity under expert parallelism is overstated by that much a rank (TestInternalH200ArmsAreTheEnginesFigures).

DeepSeek-V4.1's prefill workspace beside the activation scratch¶

Known over-charge, in (*Kernel).engineBufferBytes in new.go.

DeepSeek-V4.1's prefill gather workspace and its indexer's logits stand-in are charged in full beside the registry's activation scratch. vLLM holds the workspace in one shared scratch buffer that its MoE kernels also grow, and the stand-in is a transient at the profile peak, so the engine reserves the larger of each pair where the kernel charges their sum.

The note in the source

KNOWN over-charge: the workspace manager's buffer is shared scratch, and the MoE modular kernels grow the same one (workspace.py:49-56), so the engine holds the larger of the two where this adds the attention's request to an activation multiple that already allows for MoE intermediates; and the indexer's logits stand-in is a transient that this adds to the activation multiple's own transients rather than to their peak.

Choosing the sparse attention backend¶

Coverage limit, in sparseMLABackend in sparse_mla.go.

For DeepSeek sparse attention, which backend vLLM chooses depends on facts the model graph does not state: whether the checkpoint quantizes its KV cache, the installed FlashInfer version, and one head dimension. The kernel assumes the common case for each, and the catalog has no chip of the SM120 generation, whose backend differs.

The note in the source

COVERAGE LIMITS: an "auto" cache is taken as 16-bit, which it is unless the checkpoint states a KV quantization -- a ModelOpt checkpoint's kv_cache_quant_algo turns "auto" into fp8_e4m3 or nvfp4 before any backend is chosen (MODELOPT_TO_VLLM_KV_CACHE_DTYPE_MAP and resolve_kv_cache_dtype_string, vllm/utils/torch_utils.py:67-70, :448-521, called from vllm/engine/arg_utils.py:2199-2202), and the graph does not record it; FlashInfer's SM90 backend needs FlashInfer 0.6.18 or later (flashinfer_mla_sparse_sm90.py:194-199), which this assumes is installed; FlashInfer (SM100) checks the model's qk_nope_head_dim, which the DSA family satisfies (128 and 192, within its [128, 192]) and the graph does not state; and the SM120 parts' backend, which repacks "auto" too, has no part in the catalog.

Routed experts under tensor parallelism¶

Known over-charge, in (*Kernel).priceLayers in step.go.

Under tensor parallelism each rank is charged the arithmetic of whole routed experts, where vLLM gives it a tp-th slice of each. Correcting this makes the measured accuracy worse, because the over-charge offsets a cost not yet identified, so it is kept until that cost is found.

The note in the source

NOT divided by expertTensorShards, and that is a KNOWN over-charge rather than an oversight. Under pure tensor parallelism localExpertShare is 1 -- every rank does hold every expert -- and vLLM shards the intermediate dimension, `intermediate_size_per_partition = intermediate_size // tp_size` (vllm/model_executor/layers/fused_moe/config.py:1321-1323), so a rank's routed work is its slice's. Charging the whole expert over-prices routed compute by the tensor-parallel width on the 400 of 591 InferenceX scenarios that are pure TP. The BYTES do divide by it (routedWeightBytes, below), so the two terms disagree about the same experts.

17 notes: 1 known divergence, 3 known over-charges, 13 coverage limits.