Model graph

A model, written as the GPU work one forward pass launches. Each distinct layer is a small directed acyclic graph of cost primitives. A stack says how the layers repeat. The catalog derives one graph per model from the vendor's configuration and commits it as models/<name>/graph.yaml.

Graphs are generated by the catalog's scripts/derive_graph.py, never written by hand. Why the schema takes this form, rather than following a vendor's configuration format, is explained in Why a model is a graph.

models/qwen3-14b/graph.yaml, abbreviated

kind: ModelGraph
name: qwen3-14b
derived_from:
  format: hf_config_json
  path: config.json
  sha256: e73c3664ca09b10a673fef0c22e8a6b456201d49bd4713c9691f775720e8857a
  deriver_version: 1
global:
  hidden_size: 5120
  vocab_size: 151936
  tie_word_embeddings: false
  weight_dtype: bf16
layer_kinds:
- id: dense
  nodes:
  - {op: Elementwise, role: input_norm}
  - {op: GEMM, role: qkv_proj, n: 7168, k: 5120}
  - {op: Attention, kind: gqa, n_q: 40, n_kv: 8, d_h: 128}
  - {op: GEMM, role: o_proj, n: 5120, k: 5120}
  - {op: AllReduce, role: attn_out, emit: tensor_parallel}
  # ... the MLP: a norm, two GEMMs and a second AllReduce
  edges: [[0, 1], [1, 2], [2, 3], [3, 4], ...]
stack:
  pattern: [dense]
  repeat: 40
head:
- {op: Elementwise, role: final_norm}
- {op: GEMM, role: lm_head, n: 151936, k: 5120}

Keys

Key	Type	Required	Meaning
kind	string	yes	Always ModelGraph.
name	string	yes	Must equal the model's directory name in the catalog.
derived_from	mapping	yes	The vendor file the graph was derived from, below.
global	mapping	yes	Whole-model shapes, below.
layer_kinds	list	yes	The distinct layers, at least one.
embedding	list of nodes	no	Nodes before the stack: the vocabulary lookup, as a Gather. Omitted means a vocab_size x hidden_size table at global.weight_dtype.
stack	mapping	yes	The order the layers run in.
head	list of nodes	no	Nodes after the stack, such as the final norm and the LM head.
speculator	mapping	no	A draft module, as its own stack.
arch_overrides	list	no	Facts about the served model that differ by GPU architecture, below.
base_capability_families	list of integers	with arch_overrides	The compute-capability major versions the base graph describes as served. A family listed neither here nor in an override is refused by ForCapability.
modality	string	no	What the graph covers. Omitted means text_only. text_decoder_of_multimodal marks the decoder of a model whose encoder the graph does not price. See modalities.

derived_from

Key	Type	Required	Meaning
format	string	yes	The source format, such as hf_config_json.
path	string	yes	The source file, relative to the model directory.
sha256	string	yes	Digest of the source file, 64 characters. A change to the source without re-deriving is then detectable.
deriver_version	integer	yes	Version of the deriver that wrote the graph, at least 1.

global

Key	Type	Required	Meaning
architecture	string	no	The model class the engine serves the checkpoint with, the configuration's architectures[0], such as DeepseekV41ForCausalLM. Rules packs key model-specific engine behavior on it. Required by the sequence_parallel emit conditions and by under_sequence_parallel.
hidden_size	integer	yes	Model width, positive.
vocab_size	integer	yes	Vocabulary size, positive.
tie_word_embeddings	boolean	no	Whether the LM head shares the embedding weights.
weight_dtype	string	yes	The format parameters are stored in. One of the dtypes.
weight_scale	mapping	no	The resident scale layout of weights stored at weight_dtype, below. It does not apply to a node stored in another format.
context_window	integer	no	The longest context the engine admits for this model: the max_model_len vLLM derives from the configuration. An unstated or auto max_model_len starts from it.
default_context_window	integer	no	The window an unstated or auto max_model_len starts from, where it is shorter than context_window, as for a LongRoPE model. Needs context_window.

layer_kinds

Key	Type	Required	Meaning
id	string	yes	Unique within the graph. The stack refers to it.
nodes	list of nodes	yes	The primitives in this layer, at least one.
edges	list of pairs	no	[from, to] indices into nodes. The graph must be acyclic.

stack

Two committed stacks. A stack is a literal prologue, a pattern repeated some number of times, and a literal epilogue (neither model has one). The total is len(prologue) + len(pattern) × repeat + len(epilogue).

Key	Type	Required	Meaning
prologue	list of strings	no	Layer kinds that run once, first.
pattern	list of strings	no	Layer kinds that repeat.
repeat	integer	if pattern is set	How many times pattern runs.
epilogue	list of strings	no	Layer kinds that run once, last.

A uniform model is a one-entry pattern with a repeat. A model with no period puts its whole layer sequence in prologue and leaves repeat at zero. The stack must expand to at least one layer.

speculator

Key	Type	Required	Meaning
method	string	yes	The engine's name for the speculative method. The rules pack checks it.
num_spec_tokens	integer	yes	Draft tokens per request per step, at least 1. Not the draft layer count.
drafting	string	no	How stack runs. Omitted means once per drafted token; parallel_block means once per step over num_spec_tokens positions per request. See drafting modes.
embedding	list of nodes	no	The draft's own token lookup and its reduction, before each pass of stack, over the same rows. A lookup into the target's table aliases embedding.<role>. Outside every layer, like the top-level embedding.
stack	mapping	yes	The draft module's own stack over the graph's layer kinds. Needs a pattern and a repeat of at least 1.
context_stack	mapping	no	Draft work that runs once per step over the target's scheduled tokens, prefill tokens included.
head	list of nodes	no	Nodes after each pass of stack, over the same rows.
sequential_head	list of nodes	no	Nodes run once per draft position, in order, over one row per request. Only with parallel_block.

The draft module is a stack and not a token count because, for an MoE target, the draft pass is itself an MoE pass with its own cost. DeepSeek-V4.1's DSpark shows why the other keys exist. It drafts 5 tokens through 3 layers in one pass over 5 positions per request. Before that pass it looks up its positions' tokens in the target's table and all-reduces the result, before it shards its stream for sequence parallelism. In a layer kind, that all-reduce would read as a decoder-layer reduction, which sequence parallelism replaces. It also projects the target's hidden states over every token the target scheduled. After it, it runs a Markov head once per position, one position at a time.

Nodes

A node is one primitive with the shape parameters that primitive prices. There are ten primitives:

GEMM · GroupedGEMM · Attention · RecurrentUpdate · Elementwise · Gather · AllReduce · AllGather · ReduceScatter · All2All

The four collectives (AllReduce, AllGather, ReduceScatter, All2All) exist or not depending on the deployment's layout, so each must carry an emit condition. The tables below group a node's keys by the primitive that prices them. Some keys are rejected on a primitive that does not price them, such as a head count or an expert count on a GEMM. That check covers the likely mistakes, not every key.

Common to every node:

Key	Type	Required	Meaning
op	string	yes	The primitive.
role	string	no	A label for readers, such as qkv_proj.
emit	string	for collectives	When the node exists. One of the emit conditions; omitted means always.
tp	string	no	How the node divides across tensor-parallel ranks, on a GEMM, Elementwise or Attention node. One of the layouts; omitted leaves it to the cost model's inference.
under_sequence_parallel	string	no	How the node divides when the model's own sequence parallelism is on. One of the splits; omitted means as tp states, over every token the step schedules. Not on a collective. Needs global.architecture.
aliases	string	no	On a GEMM or Gather: the node whose weights this one reads, as <layer kind>.<role>, head.<role> or embedding.<role>. The node is priced for its compute and weight reads, at its target's format, but adds no resident bytes. It states no weight_dtype or weight_scale of its own.

GEMM and GroupedGEMM:

Key	Type	Required	Meaning
n	integer	yes	Output dimension.
k	integer	yes	Input dimension.
weight_dtype	string	no	Overrides global.weight_dtype for this node's weights.
weight_scale	mapping	no	The resident scale layout of this node's weights, below.

GroupedGEMM only, for MoE experts:

Key	Type	Required	Meaning
experts	integer	yes	Routed experts.
top_k	integer	yes	Experts each token is routed to. At most experts.
shared_experts	integer	no	Experts every token passes through as well.
shared_intermediate_size	integer	no	The shared experts' inner dimension, where it differs from n. Needs shared_experts.
latent_size	integer	no	A narrower input dimension for the experts, where a projection precedes them.

Attention:

Key	Type	Required	Meaning
kind	string	yes	One of the attention kinds.
n_q	integer	yes	Query heads.
n_kv	integer	yes	KV heads, at most n_q. Exactly 1 for mla and sparse_mla, which store one latent vector per token.
d_h	integer	yes	Head dimension.
kv_lora_rank	integer	no	MLA latent rank.
qk_rope_head_dim	integer	no	MLA rotary head dimension.
window	integer	for swa	Sliding window in tokens. On sparse_mla, one of its three bounds.
index_topk	integer	for sparse_mla, unless compress_ratio or window is set	Cache entries a sparse layer selects.
compress_ratio	integer	for sparse_mla, unless index_topk or window is set	Factor by which a sparse layer compresses its latent read.
caches	list	no	The per-request caches the node reads and which its layer owns, below. Omitted means one cache inferred from the read shape.

RecurrentUpdate, for linear-attention and state-space layers:

Key	Type	Required	Meaning
recurrent_kind	string	yes	One of the recurrent kinds.
n_heads	integer	yes	Heads.
state_size	integer	yes	State dimension.
n_groups	integer	for mamba2	Groups.
conv_kernel	integer	for mamba2	Convolution kernel width.
intermediate_size	integer	for mamba2	Inner dimension.
state_dtype	string	no	Format the state is stored in, often wider than the weights.

Elementwise, and the payload of a collective:

Key	Type	Required	Meaning
bytes_per_token	integer	no	Bytes touched per token, where hidden_size does not determine it. On an AllReduce, AllGather or ReduceScatter, the bytes per token of the collective's full tensor, where it is not one hidden-width activation. Not accepted on an All2All, whose volume follows from the routing.
per_tokens	integer	no	Run once every this many tokens rather than once per token, as a store done once per compressed entry is.
context	mapping	no	Traffic that scales with the attended context, below.
writes_cache	string	no	On an Elementwise node in a layer kind: a cache of the same layer this node writes one entry of each time it runs. The entry's record adds to bytes_per_token, and per_tokens must be the cache's tokens_per_entry.

Gather, for a table of which each token reads a few rows:

Key	Type	Required	Meaning
rows	integer	yes	Rows in the table.
row_bytes	integer	yes	Bytes of one stored row, scales included.
rows_per_token	integer	yes	Rows read per token. At most rows.
out_bytes_per_token	integer	yes	Bytes written to device memory per token, after any dequantize.
residency	string	no	Where the table lives by default: device (the default) or host. See residency.
residency_control	string	no	The engine setting that can move the table, which a deployment states. residency is then that setting's default for the release, which the rules pack checks.
shard	string	no	How the table divides across tensor-parallel ranks. One of the shards; omitted means replicated.

A matrix multiply charges its whole weight as reads and compute every step. For a gather that is wrong by the ratio of the table to the rows read. DeepSeek-V4.1 reads 24 rows of 264 bytes per token from each 384-million-row Engram table, and by default keeps the tables in host memory.

Caches

An attention node may list the caches it reads. Readers find their writer by name: a cache a layer does not own is the same-named cache of the nearest earlier layer, in the same stack, that owns one. That is DeepSeek-V4.1's rule for its compressed cache. Only owned caches add memory, transfer volume and offload volume.

Key	Type	Required	Meaning
name	string	yes	Unique within the layer. Pairs a reader with its writer.
kind	string	yes	One of the cache kinds.
owned	boolean	yes	Whether this layer allocates the cache. Required, because an omitted false would turn an owner into a reader and drop its memory.
bytes_per_entry	integer	yes	One stored entry, scales and in-record padding included, under the cache dtype the graph's base description assumes. Which dtype runs is the cost model's to resolve. On a cache the layer reads, the entry it reads, which must equal its owner's.
bytes_by_cache_dtype	mapping	no	The entry size under cache dtypes whose record differs, keyed by the engine's cache-dtype name; each positive. The rules pack checks the names.
tokens_per_entry	integer	no	Context tokens one entry covers. Omitted means 1. A kv or index cache must match the node's compress_ratio, or 1 when it has none.
window	integer	for window	Tokens a window cache retains. Equal to the node's window, except on a layer only a parallel_block draft runs, whose read is the window plus num_spec_tokens: a DSpark draft query reads its whole block, non-causally.
entries	integer	for state	Entries a state cache holds per request. A state cache is always owned.
block_size	integer	no	Page size in tokens, where the model fixes it rather than taking the engine's block_size. Owned caches only.
page_alignment	integer	no	Each page's bytes are rounded up to a multiple of this. Owned caches only.

A cache the layer reads but does not own states what it reads: its kind, its entry under every cache dtype, tokens per entry, window and entries. Each must equal its owner's, so a read is priced with the record the owner stores. How the owner pages the cache, block_size and page_alignment, is the owner's alone and is refused on a reader. Only nodes in a layer kind state caches.

An entry is the whole layer's: tensor parallelism does not divide it (a latent cache has one head), and where the engine shards a cache by decode-context parallelism each rank holds 1/dcp of the entries. A page of block_size tokens holds block_size / tokens_per_entry entries. A window cache is bounded only under vLLM's hybrid KV cache manager, which the engine turns off for a KV connector that does not support it; then it grows with context like a kv cache.

The record under a cache dtype is the one the backend that runs stores. The same name can mean different records on different backends: on DeepSeek-V4.1's packed layout fp8 is rewritten to the packed record, and on a plain-row backend it is an FP8 row. Which dtype runs is resolved by the rules pack (Pack.ResolveCacheDType). The 68-byte MXFP4 indexer record DeepSeek-V4.1 can opt into is selected by vLLM's indexer_kv_dtype, which no deployment key carries yet, so the graph states the default FP8 record (#57).

DeepSeek-V4.1-Flash is the case caches exist for. Only layers 2, 8, 14 and 20 own a compressed cache, and the ratio-2 ones hold one entry per two tokens. Every layer's sliding window is a bounded ring. Its KV is 1,790 bytes per context token, where one full-length 584-byte cache per layer would be 23,360.

Caches with different page sizes share one block pool, and every block costs the widest group's page bytes, so a narrow group pays padding in each of its blocks. The rules pack reproduces the engine's grouping: rules.OwnedKVCacheLayers lists a graph's owned caches in engine order, and Pack.KVCacheGroups returns the KV cache groups, each cache's group and the bytes per block (vLLM's _get_packed_kv_cache_groups and _get_kv_cache_bytes_per_block). A consumer sizes each group in whole blocks of that size rather than each cache at its own page. It refuses what it does not model: several cache types sharing one page size, and a state cache on 0.29.0, which stores DeepSeek-V4's compressor state as a sliding-window spec the graph does not state. A window cache's group holds Pack.WindowPagesPerRequest blocks per request, each costing BytesPerBlock; under 0.31.0, which splits a lone window bucket into several groups, that holds per group. Only the MLA specs pad a page to page_alignment; a state cache is a circular buffer the engine never pads, so an alignment on one is refused, and 0.31.0 sizes its ring from the deployment's K (KVCacheGroupsInput.NumSpecTokens): max(8, 1 << (K + 1).bit_length()) rows, so K = 7 needs 16 rows where the graph states 8.

Context work

Key	Type	Required	Meaning
reads_cache	string	no	A cache of the same layer whose entries the work reads. Each entry then costs that cache's record, on the architecture and cache dtype that run, plus bytes_per_entry, and covers that cache's tokens_per_entry.
bytes_per_entry	integer	yes	Traffic per context entry, per unit of per; beyond the read cache's record when reads_cache is set, where it may be zero.
tokens_per_entry	integer	no	Context tokens one entry covers. Omitted means 1. Not stated with reads_cache.
per	string	yes	query_token or request: what multiplies the context. See context work.
phase	string	no	prefill restricts the work to steps that schedule the request's prompt.
decode_extent	string	no	max_model_len when a decode step's work covers the engine's max_model_len rather than the context.

Weight scales

Key	Type	Required	Meaning
elements	integer	yes	Weight elements sharing one scale.
bytes	integer	yes	Bytes of one scale.

A weight's resident size per element is its dtype's payload plus bytes / elements. NVFP4 (one FP8 scale per 16) and MXFP4 (one E8M0 scale per 32) carry their own layout, which applies when none is stated. Other formats are scaled per tensor, channel, group or block depending on the checkpoint, so their layout is never assumed. WeightBytesPerParam in spec/model applies these rules, so every consumer gets the same number.

arch_overrides

Key	Type	Required	Meaning
capability_family	integer	yes	The CUDA compute-capability major version it applies to: 9 for SM90, 10 for SM100 and SM103. Once per graph.
nodes	list	no	Weight formats restated by role, below.
caches	list	no	Cache records restated by name, below.

Key	Type	Required	Meaning
role	string	yes	Every GEMM or GroupedGEMM with this role. At least one must exist.
weight_dtype	string	no	The resident format on this architecture.
weight_scale	mapping	no	The resident scale layout on this architecture.

Key	Type	Required	Meaning
name	string	yes	Every cache with this name. At least one must exist.
bytes_per_entry	integer	no	The record on this architecture, for owners and readers alike.
page_alignment	integer	no	The page alignment on this architecture, for owners.
bytes_by_cache_dtype	mapping	no	Entries merged over each named cache's, owners and readers alike.

The base graph describes the architecture the catalog prices by default. Graph.ForCapability returns the graph with the matching override applied, none left, and that family as its only base, so consumers apply the overlay one way. It refuses a family the graph describes neither as its base nor by override, including the zero a chip with no compute capability yields, rather than pricing it as the base architecture: DeepSeek-V4.1's base is SM90, and SM120 runs a plain-row KV cache the base does not describe. An override restates what is resident. Kernel selection and launch fusion are not restated here, because they change the cost of nodes the graph already states, and they also depend on the layout and the build. DeepSeek-V4.1 needs an override because its wo_a is resident in BF16 on SM90 and FP8 on SM100, and its KV record is 584 bytes on SM90 and 528 on SM100.

docs/examples/model-graph-deepseek-v4.1.yaml, abridged by hand

# DeepSeek-V4.1-Flash, abridged to show the keys a V4.1 graph needs. The catalog's
# derived graph carries every node; this one keeps the nodes whose keys are new and drops
# norms, RoPE launches, mHC, the router and similar work, the draft's confidence head, and the SM100 attention path's different launches. Its collectives are
# shown in full on the first layer kind only. Each figure here is from the
# model's config.json or from vLLM v0.31.0's source, as the schema's doc comments cite.
kind: ModelGraph
name: deepseek-v4.1-flash
derived_from:
  format: hf_config_json
  path: config.json
  sha256: 8be45ce0476004a3f529fd896115a4a2e800a129ad2d3ec05b16050f52e21879
  deriver_version: 1
modality: text_decoder_of_multimodal
global:
  architecture: DeepseekV41ForCausalLM
  hidden_size: 5120
  vocab_size: 129280
  tie_word_embeddings: false
  weight_dtype: fp8
  # The checkpoint's 32 x 32 UE8M0 blocks, expanded to 1 x 32 at load.
  weight_scale: {elements: 32, bytes: 1}
  context_window: 1048576

# BF16, not FP8: the quant config has no method for an embedding.
# Each rank looks up the rows it holds and the partial results are all-reduced, before the
# residual stream is sharded for sequence parallelism.
embedding:
- {op: Gather, role: embed_tokens, rows: 129280, row_bytes: 10240, rows_per_token: 1,
   out_bytes_per_token: 10240, shard: rows}
- {op: AllReduce, role: embed_out, emit: tensor_parallel}

layer_kinds:
# Layers 0 and 1: sliding window only, compress ratio 0. Layer 1 also looks up Engram.
# Under V4.1's own sequence parallelism the residual stream is sharded by token, so each
# attention input is all-gathered and its output reduce-scattered rather than all-reduced.
- id: swa
  nodes:
  - {op: AllGather, role: attn_in, emit: sequence_parallel}
  - {op: GEMM, role: fused_wqa_wkv, n: 1792, k: 5120, tp: replicated}
  - {op: GEMM, role: wq_b, n: 32768, k: 1280, tp: column}
  - op: Attention
    kind: sparse_mla
    n_q: 64
    n_kv: 1
    d_h: 512
    qk_rope_head_dim: 64
    window: 128
    caches:
    - {name: swa, kind: window, owned: true, bytes_per_entry: 584, window: 128,
       block_size: 32, page_alignment: 576}
  # MXFP8 is a Blackwell-only BMM, so on SM90 wo_a is dequantized to BF16 at load; its
  # E8M0 scales stay resident.
  - {op: GEMM, role: wo_a, n: 8192, k: 4096, tp: column, weight_dtype: bf16,
     weight_scale: {elements: 32, bytes: 1}}
  - {op: GEMM, role: wo_b, n: 5120, k: 8192, tp: row}
  - {op: AllReduce, role: attn_out, emit: tensor_parallel_unless_sequence_parallel}
  - {op: ReduceScatter, role: attn_out, emit: sequence_parallel}
  # The MoE runs on the residual stream, so under the mode each rank routes only its own
  # token shard. The nodes above, between the all-gather and the reduce-scatter, see every
  # token, and state no split.
  - {op: GroupedGEMM, role: experts, n: 2304, k: 5120, experts: 384, top_k: 6,
     weight_dtype: mxfp4, under_sequence_parallel: token_sharded}
  - {op: All2All, role: moe_dispatch_combine, emit: expert_parallel}
  # The MoE's tensor-parallel reduction follows the model's own sequence parallelism, not
  # the engine-wide sequence-parallel-MoE rule, which V4.1 never consults: FusedMoE skips
  # it when the model makes the MoE sequence-parallel.
  - {op: AllReduce, role: mlp_out, emit: tensor_parallel_unless_sequence_parallel}
  edges: [[0, 1], [1, 2], [2, 3], [3, 4], [4, 5], [5, 6], [5, 7], [6, 8], [7, 8], [8, 9], [9, 10]]
- id: swa_engram
  nodes:
  - {op: Gather, role: engram_lookup, rows: 384006168, row_bytes: 264, rows_per_token: 24,
     out_bytes_per_token: 12288, residency: host, residency_control: engram_cpu_offload,
     shard: heads}
  # Without sequence parallelism each rank looks up its heads for every token and the
  # rows are all-gathered; with it the gather runs along tokens instead (not shown).
  - {op: AllGather, role: engram_rows, emit: tensor_parallel_unless_sequence_parallel,
     bytes_per_token: 12288}
  - {op: GEMM, role: fused_wqa_wkv, n: 1792, k: 5120, tp: replicated}
  - op: Attention
    kind: sparse_mla
    n_q: 64
    n_kv: 1
    d_h: 512
    qk_rope_head_dim: 64
    window: 128
    caches:
    - {name: swa, kind: window, owned: true, bytes_per_entry: 584, window: 128,
       block_size: 32, page_alignment: 576}
  edges: [[0, 1], [1, 2], [2, 3]]

# Layers 2, 8 and 14 own the ratio-2 compressed cache, the indexer key cache and the
# compressor's FP32 ring; layers 3-7, 9-13 and 15-19 read the nearest one's cache.
- id: csa2_source
  nodes: &csa2_source_nodes
  - {op: GEMM, role: fused_wqa_wkv, n: 1792, k: 5120, tp: replicated}
  - {op: GEMM, role: compressor_fused_wkv_wgate, n: 1024, k: 5120, tp: replicated,
     weight_dtype: bf16}
  # Launched per token, but only the last token of each ratio-2 group writes an entry of
  # the index cache.
  - {op: Elementwise, role: index_k_norm_rope_store, per_tokens: 2, writes_cache: index,
     tp: replicated}
  - {op: GEMM, role: index_weights_proj, n: 32, k: 5120, tp: replicated, weight_dtype: bf16}
  - op: Attention
    role: block_index_scores
    kind: sparse_mla
    n_q: 32
    n_kv: 1
    d_h: 128
    compress_ratio: 2
    tp: replicated
    caches:
    - {name: index, kind: index, owned: true, bytes_per_entry: 132, tokens_per_entry: 2,
       page_alignment: 576}
  - op: Attention
    kind: sparse_mla
    n_q: 64
    n_kv: 1
    d_h: 512
    qk_rope_head_dim: 64
    window: 128
    index_topk: 512
    compress_ratio: 2
    caches:
    - {name: swa, kind: window, owned: true, bytes_per_entry: 584, window: 128,
       block_size: 32, page_alignment: 576}
    - {name: compressed, kind: kv, owned: true, bytes_per_entry: 584, tokens_per_entry: 2,
       page_alignment: 576}
    - {name: compressor_ring, kind: state, owned: true, bytes_per_entry: 4096, entries: 8}
  # The prefill path dequantizes the whole compressed stream into a BF16 workspace once
  # per step: read each record, whatever size it is on the chip and cache dtype, and
  # write 512 BF16 values. Every compressed layer runs it, readers included; this
  # abridgement shows it once.
  - op: Elementwise
    role: prefill_dequant_gather
    context: {reads_cache: compressed, bytes_per_entry: 1024, per: request, phase: prefill}
  edges: [[0, 1], [1, 2], [2, 3], [3, 4], [4, 5], [5, 6]]
- id: csa2_source_engram
  nodes:
  - {op: Gather, role: engram_lookup, rows: 384016682, row_bytes: 264, rows_per_token: 24,
     out_bytes_per_token: 12288, residency: host, residency_control: engram_cpu_offload,
     shard: heads}
  - {op: AllGather, role: engram_rows, emit: tensor_parallel_unless_sequence_parallel,
     bytes_per_token: 12288}
  - op: Attention
    role: block_index_scores
    kind: sparse_mla
    n_q: 32
    n_kv: 1
    d_h: 128
    compress_ratio: 2
    tp: replicated
    caches:
    - {name: index, kind: index, owned: true, bytes_per_entry: 132, tokens_per_entry: 2,
       page_alignment: 576}
  - op: Attention
    kind: sparse_mla
    n_q: 64
    n_kv: 1
    d_h: 512
    qk_rope_head_dim: 64
    window: 128
    index_topk: 512
    compress_ratio: 2
    caches:
    - {name: swa, kind: window, owned: true, bytes_per_entry: 584, window: 128,
       block_size: 32, page_alignment: 576}
    - {name: compressed, kind: kv, owned: true, bytes_per_entry: 584, tokens_per_entry: 2,
       page_alignment: 576}
    - {name: compressor_ring, kind: state, owned: true, bytes_per_entry: 4096, entries: 8}
  edges: [[0, 1], [1, 2], [2, 3]]
- id: csa2_reader
  nodes:
  - op: Attention
    kind: sparse_mla
    n_q: 64
    n_kv: 1
    d_h: 512
    qk_rope_head_dim: 64
    window: 128
    index_topk: 512
    compress_ratio: 2
    caches:
    - {name: swa, kind: window, owned: true, bytes_per_entry: 584, window: 128,
       block_size: 32, page_alignment: 576}
    - {name: compressed, kind: kv, owned: false, bytes_per_entry: 584, tokens_per_entry: 2}

# Layer 20 owns the ratio-1 compressed and indexer caches the rest of the stack reads.
- id: csa1_source
  nodes:
  - op: Attention
    role: block_index_scores
    kind: sparse_mla
    n_q: 32
    n_kv: 1
    d_h: 128
    compress_ratio: 1
    tp: replicated
    caches:
    - {name: index, kind: index, owned: true, bytes_per_entry: 132, page_alignment: 576}
  - op: Attention
    kind: sparse_mla
    n_q: 64
    n_kv: 1
    d_h: 512
    qk_rope_head_dim: 64
    window: 128
    index_topk: 512
    compress_ratio: 1
    caches:
    - {name: swa, kind: window, owned: true, bytes_per_entry: 584, window: 128,
       block_size: 32, page_alignment: 576}
    - {name: compressed, kind: kv, owned: true, bytes_per_entry: 584, page_alignment: 576}
  edges: [[0, 1]]
- id: csa1_reader
  nodes:
  - op: Attention
    kind: sparse_mla
    n_q: 64
    n_kv: 1
    d_h: 512
    qk_rope_head_dim: 64
    window: 128
    index_topk: 512
    compress_ratio: 1
    caches:
    - {name: swa, kind: window, owned: true, bytes_per_entry: 584, window: 128,
       block_size: 32, page_alignment: 576}
    - {name: compressed, kind: kv, owned: false, bytes_per_entry: 584}
# Layers 24, 28, 32 and 36 score the context with their own queries over layer 20's
# indexer keys, and mask the FP32 scores to layer 20's candidate blocks with -inf stores:
# one row per query token, as wide as max_model_len in decode.
- id: csa1_index
  nodes:
  - op: Attention
    role: block_index_scores
    kind: sparse_mla
    n_q: 32
    n_kv: 1
    d_h: 128
    compress_ratio: 1
    tp: replicated
    caches:
    - {name: index, kind: index, owned: false, bytes_per_entry: 132}
  - op: Elementwise
    role: candidate_mask
    tp: replicated
    context: {bytes_per_entry: 4, per: query_token, decode_extent: max_model_len}
  - op: Attention
    kind: sparse_mla
    n_q: 64
    n_kv: 1
    d_h: 512
    qk_rope_head_dim: 64
    window: 128
    index_topk: 512
    compress_ratio: 1
    caches:
    - {name: swa, kind: window, owned: true, bytes_per_entry: 584, window: 128,
       block_size: 32, page_alignment: 576}
    - {name: compressed, kind: kv, owned: false, bytes_per_entry: 584}
  edges: [[0, 1], [1, 2]]

# DSpark's three draft layers: sliding window only, 128 experts, top 3. A draft query
# reads the 128-token window plus all 5 positions of its block, non-causally.
- id: dspark_0
  nodes: &dspark_nodes
  - {op: GEMM, role: fused_wqa_wkv, n: 1792, k: 5120, tp: replicated}
  - op: Attention
    kind: sparse_mla
    n_q: 64
    n_kv: 1
    d_h: 512
    qk_rope_head_dim: 64
    window: 133
    caches:
    - {name: swa, kind: window, owned: true, bytes_per_entry: 584, window: 128,
       block_size: 32, page_alignment: 576}
  - {op: GroupedGEMM, role: experts, n: 2304, k: 5120, experts: 128, top_k: 3,
     weight_dtype: mxfp4, under_sequence_parallel: token_sharded}
  - {op: All2All, role: moe_dispatch_combine, emit: expert_parallel}
  edges: &edges4 [[0, 1], [1, 2], [2, 3]]
- {id: dspark_1, nodes: *dspark_nodes, edges: *edges4}
- {id: dspark_2, nodes: *dspark_nodes, edges: *edges4}
# The draft's work over the target's tokens: main_proj over three target layers' hidden
# states, then each draft layer's own fused_wqa_wkv for its context KV.
- id: dspark_context
  nodes:
  - {op: GEMM, role: main_proj, n: 5120, k: 15360, tp: replicated}
  - {op: GEMM, role: context_wqa_wkv_0, n: 1792, k: 5120, tp: replicated,
     aliases: dspark_0.fused_wqa_wkv}
  - {op: GEMM, role: context_wqa_wkv_1, n: 1792, k: 5120, tp: replicated,
     aliases: dspark_1.fused_wqa_wkv}
  - {op: GEMM, role: context_wqa_wkv_2, n: 1792, k: 5120, tp: replicated,
     aliases: dspark_2.fused_wqa_wkv}
  edges: [[0, 1], [0, 2], [0, 3]]

stack:
  prologue: [swa, swa_engram,
             csa2_source, csa2_reader, csa2_reader, csa2_reader, csa2_reader, csa2_reader,
             csa2_source, csa2_reader, csa2_reader, csa2_reader, csa2_reader, csa2_reader,
             csa2_source_engram, csa2_reader, csa2_reader, csa2_reader, csa2_reader, csa2_reader,
             csa1_source]
  pattern: [csa1_reader, csa1_reader, csa1_reader, csa1_index]
  repeat: 4
  epilogue: [csa1_reader, csa1_reader, csa1_reader]

head:
- {op: Elementwise, role: final_norm}
- {op: GEMM, role: lm_head, n: 129280, k: 5120, tp: column, weight_dtype: bf16}

speculator:
  method: dspark
  num_spec_tokens: 5
  drafting: parallel_block
  # The draft's own lookup, into the target's table, before each pass. It reduces before
  # the draft shards its stream, so the all-reduce runs under the mode too.
  embedding:
  - {op: Gather, role: dspark_embed_tokens, rows: 129280, row_bytes: 10240, rows_per_token: 1,
     out_bytes_per_token: 10240, shard: rows, aliases: embedding.embed_tokens}
  - {op: AllReduce, role: dspark_embed_out, emit: tensor_parallel}
  stack:
    pattern: [dspark_0, dspark_1, dspark_2]
    repeat: 1
  context_stack:
    pattern: [dspark_context]
    repeat: 1
  head:
  - {op: GEMM, role: dspark_lm_head, n: 129280, k: 5120, tp: column, aliases: head.lm_head}
  # The Markov head, once per draft position in order: a rank-256 row for the previous
  # token, projected to the vocabulary.
  sequential_head:
  - {op: Gather, role: markov_w1, rows: 129280, row_bytes: 512, rows_per_token: 1,
     out_bytes_per_token: 512}
  - {op: GEMM, role: markov_w2, n: 129280, k: 256, tp: replicated, weight_dtype: bf16}

# SM100 and SM103 run the MXFP8 BMM natively and store the 528 B MXFP8 record; under
# nvfp4_ds_mla the compressed cache becomes a 288 B NVFP4 record. Mega attention, SM100's
# default where the layout allows it, resolves an unstated cache dtype to nvfp4_ds_mla;
# which dtype runs is the cost model's to resolve.
base_capability_families: [9]
arch_overrides:
- capability_family: 10
  nodes:
  - {role: wo_a, weight_dtype: fp8}
  caches:
  - {name: swa, bytes_per_entry: 528, page_alignment: 512}
  - {name: compressed, bytes_per_entry: 528, page_alignment: 512,
     bytes_by_cache_dtype: {nvfp4_ds_mla: 288}}
  - {name: index, page_alignment: 512}

Model identity

The catalog commits models/<name>/model.yaml beside the graph. It records which model the entry is and where its config.json came from.

models/qwen3-14b/model.yaml

name: qwen3-14b
source:
  provider: huggingface
  repo: Qwen/Qwen3-14B
  revision: 40c069824f4251a91eefaf281ebe4c544efd3e18
  retrieved: 2026-09-16

Key	Type	Required	Meaning
name	string	yes	Must equal the model's directory name.
source	mapping	yes	Where the vendor configuration was fetched from, below.

source

Key	Type	Required	Meaning
provider	string	yes	The hosting service, such as huggingface.
repo	string	yes	The repository on that service.
revision	string	yes	The exact revision the configuration was taken at.
retrieved	string	no	The date it was fetched.

model.yaml and graph.yaml do not accept _comment keys, unlike the chip, fabric, storage-class and workload-shape files.

Checks

A graph is invalid if

• kind is not ModelGraph, name is empty, or modality is not recognized;
• derived_from lacks format or path, its sha256 is not 64 characters long, or its deriver_version is below 1;
• hidden_size or vocab_size is below 1, or weight_dtype is not a recognized dtype;
• there are no layer kinds, or a layer kind has an empty or duplicate id or no nodes;
• an edge indexes a node that does not exist, joins a node to itself, or closes a cycle;
• the stack expands to no layers, repeat is negative, a pattern has no repeat or a repeat no pattern, or the stack names a layer kind that is not declared;
• a speculator lacks a method, a pattern, at least one draft token or a repeat of at least 1, has an unrecognized drafting, a sequential_head without parallel_block, or a stack or context_stack naming a layer kind that is not declared;
• a cache a layer reads but does not own has no owner earlier in its stack, or differs from its owner in kind, entry size under any cache dtype, tokens per entry, window or entries;
• an aliases reference does not resolve to exactly one weight-holding node of the same shape, names a layer kind the stacks run more than once, or names a node that is itself an alias;
• context_window or default_context_window is negative, or the second exceeds the first or is stated without it;
• arch_overrides is stated without base_capability_families; a family in either is not positive, is listed twice, or is both a base and an override; or an override overrides nothing, or names a role or cache that no node has, or a role held by a node with no parameters;
• any node, in the embedding, a layer kind, the head or the speculator's heads, is invalid.

A node is invalid if

• op or emit is not recognized, or a collective has no emit;
• a key the tables mark required for its primitive is missing or not positive;
• top_k exceeds experts, shared_intermediate_size is set without shared_experts, or shared_experts, shared_intermediate_size, latent_size or bytes_per_token is negative;
• an attention kind or recurrent_kind is not recognized, n_kv exceeds n_q, or n_kv is not 1 for mla or sparse_mla;
• weight_dtype or weight_scale is set on anything other than a GEMM or GroupedGEMM, weight_dtype or state_dtype is not a recognized dtype, or a weight_scale count is not positive;
• tp is not a layout its primitive accepts, or aliases is set on anything other than a GEMM or Gather;
• caches is set on anything other than an Attention node; a cache lacks a name, has an unrecognized kind, a non-positive bytes_per_entry or a negative size; a window cache lacks a positive window, or a state cache positive entries; either key is set on another kind; a bytes_by_cache_dtype entry is unnamed or not positive; a cache the layer does not own states block_size or page_alignment, or is a state cache; or a cache disagrees with the node's window or compress_ratio;
• a Gather lacks a positive rows, row_bytes, rows_per_token or out_bytes_per_token, reads more rows per token than it has, or is split by heads with fewer than two; a Gather key is set on another primitive;
• per_tokens, context or writes_cache is set on anything other than an Elementwise node; a context lacks a positive bytes_per_entry (or a non-negative one with reads_cache) or a recognized per, states a decode_extent on prefill-only work, or states tokens_per_entry beside reads_cache;
• in a layer kind, writes_cache or reads_cache names no cache of the layer, writes_cache names a cache the layer does not own or per_tokens differs from that cache's tokens_per_entry, or per_tokens above 1 is no compress_ratio the layer reads at; on an embedding or head node, any of caches, writes_cache, reads_cache or a per_tokens above 1 is set;
• an aliases node states its own weight_dtype or weight_scale;
• bytes_per_token is set on a primitive other than Elementwise, AllReduce, AllGather or ReduceScatter;
• index_topk or compress_ratio is set on anything other than a sparse_mla attention node, or is negative, or a sparse_mla node has none of index_topk, compress_ratio and window;
• it carries one of the keys its primitive rejects, such as n_q on a GEMM.

Two problems are warnings: a layer kind that neither the stack nor the speculator uses, and an emit condition on a node that is not a collective.

A model identity is invalid if name is empty, source is absent, or source lacks a provider, repo or revision.

validate-catalog also checks that name in both graph.yaml and model.yaml equals the directory name.

Source: spec/model.