Deployment¶

The choices made against a scenario: how its cluster is divided into pools, each pool's parallel layout and engine settings, the KV offload tiers, and how KV moves from prefill to decode. A deployment is validated with its scenario, because most of what makes a layout feasible depends on the cluster.

kind: Deployment
name: 1p3d-ep8
pools:
  - role: prefill
    nodes: 1
    parallel:
      tp: 1
      pp: 1
      dp: 8
      dp_local: 8
      enable_expert_parallel: true
    engine:
      all2all_backend: deepep_high_throughput
      max_num_batched_tokens: 16384
  - role: decode
    nodes: 3
    parallel:
      tp: 1
      pp: 1
      dp: 8
      dp_local: 8
      enable_expert_parallel: true
    engine:
      all2all_backend: deepep_low_latency
      max_num_seqs: 512
offload:
  tiers:
    - tier: cpu_dram
      bytes: 549755813888
pd_transfer:
  connector: NixlConnector

Keys¶

Key	Type	Required	Meaning
`kind`	string	yes	Always `Deployment`.
`name`	string	yes	The deployment's identity.
`pools`	list	yes	At least one pool, below.
`offload`	mapping	no	KV offload tiers below HBM.
`pd_transfer`	mapping	if any pool is `prefill`	How KV moves from prefill to decode.
`router`	mapping	no	The inputs of the llm-d router that routes the deployment's requests. Absent: the router's built-in configuration over every pod.

Pools¶

Key	Type	Required	Meaning
`role`	string	yes	`colocated`, `prefill` or `decode`.
`nodes`	integer	yes	Nodes in the pool, at least 1.
`parallel`	mapping	yes	The parallel layout of each engine in the pool.
`engine`	mapping	no	Engine settings for the pool. Each omitted setting takes the engine's default.
`labels`	mapping	no	Kubernetes labels on the pool's model-server pods, as its pod template sets them. The router selects and filters endpoints by them. `llm-d.ai/role` is derived from `role` (`colocated` -> `prefill-decode`, `prefill` -> `prefill`, `decode` -> `decode`); given here, it must have that value.

A pool may run several engines with the same layout and settings. Its nodes is the pool's whole extent; one engine may occupy only part of it.

Pools of role prefill and decode come together or not at all, and neither may appear beside a colocated pool.

`parallel`¶

Key	Type	Required	Meaning
`tp`	integer	yes	Tensor-parallel width, at least 1.
`pp`	integer	yes	Pipeline-parallel width, at least 1.
`dp`	integer	yes	Data-parallel width, at least 1.
`dp_local`	integer	no	Data-parallel replicas resident on one node. Must divide `gpus_per_node` and not exceed `dp`. Zero means the launch decides.
`data_parallel_load_balancing`	string	no	How the data-parallel ranks are exposed, which decides what a router chooses between. One of the load-balancing modes. Omitted, `internal`. `hybrid` and `external` need `dp` > 1. The rules pack refuses `external` on a model without experts and `hybrid` without `dp_local`, and names the mode that runs when the engine resolves `hybrid` to another. `--api-server-count` is not part of it: it sets how many frontend processes share an endpoint, not which ranks a router can reach.
`pcp`	integer	no	Prefill-context-parallel width. Zero or 1 means off.
`dcp`	integer	no	Decode-context-parallel width. Reuses the tensor-parallel ranks; see below. Zero or 1 means off.
`enable_expert_parallel`	boolean	no	Shard MoE experts across ranks. Default false.

There is no key for the expert-parallel width. It is derived: tp × pcp × dp when expert parallelism is enabled, and 1 otherwise. A key for it would let a document state a width no engine would build. Parallel layouts explains this and the checks below in more depth.

`engine`¶

Every engine key is optional. Some are requests the engine may not honor: disable_custom_all_reduce, allreduce_backend, async_scheduling, disable_cascade_attn and the decode-context keys. A cost model reports what it resolved in its Resolution. Which values each string key accepts depends on the engine release and is checked by its rules pack.

Communication:

Key	Type	Required	Meaning
`all2all_backend`	string	no	How a mixture-of-experts (MoE) layer sends tokens to their experts and back.
`disable_custom_all_reduce`	boolean	no	Disable the custom all-reduce GPU kernel, which runs on the streaming multiprocessors (SMs) rather than the network. Omitted means false, so the kernel is used where the layout allows.
`allreduce_backend`	string	no	`custom` or `nccl`, stated directly instead of through `disable_custom_all_reduce`. Must not contradict it.
`dbo`	mapping	no	Dual-batch overlap, below.

Scheduling and memory:

Key	Type	Required	Meaning
`block_size`	integer	no	KV block size in tokens.
`max_num_batched_tokens`	integer	no	Token budget per step.
`max_num_seqs`	integer	no	Sequence cap per step.
`long_prefill_token_threshold`	integer	no	The most tokens of one prefill scheduled in a step, so a long prompt cannot take the whole token budget from the decodes beside it. Not negative; omitted or 0 leaves prefills uncapped. The engine refuses one above `max_model_len`. 0.31.0 waives it for a request alone in the scheduler; 0.29.0 does not.
`max_model_len`	integer or `auto`	no	Context length. `auto` (or `-1`) asks the engine to fit the window to the KV memory it has left, which is a different request from omitting the key: omitted, the engine uses the model's whole window and refuses to start if it does not fit.
`gpu_memory_utilization`	number	no	Fraction of GPU memory the engine may use, in [0, 1]. Zero means the engine's default.
`scheduling_policy`	string	no	`fcfs` or `priority` in vLLM 0.29.0.
`async_scheduling`	boolean	no	Omitted means the engine decides through its own disqualifying conditions.
`frontend`	string	no	The API server that renders and tokenizes each prompt: one of the frontends. Omitted, `python`. `rust` is vLLM's vllm-rs frontend, selected by the environment variable `VLLM_USE_RUST_FRONTEND=1`, not a flag. It changes how long a request waits for admission, per prompt token.
`cudagraph_mode`	string	no	CUDA graph capture strategy.
`max_cudagraph_capture_size`	integer	no	`compilation_config.max_cudagraph_capture_size`: the largest step, in tokens, a CUDA graph is captured for; a larger step runs eager. At least 1. Omitted, the release default, min(`max_num_seqs` x decode query length x 2, 512), or 1024 on SM100. The engine caps it at `max_num_batched_tokens` and truncates it to the largest size it generates. `cudagraph_mode: NONE`, or `--enforce-eager`, which has no field here, captures nothing.
`cudagraph_capture_sizes`	list of integers	no	`compilation_config.cudagraph_capture_sizes`: the exact step sizes to capture, each at least 1, not empty. Sizes above `max_num_batched_tokens` are dropped. With `max_cudagraph_capture_size` too, that must be the largest kept, or the engine refuses the pair.

Weights and caches:

Key	Type	Required	Meaning
`quantization`	string	no	Weight format the engine serves, where it differs from the checkpoint's. Omitted means the checkpoint's own format.
`moe_backend`	string	no	The expert-kernel backend, `--moe-backend`. Omitted means `auto`. A mega-MoE backend is one of the conditions under which DeepSeek-V4.1 turns on its own sequence parallelism.
`cache_dtype`	string	no	KV cache format.
`enable_prefix_caching`	boolean	no	Omitted means true, vLLM's default. `false` describes a deployment without prefix caching.
`swa_bounded_replay`	boolean	no	DeepSeek-V4.1 SWA bounded replay (`--swa-bounded-replay` / `--no-swa-bounded-replay`). `true` keeps the sliding-window cache out of prefix caching and recomputes a hit's last sliding window; `false` lets the window cache take part in prefix caching (retained window tails). Omitted, the engine's resolution: on in vLLM 0.31.0 under the V2 model runner, without prefill context parallelism and not on ROCm. Only DeepSeek-V4.1 reads it.
`mamba_cache_dtype`	string	no	Format of a recurrent layer's state.
`mamba_ssm_cache_dtype`	string	no	Format of the SSM state alone, where it differs.
`mamba_cache_mode`	string	no	Whether recurrent state is fixed per sequence or grows with context.
`disable_cascade_attn`	boolean	no	Cascade attention reads a prefix shared by a batch once rather than once per request. Omitted means `true`, so it is disabled unless this is set to `false`.

Speculation and experts:

Key	Type	Required	Meaning
`speculative`	mapping	no	Speculative decoding, below.
`eplb`	mapping	no	Expert-parallel load balancing, below.
`engram_config`	mapping	no	Where DeepSeek-V4.1's Engram tables live, below. Decides the residency of every model `Gather` whose `residency_control` is `engram_cpu_offload`.

Decode-context parallelism. These have no effect when dcp is 0 or 1:

Key	Type	Required	Meaning
`dcp_comm_backend`	string	no	The collectives a decode step runs: `ag_rs` (the default) or `a2a` in vLLM 0.29.0.
`dcp_q_replicate`	boolean	no	Replicate the query projection of multi-head latent attention (MLA) on every rank, to skip a query all-gather.
`cp_kv_cache_interleave_size`	integer	no	Consecutive tokens one rank holds before the next takes over. Zero means unstated, which is a different request from 1.

`engine.dbo`¶

Key	Type	Required	Meaning
`enable_dbo`	boolean	no	Split a batch so one half's communication overlaps the other's compute.
`dbo_decode_token_threshold`	integer	no	Batch size in tokens at or above which a uniform-decode batch is split. Omitted takes the engine's default. With `enable_dbo` on, vLLM 0.31.0 refuses a stated value below the two microbatches, 0 or 1. Not negative.
`dbo_prefill_token_threshold`	integer	no	Batch size in tokens at or above which any other batch is split. As for the decode threshold.

`engine.eplb`¶

Key	Type	Required	Meaning
`enable_eplb`	boolean	no	Enable expert-parallel load balancing.
`num_redundant_experts`	integer	no	Extra expert replicas. With load balancing on, the expert-parallel width must divide the experts plus these.
`window_size`	integer	no	Load-balancing window.
`step_interval`	integer	no	Steps between rebalances.

`engine.speculative`¶

Key	Type	Required	Meaning
`method`	string	yes	The draft method. Which names are valid depends on the engine release.
`num_spec_tokens`	integer	yes	Draft tokens per step, at least 1.

`engine.engram_config`¶

The cost-bearing keys of vLLM's --engram-config. All three are optional and take the engine's default when omitted. The rules pack for 0.31.0 refuses the block on a model with no Engram lookups.

Key	Type	Required	Meaning
`cpu_offload`	boolean	no	Keep the tables in pinned host memory, read over the host link. Omitted means true. `false` puts them in GPU memory.
`dp_shared_memory`	boolean	no	Keep one host copy per node, shared by its data-parallel replicas. Omitted lets the engine decide: it enables it when `cpu_offload` is on, `use_thp` is off, `dp` exceeds 1 and elastic EP is off. Without sharing, the node's co-located replicas split each table between them instead.
`use_thp`	boolean	no	Back a per-rank host table with transparent huge pages. Refused with `dp_shared_memory`.

Offload¶

Key	Type	Required	Meaning
`tiers`	list	yes	At least one tier, below.
`eviction_policy`	string	no	KV offload cache policy. Omitted means `lru`.
`prefetch_depth`	integer	no	How many steps ahead a fetch is issued.
`connector`	string	no	The KV connector that implements offload.
`spec`	string	no	The offloading spec, such as `CPUOffloadingSpec`.

`offload.tiers`¶

Key	Type	Required	Meaning
`tier`	string	yes	The storage class this tier uses. If the scenario's cluster lists `storage`, it must be one of those.
`bytes`	integer	yes	Capacity of the tier in bytes, at least 1.

PD transfer¶

Key	Type	Required	Meaning
`connector`	string	no	The KV connector that moves KV from prefill to decode.

There is no bandwidth key. Transfer bandwidth comes from the scenario's fabric and a registry coefficient, so it cannot be set here.

`router`¶

Key	Type	Required	Meaning
`epp_flags`	list	no	The router's (EPP's) command-line flags, verbatim, one list entry per flag (`--name=value`). They select the pool with `--pool-name` or `--endpoint-selector`. A Helm chart's `epp.flags` map is rendered to this list (`name: value` becomes `--name=value`).
`endpoint_picker_config`	string	no	The `EndpointPickerConfig` file, as the router reads it with `--config-file`; a blank path is an error. Naming the configuration here and in `epp_flags` is an error.
`manifests`	list	no	Kubernetes manifest files. The router reads their `InferencePool`, `InferenceObjective` and `InferenceModelRewrite` objects; other kinds are ignored. Objectives and rewrites take effect only for an `InferencePool` (`--pool-name`); with a standalone pool (`--endpoint-selector`) the router does not reconcile them, so a consumer may refuse them there.

Paths are relative to the directory of the file that holds the deployment. The block names the router's own inputs and nothing else: scorers, weights, admission and flow control are configured in those files, exactly as for a real deployment, so the same files configure the real router and a simulated one. There is no deployment key for a routing decision. The router here is the Endpoint Picker. The multimodal Coordinator pipeline is a separate llm-d component and is not named by this block; a simulator that cannot host it refuses it at startup.

These checks are syntactic. A deployment that passes them is well formed, not necessarily routable by a given consumer: a simulator refuses, at its startup and with the reason, router flags, plugins or manifests it cannot run faithfully.

pools[].labels are a different kind of input from the three above. In a real deployment the labels live in each pool's pod template; a deployment here has no pod spec, so the labels restate them, so that an InferencePool selector or a label filter has the pods' labels to match.

Checks¶

On its own¶

A deployment is invalid if

kind is not Deployment, name is empty, or pools is empty;
a pool's role is not recognized, or its nodes is less than 1;
tp, pp or dp is less than 1, pcp or dcp is negative, dp_local is negative, or dp_local exceeds dp;
dcp does not partition the group it reuses. With pcp at most 1, tp must be divisible by dcp. With pcp above 1, dcp must be 1, pcp or tp × pcp;
a prefill pool has no decode pool, a decode pool has no prefill pool, or a colocated pool appears beside either;
a prefill pool is present and pd_transfer is absent;
an engine count (block_size, max_num_batched_tokens, max_num_seqs, cp_kv_cache_interleave_size) is negative, or max_model_len is below -1. A max_model_len that is not a whole number, auto, or a number with a k/m/g/t or K/M/G/T suffix fails to load, and so does one that comes to zero or is too large to hold;
gpu_memory_utilization is not finite or lies outside [0, 1];
max_cudagraph_capture_size is stated below 1, or cudagraph_capture_sizes is stated empty or lists a size below 1;
disable_custom_all_reduce is true while allreduce_backend is custom;
dbo states a negative threshold, or an enabled eplb a negative num_redundant_experts;
speculative is present with an empty method or with num_spec_tokens below 1;
offload is present with no tiers, a tier is unnamed or named twice, a tier's bytes is below 1, or prefetch_depth is negative.

It draws a warning if expert parallelism is enabled but the derived width is 1, or if allreduce_backend is nccl while disable_custom_all_reduce is explicitly false.

Against its scenario's cluster¶

These run when the bundle holds both documents. A deployment without its scenario is itself reported, because these checks would otherwise be skipped.

The pools' nodes must sum to the cluster's nodes. A deployment lays out the whole cluster.
A pool's dp_local, when set, must divide gpus_per_node.
An engine needs one GPU per rank, pp × tp × pcp × dp in all, and they must fit in the pool's nodes. Decode-context and expert parallelism add no ranks.
With dp_local set, the local replicas must fit on one node in the way the engine places them. The check refuses only layouts that no launch could place. In the rare case that its bounded search cannot decide, it reports a warning that the layout's fit is unverified.
Each offload tier must be one of the cluster's storage classes, if the cluster lists any.

Engine-release checks, such as whether a backend name exists or whether prefill-context parallelism may be combined with data parallelism, are in the rules pack.

Source: spec/deployment.

Deployment¶

Keys¶

Pools¶

parallel¶

engine¶

engine.dbo¶

engine.eplb¶

engine.speculative¶

engine.engram_config¶