Concepts & Glossary¶

This page defines terminology used throughout BLIS documentation. Terms are listed alphabetically with cross-references to the relevant design pages.

Admission¶

The llm-d router's decision to accept or deny a request. With the flowControl feature gate off, the router's legacy admission controller sheds sheddable requests (priority < 0) when the saturation detector reports a saturated pool; an admitter plugin (probabilistic-admitter) may also deny requests. A denied request is counted as rejected_requests. See Admission Control.

Batch Formation¶

The process of selecting which requests from the wait queue join the running batch for the next step. BLIS implements vLLM-style continuous batching with chunked prefill and preemption. See Core Engine: Batch Formation.

blis-latency-kernel¶

The Go module (github.com/inference-sim/blis-latency-kernel) that prices everything BLIS needs a cost for: step time, KV and fixed memory, P/D KV transfer, offload tier transfer and host overheads. It reads model graphs, chips, fabrics and storage devices from blis-catalog and fitted coefficients from blis-registry. See Latency Models Guide.

Block (KV Block)¶

The unit of KV cache allocation. Each block holds a fixed number of tokens (default: 16). Requests are allocated blocks proportional to their token count. Blocks are reference-counted and can be shared across requests via prefix caching. See Core Engine: KV Cache.

Calibration Report¶

The JSON output of blis calibrate comparing real observed latencies against simulator predictions. Contains per-request TTFT and E2E deltas, aggregate error metrics (MAPE, Pearson correlation, percentile comparisons), bias assessment, and a quality rating. See Observe / Replay / Calibrate.

Chunked Prefill¶

A vLLM optimization where long prefill sequences are split into chunks that fit within the per-step token budget (max-num-batched-tokens). Controlled by --long-prefill-token-threshold. See Core Engine: Batch Formation.

Cohort¶

A workload primitive that models a population of similar clients sharing the same arrival process, token distributions, and SLO class, with an optional time-varying lifecycle modifier. A cohort specifies a population count and one of three lifecycle patterns (diurnal, spike, drain); BLIS expands it into individual ClientSpec entries at workload generation time, each seeded independently for staggered arrival times. Distinct from a single ClientSpec in that cohorts directly model populations of concurrent users rather than a single traffic source. See Workload Specifications: Cohort Dynamics and Workload Spec Schema.

Continuous Batching¶

The serving strategy where new requests can join the running batch between decode steps, rather than waiting for the entire batch to complete. BLIS models this by re-evaluating the batch composition at every step. See Core Engine: Batch Formation.

Counterfactual Regret¶

A trace-level metric from --trace-level decisions: for each routing decision, regret = best_score - chosen_score over the scores the llm-d router's primary scheduling profile gave every endpoint. With max-score-picker the chosen endpoint is always the best-scoring one, so regret is zero; it is non-zero only with pickers that may choose a lower-scoring endpoint. --counterfactual-k ranks the top-k endpoints. See Cluster Simulation: Decision Tracing.

Decode Phase¶

The token generation phase where the model produces output tokens one at a time (or in parallel within a step). Each decode token uses the KV cache from all prior tokens. Decode steps are typically memory-bandwidth-bound. Contrast with Prefill Phase.

Discrete Event Simulation (DES)¶

A simulation paradigm where the system state changes only at discrete event times. BLIS maintains a priority queue of timestamped events and advances the simulation clock by jumping between events, rather than stepping through fixed time intervals. See Core Engine: Event Queue.

Distribution Synthesis¶

The --rate mode of blis observe that generates workload from statistical distributions (prompt/output token counts, arrival rate) instead of a workload spec YAML file. Useful for quick single-client experiments without crafting a full workload specification. See Observe / Replay / Calibrate: Distribution Synthesis Flags.

E2E (End-to-End Latency)¶

Total time from request arrival to final token completion. Computed as TTFT + sum(ITLs), where each ITL includes step time plus per-token output processing overhead. See Core Engine: Metrics.

EndpointPickerConfig¶

The llm-d router's configuration file (kind: EndpointPickerConfig): the plugins, the scheduling profiles that compose them, feature gates and the flow-control block. BLIS reads it from the scenario deployment's router.endpoint_picker_config unchanged from a real deployment. See Routing.

EPP (Endpoint Picker)¶

The llm-d router component that admits, queues and routes each request to a model-server pod. BLIS embeds the real EPP in-process (package sim/llmd) and runs it on the simulation's hosting runtime; its command-line flags are the scenario deployment's router.epp_flags. See Routing.

Fitness Score¶

A single numeric value summarizing multi-objective simulation performance. Computed as a weighted combination of configurable metrics (TTFT percentiles, E2E percentiles, throughput). Latency metrics normalized via 1/(1 + value/1000); throughput metrics via value/(value + reference). See Configuration Reference.

Flow Control¶

The llm-d router's request queueing layer, enabled with featureGates: [flowControl] in the EndpointPickerConfig. Requests wait in priority bands (from InferenceObjective priorities) and are dispatched when the saturation detector allows, subject to capacity limits (flowControl.maxRequests, per-band capacity), a queue-wait budget (flowControl.defaultRequestTTL) and fairness, ordering and usage-limit plugins. Outcomes map to the INV-1 buckets gateway_queue_rejected, gateway_expired and gateway_queue_depth. See Admission Control.

GAIE (Gateway API Inference Extension)¶

The Kubernetes Gateway API extension that defines the InferencePool and InferenceObjective resources the llm-d router reads.

HOL Blocking (Head-of-Line Blocking)¶

A scheduling pathology where one long prefill monopolizes the GPU for an entire step, preventing shorter requests from entering decode until it completes. In BLIS, a 2,048-token prompt can occupy an entire step, blocking all decode-phase requests during that step. Detected and counted as "HOL Blocking Events" in the anomaly counters. The mitigation is chunked prefill (--long-prefill-token-threshold), which splits long prefills so decode requests can be interleaved. See KV Cache: Chunked Prefill and Chunked Prefill.

Horizon¶

The simulation time limit in ticks (microseconds). The simulation stops when the clock exceeds the horizon or all requests complete, whichever comes first. See Configuration Reference.

Hosting Runtime¶

The seam (hosting.Runtime, in the inference-sim fork of llm-d-router) through which the router reads time, starts background work, sets deadlines, blocks, draws randomness and makes outbound HTTP calls. BLIS's implementation (sim/llmd/runtime.go) runs the router on the cluster's event loop: a virtual clock, coroutines that run one at a time, seeded random streams, and background work as daemon events that never keep a simulation alive. See The llm-d Router Fork.

InferenceObjective¶

A GAIE resource that gives a named objective a priority. A request names its objective with the x-llm-d-inference-objective header; BLIS sends the request's SLO class there. Negative priority makes a request sheddable. The router reconciles objectives only for an InferencePool (--pool-name); examples/llm-d/inferencepool.yaml defines one per SLO class. See Admission Control.

InferencePool¶

A GAIE resource that selects the model-server pods one router serves. BLIS reads it from router.manifests when the EPP flags name it with --pool-name; pools[].labels puts the labels its selector expects on the simulated pods. Without a pool name, the router runs standalone over every simulated pod. See Routing.

ITL (Inter-Token Latency)¶

The observed time between consecutive decode steps for a single request. Without speculative decoding, a request generating N output tokens produces N-1 ITL entries (one per inter-token gap); under speculative decoding one entry covers several accepted tokens, so the entry count is lower than N-1. ITL varies with batch composition changes between steps. The per-request mean ITL is reported as TPOT (Time Per Output Token) — the ITL sum over N-1, where N is the number of output tokens the request actually emitted (for a length-capped request that is maxModelLen - inputLen, not its assigned output length). The aggregate itl_mean_ms and ITL percentiles are instead computed over the pooled per-step ITL samples across all requests (see Latency Models).

KV Cache¶

GPU memory organized as blocks that store key-value tensors computed during attention. BLIS simulates block allocation, prefix sharing, LRU eviction, and optional CPU offloading without actual GPU memory. See Core Engine: KV Cache.

Latency Model¶

The component that predicts GPU execution time for a batch step, plus host overheads around it. BLIS uses one: blis-latency-kernel, adapted to the sim.LatencyModel interface by sim/kernelmodel. See Core Engine: Latency Models and Latency Models Guide.

llm-d¶

An open-source LLM inference routing and serving framework that BLIS is designed to simulate and validate against. BLIS routes every cluster run with llm-d's own router (EPP), embedded in-process from the inference-sim fork of llm-d-router, so a deployment's router configuration runs in BLIS unchanged. See Routing.

MaxModelLen¶

Maximum total sequence length (input + output) for a single request, in tokens. Mirrors vLLM's --max-model-len and is read from each scenario pool's engine.max_model_len: a count, auto (the model graph's context window shrunk to what one rank's KV cache holds), or unstated (the graph's context window). Requests whose input alone fills the context window (input >= MaxModelLen) or whose input + output budget exceeds it are dropped before entering the wait queue. A three-part proactive cap matches vLLM scheduler.py:773-774: FormBatch clamps token scheduling to maxModelLen - 1 - ProgressIndex, executeBatchStep skips decode when 0 tokens allocated, and processCompletions force-completes at the maxModelLen - 1 boundary. Output per length-capped request: maxModelLen - inputLen, matching vLLM's check_stop (num_tokens >= max_model_len) — ProgressIndex is num_computed_tokens, one behind the generated-token count because the first output token is charged to prefill, so the boundary token is generated and counted. See Latency Models Guide.

MoE (Mixture of Experts)¶

A transformer architecture variant where each layer routes each token to a subset of specialist sub-networks (experts) rather than processing all tokens through shared weights. blis-latency-kernel reads expert geometry (expert count, top-k) from the model graph in the catalog. A scenario's dp > 1 runs one replica per data-parallel rank; enable_expert_parallel is priced by the kernel. See Latency Models Guide.

Observe / Replay / Calibrate Pipeline¶

The end-to-end workflow of blis observe → blis replay → blis calibrate for validating simulator accuracy against real inference servers. Each stage is independently useful: observe collects latency baselines, replay tests simulator behavior on real traces, and calibrate compares results. See Observe / Replay / Calibrate.

Oracle Knowledge Boundary (INV-9)¶

The principle that control-plane decisions (admission, routing, scheduling, priority) must not read Request.OutputTokens. The actual output token count is oracle knowledge — known only after generation completes. The control plane uses MaxOutputLen (client-declared output budget) or input-only checks instead. Only the execution engine (batch step processing, completion detection) may access OutputTokens for token generation and determining when a request finishes. See Standards: Invariants.

Policy Bundle¶

A YAML configuration file (--policy-config) that specifies priority, scheduling, preemption, node pools, autoscaling and instance lifecycle in one place. CLI flags override bundle values when explicitly set. Admission, routing and flow control are not in the bundle: they are the llm-d router's configuration. See Configuration Reference.

Preemption¶

KV cache eviction under memory pressure. When the batch formation algorithm cannot allocate blocks for a continuing request, it evicts requests from the batch, frees their blocks, and re-enqueues them at the front of the wait queue. By default (--preemption-policy fcfs), BLIS evicts from the batch tail. With --preemption-policy priority, the least-urgent SLO tier is evicted first. See Core Engine: Batch Formation.

Prefill Phase¶

The initial processing phase where the model computes attention over all input tokens. Prefill is compute-bound for large inputs. After prefill completes, the request transitions to decode. TTFT is recorded at this boundary.

Prefix Caching¶

Reuse of KV blocks across requests that share a common input prefix. BLIS uses hierarchical block hashing: each block's hash chains with the prior block's hash, enabling semantic prefix matching. Shared blocks are reference-counted and exempt from eviction while in use. See Core Engine: KV Cache.

Priority Policy¶

A per-instance policy that assigns a numeric priority score to each request before batch formation. Used by priority-aware schedulers to reorder the wait queue. Built-in policies: constant, slo-based, inverted-slo (testing only). Note: despite its name, slo-based currently uses only request age (favoring older requests), not per-request SLO metadata. See Core Engine: Scheduling.

Saturation Detector¶

An llm-d router plugin that reports whether the pool is saturated: utilization-detector (queue depth and KV-cache utilization thresholds), concurrency-detector, or max-saturation-detector over several. It gates the legacy admission controller's shedding and flow control's dispatch. Distinct from BLIS's post-hoc saturation analysis of a completed run (--detectors).

Scenario¶

A blis-schemas YAML file of two documents, Scenario and Deployment, naming the model, cluster hardware/fabric/nodes, coefficient sets, engine version, and each pool's role, parallelism and engine settings. Passed to blis run/blis replay with --scenario and --scenarios. See Latency Models Guide.

Scheduling Delay¶

The elapsed time between a request's arrival and the moment it enters the running batch, reported as scheduling_delay_ms (per-request JSON) and scheduling_delay_p99_ms (aggregate). Includes the host queueing overhead plus wait queue residence time. Distinct from TTFT, which additionally includes prefill compute time. High scheduling delay with low preemption rate indicates queue saturation (add instances); low scheduling delay with high TTFT indicates compute saturation (reduce batch size or enable chunked prefill). See Metrics & Results and Core Engine: Metrics.

Scheduling Profile¶

A named set of llm-d router plugins (filters, scorers with weights, a picker) in an EndpointPickerConfig. The profile handler picks which profiles run for a request: single-profile-handler runs one; disagg-profile-handler runs prefill and decode profiles to disaggregate a request. See Routing.

Scorer¶

An llm-d router plugin in a scheduling profile that scores each candidate endpoint for one signal (queue depth, KV-cache utilization, prefix-cache match, LoRA affinity, ...). A profile sums the weighted scores and its picker chooses an endpoint. The supported scorers are listed in Routing.

Seed¶

The random seed for deterministic simulation. Same seed produces byte-identical stdout across runs (INV-6). BLIS uses partitioned RNG to isolate randomness across subsystems. See Configuration Reference.

SLO Class¶

A label that categorizes each request by service-level objective sensitivity: critical, standard, batch, sheddable, background. In a cluster it reaches the llm-d router as the request's InferenceObjective name, whose priority governs shedding and flow-control ordering. Inside each instance, the engine priority from the SLO priority map (default critical 4, standard 3, batch −1, sheddable −2, background −3, overridable with priority.slo_priorities in the policy bundle) drives priority scheduling and preemption. Classes are also reported per class in metrics. Set via slo_class in the workload spec YAML. See Workload Specifications and Admission Control.

Step¶

A single iteration of the inference engine. Each step processes one batch: prefill tokens for new/continuing requests and decode tokens for generating output. Step time is predicted by the latency model. See Core Engine: Step Phases.

Tensor Parallelism (TP)¶

A parallelism strategy that shards a model's weight tensors across multiple GPUs, reducing per-GPU compute load and KV memory requirements at the cost of per-layer collective communication, which blis-latency-kernel prices. Stated in the scenario as a pool's parallel.tp. Increasing the instance count (replication) instead increases throughput without reducing per-request latency. See Latency Models Guide.

Tick¶

The fundamental time unit in BLIS, representing one microsecond. All timestamps, durations, and latencies are measured in ticks. The simulation clock advances in ticks.

Tiered KV Cache¶

An extension of the KV cache with GPU and CPU tiers. When GPU utilization exceeds a threshold, blocks are offloaded to CPU memory. On cache miss, blocks can be reloaded from CPU with a transfer latency penalty. See Core Engine: KV Cache.

TPOT (Time Per Output Token)¶

The mean inter-token latency across all decode steps for a request, computed as mean(ITL). TPOT represents the steady-state token generation speed once prefill completes; lower TPOT means faster streaming throughput. Each ITL sample includes the step time plus the per-token output-processing overhead. Reported as the itl_mean_ms field in JSON output. See ITL and Core Engine: Metrics.

TraceV2¶

The trace format used by the observe/replay/calibrate pipeline, consisting of two files: a header YAML file (recording model, server config, and observation metadata) and a data CSV file (recording per-request timing: arrival time, TTFT, E2E latency, token counts). See Observe / Replay / Calibrate.

TTFT (Time To First Token)¶

Time from request arrival to completion of the prefill phase (first output token ready). Includes queueing delay, prefill step times, and per-token output processing overhead. A key latency SLO metric for interactive applications. See Core Engine: Metrics.

Warmup Requests¶

Initial requests dispatched by blis observe that are excluded from trace output (--warmup-requests N). Warmup requests allow the server to populate its KV cache and reach steady-state scheduling before measurement begins, avoiding cold-start artifacts in the recorded trace. See Observe / Replay / Calibrate: blis observe.

Work-Conserving¶

The property that the simulator never idles while requests are waiting. After every step completion, if the wait queue is non-empty, a new StepEvent is scheduled immediately (INV-8). See Core Engine: Event Queue.

Workload Specification¶

A YAML file (--workload-spec) defining multi-client workloads with per-client arrival distributions, token length distributions, prefix groups, and SLO classes. Supports poisson, gamma, weibull, and constant arrival processes and gaussian, exponential, pareto_lognormal, constant, and empirical token distributions. See Configuration Reference.