BLIS — Blackbox Inference Simulator¶
BLIS is a discrete-event simulator of an LLM serving stack. Its goal is llm-d stack-level parity: simulating an llm-d deployment end to end — the router (EPP), admission and flow control, prefill/decode disaggregation, KV caching and offload, and autoscaling — initially with vLLM as the engine, and eventually a wider range of engines.
Its primary uses are configuration search and optimization, capacity planning, and policy discovery, done robustly and with high fidelity. BLIS is CPU-only and deterministic: the same seed produces byte-identical results, and no GPU is needed.
Quick Start¶
git clone https://github.com/inference-sim/inference-sim.git
cd inference-sim
go build -o blis main.go
git clone --branch 0.3.0 --depth 1 https://github.com/inference-sim/blis-catalog.git
git clone --branch v0.2.0 --depth 1 https://github.com/inference-sim/blis-registry.git
export BLIS_CATALOG=$PWD/blis-catalog # or pass --catalog on every run/replay
./blis run --scenarios examplescenarios \
--scenario deepseek-v4.1-flash-h100-tp8-dspark5.yaml \
--registry $PWD/blis-registry \
--rate 10 --num-requests 100
blis run and blis replay need a catalog (--catalog or BLIS_CATALOG, no default), a
registry (--registry), a scenario directory (--scenarios, any directory of scenario YAMLs)
and a scenario file name within it (--scenario). This repository ships ready-to-run scenarios
in examplescenarios/. The clones pin the releases BLIS is tested against — see
Catalog compatibility.
How BLIS fits together¶
| Repository | Role | Holds |
|---|---|---|
| inference-sim | simulates | Arrivals, the embedded llm-d router (admission, flow control, routing), scheduling, KV block accounting, placement, metrics. |
blis-latency-kernel |
prices | Step time, KV and fixed memory, P/D KV transfer, offload tier transfer, host overheads. |
blis-catalog |
states the facts | Model graphs, chips, fabrics, storage devices, workload presets. |
blis-registry |
holds the fitted numbers | The coefficient sets the kernel prices with. |
blis-schemas |
defines the formats | Scenario, deployment, engine and catalog file formats. |
A scenario names a catalog model and chip, registry coefficient sets, and the deployment (nodes, fabric, and each pool's parallelism and engine settings). inference-sim simulates traffic through the stack and asks the kernel for every cost; the kernel computes it from the catalog's facts and the registry's coefficients.
Request Arrival → llm-d router (admission, flow control, routing) → WaitQueue → Batch Formation → Step Execution → Completion
↓ ↓
KV Allocation Step pricing (blis-latency-kernel)
The llm-d router applies in cluster mode (multi-instance). Single-instance mode skips directly to WaitQueue. See Architecture.
Features¶
| Feature | In one line | Guide |
|---|---|---|
| Latency pricing | The scenario fixes the deployment; the kernel prices every step, memory budget and transfer. | Latency model |
| Workloads | Catalog presets, token distributions, multi-client YAML specs, closed-loop sessions. | Workloads |
| Routing | The real llm-d router (EPP) embedded in-process, configured by a deployment's own EndpointPickerConfig, flags and manifests, named by the scenario's router: block. |
Routing |
| Admission and flow control | The llm-d router's admission and flow control (featureGates: [flowControl]); SLO goodput. |
Admission |
| Scheduling and preemption | Per-instance schedulers and preemption policies. | Scheduling |
| P/D disaggregation | Prefill and decode pools, KV handoff priced over the scenario's fabric. | Cluster |
| KV caching and offload | Prefix caching and tiered offload over catalog storage devices. | KV cache |
| Autoscaling | Node pools with provisioning delays (--policy-config). |
Cluster |
| Trace replay | Any TraceV2 through the same deployment path; run/replay byte-identical. | Replay |
| Results and analysis | TTFT/ITL/E2E, saturation detectors, decision traces, fitness. | Results |
Documentation Guide¶
| Section | What You'll Find |
|---|---|
| Getting Started | What is BLIS, installation, quick start, capacity planning tutorial |
| Concepts | System architecture, core engine, glossary |
| User Guide | Task-oriented guides for each feature |
| Reference | Configuration reference, supported models, workload spec schema |
| Contributing | Extension recipes, PR workflow, standards, templates |
Newcomers: What is BLIS? → Quick Start → Tutorial → Glossary.
License¶
Apache License, Version 2.0. See LICENSE.