Skip to content

Quick Start

Run your first BLIS simulation in 30 seconds.

No credentials or network access are needed to run a simulation: BLIS reads each model's architecture from a local checkout of the model catalog and makes no HuggingFace requests. (HF_TOKEN matters only if you are downloading a new — possibly gated — model's config.json by hand to add a catalog entry.)

Set up the catalog, registry and scenarios

Every blis run / blis replay needs three things besides the binary:

  • the catalog — a clone of blis-catalog (models, chips, fabrics, storage devices, workload presets), located by --catalog <path> or BLIS_CATALOG (the flag wins; there is no default and no search path);
  • the registry — a clone of blis-registry (the fitted coefficients), passed as --registry;
  • a scenario — a blis-schemas Scenario + Deployment YAML naming the model, hardware, parallelism and engine settings, picked with --scenario <file name> from --scenarios <dir>.
git clone --branch 0.3.0 --depth 1 https://github.com/inference-sim/blis-catalog.git
git clone --branch v0.2.0 --depth 1 https://github.com/inference-sim/blis-registry.git
export BLIS_CATALOG=$PWD/blis-catalog   # or pass --catalog on every command
export SCENARIOS=examplescenarios

A scenario directory can be any directory of scenario YAMLs. $SCENARIOS above is this repository's ready-to-run set (examplescenarios/). The clones pin the releases BLIS is tested against — see Catalog compatibility.

Single-Instance Simulation

./blis run --scenarios $SCENARIOS \
  --scenario deepseek-v4.1-flash-h100-tp8-dspark5.yaml --registry $PWD/blis-registry \
  --rate 10 --num-requests 100

This runs 100 requests at 10 requests/second through one DeepSeek-V4.1-Flash engine on an 8-GPU H100 node (FP8, vLLM, TP8 with expert parallelism and DSpark speculative decoding), priced by blis-latency-kernel.

The scenario states the deployment

There are no --model, --hardware or --tp flags. Model, hardware, parallelism, block size, batch limits, max model length, prefix caching, cache dtype and speculative decoding all come from the scenario; the KV block budget comes from the kernel. A model runs only if it is in the catalog — nothing is fetched at run time.

Reading the Output

BLIS prints diagnostic logs to stderr and results to stdout. You'll see log lines (prefixed with INFO or WARN) followed by a === Simulation Metrics === header and pretty-printed JSON:

Latency metrics (all in milliseconds, reported as mean/p90/p95/p99):

Field What It Measures
ttft_mean_ms, ttft_p99_ms Time to First Token — how long until the first output token is generated. Lower is better for interactive use.
e2e_mean_ms, e2e_p99_ms End-to-End latency — total time from request arrival to final output token.
itl_mean_ms, itl_p99_ms Inter-Token Latency — time between consecutive output tokens. Lower means smoother streaming.
scheduling_delay_p99_ms Wait time from request arrival until processing begins (includes any queueing).

Throughput:

Field What It Measures
responses_per_sec Completed requests per second.
tokens_per_sec Output tokens generated per second.
completed_requests How many requests finished within the simulation window.
total_input_tokens, total_output_tokens Total tokens processed across all completed requests.

Health indicators:

Field What It Measures
preemption_count Number of times a running request was evicted to make room for others. Non-zero suggests the system is overloaded.
dropped_unservable Requests rejected because they were too large for the configured memory or context length.
still_queued, still_running Requests not yet completed when the simulation ended. Non-zero means the workload outlasted the simulation window.

Cluster Mode

Scale to 4 instances behind the router. --num-instances cannot exceed the scenario pool's rank capacity (pool nodes × gpus_per_node / (pp × tp × pcp)); the single-node scenario above is one rank, so this uses the two-node gpt-oss-120b-h100-fp4-vllm-tp2-2node.yaml (2 × 8 ÷ TP2 = 8 ranks, up to eight instances):

./blis run \
  --scenarios examplescenarios \
  --scenario gpt-oss-120b-h100-fp4-vllm-tp2-2node.yaml --registry $PWD/blis-registry \
  --num-instances 4 \
  --rate 100 --num-requests 500

This simulates a 4-instance cluster receiving 100 requests/second. Every cluster run routes with the llm-d router (the Endpoint Picker) embedded in the simulator; with no router configuration it uses the router's built-in default configuration (queue, KV-cache utilization and prefix-cache scorers). A real deployment's router configuration is stated by the scenario's router: block and pool labels; see Routing.

Multi-instance output format

In cluster mode, BLIS prints one JSON block per instance plus a cluster-level summary (5 blocks total for 4 instances). The cluster summary has "instance_id": "cluster". If piping to jq, use --slurp to handle multiple JSON objects: ./blis run ... 2>/dev/null | jq --slurp '.[] | select(.instance_id == "cluster")' to extract the cluster summary.

Try Different Configurations

# Higher traffic rate
./blis run --scenarios examplescenarios \
  --scenario gpt-oss-120b-h100-fp4-vllm-tp2-2node.yaml --registry $PWD/blis-registry \
  --num-instances 8 --rate 500 --num-requests 2000

# With decision tracing (see where each request was routed)
./blis run --scenarios examplescenarios \
  --scenario gpt-oss-120b-h100-fp4-vllm-tp2-2node.yaml --registry $PWD/blis-registry \
  --num-instances 4 --rate 100 --num-requests 500 \
  --trace-level decisions --summarize-trace

# A different deployment: pick another scenario (single-node TP2: up to 4 instances)
./blis run --scenarios examplescenarios \
  --scenario gpt-oss-120b-h100-fp4-vllm-tp2.yaml --registry $PWD/blis-registry \
  --num-instances 2 --rate 100 --num-requests 500

What's Next