Skip to content

Installation

Prerequisites

  • Go 1.27.2+ (the embedded llm-d router requires it) — Download Go
  • Git — for cloning the repository

Build from Source

git clone https://github.com/inference-sim/inference-sim.git
cd inference-sim
go build -o blis main.go

Environment Setup

BLIS reads model architecture configs from a local catalog checkout and makes no HuggingFace requests, so running a simulation needs no token and no network access. Set HF_TOKEN only when downloading a new model's config.json by hand to add a catalog entry, e.g. for a gated model such as Llama-2:

export HF_TOKEN=your_token_here

Public models (e.g., Qwen3) work without a token. See HuggingFace access tokens to create a token.

Air-gapped / offline environments

Nothing to do — BLIS never reaches the network to run a simulation. Every model's config.json comes from your local catalog checkout, and a model with no catalog entry is refused rather than fetched.

To simulate a model that is not yet catalogued, obtain its config.json on a machine with internet access, then commit it at <catalog>/models/<model>/config.json — where <catalog> is whatever directory you point --catalog / BLIS_CATALOG at.

Locate the Model Catalog

blis run and blis replay must be told where the model catalog is — there is no default and no search path, so a run with neither --catalog nor BLIS_CATALOG is refused naming both forms. Clone the authoritative blis-catalog repository at the pinned release tag (see Catalog compatibility below) and point BLIS at the clone root:

git clone --branch 0.3.0 --depth 1 https://github.com/inference-sim/blis-catalog.git
export BLIS_CATALOG=$PWD/blis-catalog   # or pass --catalog on every command

Catalog compatibility

blis-catalog versions independently of BLIS and keeps releasing, so the documented clone names an explicit tag rather than whatever main happens to be. This is the one place the compatible version is stated; every other page's clone command pins the same tag.

Compatible blis-catalog release: 0.3.0 — release notes.

Why pin at all: BLIS parses every catalog file strictly (KnownFields(true)), and strict parsing is one-way — an added or renamed key in a future catalog schema is a hard load error, not a silently ignored field. An unpinned clone therefore lets a catalog release turn a blis run that worked yesterday into a startup failure with no change to BLIS itself. Pinning also makes results attributable: two people running the documented commands months apart read the same catalog. (blis run --metrics-path records the catalog's git revision and whether it was dirty — see Results; a --depth 1 tag clone still carries the revision, so that provenance is unaffected.)

BLIS prices every run with blis-latency-kernel, which additionally reads a blis-registry clone for its fitted coefficients (--registry, required). Compatible blis-registry release: v0.2.0, the release blis-latency-kernel v0.3.0 (the version go.mod pins, with blis-schemas v0.5.0, whose deployment router: block configures the llm-d router) is tested against; that pair is recorded in the kernel's own testdata/upstream.lock, and this repository's test suite holds its vendored catalog and registry to it.

git clone --branch v0.2.0 --depth 1 https://github.com/inference-sim/blis-registry.git

Upgrading to a newer catalog release is deliberately opt-in: clone the newer tag, run your workload against it, and if it works, bump the tag on the line above — the guard test then requires every documented clone command to match, so no page is left behind. Nothing stops you pointing --catalog at a main checkout or a scratch clone of your own; the pin is what the documentation promises, not a restriction the binary enforces.

Scenarios

Every blis run and blis replay names a scenario: a blis-schemas Scenario + Deployment YAML (two documents) that states the model, the cluster's hardware, fabric, nodes and GPUs per node, the coefficient sets that price it, the engine version, and each pool's role, parallelism and engine settings. --scenarios <dir> is any directory of scenario YAMLs and --scenario <file name> picks one in it. This repository ships ready-to-run scenarios in examplescenarios/ (speculative ones pin their acceptance, so no --speculative-* flag is needed): deepseek-v4.1-flash-h100-tp8-dspark5.yaml (the quickstart), an MoE gpt-oss-120b-h100-fp4-vllm-tp2.yaml, a deepseek-v4-pro-b300-fp4-vllm-tp4-mtp2.yaml, a P/D glm-5-h200-3p1d-ib.yaml, and a multi-node kimi-k3-h200-tp8dp4ep32-balanced.yaml. See examplescenarios/README.md for details.

Verify the Build

./blis run --scenarios examplescenarios \
  --scenario deepseek-v4.1-flash-h100-tp8-dspark5.yaml \
  --registry $PWD/blis-registry --catalog $PWD/blis-catalog \
  --num-requests 10

blis carries the repository's defaults.yaml (the LoRA cost constants) compiled in, so it runs from any directory. A defaults.yaml in the working directory, or --defaults-filepath <file>, takes precedence over the compiled-in copy.

You should see JSON output on stdout containing fields like ttft_mean_ms, e2e_mean_ms, and responses_per_sec. This confirms BLIS is working correctly.

Optional: Local Documentation

To preview the documentation site locally:

pip install mkdocs-material==9.7.3
mkdocs serve

Then open http://localhost:8000.

Optional: Linter

For contributors, install the linter used in CI:

go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.14.0
golangci-lint run ./...

What's Next