Skip to content

What is BLIS?

BLIS (Blackbox Inference Simulator) is a discrete-event simulator for LLM inference serving systems. It models multi-instance clusters routed by the embedded llm-d router, KV-cache dynamics, scheduling policies, and token generation — all without requiring real GPUs.

Why Simulate Inference Serving?

Deploying LLM inference at scale requires answering capacity planning questions that are expensive to answer with real hardware:

  • How many instances do I need to serve 1,000 requests/second at p99 TTFT < 200ms?
  • Which router configuration minimizes tail latency for my workload mix?
  • How much KV cache memory do I need before preemptions degrade throughput?
  • What happens at 2x traffic — does latency degrade gracefully or catastrophically?

Running these experiments on real GPUs costs thousands of dollars and takes days. BLIS answers them in seconds on a laptop.

Who Should Use BLIS

Audience Use Case
Capacity planners Determine instance counts, GPU memory, and parallelism configurations before procurement
Platform engineers Compare llm-d router configurations (scorer weights, admission, flow control) before deploying them
Researchers Run controlled experiments on scheduling, batching, and caching algorithms
Developers Validate new policies against existing ones before deploying to production

What BLIS Is Not

Setting expectations

  • Not a benchmark — BLIS simulates serving behavior, it does not generate real GPU load
  • Not primarily a load generator — BLIS focuses on simulation. Real-mode traffic generation against OpenAI-compatible endpoints is available but experimental. For production load testing, use tools like inference-perf or genai-perf

Key Features

See the Home page feature table for the capabilities and the guide for each.

Next Steps