Skip to content

blis-latency-kernel

blis-latency-kernel is the cost model of BLIS. Given a model, a GPU, a set of coefficients for that GPU and a deployment, it answers the two questions a serving simulator asks: how long one forward pass over a batch of requests takes, and how much GPU memory the deployment holds.

It is a Go library. A simulator builds one kernel per pool, a set of GPUs running one engine configuration (a disaggregated deployment has a prefill pool and a decode pool), then calls it once per simulated step. The kernel prices the step from first principles: it counts the arithmetic, the bytes read from memory and the bytes moved between GPUs, and turns each count into time with coefficients for the hardware, most of them measured. It holds no data of its own; the model and the hardware come from blis-catalog, and the coefficients from blis-registry.

How a kernel is built once and then priced every step On the left, the inputs: a scenario and its deployment, a model graph and a chip from blis-catalog, and coefficient sets from blis-registry. They go into New, which runs once and returns a Kernel. A simulator then calls StepTime on that kernel once per simulated step, passing a batch and receiving a StepEstimate. The memory methods are answered from the same kernel. inputs, read once scenario + deployment what runs, and how it is laid out model graph, chip from blis-catalog coefficient sets from blis-registry New validate, fill in defaults, precompute Kernel immutable, safe to share every simulated step StepTime(batch) → Overlap, NoOverlap, Bottleneck, PerResource when sizing memory FixedBytes() SequenceVariableBytes(n) runs once; refuses a deployment the engine would not start
A kernel's life. New reads the inputs once and either refuses the deployment or returns a kernel. The kernel holds no mutable state, so a simulator can call it from any goroutine at any simulated instant and get the same answer for the same batch.

A first example

The deployment is DeepSeek-V3 in fp8 on eight H200 GPUs at tensor-parallel width 8.1 The batch is 32 requests, each decoding one token against a 4,096-token context.

// DeepSeek-V3, served in fp8 on eight H200s at tensor-parallel width 8.
const scenario = "aisimulate/deepseek-v3-h200-fp8-sglang-tp8.yaml"
k, err := latencykernel.Open(scenario, repos)
if err != nil {
    panic(err)
}

// 32 requests, each decoding one token against a context of 4,096 tokens.
batch := kernel.Batch{DecodeThreshold: 8}
for range 32 {
    batch.Reqs = append(batch.Reqs, kernel.ReqShape{
        Scheduled: 1, Computed: 4095, PromptLen: 4096,
    })
}

e := k.StepTime(batch)
fmt.Printf("between %.1f and %.1f ms, bottleneck %s\n",
    ms(e.Overlap), ms(e.NoOverlap), e.Bottleneck)
for _, r := range []kernel.Resource{kernel.ResourceSM, kernel.ResourceHBM,
    kernel.ResourceNVLink, kernel.ResourceHost} {
    fmt.Printf("%-7s %5.1f ms\n", r, ms(e.PerResource[r]))
}

between 24.1 and 25.7 ms, bottleneck hbm
sm        4.5 ms
hbm      16.5 ms
nvlink    1.2 ms
host      3.5 ms

The first number assumes the GPU's resources (arithmetic, memory, interconnect and CPU) work at the same time within each layer; the second assumes they take turns. For one number, use the second, NoOverlap; Pricing a step says why. NIC, the network between nodes, is absent because no collective here leaves the node. The output above is checked by go test, so it is what this version computes.

Where to go next

To use the kernel, start with Price a batch from Go, then Use the kernel from a simulator.

To understand what it computes, read the Concepts in order. They explain how a step is priced, how memory is counted, and what each kind of parallelism changes, without assuming you have read the code.

To judge how far to trust it, read Accuracy and the known divergences: the places where it knowingly prices something differently from the engine it models.

To change it, read Contributing.

What it is not

It is not a simulator. It does not queue requests, form batches or decide when a request is admitted; those depend on time, and the kernel is a pure function of its inputs. It prices whatever batch it is given.

It is not a lookup table. It evaluates closed-form laws, so it can price a model or a shape that was never measured. The same property bounds its accuracy against a table on shapes that were.

It does not know model names. It reads a model as a graph of a few kinds of operation, and dispatches on those. A new model reaches it as data, with no change to its code.

Versions

This kernel prices the engine behavior of vLLM v0.31.0: which kernels run, how work is split across GPUs, what an unstated setting defaults to. The exact pins of vLLM and of every other input are on Sources of truth. These pages are published per release of the kernel; the selector at the top of the page switches between them.

  1. The scenario's file name includes sglang because it comes from NVIDIA's AISimulate measurements, which ran that engine. The kernel prices vLLM v0.31.0's behavior for whatever deployment it is given. ↩