Skip to content

Deep Dives

What We've Learned About Modeling LLM Latency

The last post covered how BLIS models the engine, data plane, and control plane to predict full-pipeline latency without touching a GPU. What it didn't answer is whether the predictions can be trusted.

That comes down to one number. Everything BLIS reports at the cluster level rests on its estimate of how long a single forward pass takes, so if that's off, nothing above it can be right. This post is about how we estimate it, how well it holds up, and where it falls short.

The headline, up front: fit once on H100, BLIS predicts held-out configurations (six models across three GPU types) at 6.7% median end-to-end error, roughly 200× faster than running them for real, and those predictions have already steered serving policies we later confirmed on a physical cluster: a better admission controller and soft reflective flow control for llm-d. The rest of this post is how we get there, and where it still falls short.

The Physics of High-Fidelity Distributed Inference Platform Simulation

Production LLM inference platforms are distributed systems where routing policies, admission control, autoscaling, and engine-level scheduling all interact to determine latencies and throughput. How do you explore how different policies and configurations affect these KPIs before deploying to production? Testing a new routing policy or autoscaling threshold on live traffic risks cascading bugs across the fleet, while building separate test environments burns GPU-hours and still cannot predict interactions between cluster-level policies and engine-level batch dynamics.

The answer is end-to-end simulation: model the entire distributed inference stack to explore how policies and configurations affect latencies and throughput for your workloads. What does it take to build a simulator accurate enough to guide these decisions? The challenge lies in capturing the right mechanisms. At the engine level, batches process together — all requests wait for the slowest operation to finish, so KV cache fills trigger preemptions and long prompts stall short decodes. At the cluster level, routing policies operate on stale cache state, admission control gates overload, and prefill/decode disaggregation trades utilization for latency. At the control plane, autoscalers react to lagged metrics, creating oscillations. When these couplings are not modeled, predictions diverge: a back-of-the-envelope model might predict 50ms time-to-first-token while production measures 200ms.