Skip to content

BLIS Extension Recipes

Step-by-step guides for extending BLIS. Each recipe lists the exact files to touch, the order, and examples to follow.

Routing, Admission and Flow Control

BLIS has no routing, admission or flow-control code of its own: every cluster run uses the llm-d router embedded in-process (Routing). New routing, admission or flow-control behavior belongs in llm-d-router, as a plugin or configuration, not in this repository.

A new llm-d plugin type becomes usable in BLIS once it is audited and added to the supported list in sim/llmd/support.go: read every clock read, goroutine, blocking wait, random draw and map iteration it makes, route the decision-relevant ones through the hosting runtime in the fork, check that every signal it consumes is produced by the simulated engines, and add a test in sim/llmd. Until then it is refused at startup with the reason in the unsupported list. See The llm-d router fork.

Adding New Instance Schedulers

To add a new instance-level scheduler (wait-queue ordering):

  1. Implement the InstanceScheduler interface in sim/scheduler.go (receives requests + clock only).
  2. Register in two places (both required): add the name to validSchedulers in sim/bundle.go, and add a case to NewScheduler in sim/scheduler.go. CLI error messages derive from ValidSchedulerNames().
  3. Add tests following BDD naming: TestMyScheduler_Scenario_Behavior. Test observable behavior, not internal structure.
  4. Update documentation: the relevant docs/ guide. Touch CLAUDE.md only if you are adding an operating rule an agent needs every session (see the charter at the top of CLAUDE.md) — it holds pointers, not per-feature detail.

Examples: - See PriorityFCFSScheduler and SJFScheduler in sim/scheduler.go.

Extending KV Cache Tiers

To add a new KV tier (e.g., NVMe offloading for 3-tier GPU+CPU+NVMe):

  1. Implement the KVStore interface in sim/kv/ (11 methods: allocate, get cached, release, capacity queries, metrics, SetClock, ConsumePendingTransferLatency)
  2. Compose existing tiers — e.g., wrap TieredKVCache (GPU+CPU) with NVMe logic, following the same delegation pattern
  3. Update NewKVStore factory in sim/kv/register.go to instantiate your tier based on KVCacheConfig fields (add new fields to KVCacheConfig in sim/config.go)
  4. Add CLI flags in cmd/root.go for new parameters (e.g., --kv-nvme-blocks) and wire them into the KVCacheConfig sub-config Tier transfer time is not computed here: each tier names a catalog device_class, and blis-latency-kernel prices the transfer (TierTime). A new device class is added to blis-catalog.
  5. Aggregate metrics — combine hit/miss/thrashing counters from all tiers; see TieredKVCache.CacheHitRate() for the 2-tier pattern
  6. Add behavioral tests in sim/kv/*_test.go
  7. Check-then-act allocation (no rollback) — KVCacheState.AllocateKVBlocks uses a pre-check gate: it computes the total blocks needed (new blocks + cached blocks leaving the free list) and compares against countFreeBlocks() before any state mutation. If insufficient, it returns false immediately with zero side effects. Post-pre-check popFreeBlock() == nil is a panic (INV-4 violation, structurally unreachable in single-threaded DES). This mirrors vLLM's kv_cache_manager.py:334-336 universal pre-check. If your tier adds mutations before delegating to gpu.AllocateKVBlocks(), ensure the inner pre-check sees the updated FreeBlockCnt (e.g., commitCachedBlocks calls removeFromFreeList which decrements FreeBlockCnt before the inner call).
  8. GetCachedBlocks is a pure query — it returns cached block IDs without side effects. CacheHits are counted by AllocateKVBlocks when cached blocks are committed to an allocation. The pre-check accounts for cached blocks with !InUse (on the free list) via the cachedFromFreeList budget, mirroring vLLM's num_evictable_blocks.

Examples: - See TieredKVCache in sim/kv/tiered.go for 2-tier GPU+CPU composition - See KVCacheState in sim/kv/cache.go for single-tier baseline (also implements KVStore) - See docs/plans/archive/pr12-architectural-predesign.md for the design decisions behind the tiered architecture

Adding New Trace Record Types

To add a new trace record type (e.g., ScaleRecord for autoscaling events):

  1. Define the record struct in sim/trace/record.go (pure data, no sim/ dependency)
  2. Add a slice field to SimulationTrace in sim/trace/trace.go (e.g., Scales []ScaleRecord)
  3. Add a recording method to SimulationTrace (e.g., RecordScale(ScaleRecord))
  4. Hook recording into the cluster pipeline (e.g. sim/cluster/gateway.go for router decisions, sim/cluster/cluster_event.go for scaling events; guard with if cs.trace != nil for zero-overhead default)
  5. Update Summarize() in sim/trace/summary.go to aggregate the new record type
  6. Add behavioral tests in sim/trace/*_test.go

Examples: - See AdmissionRecord in sim/trace/record.go for a simple record - See RoutingRecord with CandidateScore for a record with nested counterfactual data - See counterfactual() in sim/cluster/gateway.go for derived computation that lives in sim/cluster/ (not sim/trace/): it ranks the scores the router's primary scheduling profile gave every endpoint

Adding Fields to TraceV2 Format

To add a new field to TraceV2 CSV output (e.g., observability metadata like vllm_priority):

  1. Add field to TraceRecord in sim/workload/tracev2.go (place logically near related fields)
  2. Add field to RequestRecord in cmd/observe.go (if captured during observation)
  3. Capture the value in RealClient.Send() in cmd/observe.go (set field on RequestRecord)
  4. Wire through Recorder in cmd/observe.go: Update RecordRequest() to copy field from RequestRecord to TraceRecord
  5. Update ExportTraceV2 in sim/workload/tracev2.go:
  6. For optional columns: scan records to determine if column is needed, conditionally add to header, conditionally write values
  7. For always-present columns: add to traceV2Columns slice and write in row construction
  8. Update LoadTraceV2/parseTraceRecord in sim/workload/tracev2.go:
  9. For optional columns: detect column presence from CSV header, apply offset to subsequent column indices
  10. For always-present columns: parse at fixed index, update all downstream indices
  11. Enforce simulation isolation (if observability-only): Verify LoadTraceV2Requests and LoadTraceV2SessionBlueprints do NOT read the field into sim.Request — add tests for this
  12. Add behavioral tests covering round-trip, conditional logic, and simulation isolation

Column position guidelines: - Observability metadata: place near related fields (e.g., vllm_priority after slo_class) - Timing data: group with other timestamps - Optional columns: prefer conditional inclusion over always-writing zero values

Examples: - See PR #1220 (VLLMPriority) — optional column, simulation-isolated, conditional on SLOClass presence - See FinishReason/ErrorMessage fields — always-present optional strings (empty when not set) - See ServerInputTokens field — observability metadata that differs from InputTokens

Extending Latency Pricing

Latency pricing is not extended in this repository. BLIS takes every step time, memory figure, P/D transfer time, offload tier transfer time and host overhead from blis-latency-kernel (adapter sim/kernelmodel); there is no LatencyModel factory or backend flag to add a case to.

  • A new pricing term, engine feature or model shape → change blis-latency-kernel, release it, and bump the pinned module version in go.mod.
  • New or refitted coefficients → publish them in blis-registry and name the coefficient sets in the scenario's coefficients: list.
  • A new model, chip, fabric or storage device → add it to blis-catalog.
  • A new scenario or engine-setting field → change the format in blis-schemas first.

On the BLIS side, a new kernel capability usually needs only the adapter in sim/kernelmodel and the scenario-to-SimConfig wiring in cmd/. See Latency models.

Adding New Batch Formation Strategies

To add a new batch formation strategy (e.g., disaggregated prefill/decode, speculative decoding, continuous batching without preemption):

  1. Implement the BatchFormation interface in sim/batch_formation.go (or a new file for complex strategies) — 1 method:
  2. FormBatch(ctx BatchContext) BatchResult — compose the running batch for the next step
  3. The implementation receives BatchContext with: RunningBatch, WaitQ, KVCache, token budget, batch size limit, chunked prefill threshold, MaxModelLen (0 = unlimited; implementations should clamp token scheduling to maxModelLen-1-ProgressIndex when > 0), simulation time, step count, and ComputedTokens map
  4. The implementation MUST update ctx.ComputedTokens[req.ID] for each request that receives new tokens (Phase 2 of Step() reads this map to advance ProgressIndex)
  5. The implementation may mutate WaitQ (dequeue/prepend) and KVCache (allocate/release) during batch formation
  6. The implementation MUST NOT schedule events or record metrics — return decisions in BatchResult, the Simulator applies them
  7. Register in NewBatchFormation factory in sim/batch_formation.go: add a selection branch. The factory signature is NewBatchFormation(preemptionPolicy string). For a new batch formation strategy (not just a preemption variant), add a BatchFormation string field to PolicyConfig and a selection branch in NewBatchFormation
  8. Add behavioral tests — token budget enforcement, batch size limits, KV conservation, preemption behavior (if applicable), FCFS ordering
  9. Extension friction: 2 touch points (implementation + factory registration)

Note: Currently only VLLMBatchFormation exists (with configurable preemption via --preemption-policy fcfs|priority). Adding a second batch formation strategy will also require: (a) a BatchFormation string field in PolicyConfig or BatchConfig (in sim/config.go), (b) a CLI flag in cmd/root.go, (c) validation in sim/bundle.go, (d) selection logic in NewBatchFormation. For adding a new preemption variant (not a new strategy), add a constant to batch_formation.go, a case to the switch in preemptForTokens, and an entry in validPreemptionPolicies in bundle.go.

Examples: - See VLLMBatchFormation in sim/batch_formation.go for the vLLM FCFS + chunked-prefill + preemption strategy - See preemptForTokens for the KV allocation + eviction loop pattern

Adding New Quantization Formats

Quantization is stated per pool in the scenario (engine.quantization, engine.cache_dtype) and priced by blis-latency-kernel. A new format is added there (and to blis-schemas if the field's allowed values change), not in this repository.

Adding a New Engine

To add a new autoscaler Engine implementation (e.g., a cost-minimizing MIP solver or an OpenEvolve-evolved policy):

  1. Implement the Engine interface in sim/cluster/engine.go (or a new file for complex engines) — 1 method:
  2. Optimize(results []AnalyzerResult, inventory GPUInventory) []ScaleDecision
  3. Inputs: one AnalyzerResult per model (supply/demand signals + per-variant breakdown) and a GPUInventory snapshot (free GPU slots per variant, pre-subtracted for Loading/Active/Draining instances).
  4. Output: at most one ScaleDecision per model per call. Delta > 0 = add replicas; Delta < 0 = remove replicas.

  5. Must not read RouterState or ModelSignals directly — only AnalyzerResult.

  6. Reuse the shared helpers when appropriate:

  7. scaleUpN(requiredCapacity float64, vcs []VariantCapacity) int — exact replica count via ceil(requiredCapacity / prc); fallback to 1 when prc == 0.
  8. scaleDownN(spareCapacity float64, vc VariantCapacity) int — exact replica count via floor(spareCapacity / prc), clamped to [1, ReplicaCount]; fallback to 1.
  9. sortedByAscCost, sortedByDescCost — deterministic variant sort (R2: copy-and-sort to avoid mutating caller data).

  10. Pass only the selected variant to scaleUpN (vcs[0:1], not the full slice) so perReplicaCapacityForScaleUp uses the chosen variant's own capacity — not a different active variant's.

  11. Wire the engine in sim/cluster/cluster.go — search for &UnlimitedEngine{} in the autoscaler pipeline construction and replace it with your engine (or add a config field + factory once multi-engine selection is implemented in the follow-up wiring PR).

  12. Add behavioral tests in sim/cluster/engine_test.go:

  13. Scale-up: correct Delta, correct Variant, inventory check pass/fail.
  14. Scale-down: correct Delta from spare capacity, ReplicaCount clamp.
  15. Edge cases: zero RequiredCapacity, zero SpareCapacity, all variants inactive, TPDegree > 1.

  16. Cross-model: multiple AnalyzerResult entries — verify decisions for each model independently.

  17. Extension friction: 2 touch points (implementation + wiring via &UnlimitedEngine{} in cluster.go). For research variants behind a config field, 4 touch points: implementation + AutoscalerConfig field + factory function + CLI flag.

Examples: - See UnlimitedEngine in sim/cluster/engine.go for a simple inventory-ignoring engine - See GreedyEngine in sim/cluster/engine.go for an inventory-respecting greedy engine with exact-N sizing

Adding New Per-Request Metric Fields

To add a new field to per-request JSON output (appears in --metrics-path output):

  1. Add field to Request in sim/request.go (runtime state, zero-value safe). When constructing Request structs, use RequestState typed constants (StateQueued, StateRunning, StateCompleted) — never bare strings.
  2. Add field to RequestMetrics in sim/metrics_utils.go (JSON output struct, use omitempty for backward compatibility)
  3. Update NewRequestMetrics() constructor in sim/metrics_utils.go to propagate the new field from Request to RequestMetrics
  4. Set the field at the appropriate point (e.g., dispatch in sim/cluster/gateway.go for cluster-level, or completion for computed metrics)
  5. Add behavioral tests covering multi-instance, single-instance, and standalone boundaries

Examples: - See HandledBy (#181) — derived from the instance a request was dispatched to, zero-value outside the cluster pipeline (suppressed from JSON via omitempty) - See SLOClass/TenantID (PR10) — set during workload generation, propagated at injection