Validate documents from Go¶

Load each document with its loader, put the documents in a bundle, and call Validate. The report keeps the field layer apart from the rule layer and says which engine release's rules ran.

go get github.com/inference-sim/blis-schemas@latest

The snippets on this page are the package's example tests. They run under go test, and the output shown is checked against what they print.

Load, then validate¶

s := must(blisschemas.LoadScenario("docs/examples/scenario.yaml"))
d := must(blisschemas.LoadDeployment("docs/examples/deployment.yaml"))
g := must(blisschemas.LoadModelGraph("testdata/models/llama-3.1-8b-instruct/graph.yaml"))
c := must(blisschemas.LoadChip("testdata/hardware/h100.yaml"))

rep := blisschemas.Validate(blisschemas.Bundle{
    Scenario: s, Deployment: d, Model: g, Chip: c,
})
for _, p := range rep.Problems() {
    fmt.Println(p)
}
fmt.Println("ok:", rep.OK(), "rules applied:", rep.RulesApplied)

Output

ok: true rules applied: 0.29.0

must stands for whatever error handling the caller uses. Loading and validating are separate calls. A load error means the file is not a document of that kind at all: it does not parse, or it has a key the type does not declare. A validation problem means the document parsed and says something wrong. They need different fixes, so they arrive through different channels.

Loaders¶

Loader	Reads	The document's name comes from
`LoadScenario`	a scenario	its `name` key
`LoadDeployment`	a deployment	its `name` key
`LoadModelGraph`	`models/<name>/graph.yaml`	its `name` key
`LoadModelIdentity`	`models/<name>/model.yaml`	its `name` key
`LoadChip`	`hardware/<name>.yaml`	the file name
`LoadFabric`	`networks/<name>.yaml`	the file name
`LoadStorageDevices`	`devices/storage.yaml`	each class's key; returned sorted by name
`LoadWorkload`	`workloads/<name>.yaml`	the file name
`LoadCoefficientSet`	a coefficient set	its `name` key (a scenario refers to it by file name)
`LoadEvaluationRun`	an evaluation run	its `name` key
`LoadMeasuredRun`	`measurements/<license_ref>/<name>.yaml`	its `name` key
`LoadMeasurementSources`	`measurements/sources.yaml`	each source's `id`

Each decodes one YAML document and rejects keys its type does not declare. The loaders do not compare a name key with the directory or file it sits in; validate-catalog does that for model entries and measured runs. A measured run is validated with ValidateWithSources(registry), which also computes its verdict; it is not part of a bundle.

The bundle¶

A Bundle holds any subset of documents: a chip alone, or everything a run needs. Every document present is checked on its own. A deployment is also checked against its scenario's cluster, which is why a bundle with a deployment and no scenario is reported as a problem rather than checked by halves. Rules run when the bundle has a scenario and the field layer passed; they read the deployment and the model graph when those are present.

Read the report¶

pool := &d.Pools[0]
pool.Parallel.TP = 8  // 8 x 4 data-parallel ranks needs 32 GPUs
pool.Parallel.DCP = 3 // does not divide tp
pool.Engine.GPUMemoryUtilization = 1.5

rep := blisschemas.Validate(blisschemas.Bundle{Scenario: s, Deployment: d})
for _, p := range rep.Problems() {
    fmt.Println(p)
}

Output

error: deployment.pools[0].engine.gpu_memory_utilization: must lie in [0, 1] (0 means unset), got 1.5
error: deployment.pools[0].parallel: needs 32 GPUs (pp 1 x tp 8 x pcp 1 x dp 4, one per rank) but the pool's 1 node(s) of 8 GPUs provide 8
error: deployment.pools[0].parallel.dcp: decode-context parallelism reuses the tensor-parallel ranks when prefill-context parallelism is off, so tp 8 must be divisible by dcp 3

All three mistakes are reported at once, sorted by path. Each Problem has Path, Message, Severity (error or warning) and Rule, which is empty for a field problem. rep.OK() is false here because errors are present; warnings alone leave it true.

To treat the layers differently, read rep.Field and rep.Rule. Each has OK(), All() for every problem and Errors() for errors only.

d.Pools[0].Engine.CacheDType = "fp8_e4m3fn" // not a name vLLM 0.29.0 accepts

rep := blisschemas.Validate(blisschemas.Bundle{Scenario: s, Deployment: d})
fmt.Println("field layer ok:", rep.Field.OK())
for _, p := range rep.Rule.All() {
    fmt.Println(p)
}

Output

field layer ok: true
error [enum-values-known]: : pools[0].engine.cache_dtype: "fp8_e4m3fn" is not accepted by engine 0.29.0

A rule problem carries its rule's name in brackets. Rules state the location inside the message, so the path between the two colons is empty.

If the field layer passed, the bundle has a scenario, and rep.RulesApplied is still empty, then no rules pack matched the scenario's engine_version. The rule layer then holds a single warning saying so.

Resolve names against a catalog¶

Validate does not read the disk, so it cannot tell whether cluster.hardware names a chip that exists. ValidateAgainstCatalog can, given a catalog root and a registry root:

s := must(blisschemas.LoadScenario("docs/examples/scenario.yaml"))
s.Cluster.Hardware = "nvidia-h100" // the catalog calls it h100

rep := blisschemas.ValidateAgainstCatalog(
    blisschemas.Bundle{Scenario: s}, "testdata", registry)
for _, p := range rep.Problems() {
    fmt.Println(p)
}

Output, with the pinned catalog in testdata/

error: references.hardware: "nvidia-h100": looked for hardware/nvidia-h100.yaml; the catalog has [a100-80 a100-sxm b200 b300 gb200-nvl72 h100 h200 l40s]

It checks only that each name resolves. To validate what the names point to, load those files and pass them to Validate.

Ask a rules pack¶

The rules package exposes the registered packs. Besides the values a release accepts, a pack answers questions about what the release does:

pack := rules.Lookup("0.29.0") // nil if no pack is registered
rules.Versions()               // every registered version

// Does the engine make the MoE input sequence-parallel, replacing an all-reduce with a
// reduce-scatter and all-gather pair?
pack.SequenceParallelMoE(all2allBackend, expertParallel, tp, dp)
// Does the custom all-reduce GPU kernel support this tensor-parallel width?
pack.CustomAllReduceSupportsWidth(tp)
// Does dual-batch overlap split a batch of this size?
pack.DBOEngages(enabled, totalTokens, uniformDecode)
// How many SMs does a concurrent offload fetch of this page size take?
pack.TritonFetchWithholdsSMs(pageBytes)

Packs are registered when the root package blisschemas is imported. A program that imports rules but not the root package has no packs, so Lookup returns nil.

The full API is on pkg.go.dev.