laya/docs/evals.md

Evaluation harness

laya.evals turns a labelled dataset into a repeatable score, and a baseline into a pass/fail gate, so a quality change is a reviewable diff instead of a hand-check.

The metric math and the dataset parser are pure Python plus numpy and never import torch, so they run with no weights. Running a dataset against a checkpoint needs the checkpoint and takes its normal load time.

Quickstart

# check the format without a model
laya-evals validate research/evals/fixture.jsonl

# score a labelled set on one checkpoint, with thresholds and a baseline
laya-evals run data.jsonl --model english --device cpu \
    --min-accuracy 0.8 --max-ece 0.05 --score-within 0.25 --slice language \
    --json report.json --markdown report.md

# compare a saved report to a baseline
laya-evals compare report.json --baseline baseline.json --tolerance choice_accuracy=0.02

laya eval ... is the same thing through the main CLI, so laya eval validate data.jsonl works too.

Exit codes: 0 on success, 1 when a threshold or a baseline tolerance fails, 2 on a usage error. run prints the overall metrics and any requested slices to stdout, and writes the full report and a Markdown summary when --json / --markdown are given.

Evaluating an ONNX export

run --onnx PATH scores an exported ONNX model through ONNXAgent instead of the torch Router, so an ONNX deployment (including an INT8 copy from scripts/export_onnx.py --quantize) gets gated by the same thresholds and baselines as the torch path:

python scripts/export_onnx.py --model convaiinnovations/laya --output laya.onnx --quantize
laya-evals run data.jsonl --onnx laya.int8.onnx --max-ece 0.05

--model names the checkpoint the export came from — a Hub id or local path, not a Router short name like english, since there is no Router on this path (default convaiinnovations/laya). Its config and tokenizer are loaded from there. The agent serves one checkpoint, so a dataset row whose model field names a different one fails with a clear error rather than being silently answered by the wrong model; --device does not apply. --batch-size uses the agent's batch API when it has one and falls back to one call per state otherwise. The report's config block records the onnx path.

Measured on research/evals/fixture.jsonl (12 labelled rows, English checkpoint, CPU):

runner	choice_acc	noul_acc	score_mae	ece	mean_conf	p50 ms
torch Router	0.75	1.00	1.3418	0.1596	0.7304	116.8
--onnx fp32	0.75	1.00	1.3418	0.1596	0.7304	66.3
--onnx int8	0.75	1.00	1.3512	0.1658	0.7304	46.3

The fp32 export reproduces the torch numbers exactly, and the quantized copy moves score_mae by 0.009 and ece by 0.006 — the kind of drift compare --tolerance is meant to gate.

Dataset format

One JSON object per line (JSONL). Blank lines and lines starting with # are ignored.

field	required	meaning
state	yes	text, email, ticket or JSON document to decide on
questions	yes	a Laya question dict, exactly as Router.predict accepts
expected	yes	ground truth keyed by question id: a label for choice, a number for score, true/false for noul
tags	no	strings to slice by
language	no	a code to slice by
model	no	force a checkpoint for this row; --model overrides it. A row that forces nothing is labelled with whatever checkpoint the Router answered with

research/evals/dataset.template.jsonl has a commented example.

Metrics

Each metric is computed per answer where it applies and aggregated over the dataset:

metric	applies to	meaning
choice_accuracy	choice	fraction whose chosen label matches
noul_accuracy	noul	fraction whose boolean (probability >= 0.5) matches
score_mae	score	mean absolute error
score_within_<tol>	score	fraction within an absolute tolerance
ece	any answer with a confidence	expected calibration error, 15 bins, computed on answer["answer_confidence"], the calibrated probability Laya reports on every answer type
mean_confidence	any answer with a confidence	mean reported answer["answer_confidence"]
latency_p50_ms, latency_p95_ms	per request	wall time each request waited, informational -- see batching
cost_per_decision_p50_ms, cost_per_decision_p95_ms	per decision	a call's wall time divided by the rows it carried, informational

Add ScoreWithin(0.25) to the evaluator list for a tolerance metric; the default set is choice_accuracy, noul_accuracy, score_mae, mean_confidence, plus ece. From the CLI the same thing is one flag: laya-evals run data.jsonl --score-within 0.25 reports score_within_0.25 beside the defaults, and the flag repeats, so --score-within 0.25 --score-within 0.5 reports both.

A tolerance metric needs a score answer with a numeric label, so on a dataset without one it has no value: run names the metric it could not compute instead of publishing a silent zero, and a --min / --max gate naming that metric fails as missing. The tolerances a run was asked for are recorded in the report's config block, so a reviewed baseline says which columns it expects.

Batching and timing

--batch-size N scores up to N consecutive rows that share a checkpoint and a question schema in one call. Both timing metrics come from the same measurements and answer different questions: every row of a batch returns when the batch does, so its latency is the whole call, while its cost_per_decision is 1/N of it. Batching therefore raises latency_* and lowers cost_per_decision_* on an unchanged set of decisions, and --max latency_p50_ms=... asks whether requests were served fast, not whether the run was cheap. With no --batch-size the two agree.

compare ignores any *_ms metric unless a tolerance names it, so these never fail a baseline on timing noise. What the harness actually did -- the batch size asked for, the runner shape it resolved to, how many rows shared a call, and the largest chunk -- is recorded in the report's config.timing, because the flag alone does not say whether anything was batched. Those counters record the calls issued, not the calls that returned: with on_error=skip, a chunk whose call raised still counts in rows_grouped and max_chunk, next to its entries in config.errored. The two *_ms metrics count only the calls that returned, so a failed call never contributes a latency it did not measure.

Slices

compare and run report overall numbers and, for --slice language|model|qid|tag, the same metrics per slice value, so a regression in one language or one question is visible without reading the aggregate. The model slice holds the checkpoint that answered each row: the Router's own choice per request, or the runner's model for a runner that does not route.

Baseline and CI gate

• Keep the dataset, a baseline report (--json output you have reviewed), and the tolerances together, committed, so a change is a reviewable diff. --tolerance METRIC=VALUE is the maximum absolute drift allowed for that metric.
• laya-evals run ... --baseline baseline.json --tolerance ... exits non-zero on drift, so it drops into CI unchanged. laya.evals.EvalReport.compare and assert_regression expose the same logic for tests.

Two CI surfaces use this:

• a weight-free job in .github/workflows/ci.yml runs tests/test_evals.py and tests/test_evals_api.py, so metric math, dataset parsing and the CLI are covered on every PR without downloading a checkpoint;
• .github/workflows/evals.yml runs weekly, before a release and on demand: it evaluates the English checkpoint on the MASSIVE English suite and compares to research/results/eval_english_51_languages.json with the tolerances in research/evals/thresholds.json. It uploads the report as an artifact and does not block a PR.

The harness is deterministic for a fixed checkpoint revision, so a report is reproducible. run records the dataset, model and device, plus the timing facts of the run, in the report's config block, and revisions: the commit each checkpoint that answered was actually loaded from. --revision <SHA> pins that commit for every checkpoint the run loads, and --revision english=<SHA> pins one checkpoint (repeatable) — which is the form an auto-routing run wants, since the three checkpoints are three repositories and one commit cannot exist in all of them. Left unpinned, the run takes the checkpoint's default branch and the report still says which commit answered, so a baseline drift can be attributed to the weights or to the code. laya/revisions.py publishes reviewed commit SHAs in PINNED_REVISIONS for callers who want to opt in. With --onnx, only a bare --revision <SHA> applies, to the config and tokenizer download.

Adding the real labelled set

Drop a JSONL in research/evals/ and a reviewed baseline beside it, then point a workflow (or research/evals/check_regression.py) at both. The format is the same as the fixture; nothing in the harness knows about MASSIVE.