laya/research

Research

Benchmark harnesses and raw results for the Laya checkpoints. This branch is the evidence behind the numbers quoted in the main README — nothing here is imported by the laya package.

Evaluation

• eval/: the independent per-language MASSIVE harness (laya_eval.py), the per-case report behind the published numbers, plus the metamorphic and presentation checks.
• evals/: the labelled datasets, thresholds and regression gate consumed by laya.evals / laya-evals and the scheduled evals workflow.

Community diagnostics

• Chinese workplace decisions (Feishu-style) — 64 synthetic scenarios, paired recorded Laya/Jev responses, English/Chinese cards, and a model-free audit. 中文入口. Start with python research/benchmarks/feishu_zh/audit.py; no downloads or API keys required. This is a contributed historical snapshot, separate from the upstream sweeps below.
• Chinese short-command routing — 18 frozen Chinese voice commands, a seven-rung ablation of the documented prompt guidance on both the six-way choice path and the four-question noul path, and every per-case decision archived. 中文入口. Start with python research/benchmarks/zh_short_commands/audit.py; the audit needs no downloads and the archive records which weights produced the numbers.

Scripts

file	what it does
scripts/laya_benchmark_colab.ipynb	the full head-to-head on a Colab T4: typed-decisions, MASSIVE intent + scenario (14 languages), XNLI (15), English suites, latency, option-order robustness, calibration repair. Writes one JSON.
scripts/build_benchmark_nb.py	generator for that notebook (edit here, not the .ipynb)
scripts/bench_local.py	CPU sweep: MASSIVE intent across all 51 languages, plus typed-decisions on all three checkpoints
scripts/bench_apps.py	the six application workflows (support triage, email + phishing, guardrails, RAG relevance, moderation, model routing) plus the datasets where public Jev numbers exist
scripts/bench_latency.py	inference speed including what routing costs: detection overhead, hot path, cold-swap, mixed-language throughput at several max_loaded settings
scripts/bench_length_batching.py	compare upstream contiguous batches with optional length sorting on synthetic tickets, including output consistency and optional fresh-process memory profiles (psutil required for memory mode)
scripts/make_plots.py	renders assets/laya_benchmark.png from the result JSONs
scripts/bench_long_context.py	laya-multilingual on long documents: 20 support requests in 8 languages, each placed after 0 to 7,000 tokens of unrelated text, scored at the default limit and at max_len=8192
scripts/plot_long_context.py	renders assets/long_context_8192.png from results/long_context_multilingual.json

Everything runs with USE_TF=0 — transformers probes for TensorFlow at import, and when TF is installed its abseil runtime can deadlock model construction on macOS/Python 3.9.

Running the harnesses

The checkpoints come from LAYA_MODELS (default ~/laya_models, a directory holding laya/, laya-multilingual/ and laya-typed-decisions/), so a checkout can point them at a local models/ copy:

# from the repository root
LAYA_MODELS=$PWD/models python research/scripts/bench_local.py --langs 10 --per-lang 60 --skip-b
LAYA_MODELS=$PWD/models BENCH_N=80 python research/scripts/bench_apps.py
LAYA_MODELS=$PWD/models python research/scripts/bench_latency.py

option	effect
bench_local.py --langs N	caps the language list (sorted, so 10 is the first ten codes; 0 = all 51)
bench_local.py --per-lang N	cases per language (default 120)
bench_local.py --skip-a / --skip-b	run only the typed-decisions half, or only the MASSIVE half
bench_apps.py with BENCH_N	cases per application suite (default 400)

bench_local.py and bench_apps.py write research/*_benchmark_results.json, which is gitignored — the curated runs live in results/. Set HF_HOME inside the checkout to keep the dataset cache local.

bench_local.py and bench_latency.py take the model root from LAYA_MODELS rather than assuming ~/laya_models, so the harnesses run against a checkout that keeps its weights under models/.

Length batching

Local CPU measurements compare the original predict_batch method at 1e28ac20c0896b1c37a744cd11f740eb98f8b178 with sort_by_length=True. On 10,000 synthetic English support tickets, the multilingual checkpoint took 1775.71 s before and 825.09 s after (2.15x), including tokenization, collation, inference and decoding. There were zero choice/action decision changes, zero usage mismatches, and a maximum returned numeric difference of 0.0001. This is a throughput workload, not a labeled accuracy test.

The machine was a Ryzen 9 5950X with 64 GiB RAM, Windows 11, CPU float32 and 16 Torch threads. An unrelated adaptation experiment was running on the GPU. Small tests alternate execution order over repeated rounds; the 1,000 and 10,000 input tests each have one full timing pair. The similar-length control showed no regression. GPU, MPS, compiled and TileLang performance were not measured. Results depend on input lengths, checkpoint, batch size and hardware.

To reproduce, use a fresh Python 3.12 environment and the pinned model snapshot:

python -m pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cpu
python -m pip install transformers==4.57.1 huggingface_hub==0.36.2 numpy==2.3.5 psutil==7.2.2
python -c "from huggingface_hub import snapshot_download; print(snapshot_download('convaiinnovations/laya', revision='aa8c91ca088ec597df95a0d1c76b3063cb2ae5e8', allow_patterns=['multilingual/*']) + '/multilingual')"

Replace MODEL_DIR below with the printed checkpoint directory. Start with --count 16 --rounds 2 --questions 3, then increase the count. The full CPU comparison takes tens of minutes:

python research/scripts/bench_length_batching.py MODEL_DIR --count 10000 --batch-size 8 --threads 16 --rounds 1 --output length-10000.json

Memory measurements run each mode in a separate process after model warm-up. They sample process RSS every 10 ms during 64-input inference, excluding model-load transients; short allocation peaks can be missed. They do not measure the peak memory of the 10,000-input run:

python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-size 8 --threads 16 --memory-mode original --output memory-original.json
python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-size 8 --threads 16 --memory-mode grouped --output memory-grouped.json

Results

file	contents
results/t4_colab_benchmark.json	17,416 questions on one T4, both checkpoints, identical questions per model
results/long_context_multilingual.json	the long-document run behind assets/long_context_8192.png: every prediction, with device and library versions
results/cpu_51_language_sweep.json	51 languages x 2 checkpoints, MASSIVE intent, 20 options
results/cpu_51_language_sweep_clamped.json	the same 51 languages and 5,100 cases re-run after the temperature clamp, raw temperatures and served temperatures side by side (#208)

Headline findings

Routing takes Laya from 23 to 45 of 51 languages. On MASSIVE intent (20 options, random = 0.050) the English checkpoint macro-averages 0.227 and clears 3x random on 23 of 51 languages; the multilingual checkpoint reaches 0.366 and clears it on 45.

The English checkpoint's confidence gives no warning when it cannot read the input. Khmer: 0.000 accuracy at 0.952 mean confidence. Macro ECE 0.733 across 51 languages, with mean confidence never dropping below 0.885 at any accuracy level. This is why routing has to happen before the forward pass — confidence gating cannot catch it.

Both checkpoints ship over-confident. Refitting one temperature per (question type, option count) on held-out data moves mean ECE 0.466 -> 0.081 (laya) and 0.314 -> 0.106 (laya-multilingual, which ships with no fitted temperatures at all).

The base checkpoints are near chance on typed-decisions zero-shot — 0.362 and 0.352 against a 0.318 random baseline and a 0.461 majority-class baseline. The published 0.766 belongs to the checkpoint fine-tuned on that benchmark's own training split.

Speed. 32.8 ms for one question and 7.2 ms/question at batch 10 on a T4; 103–332 questions/s batched.

On comparisons with Jev

For the original upstream suites listed above, Jev was not run directly. Their Jev figures are third-party published, with different sample sizes and prompts. The separate community diagnostic linked above includes paired API responses and documents its own limitations:

• AbdelStark/jev-benchmarks — AG News 0.910, Banking77 0.870, DAIR Emotion 0.480 (Brier 0.846, NLL 5.588, zero probability on the true label for 16% of examples)
• nibzard/decision-model-benchmark — ECE 0.246 (worst in that study), banking77 0.763, option-order flip rate 13%, latency 264–276 ms p50

Treat those as indicative, not a controlled head-to-head.