mirror of
https://github.com/NandhaKishorM/laya.git
synced 2026-09-28 07:52:57 +08:00
Ben FordandClaude Opus 4.8
d531dc73f2
* feat: self-hostable server + Nix flake (Jev-compatible /v1/systemone) Laya's predict() output is already schema-compatible with TypeSafe Jev's decision API, so this adds only the HTTP surface a Jev client needs to talk to a self-hosted Laya: - laya/serve.py: FastAPI app exposing POST /v1/systemone (+ /health), calling Router.predict, with optional LAYA_API_KEY bearer auth. Config via env (LAYA_HOST/PORT/DEVICE/PRELOAD/MODELS/AUTO_TASK). Heavy imports are deferred so `import laya.serve` stays GPU- and web-dep-free. - laya-serve console script + laya[serve] extra (fastapi, uvicorn). - flake.nix: overlay building laya from torch-bin (prebuilt CUDA), a laya-serve runner, a dev shell, and nixosModules.default. - nix/laya-serve.nix: hardened DynamicUser systemd service (services.laya-serve) with CUDA device access and a HF weight cache. - tests/test_serve.py: shim tests (injected router, real FastAPI TestClient). Verified end-to-end on an RTX 3090: all three checkpoints preload on CUDA, English/Hindi requests route correctly and return Jev-shaped payloads, 6/6 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe * fix(nix): build the service package from the host pkgs, not an overlay Hosts that inject `pkgs` via specialArgs (read-only nixpkgs) ignore module-level `nixpkgs.overlays`, so `pkgs.laya-serve` was missing and the NixOS toplevel failed to evaluate. Factor the package into nix/package.nix and have services.laya-serve.package default to `(pkgs.callPackage ./package.nix {}).laya-serve`, built from the host's own pkgs. The flake overlay/packages now reuse the same file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe * feat(serve): LAYA_THREADS knob + services.laya-serve.threads for CPU tuning Cap torch intra-op threads for CPU inference. serve.py reads LAYA_THREADS and calls torch.set_num_threads (torch imported only when a limit is set); the NixOS module exposes `threads` and wires it to LAYA_THREADS + OMP_NUM_THREADS. Motivated by a thread sweep on a 32-core/64-thread EPYC: oversubscribing the logical core count is a ~4x latency regression, and single-request latency is often best below the physical core count (e.g. 8-16), while batched throughput peaks at the physical count. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe * chore(nix): commit flake.lock pinning nixpkgs to ccad53c Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
62 lines
1.5 KiB
JSON
62 lines
1.5 KiB
JSON
{
|
|
"nodes": {
|
|
"flake-utils": {
|
|
"inputs": {
|
|
"systems": "systems"
|
|
},
|
|
"locked": {
|
|
"lastModified": 1731533236,
|
|
"narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
|
|
"owner": "numtide",
|
|
"repo": "flake-utils",
|
|
"rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"owner": "numtide",
|
|
"repo": "flake-utils",
|
|
"type": "github"
|
|
}
|
|
},
|
|
"nixpkgs": {
|
|
"locked": {
|
|
"lastModified": 1766166645,
|
|
"narHash": "sha256-7urOsghhTfOv2d7+2il711ej2BY5LlFHlOA7IOCD4OY=",
|
|
"owner": "NixOS",
|
|
"repo": "nixpkgs",
|
|
"rev": "ccad53cd79cf4cf3bc338805d007d68565e75bda",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"owner": "NixOS",
|
|
"repo": "nixpkgs",
|
|
"rev": "ccad53cd79cf4cf3bc338805d007d68565e75bda",
|
|
"type": "github"
|
|
}
|
|
},
|
|
"root": {
|
|
"inputs": {
|
|
"flake-utils": "flake-utils",
|
|
"nixpkgs": "nixpkgs"
|
|
}
|
|
},
|
|
"systems": {
|
|
"locked": {
|
|
"lastModified": 1681028828,
|
|
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
|
|
"owner": "nix-systems",
|
|
"repo": "default",
|
|
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"owner": "nix-systems",
|
|
"repo": "default",
|
|
"type": "github"
|
|
}
|
|
}
|
|
},
|
|
"root": "root",
|
|
"version": 7
|
|
}
|