openclaw/docs/reference/test/runner-internals.md

summary, title, read_when

summary	title	read_when
Shared build locks, isolated test state and homes, and JSON report merging	Test runner internals	A run leaked state, retained a lock, or lost a report You need machine-readable results from a multi-project run

Shared test state and process helpers

TypeScript tooling uses the native typescript package. The pnpm tsgo lanes resolve its executable directly; syntax and semantic tools use its unstable API with explicit process lifetimes. The package root exports version metadata, while AST, filesystem, and compiler APIs live under typescript/unstable/*. Packaged declaration builds run the same native compiler with semantic checking and validate its complete source and package-manifest receipts before bundling. The tsdown wrapper defaults declaration builds to one configuration at a time, bounding native compiler processes alongside the existing Node heap budget. Runtime-only builds keep their parallelism, and an explicit --concurrency value retains tsdown's own behavior. Code Mode executes JavaScript directly and does not use this compiler; its TypeScript-style tool declarations are model-facing documentation.

build-all, standalone tsdown builds, tsgo, SDK declaration preparation, package-boundary checks, and dependent lint use checkout-local ownership at .artifacts/dist-artifacts.lock. Ownership spans cleanup, generation, cache restoration, and the checks consuming those outputs; independent checkouts remain independent. Competing commands print a waiting message and wait for a live owner without an acquisition deadline. Compiler and build execution timeouts are unchanged. Standalone tsgo runs serialize, including source-only checks; the core test shard runner retains its explicit concurrency inside one owner. Do not delete dist manually while these commands are running. An abrupt owner or nested wrapper exit, or unverified child cleanup, retains the lock. A missing or unverifiable owner PID, or a recorded child-cleanup failure, fails acquisition promptly without reclaiming anything. PID death does not prove detached descendants stopped. Before manually removing an abandoned lock directory, inspect its owner.json and verify all associated build, compiler, and lint processes, including detached descendants, have stopped; then retry the command.

Lint reports its final failure on stderr after child joins and artifact ownership have settled, including retained ownership when cleanup is uncertain. Standalone Oxlint and its shard CLI end with [oxlint] FAILED (exit N); pnpm lint owns the whole pipeline and ends with one [lint] FAILED (exit N) instead. Shard progress distinguishes passed from failed (exit N), and stdout remains available for machine-readable tool output. Successful runs have no failure trailer. Signals forwarded during child execution and shard timeouts fail the command; whole-host loss or SIGKILL of the reporting process can prevent a final line.

Local plugin lint consumes native SDK declarations in packages/plugin-sdk/dist. The dedicated package-boundary compiler also consumes seven plugin API trees in .artifacts/extension-package-boundary/plugins. Each declaration and compile owner validates its consumed source content, inherited config, selected compiler, and complete output inventory. Unrelated existing source or test edits retain cache hits. Resolution-topology changes invalidate conservatively, including new module candidates outside declared roots. Stale declarations get a full native emit after clearing their private input receipt; the successful emitted inventory then drives obsolete declaration pruning. Missing or tampered outputs invalidate the owner. The content records live under .artifacts/extension-package-boundary, outside packaged build cleanup. Private .inputs.json receipts contain normalized checkout-relative source and manifest paths instead of native .tsbuildinfo state. A warm run validates the records without emitting declarations.

Packaged declaration builds and package-boundary records accept only checkout-owned input realpaths, including compiler libraries, inherited config, dependency links, and package manifests. These paths share compileNativeProject, which uses the pinned native compiler's asynchronous API for checking and in-memory declaration emission. Its filesystem callbacks make candidates outside the checkout appear missing before native resolution can read them. Source and package-manifest reads are captured directly; admission does not depend on parsing resolution traces. The compiler version is pinned in package.json because this API is unstable. Configuration and requested semantic checks run before emission. Declaration errors come from the in-memory emit result, avoiding a separate declaration transform solely for diagnostics. Any error prevents artifact publication.

Nested physical worktrees are supported with their own pnpm install --frozen-lockfile, even when ancestor directories contain node_modules. An ancestor dependency cannot satisfy a missing local input or change the emitted declarations. Local pnpm links remain supported when their targets stay inside the checkout. A local link that resolves outside still fails with Declaration input escapes checkout, without publishing a success record or pruning obsolete declarations. An outside candidate remains inaccessible even if it is a symlink back into the checkout. Warm records use the same input check. Do not filter compiler receipts or transplant declarations to bypass the checks.

Declared checkout junctions and platform path aliases map to the same native root for validation and actual snapshot reads. Local declaration preparation also aligns the compiler's PWD with its working directory so shell aliases do not change emitted inventory paths. Invocations from subdirectories still use the containing checkout as the ownership boundary. Other pnpm tsgo lanes continue to use the native CLI; its wrapper does not create or reuse a shared external install.

Packaged SDK declarations belong to one staged owner shared by full, package, and ciArtifacts builds. It serializes the two canonical tsdown SDK groups on a miss and caches their complete staged generation. Each successful compiler supplies its source and package-manifest membership through a private staged receipt; missing receipts or inputs changed during compilation prevent publication. The shared input snapshot policy validates consumed bytes, inherited configuration, generator and manifest inputs, and resolution topology without starting a compiler on hits. Cache hits restore into fresh staging and pass the same entry and relative declaration closure checks before publication. All tsdown declaration builds (the eight SDK/unified groups, workspace packages, and the AI package) use the same bounded compiler as local declaration preparation and package-boundary checks. Successful builds need checkout-owned compiler libraries and declaration dependencies; shared external installs are not supported. Compiler receipts retain complete source and package-manifest membership, including JSON inputs. Default type roots stay within the checkout, explicit type roots must be local, and a valid root package.json bounds source-package scope lookup.

The shared snapshot policy still validates consumed bytes and resolution topology. Source and namespace changes during compilation prevent acceptance. New local module candidates invalidate cached records, as do ancestor-install appearance and removal. Outside probes always see missing files, so later changes to ancestor package contents cannot enter the compiler's filesystem view. Each emitted declaration must have one source-map owner in the successful compiler membership. The bundler consumes those declarations under their original source paths; private compiler stages are removed only after their child settles. Compiler configuration, native executable and API files, input bytes, and resolution topology participate in cache invalidation. Runtime module resolution is unchanged.

Local preparation never overwrites packaged declarations or writes workspace forwarding bridges.

Plugin SDK declaration preparation and scripts/run-tsgo.mjs require child work to finish before reporting success. On POSIX, each verifies its own managed process group: leftover children are terminated and the command fails instead of allowing artifact stamps or downstream checks to proceed. POSIX process groups do not detect descendants that deliberately leave the group.

On Windows, managed commands and the Gateway test instance use a retained kernel Job. Platform code loads only when a Windows command is requested; planner imports stay independent of installed application packages. Loading finishes before spawn, so listener registration remains synchronous with child creation. Tooling resolves its worker URL with Node built-ins and reuses the native Job bindings from core. The launcher joins that Job before it can start the command. Cleanup waits for an empty Job, leader exit, and output closure; leader exit alone never proves descendant completion. Failed termination reports the observed surviving PIDs and retains resource claims while the Job remains unresolved. Normal leader exit also terminates remaining Job members. Finalization records its outcome before closing the Job handle, including on failure; closing the handle alone does not verify termination or release resource claims. Existing callers with their own IPC channel keep the direct-launch contract; failed taskkill without an owned Job stays indeterminate even if the leader and its pipes have closed.

run-vitest (including project shards), plugin batches, test-live (including live shards), run-vitest-profile, and the TUI PTY watcher give each Vitest invocation an owned temporary namespace through TMPDIR, TMP, and TEMP. Before Vitest starts, isolated invocations also receive native HOME and USERPROFILE inside that namespace. This protects home fallbacks used by worker threads, named builtin imports, and import-time captures; changing only a worker's JavaScript process.env does not change native thread home lookup. Per-worker and per-test fixture homes remain separate. Installed Corepack and Playwright browser caches retain their caller-selected locations.

Gateway port claims remain in the common temporary directory outside all enclosing Vitest namespaces, found through their explicit resource owners. Parallel invocations therefore share port ownership while a fixture hands its reserved socket to a child; removing one invocation's files cannot remove another fixture's port claim.

Live-aware setup still loads the original profile and stages live state when requested. A bounded invocation artifact carries the original home to that setup; it does not grant live access, and hermetic setup never consults it. Known hermetic selections ignore ambient live and real-home flags. Known wholly live-aware selections retain explicit OPENCLAW_LIVE_USE_REAL_HOME behavior. An explicitly real-home live invocation is refused before config loading if its selection mixes home policies or cannot be classified, including custom configs and ambiguous project selectors. Run hermetic tests without LIVE, OPENCLAW_LIVE_TEST, OPENCLAW_LIVE_GATEWAY, and OPENCLAW_LIVE_USE_REAL_HOME using node scripts/run-vitest.mjs <test-path>, then run the intended live selection separately using node scripts/test-live.mts -- <live-test-path>. The launcher does not split runs or change watch, filter, or report semantics.

The namespace contains isolated homes, their JIT caches, SDK/shared-home allocation roots, and fallback SQLite state; its lifetime spans shared-worker files and module resets. On POSIX detached launches, the parent removes only that namespace after its child process group has stopped, output pipes have closed, and nested resource owners have released their pending claims, including passing and failing runs, child crashes, caught SIGINT/SIGTERM signals, and watchdog termination where supported. Explicit state, profile output, and mirror artifacts outside the namespace remain untouched. Failed or unverified group joins or unresolved nested claims retain the namespace and report the exact path for manual recovery. Nested namespaces, fixture lifetimes, and managed commands register ephemeral filesystem ownership before admitting work. Release requires positive completion evidence; caught cleanup failures, module resets, worker exit, or an intermediate runner crash cannot release a pending claim or its ancestors. Managed commands keep their existing output-drain contract; Windows Job commands also join kernel Job completion. Failed finalization never releases ownership. Stop all remaining writers before manually removing the reported exact directory. Windows and non-detached launches allocate the same isolated native home, but retain their namespace and enclosing claims with a diagnostic after child exit and pipe closure because descendant completion cannot be verified. Raw external invocations do not gain this boundary. Forced parent or supervisor death (such as SIGKILL) can prevent cleanup; unregistered descendants that intentionally escape the owned group remain outside this contract. The wrappers do not sweep old directories or infer ownership from names, ages, or PIDs. The CI shard runner also removes its default include-file and transform-cache scratch directory after every admitted group has joined. Caller-supplied scratch and persistent cache roots remain caller-owned. Unverified descendant completion retains the shard scratch directory and reports its exact path. This is home isolation, not a filesystem sandbox: explicit absolute paths, os.userInfo() account lookup, children with stripped or replaced home variables, and intentionally real-home live execution remain outside its protection.

Codex app-server fixtures await agent and shared-state SQLite drainage between cases. Their file teardown drains the shared disk-budget scan worker, preserving reuse during the file and releasing it before isolated fork shutdown.

• src/test-utils/openclaw-test-state.ts: use from Vitest when a test needs an isolated HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH, config fixture, workspace, agent dir, or auth-profile store.
• pnpm test:env-mutations:report: non-blocking report of tests/harnesses that mutate HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH, OPENCLAW_WORKSPACE_DIR, or related env keys directly. Use it to find migration candidates for the shared test-state helper.
• test/helpers/openclaw-test-instance.ts: process-level E2E tests needing a running Gateway, CLI env, log capture, and cleanup in one place.
• Docker/Bash E2E lanes that source scripts/lib/docker-e2e-image.sh can pass docker_e2e_test_state_shell_b64 <label> <scenario> into the container and decode it with scripts/lib/openclaw-e2e-instance.sh; multi-home scripts can pass docker_e2e_test_state_function_b64 and call openclaw_test_state_create <label> <scenario> in each flow. node --import tsx scripts/lib/openclaw-test-state.mts -- create --label <name> --scenario <name> --env-file <path> --json writes a sourceable host env file (the -- before create keeps newer Node runtimes from treating --env-file as a Node flag). Lanes that launch a Gateway can source scripts/lib/openclaw-e2e-instance.sh for entrypoint resolution, mock OpenAI startup, foreground/background launch, readiness probes, state env export, log dumps, and process cleanup.

createOpenClawTestState selects and owns temporary paths and process environment selectors. It is not filesystem sandboxing and does not stop external producers. Await its asynchronous restoreEnv(); stop and join required producers before restoring selectors or removing state. Runtime reproductions of state-selection leaks require enforced storage isolation, such as a VM or container without access to operator stores, not merely temporary HOME or state-directory overrides.

Public test diagnostics

The shared Vitest reporter factory redacts credential-shaped fields in assertion messages, diffs, expected/received values, stacks, source excerpts, and annotations before forwarding them to the selected reporters. Keys remain visible and values become <redacted len=N>. This also applies to explicit --reporter selections, UI/browser configurations, and JSON/JUnit reports. Hosted logs are public, and runner-issued tokens may not be registered for GitHub masking.

Unquoted environment records use one assignment per line: spaces and punctuation on the right-hand side belong to that value. Multiline strings split into quoted fragments by Node's inspector are redacted as one value.

Redaction is unconditional and affects diagnostic output, not assertion behavior. Test console capture is outside this boundary; tests must still avoid logging credentials directly.

Configured extension fork projects use the openclaw-forks diagnostic adapter around Vitest's native fork transport. If the existing stop deadline fails while the child remains alive, the adapter spends at most two additional seconds collecting a Node report before native termination and pipe cleanup. The timeout remains a test failure. The report distinguishes a missing stop acknowledgement from a stall after acknowledgement and includes native stacks, libuv handles, and worker-thread reports. Environment variables, command arguments, and socket endpoints are omitted. A blocked event loop can prevent signal reporting; that case explicitly reports that no complete report was captured.

Successful shutdown remains quiet. An explicit --pool=forks selects Vitest's built-in pool and bypasses this adapter.

JSON reports across native processes

For a multi-project or chunked run, explicitly request native JSON with an output file, for example:

pnpm test test/vitest/vitest.unit-fast-isolated.config.ts test/vitest/vitest.agents-embedded-agent.config.ts --reporter=verbose --reporter=json --outputFile=.artifacts/test-results.json

The project runner and plugin batch runner give each attempt separate native JSON and blob files, then publish the requested JSON from Vitest's native report merge. They print a companion <output>.reports-<unique> directory. Keep that directory: it contains original reports, per-attempt coverage files when coverage is enabled, and an index.json with child exit codes, signals, timeouts and unstarted work. Each invocation runs once. A no-output timeout fails the command and leaves the report set incomplete; it never starts a replacement attempt. Blob reports are exact-version artifacts. Rerun child reports with the current Vitest version before merging artifacts produced by another version.

The aggregate preserves the accepted case inventory, but is not a lossless replacement for the originals. Native merging does not restore snapshot summaries or JSON coverageMap, and its startTime is the merge time. Passing snapshot tests still succeed. Read native originals for those details and the index for process outcomes: JSON success does not encode every wrapper or unhandled-error failure. Separate built-in coverage reports remain per attempt in the companion directory. Custom coverage providers/reporters and coverage reporter tuple options require separate invocations with unique destinations.

A complete failed-test aggregate is retained with a failing command exit. Missing or invalid evidence, cancellation, unstarted required work, or publication failure does not publish a complete aggregate; an existing output file is not proof of the new run. The diagnostic prints the retained report-set location. Report sets are not automatically swept.

Overlapping selections can share native task IDs, so merging them can replace independent failure details even when case counts match. Such report sets retain their originals and fail publication. Select each configuration once, or run overlapping selections separately with distinct output files.

This ownership applies to explicit CLI JSON file requests with named, file-based Node projects and native console reporters. Scalar --outputFile and --outputFile.json both work. Config-owned reporter options, other file formats, custom reporters and inline/browser project composition require separate native invocations with unique output destinations. Do not assume those outputs are aggregated. Single-process and console-only runs keep their existing native behavior. Native help and other non-test controls stay with the child CLI and do not allocate report sets. run --version still runs tests, as it does in native Vitest. Config-only reporters are not intercepted: multiple children can still overwrite the same configured file. Run those configurations separately with distinct paths; adding --reporter=json alone does not override a reporter tuple's own outputFile.