omarchy/docs/testing.md

Testing

How the non-graphical test suites are organized: what each runner owns, the protocol test files speak, and the conventions that keep them runnable on any machine — including headless CI sandboxes with no compositor. The graphical acceptance suite is a separate thing that drives a live session in a disposable VM; see agents/skills/acceptance-tests.md.

Suite map

./test/all runs both suites below and keeps going when one fails, so a single failure cannot hide the other suite behind it. It reports the failed suites at the end and exits non-zero.

• ./test/cli — one big script, one suite. It owns the CLI router: help and group rendering, route resolution, aliases, hidden commands, and the guarantee that a trailing --help never executes the target. It also owns the metadata lint — every omarchy-* executable under bin/ is checked for a # omarchy:summary= header and against removed or redundant fields — plus the theme pipeline: template rendering (omarchy-theme-set-templates, omarchy-theme-color, omarchy-theme-osc), the theme sync commands (tmux, GNOME, VS Code, Pi, Claude) run against stub binaries and a fake $HOME.
• ./test/shell — runs every test/shell.d/*-test.sh (except base-test.sh itself). Each file is an independent suite covering one area: a shell plugin, a bin/ command, a config invariant, or a still-live migration. This is where new tests go.
• Acceptance — everything that needs a real desktop doing real things. Deliberately excluded from ./test/all; it runs in a VM, not the development session.

A new shell test only needs the right name: drop <area>-test.sh into test/shell.d/ and ./test/shell picks it up automatically. Shared fixtures live under test/shell.d/fixtures/.

The base-test.sh contract

Every shell test starts the same way:

#!/bin/bash

set -euo pipefail

source "$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)/base-test.sh"

base-test.sh refuses to be executed directly — it is a library. It discovers the repo root from its own location and exports it as ROOT, so tests reference files as $ROOT/bin/... and never depend on the caller's working directory or an installed Omarchy.

Assertions are TAP-flavored and blunt:

• pass "description" prints ok - description.
• skip "description" prints ok - description # SKIP. Include why the check could not run. Like pass, it returns normally; use exit 0 afterwards only when the rest of the file cannot run either.
• fail "description" [detail] prints the optional detail and not ok - description to stderr, then exits the file. There is no counting or continuing within a file: the first failed assertion ends it, which keeps later assertions from reporting against state the failure already invalidated.
• require_command <cmd> fails the file when a needed tool is absent.

The runner compensates for that early exit: ./test/shell continues past a failing file and summarizes the failures at the end. Aborting the whole run at the first bad file once let a single packaging failure mask 114 of 134 files. Failure granularity is therefore per file inside a run, per assertion inside a file.

The runner also lists files with skipped checks, including files that ran some checks or later failed. Skips do not fail a run. When no files fail, a run with skips is reported as completed without failures rather than having passed every check.

Compositor-dependent tests

Some tests launch Quickshell or query Hyprland, but the suite must stay green on headless machines. require_compositor "description" handles this: when no compositor answers it calls skip with the reason and exits 0, and otherwise returns so the file proceeds. The skip marker lets the runner distinguish unavailable runtime coverage from checks that passed.

The probe is more than an environment check, because WAYLAND_DISPLAY only proves the variable was inherited. Sandboxes pass the environment through while blocking $XDG_RUNTIME_DIR, so Quickshell clears a bare variable check and then aborts inside QGuiApplication — a core dump per launch where a skip belonged. So compositor_reachable checks the socket actually exists, then asks Hyprland itself (hyprctl -j monitors, retried, and only when HYPRLAND_INSTANCE_SIGNATURE makes it askable), since a compositor that died mid-session leaves its socket behind. When the compositor is reachable, require_compositor also sets ulimit -c 0: Quickshell leaves through qFatal() if its connection drops mid-run, and the test should fail without writing a core dump as debris.

Gate only what needs gating — put require_compositor in files whose runtime half needs a live session, and keep static analysis of the same area in code that runs unconditionally before or beside it.

Unit-testing shell JavaScript from bash

The Quickshell plugins keep their logic in plain .js modules (shell/plugins/menu/MenuModel.js, bar/BarModel.js, ...) that end in a guarded if (typeof module !== "undefined") module.exports = {...} block. QML imports them directly and ignores the guard; Node loads them as CommonJS. That dual citizenship is what makes the shell's model logic unit-testable without a compositor.

run_node_test is the bridge: it prepends a JS prelude to a heredoc and pipes the result into node. The prelude mirrors the bash assertion protocol (pass, fail, assert, assertEqual, assertDeepEqual — same ok/not ok lines, same exit-on-first-failure) and provides root (from the exported ROOT), path, and requireFromRoot(relativePath):

run_node_test <<'JS'
const menu = requireFromRoot('shell/plugins/menu/MenuModel.js')

const parsed = menu.parseMenuJsonc('{ "items": { "root": { "label": "Go" }, }, }')
assertEqual(parsed.length, 1, 'menu parses JSONC with trailing commas')
JS

Roughly a quarter of the shell test files use this to test parsing, merging, and layout logic as pure functions, reserving compositor-gated tests for what only a live session can prove.

Conventions worth copying

• Redirect background output. Send background fixtures' stdout to a log or /dev/null and clean up the processes on exit. An inherited output pipe can hold the runner open after the test exits.
• Stub the world, run the real code. Tests build a scratch bin/ of stub executables (sudo, tmux, gsettings, helper commands) that log their arguments to a file, prepend it to PATH, and then run the real script under test. Assertions grep the call log and the files the script wrote.
• Fake $HOME, real $OMARCHY_PATH. Anything touching user state runs with HOME pointed at a mktemp -d directory (cleaned up via trap ... EXIT) and OMARCHY_PATH="$ROOT", so tests exercise the checkout without touching the developer's machine.
• Migrations run directly. A migration test builds the legacy state in a fake $HOME, runs bash -euo pipefail "$ROOT/migrations/<ts>.sh", and asserts the resulting state — including running it twice to prove idempotence, and once against non-legacy state to prove it leaves user customization alone. Keep that test while the migration is still being written or bugfixed, if it calls an Omarchy helper whose interface can still change, or if it is a security-sensitive privileged repair. Once a one-shot rewrite has shipped in a tagged release and is frozen, drop the test even when that rewrite used sudo, pacman, or limine-mkinitcpio. Keep the migration itself for late-updaters. Tests of omarchy-migrate, the login notifier, and omarchy-upgrade-to-quattro stay.
• Assert the invariant, not the snapshot. Config tests pin the property a test is named for (this widget stays adjacent to that one) rather than whole structures, so unrelated churn does not fail them.