fix(backup): preserve discovery under concurrent state writes (#146912)

## What Problem This Solves

Concurrent state-database writes can make `backup create` report a valid config as invalid. Following its `--no-include-workspace` advice can then produce a verified archive with unresolved agent ownership and declared plugin databases copied as opaque bytes.

## Why This Change Was Made

Discovery and config revalidation now use online SQLite read snapshots. Failed discovery refuses archive publication and reports the underlying error. `--no-include-workspace` changes file selection without changing plugin discovery; `--only-config` remains available for recovery.

#146700 introduced opaque handling, which remains unchanged for genuinely undeclared files.
#146713 made concurrent memory repair reachable from ordinary search, exposing this discovery failure.

## User Impact

Backups retain agent and plugin protection under write load. Invalid or unreadable discovery cannot silently produce a partial state archive.

## Evidence

Before: with a sustained root-database writer, full backup exited 1 blaming config; the no-workspace command exited 0 with empty agent roots and an opaque warning for a declared plugin database.
After: both commands succeed with verification, resolved agent roots, and managed plugin capture. Idle backup and legacy-audit capture remain covered.

## Compatibility

No schema, config key, flag, or archive-format change. Invalid-config state archives now fail closed; `--only-config` still exports the active JSON file. Production growth provides one scoped online-read lifetime shared by discovery and revalidation.

This intentionally replaces the schema-invalid partial state archive supported by #143693. Both state archive modes now require resolved agent and plugin ownership. `--only-config` remains a raw root-file recovery export and does not include referenced config files.

## Consumers

- `backup create`: reliable discovery under concurrent state writes and truthful failure output.
- `backup create --no-include-workspace`: the same agent/plugin ownership protection with workspace files excluded.

The decision `which declared plugin resources backup protects` is made by exactly one mechanism at `src/commands/backup-resource-inventory.ts:126`, supplied with resolved ownership before archive creation.

- Pre-migration backup (`src/commands/migrate/apply.ts:26`) calls the same verified archive command before applying a migration. It consumes the archive path and propagates discovery failures. Only absent local state/config is treated as an empty installation.
- Config staging revalidation (`src/infra/backup-config-capture.ts:119`) takes a fresh scoped state image while retaining authored config bytes, file identity, and include-graph checks.
- Archive result, manifest, and text/JSON output no longer receive unresolved-ownership skipped entries. The existing manifest reader continues to accept earlier archives.

## Invalidation

Each discovery or revalidation operation owns and closes its read snapshot. Config-file identity and include-graph checks remain active. Legacy-audit witnesses continue to read live state.

## Contention

Real CLI backups run alongside a continuous SQLite writer. Both workspace modes retain their declared owners.

## Tests

The registered backup CLI tests cover concurrent root-database writes in both workspace modes and unreadable-state refusal. Retained real CLI captures show the original full-backup failure and no-workspace ownership downgrade on the baseline, then successful verified archives on the candidate tree. Independent archive inspection confirms agent and declared plugin contents, workspace selection, and opaque handling for undeclared SQLite files.

All 79 focused tests passed on a fresh merge of `63ef2a8a4b44` with main `a8a9114fb78784cb726ba271e3b4da74a40f01a9`: four CLI cases, 23 command/publication cases, 30 state-reader cases, and 22 config-capture/legacy-audit cases. Workspace-only plugin discovery passes in both workspace modes. Restoring the previous workspace filter makes the no-workspace case fail with an opaque warning for its declared database. Eight real CLI cases also passed on that integration. The final test-only correction passed all 140 backup-create tests, and CI is green at the landing head.

AI-assisted.

Co-authored-by: Ayaan Zaidi <hi@obviy.us>

docs/cli/backup.md

@@ -409,16 +409,11 @@ Local edits inside a managed `dev/` checkout are developer source, not OpenClaw
 
 ## Invalid config behavior
 
-`openclaw backup` bypasses the normal config preflight so it can still help during recovery. Workspace discovery depends on a valid config, so `openclaw backup create` fails fast when the config file exists but is invalid and workspace backup is still enabled.
+`openclaw backup` bypasses the normal config preflight so it can still help during recovery. State archives require resolved agent and plugin ownership. If discovery fails, `backup create` reports the underlying error and refuses to publish an archive. `--no-include-workspace` excludes workspace files; it does not bypass ownership discovery.
 
-For a partial backup in that situation, rerun with
-`--no-include-workspace`: it keeps state, config, and the external credentials
-directory in scope without workspace discovery. Because malformed configuration
-also prevents resolving custom agent ownership and effectively activated plugin
-resources, the result records those unresolved scopes as skipped diagnostics;
-do not treat that recovery archive as a complete backup.
+Discovery reads shared state through an online SQLite snapshot so concurrent writers do not make a valid config appear invalid. If the state cannot be read, resolve the reported error and retry backup.
 
-`--only-config` also works when the config is malformed, since it does not parse the config for workspace discovery.
+`--only-config` still works when the config is malformed or state discovery fails. It saves the active JSON config file alone, without parsing it or including its dependencies.
 
 ## Size and performance
 

src/cli/program/register.backup.product-path.test.ts

@@ -1,20 +1,26 @@
 import { spawn } from "node:child_process";
 import fs from "node:fs/promises";
 import path from "node:path";
+import { Worker } from "node:worker_threads";
+import * as tar from "tar";
 import { describe, expect, it } from "vitest";
+import {
+  closeOpenClawStateDatabase,
+  openOpenClawStateDatabase,
+} from "../../state/openclaw-state-db.js";
 import { withOpenClawTestState } from "../../test-utils/openclaw-test-state.js";
 
 function runBackupCli(params: {
   env: NodeJS.ProcessEnv;
   outputPath: string;
-  preloadPath: string;
+  preloadPath?: string;
+  includeWorkspace?: boolean;
 }): Promise<{ code: number | null; stdout: string; stderr: string }> {
   return new Promise((resolve, reject) => {
     const child = spawn(
       process.execPath,
       [
-        "--import",
-        params.preloadPath,
+        ...(params.preloadPath ? ["--import", params.preloadPath] : []),
         "--import",
         "tsx",
         path.resolve("src/entry.ts"),
@@ -22,12 +28,12 @@ function runBackupCli(params: {
         "create",
         "--output",
         params.outputPath,
-        "--no-include-workspace",
+        ...(params.includeWorkspace ? [] : ["--no-include-workspace"]),
         "--verify",
         "--json",
       ],
       {
-        env: params.env,
+        env: { ...params.env, OPENCLAW_TEST_RUNTIME_LOG: "1" },
         stdio: ["ignore", "pipe", "pipe"],
       },
     );
@@ -45,6 +51,124 @@ function runBackupCli(params: {
 }
 
 describe("backup create CLI", () => {
+  it.each([true, false])(
+    "backup create retains workspace-discovered owners under state write load (includeWorkspace=%s)",
+    async (includeWorkspace) => {
+      await withOpenClawTestState({ layout: "state-only" }, async (state) => {
+        const pluginRoot = path.join(
+          state.workspaceDir,
+          ".openclaw",
+          "extensions",
+          "backup-fixture",
+        );
+        await fs.mkdir(pluginRoot, { recursive: true });
+        await fs.writeFile(path.join(state.workspaceDir, "workspace-only-marker.txt"), "workspace");
+        await fs.writeFile(
+          path.join(pluginRoot, "package.json"),
+          JSON.stringify({
+            name: "backup-fixture",
+            version: "1.0.0",
+            openclaw: { extensions: ["./index.cjs"] },
+          }),
+        );
+        await fs.writeFile(
+          path.join(pluginRoot, "index.cjs"),
+          "module.exports = { register() {} };",
+        );
+        await fs.writeFile(
+          path.join(pluginRoot, "openclaw.plugin.json"),
+          JSON.stringify({
+            id: "backup-fixture",
+            configSchema: { type: "object", properties: {} },
+            backupResources: [
+              { disposition: "include", scope: "state", relativePath: "plugin-data" },
+            ],
+          }),
+        );
+        await state.writeConfig({
+          agents: { entries: { main: { workspace: state.workspaceDir } } },
+          plugins: { allow: ["backup-fixture"], entries: { "backup-fixture": { enabled: true } } },
+        });
+        const root = openOpenClawStateDatabase();
+        const rootPath = root.path;
+        closeOpenClawStateDatabase();
+        await fs.mkdir(state.statePath("plugin-data"));
+        const writer = new Worker(
+          `
+          const { parentPort, workerData } = require('node:worker_threads');
+          const { DatabaseSync } = require('node:sqlite');
+          const plugin = new DatabaseSync(workerData.plugin);
+          plugin.exec("CREATE TABLE records(value TEXT); INSERT INTO records VALUES ('owned');");
+          plugin.close();
+          const db = new DatabaseSync(workerData.root, { timeout: 5000 });
+          db.exec('PRAGMA journal_mode=WAL; CREATE TABLE contention_fixture(value BLOB); INSERT INTO contention_fixture VALUES(zeroblob(1048576));');
+          const write = db.prepare('UPDATE contention_fixture SET value=randomblob(1048576)');
+          let stopped = false;
+          parentPort.on('message', () => { stopped = true; });
+          function tick() {
+            if (stopped) { db.close(); parentPort.close(); return; }
+            write.run(); setImmediate(tick);
+          }
+          tick(); parentPort.postMessage('ready');
+        `,
+          {
+            eval: true,
+            workerData: { root: rootPath, plugin: state.statePath("plugin-data/records.sqlite") },
+          },
+        );
+        const finished = new Promise<number | Error>((resolve) => {
+          writer.once("error", resolve);
+          writer.once("exit", resolve);
+        });
+        try {
+          await new Promise<void>((resolve, reject) => {
+            writer.once("message", () => resolve());
+            writer.once("error", reject);
+          });
+          const outputPath = state.path("backup.tar.gz");
+          const result = await runBackupCli({
+            env: { ...process.env, ...state.env },
+            outputPath,
+            includeWorkspace,
+          });
+          expect(result.code, result.stderr).toBe(0);
+          const archive = JSON.parse(result.stdout);
+          expect(archive.agentRoots).toEqual([expect.objectContaining({ agentId: "main" })]);
+          expect(archive.verified).toBe(true);
+          expect(archive.warnings ?? []).toEqual([]);
+          const entries: string[] = [];
+          await tar.t({
+            file: outputPath,
+            onReadEntry: (entry) => {
+              entries.push(entry.path);
+            },
+          });
+          expect(entries.some((entry) => entry.endsWith("/plugin-data/records.sqlite"))).toBe(true);
+          expect(entries.some((entry) => entry.endsWith("/workspace-only-marker.txt"))).toBe(
+            includeWorkspace,
+          );
+        } finally {
+          writer.postMessage("stop", []);
+          expect(await finished).toBe(0);
+        }
+      });
+    },
+  );
+
+  it("backup create refuses unreadable discovery state even without workspaces", async () => {
+    await withOpenClawTestState({ layout: "state-only" }, async (state) => {
+      await state.writeConfig({ agents: { entries: { main: {} } } });
+      await fs.mkdir(state.statePath("state"));
+      await fs.writeFile(state.statePath("state/openclaw.sqlite"), "unreadable database");
+      const outputPath = state.path("backup.tar.gz");
+      const result = await runBackupCli({ env: { ...process.env, ...state.env }, outputPath });
+      expect(result.code).toBe(1);
+      expect(result.stderr).not.toContain("Config invalid");
+      expect(result.stderr).toContain("Cannot read shared state for discovery");
+      await expect(fs.stat(outputPath)).rejects.toMatchObject({ code: "ENOENT" });
+    });
+  });
+
   it("completes when the SQLite snapshot outlives the audit lease", async () => {
     await withOpenClawTestState(
       { layout: "state-only", prefix: "backup-cli-audit-lease-", scenario: "minimal" },

src/commands/backup-shared.ts

@@ -29,6 +29,7 @@ import {
 } from "../skills/loading/skill-root-discovery.js";
 import { tryRealpath } from "../skills/loading/symlink-targets.js";
 import { recordBackupRunOutcome } from "../state/backup-run-records.js";
+import { withOpenClawStateDatabaseReadSnapshot } from "../state/openclaw-state-db-readonly.js";
 import { resolveOpenClawStateSqlitePath } from "../state/openclaw-state-db.paths.js";
 import { pathExists, resolveUserPath, shortenHomePath } from "../utils.js";
 import {
@@ -72,7 +73,7 @@ export function resolveRequiredBackupPath(
 }
 
 type BackupAssetKind = "state" | "config" | "credentials" | "workspace" | "agent" | "managed skill";
-type BackupSkipReason = "covered" | "missing" | "regenerable" | "unresolved" | "private";
+type BackupSkipReason = "covered" | "missing" | "regenerable" | "private";
 
 export type BackupAsset = {
   kind: BackupAssetKind;
@@ -82,7 +83,7 @@ export type BackupAsset = {
 };
 
 type SkippedBackupAsset = {
-  kind: BackupAssetKind | BackupRegenerableKind | "plugin resources";
+  kind: BackupAssetKind | BackupRegenerableKind;
   sourcePath: string;
   displayPath: string;
   reason: BackupSkipReason;
@@ -185,7 +186,6 @@ async function resolveBackupPlanFromPaths(params: {
   agentRoots?: readonly BackupAgentRoot[];
   pluginInventory?: ActivatedPluginBackupInventory;
   configCapture?: BackupConfigCapture;
-  unresolvedOwnership?: boolean;
   includeWorkspace?: boolean;
   onlyConfig?: boolean;
   skillDiscoveryLimits?: ResolvedSkillDiscoveryLimits;
@@ -442,16 +442,6 @@ async function resolveBackupPlanFromPaths(params: {
       reason: "regenerable",
     });
   }
-  if (params.unresolvedOwnership) {
-    for (const kind of ["agent", "plugin resources"] as const) {
-      skipped.push({
-        kind,
-        sourcePath: configPath,
-        displayPath: shortenHomePath(configPath),
-        reason: "unresolved",
-      });
-    }
-  }
 
   return {
     stateDir,
@@ -602,6 +592,18 @@ export async function resolveBackupPlanFromDisk(
     nowMs?: number;
   } = {},
 ): Promise<BackupPlan> {
+  if (params.onlyConfig) {
+    return await resolveBackupPlanFromState(params);
+  }
+  assertNotUpdateCapturePath(resolveOpenClawStateSqlitePath(), resolveStateDir());
+  return await withOpenClawStateDatabaseReadSnapshot(() => resolveBackupPlanFromState(params));
+}
+
+async function resolveBackupPlanFromState(params: {
+  includeWorkspace?: boolean;
+  onlyConfig?: boolean;
+  nowMs?: number;
+}): Promise<BackupPlan> {
   const includeWorkspace = params.includeWorkspace ?? true;
   const onlyConfig = params.onlyConfig ?? false;
   const stateDir = resolveStateDir();
@@ -624,23 +626,24 @@ export async function resolveBackupPlanFromDisk(
   const configSnapshot = configRead.snapshot;
   const discoverySnapshot = resolveStartupConfigSnapshot(configSnapshot) ?? configSnapshot;
   const configCapture = await resolveBackupConfigCapture(configRead);
-  if (includeWorkspace && discoverySnapshot.exists && !discoverySnapshot.valid) {
+  if (discoverySnapshot.exists && !discoverySnapshot.valid) {
     throw new Error(
-      `Config invalid at ${shortenHomePath(discoverySnapshot.path)}. OpenClaw cannot reliably discover custom workspaces for backup. Fix the config or rerun with --no-include-workspace for a partial backup.`,
+      `Backup discovery failed at ${shortenHomePath(discoverySnapshot.path)}: ${discoverySnapshot.issues.map((issue) => issue.message).join("; ")}. Agent and plugin ownership could not be resolved. Resolve the reported error and retry backup, or use --only-config to save the config alone.`,
     );
   }
-  const unresolvedOwnership = discoverySnapshot.exists && !discoverySnapshot.valid;
-  const discoveredWorkspaceDirs = unresolvedOwnership
-    ? []
-    : buildCleanupPlan({
-        cfg: discoverySnapshot.config,
-        stateDir,
-        configPath,
-        oauthDir,
-      }).workspaceDirs;
-  const agentRoots = unresolvedOwnership
-    ? []
-    : await resolveBackupAgentRoots(discoverySnapshot.config);
+  const discoveredWorkspaceDirs = buildCleanupPlan({
+    cfg: discoverySnapshot.config,
+    stateDir,
+    configPath,
+    oauthDir,
+  }).workspaceDirs;
+  const agentRoots = await resolveBackupAgentRoots(discoverySnapshot.config);
+  const pluginInventory = resolveActivatedPluginBackupInventory({
+    config: discoverySnapshot.config,
+    env: process.env,
+    stateDir,
+    workspaceDirs: discoveredWorkspaceDirs,
+  });
   // Effective agent workspaces can omit their shared base. Exclude it only here
   // so full backups and destructive cleanup retain their existing selection.
   if (!includeWorkspace && discoverySnapshot.valid) {
@@ -649,14 +652,6 @@ export async function resolveBackupPlanFromDisk(
       discoveredWorkspaceDirs.push(resolveUserPath(sharedWorkspaceBase));
     }
   }
-  const pluginInventory = unresolvedOwnership
-    ? undefined
-    : resolveActivatedPluginBackupInventory({
-        config: discoverySnapshot.config,
-        env: process.env,
-        stateDir,
-        workspaceDirs: includeWorkspace ? discoveredWorkspaceDirs : [],
-      });
   return await resolveBackupPlanFromPaths({
     stateDir,
     configPath,
@@ -665,7 +660,6 @@ export async function resolveBackupPlanFromDisk(
     agentRoots,
     pluginInventory,
     configCapture,
-    unresolvedOwnership,
     includeWorkspace,
     onlyConfig,
     skillDiscoveryLimits: resolveSkillDiscoveryLimits(discoverySnapshot.config),

src/commands/backup.test-support.ts

@@ -50,7 +50,7 @@ export async function resetBackupTempHome(tempHome: { home: string }) {
 export async function mockStateOnlyBackupPlan(stateDir: string) {
   await fs.writeFile(
     path.join(stateDir, "openclaw.json"),
-    JSON.stringify({ agents: { ownership: "explicit", entries: {} } }),
+    JSON.stringify({ agents: { ownership: "explicit", entries: { main: {} } } }),
     "utf8",
   );
   const plan = await backupShared.resolveBackupPlanFromDisk({

src/commands/backup.test.ts

@@ -535,16 +535,12 @@ describe("backup commands", () => {
             });
       await withInvalidWorkspaceBackupConfig(raw, async (runtime) => {
         await expect(backupCreateCommand(runtime, { dryRun: true })).rejects.toThrow(
-          /--no-include-workspace/i,
+          /ownership could not be resolved/i,
         );
 
-        const result = await backupCreateCommand(runtime, {
-          dryRun: true,
-          includeWorkspace: false,
-        });
-
-        expect(result.includeWorkspace).toBe(false);
-        expect(result.assets.map((asset) => asset.kind)).not.toContain("workspace");
+        await expect(
+          backupCreateCommand(runtime, { dryRun: true, includeWorkspace: false }),
+        ).rejects.toThrow(/ownership could not be resolved/i);
 
         const configOnly = await backupCreateCommand(runtime, {
           dryRun: true,

src/infra/backup-config-capture.test.ts

@@ -85,6 +85,10 @@ describe("full backup config include capture", () => {
             .replace('ownership: "explicit"', "defaults: { workspace: 42 }");
           graph.files.set(state.configPath, raw);
           await fs.writeFile(state.configPath, raw);
+          await expect(
+            createBackupArchive({ output: state.path("backup.tar.gz"), includeWorkspace: false }),
+          ).rejects.toThrow(/ownership could not be resolved/i);
+          return;
         }
         if (rootLink) {
           const authoredRoot = state.path("authored-config.json5");

src/infra/backup-config-capture.ts

@@ -6,6 +6,7 @@ import { hashConfigIncludeRaw } from "../config/includes.js";
 import { createConfigIO } from "../config/io.factory.js";
 import { containsConfigIncludeDirective } from "../config/io.read-helpers.js";
 import type { ReadConfigFileSnapshotForWriteResult } from "../config/io.types.js";
+import { withOpenClawStateDatabaseReadSnapshot } from "../state/openclaw-state-db-readonly.js";
 
 type CapturedConfigFile = {
   sourcePath: string;
@@ -117,10 +118,12 @@ export async function resolveBackupConfigCapture({
     assertRootAlias,
     revalidate: async () => {
       await assertRootAlias?.();
-      const current = await createConfigIO({
-        configPath: snapshot.path,
-        observe: false,
-      }).readConfigFileSnapshotForWrite();
+      const current = await withOpenClawStateDatabaseReadSnapshot(() =>
+        createConfigIO({
+          configPath: snapshot.path,
+          observe: false,
+        }).readConfigFileSnapshotForWrite(),
+      );
       // A file can be reached repeatedly during discovery. Comparing the resolved
       // source as well as the last hashes prevents accepting mixed observations.
       if (

src/infra/backup-create.test.ts

@@ -4144,7 +4144,7 @@ describe("createBackupArchive", () => {
             includeWorkspace: false,
             nowMs: Date.UTC(2026, 4, 9, 8, 34, 45),
           }),
-        ).rejects.toThrow(/Canonical global SQLite path must be a regular file or symlink/);
+        ).rejects.toThrow(`Cannot read shared state for discovery: ${globalDbPath}`);
         expect(await fs.readdir(outputDir)).toEqual([]);
       },
     );

src/state/openclaw-state-db-readonly.ts

@@ -30,6 +30,51 @@ const disposableStateReads = resolveGlobalSingleton(
   () => new AsyncLocalStorage<{ path: string; active: boolean }[]>(),
 );
 
+const stateSnapshotReads = resolveGlobalSingleton(
+  Symbol.for("openclaw.stateSnapshotReads"),
+  () =>
+    new AsyncLocalStorage<{
+      path: string;
+      location: string;
+      env: NodeJS.ProcessEnv;
+      active: boolean;
+    }>(),
+);
+
+/** Resolve a composite read from one online snapshot without redirecting live writers. */
+export async function withOpenClawStateDatabaseReadSnapshot<T>(
+  operation: () => Promise<T>,
+  options: OpenClawStateDatabaseOptions = {},
+): Promise<T> {
+  const pathname = resolveReadOnlyPath(options);
+  const current = stateSnapshotReads.getStore();
+  if ((current?.active && current.path === pathname) || !existingPathOrUndefined(pathname)) {
+    return await operation();
+  }
+  const env = options.env ?? process.env;
+  openClawStateDatabaseCache.assertOpenClawStateDatabaseFreshOpenAllowedAtPath(pathname, env);
+  let prepared: PreparedSqliteReadOnlyLocation;
+  try {
+    prepared = await prepareSqliteReadOnlyLocation(pathname);
+  } catch (error) {
+    throw new Error(
+      `Cannot read shared state for discovery: ${pathname}. Retry after the current state operation completes. ${String(error)}`,
+      { cause: error },
+    );
+  }
+  const scope = { path: pathname, location: prepared.location, env, active: true };
+  await using _ = {
+    async [Symbol.asyncDispose]() {
+      scope.active = false;
+      if (!(await prepared.cleanupAsync())) {
+        throw new Error(`Shared-state discovery snapshot cleanup failed: ${pathname}`);
+      }
+    },
+  };
+  openClawStateDatabaseCache.assertOpenClawStateDatabaseFreshOpenAllowedAtPath(pathname, env);
+  return await stateSnapshotReads.run(scope, operation);
+}
+
 /** The caller owns this private database and removes its files after the scope closes. */
 export async function withDisposableOpenClawStateReads<T>(
   pathname: string,
@@ -101,6 +146,17 @@ function withOpenClawStateDatabaseReadOnlyIfOpen<T>(
   operation: (database: OpenClawStateReadOnlyDatabase) => T,
   pathname: string,
 ): ReusedOpenClawStateReadOnlyDatabase<T> {
+  const snapshot = stateSnapshotReads.getStore();
+  if (snapshot?.active && snapshot.path === pathname) {
+    openClawStateDatabaseCache.assertOpenClawStateDatabaseFreshOpenAllowedAtPath(
+      pathname,
+      snapshot.env,
+    );
+    return {
+      reused: true,
+      value: withOpenClawStateReadOnlyLocation(operation, pathname, snapshot.location),
+    };
+  }
   const opened = openClawStateDatabaseCache.getOpenClawStateDatabaseIfOpenAtPath(pathname);
   if (!opened || opened.db.isTransaction) {
     return { reused: false };