The api_key_watch_interval mechanism reloads account info across replicas
by polling compute_store_signature() (a (path, size, modTime) signature
over accounts.json + users.json). On S3FS this never detected writer-side
changes: S3FS stat() serves from a sliding-TTL StatCache (default 60s,
re-armed on every get()), so the watcher's 30s poll kept the entry alive
forever and the signature never moved -> reload() never fired. localfs
stats live, so it worked there.
Thread a bypass_cache flag from the watcher's stat call down to S3FS so
signature stats read fresh backend metadata:
- FsContext(View): add bypass_cache field + builder/getter
- ragfs-python build_fs_context: parse ctx["bypass_cache"]
- S3FS stat(): skip stat_cache.get() when bypass_cache is set
- AsyncAGFSClient.stat(bypass_cache=...): inject ctx flag
- legacy _stat_signature: stat with bypass_cache=True
Co-authored-by: TRAE CLI <traecli@bytedance.com>
* feat: add freshness-aware parent aggregation
Defer wide-directory abstract/overview regeneration until the configured freshness threshold is reached while continuing changed-file semantic and vector processing.
Persist freshness metadata atomically, make parent bubbling L0-aware, preserve separate semantic/vector statuses, and keep explicit waits synchronous.
Rebuild every sampled summary on threshold refresh and always retry directory vectorization so stale sidecars or transient vector failures cannot be silently accepted.
Add focused coverage for freshness policy, pending-state consumption, sampled-summary refresh, vector retries, and parent bubbling.
Co-authored-by: TRAE CLI <traecli@bytedance.com>
* feat: ov reindex support --recursive
* feat: ov reindex support --recursive
* feat: ov reindex support --recursive
* feat: ov reindex support --recursive, and applied to memory
* feat: ov reindex support --recursive, and applied to memory
---------
Co-authored-by: TRAE CLI <traecli@bytedance.com>
The public MountableFS constructors do not initialize a pathlock manager, but
multi-write mount and raw copy used expect() on the missing manager, so calling
either fast path on such an instance panicked.
Return Error::Config with a clear message instead; tests that need the success
path already build the manager via with_test_pathlock_manager().
* chore: remove dead git tuning knobs and duplicate release/frontend files
- GitTuningConfig: drop upload_concurrency, restore_concurrency,
ref_cas_max_retry, and ref_cas_backoff_ms, which were parsed but never read
anywhere (verified no source readers). Keep commit_index_enabled and
blob_exists_precheck_enabled, the two knobs that actually take effect.
- Design doc: mark the removed knobs as roadmap items to be added back when
the behavior lands, and correct stale 'not implemented' claims
(validate_account_id is enforced at the Git service entry; blob reads are
limited via show_with_limit).
- Remove .github/workflows/release-vikingbot-first.yml: a historical one-off
PyPI release workflow whose bot/ package lacks build metadata.
- web-studio: delete pnpm-lock.yaml and the pnpm-only package.json block;
the Makefile and CI already use npm + package-lock.json as the single
install chain. Net -9,695 lines.
* docs: align cleanup notes with current behavior
---------
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
* feat(reindex): support tag updates
Add replace and append tag modes to reindex vector rebuilds, preserve omission-aware behavior, and propagate options through background tasks and namespace rebuilds. Align Python, TypeScript, Go, and CLI interfaces with tests and documentation.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(reindex): lock file targets exactly
Use an exact path lock for existing file targets while retaining tree locks for directories and prune-orphans scopes. Add a regression test for single-file reindex.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(reindex): handle prune file targets
Use exact locks for existing file targets in prune-orphans mode while retaining tree scope for missing targets. Document the existing Go ReindexOptions wait semantics and add lock regression coverage.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
---------
Co-authored-by: TRAE CLI <noreply@bytedance.com>
Replace the fixed 50 ms retry loop with bounded exponential backoff and jitter while preserving the original first-retry latency floor. Clamp each delay to the remaining timeout so retry sleeps do not add avoidable timeout overshoot.
Add deterministic interval and polling-reduction checks plus a four-waiter contention regression. The 10-second worst-case schedule drops retry probes from 200 to 29 while all waiters still acquire after the holder releases.
Tests: cargo test -p ragfs --lib lock:: -- --nocapture
* fix(pathlock): treat localfs read ENOENT as NotFound
Map ENOENT from localfs read back to NotFound when a file disappears
between metadata() and fs::read().
This avoids misclassifying lock-file delete races as plugin I/O errors
in pathlock token reads, so missing .path.ovlock is handled as an
expected absence instead of a fatal lock I/O failure.
* fix(pathlock): treat localfs read ENOENT as NotFound
Map ENOENT from localfs read back to NotFound when a file disappears
between metadata() and fs::read().
* feat(memory): support event tag filtering
Add session-level default event tags, commit-time overrides, durable queue propagation, and first-write vector index tagging. Include config update APIs and coverage for serialization, concurrency, extraction, and HTTP behavior.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* feat(memory): expose event tags in SDKs and CLI
Add session default tag configuration, config updates, and commit-time event tag overrides across embedded Python, standalone Python, TypeScript, Go, and the Rust CLI. Preserve explicit empty-tag semantics and document each public interface.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(sdk): align legacy session tag APIs
Forward commit-time event tags through the legacy Python HTTP shims and align BaseClient session signatures without adding a new abstract-method requirement for existing subclasses.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* feat(session): allow updating auto-commit policy
Extend PATCH session config to atomically update event tags and auto-commit settings. Merge policy objects by field, use explicit null to disable automatic commits, preserve omitted fields, and expose the contract across SDKs and CLI.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(session): align session config interfaces
Replace the generic session create config JSON flag with explicit event-tag and auto-commit options. Preserve omitted, object, and null auto-commit semantics across HTTP, embedded clients, SDKs, and CLI, reject ambiguous null policy fields, and handle nullable event configuration consistently.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* test(session): trim redundant event tag tests
---------
Co-authored-by: TRAE CLI <noreply@bytedance.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
- keep handed-off leases in the registry with pending_handoff so auto-refresh continues
- include lease_ref in PathLockHandoffRef for same-process adopt fast path
- rotate both lease_ref and ownership_ref on adopt to invalidate stale producer capabilities
- reject stale capability use in release, release_selected and refresh
- validate owner_id, lock_paths and covered_paths before local adopt
- reject replayed fallback adopt when the same owner/path token is already held
- add tests for pending handoff refresh, retryable adopt race, forged coverage and replay rejection
* feat: support oidc and ldap auth
* feat: support oidc and ldap auth
* fix: remove heima partner, clean up auth docs, add web-studio unsupported auth banner
- Remove heima from partner list in README (en/zh/ja)
- Remove unsupported env var references (OPENVIKING_AUTH_MODE, OPENVIKING_USERNAME,
OPENVIKING_PASSWORD) from LDAP auth docs
- Remove temporary switch bash snippets from auth docs
- Fix ldap_password description
- Add web-studio unsupported-auth-mode banner for oidc/ldap servers
* fix: address OIDC/LDAP review comments on auth plugin design
Key changes driven by PR review:
- **Role mapping**: OIDC and LDAP external identities always resolve to
USER role. Removed map_role() calls and group_membership-based role
mapping. Admin access is gated by the root API key mechanism only.
- **LDAP credential extraction**: Removed query-parameter-based username/
password extraction (security concern — passwords in URLs can leak via
shell history, proxy logs, and monitoring). Clients must use Basic Auth
header or form data.
- **OIDC identifier sanitization**: Auth0 and other providers may include
characters like "|" in the `sub` claim. These are now replaced with "_"
to produce valid OpenViking user identifiers.
- **Dead code removal**: Removed _extract_groups, memberof_attribute,
require_root_api_key_for_admin, _initialize_api_key_manager, and
get_request_context_checks from both plugins since they are no longer
needed.
- **Docs**: Removed query-parameter curl example, memberof_attribute and
require_root_api_key_for_admin config references.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* feat: support oidc and ldap auth
* feat: support oidc and ldap auth
* fix(auth): bind lazy OIDC imports at module scope
---------
Co-authored-by: TRAE CLI <noreply@bytedance.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
* docs: fix stale commands, paths, and provider claims
Sweep findings: D-03, D-04, D-05, D-06, D-07, D-08, D-09. Align setup and API examples with current configuration and CLI behavior.
(cherry picked from commit 2b21ee513b)
* fix(review): correct crypto output flag
Addresses blocking review finding on #3401.
(cherry picked from commit 85bb502709)
* docs(ov): finish stale CLI path cleanup
Complete the #3401 salvage by updating the API-writing templates and the remaining encryption guide examples to the Rust CLI surface.
* fix(ov): make local content operations race-safe
Salvage the safe-I/O portions of OpenViking#3414: reject unsupported local watch requests before upload, atomically create download targets, and write snapshot output before reporting JSON success. Add focused regression coverage.
* fix(ov): validate timeout and node-limit inputs
Salvage and complete OpenViking#3414 by validating every timeout and node-limit surface consistently while preserving config commands as a repair path for invalid persisted values.
* fix(ov): allow explicit help before language setup
Salvage OpenViking#3416 with a narrower contract: only clap-recognized -h/--help requests bypass first-run language selection. Bare command groups, legacy -help, and option values keep the existing gate.
* docs(ov): align session and snapshot command examples
Salvage OpenViking#3419 by correcting positional session and snapshot examples, documenting the canonical observer filesystem command, and keeping fs as a compatible alias.
* fix(ov): honor configured output defaults safely
Salvage and complete OpenViking#3424 with CLI-over-config precedence, runtime validation for normal commands, and a table fallback that leaves config repair commands usable. Also clarify the Python-client versus Rust-CLI upload-mode controls.
* fix(ov): preserve zero node-limit semantics
* ci: skip embedding-dependent resource test without secrets
* fix(cli): validate compile timeout consistently
---------
Co-authored-by: zhiheng.liu <zhiheng.liu@bytedance.com>
* fix(ragfs): preserve cache visibility on partial S3 deletes
Surface exact and per-object S3 deletion failures, while always invalidating the affected directory and stat cache scope after a recursive delete attempt.
Source-PR: #3407
Original-Commit: 8d6addf28e
* fix(session): preserve legacy policy and peer identity compatibility
Parse string false and other legacy boolean-like memory policy values without silently enabling extraction or breaking persisted configs. Encode mixed-script peers losslessly, while retaining their former lossy IDs as read-only retrieval and extraction aliases.
Source-PR: #3422
Original-Commit: 0dfd5a9ed9
* fix(memory): drain timer flush tasks during shutdown
Retain the shielded timer flush task and await it when close cancels the timer loop, so batch failures are observed and submitters are resolved without unhandled task exceptions.
Source-PR: #3438
Original-Commit: ca1d74e164
* fix(storage): preserve peer isolation and cache correctness
* fix(ingest): reserve encoded peer namespace
* ci: skip embedding-dependent resource test without secrets
* fix(ragfs): invalidate caches after partial remove
---------
Co-authored-by: zhiheng.liu <zhiheng.liu@bytedance.com>
- add storage.agfs.pathlock.lock_timeout_secs
- use pathlock default timeout instead of hardcoded zero in wrapper
- map legacy storage.transaction.lock_timeout when new config is unset
- remote redolog by using persistent `session_commit` queue.
* feat: implement server-resolved OpenViking Assets manifests
Add the openviking-assets/1 declaration flow with server-owned configuration parsing and native Rust CLI execution.
- Resolve one flat Manifest against one Catalog through an authenticated server endpoint with strict schema and Git semantic validation.
- Reject recursive includes and unsafe clone URLs; return a resolved plan without submitting resources or running server-side batches.
- Keep local credential aliases, manifest state, dry-run, failure isolation, and per-asset create/sync orchestration in the CLI.
- Generate normalized stable asset identities on the server and remove the CLI direct SHA-1 dependency.
- Update flat examples and add server resolver/API plus Rust CLI coverage.
* feat: implement server-resolved OpenViking Assets manifests
* feat: implement server-resolved OpenViking Assets manifests
* fix(pathlock): tolerate missing lock token after recursive delete
* feat: implement server-resolved OpenViking Assets manifests
* feat: implement server-resolved OpenViking Assets manifests
* fix(ragfs): enforce mount containment and write-flag semantics in LocalFS
Sweep findings: E-01, E-04. Reject lexical traversal and honor LocalFS write contracts.
* fix(review): reject absolute localfs remainders
Addresses blocking review finding on #3402.
* fix(ragfs): close LocalFS glob mount-escape gap
LocalFS::glob_directory only called validate_virtual_path(path), which
rejects `..` but accepts an absolute remainder. A `//`-double-slash mount
path (e.g. `/local//etc`) survives normalize_path, and find_mount hands the
remainder `//etc` to glob_directory; validate_virtual_path passes it
(components are [RootDir, Normal("etc")], no ParentDir), and glob_via_walk's
resolve_virtual_path strips one slash to `/etc` and joins it over the base —
an absolute join that overrides the base and lists the host directory.
Every other op (read/write/stat/rename/remove/grep) routes through
resolve_path, which adds the is_absolute check. glob now calls the same
resolve_path guard (discarding the returned PathBuf) so it shares the
containment contract. Directory-listing info leak only; content reads were
already covered.
Adds test_localfs_glob_mount_rejects_absolute_remainder mirroring the read
regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RbWx1T81KkNV4nucxWsPXZ
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ov): redact gateway secrets in config show and create root key with 0600
Sweep findings: E-02, E-03. Prevent credential disclosure in output and at key creation.
* test(ov): drop trivial init-key permission test
The 0600 fix is a one-line OpenOptions::mode; a dedicated tokio+tempfile
test module for a single mode assertion is not worth its weight. The
redaction test in store.rs (a real multi-case security behavior) stays.
