Commit Graph
19 Commits
Author SHA1 Message Date
baojun-zhang 7c956f23bc feat(config): add compatible default timeout for ragfs pathlock (#3641)
- add storage.agfs.pathlock.lock_timeout_secs
- use pathlock default timeout instead of hardcoded zero in wrapper
- map legacy storage.transaction.lock_timeout when new config is unset
- remote redolog by using  persistent `session_commit` queue.
2026-07-31 11:23:27 +08:00
baojun-zhang 2f9451231e refactor(pathlock):using rust implement instead python (#3602)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design

* fix(ragfs):fix(ragfs): use non-blocking fcntl locks for localfs CAS

* fix(ragfs): serialize heartbeat lease refresh with release and report real conflict kind

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding
2026-07-29 19:45:34 +08:00
baojun-zhang 1841dfed81 Revert "refactor(pathlock):using rust implement instead python (#3557)" (#3597)
This reverts commit 6b538db569.
2026-07-29 11:31:41 +08:00
baojun-zhang 6b538db569 refactor(pathlock):using rust implement instead python (#3557)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design
2026-07-29 11:08:42 +08:00
Kchenandchenpengfei 5f945fa8e6 fix(path_lock): (#3353)
# 并发精确路径锁下父目录幂等创建设计

## 问题

两个并发写请求分别写入不同文件时,可能需要在同一个父目录中创建精确路径锁文件。如果该父目录尚不存在,两个加锁流程都可能先观察到目录不存在,随后递归创建同一个目录。其中一个 `mkdir` 成功,另一个收到 `EEXIST`/already-exists;此时所需目录其实已经存在。

`PathLockEngine._ensure_directory_exists_async` 当前将所有 `mkdir` 异常都视为失败。因此,竞争失败一方的加锁结果为 `False`,内容写入链路最终向调用方返回 HTTP 409 `resource is busy`。

## 预期行为

并发场景下,目录创建应具有幂等语义:

- 如果 `mkdir` 抛出异常,但重新执行 `stat` 后确认目标已经是目录,则目录准备成功。
- 如果目标仍不存在、无法查询,或者目标是非目录条目,则保持原有失败行为。
- 本次修改只影响路径锁所需父目录的准备流程,不改变锁冲突、等待超时和 HTTP 错误语义。

## 实现方案

修改 `openviking/storage/transaction/path_lock.py` 中的 `PathLockEngine._ensure_directory_exists_async`:

1. 保留现有的首次 `stat` 和递归创建父目录流程。
2. 调用 `mkdir(path)` 创建当前目录。
3. 如果 `mkdir` 抛出异常,立即通过 `_is_existing_directory_async(path)` 重新查询目录状态。
4. 如果重新查询确认目标是目录,则返回成功。这样既能处理明确的 `EEXIST`,也能兼容存储后端对该错误的包装;只有在所需文件系统状态已经成立时才忽略异常。
5. 如果重新查询未确认目标是目录,则记录原始 `mkdir` 异常并返回失败。

不增加通用重试循环。如果竞争方在该路径创建的是文件而不是目录,不能将其视为成功。

## 回归测试

在 `tests/transaction/test_exact_path_lock.py` 中增加一个可稳定复现竞态的异步测试:

- 初始文件系统中存在 `/local/default/resources`,但不存在其下的共享子目录。
- 并发为共享子目录下两个不同文件获取精确路径锁,例如 `shared/a.md` 和 `shared/b.md`。
- 通过测试 AGFS 的同步点,保证两个加锁流程首次对 `shared` 执行 `stat` 时都看到目录不存在,然后才允许任一方执行 `mkdir(shared)`。
- 允许一个 `mkdir(shared)` 创建目录,另一个抛出 already-exists 异常。
- 断言两个精确路径锁都获取成功,并且创建了两个不同的锁文件。

该测试在现有实现上必须失败:竞争失败一方的 `mkdir` 异常会进入 `_ensure_directory_exists_async` 的失败分支。

## 非目标

- 不修改调用方 `upsertTextWithConflictFallback` 的行为。
- 不调整 `resource is busy` 的可重试分类。
- 不修改精确路径锁与目录树锁之间的冲突规则。
- 不为其他 AGFS 异常增加通用重试。

Co-authored-by: chenpengfei <chenpengfei@bytedance.com>
2026-07-19 19:17:11 +08:00
Qin Haojie d47f2106ee refactor: remove unused and deprecated APIs (#3272)
Delete dead compatibility paths and test-only helpers so unsupported APIs do not remain as accidental contracts.
2026-07-16 10:49:56 +08:00
baojun-zhang e492cbd16f refactor(encryption): using rust refactor encryption (#2444) 2026-06-05 17:26:26 +08:00
Matt Van HornandMatt Van Horn 35acbc703b fix: reindex lock acquisition fails when URI targets a file (Not a directory error) (#2228)
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-05-26 14:30:53 +08:00
Qin Haojie 2406c0e6e1 refactor(storage): 异步化存储锁与 IO (#2143)
* refactor(storage): async storage lock IO

Move AGFS and storage lock paths onto async wrappers while preserving lock handoff semantics.

* refactor: streamline async task tracking

Collapse TaskTracker lifecycle operations into async-only APIs and align callers/tests with the new boundary. Also throttle repeated memory/path lock wait warnings to reduce noisy retry logs.
2026-05-22 14:03:18 +08:00
85908e241a Feat/memory link (#2010)
* auto-commit before eval 20260509_181850

* auto-commit before eval 20260509_192618

* update

* auto-commit before eval 20260510_005109

* auto-commit before eval 20260510_011832

* auto-commit before eval 20260510_014114

* auto-commit before eval 20260510_022835

* auto-commit before eval 20260510_025048

* auto-commit before eval 20260510_031034

* auto-commit before eval 20260510_143728

* auto-commit before eval 20260510_172705

* auto-commit before eval 20260510_220133

* auto-commit before eval 20260511_115905

* auto-commit before eval 20260511_121959

* auto-commit before eval 20260511_132120

* auto-commit before eval 20260511_161430

* auto-commit before eval 20260511_163606

* auto-commit before eval 20260511_173943

* auto-commit before eval 20260511_175657

* auto-commit before eval 20260511_224347

* auto-commit before eval 20260511_233109

* auto-commit before eval 20260512_104710

* auto-commit before eval 20260512_111256

* auto-commit before eval 20260512_181905

* auto-commit before eval 20260512_191540

* auto-commit before eval 20260512_192540

* auto-commit before eval 20260512_195710

* auto-commit before eval 20260513_000746

* auto-commit before eval 20260513_004221

* auto-commit before eval 20260513_004656

* refactor: migrate logger calls to tracer in extract_loop modules

Replace logger.warning/error/info with tracer.error/info in extract_loop
related modules for better observability (console + OpenTelemetry spans).

Modules updated:
- agent_experience_context_provider.py (5 replacements)
- extract_loop.py (4 replacements)
- memory_updater.py (9 replacements)
- session_extract_context_provider.py (4 replacements)
- utils/json_parser.py (7 replacements)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* auto-commit before eval 20260513_123007

* auto-commit before eval 20260513_125305

* auto-commit before eval 20260513_135421

* auto-commit before eval 20260513_141013

* auto-commit before eval 20260513_143455

* auto-commit before eval 20260513_145401

* auto-commit before eval 20260513_163345

* auto-commit before eval 20260514_105906

* auto-commit before eval 20260514_112912

* auto-commit before eval 20260514_120308

* auto-commit before eval 20260514_122022

* auto-commit before eval 20260514_134800

* auto-commit before eval 20260514_135615

* auto-commit before eval 20260514_135818

* auto-commit before eval 20260514_142941

* auto-commit before eval 20260514_162401

* auto-commit before eval 20260514_231859

* auto-commit before eval 20260515_104122

* auto-commit before eval 20260515_122140

* auto-commit before eval 20260515_122942

* auto-commit before eval 20260515_144941

* auto-commit before eval 20260515_154736

* auto-commit before eval 20260515_181643

* auto-commit before eval 20260515_182727

* auto-commit before eval 20260515_183056

* auto-commit before eval 20260515_183652

* auto-commit before eval 20260515_183825

* auto-commit before eval 20260515_202731

* auto-commit before eval 20260516_001144

* auto-commit before eval 20260516_011749

* auto-commit before eval 20260516_015903

* auto-commit before eval 20260516_020505

* auto-commit before eval 20260516_130701

* auto-commit before eval 20260516_144342

* auto-commit before eval 20260516_151043

* Harden memory graph rendering and patch guidance.

Escape embedded graph data for script safety, add a vis-network load guard, tighten graph layout defaults, and clarify SEARCH guidance so patch content stays bound to the target file/page context.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260517_005258

* auto-commit before eval 20260517_012903

* auto-commit before eval 20260517_014036

* auto-commit before eval 20260517_015726

* auto-commit before eval 20260517_024952

* auto-commit before eval 20260517_032518

* auto-commit before eval 20260517_135114

* auto-commit before eval 20260517_143238

* auto-commit before eval 20260517_154858

* auto-commit before eval 20260517_200556

* auto-commit before eval 20260517_215025

* fix: keep memory storage plain and render graph links on display

Store memory bodies as plain text in VikingFS and move link rendering to graph display so repeated writes no longer persist nested markdown links. Also tighten link renderer path handling so cross-user relative paths are rejected and strip_links preserves viking and absolute targets.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260518_001945

* fix: invert selected graph node colors

Make the currently selected memory node use a light background with dark text so it stands out against the dark graph theme.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260518_011327

* update

* auto-commit before eval 20260518_161813

* auto-commit before eval 20260518_165104

* auto-commit before eval 20260518_174259

* update

* auto-commit before eval 20260518_224834

* auto-commit before eval 20260518_233319

* auto-commit before eval 20260518_235712

* auto-commit before eval 20260519_135952

* fix memory patch failure logging

Keep dry-run patch validation from emitting a misleading patch_handler warning, and record skipped field updates from MemoryUpdater where the failure is handled.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260519_213142

* fix(memory): fan out links for shared page ids

Expand _resolve_links so shared page ids resolve across every operation URI instead of collapsing to a single path. Align the page-id and extract-loop tests with the current API contract and the multi-URI link behavior.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260520_195141

* auto-commit before eval 20260520_215911

* auto-commit before eval 20260520_222335

* update

* style(memory): clean up formatter drift

Apply the remaining formatter-driven cleanup in the memory modules so the working tree stays clean before the next behavior changes. This keeps helper signatures and string literals aligned with current lint output.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260521_130517

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-21 14:58:24 +08:00
Qin Haojie 77b604a641 fix(storage): 优化路径锁与语义刷新并发 (#2029)
* fix(storage): refine path lock semantic refresh concurrency

Use exact path locks for source commits, tree locks only for lifecycle and schema scopes, and coalesce derived semantic writes to avoid stale summary overwrites under concurrent resource and memory updates.

* chore: split benchmark changes into separate PR

* chore: keep semantic refresh design notes out of docs

* style: format lock changes

* fix: preserve resource lifecycle locks

* Revert "fix: preserve resource lifecycle locks"

This reverts commit d2fb274f85.

* fix(resource): simplify lifecycle locking

* fix(queuefs): consolidate semantic sidecar writes
2026-05-14 20:44:28 +08:00
dingbenanddingben.db@bytedance.com <dingben.db@bytedance.com@bytedance.com> 6d533f9e91 feat(fs): expose isLocked in stat() to surface path-lock state (#1940)
Adds an `isLocked` boolean to the dict returned by `VikingFS.stat()` so
callers (and the `/api/v1/fs/stat` endpoint, which transparently passes
the dict through) can tell whether a resource is currently held by a
path lock without having to attempt a write and observe `ResourceBusyError`.

The lookup reuses the same conflict-detection semantics as the acquire
flow: a path is reported as locked when it has a valid `.path.ovlock` or
when any ancestor directory holds a SUBTREE lock; stale locks are
ignored because the next acquirer would reclaim them anyway.

To make the check available to higher layers, a public
`PathLock.is_locked()` helper is introduced and surfaced through
`LockManager.is_path_locked()`; both are best-effort and degrade to
`False` when the LockManager is unavailable, keeping `stat()` resilient.

Co-authored-by: dingben.db@bytedance.com <dingben.db@bytedance.com@bytedance.com>
2026-05-09 16:40:34 +08:00
Jiahui Zhou b84f798207 feat(transaction): add redo recovery toggle for session commits (#1934)
Allow crash-recovery redo for session commit phase 2 to be disabled via configuration while keeping the default behavior unchanged. This lets deployments skip pending redo marker writes and startup redo recovery when the mechanism is not wanted.
2026-05-09 15:47:00 +08:00
MaojiaShengandopenviking a7e5417ef2 reorg: remove golang depends (#1339)
* docs: fix docker deployment

* reorg: remove third_party/agfs

* feat(s3fs): add disable_batch_delete option for OSS compatibility

Port of PR #1333 from Go version to Rust:

- Add disable_batch_delete config option to S3Client
- When enabled, use sequential single-object deletes instead of DeleteObjects
- This is for S3-compatible services like Alibaba Cloud OSS that require
  Content-MD5 for DeleteObjects but AWS SDK v2 does not send it by default
- Add documentation and config example for OSS

* fix(s3fs): pass disable_batch_delete config from Python to Rust

Add disable_batch_delete to the s3_plugin_config dict in _generate_plugin_config
so that the Python config can properly control the Rust S3FS plugin's behavior.

* reorg: remove third_party/agfs

* reorg: remove third_party/agfs

* change some docs

* change some docs

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-10 15:16:29 +08:00
Qin Haojie 6211691826 fix(transaction): unify stale lock handling with ownership checks (#1171)
Use lock_expire as the single inactivity threshold for both lock files
and in-process handles, so long-running tasks are no longer released just
because the handle was created a long time ago. Make refresh/release and
handle lookup ownership-aware to avoid zombie handles and prevent stale
owners from deleting locks that have already been reclaimed.
2026-04-02 17:12:58 +08:00
Jiahui Zhou 673b267976 feat: add content write interface (#1151) 2026-04-01 23:19:59 +08:00
MaojiaShengandopenviking ce998873f9 lisence: change the main lisence to AGPL-3.0 (#1085)
* lisence: change the main lisence from Apache-2.0 to AGPL-v3

* lisence: change the main lisence from Apache-2.0 to AGPL-v3

* lisence: change the main lisence from Apache-2.0 to AGPL-v3

---------

Co-authored-by: openviking <openviking@example.com>
2026-03-30 14:37:42 +08:00
Zayn Jarvis 659b22cad9 fix(server): handle CancelledError during shutdown paths (#848) 2026-03-25 20:31:21 +08:00
Qin HaojieandClaude Opus 4.6 1823a7c4f7 feat(storage): add path locking and selective crash recovery for write operations (#431)
* feat(storage): add transaction support with journal, undo, and crash recovery

Implement a full transaction system for VikingFS storage operations including
write-ahead journal, path locking, undo/rollback, context manager API, and
crash recovery. Includes comprehensive tests and documentation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test(transaction): add e2e rollback tests for mv and multi-step operations

Add end-to-end tests covering rollback scenarios that were missing:
- mv rollback: file moved back to original location on failure
- mv commit: file persists at new location
- Multi-step rollback: mkdir + write + mkdir all reversed in order
- Partial step rollback: only completed entries are reversed
- Nested directory rollback: child removed before parent
- Best-effort rollback: single step failure does not block others

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(storage): add transaction support with path locking and journal

Implement transaction system for VikingFS with ACID-like guarantees:
- TransactionManager with configurable lock timeout and journal-based recovery
- PathLock supporting point, subtree, and mv lock modes
- Refactor VikingFS mv to use cp+rm to prevent lock files from being carried
- Fix stale lock detection returning false for missing lock files
- Update ragas eval to use LangchainLLMWrapper

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: tests

* fix(transaction): fix rollback and race condition bugs

- Reconstruct RequestContext from undo params for vectordb_delete/update_uri
  rollback (previously skipped silently due to missing ctx)
- Serialize ctx fields into undo params in rm/mv operations
- Fix Phase 1 undo path to target archive dir instead of session root
- Remove Phase 2 fs_write_new undo (overwrites are idempotent, checkpoint
  handles recovery)
- Add ancestor SUBTREE recheck after lock creation in acquire_subtree
- Move _collect_uris inside TransactionContext in rm/mv to close race window
- Log journal persistence failures instead of silently swallowing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(transaction): make TransactionManager required and rewrite tests with real backends

Remove all optional/fallback code paths where tx_manager could be None. get_transaction_manager()
now raises RuntimeError if not initialized. Fix undo rollback to reconstruct ctx for vectordb_upsert
and use correct agent_id default. Replace mock-based transaction tests with integration tests using
real AGFS and VectorDB backends.

* refactor(transaction): make rollback fully async and unify session commit path

- Convert execute_rollback/rollback_entry to async, removing sync run_async wrappers
- Unify Session.commit() to delegate to commit_async(), removing duplicate phase methods
- Fix SUBTREE lock to conflict with ancestor SUBTREE locks (was previously missing)
- Fix mv lock mode: directory moves now use SUBTREE on both source and destination
- Replace deprecated asyncio.get_event_loop() with get_running_loop()
- Remove max_parallel_locks config option
- Update docs (en/zh) and tests to match new async rollback signatures

* fix: tests

* refactor(transaction): simplify session commit and add redo-based crash recovery

Session commit no longer wraps archive phase in a transaction. Phase 2 uses
redo semantics so crashed memory-extraction can be replayed from archive.
PathLock stale-lock cleanup no longer redundantly re-checks timeout.
Semantic processor vectorization runs concurrently via asyncio.gather.

* fix: transaction

* fix: UserIdentifier

* refactor(transaction): replace undo-based transaction manager with lightweight lock + redo-log

Remove the heavyweight TransactionManager/Journal/UndoEntry system (~4000 lines) and
replace it with a simpler architecture: LockManager for path locking, LockContext as
the async context manager, LockHandle/LockOwner protocol, and a RedoLog for crash
recovery of session_memory operations. VikingFS rm/mv now use inline error handling
instead of rollback semantics. Updated docs, observers, and tests accordingly.

Co-Authored-By: Claude Opus 4.6

* fix(transaction): remove checkpoint dead code, fix TOCTOU race, clarify mv lock param

- Remove unused _write_checkpoint/_write_checkpoint_async/_read_checkpoint
  from Session (superseded by redo-log)
- Re-resolve URI inside lock in resource_processor Phase 3.5 to prevent
  concurrent add_resource calls from resolving to the same final_uri
- Rename acquire_mv dst_path to dst_parent_path with docstring to clarify
  that callers pass the destination parent directory

* fix: path

* fix: resource lock

* fix: test

* docs: update

* fix: tests

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-18 14:50:19 +08:00