Commit Graph
26 Commits
Author SHA1 Message Date
baojun-zhang 2f9451231e refactor(pathlock):using rust implement instead python (#3602)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design

* fix(ragfs):fix(ragfs): use non-blocking fcntl locks for localfs CAS

* fix(ragfs): serialize heartbeat lease refresh with release and report real conflict kind

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding
2026-07-29 19:45:34 +08:00
baojun-zhang 1841dfed81 Revert "refactor(pathlock):using rust implement instead python (#3557)" (#3597)
This reverts commit 6b538db569.
2026-07-29 11:31:41 +08:00
baojun-zhang 6b538db569 refactor(pathlock):using rust implement instead python (#3557)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design
2026-07-29 11:08:42 +08:00
ef4d97ebe3 feat(snapshot): support path-filtered commit history (#3271)
* feat(snapshot): support path-filtered commit history

* refactor(snapshot): reuse SDK git log implementation

* fix(snapshot): harden path-filtered log resource limits

* chore: remove stale SDK lock entry

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
2026-07-16 19:53:40 +08:00
Qin Haojie d47f2106ee refactor: remove unused and deprecated APIs (#3272)
Delete dead compatibility paths and test-only helpers so unsupported APIs do not remain as accidental contracts.
2026-07-16 10:49:56 +08:00
baojun-zhang 22131233a9 Fix dir ops locks (#3054)
* feat(python-sdk): format code

* fix: reuse active path lock handle in encrypted mv and compressor fallback writes
2026-07-07 14:05:57 +08:00
17631bd353 feat: support ignoring certain files in the multi-version management function, similar to the git ignore feature (#2930)
* docs: add ovgitignore design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: add ovgitignore implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(git): add ovgitignore matcher

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(git): apply ovgitignore during commits

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(pyagfs): expose ovgitignore commit results

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(vikingfs): manage account ovgitignore

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(git): document ovgitignore semantics

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(git): enable ovgitignore restore assertion and document python api

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: fix gitignore

* fix: fix gitignore

* feat: 新增 git ignore 相关的接口

* fix: 修复 cli 命令渲染

* doc: 删除执行计划文档

* doc: 删除执行计划文档

* feat: 在 http client 中新增 git ignore 相关接口

* fix: 修改文档中关于 .ovgitignore 被版本化的内容

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-02 11:45:38 +08:00
Haoyu Zhangandzhanghaoyu.la aa53e7aede feat: 实现 commit、restore、show 文件系统多版本管理功能 (#2756)
* feat: 实现 commit、restore、show 文件系统多版本管理功能

fix: 修复commit时删除文件

fix: commit 的 fast path 1 添加 Racy-clean 机制

fix: 将 sdk 中的 git 命令改为 snapshot 命令,同步修改单测

fix: 多版本管理的文件存储目录改为 .ovgit

feat: snapshot cli 渲染

fix: 修复 restore 时将删除的文件回滚时,目录不存在的问题

feat: restore 命令的 project_dir 参数改为可选,不传时默认全目录回滚

feat: 更新文档

fix: 删除暂未使用的配置参数

feat: 新增示例脚本

fix: 修复示例代码

fix: 修复 restore 返回的 task id 任务完成状态

feat: 在 restore 修改文件系统时加锁

fix: fix openviking_sdk

* feat: 将git多版本管理功能改为默认打开,并复用agfs的配置参数作为默认值

* fix: restore 命令改为先完成 ref 一致性协议再写回 VFS;object store 并发改为使用唯一 temp path

* fix: 在 git 配置检验层去除未实现的cas_mode = "redis_lock"模式

* fix: 在 Rust GitService 边界统一校验 account

* fix: 当前commit不支持通过 path 传入目录,增加报错信息

* fix: 将git文件默认存储路径统一为 .ovgit

* fix: restore 时写入 VFS 失败时返回详细的报错,并继续触发 reindex

* fix: 校验 commit、restore、show 的路径

* feat: 实现 commit 时指定目录

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
2026-06-25 11:28:04 +08:00
yufeng 37507571a9 fix: harden Studio resource search (#2521)
* fix studio playground resource search

* fix: harden studio resource search

* test: handle missing user skills directory

* fix: preserve lazy user directory listing

* test: prepare user skills directory
2026-06-09 19:11:33 +08:00
baojun-zhang e492cbd16f refactor(encryption): using rust refactor encryption (#2444) 2026-06-05 17:26:26 +08:00
Qin Haojie 2406c0e6e1 refactor(storage): 异步化存储锁与 IO (#2143)
* refactor(storage): async storage lock IO

Move AGFS and storage lock paths onto async wrappers while preserving lock handoff semantics.

* refactor: streamline async task tracking

Collapse TaskTracker lifecycle operations into async-only APIs and align callers/tests with the new boundary. Also throttle repeated memory/path lock wait warnings to reduce noisy retry logs.
2026-05-22 14:03:18 +08:00
85908e241a Feat/memory link (#2010)
* auto-commit before eval 20260509_181850

* auto-commit before eval 20260509_192618

* update

* auto-commit before eval 20260510_005109

* auto-commit before eval 20260510_011832

* auto-commit before eval 20260510_014114

* auto-commit before eval 20260510_022835

* auto-commit before eval 20260510_025048

* auto-commit before eval 20260510_031034

* auto-commit before eval 20260510_143728

* auto-commit before eval 20260510_172705

* auto-commit before eval 20260510_220133

* auto-commit before eval 20260511_115905

* auto-commit before eval 20260511_121959

* auto-commit before eval 20260511_132120

* auto-commit before eval 20260511_161430

* auto-commit before eval 20260511_163606

* auto-commit before eval 20260511_173943

* auto-commit before eval 20260511_175657

* auto-commit before eval 20260511_224347

* auto-commit before eval 20260511_233109

* auto-commit before eval 20260512_104710

* auto-commit before eval 20260512_111256

* auto-commit before eval 20260512_181905

* auto-commit before eval 20260512_191540

* auto-commit before eval 20260512_192540

* auto-commit before eval 20260512_195710

* auto-commit before eval 20260513_000746

* auto-commit before eval 20260513_004221

* auto-commit before eval 20260513_004656

* refactor: migrate logger calls to tracer in extract_loop modules

Replace logger.warning/error/info with tracer.error/info in extract_loop
related modules for better observability (console + OpenTelemetry spans).

Modules updated:
- agent_experience_context_provider.py (5 replacements)
- extract_loop.py (4 replacements)
- memory_updater.py (9 replacements)
- session_extract_context_provider.py (4 replacements)
- utils/json_parser.py (7 replacements)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* auto-commit before eval 20260513_123007

* auto-commit before eval 20260513_125305

* auto-commit before eval 20260513_135421

* auto-commit before eval 20260513_141013

* auto-commit before eval 20260513_143455

* auto-commit before eval 20260513_145401

* auto-commit before eval 20260513_163345

* auto-commit before eval 20260514_105906

* auto-commit before eval 20260514_112912

* auto-commit before eval 20260514_120308

* auto-commit before eval 20260514_122022

* auto-commit before eval 20260514_134800

* auto-commit before eval 20260514_135615

* auto-commit before eval 20260514_135818

* auto-commit before eval 20260514_142941

* auto-commit before eval 20260514_162401

* auto-commit before eval 20260514_231859

* auto-commit before eval 20260515_104122

* auto-commit before eval 20260515_122140

* auto-commit before eval 20260515_122942

* auto-commit before eval 20260515_144941

* auto-commit before eval 20260515_154736

* auto-commit before eval 20260515_181643

* auto-commit before eval 20260515_182727

* auto-commit before eval 20260515_183056

* auto-commit before eval 20260515_183652

* auto-commit before eval 20260515_183825

* auto-commit before eval 20260515_202731

* auto-commit before eval 20260516_001144

* auto-commit before eval 20260516_011749

* auto-commit before eval 20260516_015903

* auto-commit before eval 20260516_020505

* auto-commit before eval 20260516_130701

* auto-commit before eval 20260516_144342

* auto-commit before eval 20260516_151043

* Harden memory graph rendering and patch guidance.

Escape embedded graph data for script safety, add a vis-network load guard, tighten graph layout defaults, and clarify SEARCH guidance so patch content stays bound to the target file/page context.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260517_005258

* auto-commit before eval 20260517_012903

* auto-commit before eval 20260517_014036

* auto-commit before eval 20260517_015726

* auto-commit before eval 20260517_024952

* auto-commit before eval 20260517_032518

* auto-commit before eval 20260517_135114

* auto-commit before eval 20260517_143238

* auto-commit before eval 20260517_154858

* auto-commit before eval 20260517_200556

* auto-commit before eval 20260517_215025

* fix: keep memory storage plain and render graph links on display

Store memory bodies as plain text in VikingFS and move link rendering to graph display so repeated writes no longer persist nested markdown links. Also tighten link renderer path handling so cross-user relative paths are rejected and strip_links preserves viking and absolute targets.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260518_001945

* fix: invert selected graph node colors

Make the currently selected memory node use a light background with dark text so it stands out against the dark graph theme.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260518_011327

* update

* auto-commit before eval 20260518_161813

* auto-commit before eval 20260518_165104

* auto-commit before eval 20260518_174259

* update

* auto-commit before eval 20260518_224834

* auto-commit before eval 20260518_233319

* auto-commit before eval 20260518_235712

* auto-commit before eval 20260519_135952

* fix memory patch failure logging

Keep dry-run patch validation from emitting a misleading patch_handler warning, and record skipped field updates from MemoryUpdater where the failure is handled.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260519_213142

* fix(memory): fan out links for shared page ids

Expand _resolve_links so shared page ids resolve across every operation URI instead of collapsing to a single path. Align the page-id and extract-loop tests with the current API contract and the multi-URI link behavior.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260520_195141

* auto-commit before eval 20260520_215911

* auto-commit before eval 20260520_222335

* update

* style(memory): clean up formatter drift

Apply the remaining formatter-driven cleanup in the memory modules so the working tree stays clean before the next behavior changes. This keeps helper signatures and string literals aligned with current lint output.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260521_130517

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-21 14:58:24 +08:00
yepper ddcd3fb9c8 chore(format): align python and c++ file formatting (#2001)
* chore(format): align python and c++ file formatting

* chore: update urllib3 to 2.7.0 and clean test imports

1. bump urllib3 dependency from 2.6.3 to 2.7.0
2. remove unused pytest import and RoleScope import from test file

* style: format list comprehensions and lambda function for readability

Adjust the line breaks in the list comprehension in the VikingSearchTool class to follow standard Python formatting conventions, and rewrap the lambda assignment in the test case to improve code readability without changing functionality.

* style: fix line wrapping and remove extra blank line

- remove stray blank line in ov_server.py
- wrap long logger.info line in memory.py for better readability

* style: fix targeted ruff lint violations

* chore: clean up unused imports and reorder code

This commit removes unused imports, reorders import statements for better consistency,
and simplifies some test file imports. Changes include:
- Remove redundant blank lines and unused imports across multiple test files and core modules
- Reorder imports in openviking hooks module to follow standard layout
- Fix import ordering in memory isolation handler
- Simplify php parser type imports
- Move volcengine mock import to correct position in test file

* refactor(uri utils): remove extra blank lines in uri.py

clean up redundant whitespace to improve code readability
2026-05-13 17:53:09 +08:00
MaojiaShengandopenviking e15f95eb46 reorg: collect all envs from everywhere, and defined in consts.py (#1490)
* refactor: all env are listed in openviking_cli/utils/config/consts.py

* refactor: all env are listed in openviking_cli/utils/config/consts.py

* refactor: all env are listed in openviking_cli/utils/config/consts.py

* refactor: all env are listed in openviking_cli/utils/config/consts.py

* refactor: all env are listed in openviking_cli/utils/config/consts.py

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-16 16:58:20 +08:00
chuanbao666 fc230cf9c2 fix(agfs): enable agfs s3 plugin default (#1408) 2026-04-13 21:06:39 +08:00
MaojiaShengandopenviking a7e5417ef2 reorg: remove golang depends (#1339)
* docs: fix docker deployment

* reorg: remove third_party/agfs

* feat(s3fs): add disable_batch_delete option for OSS compatibility

Port of PR #1333 from Go version to Rust:

- Add disable_batch_delete config option to S3Client
- When enabled, use sequential single-object deletes instead of DeleteObjects
- This is for S3-compatible services like Alibaba Cloud OSS that require
  Content-MD5 for DeleteObjects but AWS SDK v2 does not send it by default
- Add documentation and config example for OSS

* fix(s3fs): pass disable_batch_delete config from Python to Rust

Add disable_batch_delete to the s3_plugin_config dict in _generate_plugin_config
so that the Python config can properly control the Rust S3FS plugin's behavior.

* reorg: remove third_party/agfs

* reorg: remove third_party/agfs

* change some docs

* change some docs

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-10 15:16:29 +08:00
bot-of-qin-ctxandqin-ctx e8c129c51f Chore/pr agent ark token billing (#1117)
* chore(ci): switch Qodo PR agent to Ark token billing

* Update conftest.py

---------

Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
2026-03-31 15:16:45 +08:00
MaojiaShengandopenviking ce998873f9 lisence: change the main lisence to AGPL-3.0 (#1085)
* lisence: change the main lisence from Apache-2.0 to AGPL-v3

* lisence: change the main lisence from Apache-2.0 to AGPL-v3

* lisence: change the main lisence from Apache-2.0 to AGPL-v3

---------

Co-authored-by: openviking <openviking@example.com>
2026-03-30 14:37:42 +08:00
Qin HaojieandClaude Opus 4.6 1823a7c4f7 feat(storage): add path locking and selective crash recovery for write operations (#431)
* feat(storage): add transaction support with journal, undo, and crash recovery

Implement a full transaction system for VikingFS storage operations including
write-ahead journal, path locking, undo/rollback, context manager API, and
crash recovery. Includes comprehensive tests and documentation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test(transaction): add e2e rollback tests for mv and multi-step operations

Add end-to-end tests covering rollback scenarios that were missing:
- mv rollback: file moved back to original location on failure
- mv commit: file persists at new location
- Multi-step rollback: mkdir + write + mkdir all reversed in order
- Partial step rollback: only completed entries are reversed
- Nested directory rollback: child removed before parent
- Best-effort rollback: single step failure does not block others

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(storage): add transaction support with path locking and journal

Implement transaction system for VikingFS with ACID-like guarantees:
- TransactionManager with configurable lock timeout and journal-based recovery
- PathLock supporting point, subtree, and mv lock modes
- Refactor VikingFS mv to use cp+rm to prevent lock files from being carried
- Fix stale lock detection returning false for missing lock files
- Update ragas eval to use LangchainLLMWrapper

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: tests

* fix(transaction): fix rollback and race condition bugs

- Reconstruct RequestContext from undo params for vectordb_delete/update_uri
  rollback (previously skipped silently due to missing ctx)
- Serialize ctx fields into undo params in rm/mv operations
- Fix Phase 1 undo path to target archive dir instead of session root
- Remove Phase 2 fs_write_new undo (overwrites are idempotent, checkpoint
  handles recovery)
- Add ancestor SUBTREE recheck after lock creation in acquire_subtree
- Move _collect_uris inside TransactionContext in rm/mv to close race window
- Log journal persistence failures instead of silently swallowing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(transaction): make TransactionManager required and rewrite tests with real backends

Remove all optional/fallback code paths where tx_manager could be None. get_transaction_manager()
now raises RuntimeError if not initialized. Fix undo rollback to reconstruct ctx for vectordb_upsert
and use correct agent_id default. Replace mock-based transaction tests with integration tests using
real AGFS and VectorDB backends.

* refactor(transaction): make rollback fully async and unify session commit path

- Convert execute_rollback/rollback_entry to async, removing sync run_async wrappers
- Unify Session.commit() to delegate to commit_async(), removing duplicate phase methods
- Fix SUBTREE lock to conflict with ancestor SUBTREE locks (was previously missing)
- Fix mv lock mode: directory moves now use SUBTREE on both source and destination
- Replace deprecated asyncio.get_event_loop() with get_running_loop()
- Remove max_parallel_locks config option
- Update docs (en/zh) and tests to match new async rollback signatures

* fix: tests

* refactor(transaction): simplify session commit and add redo-based crash recovery

Session commit no longer wraps archive phase in a transaction. Phase 2 uses
redo semantics so crashed memory-extraction can be replayed from archive.
PathLock stale-lock cleanup no longer redundantly re-checks timeout.
Semantic processor vectorization runs concurrently via asyncio.gather.

* fix: transaction

* fix: UserIdentifier

* refactor(transaction): replace undo-based transaction manager with lightweight lock + redo-log

Remove the heavyweight TransactionManager/Journal/UndoEntry system (~4000 lines) and
replace it with a simpler architecture: LockManager for path locking, LockContext as
the async context manager, LockHandle/LockOwner protocol, and a RedoLog for crash
recovery of session_memory operations. VikingFS rm/mv now use inline error handling
instead of rollback semantics. Updated docs, observers, and tests accordingly.

Co-Authored-By: Claude Opus 4.6

* fix(transaction): remove checkpoint dead code, fix TOCTOU race, clarify mv lock param

- Remove unused _write_checkpoint/_write_checkpoint_async/_read_checkpoint
  from Session (superseded by redo-log)
- Re-resolve URI inside lock in resource_processor Phase 3.5 to prevent
  concurrent add_resource calls from resolving to the same final_uri
- Rename acquire_mv dst_path to dst_parent_path with docstring to clarify
  that callers pass the destination parent directory

* fix: path

* fix: resource lock

* fix: test

* docs: update

* fix: tests

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-18 14:50:19 +08:00
chuanbao666 c1537a5dc5 chore: agfs-client默认切到binding-client模式 (#430) 2026-03-05 15:45:05 +08:00
chuanbao666 ac17311b55 fix(agfs): agfs sdk默认安装从本地安装 (#355) 2026-02-28 17:58:05 +08:00
Qin HaojieandClaude Opus 4.6 4d7d222658 fix(test): 修复测试稳定性问题,清理废弃代码 (#353)
* fix(test): 修复测试稳定性问题,清理废弃代码

- 重构 is_closing 为基类正式 property,移除 getattr 防御性调用
- 测试用例使用唯一 account ID 避免跨测试数据冲突
- 服务器测试使用独立临时目录避免并发竞争
- 修复 ResourceWarning 测试因历史警告导致的误报
- 修复 CLI grep 命令参数顺序
- 更新路径遍历测试断言为 ValueError
- 移除已废弃的 CompressManager 测试
- 改进 AGFS_LIB_PATH 环境变量路径发现逻辑
- 新增 ragas/datasets/pandas 测试依赖
- 改进 CLI 服务器启动失败时的错误输出

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* revert: 回退 binding_client.py 的改动,由其他人负责修改

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 17:04:44 +08:00
chuanbao666 4c9340fc10 feat(agfs): agfs新增binding client (#304)
* feat(agfs): agfs新增binding client

fix: viking_fs适配binding client, 简化agfs相关参数

* feat(agfs): agfs binding-client支持windows

* docs: 补充agfs binding-client用法

* fix(agfs): 默认带上agfs binding-client依赖库; setup.py支持编译该库

* fix(agfs): add agfs binding-client linux so

* fix(agfs): code format
2026-02-26 20:27:00 +08:00
MaojiaSheng db7319f4a2 refactor: to accelerate cli launch speed, refactor openviking_cli dir (#150)
* fix: remove await asyncio and call agfs directly

* feat: mv cli out of openviking

* refactor: mv cli out of openviking

* refactor: mv cli out of openviking
2026-02-12 22:12:50 +08:00
chuanbao666 b5cd4e2ecb fix(depends): 修复测试引入的依赖 (#130) 2026-02-11 15:22:59 +08:00
chuanbao666 5395854385 refactor: refactor agfs s3 backend config (#113)
add some test for agfs s3 backend
reformat openviking/
2026-02-09 21:06:59 +08:00