Commit Graph
29 Commits
Author SHA1 Message Date
baojun-zhang 9791c87554 feat(pathlock): keep lease alive across handoff and rotate adopt capability (#3903)
- keep handed-off leases in the registry with pending_handoff so auto-refresh continues
- include lease_ref in PathLockHandoffRef for same-process adopt fast path
- rotate both lease_ref and ownership_ref on adopt to invalidate stale producer capabilities
- reject stale capability use in release, release_selected and refresh
- validate owner_id, lock_paths and covered_paths before local adopt
- reject replayed fallback adopt when the same owner/path token is already held
- add tests for pending handoff refresh, retryable adopt race, forged coverage and replay rejection
2026-08-10 11:40:56 +08:00
baojun-zhang bd5cce09c7 feat(pathlock): adjust pathlock config (#3854) 2026-08-07 13:21:30 +08:00
baojun-zhang df1557901e feat(ragfs): add structured lock tracing logs (#3693)
* feat(ragfs): add structured lock tracing logs

* feat(ragfs): add log async writer

* fix(ragfs): add CRITICAL trans && add rust hook to receive rotate log file
2026-08-03 15:58:55 +08:00
baojun-zhang 89c2705192 feat(pathlock): split lock busy from io for retryable stale cleanup (#3659) 2026-07-31 16:08:18 +08:00
baojun-zhang 7c956f23bc feat(config): add compatible default timeout for ragfs pathlock (#3641)
- add storage.agfs.pathlock.lock_timeout_secs
- use pathlock default timeout instead of hardcoded zero in wrapper
- map legacy storage.transaction.lock_timeout when new config is unset
- remote redolog by using  persistent `session_commit` queue.
2026-07-31 11:23:27 +08:00
Kchenandchenpengfei 86771b467e fix(pathlock): tolerate concurrent lock directory creation (#3635)
Co-authored-by: chenpengfei <chenpengfei@bytedance.com>
2026-07-30 20:20:09 +08:00
baojun-zhang 2f9451231e refactor(pathlock):using rust implement instead python (#3602)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design

* fix(ragfs):fix(ragfs): use non-blocking fcntl locks for localfs CAS

* fix(ragfs): serialize heartbeat lease refresh with release and report real conflict kind

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding
2026-07-29 19:45:34 +08:00
baojun-zhang 1841dfed81 Revert "refactor(pathlock):using rust implement instead python (#3557)" (#3597)
This reverts commit 6b538db569.
2026-07-29 11:31:41 +08:00
baojun-zhang 6b538db569 refactor(pathlock):using rust implement instead python (#3557)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design
2026-07-29 11:08:42 +08:00
agent 27debfe7a8 feat(snapshot): add path diff API, SDK, and CLI (#3451)
* feat(snapshot): add path diff API

* feat(snapshot): expose path diff in clients and CLI

* fix(snapshot): pin path diff to resolved commits

* fix(snapshot): bound path diff resource usage

* fix(snapshot): bound diff blob reads

* fix(snapshot): harden path diff API

* fix(snapshot): bound path diff resource usage

* fix(build): sync native binding dependencies
2026-07-23 17:45:36 +08:00
ef4d97ebe3 feat(snapshot): support path-filtered commit history (#3271)
* feat(snapshot): support path-filtered commit history

* refactor(snapshot): reuse SDK git log implementation

* fix(snapshot): harden path-filtered log resource limits

* chore: remove stale SDK lock entry

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
2026-07-16 19:53:40 +08:00
baojun-zhang 6a33ebb7ca Optimize glob walkdir (#3013)
* feat(storage): optimize glob func

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* fix(localfs): offload blocking fs operations to spawn_blocking

* feat(glob): cap glob api default node_limit at 256

* feat(sdk): add node_limit options for glob in python and go SDKs
2026-07-06 21:39:16 +08:00
17631bd353 feat: support ignoring certain files in the multi-version management function, similar to the git ignore feature (#2930)
* docs: add ovgitignore design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: add ovgitignore implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(git): add ovgitignore matcher

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(git): apply ovgitignore during commits

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(pyagfs): expose ovgitignore commit results

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(vikingfs): manage account ovgitignore

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(git): document ovgitignore semantics

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(git): enable ovgitignore restore assertion and document python api

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: fix gitignore

* fix: fix gitignore

* feat: 新增 git ignore 相关的接口

* fix: 修复 cli 命令渲染

* doc: 删除执行计划文档

* doc: 删除执行计划文档

* feat: 在 http client 中新增 git ignore 相关接口

* fix: 修改文档中关于 .ovgitignore 被版本化的内容

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-02 11:45:38 +08:00
Haoyu Zhangandzhanghaoyu.la aa53e7aede feat: 实现 commit、restore、show 文件系统多版本管理功能 (#2756)
* feat: 实现 commit、restore、show 文件系统多版本管理功能

fix: 修复commit时删除文件

fix: commit 的 fast path 1 添加 Racy-clean 机制

fix: 将 sdk 中的 git 命令改为 snapshot 命令,同步修改单测

fix: 多版本管理的文件存储目录改为 .ovgit

feat: snapshot cli 渲染

fix: 修复 restore 时将删除的文件回滚时,目录不存在的问题

feat: restore 命令的 project_dir 参数改为可选,不传时默认全目录回滚

feat: 更新文档

fix: 删除暂未使用的配置参数

feat: 新增示例脚本

fix: 修复示例代码

fix: 修复 restore 返回的 task id 任务完成状态

feat: 在 restore 修改文件系统时加锁

fix: fix openviking_sdk

* feat: 将git多版本管理功能改为默认打开,并复用agfs的配置参数作为默认值

* fix: restore 命令改为先完成 ref 一致性协议再写回 VFS;object store 并发改为使用唯一 temp path

* fix: 在 git 配置检验层去除未实现的cas_mode = "redis_lock"模式

* fix: 在 Rust GitService 边界统一校验 account

* fix: 当前commit不支持通过 path 传入目录,增加报错信息

* fix: 将git文件默认存储路径统一为 .ovgit

* fix: restore 时写入 VFS 失败时返回详细的报错,并继续触发 reindex

* fix: 校验 commit、restore、show 的路径

* feat: 实现 commit 时指定目录

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
2026-06-25 11:28:04 +08:00
tuofangandfang 2d56bd71f4 feat(ragfs): add CachedFileSystem and Redis/Mooncake/Yuanrong cache providers (#2520)
* add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

* Handle poisoned known_keys mutexes in cache providers

* Add configurable cache support to ragfs python

* Add Redis cache provider support

* Prune native cache providers from default build

* Add RAGFS cache guides in Chinese and English

* delete .cargo/config.toml

* Fix stale cache invalidation across shared wrappers

* Document Yuanrong native concurrency limits

* add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

* Handle poisoned known_keys mutexes in cache providers

* Add configurable cache support to ragfs python

* Add Redis cache provider support

* Prune native cache providers from default build

* Add RAGFS cache guides in Chinese and English

* delete .cargo/config.toml

* Fix stale cache invalidation across shared wrappers

* Document Yuanrong native concurrency limits

* Limit directory cache entries and add regression test

* docs: add TOS s3fs backend test design

* Add runtime cache config override

* add build guides for mooncake and yuanrong

* fix _FakeConfig test with cache

* fix(ragfs): cache encrypted data below the encryption layer

- pass runtime cache config to the Rust binding
- cache ciphertext instead of decrypted content
- disable cache for encrypted multi-write mounts
- update tests and provider build documentation

* fix(ragfs): invalidate cache after same-mount raw copy

Ensure copy_within_mount invalidates the cached destination file and
parent directory after the raw backend fast path writes data directly.
This prevents stale reads and stale directory metadata when cache is
enabled and Python cp() uses the same-mount copy optimization.

Add a regression test covering overwrite copy followed by read/list.

* fix(ragfs): preserve multi-write discovery through cache layer

Allow MountableFS::as_multiwrite() to unwrap CachedFileSystem when
discovering the underlying MultiWriteWrappedFS. This preserves
multi-write admin paths and same-mount copy behavior for cache-enabled,
unencrypted multi-write mounts.

Add a regression test covering sync status, sync retry, same-mount copy,
and unmount behavior for cached unencrypted multi-write mounts.

* feat(ragfs): add cache-aware tree traversal mode

- add configurable tree traversal mode to cache policy
- keep default tree behavior delegated to backend
- allow cached traversal to reuse read_dir directory cache
- bypass cached traversal for multi-write backends
- add regression coverage for tree cache behavior and fallbacks

* docs: design cache-aware grep traversal

* feat(ragfs): add cache-aware grep traversal

Introduce a shared cache traversal mode for recursive APIs and use it to
optionally run grep through CachedFileSystem.

- add CacheTraversalMode with backend and cached_traversal modes
- keep CacheTreeMode as a compatibility alias
- route tree and grep through cached traversal only when explicitly enabled
- reuse cached read_dir entries and full-file reads during grep traversal
- keep multi-write traversal on the backend path
- expose storage.agfs.cache.traversal_mode in Python config
- raise max cached directory entries threshold to 4096
- add regression tests for grep cache traversal and traversal config

* Optimize cached grep generation validation

* Parallelize cached grep file scanning

---------

Co-authored-by: fang <fang@fangMacBook-Air.local>
2026-06-17 16:01:52 +08:00
baojun-zhang 7ee1481e1d feature(storage): support multi write storage (#2466)
* feature(storage): support multi write storage

* refactor(storage): simplify multiwrite logic and consolidate test helpers

* refactor(storage): extract multibackend and shape modules and tighten multi-write wrapper boundaries

* refactor(storage): refactor write pipeline
2026-06-08 11:12:15 +08:00
baojun-zhang e492cbd16f refactor(encryption): using rust refactor encryption (#2444) 2026-06-05 17:26:26 +08:00
baojun-zhang 073e929a1a feat(storage): optimize tree func (#2372)
* feat(storage): optimize tree func

* feat(storage): add test case && fix tree show parent dir problem

* feat(storage): format code

* feat(storage): fix check problem

* feat(storage): remove redundant unit test

* feat(storage): fix code review issue

* feat(storage): fix code review issue
2026-06-03 18:21:55 +08:00
MaojiaSheng a1c9e5080b feat: add ov observer filesystem, and optimize mkdir frequency (#2045)
* feat: ov add-resource (spec -L --level), ov stat (return count for dir)

* feat: ov add-resource (spec -L --level), ov stat (return count for dir)

* feat: Add VLM backup configuration for automatic failover

- Add backup field to VLMConfig with recursive backup prevention
- Implement FailoverVLM wrapper class for automatic failover
- Support rate limit, timeout, server error triggers
- Add comprehensive unit tests

* feat: Add VLM backup configuration for automatic failover

* feat: Add VLM backup configuration for automatic failover

* ov observer filesystem

* ov observer filesystem
2026-05-14 18:21:26 +08:00
baojun-zhang c8ff9e3575 perf(storage): optimize VikingFS grep implementation (#1731)
* perf(storage): optimize VikingFS grep implementation
- add ripgrep support and native local grep fallback for LocalFS mode
- normalize grep path handling and improve regression test coverage

* perf(storage): optimize VikingFS grep implementation
- add ripgrep support and native local grep fallback for LocalFS mode
- normalize grep path handling and improve regression test coverage

* perf(storage): format code

* perf(storage): support async grep

* feat(encryption): Push `exclude_uri` and `level_limit` down to the backend to ensure that the limits take effect after the same set of filtering conditions. &&  The default implementation also follows the same path contract as LocalFS, returning the path relative to the query root

* feat(encryption): Push `exclude_uri` and `level_limit` down to the backend to ensure that the limits take effect after the same set of filtering conditions. &&  The default implementation also follows the same path contract as LocalFS, returning the path relative to the query root

* fix(storage): relativize localfs exclude_path against grep query root && disable parent .gitignore inheritance in localfs rg fast path && add HTTP AGFS grep support for exclude_path and level_limit
2026-05-07 19:41:33 +08:00
Jiahui Zhou b9753385ec feat: parallelize grep reads for remote storage (#1761) 2026-04-28 14:57:11 +08:00
zgy 1567707856 feat(queuefs): Add SQLite backend with ack/recover and unified control-file/config specs (#1500)
* feat(queuefs): add sqlite backend + unify control/config specs

* fix queuefs legacy sqlite dequeue compatibility

* fix queuefs sqlite metadata and queue semantics
2026-04-20 11:54:18 +08:00
kaisongli 8d39834795 fix(ci): fix ragfs-python native extension build in CI pipelines (#1532)
* fix(ci): fix ragfs-python native extension build in CI pipelines

* fix(ci): skip sessions/ tests when no VLM/Embedding secrets available

Sessions tests (add_message, commit, used) depend on storage backend
that requires proper initialization with valid API keys. When running
without secrets (PR from fork), these tests fail with 400 because the
storage layer cannot fully initialize. Skip them in no-secrets mode;
they will still run on main branch pushes where secrets are available.
2026-04-17 17:13:23 +08:00
MaojiaShengandopenviking 21ba377a9c fix: use abi3 for rust package (#1517)
* fix: use abi3 for rust package

* fix: modify list scope

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-17 13:58:23 +08:00
MaojiaSheng 95cc0f84d0 reorg: split parser layer to 2-layer: accessor and parser, so that we can reuse more code (#1428) 2026-04-15 10:33:12 +08:00
Qin Haojie 62c023307d fix(docker): raise Rust MSRV for ragfs builds (#1448)
Align the Docker image, local build checks, and Rust crate metadata with the 1.91.1 toolchain required by ragfs-python's S3 dependencies.
2026-04-14 20:36:31 +08:00
chuanbao666 fc230cf9c2 fix(agfs): enable agfs s3 plugin default (#1408) 2026-04-13 21:06:39 +08:00
Jiahui Zhou b174deb410 Fix ci (#1307)
* fix(ci): fix build ci

* build: move ragfs-python packaging into setup.py
2026-04-09 00:05:18 +08:00
MaojiaShengandopenviking dc052ce3e3 reorg: Rewrite agfs to ragfs with rust (#1221)
* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* fix: grep level limit

* fix: grep root

* fix: import error

* fix: rust code optimazation

* fix: CI error

* fix: CI go mod cache

* fix: grep level limit

* fix: CI

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-05 21:17:45 +08:00