Commit Graph
16 Commits
Author SHA1 Message Date
baojun-zhang 8c9c2282a6 feat(queuefs): support redis as queufs backend ,redis mode support singleton 、cluster 、 sentinel (#3741) 2026-08-05 17:19:42 +08:00
Hao Zheandzhiheng.liu 4237c66031 fix(bot): harden scheduling, sessions, sandbox, and packaging (#3549)
* fix(bot): make SRT sandbox constructible and surface startup failures

Sweep findings: C-02, C-04. Read SRT settings from the correct config and never cache failed startups.

(cherry picked from commit 0d3798399a)

* fix(bot): reject unschedulable cron jobs and honest manual runs

Sweep findings: C-07, C-08. Reject impossible schedules and fail no-op manual execution.

(cherry picked from commit 47413fdffe)

* build: ship VikingBot workspace and bridge assets in wheels

Sweep findings: F-07. Copy required bot assets into the package and resolve installed workspace templates correctly.

(cherry picked from commit 85e9ff2d06)

* fix(bot): OpenAPI channel concurrency, session deletion, and context handling

Sweep findings: C-09, C-10, C-11. Reject ambiguous requests and make session deletion durable.

(cherry picked from commit 68fd70c5be)

* docs(bot): correct channel setup config paths and license metadata

Sweep findings: C-13, C-15. Point channel setup at ov.conf and align package metadata with MIT.

(cherry picked from commit b15c241037)

* fix(review): align bot license metadata

Addresses blocking review finding on #3426.

(cherry picked from commit 001e0fa7ed)

* fix(bot): preserve cron scheduler liveness

Record callback-less scheduled runs as job errors so recurring timers persist and rearm, while manual runs still fail without mutating state. Document the OpenAPI context rejection and same-session concurrency contract.

* fix(ragfs): use stable Windows file identity APIs

---------

Co-authored-by: zhiheng.liu <zhiheng.liu@bytedance.com>
2026-07-31 15:57:43 +08:00
baojun-zhang 2f9451231e refactor(pathlock):using rust implement instead python (#3602)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design

* fix(ragfs):fix(ragfs): use non-blocking fcntl locks for localfs CAS

* fix(ragfs): serialize heartbeat lease refresh with release and report real conflict kind

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding
2026-07-29 19:45:34 +08:00
baojun-zhang 1841dfed81 Revert "refactor(pathlock):using rust implement instead python (#3557)" (#3597)
This reverts commit 6b538db569.
2026-07-29 11:31:41 +08:00
baojun-zhang 6b538db569 refactor(pathlock):using rust implement instead python (#3557)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design
2026-07-29 11:08:42 +08:00
baojun-zhang 6a33ebb7ca Optimize glob walkdir (#3013)
* feat(storage): optimize glob func

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* fix(localfs): offload blocking fs operations to spawn_blocking

* feat(glob): cap glob api default node_limit at 256

* feat(sdk): add node_limit options for glob in python and go SDKs
2026-07-06 21:39:16 +08:00
Haoyu Zhangandzhanghaoyu.la aa53e7aede feat: 实现 commit、restore、show 文件系统多版本管理功能 (#2756)
* feat: 实现 commit、restore、show 文件系统多版本管理功能

fix: 修复commit时删除文件

fix: commit 的 fast path 1 添加 Racy-clean 机制

fix: 将 sdk 中的 git 命令改为 snapshot 命令,同步修改单测

fix: 多版本管理的文件存储目录改为 .ovgit

feat: snapshot cli 渲染

fix: 修复 restore 时将删除的文件回滚时,目录不存在的问题

feat: restore 命令的 project_dir 参数改为可选,不传时默认全目录回滚

feat: 更新文档

fix: 删除暂未使用的配置参数

feat: 新增示例脚本

fix: 修复示例代码

fix: 修复 restore 返回的 task id 任务完成状态

feat: 在 restore 修改文件系统时加锁

fix: fix openviking_sdk

* feat: 将git多版本管理功能改为默认打开,并复用agfs的配置参数作为默认值

* fix: restore 命令改为先完成 ref 一致性协议再写回 VFS;object store 并发改为使用唯一 temp path

* fix: 在 git 配置检验层去除未实现的cas_mode = "redis_lock"模式

* fix: 在 Rust GitService 边界统一校验 account

* fix: 当前commit不支持通过 path 传入目录,增加报错信息

* fix: 将git文件默认存储路径统一为 .ovgit

* fix: restore 时写入 VFS 失败时返回详细的报错,并继续触发 reindex

* fix: 校验 commit、restore、show 的路径

* feat: 实现 commit 时指定目录

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
2026-06-25 11:28:04 +08:00
tuofangandfang 2d56bd71f4 feat(ragfs): add CachedFileSystem and Redis/Mooncake/Yuanrong cache providers (#2520)
* add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

* Handle poisoned known_keys mutexes in cache providers

* Add configurable cache support to ragfs python

* Add Redis cache provider support

* Prune native cache providers from default build

* Add RAGFS cache guides in Chinese and English

* delete .cargo/config.toml

* Fix stale cache invalidation across shared wrappers

* Document Yuanrong native concurrency limits

* add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

* Handle poisoned known_keys mutexes in cache providers

* Add configurable cache support to ragfs python

* Add Redis cache provider support

* Prune native cache providers from default build

* Add RAGFS cache guides in Chinese and English

* delete .cargo/config.toml

* Fix stale cache invalidation across shared wrappers

* Document Yuanrong native concurrency limits

* Limit directory cache entries and add regression test

* docs: add TOS s3fs backend test design

* Add runtime cache config override

* add build guides for mooncake and yuanrong

* fix _FakeConfig test with cache

* fix(ragfs): cache encrypted data below the encryption layer

- pass runtime cache config to the Rust binding
- cache ciphertext instead of decrypted content
- disable cache for encrypted multi-write mounts
- update tests and provider build documentation

* fix(ragfs): invalidate cache after same-mount raw copy

Ensure copy_within_mount invalidates the cached destination file and
parent directory after the raw backend fast path writes data directly.
This prevents stale reads and stale directory metadata when cache is
enabled and Python cp() uses the same-mount copy optimization.

Add a regression test covering overwrite copy followed by read/list.

* fix(ragfs): preserve multi-write discovery through cache layer

Allow MountableFS::as_multiwrite() to unwrap CachedFileSystem when
discovering the underlying MultiWriteWrappedFS. This preserves
multi-write admin paths and same-mount copy behavior for cache-enabled,
unencrypted multi-write mounts.

Add a regression test covering sync status, sync retry, same-mount copy,
and unmount behavior for cached unencrypted multi-write mounts.

* feat(ragfs): add cache-aware tree traversal mode

- add configurable tree traversal mode to cache policy
- keep default tree behavior delegated to backend
- allow cached traversal to reuse read_dir directory cache
- bypass cached traversal for multi-write backends
- add regression coverage for tree cache behavior and fallbacks

* docs: design cache-aware grep traversal

* feat(ragfs): add cache-aware grep traversal

Introduce a shared cache traversal mode for recursive APIs and use it to
optionally run grep through CachedFileSystem.

- add CacheTraversalMode with backend and cached_traversal modes
- keep CacheTreeMode as a compatibility alias
- route tree and grep through cached traversal only when explicitly enabled
- reuse cached read_dir entries and full-file reads during grep traversal
- keep multi-write traversal on the backend path
- expose storage.agfs.cache.traversal_mode in Python config
- raise max cached directory entries threshold to 4096
- add regression tests for grep cache traversal and traversal config

* Optimize cached grep generation validation

* Parallelize cached grep file scanning

---------

Co-authored-by: fang <fang@fangMacBook-Air.local>
2026-06-17 16:01:52 +08:00
baojun-zhang eff7e67037 feat(storage): support content-type auto-detecting in S3 case (#2668)
* feat(storage): support content-type auto-detecting in S3 case

* feat(storage): update code doc
2026-06-16 18:24:17 +08:00
baojun-zhang e492cbd16f refactor(encryption): using rust refactor encryption (#2444) 2026-06-05 17:26:26 +08:00
baojun-zhang 7e43428f04 refactor(storage): deprecated and remove agfs http mode client (#2425)
* refactor(storage): delete agfs http mode client

* refactor(storage): refactor protocol to support diff return type

* refactor(storage): refactor code format from lint
2026-06-04 17:36:43 +08:00
Hao Zheandhaozhelee 1194866e8f fix(rust): clean build warnings (#2362)
Co-authored-by: haozhelee <haozhelee@bytedance.com>
2026-06-01 20:53:55 +08:00
baojun-zhang 86f61ee16e perf(storage): optimize s3 backend grep performance (#2078)
* perf(storage): optimize s3 backend grep performance

* feat(storage): optimize git pr suggestion
2026-05-20 16:53:51 +08:00
baojun-zhang c8ff9e3575 perf(storage): optimize VikingFS grep implementation (#1731)
* perf(storage): optimize VikingFS grep implementation
- add ripgrep support and native local grep fallback for LocalFS mode
- normalize grep path handling and improve regression test coverage

* perf(storage): optimize VikingFS grep implementation
- add ripgrep support and native local grep fallback for LocalFS mode
- normalize grep path handling and improve regression test coverage

* perf(storage): format code

* perf(storage): support async grep

* feat(encryption): Push `exclude_uri` and `level_limit` down to the backend to ensure that the limits take effect after the same set of filtering conditions. &&  The default implementation also follows the same path contract as LocalFS, returning the path relative to the query root

* feat(encryption): Push `exclude_uri` and `level_limit` down to the backend to ensure that the limits take effect after the same set of filtering conditions. &&  The default implementation also follows the same path contract as LocalFS, returning the path relative to the query root

* fix(storage): relativize localfs exclude_path against grep query root && disable parent .gitignore inheritance in localfs rg fast path && add HTTP AGFS grep support for exclude_path and level_limit
2026-05-07 19:41:33 +08:00
Qin Haojie 62c023307d fix(docker): raise Rust MSRV for ragfs builds (#1448)
Align the Docker image, local build checks, and Rust crate metadata with the 1.91.1 toolchain required by ragfs-python's S3 dependencies.
2026-04-14 20:36:31 +08:00
MaojiaShengandopenviking dc052ce3e3 reorg: Rewrite agfs to ragfs with rust (#1221)
* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* fix: grep level limit

* fix: grep root

* fix: import error

* fix: rust code optimazation

* fix: CI error

* fix: CI go mod cache

* fix: grep level limit

* fix: CI

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-05 21:17:45 +08:00