Commit Graph
25 Commits
Author SHA1 Message Date
baojun-zhang df1557901e feat(ragfs): add structured lock tracing logs (#3693)
* feat(ragfs): add structured lock tracing logs

* feat(ragfs): add log async writer

* fix(ragfs): add CRITICAL trans && add rust hook to receive rotate log file
2026-08-03 15:58:55 +08:00
baojun-zhang 2f9451231e refactor(pathlock):using rust implement instead python (#3602)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design

* fix(ragfs):fix(ragfs): use non-blocking fcntl locks for localfs CAS

* fix(ragfs): serialize heartbeat lease refresh with release and report real conflict kind

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding

* fix(ragfs): preserve conflict kind snapshot and drop unused test scaffolding
2026-07-29 19:45:34 +08:00
baojun-zhang 1841dfed81 Revert "refactor(pathlock):using rust implement instead python (#3557)" (#3597)
This reverts commit 6b538db569.
2026-07-29 11:31:41 +08:00
baojun-zhang 6b538db569 refactor(pathlock):using rust implement instead python (#3557)
* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):using rust implement instead python

* refactor(pathlock):optimize unit test code

* refactor(pathlock):optimize encryption create func

* refactor(pathlock):avoid releasing handoffed pathlock on enqueue errors

* fix(pathlock): use owned lease capability and handle S3 create-new 409 as conflict

* fix(ragfs): keep original FsContext for multi-write metadata

* fix(pathlock): resolve lease coverage and CAS handling issues

- detect S3 conditional conflicts from structured service errors
- pass transaction leases when deleting skill roots
- let temp cleanup acquire locks for temp paths
- disambiguate cache and pathlock providers in cache tests
- update temp cleanup lease assertions

* fix(ragfs): bypass pathlock for multi-write metadata

* fix(ragfs): revert pathlock fail-fast design
2026-07-29 11:08:42 +08:00
agent 27debfe7a8 feat(snapshot): add path diff API, SDK, and CLI (#3451)
* feat(snapshot): add path diff API

* feat(snapshot): expose path diff in clients and CLI

* fix(snapshot): pin path diff to resolved commits

* fix(snapshot): bound path diff resource usage

* fix(snapshot): bound diff blob reads

* fix(snapshot): harden path diff API

* fix(snapshot): bound path diff resource usage

* fix(build): sync native binding dependencies
2026-07-23 17:45:36 +08:00
Qin Haojie d47f2106ee refactor: remove unused and deprecated APIs (#3272)
Delete dead compatibility paths and test-only helpers so unsupported APIs do not remain as accidental contracts.
2026-07-16 10:49:56 +08:00
baojun-zhang 6a33ebb7ca Optimize glob walkdir (#3013)
* feat(storage): optimize glob func

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* feat(rgafs): implement paged glob traversal without full tree materialization

* fix(localfs): offload blocking fs operations to spawn_blocking

* feat(glob): cap glob api default node_limit at 256

* feat(sdk): add node_limit options for glob in python and go SDKs
2026-07-06 21:39:16 +08:00
Haoyu Zhangandzhanghaoyu.la aa53e7aede feat: 实现 commit、restore、show 文件系统多版本管理功能 (#2756)
* feat: 实现 commit、restore、show 文件系统多版本管理功能

fix: 修复commit时删除文件

fix: commit 的 fast path 1 添加 Racy-clean 机制

fix: 将 sdk 中的 git 命令改为 snapshot 命令,同步修改单测

fix: 多版本管理的文件存储目录改为 .ovgit

feat: snapshot cli 渲染

fix: 修复 restore 时将删除的文件回滚时,目录不存在的问题

feat: restore 命令的 project_dir 参数改为可选,不传时默认全目录回滚

feat: 更新文档

fix: 删除暂未使用的配置参数

feat: 新增示例脚本

fix: 修复示例代码

fix: 修复 restore 返回的 task id 任务完成状态

feat: 在 restore 修改文件系统时加锁

fix: fix openviking_sdk

* feat: 将git多版本管理功能改为默认打开,并复用agfs的配置参数作为默认值

* fix: restore 命令改为先完成 ref 一致性协议再写回 VFS;object store 并发改为使用唯一 temp path

* fix: 在 git 配置检验层去除未实现的cas_mode = "redis_lock"模式

* fix: 在 Rust GitService 边界统一校验 account

* fix: 当前commit不支持通过 path 传入目录,增加报错信息

* fix: 将git文件默认存储路径统一为 .ovgit

* fix: restore 时写入 VFS 失败时返回详细的报错,并继续触发 reindex

* fix: 校验 commit、restore、show 的路径

* feat: 实现 commit 时指定目录

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
2026-06-25 11:28:04 +08:00
Jiahui Zhou 27e90ad9cd Feature/search telemetry relations control (#2547)
* Add search retrieval telemetry breakdown

* Fix pyagfs helper annotation imports

* docs: document search relation controls and telemetry fields

Document the new include_relations request parameter and the search telemetry summary fields so the API docs stay aligned with the latest retrieval changes.

* refactor(search): drop relation enrichment and trim telemetry

Remove relation fetching from the retrieval path and delete low-value search telemetry fields so retrieval stays simpler and the telemetry summary focuses on actionable diagnostics.
2026-06-10 20:00:47 +08:00
baojun-zhang 7ee1481e1d feature(storage): support multi write storage (#2466)
* feature(storage): support multi write storage

* refactor(storage): simplify multiwrite logic and consolidate test helpers

* refactor(storage): extract multibackend and shape modules and tighten multi-write wrapper boundaries

* refactor(storage): refactor write pipeline
2026-06-08 11:12:15 +08:00
baojun-zhang e492cbd16f refactor(encryption): using rust refactor encryption (#2444) 2026-06-05 17:26:26 +08:00
baojun-zhang 7e43428f04 refactor(storage): deprecated and remove agfs http mode client (#2425)
* refactor(storage): delete agfs http mode client

* refactor(storage): refactor protocol to support diff return type

* refactor(storage): refactor code format from lint
2026-06-04 17:36:43 +08:00
baojun-zhang 073e929a1a feat(storage): optimize tree func (#2372)
* feat(storage): optimize tree func

* feat(storage): add test case && fix tree show parent dir problem

* feat(storage): format code

* feat(storage): fix check problem

* feat(storage): remove redundant unit test

* feat(storage): fix code review issue

* feat(storage): fix code review issue
2026-06-03 18:21:55 +08:00
Qin Haojie 2406c0e6e1 refactor(storage): 异步化存储锁与 IO (#2143)
* refactor(storage): async storage lock IO

Move AGFS and storage lock paths onto async wrappers while preserving lock handoff semantics.

* refactor: streamline async task tracking

Collapse TaskTracker lifecycle operations into async-only APIs and align callers/tests with the new boundary. Also throttle repeated memory/path lock wait warnings to reduce noisy retry logs.
2026-05-22 14:03:18 +08:00
MaojiaSheng a1c9e5080b feat: add ov observer filesystem, and optimize mkdir frequency (#2045)
* feat: ov add-resource (spec -L --level), ov stat (return count for dir)

* feat: ov add-resource (spec -L --level), ov stat (return count for dir)

* feat: Add VLM backup configuration for automatic failover

- Add backup field to VLMConfig with recursive backup prevention
- Implement FailoverVLM wrapper class for automatic failover
- Support rate limit, timeout, server error triggers
- Add comprehensive unit tests

* feat: Add VLM backup configuration for automatic failover

* feat: Add VLM backup configuration for automatic failover

* ov observer filesystem

* ov observer filesystem
2026-05-14 18:21:26 +08:00
baojun-zhang c8ff9e3575 perf(storage): optimize VikingFS grep implementation (#1731)
* perf(storage): optimize VikingFS grep implementation
- add ripgrep support and native local grep fallback for LocalFS mode
- normalize grep path handling and improve regression test coverage

* perf(storage): optimize VikingFS grep implementation
- add ripgrep support and native local grep fallback for LocalFS mode
- normalize grep path handling and improve regression test coverage

* perf(storage): format code

* perf(storage): support async grep

* feat(encryption): Push `exclude_uri` and `level_limit` down to the backend to ensure that the limits take effect after the same set of filtering conditions. &&  The default implementation also follows the same path contract as LocalFS, returning the path relative to the query root

* feat(encryption): Push `exclude_uri` and `level_limit` down to the backend to ensure that the limits take effect after the same set of filtering conditions. &&  The default implementation also follows the same path contract as LocalFS, returning the path relative to the query root

* fix(storage): relativize localfs exclude_path against grep query root && disable parent .gitignore inheritance in localfs rg fast path && add HTTP AGFS grep support for exclude_path and level_limit
2026-05-07 19:41:33 +08:00
euyua9 2fb5cc1ec6 fix: reject mismatched ragfs cpython extensions (#1854)
* fix: reject mismatched ragfs cpython extensions

* fix: reject windows ragfs cpython tag drift
2026-05-06 11:31:11 +08:00
Qin Haojie 964998daca fix(ragfs): load Windows abi3 pyd artifact (#1801) 2026-04-29 20:10:07 +08:00
Qin Haojieanddingben.db@bytedance.com <dingben.db@bytedance.com@bytedance.com> 6aa62c1b87 fix(ragfs): 仅加载 abi3 binding 产物 (#1756)
* feat(agfs): support overriding queuefs db_path via config

Add a new optional `storage.agfs.queue_db_path` field to AGFSConfig so
the queuefs sqlite database can be relocated outside the workspace via
config. This helps when the workspace volume does not support sqlite
(e.g. some network filesystems returning disk I/O errors on PRAGMA/WAL),
allowing the queue db to be placed on a local path like `/tmp/queue.db`.

When not set, the default path `{storage.workspace}/_system/queue/queue.db`
is preserved, keeping backwards compatibility.

* fix(ragfs): load only abi3 binding artifacts

Avoid stale cpython-specific ragfs_python artifacts shadowing rebuilt abi3 extensions so queuefs sqlite persistence uses the current Rust binding.

---------

Co-authored-by: dingben.db@bytedance.com <dingben.db@bytedance.com@bytedance.com>
2026-04-27 21:30:44 +08:00
MaojiaShengandopenviking 21ba377a9c fix: use abi3 for rust package (#1517)
* fix: use abi3 for rust package

* fix: modify list scope

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-17 13:58:23 +08:00
MaojiaSheng 95cc0f84d0 reorg: split parser layer to 2-layer: accessor and parser, so that we can reuse more code (#1428) 2026-04-15 10:33:12 +08:00
MaojiaShengandopenviking a7e5417ef2 reorg: remove golang depends (#1339)
* docs: fix docker deployment

* reorg: remove third_party/agfs

* feat(s3fs): add disable_batch_delete option for OSS compatibility

Port of PR #1333 from Go version to Rust:

- Add disable_batch_delete config option to S3Client
- When enabled, use sequential single-object deletes instead of DeleteObjects
- This is for S3-compatible services like Alibaba Cloud OSS that require
  Content-MD5 for DeleteObjects but AWS SDK v2 does not send it by default
- Add documentation and config example for OSS

* fix(s3fs): pass disable_batch_delete config from Python to Rust

Add disable_batch_delete to the s3_plugin_config dict in _generate_plugin_config
so that the Python config can properly control the Rust S3FS plugin's behavior.

* reorg: remove third_party/agfs

* reorg: remove third_party/agfs

* change some docs

* change some docs

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-10 15:16:29 +08:00
MaojiaShengandopenviking dc052ce3e3 reorg: Rewrite agfs to ragfs with rust (#1221)
* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* reorg: rewrite agfs with rust, and named with ragfs, keep License

* fix: grep level limit

* fix: grep root

* fix: import error

* fix: rust code optimazation

* fix: CI error

* fix: CI go mod cache

* fix: grep level limit

* fix: CI

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-05 21:17:45 +08:00
Maojia ShengandMaojia Sheng 50323dfa6c fix: grep for binding-client (#474)
Co-authored-by: Maojia Sheng <shengmaojia@bytedance.com>
2026-03-08 00:59:12 +08:00
chuanbao666 13a3b74188 fix(agfs): fix agfs binding-client import error (#458)
* fix(agfs): fix agfs binding-client import error

* fix: agfs-client默认使用binding-client
2026-03-06 14:55:16 +08:00