Commit Graph
4 Commits
Author SHA1 Message Date
Hao Zheandzhiheng.liu de9c3cec15 fix(storage): preserve deletion and session durability (#3553)
* fix(ragfs): preserve cache visibility on partial S3 deletes

Surface exact and per-object S3 deletion failures, while always invalidating the affected directory and stat cache scope after a recursive delete attempt.

Source-PR: #3407
Original-Commit: 8d6addf28e

* fix(session): preserve legacy policy and peer identity compatibility

Parse string false and other legacy boolean-like memory policy values without silently enabling extraction or breaking persisted configs. Encode mixed-script peers losslessly, while retaining their former lossy IDs as read-only retrieval and extraction aliases.

Source-PR: #3422
Original-Commit: 0dfd5a9ed9

* fix(memory): drain timer flush tasks during shutdown

Retain the shielded timer flush task and await it when close cancels the timer loop, so batch failures are observed and submitters are resolved without unhandled task exceptions.

Source-PR: #3438
Original-Commit: ca1d74e164

* fix(storage): preserve peer isolation and cache correctness

* fix(ingest): reserve encoded peer namespace

* ci: skip embedding-dependent resource test without secrets

* fix(ragfs): invalidate caches after partial remove

---------

Co-authored-by: zhiheng.liu <zhiheng.liu@bytedance.com>
2026-07-31 21:45:29 +08:00
17631bd353 feat: support ignoring certain files in the multi-version management function, similar to the git ignore feature (#2930)
* docs: add ovgitignore design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: add ovgitignore implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(git): add ovgitignore matcher

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(git): apply ovgitignore during commits

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(pyagfs): expose ovgitignore commit results

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(vikingfs): manage account ovgitignore

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(git): document ovgitignore semantics

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(git): enable ovgitignore restore assertion and document python api

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: fix gitignore

* fix: fix gitignore

* feat: 新增 git ignore 相关的接口

* fix: 修复 cli 命令渲染

* doc: 删除执行计划文档

* doc: 删除执行计划文档

* feat: 在 http client 中新增 git ignore 相关接口

* fix: 修改文档中关于 .ovgitignore 被版本化的内容

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-02 11:45:38 +08:00
Haoyu Zhangandzhanghaoyu.la aa53e7aede feat: 实现 commit、restore、show 文件系统多版本管理功能 (#2756)
* feat: 实现 commit、restore、show 文件系统多版本管理功能

fix: 修复commit时删除文件

fix: commit 的 fast path 1 添加 Racy-clean 机制

fix: 将 sdk 中的 git 命令改为 snapshot 命令,同步修改单测

fix: 多版本管理的文件存储目录改为 .ovgit

feat: snapshot cli 渲染

fix: 修复 restore 时将删除的文件回滚时,目录不存在的问题

feat: restore 命令的 project_dir 参数改为可选,不传时默认全目录回滚

feat: 更新文档

fix: 删除暂未使用的配置参数

feat: 新增示例脚本

fix: 修复示例代码

fix: 修复 restore 返回的 task id 任务完成状态

feat: 在 restore 修改文件系统时加锁

fix: fix openviking_sdk

* feat: 将git多版本管理功能改为默认打开,并复用agfs的配置参数作为默认值

* fix: restore 命令改为先完成 ref 一致性协议再写回 VFS;object store 并发改为使用唯一 temp path

* fix: 在 git 配置检验层去除未实现的cas_mode = "redis_lock"模式

* fix: 在 Rust GitService 边界统一校验 account

* fix: 当前commit不支持通过 path 传入目录,增加报错信息

* fix: 将git文件默认存储路径统一为 .ovgit

* fix: restore 时写入 VFS 失败时返回详细的报错,并继续触发 reindex

* fix: 校验 commit、restore、show 的路径

* feat: 实现 commit 时指定目录

---------

Co-authored-by: zhanghaoyu.la <zhanghaoyu.la@bytedance.com>
2026-06-25 11:28:04 +08:00
tuofangandfang 2d56bd71f4 feat(ragfs): add CachedFileSystem and Redis/Mooncake/Yuanrong cache providers (#2520)
* add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

* Handle poisoned known_keys mutexes in cache providers

* Add configurable cache support to ragfs python

* Add Redis cache provider support

* Prune native cache providers from default build

* Add RAGFS cache guides in Chinese and English

* delete .cargo/config.toml

* Fix stale cache invalidation across shared wrappers

* Document Yuanrong native concurrency limits

* add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide

* Handle poisoned known_keys mutexes in cache providers

* Add configurable cache support to ragfs python

* Add Redis cache provider support

* Prune native cache providers from default build

* Add RAGFS cache guides in Chinese and English

* delete .cargo/config.toml

* Fix stale cache invalidation across shared wrappers

* Document Yuanrong native concurrency limits

* Limit directory cache entries and add regression test

* docs: add TOS s3fs backend test design

* Add runtime cache config override

* add build guides for mooncake and yuanrong

* fix _FakeConfig test with cache

* fix(ragfs): cache encrypted data below the encryption layer

- pass runtime cache config to the Rust binding
- cache ciphertext instead of decrypted content
- disable cache for encrypted multi-write mounts
- update tests and provider build documentation

* fix(ragfs): invalidate cache after same-mount raw copy

Ensure copy_within_mount invalidates the cached destination file and
parent directory after the raw backend fast path writes data directly.
This prevents stale reads and stale directory metadata when cache is
enabled and Python cp() uses the same-mount copy optimization.

Add a regression test covering overwrite copy followed by read/list.

* fix(ragfs): preserve multi-write discovery through cache layer

Allow MountableFS::as_multiwrite() to unwrap CachedFileSystem when
discovering the underlying MultiWriteWrappedFS. This preserves
multi-write admin paths and same-mount copy behavior for cache-enabled,
unencrypted multi-write mounts.

Add a regression test covering sync status, sync retry, same-mount copy,
and unmount behavior for cached unencrypted multi-write mounts.

* feat(ragfs): add cache-aware tree traversal mode

- add configurable tree traversal mode to cache policy
- keep default tree behavior delegated to backend
- allow cached traversal to reuse read_dir directory cache
- bypass cached traversal for multi-write backends
- add regression coverage for tree cache behavior and fallbacks

* docs: design cache-aware grep traversal

* feat(ragfs): add cache-aware grep traversal

Introduce a shared cache traversal mode for recursive APIs and use it to
optionally run grep through CachedFileSystem.

- add CacheTraversalMode with backend and cached_traversal modes
- keep CacheTreeMode as a compatibility alias
- route tree and grep through cached traversal only when explicitly enabled
- reuse cached read_dir entries and full-file reads during grep traversal
- keep multi-write traversal on the backend path
- expose storage.agfs.cache.traversal_mode in Python config
- raise max cached directory entries threshold to 4096
- add regression tests for grep cache traversal and traversal config

* Optimize cached grep generation validation

* Parallelize cached grep file scanning

---------

Co-authored-by: fang <fang@fangMacBook-Air.local>
2026-06-17 16:01:52 +08:00