mirror of
https://github.com/volcengine/OpenViking.git
synced 2026-09-30 09:17:51 +08:00
tuofangandfang
2d56bd71f4
* add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide * Handle poisoned known_keys mutexes in cache providers * Add configurable cache support to ragfs python * Add Redis cache provider support * Prune native cache providers from default build * Add RAGFS cache guides in Chinese and English * delete .cargo/config.toml * Fix stale cache invalidation across shared wrappers * Document Yuanrong native concurrency limits * add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide * Handle poisoned known_keys mutexes in cache providers * Add configurable cache support to ragfs python * Add Redis cache provider support * Prune native cache providers from default build * Add RAGFS cache guides in Chinese and English * delete .cargo/config.toml * Fix stale cache invalidation across shared wrappers * Document Yuanrong native concurrency limits * Limit directory cache entries and add regression test * docs: add TOS s3fs backend test design * Add runtime cache config override * add build guides for mooncake and yuanrong * fix _FakeConfig test with cache * fix(ragfs): cache encrypted data below the encryption layer - pass runtime cache config to the Rust binding - cache ciphertext instead of decrypted content - disable cache for encrypted multi-write mounts - update tests and provider build documentation * fix(ragfs): invalidate cache after same-mount raw copy Ensure copy_within_mount invalidates the cached destination file and parent directory after the raw backend fast path writes data directly. This prevents stale reads and stale directory metadata when cache is enabled and Python cp() uses the same-mount copy optimization. Add a regression test covering overwrite copy followed by read/list. * fix(ragfs): preserve multi-write discovery through cache layer Allow MountableFS::as_multiwrite() to unwrap CachedFileSystem when discovering the underlying MultiWriteWrappedFS. This preserves multi-write admin paths and same-mount copy behavior for cache-enabled, unencrypted multi-write mounts. Add a regression test covering sync status, sync retry, same-mount copy, and unmount behavior for cached unencrypted multi-write mounts. * feat(ragfs): add cache-aware tree traversal mode - add configurable tree traversal mode to cache policy - keep default tree behavior delegated to backend - allow cached traversal to reuse read_dir directory cache - bypass cached traversal for multi-write backends - add regression coverage for tree cache behavior and fallbacks * docs: design cache-aware grep traversal * feat(ragfs): add cache-aware grep traversal Introduce a shared cache traversal mode for recursive APIs and use it to optionally run grep through CachedFileSystem. - add CacheTraversalMode with backend and cached_traversal modes - keep CacheTreeMode as a compatibility alias - route tree and grep through cached traversal only when explicitly enabled - reuse cached read_dir entries and full-file reads during grep traversal - keep multi-write traversal on the backend path - expose storage.agfs.cache.traversal_mode in Python config - raise max cached directory entries threshold to 4096 - add regression tests for grep cache traversal and traversal config * Optimize cached grep generation validation * Parallelize cached grep file scanning --------- Co-authored-by: fang <fang@fangMacBook-Air.local>
RAGFS Mooncake Cache Provider
This crate maps the RAGFS CacheProvider contract to Mooncake Store without
changing CachedFileSystem.
The optional official binding is pinned to Mooncake commit:
1352bbec43081e461356aaecf6c70cddd826b455
Default tests use the same synchronous object boundary as the native binding, so they run without Mooncake services or C++ libraries:
cargo test -p ragfs-cache-mooncake
Native Build
Build the matching Mooncake checkout with Store Rust support:
cmake -S /path/to/Mooncake -B /path/to/Mooncake/build \
-DWITH_STORE=ON \
-DWITH_STORE_RUST=ON
cmake --build /path/to/Mooncake/build \
--target build_mooncake_store_rust -j
Before building OpenViking, provide the paths expected by the official crate:
export MOONCAKE_BUILD_DIR=/path/to/Mooncake/build
export MOONCAKE_STORE_LIB_DIR=$MOONCAKE_BUILD_DIR/mooncake-store/src
export MOONCAKE_STORE_INCLUDE_DIR=/path/to/Mooncake/mooncake-store/include
export LD_LIBRARY_PATH="$MOONCAKE_STORE_LIB_DIR:$LD_LIBRARY_PATH"
Compile the adapter against the official binding:
cargo check -p ragfs-cache-mooncake --features mooncake-native
TCP Smoke Test
With Mooncake Master and its metadata service running:
OPENVIKING_RUN_MOONCAKE_INTEGRATION=true \
MOONCAKE_LOCAL_HOSTNAME=127.0.0.1 \
MOONCAKE_METADATA_SERVER=http://127.0.0.1:8080/metadata \
MOONCAKE_MASTER_SERVER_ADDR=127.0.0.1:50051 \
MOONCAKE_PROTOCOL=tcp \
cargo test -p ragfs-cache-mooncake \
--features mooncake-native \
--test native_smoke -- --nocapture
The test writes one complete object, reads it back, removes it, verifies the miss result, and closes the provider.
Docker Smoke Test
The repository includes a Linux ARM64 image that builds the pinned Mooncake commit, starts its HTTP metadata service and Master, runs Mooncake's official Rust smoke test, and then runs the OpenViking provider smoke test:
docker build \
-t openviking-mooncake-test:arm64 \
docker/mooncake-test
docker run --rm --shm-size=4g \
-v "$PWD:/workspace/OpenViking" \
-v openviking-cargo-registry:/root/.cargo/registry \
-v openviking-cargo-git:/root/.cargo/git \
-v openviking-cargo-target:/workspace/OpenViking/target-docker \
-e CARGO_TARGET_DIR=/workspace/OpenViking/target-docker \
openviking-mooncake-test:arm64