[EN]
viking_fs.ls() defaults to node_limit=1000 to keep agent-facing tool
output from flooding the model's context. Several internal system
operations on the ingest / summary / vectorize path call ls() to
enumerate a directory's children and inherited that 1000 cap, so
importing a directory with more than 1000 entries silently processed
only the first 1000 and dropped the rest.
Observed: a 6,221-document import produced exactly 1000 subdirectories.
The namespace already held an earlier import, so temp->final
materialization took the incremental sync path
(_sync_topdown_recursive -> list_children) instead of the atomic
whole-directory move; list_children's ls() truncated the 6,221 temp
children to 1000 and the remaining 5,221 were dropped when temp was
deleted.
Fix: introduce a shared LS_ALL_NODES sentinel in viking_fs and pass it
explicitly at every internal call site that must observe every child.
ls()'s default stays 1000, so agent-facing listings are unchanged.
Call sites fixed:
- DirectoryParser._merge_temp / _recursive_move (parser temp merge)
- SemanticProcessor._sync_topdown_recursive (temp->final sync)
- SemanticProcessor._process_memory_directory (memory dirs)
- SemanticDagExecutor._list_dir (summary DAG dispatch + recursion)
- Summarizer.list_top_children (semantic-unit enqueue)
- embedding_utils.index_resource (per-directory file indexing)
Tests: tests/storage/test_ingest_ls_node_limit.py reproduces the >1000
truncation for both the temp->final sync materialization and the summary
DAG enumeration (red before, green after). Existing fakes updated to
accept the node_limit kwarg production now passes.
[中文]
viking_fs.ls() 默认 node_limit=1000,用于避免 agent 工具输出刷爆模型上下文。
但入库 / 摘要 / 向量化链路上多处内部系统调用 ls() 枚举目录子节点时也继承了
这个上限,导致目录条目超过 1000 时只处理前 1000 个、其余被静默丢弃。
现象:6221 篇文档入库后,目标命名空间下只剩正好 1000 个子目录。由于该命名
空间已存在更早的入库结果,temp->final 物化走了增量同步路径
(_sync_topdown_recursive -> list_children)而非原子整目录搬移;list_children
的 ls() 把 6221 个 temp 子节点截断到 1000,其余 5221 个在 temp 清理时丢失。
修复:在 viking_fs 中引入共享哨兵 LS_ALL_NODES,在每一处必须枚举全部子节点的
内部调用显式传入;ls() 默认值仍为 1000,agent-facing 的列目录行为不变。修复的
调用点见上方 Call sites。若不一并修摘要 DAG / sync,即使 raw 物化修好,1001 篇
之后的 L0/L1 摘要与向量索引仍会卡在 1000。
测试:tests/storage/test_ingest_ls_node_limit.py 复现 temp->final 物化同步与摘要
DAG 枚举两处的 >1000 截断(修复前 red、修复后 green);现有 fake 已更新以接受生产
代码新传入的 node_limit 参数。
Co-authored-by: chenpengfei <chenpengfei@bytedance.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Cache async SDK clients per event loop to avoid cross-loop reuse in worker threads.
Move memory vectorization into semantic queue refresh and preserve target sync state for resource updates.
Introduce typed lock leases for semantic queue handoff so resource, memory, and reindex flows can share or transfer lock ownership without releasing caller-owned locks prematurely.
* fix(resources): improve handling of resource imports and naming
- Add source_name to file upload requests for preserving original filenames
- Handle single-directory zip files by using their root directory directly
- Support viking://resources as parent directory for imports
- Split summarization for resources root imports into individual child items
- Add tests for new resource import behaviors
* style(tests): format test files with consistent line breaks
Improve readability by applying consistent line breaks in test file patches and removing trailing whitespace
* fix: decrypt raises 'Ciphertext too short' on plaintext files shorter than 4 bytes
The decrypt() method checked ciphertext length before checking the magic
header. This caused plaintext files shorter than 4 bytes (including empty
files) to raise InvalidMagicError('Ciphertext too short') before the
'is this plaintext?' check could return them as-is.
Fix: swap the order — check if content starts with OVE1 magic first,
then only check length for actual encrypted content.
This fixes failures when append_file() reads an empty messages.jsonl
session file and tries to decrypt it.
* fix: add strict=True to zip() in summarizer (B905 lint)
---------
Co-authored-by: yc111233 <yc111233@gmail.com>
The summarizer used uri.startswith("viking://memory/") to detect memory
URIs, but actual memory URIs are viking://user/{space}/memories/... and
viking://agent/{space}/memories/... — the prefix never matched, causing
all memories to be classified as context_type="resource" during reindex.
Replace the broken inline prefix check with a call to the existing
get_context_type_for_uri() from core/directories.py, which correctly
uses substring matching ("/memories" in uri) and handles all URI types
including session URIs.
Fixes#1060
* lisence: change the main lisence from Apache-2.0 to AGPL-v3
* lisence: change the main lisence from Apache-2.0 to AGPL-v3
* lisence: change the main lisence from Apache-2.0 to AGPL-v3
---------
Co-authored-by: openviking <openviking@example.com>
* feat(embedder): use summary for file embedding in semantic pipeline
When files are processed through the semantic pipeline (SemanticDag),
use the pre-generated summary (AST skeleton or LLM summary) for
embedding instead of reading raw file content. This ensures code files,
markdown, and other text files within a repository are indexed by their
semantic summary rather than truncated raw content.
- Add use_summary flag to VectorizeTask, _vectorize_single_file, and vectorize_file
- Set use_summary=True in _file_summary_task when a non-empty summary is available
- Truncate AST skeleton to max_skeleton_chars (12000 chars, ~3000 tokens) before embedding
- Add max_skeleton_chars config field to SemanticConfig
- index_resource and memory paths are unaffected (use_summary defaults to False)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(embedding): only use summary for code repo embedding, not plain text/doc files
Add `is_code_repo` flag to `SemanticMsg` and propagate it through the
pipeline so that summary-based embedding (AST skeleton) is only applied
when processing a code repository (`source_format == "repository"`).
For plain text, markdown, and other non-repo resources, raw file content
is used for embedding as before.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(storage): add transaction support with journal, undo, and crash recovery
Implement a full transaction system for VikingFS storage operations including
write-ahead journal, path locking, undo/rollback, context manager API, and
crash recovery. Includes comprehensive tests and documentation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(transaction): add e2e rollback tests for mv and multi-step operations
Add end-to-end tests covering rollback scenarios that were missing:
- mv rollback: file moved back to original location on failure
- mv commit: file persists at new location
- Multi-step rollback: mkdir + write + mkdir all reversed in order
- Partial step rollback: only completed entries are reversed
- Nested directory rollback: child removed before parent
- Best-effort rollback: single step failure does not block others
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(storage): add transaction support with path locking and journal
Implement transaction system for VikingFS with ACID-like guarantees:
- TransactionManager with configurable lock timeout and journal-based recovery
- PathLock supporting point, subtree, and mv lock modes
- Refactor VikingFS mv to use cp+rm to prevent lock files from being carried
- Fix stale lock detection returning false for missing lock files
- Update ragas eval to use LangchainLLMWrapper
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: tests
* fix(transaction): fix rollback and race condition bugs
- Reconstruct RequestContext from undo params for vectordb_delete/update_uri
rollback (previously skipped silently due to missing ctx)
- Serialize ctx fields into undo params in rm/mv operations
- Fix Phase 1 undo path to target archive dir instead of session root
- Remove Phase 2 fs_write_new undo (overwrites are idempotent, checkpoint
handles recovery)
- Add ancestor SUBTREE recheck after lock creation in acquire_subtree
- Move _collect_uris inside TransactionContext in rm/mv to close race window
- Log journal persistence failures instead of silently swallowing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(transaction): make TransactionManager required and rewrite tests with real backends
Remove all optional/fallback code paths where tx_manager could be None. get_transaction_manager()
now raises RuntimeError if not initialized. Fix undo rollback to reconstruct ctx for vectordb_upsert
and use correct agent_id default. Replace mock-based transaction tests with integration tests using
real AGFS and VectorDB backends.
* refactor(transaction): make rollback fully async and unify session commit path
- Convert execute_rollback/rollback_entry to async, removing sync run_async wrappers
- Unify Session.commit() to delegate to commit_async(), removing duplicate phase methods
- Fix SUBTREE lock to conflict with ancestor SUBTREE locks (was previously missing)
- Fix mv lock mode: directory moves now use SUBTREE on both source and destination
- Replace deprecated asyncio.get_event_loop() with get_running_loop()
- Remove max_parallel_locks config option
- Update docs (en/zh) and tests to match new async rollback signatures
* fix: tests
* refactor(transaction): simplify session commit and add redo-based crash recovery
Session commit no longer wraps archive phase in a transaction. Phase 2 uses
redo semantics so crashed memory-extraction can be replayed from archive.
PathLock stale-lock cleanup no longer redundantly re-checks timeout.
Semantic processor vectorization runs concurrently via asyncio.gather.
* fix: transaction
* fix: UserIdentifier
* refactor(transaction): replace undo-based transaction manager with lightweight lock + redo-log
Remove the heavyweight TransactionManager/Journal/UndoEntry system (~4000 lines) and
replace it with a simpler architecture: LockManager for path locking, LockContext as
the async context manager, LockHandle/LockOwner protocol, and a RedoLog for crash
recovery of session_memory operations. VikingFS rm/mv now use inline error handling
instead of rollback semantics. Updated docs, observers, and tests accordingly.
Co-Authored-By: Claude Opus 4.6
* fix(transaction): remove checkpoint dead code, fix TOCTOU race, clarify mv lock param
- Remove unused _write_checkpoint/_write_checkpoint_async/_read_checkpoint
from Session (superseded by redo-log)
- Re-resolve URI inside lock in resource_processor Phase 3.5 to prevent
concurrent add_resource calls from resolving to the same final_uri
- Rename acquire_mv dst_path to dst_parent_path with docstring to clarify
that callers pass the destination parent directory
* fix: path
* fix: resource lock
* fix: test
* docs: update
* fix: tests
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(semantic): add embedding task tracker and incremental update support
Implement embedding task tracker to monitor completion of vectorization tasks
Add support for incremental updates with file change status tracking
Optimize vectorization by reducing file reads with pre-computed status
Introduce VectorizeTask structure for better task management
Update semantic DAG executor to handle both full and incremental updates
* fix(semantic_processor): correct directory sync order and rename abstract to overview
Update the directory synchronization process to properly handle vector data files before directory deletion. Also rename 'abstract' parameter to 'overview' for clarity in the embedding utils.
* feat(embedding): add tracker decrement for failed enqueues
refactor(semantic_processor): simplify sync diff implementation
fix(summarizer): improve temp_uris empty check
test(embedding): add tests for tracker decrement cases
style(embedding_tracker): remove dataclass decorator
* fix(semantic_processor): change log level from warning to error for sync failures
refactor(tests): update vectorize methods to include semantic_msg_id parameter
test: add dummy tracker for embedding task tests
chore: remove obsolete test files
* fix: context overview
---------
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
* feat(resource): implement incremental update with COW pattern
Add support for incremental updates using copy-on-write pattern. Key changes include:
- Add ResourceLockManager for managing concurrent updates
- Introduce EmbeddingTaskTracker to track embedding task completion
- Modify TreeBuilder to support temp URIs and skip conflict resolution
- Update SemanticDagExecutor to handle incremental updates
- Extend memory extractor/compressor to work with temp URIs
- Add exists() method to VikingFS for URI existence checks
- Update Context to include temp_uri field
* refactor(resource_lock): clean up imports and improve code formatting
style: fix code formatting and whitespace issues across multiple files
feat(viking_fs): add copy_directory method for recursive directory copying
refactor(session): simplify temp URI creation and cleanup logic
style(memory_extractor): improve code formatting and line wrapping
refactor(semantic_processor): clean up imports and improve sync diff logic
style(compressor): fix code formatting and line wrapping
refactor(embedding_tracker): clean up code and improve logging
style(session): fix code formatting and whitespace issues
refactor(resource_lock): improve error handling and code organization
* refactor(storage): remove resource lock and improve semantic processing
- Remove ResourceLockManager and related lock handling code
- Simplify semantic processor by removing path locking mechanism
- Improve error handling and logging in sync operations
- Add new test files for storage components
- Clean up unused imports and update dependencies
* style(tests): clean up unused imports in test files
Remove unused imports across multiple test files to improve code cleanliness and reduce potential confusion. This includes removing unused mock objects, context classes, and constants that are not referenced in the tests.
* style: reformat code for better readability and consistency
Refactor long lines and adjust formatting to improve code readability. Changes include:
- Breaking long lines to adhere to line length limits
- Reformatting dictionary and list literals for consistency
- Adjusting indentation in multi-line statements
* refactor(sandbox): remove docker/aiosandbox backends, simplify SRT config
- Remove docker and aiosandbox from available backends
- Remove settings_path from SrtBackendConfig (now auto-generated in workspace)
- Update SRT settings path to workspace/sandboxes/{session}-srt-settings.json
- Update README examples to use srt backend and remove settingsPath
* refactor(sandbox): remove docker/aiosandbox backends, simplify SRT config
- Remove docker and aiosandbox from available backends
- Remove settings_path from SrtBackendConfig (now auto-generated in workspace)
- Update SRT settings path to workspace/sandboxes/{session}-srt-settings.json
- Update README examples to use srt backend and remove settingsPath
* fix: remove unused handle_chat_direct function and fix unused logs variable
* Fix UTF-8 issues in chat command
* Add tab indentation to Think, Calling, and Result lines in CLI output
* Add first release workflow
* Update release workflow with correct working directory
* 修改 SessionKey 构建逻辑：统一使用 type="cli"，channel_id 默认 "default"，chat_id 作为 session_id 使用
* Implement machine unique ID as default session ID for ov chat
* Remove unsupported --logs parameter from chat command
* 统一 Python 和 Rust CLI 的默认 session ID 生成逻辑
* 修改日志
* 去掉log依赖
* docs: add VikingBot quick start section to READMEs
* fix: use vikingbot chat instead of ov chat in READMEs
* Revert "fix: use vikingbot chat instead of ov chat in READMEs"
This reverts commit 59f4e87ba0.
* fix: use UUID v4 for machine ID generation in both Rust and Python
* refactor: move truncate_utf8 to utils, fix chat history path, and use BotProcess dataclass
* refactor: update machine ID generation and remove unused chat_v2
- Update Python to use py-machineid library
- Update Rust to use machine-uid crate
- Remove unused chat_v2.rs
- Move machine ID from file storage to system-provided IDs
- Add fallback to "default" if system ID is unavailable
* 优化格式
* ruff format .
Kchen