chenjwandTRAE CLI a09a9d20a8 feat(compile): 通过 --skill memory 支持记忆整理模式 (#5178)
* feat(compile): add memory consolidation mode via `--skill memory`

Add an in-process memory consolidation mode to `ov compile`. When `--skill`
is the sentinel value `memory`, CompileService runs the existing memory
framework (ConsolidationExtractContextProvider -> ExtractLoop -> MemoryUpdater)
directly inside OpenViking core to dedup/merge/split/compact an existing
memory-type directory in place, conforming to that type's schema. The default
skill path (VikingBot Wiki compile) is unchanged.

Highlights:
- New ConsolidationExtractContextProvider: agentic exploration with ls/search/read
  tools seeded by a recursive listing; single schema inferred from --to; space
  (self/peer) taken from the canonical --to URI so listing never depends on an
  empty ctx.user_id.
- MemoryCompileRunner: session.commit-lite task shape (task_tracker + one
  in-process asyncio.Task, no QueueFS re-delivery), bound to a root span so a
  trace_id is recorded; result reports adds/updates/deletes (file URIs only,
  memory_diff.json semantics) classified via read_file_contents.
- MemoryLsTool: add recursive listing (relative paths, 500-node cap with
  truncation) and stop hiding subdirectories so subfoldered dirs are not
  misreported as empty. Only the compile provider exposes ls, so session.commit
  is unaffected.
- CLI: --from optional (required for normal mode, rejected for memory mode);
  help gains a memory example.
- Fix a syntax regression in crates/ragfs/src/lock/provider.rs test module that
  blocked `make build` (unrelated to compile; introduced by #4908).
- Docs: document the memory mode in ov-compile-design.md.
- Tests: unit tests for provider/runner/request validation; integration script
  test_compile_memory_xiaomei.py with merge/split/dedup/preferences cases.

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* docs(compile): document memory consolidation mode

Add a dedicated user-facing page (zh + en) for `ov compile --skill memory`
covering when to use it, usage, parameters, behavior, and the adds/updates/
deletes result. Link it from the context-compilation overview. The VitePress
sidebar picks the new page up automatically from the directory listing.

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* fix(compile): honor language and cancellation

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: migrate files when URI fields change

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: materialize URI moves at apply time

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: inherit source links on explicit merge

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* test(memory): cover rename conflict and streaming migration

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* test(compile): assert URI migration diff semantics

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* docs(compile): clarify cross-type link migration

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: fail rename on target read errors

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: preserve omitted URI identity fields

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: rebase URI moves on latest source

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: protect occupied empty rename targets

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: reject case-only URI moves

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* docs(compile): document URI migration semantics

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: validate explicit replacement targets

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: reject case-only replacement moves

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: rerender managed links after URI moves

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* test(compile): add repeatable Chinese URI rename case

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* test(compile): cover bidirectional URI renames

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: remove empty directories after URI moves

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory(prompts): lowercase filename identity segments

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: consolidate memories-root in one extract loop

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* docs(compile): document memory-root consolidation and failure semantics

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* test(compile): add memory_root and deterministic rename cases

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory(compile): honor Account templates and preserve merged duplicate links

- MemoryCompileRunner._consolidate now resolves the Account-level memory
  template snapshot via resolve_account_memory_registry(), matching the
  session.commit extract path, so consolidation no longer overwrites
  operator-customized content templates with deployment defaults.
- _inherit_deleted_link_relations now tracks the deleted source URI for
  each inherited link. Implicit rename targets only exclude contributions
  copied from their own migration source, so links unique to a duplicate
  merged into the same target (delete_replacements) are folded in and the
  neighbor backlinks match.
- Regression tests cover the Account-template snapshot and a same-batch
  rename+merge where only the duplicate holds a link to a third file.

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: preserve rename sources through session commit queue

- ExtractLoop._updated_uri_for_existing_operation only considers an
  identity-field change a rename when the merged value actually differs
  from the current one, and returns the source URI unchanged when the
  regenerated candidate matches it. Legacy paths whose new template
  differs only in case (e.g. preferences user "Alice") no longer trip
  the case-only migration guard on plain content updates.
- clone_operation_for_uri no longer drops old_memory_file_content when
  the target URI differs from the source. The upstream
  _materialize_uri_migrations still detects the mismatch and generates
  a write-new + delete-old migration, but the queue clone keeps the
  source content so the migration can inherit prior body and links
  instead of turning a rename into an empty new record.
- python_protocol contract preamble drops the misleading "Unknown
  business fields are ignored" clause; unknown fields raise at parse
  time, so the note was inaccurate.
- Regression tests cover legacy case-only preferences updates going
  through ExtractLoop and split_request_by_merge_group preserving the
  rename source for the add + delete pair.

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* vikingbot(cli): pin openviking log stream to stderr before eager imports

vikingbot chat --eval / -e commits to a stdout JSON contract. Downstream
consumers like benchmark/locomo/vikingbot/run_eval.py parse stdout with
json.loads, so any log line on stdout breaks the parse and drops
token_usage and iteration to defaults (0). That is what caused
Total prompt tokens=0 and Avg iteration=0 in the LoCoMo summary.

Move the openviking / openviking_cli log redirection into a module-level
_preimport_redirect_openviking_logs_to_stderr() that runs before any
vikingbot.agent.* or openviking.* imports. Those imports call get_logger()
at module load time, which loads ov.conf and can emit warnings (e.g.
"Ignoring unknown config field") through openviking_cli's shared
QueueListener whose default output is stdout. Force the "stdout" listener
plus real StreamHandler pair into existence up front and rebind its
stream to stderr. Also move the in-chat() redirect ahead of ensure_config
and extend it to swap the shared stdout handler as a second-line guard.

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory(python-protocol): silently drop unknown-field edits

Extraction DSL programs occasionally reference a field name that does not
exist in the memory schema. Failing the whole program on this is
brittle: unrelated valid statements in the same commit are lost. Match
the tolerance kwargs already have on create()/set()/update() and treat
unknown-field attribute access as a compile-time no-op: return a
_FieldHandle flagged is_noop=True, and skip any .update()/.edit()/.drop()
chained on it plus the final _apply_field_handle. Sibling operations on
real fields keep applying.

Update the corresponding regression tests: replace the literal-`field`
placeholder rejection test with two new cases asserting the whole program
still commits and a real content.edit() still lands when a bogus field
appears in the same batch.

Co-authored-by: TRAE CLI <traecli@bytedance.com>

* memory: tree-lock delete parents so overview cleanup covers empty dir rm

`generate_overview` recursively removes a memory directory once the last file
is deleted, but `_operation_tree_lock_paths` only tree-locked rename source
parents. Batches that plain-deleted the last file in a directory ran the
follow-up `rm -r` under a lease that did not cover the parent, and RAGFS
rejected the request with "pathlock lease ref does not cover the requested
operation". Add each delete's parent directory (unless a same-directory
rename replaces it) to the tree-lock set.

Co-authored-by: TRAE CLI <traecli@bytedance.com>

---------

Co-authored-by: TRAE CLI <traecli@bytedance.com>
2026-09-26 16:30:28 +08:00
2026-01-29 20:29:19 +08:00

OpenViking

OpenViking: The Context Database for AI Agents

English / 中文 / 日本語

Website · Live Demo · GitHub · Issues · Docs

release stars issues contributors license last commit

Deploy on Railway

Lark Lark · WeChat WeChat · Discord Discord · X X

volcengine%2FOpenViking | Trendshift


What is OpenViking

OpenViking is an open-source context database for AI agents — one filesystem for everything an agent knows: knowledge, memory, and skills.

Most agent memory is a black box: text goes in, embeddings come out, and nobody can see what was actually stored. OpenViking organizes context as a virtual filesystem under viking:// instead. Agents navigate it like files — ls, tree, read, write, grep — and you can open any directory to inspect and edit what your agent knows. Every directory carries a generated summary, so agents can scan summaries first and decide what to read.

OpenViking Studio: browse context and try semantic search

Try OpenViking Studio in your browser, no installation required. Self-host Web Studio.

Why OpenViking

  • One filesystem for knowledge, memory, and skills. Resources hold documents and code; memories retain user preferences and experience; skills define how to perform tasks — not just extracted facts, but the full context, each with a viking:// URI for browsing and retrieval. → Viking URI · Context types
  • Search a directory, not the whole index. Scope semantic search to a project or memory subtree instead of scanning a flat vector pool. find runs a query directly; search plans retrieval from session context. → Retrieval
  • Read the summary before the source. Generated directory abstracts (L0) and overviews (L1) let agents judge relevance before opening full content (L2). → Context layers
  • Sessions become files you can read. Committing a session archives the conversation and extracts memories as Markdown you can inspect, edit, and merge. With VikingBot enabled, ov compile organizes source material into a wiki, knowledge graph, or report. → Sessions · Context compilation

Architecture · Design rationale

viking://
├── resources/              # Resources: project docs, repos, web pages, etc.
│   └── my_project/
│       ├── docs/
│       │   ├── api/
│       │   └── tutorials/
│       └── src/
└── user/
    └── {user_id}/
        ├── memories/
        │   └── preferences/
        │       ├── writing_style
        │       └── coding_habits
        ├── resources/
        │   └── private_project/
        ├── skills/
        │   ├── search_code
        │   └── analyze_data
        └── peers/
            └── web-visitor-alice/

The three loading tiers:

  • L0 (Abstract): a one-sentence summary for quick relevance checks.
  • L1 (Overview): core information and usage scenarios for planning.
  • L2 (Details): the full original data, read only when needed.

Semantically processed directories carry L0/L1 summaries, so agents can judge relevance before reading full files:

viking://resources/my_project/
├── .abstract.md           # L0: quick relevance check
├── .overview.md           # L1: structure and key points
└── docs/
    ├── .abstract.md
    ├── .overview.md
    └── api/
        ├── auth.md         # L2: full content, loaded on demand
        └── endpoints.md

Proof it works

OpenViking 0.3.22 has been evaluated on long-conversation user memory (LoCoMo) and multi-turn agent tasks (tau2-bench). Full results and setup details, including knowledge-base QA, are in the benchmark report; reproduction scripts live in ./benchmark.

The memory evaluation used Doubao 2.0 Pro as the VLM and Doubao-embedding-vision-251215 as the embedding model.

Benchmark results. LoCoMo accuracy: OpenClaw 24.20% native vs 82.08% with OpenViking; Hermes 33.38% vs 82.86%; Claude Code 57.21% vs 80.32%. tau2-bench task success: Retail 70.94% vs 77.81%; Airline 54.38% vs 66.25%.
  • User memory (LoCoMo): with OpenViking, all three agent integrations land at 80–83% accuracy — up from 24–57% on their native memory — while input tokens drop by 34.3–91.0% and query latency by 58.45–66.10%.
  • Agent experience (tau2-bench): experience memory lifts task success by +6.87pp (retail) and +11.87pp (airline) over the same LLM without memory.

Quick start

Requires Python 3.10+ and access to an embedding model and a VLM (cloud or local).

pip install openviking --upgrade
openviking-server init      # configure providers and models
openviking-server doctor    # check configuration and connectivity
openviking-server           # start the server

init writes ~/.openviking/ov.conf. Supported options include Volcengine, OpenAI, Codex OAuth, Kimi, GLM, and local Ollama. See the configuration guide for provider setup and the quick start docs for platform instructions.

The package includes the ov CLI. In another terminal, import a repository and search it:

ov status
ov add-resource https://github.com/volcengine/OpenViking
# Replace TASK_ID with the returned task_id; repeat until status is completed
ov task status TASK_ID
ov ls viking://resources/
ov tree viking://resources/volcengine -L 2
ov find "what is openviking"
ov grep "openviking" --uri viking://resources/volcengine/OpenViking/docs/en

ov find returns matching context with URIs you can inspect. For client configuration (ov config), standalone CLI installs, and index maintenance, see CLI setup.

Build your own integration with the Python, Go, or TypeScript SDK, or the HTTP API.

Use it with your agent

Connect your agent to OpenViking for cross-session memory. Choose a native integration for automatic recall and session capture, or use MCP to give your agent memory and context tools.


Claude

Hooks + MCP

Codex

Hooks + MCP

Cursor

Hooks + MCP

TRAE

Hooks + MCP

OpenClaw

Context engine

Hermes

Built-in

OpenCode

Plugin + MCP

pi

Native extension

DeerFlow

Plugin + MCP

DSH

Plugin + MCP

Doubao Work

Connector

LangChain

Tools + store

General integrations


Agent Plugins 1.0

MCP clients

For setup instructions and integration details, see Integrations.

Desktop App (Beta)

The desktop app is a console for macOS and Windows x64 (beta). It configures supported local agent integrations, inspects recall and capture events in sessions, and syncs local memories and skills to OpenViking.

Download:

VikingBot

VikingBot is an AI agent framework built on top of OpenViking:

pip install "openviking[bot]"
openviking-server --with-bot
ov chat   # in another terminal

The official Docker image bundles VikingBot and starts it by default alongside the server and console UI. Details: VikingBot guide.

Deploy in production

Run the open-source server in your own environment under AGPLv3. It requires no activation key. Start with server setup or the Docker and deployment guide.

The server supports accounts and user isolation and opt-in resource ACLs. Configure authentication before exposing it beyond localhost.

Commercial editions

Managed SaaS

☁️ Managed SaaS

Volcano Engine hosts and operates OpenViking. Personal and Enterprise plans cover individual and team use, with migration tooling for open-source deployments. See the service documentation for plans and limits. Hosting outside China is planned on BytePlus.

Self-Managed

🏢 Self-Managed

Deploy in your own cloud account / VPC (BYOC) or an offline environment. This edition adds distributed deployment and official support, activated by a license key. Contact the team.

Research

Memory that evolves with your agent. VikingMem develops an event-driven approach to extracting, updating, and consolidating long-term memory, giving stateful agents a way to retain useful experience as interactions accumulate. OpenViking open-sources a subset of these core capabilities.

VikingMem: A Memory Base Management System for Stateful LLM-based Applications
Jiajie Fu, Junwen Chen, Mengzhao Wang, Aoxiang He, Maojia Sheng, Xiangyu Ke, Yifan Zhu, and Yunjun Gao.
arXiv:2605.29640, 2026. Presented at VLDB 2026 in September.
📄 Read the paper on arXiv · Read PDF

Directory structure as retrieval context. This paper provides the formal foundations, index design, and experimental evidence behind OpenViking’s directory-aware retrieval. It defines directory-scoped query and maintenance operations and introduces TrieHI, which OpenViking integrates to resolve directory scopes before vector ranking. This connects the filesystem paradigm to retrieval: agents can search a project or memory subtree, retain its surrounding context, and reorganize it as knowledge evolves.

Directory-Aware Query and Maintenance in Vector Databases
Mengzhao Wang, Zheng Gong, Jingpei Hu, Jiajie Fu, Maojia Sheng, Junwen Chen, and Yifan Zhu.
arXiv:2606.16903, 2026. Accepted by ICDE.
📄 Read the paper on arXiv · Read PDF

Retrieve the evidence you need with fewer tokens. VikingRAG combines semantic search with document structure, exposing relevant directory segments as evidence gaps arise. Its core mechanisms are integrated into OpenViking. The paper further explores reusing retrieval traces and escalating to multi-round retrieval only when needed, reducing repeated exploration while preserving answer quality.

VikingRAG: Accurate and Token-efficient Retrieval-augmented Generation over Structured Documents
Peiyuan Gao, Gaoyuan Zhang, Haojie Qin, Yahui Sun, Qianyi Zhang, Yunhao Zhang, Zeyu Wang, and Wei Lu.
arXiv:2609.11390, 2026. Submitted.
📄 Read the paper on arXiv · Read PDF

Partner Projects

  • deer-flow - Open-source long-horizon SuperAgent harness
  • NoKV - AI native distributed file system
  • loopx - Lightweight loop engineering state kernel
  • Hermes Agent - The agent that grows with you

To propose a partnership, open an issue.

Community & Contributing

OpenViking contributors

Security and privacy

For vulnerability reporting and supported versions, see SECURITY.md

License

The OpenViking project uses different licenses for different components:

  • Main Project: AGPLv3 - see the LICENSE file for details
  • crates/ov_cli: Apache 2.0 - see the LICENSE for details
  • examples: Apache 2.0 - see the LICENSE for details. The Hermes plugin in examples/hermes-plugin retains its MIT license.
  • third_party: Respective original licenses of third-party projects
Languages
Python 74.6%
Rust 12.4%
TypeScript 8.7%
C++ 1.6%
Shell 0.9%
Other 1.6%