Commit Graph
51 Commits
Author SHA1 Message Date
MaojiaShengandqin-ctx 3c70f8d370 feat(parse): add large image processing for image parser (#3265)
* feat(parse): add large image processing for image parser

- Add large_image_processor.py: detect large images (>10MB or >4096px),
  create low-res previews, split into grid tiles, and generate grid
  overlay images with tile labels
- Refactor ImageParser.parse() to integrate large image processing pipeline
- Enable SVG-to-PNG conversion in utils.py (cairosvg/wand)
- Rename ImageConfig.max_dimension to preview_max_dimension and add new
  config fields: max_file_size_mb, max_tile_size_mb, max_tile_dimension_px,
  tile_overlap_px, large_image_threshold_dimension
- Update ov.conf.example with new image config options

* fix(parse): correct tile dimension comment from 1024px to 2048px

* fix(parse): fix tile label path in grid overlay to include tiles/ directory

* fix(parse): register missing image extensions for ImageParser

TIFF, ICO, DIB, ICNS, SGI, JP2 were not in IMAGE_EXTENSIONS, causing
them to fallback to TextParser. All are supported by PIL.

* fix(parse): preserve PNG format for tiles instead of always converting to JPEG

* fix(parse): address review feedback for large image processing

- Wire config.image to ImageParser in ParserRegistry (was missing)
- Remove unnecessary preview creation for small images (broke LA mode PNG)
- Enforce max_tile_size_mb on tiles with quality reduction and resize fallback
- Remove 64-tile hard cap that conflicted with max_tile_dimension_px
- Add comment explaining why original file is not saved for large images

* refactor(parse): remove max_tile_size_mb as it is a soft suggestion

max_tile_size_mb was a soft constraint that was not enforced
consistently. Remove it from config, constants, and all enforcement
logic. Tile dimension (max_tile_dimension_px) remains the sole constraint.

* fix(parse): use CJK-capable font for grid overlay labels

The old font loading only tried macOS-specific paths and fell back to
PIL's default bitmap font, which cannot render CJK characters in
filenames. Add a cross-platform CJK font lookup that covers Linux
(Noto/Droid/WQY/DejaVu), macOS (PingFang), and Windows (MSYH/SimSun).

* fix(parse): convert non-VLM-supported image formats to PNG on save

Image formats like TIFF, ICO, DIB, ICNS, SGI, JP2 are not recognized
by VLM backends (OpenAI/LiteLLM/VolcEngine only support PNG/JPEG/GIF/
WebP/BMP) or by embedding_utils for image vectorization. When a file
with one of these extensions is parsed, convert it to PNG and use a
.png extension so that downstream pipelines see consistent data.

SVG files (already PNG-converted via cairosvg) also get the .png
extension for the same reason.

* fix(parse): import io for SVG conversion

---------

Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
2026-07-16 17:37:05 +08:00
DuTao cbfb387dc7 feat(bot): add unified VikingBot gateway routing and OpenViking auth (#3119)
* bot opt config\api check

* 优化vikingbot的启动链路

* 美化颜色

* docs: add VikingBot gateway routing diagram

* fix(bot): harden gateway auth and proxy routing
2026-07-10 17:45:41 +08:00
MaojiaSheng 3c419ed09a chore: replace doubao 2.0 pro with doubao 2.0 lite as recomended VLM model (#2984) 2026-07-03 16:09:20 +08:00
t0sakiandbaobaodae 72d04cd488 feat(ingest): replay local agent-harness logs into OpenViking sessions (#2892)
* feat(ingest): replay local agent-harness logs into OV sessions / 本地 agent harness 日志重放入库

Add openviking/ingest/: parse Claude Code / Codex / OpenCode / Hermes / OpenClaw conversation logs into normalized messages and replay them through OpenViking's existing session pipeline (create_session -> batch_add_messages -> commit -> async memory extraction), instead of a bespoke ETL.

Supports one-shot backfill ("存量") and cursor-driven incremental polling ("新增", WatchScheduler-style, no fs-event dependency), per-harness enable/mode/paths config, and meaningful peer_id on every turn (assistant = {harness}/{model}; user = git identity for single-user harnesses, original username for group-chat harnesses). Cursor IDE is a registered but deferred stub.

Read-position cursors persist under ~/.openviking/ingest/state.db for crash-safe, idempotent resume. New openviking-ingest CLI (backfill/watch/run/status/list-sources) and an "ingest" section on OpenVikingConfig. Verified end-to-end against a local server: 3-message fixture -> session commit -> 10 memories extracted -> idempotent re-run.

Inspired by / supersedes volcengine/OpenViking#2674.

Co-authored-by: baobaodae <2014596548@qq.com>

* docs(ingest): bilingual guide + ov.conf.example for openviking-ingest / 本地日志入库双语文档与配置示例

Add docs/{zh,en}/agent-integrations/09-log-ingestion.md (auto-registered in the VitePress sidebar) and an `ingest` section in examples/ov.conf.example (off by default).

* fix(ingest): address review — gating, crash-safe batch replay, commit recovery, single-instance lock / 修复评审问题

Fixes the merge-blockers from the adversarial review:
- master switch ingest.enabled now actually gates enabled_harnesses();
- idempotent per-batch append with a durable pending-intent reconciled against the server message count on restart (no duplicate imports after a mid-append crash);
- bounded reads (<=100 msgs/call) so huge sessions don't materialize at once;
- needs_commit flag + commit_if_needed so appended-but-uncommitted sessions still get extracted (commit even when no new source rows);
- poller keeps dirty sessions until a commit actually succeeds;
- OpenCode advances its SQLite cursor only past complete rows (late part text no longer skipped);
- single-instance file lock guards concurrent ingest processes;
- positive-value config validation (no poll busy-loop); malformed ov.conf surfaces instead of silently defaulting.

Adds 6 tests (config gating/validation, crash reconcile both ways, commit recovery).

* refactor(ingest): expose as 'openviking-server ingest' subcommand; English-only code/docs

- Route the ingest CLI through 'openviking-server ingest ...' (same dispatch as 'init'/'doctor') and drop the separate 'openviking-ingest' console_script.
- Remove mixed-in Chinese terms (存量/新增) from source docstrings, CLI help, and the English doc; the Chinese doc keeps them.

* style(ingest): ruff format + import sort (isort I)

Run ruff 0.15.16 (from the uv cache) with the repo config: fixes 5 I001 import-order errors in tests and reformats 9 files. 'ruff check' and 'ruff format --check' now pass on all added/edited files.

---------

Co-authored-by: baobaodae <2014596548@qq.com>
2026-06-30 12:14:31 +08:00
t0saki 2846bb6e76 feat(resources): ingest whole sites via sitemap / RSS / Atom (#2858)!
Add WebFeedAccessor (priority 60) that turns a single sitemap /
sitemapindex / RSS / Atom URL into ONE resource tree: it mirrors every
listed page into a temp directory and reuses the existing DirectoryParser
pipeline (the same "fetch-many -> dir -> tree" contract as GitAccessor).
A watch on the feed URL keeps the whole site refreshed (new pages added,
removed pages dropped on each rebuild).

- New openviking/parse/accessors/web_feed_accessor.py: WebFeedAccessor +
  sitemap/feed extractors (nested sitemapindex recursion with depth cap,
  RSS 2.0 / Atom via feedparser), bounded concurrent polite mirroring,
  robots.txt, same-host / include / exclude / max_pages limits.
- args={"site": true} forces whole-site ingestion from a bare domain or
  page by auto-discovering the sitemap/RSS (robots.txt, HTML
  <link rel=alternate>, conventional paths); {"site": false} opts a
  feed-looking URL back out to HTTPAccessor.
- Thread accessor-selection kwargs through can_handle; the registry
  tolerates accessors whose can_handle lacks **kwargs (back-compatible).
- Single-page adds get a non-blocking "this site exposes a sitemap/RSS"
  suggestion appended to the MCP add_resource response, gated to the
  site root only; never auto-crawls.
- New WebFeedConfig (parsers.webfeed): max_pages, concurrency, politeness
  delay, same_host_only, respect_robots, max_depth, suggest_feed.
- Dependencies: feedparser (robust RSS/Atom), defusedxml (XXE-safe XML).
- Docs: zh/en resources API, MCP/CLI/SDK help, ov.conf.example.
- Tests: 52 unit tests (fake httpx, no network).
2026-06-26 21:13:03 +08:00
87329714dd feat(grep): integrate VikingDB bm25 keyword search for grep engine (#2144)
* feat(grep): integrate VikingDB bm25 keyword search for grep engine

* fix(grep): address CI review feedback: max-size eviction to _count_cache, use Literal, Split regex alternation into individual keywords for bm25 (max 10)

* fix(schema): use dynamic __version__ for schema_version and handle dev suffixes in version comparison

* fix(schema): upsert data to vikingdb lack of content

* chore: add benchmark for retrieval

* fix(grep): vikingdb return 200 and no results means no matching content, not necessary to fallback to local fs

* fix(benchmark): sub uri args; add report

* refactor: code format by ruff

* optimize: move grep config (engine and switch_to_remote_threshold) to ov.conf

* optimize: auto adapt remote_return_limit by agg API; rm unnecessary params in keywords search

* fix: adjust benchmark scripts

* fix(grep): store full content for BM25; use PathScope depth; reduce redundant API calls

* refactor: new benchmark

* fix: step1 add resource by real code data

* feat(benchmark): split grep benchmark into effectiveness/performance suites with async reindex

* optimize (benchmark): adjust keywords and ground truth for testing

* fix: truncate 64KB for content field

* optimize: effectiveness add resource plainly

* optimize: change param use of SearchByKeywords from "keywords" to "query"

* optimize(benchmark): refactor effectiveness scripts

* optimize: ensure raw data for content field

* optimize: fulltext analyzer's stop-words only use symbols

* fix: adapt to new ov cli for benchmark

* optimize: reuse file content to avoid re-read AGFS file

* optimize: tune grep vikingdb defaults and refresh bm25 benchmark scripts

* optimize: benchmark client timeout

* update README

* fix: rm unused param

* fix: default values in docs

* optimize: increase truncate byte size to 1MB for content field for VikingDB

* fix(logger): harden queued stream logging (#2786)

* fix(logger): replace StreamHandler with QueueHandler+QueueListener to prevent thread deadlock

When log.output='stdout' (default) and the server is managed by systemd,
concurrent log writes can deadlock because logging.StreamHandler holds a
thread lock across stream.flush() which blocks on systemd-piped file I/O.

During session.commit() phase 2, multiple async coroutines (memory
extraction, summarization) concurrently call logger.info()/warning()
with large payloads. The first thread's flush() blocks on the pipe,
while all subsequent threads block on handler.acquire() forever.
This permanently silences the server log and prevents _write_done_file()
from executing, leaving phase 2 hanging without .done.

Fix: use QueueHandler + QueueListener from stdlib logging.handlers
(Python 3.2+). QueueHandler.emit() does queue.put(record) with no lock
or I/O, returning immediately. QueueListener has a dedicated single
thread as the sole consumer touching the real StreamHandler, making
lock contention impossible.

Changes in _create_log_handler(): stdout/stderr branches now create
a shared QueueListener with unbounded queue, returning QueueHandler
instances to callers. _build_standard_handler() delegates formatter
and filter setup to the real handler in the listener thread.

Closes: #2752

* fix(logger): harden queued stream logging

---------

Co-authored-by: njuboy11 <njuboy11@users.noreply.github.com>

---------

Co-authored-by: Qin Haojie <qinhaojie.exe@bytedance.com>
Co-authored-by: njuboy11 <njuboy11@users.noreply.github.com>
2026-06-24 18:46:02 +08:00
baojun-zhang eff7e67037 feat(storage): support content-type auto-detecting in S3 case (#2668)
* feat(storage): support content-type auto-detecting in S3 case

* feat(storage): update code doc
2026-06-16 18:24:17 +08:00
Qin Haojie a6fc0424bc fix(session): apply memory type policy whitelist (#2530)
* fix(session): apply memory type policy whitelist

Restore top-level memory_types filtering for session memory extraction and validate it against enabled registry schemas. Ensure initialization and peer-aware smoke coverage honor the whitelist.

* fix(session): scope session skills to execution memory policy

* refactor(session): remove per-commit memory policy
2026-06-10 14:54:24 +08:00
yangxinxin-7andClaude Sonnet 4.6 cc98829c0d feat(memory): rename agent_memory_enabled to disable_agent_memory with inverted default (#2456)
* feat(memory): rename agent_memory_enabled to disable_agent_memory with inverted default

Agent memory (trajectory/experience extraction) is now on by default.
Use `disable_agent_memory: true` in ov.conf to opt out.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(memory): add backward compat for deprecated agent_memory_enabled config field

Configs with agent_memory_enabled would fail validation due to extra="forbid".
Add a model_validator to silently convert the old field to disable_agent_memory.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* revert: remove unnecessary backward compat for agent_memory_enabled

No existing users, no migration needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(memory): keep agent_memory_enabled name, change default to true

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: gitignore integration test tmp dirs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test: restore RUN_AGENT_MEMORY_TESTS guard for agent memory e2e

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 15:34:45 +08:00
Qin Haojie ff258768c2 feat(memory): 引入 User/Peer 记忆隔离模型 (#2236)
* feat(memory): introduce user and peer memory isolation

Unify agent-scoped memory behavior into user-owned memory spaces, add peer_id compatibility for session and retrieval paths, and wire memory_policy through session commit flows.

* feat(memory): align session identity around peer IDs

* feat(search): pass peer id through retrieval

* refactor(memory): remove agent identity from integrations

* fix(memory): isolate peer identity from self extraction

* fix(tau2): provision benchmark user configs

* fix(auth): allow admin keys to access data APIs

* fix(openclaw): enable peer memory policy for peer roles

* fix(openclaw): resolve sender for peer recall

* refactor(session): simplify memory extraction routing

* refactor(ov-cli): reduce formatting-only diff

* refactor(message): remove unused message helpers

* refactor(retrieval): simplify peer target resolution

* refactor(namespace): remove deprecated agent namespace policy

* fix(agent): propagate peer id through integrations

* fix(auth): align integration clients with api-key mode
2026-06-05 10:55:48 +08:00
chenjwandAiden 82225006fb Feat/searchable template (#2193)
* feat: replace searchable memory fields with embedding templates

Use memory-type embedding templates for vectorization, share template rendering with content serialization, and fall back to plain content when embedding rendering cannot be resolved.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* fix(memory): rename role-id isolation config flag

Use role_id_memory_isolation_enabled consistently across config, handler, and tests so the rename matches current prepare_messages behavior.

🤖 Generated with [Aiden x Claude Code]

Co-Authored-By: Aiden

* auto-commit before eval 20260522_190730

* auto-commit before eval 20260522_194846

* auto-commit before eval 20260522_200235

* auto-commit before eval 20260522_200720

* update

* auto-commit before eval 20260525_013620
2026-05-25 13:57:26 +08:00
DuTao 96431c17b4 feat(skill): Add extracting skills from session commit processing (#2182)
* add skill extract

* add skill extract

* add skill extract

* add skill extract

* add skill extract

* add skill extract
2026-05-22 16:48:33 +08:00
Evo 4237d5018f docs(retrieval): align score_propagation_alpha default 0.5→1.0 with #1947 (#1948)
* docs(retrieval): align score_propagation_alpha default 0.5→1.0 with #1947

* docs(retrieval): align score_propagation_alpha default 0.5→1.0 with #1947

* docs(retrieval): align score_propagation_alpha default 0.5→1.0 with #1947

* docs(retrieval): align score_propagation_alpha default 0.5→1.0 with #1947

* docs(retrieval): align score_propagation_alpha default 0.5→1.0 with #1947
2026-05-11 18:00:39 +08:00
chenjw 44d3cc41b1 Feat/memory isolation 支持群聊模式 (#1711) 2026-05-06 10:45:06 +08:00
baojun-zhang d7fdb489ce feat(observability): support header param while OTLP export (#1805)
* feat(observability): support header param while OTLP export

* feat(observability): support header param while OTLP export
2026-04-29 20:14:31 +08:00
Qin Haojie b35d38a323 feat(config): 配置检索打分和 embedding 输入 (#1770)
* feat(retrieval): configure hotness score blending

* feat(retrieval): configure score propagation alpha

* test(retrieval): trim redundant propagation coverage

* feat(embedding): centralize token estimation

* fix(embedding): use shared token estimator

* fix(embedding): narrow token truncation scope
2026-04-28 19:04:24 +08:00
baojun-zhang 64682ae189 feat(encryption): make apikey hash encryption as single switch (#1736)
* feat(encryption): make apikey hash encryption as single switch

* feat(encryption): add break change note
2026-04-27 19:23:23 +08:00
baojun-zhangandMaojiaSheng 17d2c5603e feat(observability): unify observability context && support otel && etc. (#1666)
* feat(observability): unify OTLP metrics export, log/trace context, and telemetry bridging
- - Add OTLP metrics http/grpc exporter that pushes MetricRegistry snapshots
- - Decouple telemetry response payload from telemetry collection; always finish() and bridge summary to metrics
- - Unify observability config under server.observability (metrics/traces/logs siblings); update ov.conf.example and docs (zh/en)
- - Improve log/trace correlation via structured context injection
- - Add/adjust tests for exporter lifecycle, config loader, metrics/telemetry runtime
- BREAKING CHANGE: remove legacy telemetry.* config path; use server.observability.*

* feat(observability): import Status/StatusCode for LogToSpanEventFilter

* feat(observability): fix check issue

* feat(observability): format code

---------

Co-authored-by: MaojiaSheng <shengmaojia@bytedance.com>
2026-04-24 21:32:25 +08:00
dingbenanddingben.db@bytedance.com <dingben.db@bytedance.com@bytedance.com> 557b173515 feat(vectordb): add volcengine_api_key data-plane backend for VikingDB (#1588)
* feat(vectordb): add volcengine_api_key data-plane backend for VikingDB

* fix: comment

---------

Co-authored-by: dingben.db@bytedance.com <dingben.db@bytedance.com@bytedance.com>
2026-04-22 14:55:10 +08:00
Hao ZheandZayn Jarvis 01403312ea feat(vlm): add Codex, Kimi, and GLM VLM support (#1444)
* feat(vlm): add Codex OAuth-backed VLM setup and docs

* fix(codex): address PR review follow-up issues

* feat(vlm): add Kimi and GLM backends

* refactor(vlm): simplify codex auth flow and docs

* fix: update code comments and doctor validation

* chore: update uv.lock after merge

* Refine Codex auth flow and VLM backend integrations

* Take over mirrored Codex auth on refresh

* feat(vlm): refine provider setup and auth flow

* style: format VLM and setup files

* style: fix lint import ordering

* fix(codex): harden auth refresh and disable streaming

* fix(codex): translate tool history and refresh auth safely

* style(lint): fix changed-file ruff violations

* fix(init): refine cloud VLM setup prompts

* style(lint): format setup wizard changes

---------

Co-authored-by: Zayn Jarvis <zhiheng.liu@bytedance.com>
2026-04-22 11:01:23 +08:00
baojun-zhang 629fc241e4 feat(metric): add token-full-cycle metric (#1488)
* feat(metric): add token full-cycle metric && support token dashboard && optimize metric guide && deprecated 'server.telemetry.prometheus.enabled' configuration && change metric config to observability.metric

* feat(metric): add token full-cycle metric && support token dashboard && optimize metric guide && deprecated 'server.telemetry.prometheus.enabled' configuration && change metric config to observability.metric

* feat(metric): add debug log

* feat(metric): format code

* feat(metric): format code
2026-04-16 18:22:40 +08:00
MaojiaSheng 95cc0f84d0 reorg: split parser layer to 2-layer: accessor and parser, so that we can reuse more code (#1428) 2026-04-15 10:33:12 +08:00
baojun-zhang b441622ee6 feat(metric): add metric system (#1357)
* feat(metric): add metric system

* feat(metric): add metric system

* feat(metric): add metric system

* fix(metric): do not cancel refresh tasks on deadline; trust only authenticated account id; avoid per-scrape rerank clients

* doc(metric): add metric guide doc

* doc(metric): add metric guide doc

* doc(metric): add metric guide doc

* doc(metric): add metric guide doc

* feat(metric): fix bug & change account dimension switch to default true
2026-04-14 20:55:02 +08:00
Qin Haojie 3c2f095c4f fix(volcengine): update default doubao embedding model (#1438)
Align docs, examples, the setup wizard, and telemetry tests with the
current Doubao multimodal embedding model so generated configs keep
referencing the supported default consistently.
2026-04-14 15:51:06 +08:00
MaojiaShengandopenviking a7e5417ef2 reorg: remove golang depends (#1339)
* docs: fix docker deployment

* reorg: remove third_party/agfs

* feat(s3fs): add disable_batch_delete option for OSS compatibility

Port of PR #1333 from Go version to Rust:

- Add disable_batch_delete config option to S3Client
- When enabled, use sequential single-object deletes instead of DeleteObjects
- This is for S3-compatible services like Alibaba Cloud OSS that require
  Content-MD5 for DeleteObjects but AWS SDK v2 does not send it by default
- Add documentation and config example for OSS

* fix(s3fs): pass disable_batch_delete config from Python to Rust

Add disable_batch_delete to the s3_plugin_config dict in _generate_plugin_config
so that the Python config can properly control the Rust S3FS plugin's behavior.

* reorg: remove third_party/agfs

* reorg: remove third_party/agfs

* change some docs

* change some docs

---------

Co-authored-by: openviking <openviking@example.com>
2026-04-10 15:16:29 +08:00
baojun-zhang 8271ef362c fix(security): configurable embedding circuit breaker & log suppression (#1277) 2026-04-07 18:27:48 +08:00
baojun-zhang 94468d445d feat(storage): volcengine vector db support sts token (#1268)
* feat(storage):  volcengine vector db support sts token  https://www.volcengine.com/docs/7139/1302258?lang=zh

* feat(storage):  volcengine vector db support sts token  https://www.volcengine.com/docs/7139/1302258?lang=zh
2026-04-07 17:04:30 +08:00
chenjwandClaude Opus 4.6 2771765298 Refactor memory extract (#916)
* docs: add memory extractor templating and update mechanism optimization design document

- Add bilingual (English/Chinese) design document for memory templating system
- Include YAML-based MemoryTypeRegistry with 8 built-in types
- Detail ReAct 3+1 phase flow with pre-fetch optimization
- Describe 3-operation Schema: write/edit/delete
- Document RoocodePatch SEARCH/REPLACE format
- Explain dual-mode design: simple mode vs template mode
- Cover pre-fetch optimization: ls directories + read .abstract.md/.overview.md + search once
- Include merge operations: patch, sum, avg, immutable
- Address #578: allow custom prompt template addition and specification

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add memory templating system with ReAct orchestrator

- Add YAML-configurable memory schemas (cards, events, entities, etc.)
- Implement MemoryReAct with tool use (read/find/ls)
- Add schema-driven memory operations (write_uris/edit_uris/delete_uris)
- Implement memory patch handler for incremental updates
- Add comprehensive test suite

* refactor: memory extractor templating system with ReAct orchestrator

## Summary

Implement memory templating system (GitHub Issue #578) - a complete
rewrite of the memory extractor subsystem to support YAML-configurable
memory types instead of hardcoded categories.

## Key Changes

### Architecture
- Replace hardcoded 8 memory types with YAML-configurable schema system
- Add MemoryTypeRegistry to load memory type definitions from YAML files
- Dynamic Pydantic model generation from schema for type safety
- Field-level merge operations: PATCH, SUM, IMMUTABLE

### Memory Extraction Flow
- Implement ReAct orchestrator for single-pass memory updates
- MemoryUpdater for applying operations to storage
- Memory tools (read, search, ls) for ReAct loop
- Stable JSON parser with 5-layer fault tolerance

### File Naming & Storage
- Semantic filenames from template ({topic}.md instead of random IDs)
- Two memory modes: simple mode and template mode
- MEMORY_FIELDS HTML comment for structured metadata

### Configuration
- 9 YAML templates in openviking/prompts/templates/memory/
- memory_config.py for memory system configuration
- Dual-threshold compact upload mechanism in design doc

### Deletions
- Remove old memory_content.py, memory_data.py, memory_operations.py
- Remove memory_types.py, memory_utils.py, memory_patch.py
- Remove corresponding old test files

### Updated Components
- VLM backends (litellm, openai, volcengine) for new interfaces
- Session and service core integration
- Test suite for new architecture

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: pass ctx/user/session_id in commit_async for memory extraction

## Summary

Fix missing parameters in commit_async() when calling extract_long_term_memories().
The synchronous commit() method correctly passes these parameters, but the async
version was missing them, causing memory extraction to be skipped.

## Changes

- Pass user=self.user, session_id=self.session_id, ctx=self.ctx
  in commit_async() when calling extract_long_term_memories()

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: convert FindResult to dict before returning from search tool

## Summary

Fix JSON serialization error by converting FindResult object to dict
using its to_dict() method before returning from MemorySearchTool.

## Changes

- In MemorySearchTool.execute(), return search_result.to_dict()
  instead of search_result directly

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: swap None check before accessing final_operations in memory_react

Also rename schema_models.py to schema_model_generator.py for clarity.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add edit_overview support and optimize memory registry initialization

- Add edit_overview_operations to MemoryUpdater for updating .overview.md files
- Optimize MemoryTypeRegistry initialization in SessionCompressorV2 (load once)
- Various memory templating system improvements

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: remove unnecessary indent in JSON schema output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add markdown link format hint to overview field description

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add pre-fetch search based on user messages in conversation

Also fix duplicate line in system prompt.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* rebase

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-24 15:28:46 +08:00
baojun-zhang 5bbb22a6ef feat: add encrypt doc && refactoring encrypt code (#893)
* feat: add encrypt doc && fix Partial reads/wrong location problem && refactoring duplicated code && add

* feat: reformat code

* feat: resolve comment

* feat: resolve comment

* feat: resolve comment
2026-03-23 19:23:11 +08:00
ChenXiaofeiandClaude Sonnet 4.6 3064725bd6 fix(parser): enforce hard character limit per section in MarkdownParser (#826)
The token estimator (_estimate_token_count) uses a simplified heuristic
(0.3 tokens/non-CJK-non-space char) that can underestimate actual token
counts for dense code or technical content. This caused large sections to
pass the token check and be written as single L2 files, which then
exceeded embedding API limits (e.g., text-embedding-v4 8192 token cap).

Changes:
- Add `max_section_chars: int = 6000` to ParserConfig as a hard character
  limit per section, guarding against token estimation errors
- `_smart_split_content`: enforces char limit per split chunk alongside
  the existing token estimate limit; falls back to token-only splitting
  when max_section_chars <= 0 to avoid range(n, 0) crash
- `_save_section`: requires both token AND char limits satisfied before
  writing a section as a single file
- `_save_merged`: splits joined content via _smart_split_content when the
  merged result exceeds max_section_chars, preventing token-small but
  char-large sections from accumulating into oversized files
- `_parse_and_create_structure`: small-document fast-path now also checks
  char count before writing as a single file
- `ParserConfig.validate()`: rejects max_section_chars <= 0
- `ParserConfig` class docstring: add max_section_chars attribute,
  fix max_section_size description ("characters" -> "tokens")
- `ov.conf.example`: add max_section_chars to pdf parser block

Add 21 unit tests covering ParserConfig defaults and validation,
_smart_split_content char-limit enforcement, _save_section and
_save_merged char-limit gates, and _parse_and_create_structure
fast-path with and without headings (VikingFS mocked throughout).

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-21 14:06:02 +08:00
baojun-zhang 8cdecd9163 feat: Add multi-tenant file encryption capability (#828)
* feat: Add multi-tenant file encryption capability

* feat: move cli command `ov crypto` to `ov system crypto` && ruff format && add encryption config example

* feat: reformat code

* feat: adjust crypto.rs to support diff platform

* feat: reformat code
2026-03-21 13:35:46 +08:00
ChenXiaofeiandClaude Sonnet 4.6 44d9542fc3 feat(rerank): add OpenAI-compatible rerank provider (#785)
- Add OpenAIRerankClient using standard flat request/response format
  compatible with DashScope compatible-api and other OpenAI/Cohere-style
  rerank APIs (no input/output wrappers)
- Fix silent data corruption: add index bounds-checking so out-of-bounds
  or missing index returns None with a warning
- Add provider allow-list validation in RerankConfig ('vikingdb'|'openai')
- Remove unnecessary getattr() in RerankClient.from_config()
- Update ov.conf.example: keep vikingdb (doubao) as primary rerank config,
  add rerank_openai_example section for DashScope qwen3-rerank
- Update docs (en/zh): add OpenAI-compatible provider example alongside
  existing volcengine example in configuration guide and schema
- Add 24 tests covering success, edge cases, and factory dispatch

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-20 14:22:25 +08:00
ChenXiaofei e5abe88f04 feat(embedding): add Ollama provider support for local embedding (#644)
- Add 'ollama' as a supported embedding provider

- Ollama runs locally via OpenAI-compatible API, no API key required

- Allow OpenAI provider to work without api_key when api_base is set

  (supports local OpenAI-compatible servers like vLLM, LocalAI)

- Add configuration example and tests for Ollama provider

This enables fully local embedding deployment without cloud API keys.
2026-03-16 17:55:50 +08:00
zhoujiahui f7607408ce fix: fix ov.conf.example (#488) 2026-03-09 14:21:31 +08:00
MaojiaShengandopenviking c4209da6cc chore: downgrade golang version limit, update vlm version to seed 2.0 (#425)
* feat: define a system path for future deployment

* feat: define a system path for future deployment

* fix: golang downgrade to 1.19

* fix: golang downgrade to 1.19, and change doc

* docs: change model recommendation

* fix: golang downgrade to 1.19, and change doc

* fix: mv test files

* fix: loguru

---------

Co-authored-by: openviking <openviking@example.com>
2026-03-05 14:34:02 +08:00
yangxinxin-7 fac6068b26 feat(parse): add AST-based code skeleton extraction mode (#334) 2026-02-27 20:03:34 +08:00
MaojiaShengandopenviking c943fb0ea9 feat: allow private gitlab domain for code repo (#285)
Co-authored-by: openviking <openviking@example.com>
2026-02-25 18:57:46 +08:00
Qin Haojie b92f2c0fc7 feat: 多租户 Phase 1 - API 层多租户能力 (#260)
* doc: update design

* feat: multi_tenant

* feat: multi_tenant

* feat: multi tenant

* feat: multi tenant

* fix: tests

* fix: rust cli
2026-02-24 13:40:44 +08:00
MaojiaShengandopenviking 3d2d05a370 feat: wrap log configs in LogConfig, add log rotation, fix empty file creation (#261)
- Create new LogConfig class to wrap all log-related configuration
- Update OpenVikingConfig to use nested log config instead of separate fields
- Add log rotation support with TimedRotatingFileHandler in logger.py
- Add configure_uvicorn_logging to make uvicorn use OpenViking's logging config
- Fix empty file being created in current directory by properly handling log.output=file in logging_init.py
- Update examples/ov.conf.example to use new nested log config structure
- Update __init__.py to export LogConfig and initialize_openviking_config

Co-authored-by: openviking <openviking@example.com>
2026-02-23 18:25:14 +08:00
e7d3610f76 feat: openviking upload zip and extract on serverside, for add-resource a local dir (#249)
* fix: make rust CLI (ov) commands match python CLI (openviking) exactly - add top-level wait/status/health commands

* fix: ov cli plays same as py cli (ls, tree)

* Refactor media parsers to subdirectory structure with validation

* Enhance CLI robustness: validate add-resource path exists and detect unquoted spaces

* Fix unescaped spaces in paths by replacing \  with space

* Sanitize URI components to replace spaces and special chars with underscores

* feat: auto organize audio and image and video files

* Update media parsers to use original filenames and folder names with extensions

* Optimize MediaParser section for readability

* feat: vlm optimization for image

* feat: vlm optimization for image

* feat: vlm optimization for image

* refactor: move media content understanding to SemanticProcessor

- Add parse/parsers/media/utils.py with media helpers
- Refactor ImageParser.parse(), AudioParser.parse(), VideoParser.parse() to remove content understanding, keep only metadata extraction
- Update SemanticProcessor._generate_single_file_summary() to handle media types and call media utils for summary generation
- Update TreeBuilder._get_base_uri() to use media utils
- Update ResourceNode.get_abstract() and get_overview() to check meta for abstract/overview
- Add debug logs and error handling

* refactor: split _generate_single_file_summary to add _generate_text_summary

- Add _generate_text_summary function for text file processing
- Update media utils functions to accept llm_sem and use it to limit concurrent calls
- Update _generate_single_file_summary to call _generate_text_summary and media utils functions
- Fix import ordering
- Fix issue where _generate_file_summaries was creating a new semaphore, now each _generate_single_file_summary handles its own

* feat: vlm optimization for image

* feat: vlm optimization for image

* Implement smart dual-mode for add-resource and import-ovpack, and config system improvements

* feat: support local upload

---------

Co-authored-by: openviking <openviking@example.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
2026-02-23 15:52:07 +08:00
baojun-zhang cab8ea540b feat: support dynamic project_name config in VectorDB / volcengine (#253)
* feat: support dynamic project_name config  in VectorDB / volcengine

* feat: support dynamic project_name config  in VectorDB / volcengine

* feat: support dynamic project_name config  in VectorDB / volcengine

* feat: support dynamic project_name config  in VectorDB / volcengine

* feat: support dynamic project_name config  in VectorDB / volcengine
2026-02-22 23:37:17 +08:00
MaojiaShengandopenviking dc7bc956f3 feat: update media parsers (#196)
* fix: make rust CLI (ov) commands match python CLI (openviking) exactly - add top-level wait/status/health commands

* fix: ov cli plays same as py cli (ls, tree)

* Refactor media parsers to subdirectory structure with validation

* Enhance CLI robustness: validate add-resource path exists and detect unquoted spaces

* Fix unescaped spaces in paths by replacing \  with space

* Sanitize URI components to replace spaces and special chars with underscores

* feat: auto organize audio and image and video files

* Update media parsers to use original filenames and folder names with extensions

* Optimize MediaParser section for readability

* feat: vlm optimization for image

* feat: vlm optimization for image

* feat: vlm optimization for image

* refactor: move media content understanding to SemanticProcessor

- Add parse/parsers/media/utils.py with media helpers
- Refactor ImageParser.parse(), AudioParser.parse(), VideoParser.parse() to remove content understanding, keep only metadata extraction
- Update SemanticProcessor._generate_single_file_summary() to handle media types and call media utils for summary generation
- Update TreeBuilder._get_base_uri() to use media utils
- Update ResourceNode.get_abstract() and get_overview() to check meta for abstract/overview
- Add debug logs and error handling

* refactor: split _generate_single_file_summary to add _generate_text_summary

- Add _generate_text_summary function for text file processing
- Update media utils functions to accept llm_sem and use it to limit concurrent calls
- Update _generate_single_file_summary to call _generate_text_summary and media utils functions
- Fix import ordering
- Fix issue where _generate_file_summaries was creating a new semaphore, now each _generate_single_file_summary handles its own

* feat: vlm optimization for image

* feat: vlm optimization for image

---------

Co-authored-by: openviking <openviking@example.com>
2026-02-20 15:42:19 +08:00
MaojiaSheng 1b42fec179 fix: tree uri output error, and validate ov.conf before start (#169)
* fix: temp dir check failed

* feat: make temp uri readable, and enlarge timeout of add-resource

* fix: server uvloop conflicts with nest_asyncio

* fix: tree uri output error

* fix: forbid unknown config field, early failed

* docs: update readme
2026-02-14 11:12:13 +08:00
MaojiaSheng ead9817791 Revert "feat: support dynamic project_name config in VectorDB / volcengine (…" (#167)
This reverts commit a469c416f9.
2026-02-13 23:09:20 +08:00
baojun-zhang a469c416f9 feat: support dynamic project_name config in VectorDB / volcengine (#161)
* feat: support dynamic project_name config  in VectorDB / volcengine

* feat: support dynamic project_name config  in VectorDB / volcengine

* feat: support dynamic project_name config  in VectorDB / volcengine

* feat: support dynamic project_name config  in VectorDB / volcengine
2026-02-13 20:40:41 +08:00
qin-ctx 6d55bf4aa1 feat: 新增 Bash CLI 基础框架与完整命令实现 (T3 + T5) (#132)
* feat: 新增 Typer CLI 并统一配置加载机制

引入基于 Typer 的完整 CLI 模块 (openviking/cli),支持 resources、
sessions、search、filesystem、content、relations、pack、debug、
observer、system 等子命令。

新增统一配置加载器 (config_loader),采用三级解析链(显式路径 →
环境变量 → ~/.openviking/),同时服务于 server (ov.conf) 和
CLI (ovcli.conf)。

精简 AsyncOpenViking / SyncOpenViking 客户端,移除 service 模式
(vectordb_url / agfs_url) 和环境变量回退逻辑,仅保留 embedded
和 HTTP 两种模式。

同步更新中英文文档、示例和测试。

* fix: remove user  when create session

* fix: tests

* fix: tests
2026-02-11 15:53:10 +08:00
qin-ctx 3165ffa0da feat: add HTTP Server and Python HTTP Client (T2 & T4) (#109)
* feat: add Server/Client architecture with HTTP API and restructure documentation

  - Implement FastAPI-based HTTP server (openviking/server/) with REST API
  - Add client abstraction layer (LocalClient, HTTPClient, BaseClient)
  - Add CLI entry point (python -m openviking serve)
  - Fix bugs: session.session_id, link/unlink param names, hmac.compare_digest
  - Restructure docs: remove numbered prefixes, add guides/, rewrite API reference
    with both Python SDK and HTTP API (curl) examples (en/zh)
  - Add quickstart-server, deployment, authentication, monitoring guides
  - Update examples and design docs to reflect implementation

* 提供单测 和 文档

* Merge branch 'main' into feature/server_client

* feat: add server/client examples and server tests

* fix: cross-references

* fix : tests
2026-02-09 21:12:15 +08:00
kkkwjx 55d1690a85 feat: rename backend to provider for embedding and vlm (#24) 2026-02-02 17:00:41 +08:00
qin-ctx b5d8149dd9 Revert "fix: use provider instead of backend"
This reverts commit 69e141fd88.
2026-01-29 20:45:33 +08:00
qin-ctx 69e141fd88 fix: use provider instead of backend 2026-01-29 20:39:06 +08:00