Commit Graph
13 Commits
Author SHA1 Message Date
87329714dd feat(grep): integrate VikingDB bm25 keyword search for grep engine (#2144)
* feat(grep): integrate VikingDB bm25 keyword search for grep engine

* fix(grep): address CI review feedback: max-size eviction to _count_cache, use Literal, Split regex alternation into individual keywords for bm25 (max 10)

* fix(schema): use dynamic __version__ for schema_version and handle dev suffixes in version comparison

* fix(schema): upsert data to vikingdb lack of content

* chore: add benchmark for retrieval

* fix(grep): vikingdb return 200 and no results means no matching content, not necessary to fallback to local fs

* fix(benchmark): sub uri args; add report

* refactor: code format by ruff

* optimize: move grep config (engine and switch_to_remote_threshold) to ov.conf

* optimize: auto adapt remote_return_limit by agg API; rm unnecessary params in keywords search

* fix: adjust benchmark scripts

* fix(grep): store full content for BM25; use PathScope depth; reduce redundant API calls

* refactor: new benchmark

* fix: step1 add resource by real code data

* feat(benchmark): split grep benchmark into effectiveness/performance suites with async reindex

* optimize (benchmark): adjust keywords and ground truth for testing

* fix: truncate 64KB for content field

* optimize: effectiveness add resource plainly

* optimize: change param use of SearchByKeywords from "keywords" to "query"

* optimize(benchmark): refactor effectiveness scripts

* optimize: ensure raw data for content field

* optimize: fulltext analyzer's stop-words only use symbols

* fix: adapt to new ov cli for benchmark

* optimize: reuse file content to avoid re-read AGFS file

* optimize: tune grep vikingdb defaults and refresh bm25 benchmark scripts

* optimize: benchmark client timeout

* update README

* fix: rm unused param

* fix: default values in docs

* optimize: increase truncate byte size to 1MB for content field for VikingDB

* fix(logger): harden queued stream logging (#2786)

* fix(logger): replace StreamHandler with QueueHandler+QueueListener to prevent thread deadlock

When log.output='stdout' (default) and the server is managed by systemd,
concurrent log writes can deadlock because logging.StreamHandler holds a
thread lock across stream.flush() which blocks on systemd-piped file I/O.

During session.commit() phase 2, multiple async coroutines (memory
extraction, summarization) concurrently call logger.info()/warning()
with large payloads. The first thread's flush() blocks on the pipe,
while all subsequent threads block on handler.acquire() forever.
This permanently silences the server log and prevents _write_done_file()
from executing, leaving phase 2 hanging without .done.

Fix: use QueueHandler + QueueListener from stdlib logging.handlers
(Python 3.2+). QueueHandler.emit() does queue.put(record) with no lock
or I/O, returning immediately. QueueListener has a dedicated single
thread as the sole consumer touching the real StreamHandler, making
lock contention impossible.

Changes in _create_log_handler(): stdout/stderr branches now create
a shared QueueListener with unbounded queue, returning QueueHandler
instances to callers. _build_standard_handler() delegates formatter
and filter setup to the real handler in the listener thread.

Closes: #2752

* fix(logger): harden queued stream logging

---------

Co-authored-by: njuboy11 <njuboy11@users.noreply.github.com>

---------

Co-authored-by: Qin Haojie <qinhaojie.exe@bytedance.com>
Co-authored-by: njuboy11 <njuboy11@users.noreply.github.com>
2026-06-24 18:46:02 +08:00
Dechao Sun c1cf1c3d06 add Qdrant vectordb support (#2350) 2026-06-01 16:46:48 +08:00
Qin HaojieandClaude Opus 4.6 cad9ec1408 refactor(model): unify config-driven retry across VLM and embedding (#926)
* refactor(model): unify config-driven retry across VLM and embedding

Move retry behavior into shared model-call utilities and config defaults so VLM and embedding providers handle transient failures consistently.

Co-Authored-By: Claude Opus 4.6

* fix

* docs(config): document model retry settings
2026-03-31 16:17:28 +08:00
MaojiaShengandopenviking ce998873f9 lisence: change the main lisence to AGPL-3.0 (#1085)
* lisence: change the main lisence from Apache-2.0 to AGPL-v3

* lisence: change the main lisence from Apache-2.0 to AGPL-v3

* lisence: change the main lisence from Apache-2.0 to AGPL-v3

---------

Co-authored-by: openviking <openviking@example.com>
2026-03-30 14:37:42 +08:00
baojun-zhang 5bbb22a6ef feat: add encrypt doc && refactoring encrypt code (#893)
* feat: add encrypt doc && fix Partial reads/wrong location problem && refactoring duplicated code && add

* feat: reformat code

* feat: resolve comment

* feat: resolve comment

* feat: resolve comment
2026-03-23 19:23:11 +08:00
chethanuk 33f1a8f42b feat(gemini): add GeminiDenseEmbedder text embedding provider (#751)
* feat(gemini): add GeminiDenseEmbedder with is_query routing per #702 pattern

- GeminiDenseEmbedder: accept query_param/document_param, use is_query
  in embed() and embed_batch() to select task_type at call time
- EmbeddingConfig: add Gemini provider, factory, validation, dimension
- No get_query_embedder/get_document_embedder/_get_contextual_embedder
  (removed in #702; embed(is_query=True/False) is the pattern)
- Tests use embed(text, is_query=True/False) pattern throughout
- Rebased onto current upstream/main

* fix(gemini): remove task_type config field, fix conditional import for CI

- Remove task_type from EmbeddingModelConfig (query_param/document_param suffice)
- Wrap GeminiDenseEmbedder import in try/except (google-genai is optional)
- Update tests for removed field
2026-03-20 21:26:23 +08:00
Qin HaojieandClaude Opus 4.6 8274cc31ef fix: 修复单测以适配 vectordb 接口重构,统一测试数据路径 (#333)
- 修复 filesystem stat 错误匹配,增加 "not found" 判断
- 为 AddResourceRequest 添加 model_validator 校验 path 参数
- 处理 queue manager 未初始化时 ObserverService 的异常
- 简化 vectordb record ID 生成逻辑,移除 owner_space
- 捕获 HTTP client close 时的 RuntimeError
- 统一测试数据路径至 test_data/ 目录
- 更新测试用例使用 get_context_by_uri() 等新接口
- 移除测试 mock 对 VikingDBInterface 的依赖

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 19:40:12 +08:00
kkkwjxandqin-ctx 37b5d0d359 Multi tenant (#283)
* test: add multi-tenant isolation unit coverage

* feat: implement phase2 multi-tenant context isolation

* feat: multi tenant

* fix: resolve multi-tenant merge conflicts

* feat: multi tenant

* fix: enforce tenant isolation for session get

* tos config

---------

Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
2026-02-25 18:25:36 +08:00
Qin Haojie b92f2c0fc7 feat: 多租户 Phase 1 - API 层多租户能力 (#260)
* doc: update design

* feat: multi_tenant

* feat: multi_tenant

* feat: multi tenant

* feat: multi tenant

* fix: tests

* fix: rust cli
2026-02-24 13:40:44 +08:00
MaojiaSheng db7319f4a2 refactor: to accelerate cli launch speed, refactor openviking_cli dir (#150)
* fix: remove await asyncio and call agfs directly

* feat: mv cli out of openviking

* refactor: mv cli out of openviking

* refactor: mv cli out of openviking
2026-02-12 22:12:50 +08:00
qin-ctx 002e14087f refactor(client): 拆分 HTTP 客户端,分离嵌入模式与 HTTP 模式 (#141)
* refactor(client): 拆分 HTTP 客户端,分离嵌入模式与 HTTP 模式

- 将 HTTPClient 重命名为 AsyncHTTPClient,新增 SyncHTTPClient 同步封装
- AsyncOpenViking/SyncOpenViking 仅保留嵌入模式,HTTP 模式使用独立的 AsyncHTTPClient/SyncHTTPClient
- HTTP 客户端支持从 ovcli.conf 自动加载 url/api_key
- 移除客户端构造函数中的 user 参数,统一使用默认用户
- 简化 CLIContext,直接使用 SyncHTTPClient
- 简化 VikingFS._uri_to_path 实现
- 更新文档、示例和测试适配新 API

* fix queue
2026-02-12 12:25:54 +08:00
qin-ctx 6d55bf4aa1 feat: 新增 Bash CLI 基础框架与完整命令实现 (T3 + T5) (#132)
* feat: 新增 Typer CLI 并统一配置加载机制

引入基于 Typer 的完整 CLI 模块 (openviking/cli),支持 resources、
sessions、search、filesystem、content、relations、pack、debug、
observer、system 等子命令。

新增统一配置加载器 (config_loader),采用三级解析链(显式路径 →
环境变量 → ~/.openviking/),同时服务于 server (ov.conf) 和
CLI (ovcli.conf)。

精简 AsyncOpenViking / SyncOpenViking 客户端,移除 service 模式
(vectordb_url / agfs_url) 和环境变量回退逻辑,仅保留 embedded
和 HTTP 两种模式。

同步更新中英文文档、示例和测试。

* fix: remove user  when create session

* fix: tests

* fix: tests
2026-02-11 15:53:10 +08:00
qin-ctx 3165ffa0da feat: add HTTP Server and Python HTTP Client (T2 & T4) (#109)
* feat: add Server/Client architecture with HTTP API and restructure documentation

  - Implement FastAPI-based HTTP server (openviking/server/) with REST API
  - Add client abstraction layer (LocalClient, HTTPClient, BaseClient)
  - Add CLI entry point (python -m openviking serve)
  - Fix bugs: session.session_id, link/unlink param names, hmac.compare_digest
  - Restructure docs: remove numbered prefixes, add guides/, rewrite API reference
    with both Python SDK and HTTP API (curl) examples (en/zh)
  - Add quickstart-server, deployment, authentication, monitoring guides
  - Update examples and design docs to reflect implementation

* 提供单测 和 文档

* Merge branch 'main' into feature/server_client

* feat: add server/client examples and server tests

* fix: cross-references

* fix : tests
2026-02-09 21:12:15 +08:00