FastMCP derives tool input schemas from Python type hints, so Optional/
Union parameters become anyOf nodes with no top-level type, and nested
models become $ref/$defs. Valid JSON Schema, but clients that forward
these schemas verbatim to strict function-calling APIs break: Gemini's
OpenAPI 3.0 subset rejects the whole request (400: schema didn't specify
the schema type field), and n8n's JSON-schema-to-Zod conversion can
silently fall back and drop tool arguments entirely.
Rewrite the advertised schemas after registration: drop null branches,
collapse unions to their most general branch, inline $refs, and ensure
every node carries an explicit type. Runtime argument validation still
uses the original function signatures, so union parameters keep
accepting every branch (e.g. read still takes a bare URI string even
though the schema advertises an array).
Also type recall's other_peer_penalty honestly as
Optional[Union[float, Dict[str, float]]] instead of Optional[Any],
which produced an empty {} schema node.
* feat(connector): delegate add_resource imports to external Connector
Opt-in integration that routes add_resource data fetching and parsing
to external Connector service; the Connector stages source data and
calls back into OV through the standard add_resource pipeline.
- add ConnectorClient wrapping the control plane's inner doc/add and
task/info endpoints
- add [connector] config section: enable, connector/tracker endpoint
URLs, timeout_seconds, poll_interval_ms, allowed_add_types
- route add_resource via Connector when enabled and args.add_type is
in allowed_add_types; otherwise fall back to the standard pipeline
with an info log
- track imports as connector_import TaskRecords and poll Connector
task status in the background until terminal state or timeout
* fix(connector): delegate add_resource imports to external Connector
#2921 persisted the DCR scope when the registrar sends one and started
advertising scopes_supported=["mcp"] in the PRM document. ChatGPT's DCR
omits the scope field, so its client registers scope-less, then requests
the advertised scope=mcp at /authorize and gets bounced back with
error=invalid_scope before any consent page renders.
- app.py: pass default_scopes=["mcp"] to ClientRegistrationOptions so
scope-less registrations get the default grant; valid_scopes stays
unset so clients that register their own scope strings are not
rejected at DCR time
- provider.py: single-source the scope as MCP_SCOPE; get_client() falls
back to it for NULL-scope rows, repairing already-registered clients
without migration or re-registration
- router.py: reuse MCP_SCOPE in the PRM scopes_supported
- tests: provider fallback unit tests + end-to-end scope-less DCR and
legacy NULL-scope client authorize regressions
中文：抢写锁失败时改抛 ResourceBusyError，复用现有 CONFLICT 到 HTTP 409 的错误映射。
English: align ov write lock contention with existing conflict handling used by other resource operations.
Co-authored-by: chenpengfei <chenpengfei@bytedance.com>
* feat: add memory plugin mcp harness
* refactor: vendor shared memory plugin modules
* feat: add type quota recall api
* feat: commit codex memory by token threshold
* feat: capture codex tool calls as parts
* feat: add claude skill experience recall
* chore: fix lint in type quota recall server files
* feat: remote marketplace install with unified openviking naming
- Fix root .claude-plugin/marketplace.json git-subdir discriminator key
("type" -> "source"); claude plugin validate now passes.
- Unified installer gains --source remote|archive|dev: remote registers a
synthesized git-subdir marketplace for Claude Code and a git marketplace
for Codex (no repo clone); archive consumes the slim TOS marketplace zip;
dev registers the checkout's examples/ directory for both harnesses.
- One marketplace name (openviking) across all modes and harnesses, so the
plugin id is always openviking-memory@openviking; installer migrates old
openviking-plugins-local registrations and config.toml sections.
- Restore legacy Claude Code (<2.0) support: claude mcp add (stdio proxy)
plus node-based hooks merge into ~/.claude/settings.json.
- Restore optional statusline registration (fetches sources on opt-in).
- Checkbox TUI harness selection via /dev/tty with non-tty fallback.
- Add examples/.agents/plugins/marketplace.json so Codex directory installs
drop the synthetic symlink marketplace.
- Add shared setup wizard (scripts/setup.mjs) for pure-marketplace installs.
- release-tos.yml: upload memory-plugin-shared/install.sh and build/upload
the memory-plugin-marketplace zip; tos-install.sh prefers it and pins all
fetches to TOS via OPENVIKING_SHARED_INSTALL_URL.
- CI: bash -n on installer scripts; marketplace contract tests updated.
* fix(installer): register Claude remote marketplace as a directory
File-type marketplaces (bare marketplace.json path) make Claude Code derive
a wrong installLocation and 'marketplace update' fails with EISDIR. Write
the synthesized manifest to <dir>/.claude-plugin/marketplace.json and add
the directory instead; compare registered sources by exact match so the
old file registration migrates cleanly.
* feat(statusline): show model name and native-style context percentage
A custom statusLine replaces Claude Code's native line including its context
indicator, so reproduce it from the statusline stdin payload: 'Fable 5 ·
ctx 42%' right after the health segment, with native color thresholds
(<70% dim, 70-89% yellow, >=90% red). Falls back from used_percentage to
remaining_percentage to token counts, and stays visible in bypass mode
since it describes the CC conversation, not OV. Opt out with
OPENVIKING_STATUSLINE_CTX=off. Line cap raised 80 -> 100 visible chars.
* fix(installer): keep checkout progress off stdout in plugin_dir_on_disk
Callers capture the function's stdout, so ensure_checkout's info lines were
concatenated into the statusline command registered in settings.json.
* fix(installer): re-register codex git marketplace instead of upgrading
Codex doesn't expose which --ref a git marketplace was added with, and
'marketplace upgrade' refreshes the old ref — so a URL match must not skip
re-registration or a ref override installs the wrong snapshot. Also remove
the stale pre-unification plugin cache directory during migration.
* fix(installer): include .agents in codex sparse checkout
A plugin-dir-only sparse checkout omits the repo-root marketplace manifest
and fails with 'marketplace root does not contain a supported manifest'.
Adding --sparse .agents keeps the snapshot slim (~7.5M vs full repo).
* feat(installer): bilingual prompts, dist channel selection, and TOS git marketplace for codex
- Interactive language selection (English/中文, --lang, auto-detected from
locale); every user-facing prompt is bilingual.
- Download-source selection (--dist github|tos, prompted interactively):
github keeps the remote marketplaces; tos serves GitHub-blocked regions.
- Credentials step now always shows the current ovcli.conf values (masked
key) and offers keep-or-reconfigure instead of silently reusing them.
- Codex on TOS installs from a TOS-hosted git repo over dumb HTTP and keeps
remote updates (codex plugin marketplace upgrade); falls back to the
archive directory if the repo is unavailable. release-tos.yml builds and
uploads the single-commit bare repo (repack + update-server-info).
- Claude Code on TOS warns that directory marketplaces cannot auto-update.
- tos-install.sh bootstraps shrink to TOS_BASE + --dist tos.
- Docs (READMEs, agent-integrations pages, image cards, en+zh) now all use
the single shared installer and drop the deleted wrapper instructions.
* feat(installer): unify all choice prompts on an arrow-key TUI menu
Language, download source, connection mode, keep-or-reconfigure
credentials, statusline enable/replace, and legacy-mode confirmation all
render as the same single-select menu (arrow keys / digit shortcuts /
enter, radio-style highlight) instead of mixed numbered and y/N prompts.
Falls back to numbered input when /dev/tty can't be drawn on and to the
default choice when non-interactive. Free-text fields (URL, API key) stay
line inputs; the harness picker keeps its checkbox multi-select.
* fix(installer): stop piping plugin lists into grep -q under pipefail
grep -q exits on first match and SIGPIPEs the producer, so with pipefail
the 'codex plugin list | grep -q' check read as a miss every time (codex's
list is long; claude's short list masked the bug). Capture the output and
substring-match in bash instead — validation no longer false-warns.
Also: drop the stdio-proxy line from the Done summary; always offer the
install-source menu unless --dist/--source was given (with a checkout the
menu gains a dev option and defaults to it); surface the Claude-on-TOS
no-auto-update warning at source resolution instead of after install.
* fix: unignore examples/memory-plugin-shared/lib and commit the shared modules
The Python build-artifact 'lib/' gitignore rule silently swallowed the
shared plugin module source, so CI checkouts had only the vendored copies
and sync.test.mjs failed with ENOENT on the source directory.
* fix(recall): budget summary/uri fallbacks and sanitize non-finite scores
max_chars is the recall API's contract, but only full fragments counted
toward it — VikingBot's client-side heuristic, faithfully ported, lets
summary and uri fallbacks render far past the budget (repro: max_chars=100
rendered 548 chars). Every fragment now counts; oversized summaries degrade
to uri fragments and entries that can't even fit a uri line are dropped
(reported via stats.dropped). VikingBot itself is intentionally unchanged.
Also run _sanitize_floats over the /recall response like the neighboring
/find and /search routes, so inf/nan scores return 0.0 instead of a 500.
* feat(storage): optimize glob func
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* fix(localfs): offload blocking fs operations to spawn_blocking
* feat(glob): cap glob api default node_limit at 256
* feat(sdk): add node_limit options for glob in python and go SDKs
* Refactor recursive web import into HTTP accessor
Move ordinary web page import routing into HTTPAccessor and materialize crawled pages as a temporary directory via WebImporter.
Relocate Scrapy/Playwright crawling under parse.accessors.web_crawler, keep trafilatura extraction inside HTMLParser, and avoid repeated ResourceService.add_resource calls.
Add recursive crawl controls, safe request validation, page/download classification, and focused unit coverage.
* Document recursive web crawler options
* fix(web-crawler): stop SSRF sub-resource block from failing whole render
The playwright fallback validated every sub-resource request against the
SSRF guard and raised on the first disallowed host, failing the entire
page render. volcengine docs load a probe resource on an internal host,
so rendering always failed and the crawler stored the static anti-bot
"Please wait..." challenge page as content.
Now a blocked sub-resource is only aborted; the main document and final
URL still gate the result. Also wait past JS interstitials, retry reads
through in-flight navigation, and reject shell/challenge pages instead of
storing them.
* fix(web-crawler): surface renderer error hint on entry-page failure
When Playwright is unavailable, the renderer returns an actionable install
hint via RenderResult.error, but the spider silently kept the static shell
and WebImporter raised only the generic "Failed to fetch entry page". The
hint never reached the user.
Now the spider records rendered.error on the failed page, and WebImporter
appends the entry page's failure reason to the raised message so the CLI
shows the Playwright install instructions.
* fix(web-crawler): surface render hints and enforce crawl limits
* fix(web-crawler): avoid rendering SSR app pages
* perf(web-crawler): bound render concurrency and cap networkidle wait
Playwright renders were dispatched from parse callbacks without any
concurrency limit, so a page with many child links could spawn dozens of
Chromium pages at once (observed peak 28 for a 20-page crawl), risking OOM
on large sites and starting ~2.3x more renders than needed before
max_pages stopped the crawl. Gate renders with a semaphore sized to
config.concurrency and re-check the success limit after acquiring a slot
so queued callbacks skip rendering once the crawl is already done.
Also cap the networkidle wait at 8s: pages with continuous background
activity (e.g. GraphiQL) never go idle and previously blocked until the
full render timeout, turning a ~3s page into ~38s. Content is ready after
domcontentloaded and _wait_past_challenge covers late-arriving text.
Bump default concurrency 5 -> 10.
* fix(web-crawler): route .html/.htm URLs through recursive WebImporter
An explicit .html/.htm URL is detected as DOWNLOAD_HTML via the extension
map, so access() previously only routed URLType.WEBPAGE to WebImporter and
these URLs fell through to single-file download, silently ignoring
depth/max_pages. Route DOWNLOAD_HTML through WebImporter too, treating a
single-page import as the depth=0 case.
* fix(web-crawler): improve HTML extraction and rendering heuristics
- Drop trafilatura favor_precision=True: it stripped the full body of
link-dense pages, keeping only headers.
- Only render __NEXT_DATA__ pages with Playwright when their static body
is too thin; SSR/SSG Next.js pages already ship full text.
- Disable Scrapy telnet console to avoid opening port 6023.
* fix(web-crawler): keep code-hosting single-file URLs off recursive crawler
GitHub/GitLab blob and GitHub raw URLs resolve to a single file, not a
site. Route them through the single-file download path instead of the
recursive WebImporter, which otherwise crawls the hosting UI shell.
* docs(resources): add recursive web crawler usage examples
Add depth/max_pages crawl examples to the HTTP, Python SDK, and CLI
blocks in both the zh and en resource API docs, plus path-prefix
filtering and skip_download_links variants.
* fix(server): restore identity resolution in /health endpoint
The /health endpoint was refactored in #2503 which removed the identity
resolution logic. This caused the frontend dashboard to show 'Usage/Audit
未初始化' because the role field was missing from the response.
Restored the identity resolution so that /health returns account_id,
user_id, and role when an API key is provided.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(server): update health endpoint tests for identity resolution
- Test that /health returns identity info when API key is provided
- Test that /health omits identity info when no API key is provided
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: wugj <wugj@g-bits.com>
Co-authored-by: Claude <noreply@anthropic.com>
OAuth Dynamic Client Registration silently dropped the client `scope`, so any
spec-compliant MCP client that requests a scope (e.g. scope=mcp) failed
/authorize with invalid_scope. The MCP SDK's client.validate_scope() requires
requested scopes to be a subset of the registered client.scope, which was always
None because register_client() never stored it.
- storage.py: add `scope` column to oauth_clients (+ idempotent migration) and
persist it in register_client()
- provider.py: pass client_info.scope on registration; return scope from
get_client() so validate_scope() sees the registered scope
- router.py: advertise scopes_supported=["mcp"] in the RFC 9728 PRM document
- tests: assert scope round-trips through register/get
Co-authored-by: wugj <wugj@g-bits.com>
* feat(auth): support seeded API key generation
Allow admin key issuance flows to accept an optional seed so clients can derive predictable user API keys when needed, while preserving random generation by default.
* fix(go-sdk): preserve explicit empty seed payloads
Use pointer seed options so callers can distinguish omitted seeds from explicit empty seeds, matching the Admin API behavior.
Collapse the MCP add_resource local-file flow to a single step: the agent
POSTs the file to a token-authorized temp_upload URL and the server finishes
ingestion in the same request, so no second add_resource(temp_file_id) call
is needed.
- Merge the signed upload into POST /api/v1/resources/temp_upload via a
two-layer auth dependency (API key first, else a one-time ?token=), and
remove the dedicated temp_upload_signed route. The API-key path is
unchanged (still returns temp_file_id) for the CLI and import_ovpack.
- Bind to/reason/actor_peer_id into the upload token so auto-ingest keeps
the caller's target, reason, and peer scope; on the token path identity
and actor peer come only from the token, never from upload request
headers.
- Extract ingest_temp_upload() helper and surface add_resource business
errors instead of reporting a false success (mark_failed on error, and
route the result through response_from_result / the MCP error string).
- Update en/zh docs for the single-step flow.
[EN]
#2863 made content-write refresh recursive but kept the anchor collapsed to the
resource project root. As a result, writing a single deeply-nested file makes the DAG
recursively walk the ENTIRE project subtree; and because vectorize tasks are
batch-dispatched only at the end of a DAG run, the changed file's L2 vector is not
enqueued for embedding until that whole subtree walk plus the full bottom-up overview
chain (leaf -> ... -> project root) completes.
This change anchors the refresh at the written file's DIRECT PARENT directory instead.
The changed file is then a direct child of the DAG run root, so its own L2 vector and the
parent's L0/L1 are produced from a single-directory run; ancestor summaries still refresh
via the existing parent bubble. The final set of refreshed directories and re-embedded
vectors is unchanged, but the changed file's vector latency drops from a
whole-project-subtree run to a single-directory run.
Details:
- New `anchor_to_parent` flag on `_resolve_root_uri`, used only by the two write paths
(`write` / `_create_and_write`). `set_tags` keeps the project-root collapse that the
derived `.abstract.md` sidecar mapping relies on.
- Removes the `recursive=True` introduced by #2863, which is no longer needed once the
refresh is anchored at the correct (direct-parent) directory.
[中文]
#2863 把 content-write 的刷新改成了递归,但锚点仍然塌缩到资源项目根。于是写一个深层嵌套
文件会让 DAG 递归遍历整棵项目子树;又因为向量化任务是在一趟 DAG run 结束时才批量派发的,
被改文件的 L2 向量要等到整棵子树遍历 + 自底向上的 overview 链(叶子 -> … -> 项目根)全部
跑完之后,才会入嵌入队列。
本改动把刷新锚点改为被写文件的【直接父目录】。这样被改文件就是 DAG run 根目录的直接子文件,
它自己的 L2 向量和父目录的 L0/L1 只需一趟单目录 run 即可生成;祖先目录摘要仍然走现有的父
目录冒泡刷新。最终被刷新的目录集合、被重嵌的向量集合都不变,但被改文件向量的延迟从"整棵
项目子树 run"降到了"单目录 run"。
细节:
- 在 `_resolve_root_uri` 上新增 `anchor_to_parent` 开关,仅两个写路径(`write` /
`_create_and_write`)启用;`set_tags` 保留塌缩到项目根的语义(派生 `.abstract.md` 边车
映射依赖它)。
- 移除 #2863 引入的 `recursive=True`——锚点定位正确后不再需要它。
Co-authored-by: chenpengfei <chenpengfei@bytedance.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(grep): integrate VikingDB bm25 keyword search for grep engine
* fix(grep): address CI review feedback: max-size eviction to _count_cache, use Literal, Split regex alternation into individual keywords for bm25 (max 10)
* fix(schema): use dynamic __version__ for schema_version and handle dev suffixes in version comparison
* fix(schema): upsert data to vikingdb lack of content
* chore: add benchmark for retrieval
* fix(grep): vikingdb return 200 and no results means no matching content, not necessary to fallback to local fs
* fix(benchmark): sub uri args; add report
* refactor: code format by ruff
* optimize: move grep config (engine and switch_to_remote_threshold) to ov.conf
* optimize: auto adapt remote_return_limit by agg API; rm unnecessary params in keywords search
* fix: adjust benchmark scripts
* fix(grep): store full content for BM25; use PathScope depth; reduce redundant API calls
* refactor: new benchmark
* fix: step1 add resource by real code data
* feat(benchmark): split grep benchmark into effectiveness/performance suites with async reindex
* optimize (benchmark): adjust keywords and ground truth for testing
* fix: truncate 64KB for content field
* optimize: effectiveness add resource plainly
* optimize: change param use of SearchByKeywords from "keywords" to "query"
* optimize(benchmark): refactor effectiveness scripts
* optimize: ensure raw data for content field
* optimize: fulltext analyzer's stop-words only use symbols
* fix: adapt to new ov cli for benchmark
* optimize: reuse file content to avoid re-read AGFS file
* optimize: tune grep vikingdb defaults and refresh bm25 benchmark scripts
* optimize: benchmark client timeout
* update README
* fix: rm unused param
* fix: default values in docs
* optimize: increase truncate byte size to 1MB for content field for VikingDB
* fix(logger): harden queued stream logging (#2786)
* fix(logger): replace StreamHandler with QueueHandler+QueueListener to prevent thread deadlock
When log.output='stdout' (default) and the server is managed by systemd,
concurrent log writes can deadlock because logging.StreamHandler holds a
thread lock across stream.flush() which blocks on systemd-piped file I/O.
During session.commit() phase 2, multiple async coroutines (memory
extraction, summarization) concurrently call logger.info()/warning()
with large payloads. The first thread's flush() blocks on the pipe,
while all subsequent threads block on handler.acquire() forever.
This permanently silences the server log and prevents _write_done_file()
from executing, leaving phase 2 hanging without .done.
Fix: use QueueHandler + QueueListener from stdlib logging.handlers
(Python 3.2+). QueueHandler.emit() does queue.put(record) with no lock
or I/O, returning immediately. QueueListener has a dedicated single
thread as the sole consumer touching the real StreamHandler, making
lock contention impossible.
Changes in _create_log_handler(): stdout/stderr branches now create
a shared QueueListener with unbounded queue, returning QueueHandler
instances to callers. _build_standard_handler() delegates formatter
and filter setup to the real handler in the listener thread.
Closes: #2752
* fix(logger): harden queued stream logging
---------
Co-authored-by: njuboy11 <njuboy11@users.noreply.github.com>
---------
Co-authored-by: Qin Haojie <qinhaojie.exe@bytedance.com>
Co-authored-by: njuboy11 <njuboy11@users.noreply.github.com>
* fix(queuefs): skip semantic generation for non-directory memory URIs
A memory file reindexed with mode=semantic_and_vectors enqueues a
context_type="memory" semantic message whose URI is a file. on_dequeue
routes it to _process_memory_directory, which ls()'d the file, raised, and
the outer handler re-enqueued it as a transient error — forever, starving
the semantic queue. The entry is AGFS-persisted, so it survives a restart
and blocks `reindex --wait` and memory writes that wait on the queue.
_process_memory_directory now stat()s the URI first: a confirmed
non-directory (or missing) URI is marked done and skipped instead of
listed, so the message acks and the queue drains. When stat is unavailable
it falls through to the existing ls() path, leaving current behavior
unchanged.
Fixes#2734
* fix(queuefs): preserve retries for stat failures
* fix(reindex): skip memory semantics for file targets
---------
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
Treat failed session archives as terminal skipped state so later commits can continue, and replace OpenClaw's fixed auto-commit token threshold with a context-window ratio while tolerating the deprecated config key.
Co-authored-by: LinQiang391 <linqiang391@users.noreply.github.com>
* chore: clear unused files
* fix(tests): fix unit test
* refactor(auth): introduce plugin-based authentication architecture
Replace the monolithic `openviking/server/auth.py` with an extensible
plugin-based auth system. This refactor extracts the three built-in modes
(`dev`, `api_key`, `trusted`) into separate `AuthPlugin` implementations,
adds a registry for third-party plugins, and preserves all existing behavior
while enabling custom authentication backends (e.g. LDAP, OIDC, mTLS).
Key changes:
- **New public API**: `AuthPlugin` (ABC) and `register_auth_plugin` decorator.
- **New registry**: `AuthPluginRegistry` supports runtime registration.
- **Built-in plugins**: `DevAuthPlugin`, `ApiKeyAuthPlugin`, `TrustedAuthPlugin`.
- **Config change**: `auth_mode` widened from `Literal` to `str` for custom modes.
- **Validation delegated**: `validate_server_config()` now delegates to the active
plugin's `validate_config()`, preserving existing validation semantics.
- **Router compatibility**: All existing `require_*` decorators and `resolve_identity`
/ `get_request_context` dependencies remain unchanged. Routers import the same
symbols from `openviking.server.auth`.
- **Tests**: `conftest.py` manually wires the DevAuthPlugin in ASGI tests (lifespan
not triggered). `test_auth.py` expanded with plugin registration and validation tests.
- **Docs**: `04-authentication.md` (en/zh) updated with plugin registration examples.
Co-Authored-By: claude-sonnet-4-6 <noreply@anthropic.com>
* fix(tests): fix trusted mode test
* fix(tests): fix unit test
---------
Co-authored-by: claude-sonnet-4-6 <noreply@anthropic.com>
Add HTTP APIs for code outline, search, and expansion backed by the existing AST tooling.
Expose the same capabilities through the opencode plugin and cover the new routes, parser behavior, and plugin wiring with tests.
Qin Haojie