* feat(plugins): align opencode and pi memory integrations
* fix(installer): tolerate missing optional harness CLIs
* fix(installer): install opencode file wrapper
* fix(opencode): import path for logger initialization
* fix(installer): register pi extension after copy
* feat(plugins): use MCP for opencode integration
* docs: move OpenCode and pi integrations to dedicated pages
Promote the OpenCode plugin and pi extension out of the community-plugins
page into their own numbered agent-integrations pages (10-opencode, 11-pi,
en + zh), update the overview routing table, and refresh the OpenCode image
cards to the hybrid MCP architecture (unified installer, openviking_* MCP
tools, ovcli.conf credentials).
* docs: bare TOS installer commands and reference more examples
Drop --harness from TOS-mirror install commands (image cards use the bare
installer URL, matching the claude-code/codex cards); add Open WebUI tool
server and an examples/ pointer to the community-plugins page (en + zh).
* docs: bare TOS installer commands across agent-integration pages
TOS-mirror install commands carry no flags anywhere; the installer wizard
asks for source, harnesses, language, and credentials.
* feat: add memory plugin mcp harness
* refactor: vendor shared memory plugin modules
* feat: add type quota recall api
* feat: commit codex memory by token threshold
* feat: capture codex tool calls as parts
* feat: add claude skill experience recall
* chore: fix lint in type quota recall server files
* feat: remote marketplace install with unified openviking naming
- Fix root .claude-plugin/marketplace.json git-subdir discriminator key
("type" -> "source"); claude plugin validate now passes.
- Unified installer gains --source remote|archive|dev: remote registers a
synthesized git-subdir marketplace for Claude Code and a git marketplace
for Codex (no repo clone); archive consumes the slim TOS marketplace zip;
dev registers the checkout's examples/ directory for both harnesses.
- One marketplace name (openviking) across all modes and harnesses, so the
plugin id is always openviking-memory@openviking; installer migrates old
openviking-plugins-local registrations and config.toml sections.
- Restore legacy Claude Code (<2.0) support: claude mcp add (stdio proxy)
plus node-based hooks merge into ~/.claude/settings.json.
- Restore optional statusline registration (fetches sources on opt-in).
- Checkbox TUI harness selection via /dev/tty with non-tty fallback.
- Add examples/.agents/plugins/marketplace.json so Codex directory installs
drop the synthetic symlink marketplace.
- Add shared setup wizard (scripts/setup.mjs) for pure-marketplace installs.
- release-tos.yml: upload memory-plugin-shared/install.sh and build/upload
the memory-plugin-marketplace zip; tos-install.sh prefers it and pins all
fetches to TOS via OPENVIKING_SHARED_INSTALL_URL.
- CI: bash -n on installer scripts; marketplace contract tests updated.
* fix(installer): register Claude remote marketplace as a directory
File-type marketplaces (bare marketplace.json path) make Claude Code derive
a wrong installLocation and 'marketplace update' fails with EISDIR. Write
the synthesized manifest to <dir>/.claude-plugin/marketplace.json and add
the directory instead; compare registered sources by exact match so the
old file registration migrates cleanly.
* feat(statusline): show model name and native-style context percentage
A custom statusLine replaces Claude Code's native line including its context
indicator, so reproduce it from the statusline stdin payload: 'Fable 5 ·
ctx 42%' right after the health segment, with native color thresholds
(<70% dim, 70-89% yellow, >=90% red). Falls back from used_percentage to
remaining_percentage to token counts, and stays visible in bypass mode
since it describes the CC conversation, not OV. Opt out with
OPENVIKING_STATUSLINE_CTX=off. Line cap raised 80 -> 100 visible chars.
* fix(installer): keep checkout progress off stdout in plugin_dir_on_disk
Callers capture the function's stdout, so ensure_checkout's info lines were
concatenated into the statusline command registered in settings.json.
* fix(installer): re-register codex git marketplace instead of upgrading
Codex doesn't expose which --ref a git marketplace was added with, and
'marketplace upgrade' refreshes the old ref — so a URL match must not skip
re-registration or a ref override installs the wrong snapshot. Also remove
the stale pre-unification plugin cache directory during migration.
* fix(installer): include .agents in codex sparse checkout
A plugin-dir-only sparse checkout omits the repo-root marketplace manifest
and fails with 'marketplace root does not contain a supported manifest'.
Adding --sparse .agents keeps the snapshot slim (~7.5M vs full repo).
* feat(installer): bilingual prompts, dist channel selection, and TOS git marketplace for codex
- Interactive language selection (English/中文, --lang, auto-detected from
locale); every user-facing prompt is bilingual.
- Download-source selection (--dist github|tos, prompted interactively):
github keeps the remote marketplaces; tos serves GitHub-blocked regions.
- Credentials step now always shows the current ovcli.conf values (masked
key) and offers keep-or-reconfigure instead of silently reusing them.
- Codex on TOS installs from a TOS-hosted git repo over dumb HTTP and keeps
remote updates (codex plugin marketplace upgrade); falls back to the
archive directory if the repo is unavailable. release-tos.yml builds and
uploads the single-commit bare repo (repack + update-server-info).
- Claude Code on TOS warns that directory marketplaces cannot auto-update.
- tos-install.sh bootstraps shrink to TOS_BASE + --dist tos.
- Docs (READMEs, agent-integrations pages, image cards, en+zh) now all use
the single shared installer and drop the deleted wrapper instructions.
* feat(installer): unify all choice prompts on an arrow-key TUI menu
Language, download source, connection mode, keep-or-reconfigure
credentials, statusline enable/replace, and legacy-mode confirmation all
render as the same single-select menu (arrow keys / digit shortcuts /
enter, radio-style highlight) instead of mixed numbered and y/N prompts.
Falls back to numbered input when /dev/tty can't be drawn on and to the
default choice when non-interactive. Free-text fields (URL, API key) stay
line inputs; the harness picker keeps its checkbox multi-select.
* fix(installer): stop piping plugin lists into grep -q under pipefail
grep -q exits on first match and SIGPIPEs the producer, so with pipefail
the 'codex plugin list | grep -q' check read as a miss every time (codex's
list is long; claude's short list masked the bug). Capture the output and
substring-match in bash instead — validation no longer false-warns.
Also: drop the stdio-proxy line from the Done summary; always offer the
install-source menu unless --dist/--source was given (with a checkout the
menu gains a dev option and defaults to it); surface the Claude-on-TOS
no-auto-update warning at source resolution instead of after install.
* fix: unignore examples/memory-plugin-shared/lib and commit the shared modules
The Python build-artifact 'lib/' gitignore rule silently swallowed the
shared plugin module source, so CI checkouts had only the vendored copies
and sync.test.mjs failed with ENOENT on the source directory.
* fix(recall): budget summary/uri fallbacks and sanitize non-finite scores
max_chars is the recall API's contract, but only full fragments counted
toward it — VikingBot's client-side heuristic, faithfully ported, lets
summary and uri fallbacks render far past the budget (repro: max_chars=100
rendered 548 chars). Every fragment now counts; oversized summaries degrade
to uri fragments and entries that can't even fit a uri line are dropped
(reported via stats.dropped). VikingBot itself is intentionally unchanged.
Also run _sanitize_floats over the /recall response like the neighboring
/find and /search routes, so inf/nan scores return 0.0 instead of a 500.
* feat(storage): optimize glob func
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* feat(rgafs): implement paged glob traversal without full tree materialization
* fix(localfs): offload blocking fs operations to spawn_blocking
* feat(glob): cap glob api default node_limit at 256
* feat(sdk): add node_limit options for glob in python and go SDKs
* refactor(web-crawler): remove Playwright rendering, keep static SPA shells
Drop the Playwright-based fallback rendering path so SPA pages are stored
as their static shell HTML instead of being rendered headlessly. SPA
shells now surface the <noscript> notice (e.g. "You need to enable
JavaScript to run this app.") rather than producing an empty document,
and robots.txt-blocked entry pages get a human-readable error message.
- Delete playwright_renderer.py, render_heuristics.py and their tests
- Strip fallback_playwright/playwright_timeout config, fallback_rendered
counter, and render-hint plumbing from the crawler and web importer
- HTMLParser: drop the SPA-empty-pattern stripping and fall back to
<noscript> text when trafilatura extracts nothing
- Humanize the robots.txt entry-failure message
- Migrate the orphaned _convert_to_raw_url tests to HTTPAccessor (the
method moved there in an earlier reorg) and remove the stale file
* refactor(web-importer): simplify robots.txt-blocked import message
Replace the verbose robots.txt explanation with a short compliance hint
that omits the URL and points users to local-file import instead.
* Refactor recursive web import into HTTP accessor
Move ordinary web page import routing into HTTPAccessor and materialize crawled pages as a temporary directory via WebImporter.
Relocate Scrapy/Playwright crawling under parse.accessors.web_crawler, keep trafilatura extraction inside HTMLParser, and avoid repeated ResourceService.add_resource calls.
Add recursive crawl controls, safe request validation, page/download classification, and focused unit coverage.
* Document recursive web crawler options
* fix(web-crawler): stop SSRF sub-resource block from failing whole render
The playwright fallback validated every sub-resource request against the
SSRF guard and raised on the first disallowed host, failing the entire
page render. volcengine docs load a probe resource on an internal host,
so rendering always failed and the crawler stored the static anti-bot
"Please wait..." challenge page as content.
Now a blocked sub-resource is only aborted; the main document and final
URL still gate the result. Also wait past JS interstitials, retry reads
through in-flight navigation, and reject shell/challenge pages instead of
storing them.
* fix(web-crawler): surface renderer error hint on entry-page failure
When Playwright is unavailable, the renderer returns an actionable install
hint via RenderResult.error, but the spider silently kept the static shell
and WebImporter raised only the generic "Failed to fetch entry page". The
hint never reached the user.
Now the spider records rendered.error on the failed page, and WebImporter
appends the entry page's failure reason to the raised message so the CLI
shows the Playwright install instructions.
* fix(web-crawler): surface render hints and enforce crawl limits
* fix(web-crawler): avoid rendering SSR app pages
* perf(web-crawler): bound render concurrency and cap networkidle wait
Playwright renders were dispatched from parse callbacks without any
concurrency limit, so a page with many child links could spawn dozens of
Chromium pages at once (observed peak 28 for a 20-page crawl), risking OOM
on large sites and starting ~2.3x more renders than needed before
max_pages stopped the crawl. Gate renders with a semaphore sized to
config.concurrency and re-check the success limit after acquiring a slot
so queued callbacks skip rendering once the crawl is already done.
Also cap the networkidle wait at 8s: pages with continuous background
activity (e.g. GraphiQL) never go idle and previously blocked until the
full render timeout, turning a ~3s page into ~38s. Content is ready after
domcontentloaded and _wait_past_challenge covers late-arriving text.
Bump default concurrency 5 -> 10.
* fix(web-crawler): route .html/.htm URLs through recursive WebImporter
An explicit .html/.htm URL is detected as DOWNLOAD_HTML via the extension
map, so access() previously only routed URLType.WEBPAGE to WebImporter and
these URLs fell through to single-file download, silently ignoring
depth/max_pages. Route DOWNLOAD_HTML through WebImporter too, treating a
single-page import as the depth=0 case.
* fix(web-crawler): improve HTML extraction and rendering heuristics
- Drop trafilatura favor_precision=True: it stripped the full body of
link-dense pages, keeping only headers.
- Only render __NEXT_DATA__ pages with Playwright when their static body
is too thin; SSR/SSG Next.js pages already ship full text.
- Disable Scrapy telnet console to avoid opening port 6023.
* fix(web-crawler): keep code-hosting single-file URLs off recursive crawler
GitHub/GitLab blob and GitHub raw URLs resolve to a single file, not a
site. Route them through the single-file download path instead of the
recursive WebImporter, which otherwise crawls the hosting UI shell.
* docs(resources): add recursive web crawler usage examples
Add depth/max_pages crawl examples to the HTTP, Python SDK, and CLI
blocks in both the zh and en resource API docs, plus path-prefix
filtering and skip_download_links variants.
* feat(auth): support seeded API key generation
Allow admin key issuance flows to accept an optional seed so clients can derive predictable user API keys when needed, while preserving random generation by default.
* fix(go-sdk): preserve explicit empty seed payloads
Use pointer seed options so callers can distinguish omitted seeds from explicit empty seeds, matching the Admin API behavior.
Collapse the MCP add_resource local-file flow to a single step: the agent
POSTs the file to a token-authorized temp_upload URL and the server finishes
ingestion in the same request, so no second add_resource(temp_file_id) call
is needed.
- Merge the signed upload into POST /api/v1/resources/temp_upload via a
two-layer auth dependency (API key first, else a one-time ?token=), and
remove the dedicated temp_upload_signed route. The API-key path is
unchanged (still returns temp_file_id) for the CLI and import_ovpack.
- Bind to/reason/actor_peer_id into the upload token so auto-ingest keeps
the caller's target, reason, and peer scope; on the token path identity
and actor peer come only from the token, never from upload request
headers.
- Extract ingest_temp_upload() helper and surface add_resource business
errors instead of reporting a false success (mark_failed on error, and
route the result through response_from_result / the MCP error string).
- Update en/zh docs for the single-step flow.
* feat(ingest): replay local agent-harness logs into OV sessions / 本地 agent harness 日志重放入库
Add openviking/ingest/: parse Claude Code / Codex / OpenCode / Hermes / OpenClaw conversation logs into normalized messages and replay them through OpenViking's existing session pipeline (create_session -> batch_add_messages -> commit -> async memory extraction), instead of a bespoke ETL.
Supports one-shot backfill ("存量") and cursor-driven incremental polling ("新增", WatchScheduler-style, no fs-event dependency), per-harness enable/mode/paths config, and meaningful peer_id on every turn (assistant = {harness}/{model}; user = git identity for single-user harnesses, original username for group-chat harnesses). Cursor IDE is a registered but deferred stub.
Read-position cursors persist under ~/.openviking/ingest/state.db for crash-safe, idempotent resume. New openviking-ingest CLI (backfill/watch/run/status/list-sources) and an "ingest" section on OpenVikingConfig. Verified end-to-end against a local server: 3-message fixture -> session commit -> 10 memories extracted -> idempotent re-run.
Inspired by / supersedes volcengine/OpenViking#2674.
Co-authored-by: baobaodae <2014596548@qq.com>
* docs(ingest): bilingual guide + ov.conf.example for openviking-ingest / 本地日志入库双语文档与配置示例
Add docs/{zh,en}/agent-integrations/09-log-ingestion.md (auto-registered in the VitePress sidebar) and an `ingest` section in examples/ov.conf.example (off by default).
* fix(ingest): address review — gating, crash-safe batch replay, commit recovery, single-instance lock / 修复评审问题
Fixes the merge-blockers from the adversarial review:
- master switch ingest.enabled now actually gates enabled_harnesses();
- idempotent per-batch append with a durable pending-intent reconciled against the server message count on restart (no duplicate imports after a mid-append crash);
- bounded reads (<=100 msgs/call) so huge sessions don't materialize at once;
- needs_commit flag + commit_if_needed so appended-but-uncommitted sessions still get extracted (commit even when no new source rows);
- poller keeps dirty sessions until a commit actually succeeds;
- OpenCode advances its SQLite cursor only past complete rows (late part text no longer skipped);
- single-instance file lock guards concurrent ingest processes;
- positive-value config validation (no poll busy-loop); malformed ov.conf surfaces instead of silently defaulting.
Adds 6 tests (config gating/validation, crash reconcile both ways, commit recovery).
* refactor(ingest): expose as 'openviking-server ingest' subcommand; English-only code/docs
- Route the ingest CLI through 'openviking-server ingest ...' (same dispatch as 'init'/'doctor') and drop the separate 'openviking-ingest' console_script.
- Remove mixed-in Chinese terms (存量/新增) from source docstrings, CLI help, and the English doc; the Chinese doc keeps them.
* style(ingest): ruff format + import sort (isort I)
Run ruff 0.15.16 (from the uv cache) with the repo config: fixes 5 I001 import-order errors in tests and reformats 9 files. 'ruff check' and 'ruff format --check' now pass on all added/edited files.
---------
Co-authored-by: baobaodae <2014596548@qq.com>
#2879 added a Codex marketplace section to the English integration docs. Revert
those docs-only additions so docs/ matches the pre-#2879 state — no marketplace
content in the agent-integration pages.
Keep the marketplace path in the plugin README, but demote it: it is local-only
(unauthenticated http://127.0.0.1:1933), does not support authenticated or
remote/cloud servers, and is not recommended. Reorder the Quick Start so the
one-line installer is path A (recommended, supports remote/cloud) and the
marketplace install is path B (local-only, not recommended).
Add WebFeedAccessor (priority 60) that turns a single sitemap /
sitemapindex / RSS / Atom URL into ONE resource tree: it mirrors every
listed page into a temp directory and reuses the existing DirectoryParser
pipeline (the same "fetch-many -> dir -> tree" contract as GitAccessor).
A watch on the feed URL keeps the whole site refreshed (new pages added,
removed pages dropped on each rebuild).
- New openviking/parse/accessors/web_feed_accessor.py: WebFeedAccessor +
sitemap/feed extractors (nested sitemapindex recursion with depth cap,
RSS 2.0 / Atom via feedparser), bounded concurrent polite mirroring,
robots.txt, same-host / include / exclude / max_pages limits.
- args={"site": true} forces whole-site ingestion from a bare domain or
page by auto-discovering the sitemap/RSS (robots.txt, HTML
<link rel=alternate>, conventional paths); {"site": false} opts a
feed-looking URL back out to HTTPAccessor.
- Thread accessor-selection kwargs through can_handle; the registry
tolerates accessors whose can_handle lacks **kwargs (back-compatible).
- Single-page adds get a non-blocking "this site exposes a sitemap/RSS"
suggestion appended to the MCP add_resource response, gated to the
site root only; never auto-crawls.
- New WebFeedConfig (parsers.webfeed): max_pages, concurrency, politeness
delay, same_host_only, respect_robots, max_depth, suggest_feed.
- Dependencies: feedparser (robust RSS/Atom), defusedxml (XXE-safe XML).
- Docs: zh/en resources API, MCP/CLI/SDK help, ov.conf.example.
- Tests: 52 unit tests (fake httpx, no network).
* chore: clear unused files
* fix(tests): fix unit test
* refactor(auth): introduce plugin-based authentication architecture
Replace the monolithic `openviking/server/auth.py` with an extensible
plugin-based auth system. This refactor extracts the three built-in modes
(`dev`, `api_key`, `trusted`) into separate `AuthPlugin` implementations,
adds a registry for third-party plugins, and preserves all existing behavior
while enabling custom authentication backends (e.g. LDAP, OIDC, mTLS).
Key changes:
- **New public API**: `AuthPlugin` (ABC) and `register_auth_plugin` decorator.
- **New registry**: `AuthPluginRegistry` supports runtime registration.
- **Built-in plugins**: `DevAuthPlugin`, `ApiKeyAuthPlugin`, `TrustedAuthPlugin`.
- **Config change**: `auth_mode` widened from `Literal` to `str` for custom modes.
- **Validation delegated**: `validate_server_config()` now delegates to the active
plugin's `validate_config()`, preserving existing validation semantics.
- **Router compatibility**: All existing `require_*` decorators and `resolve_identity`
/ `get_request_context` dependencies remain unchanged. Routers import the same
symbols from `openviking.server.auth`.
- **Tests**: `conftest.py` manually wires the DevAuthPlugin in ASGI tests (lifespan
not triggered). `test_auth.py` expanded with plugin registration and validation tests.
- **Docs**: `04-authentication.md` (en/zh) updated with plugin registration examples.
Co-Authored-By: claude-sonnet-4-6 <noreply@anthropic.com>
* fix(tests): fix trusted mode test
* fix(tests): fix unit test
* fix(cli): remove unexisted transaction observer
* docs: update skills definition
* docs: update skills definition
* docs: update skills definition
* docs: update skills definition
* fix(skills): now we allow viking://agent/skills again, and optimize CLI for skills
* docs(skills): use -p instead of --parent in agent skills examples
Align the `ov skills add` examples in the context-types and viking-uri
docs with the short flag `-p` introduced for `ov skills list/find/show`,
so all four user-facing examples consistently demonstrate the short form
when targeting `viking://agent/skills`.
Co-Authored-By: claude-sonnet-4-6 <noreply@anthropic.com>
* fix(tests): error check for api key
* fix(tests): unit test wait until resource not busy
* fix(tests): unit test wait until resource not busy
* fix(sdk): args form in skills find
* fix(skills): pass target uri in request body
---------
Co-authored-by: claude-sonnet-4-6 <noreply@anthropic.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
* feat(grep): integrate VikingDB bm25 keyword search for grep engine
* fix(grep): address CI review feedback: max-size eviction to _count_cache, use Literal, Split regex alternation into individual keywords for bm25 (max 10)
* fix(schema): use dynamic __version__ for schema_version and handle dev suffixes in version comparison
* fix(schema): upsert data to vikingdb lack of content
* chore: add benchmark for retrieval
* fix(grep): vikingdb return 200 and no results means no matching content, not necessary to fallback to local fs
* fix(benchmark): sub uri args; add report
* refactor: code format by ruff
* optimize: move grep config (engine and switch_to_remote_threshold) to ov.conf
* optimize: auto adapt remote_return_limit by agg API; rm unnecessary params in keywords search
* fix: adjust benchmark scripts
* fix(grep): store full content for BM25; use PathScope depth; reduce redundant API calls
* refactor: new benchmark
* fix: step1 add resource by real code data
* feat(benchmark): split grep benchmark into effectiveness/performance suites with async reindex
* optimize (benchmark): adjust keywords and ground truth for testing
* fix: truncate 64KB for content field
* optimize: effectiveness add resource plainly
* optimize: change param use of SearchByKeywords from "keywords" to "query"
* optimize(benchmark): refactor effectiveness scripts
* optimize: ensure raw data for content field
* optimize: fulltext analyzer's stop-words only use symbols
* fix: adapt to new ov cli for benchmark
* optimize: reuse file content to avoid re-read AGFS file
* optimize: tune grep vikingdb defaults and refresh bm25 benchmark scripts
* optimize: benchmark client timeout
* update README
* fix: rm unused param
* fix: default values in docs
* optimize: increase truncate byte size to 1MB for content field for VikingDB
* fix(logger): harden queued stream logging (#2786)
* fix(logger): replace StreamHandler with QueueHandler+QueueListener to prevent thread deadlock
When log.output='stdout' (default) and the server is managed by systemd,
concurrent log writes can deadlock because logging.StreamHandler holds a
thread lock across stream.flush() which blocks on systemd-piped file I/O.
During session.commit() phase 2, multiple async coroutines (memory
extraction, summarization) concurrently call logger.info()/warning()
with large payloads. The first thread's flush() blocks on the pipe,
while all subsequent threads block on handler.acquire() forever.
This permanently silences the server log and prevents _write_done_file()
from executing, leaving phase 2 hanging without .done.
Fix: use QueueHandler + QueueListener from stdlib logging.handlers
(Python 3.2+). QueueHandler.emit() does queue.put(record) with no lock
or I/O, returning immediately. QueueListener has a dedicated single
thread as the sole consumer touching the real StreamHandler, making
lock contention impossible.
Changes in _create_log_handler(): stdout/stderr branches now create
a shared QueueListener with unbounded queue, returning QueueHandler
instances to callers. _build_standard_handler() delegates formatter
and filter setup to the real handler in the listener thread.
Closes: #2752
* fix(logger): harden queued stream logging
---------
Co-authored-by: njuboy11 <njuboy11@users.noreply.github.com>
---------
Co-authored-by: Qin Haojie <qinhaojie.exe@bytedance.com>
Co-authored-by: njuboy11 <njuboy11@users.noreply.github.com>
yufeng