* feat: support oidc and ldap auth
* feat: support oidc and ldap auth
* fix: remove heima partner, clean up auth docs, add web-studio unsupported auth banner
- Remove heima from partner list in README (en/zh/ja)
- Remove unsupported env var references (OPENVIKING_AUTH_MODE, OPENVIKING_USERNAME,
OPENVIKING_PASSWORD) from LDAP auth docs
- Remove temporary switch bash snippets from auth docs
- Fix ldap_password description
- Add web-studio unsupported-auth-mode banner for oidc/ldap servers
* fix: address OIDC/LDAP review comments on auth plugin design
Key changes driven by PR review:
- **Role mapping**: OIDC and LDAP external identities always resolve to
USER role. Removed map_role() calls and group_membership-based role
mapping. Admin access is gated by the root API key mechanism only.
- **LDAP credential extraction**: Removed query-parameter-based username/
password extraction (security concern — passwords in URLs can leak via
shell history, proxy logs, and monitoring). Clients must use Basic Auth
header or form data.
- **OIDC identifier sanitization**: Auth0 and other providers may include
characters like "|" in the `sub` claim. These are now replaced with "_"
to produce valid OpenViking user identifiers.
- **Dead code removal**: Removed _extract_groups, memberof_attribute,
require_root_api_key_for_admin, _initialize_api_key_manager, and
get_request_context_checks from both plugins since they are no longer
needed.
- **Docs**: Removed query-parameter curl example, memberof_attribute and
require_root_api_key_for_admin config references.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* feat: support oidc and ldap auth
* feat: support oidc and ldap auth
* fix(auth): bind lazy OIDC imports at module scope
---------
Co-authored-by: TRAE CLI <noreply@bytedance.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
* Refactor recursive web import into HTTP accessor
Move ordinary web page import routing into HTTPAccessor and materialize crawled pages as a temporary directory via WebImporter.
Relocate Scrapy/Playwright crawling under parse.accessors.web_crawler, keep trafilatura extraction inside HTMLParser, and avoid repeated ResourceService.add_resource calls.
Add recursive crawl controls, safe request validation, page/download classification, and focused unit coverage.
* Document recursive web crawler options
* fix(web-crawler): stop SSRF sub-resource block from failing whole render
The playwright fallback validated every sub-resource request against the
SSRF guard and raised on the first disallowed host, failing the entire
page render. volcengine docs load a probe resource on an internal host,
so rendering always failed and the crawler stored the static anti-bot
"Please wait..." challenge page as content.
Now a blocked sub-resource is only aborted; the main document and final
URL still gate the result. Also wait past JS interstitials, retry reads
through in-flight navigation, and reject shell/challenge pages instead of
storing them.
* fix(web-crawler): surface renderer error hint on entry-page failure
When Playwright is unavailable, the renderer returns an actionable install
hint via RenderResult.error, but the spider silently kept the static shell
and WebImporter raised only the generic "Failed to fetch entry page". The
hint never reached the user.
Now the spider records rendered.error on the failed page, and WebImporter
appends the entry page's failure reason to the raised message so the CLI
shows the Playwright install instructions.
* fix(web-crawler): surface render hints and enforce crawl limits
* fix(web-crawler): avoid rendering SSR app pages
* perf(web-crawler): bound render concurrency and cap networkidle wait
Playwright renders were dispatched from parse callbacks without any
concurrency limit, so a page with many child links could spawn dozens of
Chromium pages at once (observed peak 28 for a 20-page crawl), risking OOM
on large sites and starting ~2.3x more renders than needed before
max_pages stopped the crawl. Gate renders with a semaphore sized to
config.concurrency and re-check the success limit after acquiring a slot
so queued callbacks skip rendering once the crawl is already done.
Also cap the networkidle wait at 8s: pages with continuous background
activity (e.g. GraphiQL) never go idle and previously blocked until the
full render timeout, turning a ~3s page into ~38s. Content is ready after
domcontentloaded and _wait_past_challenge covers late-arriving text.
Bump default concurrency 5 -> 10.
* fix(web-crawler): route .html/.htm URLs through recursive WebImporter
An explicit .html/.htm URL is detected as DOWNLOAD_HTML via the extension
map, so access() previously only routed URLType.WEBPAGE to WebImporter and
these URLs fell through to single-file download, silently ignoring
depth/max_pages. Route DOWNLOAD_HTML through WebImporter too, treating a
single-page import as the depth=0 case.
* fix(web-crawler): improve HTML extraction and rendering heuristics
- Drop trafilatura favor_precision=True: it stripped the full body of
link-dense pages, keeping only headers.
- Only render __NEXT_DATA__ pages with Playwright when their static body
is too thin; SSR/SSG Next.js pages already ship full text.
- Disable Scrapy telnet console to avoid opening port 6023.
* fix(web-crawler): keep code-hosting single-file URLs off recursive crawler
GitHub/GitLab blob and GitHub raw URLs resolve to a single file, not a
site. Route them through the single-file download path instead of the
recursive WebImporter, which otherwise crawls the hosting UI shell.
* docs(resources): add recursive web crawler usage examples
Add depth/max_pages crawl examples to the HTTP, Python SDK, and CLI
blocks in both the zh and en resource API docs, plus path-prefix
filtering and skip_download_links variants.
* feat(ingest): replay local agent-harness logs into OV sessions / 本地 agent harness 日志重放入库
Add openviking/ingest/: parse Claude Code / Codex / OpenCode / Hermes / OpenClaw conversation logs into normalized messages and replay them through OpenViking's existing session pipeline (create_session -> batch_add_messages -> commit -> async memory extraction), instead of a bespoke ETL.
Supports one-shot backfill ("存量") and cursor-driven incremental polling ("新增", WatchScheduler-style, no fs-event dependency), per-harness enable/mode/paths config, and meaningful peer_id on every turn (assistant = {harness}/{model}; user = git identity for single-user harnesses, original username for group-chat harnesses). Cursor IDE is a registered but deferred stub.
Read-position cursors persist under ~/.openviking/ingest/state.db for crash-safe, idempotent resume. New openviking-ingest CLI (backfill/watch/run/status/list-sources) and an "ingest" section on OpenVikingConfig. Verified end-to-end against a local server: 3-message fixture -> session commit -> 10 memories extracted -> idempotent re-run.
Inspired by / supersedes volcengine/OpenViking#2674.
Co-authored-by: baobaodae <2014596548@qq.com>
* docs(ingest): bilingual guide + ov.conf.example for openviking-ingest / 本地日志入库双语文档与配置示例
Add docs/{zh,en}/agent-integrations/09-log-ingestion.md (auto-registered in the VitePress sidebar) and an `ingest` section in examples/ov.conf.example (off by default).
* fix(ingest): address review — gating, crash-safe batch replay, commit recovery, single-instance lock / 修复评审问题
Fixes the merge-blockers from the adversarial review:
- master switch ingest.enabled now actually gates enabled_harnesses();
- idempotent per-batch append with a durable pending-intent reconciled against the server message count on restart (no duplicate imports after a mid-append crash);
- bounded reads (<=100 msgs/call) so huge sessions don't materialize at once;
- needs_commit flag + commit_if_needed so appended-but-uncommitted sessions still get extracted (commit even when no new source rows);
- poller keeps dirty sessions until a commit actually succeeds;
- OpenCode advances its SQLite cursor only past complete rows (late part text no longer skipped);
- single-instance file lock guards concurrent ingest processes;
- positive-value config validation (no poll busy-loop); malformed ov.conf surfaces instead of silently defaulting.
Adds 6 tests (config gating/validation, crash reconcile both ways, commit recovery).
* refactor(ingest): expose as 'openviking-server ingest' subcommand; English-only code/docs
- Route the ingest CLI through 'openviking-server ingest ...' (same dispatch as 'init'/'doctor') and drop the separate 'openviking-ingest' console_script.
- Remove mixed-in Chinese terms (存量/新增) from source docstrings, CLI help, and the English doc; the Chinese doc keeps them.
* style(ingest): ruff format + import sort (isort I)
Run ruff 0.15.16 (from the uv cache) with the repo config: fixes 5 I001 import-order errors in tests and reformats 9 files. 'ruff check' and 'ruff format --check' now pass on all added/edited files.
---------
Co-authored-by: baobaodae <2014596548@qq.com>
Add WebFeedAccessor (priority 60) that turns a single sitemap /
sitemapindex / RSS / Atom URL into ONE resource tree: it mirrors every
listed page into a temp directory and reuses the existing DirectoryParser
pipeline (the same "fetch-many -> dir -> tree" contract as GitAccessor).
A watch on the feed URL keeps the whole site refreshed (new pages added,
removed pages dropped on each rebuild).
- New openviking/parse/accessors/web_feed_accessor.py: WebFeedAccessor +
sitemap/feed extractors (nested sitemapindex recursion with depth cap,
RSS 2.0 / Atom via feedparser), bounded concurrent polite mirroring,
robots.txt, same-host / include / exclude / max_pages limits.
- args={"site": true} forces whole-site ingestion from a bare domain or
page by auto-discovering the sitemap/RSS (robots.txt, HTML
<link rel=alternate>, conventional paths); {"site": false} opts a
feed-looking URL back out to HTTPAccessor.
- Thread accessor-selection kwargs through can_handle; the registry
tolerates accessors whose can_handle lacks **kwargs (back-compatible).
- Single-page adds get a non-blocking "this site exposes a sitemap/RSS"
suggestion appended to the MCP add_resource response, gated to the
site root only; never auto-crawls.
- New WebFeedConfig (parsers.webfeed): max_pages, concurrency, politeness
delay, same_host_only, respect_robots, max_depth, suggest_feed.
- Dependencies: feedparser (robust RSS/Atom), defusedxml (XXE-safe XML).
- Docs: zh/en resources API, MCP/CLI/SDK help, ov.conf.example.
- Tests: 52 unit tests (fake httpx, no network).
* chore: clear unused files
* fix(tests): fix unit test
* refactor(auth): introduce plugin-based authentication architecture
Replace the monolithic `openviking/server/auth.py` with an extensible
plugin-based auth system. This refactor extracts the three built-in modes
(`dev`, `api_key`, `trusted`) into separate `AuthPlugin` implementations,
adds a registry for third-party plugins, and preserves all existing behavior
while enabling custom authentication backends (e.g. LDAP, OIDC, mTLS).
Key changes:
- **New public API**: `AuthPlugin` (ABC) and `register_auth_plugin` decorator.
- **New registry**: `AuthPluginRegistry` supports runtime registration.
- **Built-in plugins**: `DevAuthPlugin`, `ApiKeyAuthPlugin`, `TrustedAuthPlugin`.
- **Config change**: `auth_mode` widened from `Literal` to `str` for custom modes.
- **Validation delegated**: `validate_server_config()` now delegates to the active
plugin's `validate_config()`, preserving existing validation semantics.
- **Router compatibility**: All existing `require_*` decorators and `resolve_identity`
/ `get_request_context` dependencies remain unchanged. Routers import the same
symbols from `openviking.server.auth`.
- **Tests**: `conftest.py` manually wires the DevAuthPlugin in ASGI tests (lifespan
not triggered). `test_auth.py` expanded with plugin registration and validation tests.
- **Docs**: `04-authentication.md` (en/zh) updated with plugin registration examples.
Co-Authored-By: claude-sonnet-4-6 <noreply@anthropic.com>
* fix(tests): fix trusted mode test
* fix(tests): fix unit test
* fix(cli): remove unexisted transaction observer
* docs: update skills definition
* docs: update skills definition
* docs: update skills definition
* docs: update skills definition
* fix(skills): now we allow viking://agent/skills again, and optimize CLI for skills
* docs(skills): use -p instead of --parent in agent skills examples
Align the `ov skills add` examples in the context-types and viking-uri
docs with the short flag `-p` introduced for `ov skills list/find/show`,
so all four user-facing examples consistently demonstrate the short form
when targeting `viking://agent/skills`.
Co-Authored-By: claude-sonnet-4-6 <noreply@anthropic.com>
* fix(tests): error check for api key
* fix(tests): unit test wait until resource not busy
* fix(tests): unit test wait until resource not busy
* fix(sdk): args form in skills find
* fix(skills): pass target uri in request body
---------
Co-authored-by: claude-sonnet-4-6 <noreply@anthropic.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
* fix(parse): normalize text file encodings
* fix(parser): harden text encoding normalization
* fix(parse): normalize text encodings with charset-normalizer
* test(parse): use synthetic gb18030 fixture text
* fix(parse): respect detector rank for non-cjk text
* fix(parse): rescue short simplified chinese text
* fix(parse): preserve korean hanja text
* style(parse): format text encoding tests
The OpenViking docker image still launched the legacy `openviking/console`
standalone service on port 8020. Now that web-studio is bundled into the OV
server itself at /studio (see #2156), that process is redundant and the
port is just a confusing artefact.
This change retires the old console (python package + 8020 + console-frontend
favicons) but **keeps the in-compose Caddy as a stable single-ingress on
port 1934**, just simplified to one upstream now that there's no 8020. The
server-side BFF at `openviking/server/routers/console.py` (under
`/api/v1/console/*`) is also kept — web-studio uses the same endpoints.
**The OAuth authorize page (`openviking/server/oauth/router.py`) is
deliberately untouched in this PR** — the console-link button and Quick
authorize same-origin panel will be re-pointed at web-studio in a focused
follow-up.
BREAKING CHANGES:
- Port 8020 is gone from the docker image and docker-compose.yml; Caddy at
1934 now forwards everything to 1933 (web-studio lives at /studio there).
Anything bookmarked at `http://host:8020/...` must migrate to
`http://host:1933/studio/`.
- `python -m openviking.console.bootstrap` no longer exists; the python
package `openviking.console` has been removed.
Pip packaging:
- web-studio dist is now shipped inside the wheel under
`openviking/web_studio/dist/` (mirroring the old `openviking/console/static/`
layout). The dockerfile copies `--from=web-studio-builder /web-studio/dist`
into the source tree before `uv sync`, so the wheel produced by the
default docker build always carries the SPA. Building the wheel without
running `npm run build` first leaves the directory empty, which gracefully
degrades /studio to a 404 without breaking server startup.
- Favicon assets (`favicon.ico` / `favicon-32.png` / `apple-touch-icon.png`,
~11 KB total) are duplicated into `openviking/server/static/` and shipped
via package-data so `/favicon.*` and `/mcp/favicon.*` routes are always
registered, regardless of whether the web-studio dist is bundled.
- `pyproject.toml` and `setup.py` `package-data` drop `console/static/**`
and add `server/static/**` + `web_studio/dist/**`.
- New favicons (the 16/32/180 set in both `openviking/server/static/` and
`web-studio/public/`) are downscaled from the canonical
`web-studio/public/openviking-icon.png`, so the small-icon family matches
the SPA's high-res rel="icon" target — the studio tab icon now stays
consistent whether the browser uses the HTML link tag or falls back to
auto-fetching `/favicon.ico`.
Server:
- `openviking/server/app.py` now reads `/studio` from
`Path(__file__).parent.parent / 'web_studio' / 'dist'` by default;
`OPENVIKING_WEB_STUDIO_DIR` still wins for dev mode pointing at a
repo-local build. Favicon routes are unconditionally registered and
load from `openviking/server/static/`.
- `openviking/observability/usage_audit/projection.py` drops the legacy
`/console/*` skip prefix (the BFF prefix `/api/v1/console/*` remains).
Docker:
- `web-studio-builder` stage moved earlier (Stage 2) so its dist can flow
into `py-builder` before `uv sync` runs.
- Runtime stage no longer separately copies the dist or sets
`OPENVIKING_WEB_STUDIO_DIR`; the in-package path is the default.
- Entrypoint renamed `openviking-console-entrypoint.sh` -> `openviking-entrypoint.sh`
and stripped of the `python -m openviking.console.bootstrap` launch.
- `EXPOSE 1933 8020` -> `EXPOSE 1933`.
- `docker-compose.yml` drops the openviking service's 8020 port mapping;
the caddy service stays but no longer needs port 8020 exposed.
- `Caddyfile` simplified to a single `:1934 { reverse_proxy openviking:1933 }`
— the legacy `/console/*` route to :8020 is gone.
Docs:
- en/zh quickstart updated to drop the 8020 mapping and explain that the
API server now also serves `/studio`.
- Other guides (`12-public-access.md`, `11-oauth.md`, `05-observability.md`,
`04-setup-for-agent.md`, `03-deployment.md`) are intentionally left for a
focused follow-up PR alongside the OAuth quick-authorize reintroduction.
Tests:
- Deleted `tests/misc/test_console_{proxy,static_assets}.py` (covered the
removed console package). `tests/observability/test_console_router.py`
stays — it covers the BFF, which remains.
- Add CLI integration tests (10 test files under tests/cli/)
- Extend api_test.yml with CLI install + test steps
- Run filesystem + scenarios/resources_retrieval serially to avoid 409 conflicts
- Other tests parallel with -n 4
- Add release prereleased trigger to api_test.yml and api_test_effect.yml
- Deduplicate oc2ov_test P0 cases (20→12, ~30-55min saved):
- Delete test_memory_write.py (covered by V2 suite)
- Remove events/tools from V2 suite (structurally identical to entities/skills)
- Remove test_memory_read_verify (covered by V2 suite)
- Remove test_cross_session_recall (overlaps with recall_explicit_search)
- Add ensure_resources_dir fixture to prevent NOT_FOUND on fresh environments
- Add retry logic for 429/500/403 rate-limit in api_client.py
- Add retry for commit when task_id is None in test_memory_v2_full_suite.py
- Add exponential backoff retry for GitHub platform test 5xx errors
Close GHSA-mf9v-mfxr-j63j (decompression-bomb safeguards bypass, HIGH)
and GHSA-qccp-gfcp-xxvc (sensitive headers forwarded across origins in
proxied redirects, HIGH). Both advisories published 2026-05-11, both
first_patched_version=2.7.0.
Close 5 GHSA advisories (2 CRITICAL, 3 HIGH) by raising the lower bound
from >=1.0.0 to >=1.83.7. The existing ceiling <1.83.13 is unchanged.
- GHSA-r75f-5x8p-qvmc (CRITICAL): SQL Injection in Proxy API key verification
- GHSA-jjhc-v7c2-5hh6 (CRITICAL): OIDC userinfo cache key collision auth bypass
- GHSA-v4p8-mg3p-g94g (HIGH): Authenticated RCE via MCP stdio test endpoints
- GHSA-xqmj-j6mv-4862 (HIGH): SSTI in /prompts/test endpoint
- GHSA-69x8-hrgq-fjj8 (HIGH): Password hash exposure / pass-the-hash bypass
All 5 advisories are patched in litellm 1.83.7 (or earlier 1.83.0,
covered by the new floor).
Co-authored-by: MaojiaSheng <shengmaojia@bytedance.com>
* feat(server): add native MCP endpoint at /mcp
Serve 5 MCP tools (search, read, store, forget, health) directly from
the OV FastAPI server via streamable HTTP transport. This eliminates the
need for the Node.js MCP subprocess — the plugin's .mcp.json now points
to the server URL instead of spawning a process.
Identity headers (X-OpenViking-Account/User/Agent) are propagated to
service-layer calls via contextvars ASGI middleware.
* fix(mcp): disable DNS rebinding protection for reverse proxy compatibility
MCP SDK auto-enables host validation for localhost, rejecting requests
with external Host headers (e.g. from Cloudflare/Nginx reverse proxy).
* fix(mcp): reuse auth.resolve_identity for MCP endpoint authentication
MCP endpoint previously had no authentication — requests fell through
with default/default identity. Now delegates to the same resolve_identity
used by all REST routes, so auth_mode, API key validation, and identity
resolution are handled identically.
* fix(mcp): fix import path for TextPart in store tool
openviking.session.parts does not exist; the correct module is
openviking.message.part.
* fix(mcp): store tool now creates a new session and commits immediately
Each store call creates a unique session, adds the message, and commits
right away so memories are extracted and searchable without waiting for
a token threshold.
* chore: add mcp>=1.27.0 dependency for native MCP endpoint
* fix(mcp): align search/forget tools with REST API, fix forget crash
- Remove SEARCH_TARGETS and per-scope loop; use single
service.search.find(target_uri="") call matching REST API behavior
- Fix forget crash: FSService has no delete(), use rm() instead
- Replace fragile _is_memory_uri() substring check with ContextType
- search tool: replace scope param with target_uri for direct passthrough
- Work directly with FindResult/MatchedContext objects instead of
dict-munging via to_dict()
* fix(mcp): fail-closed on missing identity, remove unused Role import
- _get_ctx() now raises UnauthenticatedError instead of defaulting to
ROOT when identity contextvar is not set
- Remove unused Role import
- Clean up comments in create_mcp_app
* test(mcp): add unit tests for MCP endpoint tools
17 tests covering all 5 MCP tools and identity propagation:
- _get_ctx: returns context when set, raises UnauthenticatedError when not
- health: healthy/unhealthy responses
- search: no results, with resource, with target_uri
- read: nonexistent URI, directory listing, batch reads
- store: user and assistant roles
- forget: input validation, non-memory guard, URI deletion, query fallback
- Route registration: /mcp route exists in app
* docs(mcp): update integration guide with verified platforms and correct tools
- Add verified platforms table (Claude Code, ChatGPT/Codex, Claude.ai,
Manus, Trae)
- Document authentication (X-Api-Key / Bearer token)
- Add Claude.ai OAuth proxy (MCP-Key2OAuth) instructions
- Update tool table to match actual implementation (search, read, store,
forget, health) — remove stale tool names
- Reorganize client config: generic first, then platform-specific
* feat(mcp): expand to 7 tools aligned with vikingbot, split read/list
Align MCP tool surface with vikingbot/agent/tools/ov_file.py:
- Split read/list: read is file-only with semaphore(10) concurrency;
list is directory-only with recursive support
- store: accept batch messages[] (was single text), matching
VikingMemoryCommitTool
- search: add min_score parameter (default 0.35), matching
VikingSearchTool
- add_resource: new tool for adding files/URLs to resources
- Use @mcp.tool(name="list") to avoid shadowing Python builtin
7 tools: search, read, list, store, add_resource, forget, health
* feat(mcp): add grep and glob tools, update docs to 9 tools
Add grep (multi-pattern regex search) and glob (file pattern matching)
MCP tools to align with VikingBot's full tool surface. Update EN/ZH
integration docs to reflect all 9 tools with correct parameters.
* fix(mcp): store schema, forget safety, remove memories-only restriction
- store: use Pydantic StoreMessage model so MCP schema includes
required role/content field definitions (was bare dict[str, str])
- forget: remove query parameter entirely — deletion requires exact URI,
use search tool first to find candidates
- forget: remove /memories/ path restriction, allow deleting any URI
* docs(mcp): update forget tool description — exact URI only, no query
* fix(mcp): rename list_dir to ls, add forget safeguard, use Bearer in docs
- Rename list_dir → ls (MCP tool name stays "list") to avoid confusion
with "only lists directories"
- Add safeguard to forget tool description: irreversible, requires user
confirmation
- Docs: use Authorization: Bearer in all examples (standard, consistent
with OAuth proxy flow)
- Fix ruff format on mcp_endpoint.py and test_mcp_endpoint.py
Port MCP (Model Context Protocol) client support from HKUDS/nanobot v0.1.5 so
vikingbot can connect to third-party MCP servers (filesystem, GitHub, browsers,
databases, etc.) and expose their tools to the agent alongside native tools.
Implementation is essentially verbatim nanobot v0.1.5 with two small adaptations:
- Import paths rewritten from nanobot.* to vikingbot.*
- MCPToolWrapper.execute signature extended with tool_context: ToolContext as
the required first positional arg to match vikingbot's Tool.execute contract
(the context is unused since MCP servers receive inputs via kwargs only, but
the parameter is required for registry dispatch)
Credits to upstream nanobot:
- @SergioSV96 — HKUDS/nanobot#554 (initial MCP support)
- @Qinnnnnn — HKUDS/nanobot#1488 (SSE + streamableHttp transports with
auto-detection)
Supports three transports: stdio / sse / streamableHttp. The type field is
inferred from config when omitted (command implies stdio; url ending in /sse
implies SSE, otherwise streamableHttp).
Behavior is unchanged when mcp_servers is unset or empty — _connect_mcp
short-circuits and the agent loop runs exactly as before.
* docs: fix docker deployment
* reorg: remove third_party/agfs
* feat(s3fs): add disable_batch_delete option for OSS compatibility
Port of PR #1333 from Go version to Rust:
- Add disable_batch_delete config option to S3Client
- When enabled, use sequential single-object deletes instead of DeleteObjects
- This is for S3-compatible services like Alibaba Cloud OSS that require
Content-MD5 for DeleteObjects but AWS SDK v2 does not send it by default
- Add documentation and config example for OSS
* fix(s3fs): pass disable_batch_delete config from Python to Rust
Add disable_batch_delete to the s3_plugin_config dict in _generate_plugin_config
so that the Python config can properly control the Rust S3FS plugin's behavior.
* reorg: remove third_party/agfs
* reorg: remove third_party/agfs
* change some docs
* change some docs
---------
Co-authored-by: openviking <openviking@example.com>
* reorg: rewrite agfs with rust, and named with ragfs, keep License
* reorg: rewrite agfs with rust, and named with ragfs, keep License
* reorg: rewrite agfs with rust, and named with ragfs, keep License
* reorg: rewrite agfs with rust, and named with ragfs, keep License
* reorg: rewrite agfs with rust, and named with ragfs, keep License
* reorg: rewrite agfs with rust, and named with ragfs, keep License
* reorg: rewrite agfs with rust, and named with ragfs, keep License
* reorg: rewrite agfs with rust, and named with ragfs, keep License
* fix: grep level limit
* fix: grep root
* fix: import error
* fix: rust code optimazation
* fix: CI error
* fix: CI go mod cache
* fix: grep level limit
* fix: CI
---------
Co-authored-by: openviking <openviking@example.com>
* fix: add models observer info for embedder and rerank
* fix: make build deps
* fix: ov observer
* fix: ov observer
---------
Co-authored-by: openviking <openviking@example.com>
* Add RAGbenchmark: RAG system evaluation framework
* Update README.md
* Update README.md
* Update README.md
* Code structure refactoring
* feat: improve RAG benchmark with dataset sampling and configuration updates
- Add complete dataset sampling scripts with document-level sampling
- Implement filtering logic consistent with adapters (exclude category 5 for Locomo, no answer for SyllabusQA, unanswerable for Qasper)
- Update configuration from raw_data/dataset_dir to dataset_path for clarity
- Enhance adapters with improved path handling and data loading
- Add gitignore for data and output directories
- Add dependencies (datasets, pandas, tavily-python)
- Add test files and documentation
* feat: add stratified sampling support to all datasets
- Implement stratified sampling for Locomo (by category 1-4)
- Implement stratified sampling for SyllabusQA (by question_type)
- Implement stratified sampling for Qasper (by answer type: extractive/free_form/yes_no)
- Implement stratified sampling for FinanceBench (by question_type)
- Add proper handling when sample size cannot be evenly split:
- Display warning message
- Distribute remaining QAs to first N categories
- Fall back to random sampling if sample size too small
- Update prepare_dataset.py to support both 'random' and 'stratified' modes
- Set default sampling mode to 'random'
* Update locomo adapter to support image attachments and other improvements
* Update dataset documentation with actual document counts
* Add benchmark results reference and reproduction steps
* Improve sampling scripts for benchmark reproducibility
* Refactor sample_dataset.py: extract common sampling logic
- Fix two bugs:
1. num_docs + sample_size + random path: use int indices instead of dict tuples
2. pure stratified path: use len() for list length calculation
- Extract common sampling utilities:
- calculate_category_targets()
- stratified_sample_with_reallocation()
- random_sample_qas()
- sample_docs_stratified()
- sample_docs_random()
- Reduce code duplication by ~60-70%
- Improve maintainability and readability
- Keep full backward compatibility
* Update config.yaml: improve configuration structure
- Add FinanceBench to supported datasets list
- Change to template configuration format
- Add execution: section for better organization
* Fix bug: duplicate worker_end() call in generation failure path
- Remove duplicate monitor.worker_end(success=False) call in run_generation()
- The _process_generation_task() already calls worker_end() in its exception handler
- This prevents double-counting of failed tasks and distorted statistics
* Fix bug: _get_required_syllabi() doesn't support JSON input
- Add JSON file support to _get_required_syllabi()
- Extract syllabus names from JSON keys (same format as _load_from_json())
- This ensures data_prepare() processes correct docx files when using JSON input
* Improve exception re-raising: use bare raise to preserve traceback
- Replace 'raise e' with bare 'raise' to preserve original traceback
- Also remove unused 'e' variable since we don't need it
- This makes debugging easier by showing where the exception actually occurred
* Fix bug: Locomo prompt uses raw gold_answer instead of gold_answer_str
- In Locomo prompt, use gold_answer_str instead of gold_answer
- This ensures consistent formatting when gold_answer is a list
- Both Locomo and Generic prompts now use the same ' | ' separated format
* Improve directory ingest: use os.path.commonpath() for robustness
- Replace manual common ancestor calculation with os.path.commonpath()
- os.path.commonpath() handles all OS path separators correctly
- Add try-except to handle ValueError when no common path exists
- More robust than manual split(os.sep) approach
* benchmark: honor skip_ingestion and fail on LLM retry exhaustion
Jiahui Zhou