* feat(pdf): refactor MinerU parsing to the official file_parse API
* feat(pdf): remove mineru_api_key from configuration and examples
* feat(pdf): preflight MinerU /health during service initialization
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(vikingdb): normalize all date_time range filters in API key client
OpenViking compiles TimeRange down to the internal `range` DSL, but the
commercial VikingDB data plane (Bearer API-key auth) expects `time_range`
for date_time fields and `range` only for numeric fields. The API-key
client does not run the local engine's filter conversion, so `range`
nodes on date_time fields were sent verbatim and mis-handled.
Normalize `range` -> `time_range` for every schema date_time field by
reusing the canonical VALID_TIME_FIELDS constant, covering both
`created_at` and `updated_at` instead of hardcoding a single field name.
Numeric `range` nodes and nested boolean filter structure are preserved,
and filters already emitted as `time_range` pass through unchanged. Only
the request body `filter` is rewritten; upsert/update data is untouched.
Add regression tests covering the converted created_at/updated_at date
filters, an unchanged numeric filter, and time_range idempotency.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(vikingdb): normalize date_time filters in AK/SK client
The API-key client already rewrites `range` filter nodes on date_time
fields to VikingDB's `time_range` operator, but the AK/SK-signed
`VolcengineCollection` shares the same commercial data-plane endpoints
and had the identical latent bug: `TimeRange` expressions compile down
to the internal `range` DSL, which the commercial API only accepts for
numeric fields.
Mirror the API-key fix in `VolcengineCollection._data_post` so both
auth modes normalize `range` -> `time_range` for `created_at`/`updated_at`
while leaving numeric `range` nodes untouched. Add AK/SK coverage for
both date_time fields and for idempotency of already-`time_range` input.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
---------
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* feat(reindex): support tag updates
Add replace and append tag modes to reindex vector rebuilds, preserve omission-aware behavior, and propagate options through background tasks and namespace rebuilds. Align Python, TypeScript, Go, and CLI interfaces with tests and documentation.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(reindex): lock file targets exactly
Use an exact path lock for existing file targets while retaining tree locks for directories and prune-orphans scopes. Add a regression test for single-file reindex.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(reindex): handle prune file targets
Use exact locks for existing file targets in prune-orphans mode while retaining tree scope for missing targets. Document the existing Go ReindexOptions wait semantics and add lock regression coverage.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
---------
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(embedding): downsample oversized image inputs
Keep imported image resources unchanged while avoiding provider-side multimodal embedding failures for oversized images. The embedding path now builds a temporary downsampled image data URI when image bytes exceed the shared large-image limits.
Move reusable image size thresholds into media_limits so both parser-side large image handling and embedding-side input preparation depend on a common utility instead of embedding_utils importing parser internals.
Add vectorize_file coverage confirming large image embedding inputs are resized and the stored resource bytes are preserved.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(media): downsample oversized image model inputs
Keep imported image resources unchanged while avoiding provider-side multimodal failures for oversized images. Shared image input preparation now builds temporary downsampled bytes for model requests when image bytes exceed the configured large-image limits.
Apply the model-input downsampling to both semantic image summary generation and embedding image data URI construction, so directory and code repository imports can preserve original images while sending provider-compatible inputs.
Move reusable image size thresholds into media_limits so parser-side large image handling, VLM summary generation, and embedding preparation share common limits without embedding_utils importing parser internals.
Always convert downsampled model images to RGB before JPEG encoding so Pillow-openable modes such as LA or I;16 do not fall back to the original oversized bytes.
Add coverage confirming VLM image summaries, vectorize_file embedding inputs, and JPEG-incompatible image modes are resized while stored resource bytes are preserved.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
---------
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(storage): keep non-memory appends free of memory trailers
ContentWriteCoordinator._write_in_place routed every append through
MemoryFileUtils, which strips the existing trailing newline and appends
a reserved MEMORY_FIELDS metadata trailer, even for resource/skill files
where MEMORY_FIELDS is not a reserved format (see content_visibility).
Append to non-memory files now concatenates raw content instead, matching
POSIX append semantics and the documented visibility rules.
* feat(mcp): add write tool with exact-string edit support
Agents could not use viking:// as a working directory through MCP: no
tool could create or update file content. Add a write tool covering full
writes (mode=replace as create-or-overwrite, append, strict create) and
targeted edits (a list of {old_string, new_string, replace_all}
exact-string replacements applied in order, all-or-nothing), following
the Write/Edit conventions of common agent harnesses.
Edits read via read_visible and write back through the content-write
coordinator, so memory metadata trailers are preserved and semantic /
vector re-indexing triggers as with any other write. Parent directories
are created automatically by the storage layer. Descriptions spell out
writable scopes (resources, user memories/resources, agent) and the
wait=true knob for read-after-write search consistency.
Also update the stale tool-count comment in app.py and the MCP tool
tables in the en/zh guides (13 -> 14 tools).
* feat(mcp): add tree tool, split targeted edits into edit tool
tree renders the recursive directory tree under a viking:// URI,
indented by depth with file sizes, for whole-layout orientation;
level_limit/node_limit bound the output and include_abstract adds
per-file summaries. Missing directories report "(nothing under ...)"
instead of an error, matching the read tool's convention.
edit(uri, old_string, new_string, replace_all) takes over the targeted
exact-string replacement that previously lived in write's edits array,
matching the classic Edit tool signature harnesses already train on.
write now only does full-content writes (content + mode), removing the
mutually-exclusive content/edits schema ambiguity. Edits still read via
read_visible and write back through the content-write coordinator, so
memory metadata trailers are preserved and re-indexing triggers as with
any other write.
* test(plugin): update canonical MCP tool list for tree/write/edit
The marketplace test pins the server-registered MCP tool list; add the
new tree, write, and edit tools to fix plugin-tests CI.
* feat(storage): support plain files at the user scope root
Agents treating viking:// as a working directory naturally drop files
like viking://user/zeus-persona.md at the user root, but the write
coordinator only accepted the memories/ and resources/ subtrees.
Two changes make that work:
- Namespace shorthand: a dotted first segment under viking://user/ is a
file name, not a user id (canonical user ids are dot-free by
convention), so viking://user/zeus-persona.md now canonicalizes to
viking://user/<current-user>/zeus-persona.md, matching how the
reserved memories/resources/skills segments already shorthand.
Dot-free segments still address an explicit user, and an exact match
with the current user id still wins.
- Coordinator: plain files directly under the user root (or in
non-managed subdirectories) anchor their semantic refresh at the
parent directory. The managed subtrees skills/, peers/, privacy/ and
sessions/ remain read-only with an actionable error message.
* fix(namespace): narrow user-root shorthand to text-file extensions
Review on #3936 (codex /review-pr) flagged that treating any dotted
segment as a user-root file shorthand would silently re-route canonical
URIs for valid dotted user ids (e.g. alice.smith) into the current
user space. Shorthand now triggers only when the first segment ends
in a common text-file extension; dotted or email-style user ids keep
resolving as canonical user ids. Adds regression tests pinning both
behaviors.
* fix(mcp): resolve user URIs against current user
* test(mcp): pin plain-file writes directly at the user root
The user-root shorthand exists so an agent can drop viking://user/persona.md
into its workspace, but every new test went through an intermediate directory
(viking://user/project/zeus-persona.md), leaving the no-directory shape — the
one that anchors the write coordinator's refresh at the user root itself —
uncovered. Add the missing case.
Also correct the write tool docstring: the create-extension allowlist applies
to any newly created file, including one created by mode="replace" falling
back to create, not only to an explicit mode="create".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Frontmatter was parsed into ParseResult.meta and removed from the body, but
that metadata is never persisted, so every ingested markdown file lost its
frontmatter with no way to read the fields back.
Parse frontmatter into meta unconditionally (it still drives doc_title) and
only remove it from the stored body when explicitly configured; that removal
is now off by default.
* fix(watch): avoid global lock during resource moves
Allow unrelated watch operations to continue while viking_fs.mv is running, while serializing overlapping resource paths with a move fence. Preserve transaction integrity across persistence failures, rollback, and caller cancellation.
* feat(resource): add URI mutation coordinator
* refactor(watch): separate target rewrites from resource moves
* refactor(fs): own resource move watch transaction
* feat(watch): coordinate refreshes with URI mutations
* refactor(core): share URI mutation coordinator
* test(watch): focus URI move coverage
* test(watch): reuse existing URI move coverage
---------
Co-authored-by: chenpengfei <chenpengfei@bytedance.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
Keep expired terminal tasks cached when persistent deletion fails so cleanup can retry without resurrecting stale records after restart.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Materialize session-aware context requests through SessionService before
loading the recall ledger, while retaining the messages.jsonl guard for
partially initialized sessions.
Add coverage for first-turn ledger writes, same-session message capture,
materialization failure recovery, and stateless requests with session
features disabled.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* feat(memory): support event tag filtering
Add session-level default event tags, commit-time overrides, durable queue propagation, and first-write vector index tagging. Include config update APIs and coverage for serialization, concurrency, extraction, and HTTP behavior.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* feat(memory): expose event tags in SDKs and CLI
Add session default tag configuration, config updates, and commit-time event tag overrides across embedded Python, standalone Python, TypeScript, Go, and the Rust CLI. Preserve explicit empty-tag semantics and document each public interface.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(sdk): align legacy session tag APIs
Forward commit-time event tags through the legacy Python HTTP shims and align BaseClient session signatures without adding a new abstract-method requirement for existing subclasses.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* feat(session): allow updating auto-commit policy
Extend PATCH session config to atomically update event tags and auto-commit settings. Merge policy objects by field, use explicit null to disable automatic commits, preserve omitted fields, and expose the contract across SDKs and CLI.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* fix(session): align session config interfaces
Replace the generic session create config JSON flag with explicit event-tag and auto-commit options. Preserve omitted, object, and null auto-commit semantics across HTTP, embedded clients, SDKs, and CLI, reject ambiguous null policy fields, and handle nullable event configuration consistently.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
* test(session): trim redundant event tag tests
---------
Co-authored-by: TRAE CLI <noreply@bytedance.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
Read replicas load the API key store once at startup and never rewrite
it, so a user registered/rotated/removed on the writer stays invisible
(new key -> "Invalid API Key"; removed key -> still accepted).
Add an optional background watcher that polls the shared key store and
reloads the in-memory index only when it actually changes:
- APIKeyManager.reload(): strictly read-only refresh that rebuilds state
and swaps it in atomically, never writing or migrating plaintext keys.
- compute_store_signature(): cheap (path, size, modTime) signature over
accounts.json + every users.json so the watcher skips unchanged polls.
- ApiKeyAuthPlugin starts/stops the watcher behind api_key_watch_enabled
(default off) with api_key_watch_interval_seconds; AuthPlugin.shutdown()
is wired into app shutdown to cancel it cleanly.
Add coverage for reload convergence, read-only/no-migrate guarantees,
uninitialized-store tolerance, signature change detection, and watcher
reload/skip/shutdown behavior.
Co-authored-by: TRAE CLI <noreply@bytedance.com>
_smart_split_content documents that it enforces both a token limit
(max_size) and a hard character limit. But when a single paragraph was
oversized by tokens yet under the character limit, the force-split loop
stepped through it by max_chars only, so it emitted a chunk that still
exceeded max_size tokens.
This is reachable with ordinary long-form CJK text: _estimate_token_count
weights CJK at ~0.7 token/char, so a ~5000-char Chinese paragraph is
~3500 tokens (over the 2048 default) while staying under the char limit,
and was returned as a single over-budget chunk.
Bound the force-split step by min(max_chars, max_size / MAX_TOKENS_PER_CHAR),
where MAX_TOKENS_PER_CHAR is the worst-case (CJK) density already used by
_estimate_token_count, now extracted into a shared constant so the two stay
in sync.
Add regression tests for the token budget and content preservation.
fix_fields_data backfills schema fields absent from a row's data with their
defaults, but it first short-circuited on
`len(field_data_dict) >= len(field_meta_dict)`, using field count as a proxy
for "all schema fields are present".
That proxy is wrong: a row can have as many keys as the schema (or more) while
still missing a specific field — e.g. a row written before a new field was
added that also carries an extra internal/non-schema key. In that case the
fill loop was skipped and the missing field was silently omitted rather than
defaulted, so downstream reads see an incomplete record.
Remove the count guard. The loop already skips fields that are present, so
complete inputs are returned unchanged; only genuinely missing fields are now
filled.
Add regression tests: a missing field with matching key count is backfilled
(default value and type default), and complete data is returned unchanged.
_gh_slug claims to produce "GitHub-style" heading slugs but collapsed runs
of whitespace (`re.sub(r"\s+", "-", s)`). GitHub's reference slugger
(github-slugger) maps each space to its own hyphen (`.replace(/ /g, '-')`)
and does not collapse.
Because punctuation is stripped before spaces are converted, a heading like
"Foo & Bar" leaves two adjacent spaces where "&" was. GitHub renders this as
"foo--bar", but _gh_slug produced "foo-bar". The intra-document link rewriter
(_rewrite_link) compares _gh_slug(heading) against the link fragment, so an
author-written link such as `guide.md#foo--bar` failed to match its heading
and was left unrewritten after the target doc was split into sections.
Replace `\s+` with `\s` so each whitespace character maps to one hyphen,
matching GitHub. Simple single-space headings are unaffected.
Add regression tests covering punctuation headings and ordinary headings.
The range checks used strict inequalities (< instead of <=), rejecting
the valid boundary values defined by the WGS-84 geographic standard:
- latitude ±90 (the North and South Poles)
- longitude ±180 (the antimeridian / International Date Line)
Any resource whose geo_point is exactly on a pole or the date line cannot
be indexed, and a geo_range query centred on those boundaries raises
ValueError instead of executing the search.
Change both checks to <=. Values strictly outside the valid interval
(e.g. 181, -91) continue to raise ValueError as before.
Add a regression test covering all four boundary endpoints and all four
out-of-range coordinates.
The except block in VikingFS.mkdir() had no re-raise, so any backend
failure that was not an already-exists error (permission denied, quota
exceeded, I/O errors, lock-lease violations) — and even already-exists
errors with exist_ok=False — was silently discarded and mkdir() returned
as if the directory had been created. Callers on the write hot path
(ovpack import, parsers, session, privacy) then write into a directory
that may not exist, and the original actionable error is lost.
Re-raise the original exception unless it is an already-exists error
tolerated by exist_ok=True.
Also update tests/misc/test_mkdir.py, which still mocked fs.agfs.mkdir
even though mkdir() now goes through the AsyncAGFSClient wrapper
(self._async_agfs) — the swallowed-attribute-error made the stale tests
pass/fail for the wrong reasons. Add regression tests covering error
propagation for both exist_ok values.
_copy_dir_through_vikingfs() drives the copy phase of mv() for non-temp
directories, but enumerated the source with the agent-facing ls() default
node_limit=1000. Any directory level with more than 1000 visible entries
was copied only partially, and mv() then unconditionally deleted the
source recursively — permanently losing every entry past the cap, while
the vector index (remapped via the uncapped _collect_uris) kept pointing
at URIs that no longer exist anywhere.
Pass the module's LS_ALL_NODES sentinel, which exists precisely for
internal callers that must enumerate an entire directory.
Add a regression test that fails without the fix.
The row serializer writes each string's byte length as a UINT16 prefix. The
scalar `string` field guards this contract — a >65535-byte value raises a
clean, field-attributed ValueError. The `list<string>` element path uses the
identical UINT16 prefix but had no such guard, so an oversized element instead
raised a raw `struct.error: 'H' format requires 0 <= number <= 65535` from
deep inside struct.pack_into — a different exception type, naming no field.
Callers that catch ValueError (matching the documented scalar contract) do not
catch this, and the error gives no clue which field/element overflowed. Apply
the same bounds check to list<string> elements so inclusion in a list does not
silently downgrade the type-checked contract the scalar path upholds.
Add a regression test asserting the clean ValueError for an oversized element
and that an in-bounds (incl. multibyte) list still round-trips.
Several routers returned Response(status='error', error=ErrorInfo(...))
directly, so FastAPI shipped them with HTTP 200 instead of the mapped
status from ERROR_CODE_TO_HTTP_STATUS. Web-studio's sessions API only
survived because of a code-based fallback; other clients (e.g. a generic
HTTP retry layer) would treat these as success and never surface the
error.
Switch the seven return sites to error_response() so the canonical
mapping drives the HTTP status, and add the three previously-unmapped
codes (INTERNAL_ERROR, NO_VECTOR_DB, INVALID_FILTER) to the map.
Residual follow-up from PR #1764 (ac9f679a).
Co-authored-by: ming <silverchris@foxmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
