* fix(server): stop exporting raw query strings and buffering zip responses in observability
Sweep findings: B-03, B-13. Prevent query secrets from reaching traces and keep ZIP responses streaming.
(cherry picked from commit d8ac3dc33b)
* fix(session): tolerate missing/corrupt archive in Phase-2 replay
(NotFoundError / _ArchiveMessagesCorruptError) on a missing or corrupt
archive messages.jsonl instead of returning []. That PR added skip-on-
missing tolerance to the read path (_get_uncovered_archive_messages) and to
resume_queued_commit, but not to the Phase-2 commit replay path
(_prepare_phase2_archive_messages), which calls _read_archive_messages
unguarded while rolling earlier failed archives into the current commit.
Consequence: a terminally-failed earlier archive whose messages.jsonl is
missing/corrupt (legacy "no messages" terminal data, or produced by #3417's
own archive_read terminal path) makes every subsequent commit's Phase-2
extraction raise -> caught by _run_memory_extraction's except -> the current
archive is terminal-failed too. Because the poisoned archive is only removed
from replay once "covered" (which requires a later archive to complete), and
no later archive can ever complete, the session's memory extraction is
permanently poisoned. Raw messages are safe, but extraction is stuck.
Fix: wrap the replay-loop _read_archive_messages call in the same tolerance
_get_uncovered_archive_messages already uses -- skip + warn on not-found
(_is_storage_not_found) and on _ArchiveMessagesCorruptError, re-raise real
storage failures. The skipped archive stays in covered_failed so the current
archive's .done marks it covered, clearing the poison permanently.
Adds a regression test asserting the replay skips a failed archive with a
missing messages.jsonl (and marks it covered) instead of raising, and that a
real storage failure still propagates.
Follow-up to #3417.
(cherry picked from commit 5b8ec9e68a)
* fix(client): align client surfaces without leaking memory metadata
Reconstructs the client-parity work from upstream PR #3439 on current main and strips reserved memory metadata before line slicing in both embedded and HTTP reads.
Based-on: 48b411d58c
Co-authored-by: zhiheng.liu <zhiheng.liu@bytedance.com>
* fix(index): propagate semantic vectorization failures safely
Reconstructs upstream PR #3437 on current main, carries enqueue failures through SemanticDagExecutor, and drains the attempt's embedding tracker before retry-visible failure propagation.
Based-on: 02387deb09
Co-authored-by: zhiheng.liu <zhiheng.liu@bytedance.com>
* fix(core): close privacy and embedding failure gaps
* fix(memory): strip repeated metadata trailers
* fix(core): close public memory visibility gaps
* ci: skip embedding-dependent resource test without secrets
---------
Co-authored-by: zhiheng.liu <zhiheng.liu@bytedance.com>
* fix(observability): persist usage/audit in UTC and bucket per request tz
Dashboard, token-trend, and context-commit-heatmap previously bucketed by
server-process local timezone (`server/config.py` default `local`, which
on Railway / Docker without `TZ` is UTC). UTC+8 viewers saw "today" and
4h heatmap edges shifted relative to their wall clock.
Root cause: `projection.py` applied `astimezone(self._tz)` before writing
the rollup PKs, so date/hour columns were already locked to the
container's tz; a read-side `?timezone=` could not recover the buckets
because the raw UTC instants were no longer in storage.
Fix: persist all time-keyed columns (`date_utc`, `hour_utc`,
`created_at`) in UTC and accept `?timezone=` per request on
`/api/v1/console/{dashboard/summary,tokens,context-commits}`. The viewer
tz is resolved with `zoneinfo`, then the SQLite reads pull the spanning
UTC window and rebucket in Python to user-local days / 4h buckets. DST
is handled by `ZoneInfo` automatically.
Schema bump (v1 -> v2): `usage_token_daily` becomes `usage_token_hourly`
and `usage_retrieval_daily` becomes `usage_retrieval_hourly` (extra
`hour_utc` column in PK so cross-tz "today" slicing is precise). Other
tables keep their shape; `date` is renamed to `date_utc` to make the
semantic shift unambiguous. On boot, the store DROPs the legacy tables
once and recreates the new layout - acceptable because retention is 14
days and the product is pre-GA.
Frontend (`web-studio`) now passes
`Intl.DateTimeFormat().resolvedOptions().timeZone` on every console
fetch; `getLastDaysRange` derives the date range in the viewer's tz so
its boundaries match the backend interpretation. No visual / component
changes were needed.
Tests:
- Backend: existing 15 cases ported to the new keyword-only `tz=` API;
three new cases exercise UTC+8 day-boundary, America/New_York day
rebucketing across UTC midnight, and Asia/Shanghai 4h heatmap hour
shift. All 20 observability tests pass.
- Frontend: typecheck + lint clean (no new errors).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(observability): reset legacy usage audit sqlite schema
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
The OpenViking docker image still launched the legacy `openviking/console`
standalone service on port 8020. Now that web-studio is bundled into the OV
server itself at /studio (see #2156), that process is redundant and the
port is just a confusing artefact.
This change retires the old console (python package + 8020 + console-frontend
favicons) but **keeps the in-compose Caddy as a stable single-ingress on
port 1934**, just simplified to one upstream now that there's no 8020. The
server-side BFF at `openviking/server/routers/console.py` (under
`/api/v1/console/*`) is also kept — web-studio uses the same endpoints.
**The OAuth authorize page (`openviking/server/oauth/router.py`) is
deliberately untouched in this PR** — the console-link button and Quick
authorize same-origin panel will be re-pointed at web-studio in a focused
follow-up.
BREAKING CHANGES:
- Port 8020 is gone from the docker image and docker-compose.yml; Caddy at
1934 now forwards everything to 1933 (web-studio lives at /studio there).
Anything bookmarked at `http://host:8020/...` must migrate to
`http://host:1933/studio/`.
- `python -m openviking.console.bootstrap` no longer exists; the python
package `openviking.console` has been removed.
Pip packaging:
- web-studio dist is now shipped inside the wheel under
`openviking/web_studio/dist/` (mirroring the old `openviking/console/static/`
layout). The dockerfile copies `--from=web-studio-builder /web-studio/dist`
into the source tree before `uv sync`, so the wheel produced by the
default docker build always carries the SPA. Building the wheel without
running `npm run build` first leaves the directory empty, which gracefully
degrades /studio to a 404 without breaking server startup.
- Favicon assets (`favicon.ico` / `favicon-32.png` / `apple-touch-icon.png`,
~11 KB total) are duplicated into `openviking/server/static/` and shipped
via package-data so `/favicon.*` and `/mcp/favicon.*` routes are always
registered, regardless of whether the web-studio dist is bundled.
- `pyproject.toml` and `setup.py` `package-data` drop `console/static/**`
and add `server/static/**` + `web_studio/dist/**`.
- New favicons (the 16/32/180 set in both `openviking/server/static/` and
`web-studio/public/`) are downscaled from the canonical
`web-studio/public/openviking-icon.png`, so the small-icon family matches
the SPA's high-res rel="icon" target — the studio tab icon now stays
consistent whether the browser uses the HTML link tag or falls back to
auto-fetching `/favicon.ico`.
Server:
- `openviking/server/app.py` now reads `/studio` from
`Path(__file__).parent.parent / 'web_studio' / 'dist'` by default;
`OPENVIKING_WEB_STUDIO_DIR` still wins for dev mode pointing at a
repo-local build. Favicon routes are unconditionally registered and
load from `openviking/server/static/`.
- `openviking/observability/usage_audit/projection.py` drops the legacy
`/console/*` skip prefix (the BFF prefix `/api/v1/console/*` remains).
Docker:
- `web-studio-builder` stage moved earlier (Stage 2) so its dist can flow
into `py-builder` before `uv sync` runs.
- Runtime stage no longer separately copies the dist or sets
`OPENVIKING_WEB_STUDIO_DIR`; the in-package path is the default.
- Entrypoint renamed `openviking-console-entrypoint.sh` -> `openviking-entrypoint.sh`
and stripped of the `python -m openviking.console.bootstrap` launch.
- `EXPOSE 1933 8020` -> `EXPOSE 1933`.
- `docker-compose.yml` drops the openviking service's 8020 port mapping;
the caddy service stays but no longer needs port 8020 exposed.
- `Caddyfile` simplified to a single `:1934 { reverse_proxy openviking:1933 }`
— the legacy `/console/*` route to :8020 is gone.
Docs:
- en/zh quickstart updated to drop the 8020 mapping and explain that the
API server now also serves `/studio`.
- Other guides (`12-public-access.md`, `11-oauth.md`, `05-observability.md`,
`04-setup-for-agent.md`, `03-deployment.md`) are intentionally left for a
focused follow-up PR alongside the OAuth quick-authorize reintroduction.
Tests:
- Deleted `tests/misc/test_console_{proxy,static_assets}.py` (covered the
removed console package). `tests/observability/test_console_router.py`
stays — it covers the BFF, which remains.
Add an opt-in middleware that attaches the request and response bodies
(truncated, content-type filtered) onto the active OpenTelemetry root span,
and surface the URL query string as `url.query`. Off by default — bodies
may contain secrets and high-cardinality content; enable via
`server.observability.dump_body.enabled` and bound payload size with
`max_bytes`.
The dump middleware is registered before the HTTP observability middleware
so it nests inside the trace span (Starlette executes later-registered
middleware first). Streaming, multipart, and binary content types are
skipped, and any capture failure is swallowed so the request path is never
affected.
Co-authored-by: chenpengfei <chenpengfei@bytedance.com>
Hao Zhe