Qin Haojie
ff258768c2
* feat(memory): introduce user and peer memory isolation Unify agent-scoped memory behavior into user-owned memory spaces, add peer_id compatibility for session and retrieval paths, and wire memory_policy through session commit flows. * feat(memory): align session identity around peer IDs * feat(search): pass peer id through retrieval * refactor(memory): remove agent identity from integrations * fix(memory): isolate peer identity from self extraction * fix(tau2): provision benchmark user configs * fix(auth): allow admin keys to access data APIs * fix(openclaw): enable peer memory policy for peer roles * fix(openclaw): resolve sender for peer recall * refactor(session): simplify memory extraction routing * refactor(ov-cli): reduce formatting-only diff * refactor(message): remove unused message helpers * refactor(retrieval): simplify peer target resolution * refactor(namespace): remove deprecated agent namespace policy * fix(agent): propagate peer id through integrations * fix(auth): align integration clients with api-key mode
Hermes LoCoMo Benchmark
This directory runs LoCoMo QA through Hermes Agent with three memory paths:
native: imports transcripts into Hermes native memory, then evaluates through Hermes.e2e: imports transcripts through Hermes with the OpenViking memory plugin enabled, commits the resulting OpenViking sessions, then evaluates through Hermes.preingest: imports transcripts directly into OpenViking, then evaluates through Hermes against the preloaded OpenViking state.
Use run_full_eval.sh for normal runs:
cd benchmark/locomo/hermes
./run_full_eval.sh --suite native
./run_full_eval.sh --suite e2e
./run_full_eval.sh --suite preingest
Required Services
- Hermes gateway running at
HERMES_URL(default:http://127.0.0.1:8642). - OpenViking server running at
OPENVIKING_URL(default:http://127.0.0.1:1933) fore2eandpreingest. - LoCoMo dataset at
../data/locomo10.json, or setLOCOMO_JSON=/path/to/locomo10.json. - Judge model credentials through
JUDGE_TOKENorARK_API_KEY.
The default OpenViking benchmark setup is local and does not require an API key. For non-default namespaces or authenticated servers, set OPENVIKING_ACCOUNT, OPENVIKING_USER, and OPENVIKING_API_KEY consistently for import and eval.
Fresh Runs
For OpenViking suites, start from a fresh OpenViking workspace when using --force-ingest:
rm -rf ~/.openviking/data/*
./run_full_eval.sh --suite e2e --force-ingest --force-eval
--force-ingest resets benchmark CSV/token files. It does not delete OpenViking server state. If old OpenViking archives remain, deterministic benchmark session IDs can reuse stale extracted memories.
Use -cp or --checkpoint on e2e and preingest runs to copy the OpenViking state after import:
./run_full_eval.sh --suite e2e -cp
Results And Resume Behavior
Each run writes to a timestamped result_* directory unless RESULT_DIR or --result-dir is provided. Important files:
import_success.csv: successful ingest sessions and Hermes ingest usage where applicable.qa_results.csv: answers, Hermes QA usage, tool calls, judge result, and reasoning.import_true_tokens.csv: OpenViking model-token delta observed after import.eval_true_tokens.csv: OpenViking model-token delta observed after QA.stats.log: final score and token summary.
Interrupted runs can be resumed in the same result directory. OpenViking token delta CSVs append one row per completed import/eval pass, so final stats sum all rows rather than reading only the last row.
Hermes token accounting prefers state.db when available through HERMES_STATE_DB or HERMES_HOME/state.db; otherwise stats fall back to the benchmark CSV usage fields.