--- name: hermes-state-db-repair description: "Diagnose and repair corrupted Hermes state.db files." version: 1.0.0 --- # Hermes State DB Repair Diagnose and repair corrupted Hermes `state.db` (SQLite) files. Covers the `~/.hermes/state.db` and per-profile `~/.hermes/profiles//state.db`. ## Quick Diagnosis ```bash python3 -c " import sqlite3 db = sqlite3.connect('/home/n8n/.hermes/profiles//state.db') cur = db.execute('PRAGMA integrity_check') print(cur.fetchone()[0]) db.close() " ``` A healthy DB returns `ok`. Corruption produces btree page errors. ## Key Insight: FTS5 vs Direct Queries `state.db` has two access paths: | Path | Mechanism | Corruption behavior | |------|-----------|-------------------| | `session_search` tool | FTS5 virtual table (`messages_fts`) | **Fails** — "database disk image is malformed" | | Direct SQLite queries | `sessions` / `messages` tables | **Often still works** — btree corruption may only hit FTS indexes | When `session_search` fails but `hermes sessions list` works, the FTS5 index is corrupted but the base tables are intact. Use direct Python SQLite queries as a workaround for inspection. ## Why Sessions Have 0 Messages A session row is inserted into the `sessions` table the moment Hermes starts — before the user types anything. If the user exits immediately (Ctrl+C, `/exit`, process killed), the row stays with `message_count=0`. These are not bugs; they are abandoned session starts. ## `sessions.json` Is NOT the Session List `~/.hermes/profiles//sessions/sessions.json` is a **gateway routing index only** — it maps messaging session keys to active session IDs. All sessions (CLI, TUI, and gateway) live in `state.db`. Seeing only gateway entries in `sessions.json` is expected. ## Recovery Procedures ### Option 1: Salvage Script (row-by-row, preserves maximum data) Use the bundled `scripts/salvage_state_db.py`: ```bash python3 ~/.hermes/profiles/general/skills/devops/hermes-state-db-repair/scripts/salvage_state_db.py \ /path/to/corrupted.db /path/to/output.db ``` The script: - Creates a fresh DB with clean schema (no inherited corruption) - Copies every table row-by-row, skipping only corrupted rows - Handles composite primary keys (session_model_usage, gateway_routing, etc.) - Rebuilds all indexes and FTS5 from scratch - Reports exact loss count per table - Runs integrity check on the output **When to use this over Option 2:** When the corruption is in data pages (not just FTS indexes), or when `sqlite3 .recover` isn't available. This was the only option that worked when `iterdump()` and `SELECT *` both failed on the messages table. ### Option 2: SQLite `.recover` ```bash sqlite3 corrupted.db ".recover" | sqlite3 fresh.db ``` Salvages all readable pages. Some data may be lost from corrupted pages. ### Option 3: Delete and Start Fresh Lose session history but fix corruption instantly. Only the `state.db` — never delete `memory_store.db` or other DBs. ## Schema Reference Key tables in `state.db`: - `sessions` — id, source, started_at, message_count, title, model, tokens, cost - `messages` — id, session_id, role, content, tool_calls - `messages_fts` — FTS5 virtual table over messages.content - `gateway_routing` — platform session key mappings - `session_model_usage` — per-session token/cost tracking - `schema_version` — migration tracking Full column listing: see `references/schema-dump.md`. ## Pitfalls - **Don't delete `state.db` while Hermes is running.** The process holds a lock and may recreate it mid-session, causing data loss. - **FTS5 corruption doesn't mean the whole DB is lost.** Always try direct queries before assuming data is gone. - **`session_search` failure is a symptom, not the root cause.** The root cause is btree corruption in the FTS5 index pages. - **After swapping the DB file, the running Hermes process still uses the old inode.** Even after a successful rebuild, `session_search` will keep failing in the current session because the process opened the DB at startup. A `/reset` or restart is required for the new file to take effect. Verify with a direct Python SQLite query — if that passes integrity but `session_search` still fails, the swap worked and the process just needs to reopen. - **WAL/SHM files from a corrupted DB can poison reads of a clean replacement.** When swapping state.db, also delete any lingering `state.db-wal` and `state.db-shm` files. They belong to the old (corrupted) database and can cause the new file to appear corrupted. - **`iterdump()` fails on corrupted data pages, not just FTS.** When corruption is in the messages table data pages (not just the FTS index), `iterdump()` and `SELECT *` both fail. The salvage script handles this by querying row-by-row via primary key, skipping only the corrupted rows. - **Tables with composite primary keys need special handling.** The salvage script's `TABLE_PKS` mapping covers all Hermes tables including `session_model_usage` (6-column PK) and `gateway_routing` (2-column PK). Generic `WHERE id = ?` approaches will miss these tables entirely.