hermes-skills/hermes-state-db-repair/SKILL.md

---
name: hermes-state-db-repair
description: "Diagnose and repair corrupted Hermes state.db files."
version: 1.0.0
---

# Hermes State DB Repair

Diagnose and repair corrupted Hermes `state.db` (SQLite) files. Covers the
`~/.hermes/state.db` and per-profile `~/.hermes/profiles/<name>/state.db`.

## Quick Diagnosis

```bash
python3 -c "
import sqlite3
db = sqlite3.connect('/home/n8n/.hermes/profiles/<profile>/state.db')
cur = db.execute('PRAGMA integrity_check')
print(cur.fetchone()[0])
db.close()
"
```

A healthy DB returns `ok`. Corruption produces btree page errors.

## Key Insight: FTS5 vs Direct Queries

`state.db` has two access paths:

| Path | Mechanism | Corruption behavior |
|------|-----------|-------------------|
| `session_search` tool | FTS5 virtual table (`messages_fts`) | **Fails** — "database disk image is malformed" |
| Direct SQLite queries | `sessions` / `messages` tables | **Often still works** — btree corruption may only hit FTS indexes |

When `session_search` fails but `hermes sessions list` works, the FTS5 index is
corrupted but the base tables are intact. Use direct Python SQLite queries as a
workaround for inspection.

## Why Sessions Have 0 Messages

A session row is inserted into the `sessions` table the moment Hermes starts —
before the user types anything. If the user exits immediately (Ctrl+C, `/exit`,
process killed), the row stays with `message_count=0`. These are not bugs; they
are abandoned session starts.

## `sessions.json` Is NOT the Session List

`~/.hermes/profiles/<name>/sessions/sessions.json` is a **gateway routing index
only** — it maps messaging session keys to active session IDs. All sessions
(CLI, TUI, and gateway) live in `state.db`. Seeing only gateway entries in
`sessions.json` is expected.

## Recovery Procedures

### Option 1: Salvage Script (row-by-row, preserves maximum data)

Use the bundled `scripts/salvage_state_db.py`:

```bash
python3 ~/.hermes/profiles/general/skills/devops/hermes-state-db-repair/scripts/salvage_state_db.py \
    /path/to/corrupted.db /path/to/output.db
```

The script:
- Creates a fresh DB with clean schema (no inherited corruption)
- Copies every table row-by-row, skipping only corrupted rows
- Handles composite primary keys (session_model_usage, gateway_routing, etc.)
- Rebuilds all indexes and FTS5 from scratch
- Reports exact loss count per table
- Runs integrity check on the output

**When to use this over Option 2:** When the corruption is in data pages (not just FTS indexes), or when `sqlite3 .recover` isn't available. This was the only option that worked when `iterdump()` and `SELECT *` both failed on the messages table.

### Option 2: SQLite `.recover`

```bash
sqlite3 corrupted.db ".recover" | sqlite3 fresh.db
```

Salvages all readable pages. Some data may be lost from corrupted pages.

### Option 3: Delete and Start Fresh

Lose session history but fix corruption instantly. Only the `state.db` — never
delete `memory_store.db` or other DBs.

## Schema Reference

Key tables in `state.db`:

- `sessions` — id, source, started_at, message_count, title, model, tokens, cost
- `messages` — id, session_id, role, content, tool_calls
- `messages_fts` — FTS5 virtual table over messages.content
- `gateway_routing` — platform session key mappings
- `session_model_usage` — per-session token/cost tracking
- `schema_version` — migration tracking

Full column listing: see `references/schema-dump.md`.

## Pitfalls

- **Don't delete `state.db` while Hermes is running.** The process holds a lock
  and may recreate it mid-session, causing data loss.
- **FTS5 corruption doesn't mean the whole DB is lost.** Always try direct
  queries before assuming data is gone.
- **`session_search` failure is a symptom, not the root cause.** The root cause
  is btree corruption in the FTS5 index pages.
- **After swapping the DB file, the running Hermes process still uses the old inode.**
  Even after a successful rebuild, `session_search` will keep failing in the
  current session because the process opened the DB at startup. A `/reset` or
  restart is required for the new file to take effect. Verify with a direct
  Python SQLite query — if that passes integrity but `session_search` still
  fails, the swap worked and the process just needs to reopen.
- **WAL/SHM files from a corrupted DB can poison reads of a clean replacement.**
  When swapping state.db, also delete any lingering `state.db-wal` and
  `state.db-shm` files. They belong to the old (corrupted) database and can
  cause the new file to appear corrupted.
- **`iterdump()` fails on corrupted data pages, not just FTS.** When corruption
  is in the messages table data pages (not just the FTS index), `iterdump()`
  and `SELECT *` both fail. The salvage script handles this by querying
  row-by-row via primary key, skipping only the corrupted rows.
- **Tables with composite primary keys need special handling.** The salvage
  script's `TABLE_PKS` mapping covers all Hermes tables including
  `session_model_usage` (6-column PK) and `gateway_routing` (2-column PK).
  Generic `WHERE id = ?` approaches will miss these tables entirely.