hermes-skills/hermes-health-diagnostics/SKILL.md

---
name: hermes-health-diagnostics
description: "Comprehensive Hermes Agent health checks, diagnostics, and root-cause analysis. Covers the full diagnostic methodology: checklist building, peer cross-validation, source-code inspection, and the threading-shutdown hang on SSH logout."
version: 1.0.0
author: agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [hermes, diagnostics, maintenance, health-check, debugging, devops]
    related_skills: [hermes-agent, ask-hermes, skill-library-maintenance]
---

# Hermes Agent Health Diagnostics

Comprehensive health checks, diagnostics, and root-cause analysis for Hermes Agent. Use when the user reports a problem ("hermes is broken", "hangs on exit", "gateway not responding") or wants a full system audit.

## Trigger Conditions

- User reports a Hermes malfunction (hang, crash, error, unresponsive)
- User asks "check what's wrong with hermes"
- User wants a maintenance checklist or health audit
- User reports SSH logout hang or Ctrl+C traceback

## Methodology

### 1. Load the hermes-agent skill first

Always start with `skill_view(name='hermes-agent')` — it has the authoritative CLI commands, config keys, and known pitfalls. The official docs at https://hermes-agent.nousresearch.com/docs are the source of truth.

### 2. Search the official docs

Use `mcp_searxng_searxng_web_search` to find relevant sections:
- CLI commands reference: `/docs/reference/cli-commands`
- FAQ: `/docs/reference/faq`
- Configuration: `/docs/user-guide/configuration`
- Sessions: `/docs/user-guide/sessions`
- Security: `/docs/user-guide/security`
- GitHub issues: `site:github.com/NousResearch/hermes-agent <symptom>`

Fetch specific sections via `mcp_searxng_web_url_read` with the `section` parameter to avoid loading entire pages.

### 3. Inspect the local source

The hermes source lives at `~/.hermes/hermes-agent/`. When a traceback cites specific lines, verify them:
```bash
grep -n "<function_name>" ~/.hermes/hermes-agent/cli.py
sed -n '<start>,<end>p' ~/.hermes/hermes-agent/cli.py
```

### 4. Build a checklist

Structure by category (core health, logs, providers, tools, gateway, cron, sessions, security, performance). Every command should be marked read-only or [FIX]. Cite sources inline.

### 5. Peer cross-validation

Dispatch a peer Hermes to build the same checklist independently:
```
hermes -p general chat -q "Read /home/n8n/hermes_checklist_Maint.md first.
Build your own enriched checklist. Find what I missed. Use mcp_searxng_searxng_web_search
for every claim. Write to /home/n8n/hermes_checklist_Maint_peer.md." -Q --max-turns 20 --yolo
```

Then spot-check the peer's claims against source. Peers find real gaps AND may overstate findings — verify each independently.

### 6. Root-cause analysis

When a traceback is available:
1. Identify the exact file:line from the traceback
2. Read the surrounding source code (50+ lines of context)
3. Trace the call path: what spawns the thread/process, what joins it, what doesn't
4. Check for existing fixes in sibling code paths (e.g., `os._exit(0)` bypass)
5. Search GitHub issues for the same symptom
6. Identify the specific mechanism, not just the symptom

## Diagnostic Checklist Structure

The canonical checklist covers these sections (see `references/checklist-template.md`):

1. Core Health — version, doctor, status, config sanity, git/venv integrity
2. Logs — gateway.log, cron.log, errors.log, /debug, PID/lockfile orphans
3. Provider & Model — connectivity smoke test, auth pools, .env, auxiliary models
4. Tools, Skills, MCP — tools list, skills check/update, mcp list/test, stdio leaks
5. Gateway — status, /platforms, SSH-linger, crash-loop reset
6. Cron — status, list, job-level checks, .tick.lock orphans
7. Sessions & Memory — list/stats, prune, VACUUM, WAL checkpoint, active_profile
8. Security — approvals mode, redaction, file perms, Tirith
9. Analytics & Performance — insights, /usage, /compress, timeouts
10. Disk Space & Storage — ~/.hermes size, state.db, log rotation
11. Process & Thread Hygiene — zombies, background leaks, MCP orphans
12. Hindsight & Qdrant — bank health, collection health, daemon status
13. WebUI — cache staleness, backend connectivity
14. Cross-Profile — config drift, active_profile mismatch

## Known Root Causes

### SSH Logout Hang / Ctrl+C Traceback

**Symptom:** After `hermes` CLI session ends, typing `exit` or logging out of SSH hangs. Ctrl+C produces:
```
Exception ignored on threading shutdown:
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 31, in _python_exit
  File "/usr/lib/python3.13/threading.py", line 1094, in join
  File "cli.py", line 15683, in _signal_handler_q  -> time.sleep(_grace)
  File "cli.py", line 15719, in _signal_handler_q  -> raise KeyboardInterrupt()
```

**Root cause:** Python 3.13's `concurrent.futures.thread._python_exit()` calls `t.join()` on non-daemon ThreadPoolExecutor worker threads at interpreter shutdown. The primary non-daemon threads come from `agent/tool_executor.py:641` (concurrent tool execution, abandoned with `wait=False` on interrupt per line 772). When a signal arrives during that join, `_signal_handler_q` in `cli.py` (line 15683: `time.sleep(_grace)`, line 15719: `raise KeyboardInterrupt()`) interrupts the join. The non-daemon workers are still alive — Python prints "Exception ignored on threading shutdown" and blocks.

**Evidence:**
- `agent/tool_executor.py:641`: `executor = concurrent.futures.ThreadPoolExecutor(max_workers=max_workers)` — non-daemon by default
- `agent/tool_executor.py:772`: comment confirms "wait=False returns immediately" — executor abandoned on interrupt
- `cli.py:15683`: `time.sleep(_grace)` — signal handler sleeps before raising
- `cli.py:15719`: `raise KeyboardInterrupt()` — interrupts the join
- `cli.py:15718`: `os._exit(0)` bypass exists for kanban workers but NOT for the general CLI path

**Workarounds:**
- Run inside tmux: `tmux new -s hermes; hermes` — detach with Ctrl+B D, SSH logout is instant
- Enable SSH linger: `sudo loginctl enable-linger $USER`
- Tune grace period: `export HERMES_SIGTERM_GRACE=0.1` (reduces hang to 0.1s, may not fully fix)

**Upstream:** GitHub Issue #11347 ("/detach — Run Hermes Agent in Background After Exiting CLI"), labeled P3/type/feature. Not yet fixed.

## Pitfalls

- **Don't rely on parametric knowledge.** Every config key, CLI flag, and version number must be verified against the live docs or source.
- **Peers overstate findings.** A peer may correctly identify real gaps AND incorrectly flag things that aren't broken. Verify each claim independently.
- **The hermes-agent skill is bundled/protected — don't edit it.** Load it for reference, but create/update this skill or others for new findings.
- **`hermes status` output is redacted by default.** Use `--all` for full shareable output, `--deep` for slower thorough checks.
- **Tool changes need `/reset`** — they don't apply mid-conversation.
- **Cron in-process ticker only runs while a gateway/CLI process is alive.** No gateway + built-in provider = jobs won't fire.
- **Cron job self-reported stats are often wrong.** A job may claim "100% organized" while the backend shows 22%. Always verify job claims against the actual backend (Qdrant counts, file line counts, etc.) — see `references/cron-deep-diagnostic.md` for the full pattern.

## Verification

After running diagnostics:
```bash
# Quick daily health one-liner
hermes --version && hermes doctor && hermes status --all && \
hermes gateway list && hermes cron status && hermes sessions stats && \
hermes auth list && tail -n 20 ~/.hermes/logs/gateway.log
```

For the SSH-hang issue specifically:
```bash
# Identify lingering children
ps -ef | grep -E 'hermes|uv|python' | grep -v grep
ps -t <tty> -o pid,ppid,stat,cmd
ls ~/.hermes/*.pid 2>/dev/null
```

## Support Files

- `references/checklist-template.md` — Full diagnostic checklist template with all 14 sections.
- `references/cron-deep-diagnostic.md` — Pattern for deep-diving a cron job: read run outputs, inspect the job's own log, check its error log, and verify claims directly against the backend (Qdrant, etc.).