hermes-skills/hermes-health-diagnostics/SKILL.md

name, description, version, author, license, platforms, metadata

name	description	version	author	license	platforms	metadata
hermes-health-diagnostics	Comprehensive Hermes Agent health checks, diagnostics, and root-cause analysis. Covers the full diagnostic methodology: checklist building, peer cross-validation, source-code inspection, and the threading-shutdown hang on SSH logout.	1.0.0	agent	MIT	linux	hermes tags related_skills hermes diagnostics maintenance health-check debugging devops hermes-agent ask-hermes skill-library-maintenance

Hermes Agent Health Diagnostics

Comprehensive health checks, diagnostics, and root-cause analysis for Hermes Agent. Use when the user reports a problem ("hermes is broken", "hangs on exit", "gateway not responding") or wants a full system audit.

Trigger Conditions

• User reports a Hermes malfunction (hang, crash, error, unresponsive)
• User asks "check what's wrong with hermes"
• User wants a maintenance checklist or health audit
• User reports SSH logout hang or Ctrl+C traceback

Methodology

1. Load the hermes-agent skill first

Always start with skill_view(name='hermes-agent') — it has the authoritative CLI commands, config keys, and known pitfalls. The official docs at https://hermes-agent.nousresearch.com/docs are the source of truth.

2. Search the official docs

Use mcp_searxng_searxng_web_search to find relevant sections:

• CLI commands reference: /docs/reference/cli-commands
• FAQ: /docs/reference/faq
• Configuration: /docs/user-guide/configuration
• Sessions: /docs/user-guide/sessions
• Security: /docs/user-guide/security
• GitHub issues: site:github.com/NousResearch/hermes-agent <symptom>

Fetch specific sections via mcp_searxng_web_url_read with the section parameter to avoid loading entire pages.

3. Inspect the local source

The hermes source lives at ~/.hermes/hermes-agent/. When a traceback cites specific lines, verify them:

grep -n "<function_name>" ~/.hermes/hermes-agent/cli.py
sed -n '<start>,<end>p' ~/.hermes/hermes-agent/cli.py

4. Build a checklist

Structure by category (core health, logs, providers, tools, gateway, cron, sessions, security, performance). Every command should be marked read-only or [FIX]. Cite sources inline.

5. Peer cross-validation

Dispatch a peer Hermes to build the same checklist independently:

hermes -p general chat -q "Read /home/n8n/hermes_checklist_Maint.md first.
Build your own enriched checklist. Find what I missed. Use mcp_searxng_searxng_web_search
for every claim. Write to /home/n8n/hermes_checklist_Maint_peer.md." -Q --max-turns 20 --yolo

Then spot-check the peer's claims against source. Peers find real gaps AND may overstate findings — verify each independently.

6. Root-cause analysis

When a traceback is available:

1. Identify the exact file:line from the traceback
2. Read the surrounding source code (50+ lines of context)
3. Trace the call path: what spawns the thread/process, what joins it, what doesn't
4. Check for existing fixes in sibling code paths (e.g., os._exit(0) bypass)
5. Search GitHub issues for the same symptom
6. Identify the specific mechanism, not just the symptom

Diagnostic Checklist Structure

The canonical checklist covers these sections (see references/checklist-template.md):

1. Core Health — version, doctor, status, config sanity, git/venv integrity
2. Logs — gateway.log, cron.log, errors.log, /debug, PID/lockfile orphans
3. Provider & Model — connectivity smoke test, auth pools, .env, auxiliary models
4. Tools, Skills, MCP — tools list, skills check/update, mcp list/test, stdio leaks
5. Gateway — status, /platforms, SSH-linger, crash-loop reset
6. Cron — status, list, job-level checks, .tick.lock orphans
7. Sessions & Memory — list/stats, prune, VACUUM, WAL checkpoint, active_profile
8. Security — approvals mode, redaction, file perms, Tirith
9. Analytics & Performance — insights, /usage, /compress, timeouts
10. Disk Space & Storage — ~/.hermes size, state.db, log rotation
11. Process & Thread Hygiene — zombies, background leaks, MCP orphans
12. Hindsight & Qdrant — bank health, collection health, daemon status
13. WebUI — cache staleness, backend connectivity
14. Cross-Profile — config drift, active_profile mismatch

Known Root Causes

SSH Logout Hang / Ctrl+C Traceback

Symptom: After hermes CLI session ends, typing exit or logging out of SSH hangs. Ctrl+C produces:

Exception ignored on threading shutdown:
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 31, in _python_exit
  File "/usr/lib/python3.13/threading.py", line 1094, in join
  File "cli.py", line 15683, in _signal_handler_q  -> time.sleep(_grace)
  File "cli.py", line 15719, in _signal_handler_q  -> raise KeyboardInterrupt()

Root cause: Python 3.13's concurrent.futures.thread._python_exit() calls t.join() on non-daemon ThreadPoolExecutor worker threads at interpreter shutdown. The primary non-daemon threads come from agent/tool_executor.py:641 (concurrent tool execution, abandoned with wait=False on interrupt per line 772). When a signal arrives during that join, _signal_handler_q in cli.py (line 15683: time.sleep(_grace), line 15719: raise KeyboardInterrupt()) interrupts the join. The non-daemon workers are still alive — Python prints "Exception ignored on threading shutdown" and blocks.

Evidence:

• agent/tool_executor.py:641: executor = concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) — non-daemon by default
• agent/tool_executor.py:772: comment confirms "wait=False returns immediately" — executor abandoned on interrupt
• cli.py:15683: time.sleep(_grace) — signal handler sleeps before raising
• cli.py:15719: raise KeyboardInterrupt() — interrupts the join
• cli.py:15718: os._exit(0) bypass exists for kanban workers but NOT for the general CLI path

Workarounds:

• Run inside tmux: tmux new -s hermes; hermes — detach with Ctrl+B D, SSH logout is instant
• Enable SSH linger: sudo loginctl enable-linger $USER
• Tune grace period: export HERMES_SIGTERM_GRACE=0.1 (reduces hang to 0.1s, may not fully fix)

Upstream: GitHub Issue #11347 ("/detach — Run Hermes Agent in Background After Exiting CLI"), labeled P3/type/feature. Not yet fixed.

Pitfalls

• Don't rely on parametric knowledge. Every config key, CLI flag, and version number must be verified against the live docs or source.
• Peers overstate findings. A peer may correctly identify real gaps AND incorrectly flag things that aren't broken. Verify each claim independently.
• The hermes-agent skill is bundled/protected — don't edit it. Load it for reference, but create/update this skill or others for new findings.
• hermes status output is redacted by default. Use --all for full shareable output, --deep for slower thorough checks.
• Tool changes need /reset — they don't apply mid-conversation.
• Cron in-process ticker only runs while a gateway/CLI process is alive. No gateway + built-in provider = jobs won't fire.
• Cron job self-reported stats are often wrong. A job may claim "100% organized" while the backend shows 22%. Always verify job claims against the actual backend (Qdrant counts, file line counts, etc.) — see references/cron-deep-diagnostic.md for the full pattern.

Verification

After running diagnostics:

# Quick daily health one-liner
hermes --version && hermes doctor && hermes status --all && \
hermes gateway list && hermes cron status && hermes sessions stats && \
hermes auth list && tail -n 20 ~/.hermes/logs/gateway.log

For the SSH-hang issue specifically:

# Identify lingering children
ps -ef | grep -E 'hermes|uv|python' | grep -v grep
ps -t <tty> -o pid,ppid,stat,cmd
ls ~/.hermes/*.pid 2>/dev/null

Support Files

• references/checklist-template.md — Full diagnostic checklist template with all 14 sections.
• references/cron-deep-diagnostic.md — Pattern for deep-diving a cron job: read run outputs, inspect the job's own log, check its error log, and verify claims directly against the backend (Qdrant, etc.).