Add all 104 active skills from all 16 Hermes profiles
12 unversioned skills now versioned at 1.0.0: agent-communication, ascii-video, external-reasoning-augmentation, jotty-notes-api, minecraft-modpack-server, obsidian, pokemon-player, powerpoint, social-search, songwriting-and-ai-music, workspace-context-organization, youtube-content Total repo: 141 skills across all profile scopes
This commit is contained in:
Hermes Agent
155
hermes-health-diagnostics/SKILL.md
Normal file
155
hermes-health-diagnostics/SKILL.md
Normal file
@@ -0,0 +1,155 @@
|
||||
---
|
||||
name: hermes-health-diagnostics
|
||||
description: "Comprehensive Hermes Agent health checks, diagnostics, and root-cause analysis. Covers the full diagnostic methodology: checklist building, peer cross-validation, source-code inspection, and the threading-shutdown hang on SSH logout."
|
||||
version: 1.0.0
|
||||
author: agent
|
||||
license: MIT
|
||||
platforms: [linux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [hermes, diagnostics, maintenance, health-check, debugging, devops]
|
||||
related_skills: [hermes-agent, ask-hermes, skill-library-maintenance]
|
||||
---
|
||||
|
||||
# Hermes Agent Health Diagnostics
|
||||
|
||||
Comprehensive health checks, diagnostics, and root-cause analysis for Hermes Agent. Use when the user reports a problem ("hermes is broken", "hangs on exit", "gateway not responding") or wants a full system audit.
|
||||
|
||||
## Trigger Conditions
|
||||
|
||||
- User reports a Hermes malfunction (hang, crash, error, unresponsive)
|
||||
- User asks "check what's wrong with hermes"
|
||||
- User wants a maintenance checklist or health audit
|
||||
- User reports SSH logout hang or Ctrl+C traceback
|
||||
|
||||
## Methodology
|
||||
|
||||
### 1. Load the hermes-agent skill first
|
||||
|
||||
Always start with `skill_view(name='hermes-agent')` — it has the authoritative CLI commands, config keys, and known pitfalls. The official docs at https://hermes-agent.nousresearch.com/docs are the source of truth.
|
||||
|
||||
### 2. Search the official docs
|
||||
|
||||
Use `mcp_searxng_searxng_web_search` to find relevant sections:
|
||||
- CLI commands reference: `/docs/reference/cli-commands`
|
||||
- FAQ: `/docs/reference/faq`
|
||||
- Configuration: `/docs/user-guide/configuration`
|
||||
- Sessions: `/docs/user-guide/sessions`
|
||||
- Security: `/docs/user-guide/security`
|
||||
- GitHub issues: `site:github.com/NousResearch/hermes-agent <symptom>`
|
||||
|
||||
Fetch specific sections via `mcp_searxng_web_url_read` with the `section` parameter to avoid loading entire pages.
|
||||
|
||||
### 3. Inspect the local source
|
||||
|
||||
The hermes source lives at `~/.hermes/hermes-agent/`. When a traceback cites specific lines, verify them:
|
||||
```bash
|
||||
grep -n "<function_name>" ~/.hermes/hermes-agent/cli.py
|
||||
sed -n '<start>,<end>p' ~/.hermes/hermes-agent/cli.py
|
||||
```
|
||||
|
||||
### 4. Build a checklist
|
||||
|
||||
Structure by category (core health, logs, providers, tools, gateway, cron, sessions, security, performance). Every command should be marked read-only or [FIX]. Cite sources inline.
|
||||
|
||||
### 5. Peer cross-validation
|
||||
|
||||
Dispatch a peer Hermes to build the same checklist independently:
|
||||
```
|
||||
hermes -p general chat -q "Read /home/n8n/hermes_checklist_Maint.md first.
|
||||
Build your own enriched checklist. Find what I missed. Use mcp_searxng_searxng_web_search
|
||||
for every claim. Write to /home/n8n/hermes_checklist_Maint_peer.md." -Q --max-turns 20 --yolo
|
||||
```
|
||||
|
||||
Then spot-check the peer's claims against source. Peers find real gaps AND may overstate findings — verify each independently.
|
||||
|
||||
### 6. Root-cause analysis
|
||||
|
||||
When a traceback is available:
|
||||
1. Identify the exact file:line from the traceback
|
||||
2. Read the surrounding source code (50+ lines of context)
|
||||
3. Trace the call path: what spawns the thread/process, what joins it, what doesn't
|
||||
4. Check for existing fixes in sibling code paths (e.g., `os._exit(0)` bypass)
|
||||
5. Search GitHub issues for the same symptom
|
||||
6. Identify the specific mechanism, not just the symptom
|
||||
|
||||
## Diagnostic Checklist Structure
|
||||
|
||||
The canonical checklist covers these sections (see `references/checklist-template.md`):
|
||||
|
||||
1. Core Health — version, doctor, status, config sanity, git/venv integrity
|
||||
2. Logs — gateway.log, cron.log, errors.log, /debug, PID/lockfile orphans
|
||||
3. Provider & Model — connectivity smoke test, auth pools, .env, auxiliary models
|
||||
4. Tools, Skills, MCP — tools list, skills check/update, mcp list/test, stdio leaks
|
||||
5. Gateway — status, /platforms, SSH-linger, crash-loop reset
|
||||
6. Cron — status, list, job-level checks, .tick.lock orphans
|
||||
7. Sessions & Memory — list/stats, prune, VACUUM, WAL checkpoint, active_profile
|
||||
8. Security — approvals mode, redaction, file perms, Tirith
|
||||
9. Analytics & Performance — insights, /usage, /compress, timeouts
|
||||
10. Disk Space & Storage — ~/.hermes size, state.db, log rotation
|
||||
11. Process & Thread Hygiene — zombies, background leaks, MCP orphans
|
||||
12. Hindsight & Qdrant — bank health, collection health, daemon status
|
||||
13. WebUI — cache staleness, backend connectivity
|
||||
14. Cross-Profile — config drift, active_profile mismatch
|
||||
|
||||
## Known Root Causes
|
||||
|
||||
### SSH Logout Hang / Ctrl+C Traceback
|
||||
|
||||
**Symptom:** After `hermes` CLI session ends, typing `exit` or logging out of SSH hangs. Ctrl+C produces:
|
||||
```
|
||||
Exception ignored on threading shutdown:
|
||||
File "/usr/lib/python3.13/concurrent/futures/thread.py", line 31, in _python_exit
|
||||
File "/usr/lib/python3.13/threading.py", line 1094, in join
|
||||
File "cli.py", line 15683, in _signal_handler_q -> time.sleep(_grace)
|
||||
File "cli.py", line 15719, in _signal_handler_q -> raise KeyboardInterrupt()
|
||||
```
|
||||
|
||||
**Root cause:** Python 3.13's `concurrent.futures.thread._python_exit()` calls `t.join()` on non-daemon ThreadPoolExecutor worker threads at interpreter shutdown. The primary non-daemon threads come from `agent/tool_executor.py:641` (concurrent tool execution, abandoned with `wait=False` on interrupt per line 772). When a signal arrives during that join, `_signal_handler_q` in `cli.py` (line 15683: `time.sleep(_grace)`, line 15719: `raise KeyboardInterrupt()`) interrupts the join. The non-daemon workers are still alive — Python prints "Exception ignored on threading shutdown" and blocks.
|
||||
|
||||
**Evidence:**
|
||||
- `agent/tool_executor.py:641`: `executor = concurrent.futures.ThreadPoolExecutor(max_workers=max_workers)` — non-daemon by default
|
||||
- `agent/tool_executor.py:772`: comment confirms "wait=False returns immediately" — executor abandoned on interrupt
|
||||
- `cli.py:15683`: `time.sleep(_grace)` — signal handler sleeps before raising
|
||||
- `cli.py:15719`: `raise KeyboardInterrupt()` — interrupts the join
|
||||
- `cli.py:15718`: `os._exit(0)` bypass exists for kanban workers but NOT for the general CLI path
|
||||
|
||||
**Workarounds:**
|
||||
- Run inside tmux: `tmux new -s hermes; hermes` — detach with Ctrl+B D, SSH logout is instant
|
||||
- Enable SSH linger: `sudo loginctl enable-linger $USER`
|
||||
- Tune grace period: `export HERMES_SIGTERM_GRACE=0.1` (reduces hang to 0.1s, may not fully fix)
|
||||
|
||||
**Upstream:** GitHub Issue #11347 ("/detach — Run Hermes Agent in Background After Exiting CLI"), labeled P3/type/feature. Not yet fixed.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Don't rely on parametric knowledge.** Every config key, CLI flag, and version number must be verified against the live docs or source.
|
||||
- **Peers overstate findings.** A peer may correctly identify real gaps AND incorrectly flag things that aren't broken. Verify each claim independently.
|
||||
- **The hermes-agent skill is bundled/protected — don't edit it.** Load it for reference, but create/update this skill or others for new findings.
|
||||
- **`hermes status` output is redacted by default.** Use `--all` for full shareable output, `--deep` for slower thorough checks.
|
||||
- **Tool changes need `/reset`** — they don't apply mid-conversation.
|
||||
- **Cron in-process ticker only runs while a gateway/CLI process is alive.** No gateway + built-in provider = jobs won't fire.
|
||||
- **Cron job self-reported stats are often wrong.** A job may claim "100% organized" while the backend shows 22%. Always verify job claims against the actual backend (Qdrant counts, file line counts, etc.) — see `references/cron-deep-diagnostic.md` for the full pattern.
|
||||
|
||||
## Verification
|
||||
|
||||
After running diagnostics:
|
||||
```bash
|
||||
# Quick daily health one-liner
|
||||
hermes --version && hermes doctor && hermes status --all && \
|
||||
hermes gateway list && hermes cron status && hermes sessions stats && \
|
||||
hermes auth list && tail -n 20 ~/.hermes/logs/gateway.log
|
||||
```
|
||||
|
||||
For the SSH-hang issue specifically:
|
||||
```bash
|
||||
# Identify lingering children
|
||||
ps -ef | grep -E 'hermes|uv|python' | grep -v grep
|
||||
ps -t <tty> -o pid,ppid,stat,cmd
|
||||
ls ~/.hermes/*.pid 2>/dev/null
|
||||
```
|
||||
|
||||
## Support Files
|
||||
|
||||
- `references/checklist-template.md` — Full diagnostic checklist template with all 14 sections.
|
||||
- `references/cron-deep-diagnostic.md` — Pattern for deep-diving a cron job: read run outputs, inspect the job's own log, check its error log, and verify claims directly against the backend (Qdrant, etc.).
|
||||
Reference in New Issue
Block a user