Update ask-claude (v2.4.0) and ask-hermes (v4.1.0) with Critical Evaluation + Limitations sections
ask-claude v2.4.0: - Add 'Critical Evaluation — You Are the Final Authority' (5 rules) - Add 'Claude's Limitations' (file access, fabrication, environment) - Add ask.sh fallback pitfall ask-hermes v4.1.0: - Add 'Critical Evaluation — You Are the Final Authority' (5 rules) - Add 'Peer Limitations' (no memory, fabrication, environment, self-reports)
This commit is contained in:
Hermes Agent
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: ask-claude
|
||||
description: "Consult Claude Opus 4.8 on 10.0.0.28 via SSH print mode with session resumption. Multi-turn back-and-forth without tmux or polling."
|
||||
version: 2.3.0
|
||||
version: 2.4.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux]
|
||||
@@ -19,15 +19,19 @@ Consult Claude Opus on 10.0.0.28 using headless print mode (`-p`) with session r
|
||||
|
||||
## ⚡ Quick Reference
|
||||
|
||||
**Primary method (scp + ask.sh) — works with ANY question content, no shell escaping issues:**
|
||||
|
||||
| Step | What | Command Pattern |
|
||||
|------|------|----------------|
|
||||
| 1. Write question | Temp file on 10.0.0.28 via mktemp | `ssh ... 'mktemp ... && cat > $QFILE << "EOF"\nquestion\nEOF'` |
|
||||
| 2. First question | No `--resume`, `|| true` to survive errors | `timeout 280 claude -p "$(cat $QFILE)" --output-format json --model claude-opus-4-8 --max-turns 30 || true` |
|
||||
| 3. Parse result | Extract `session_id`, `result`, `is_error`, detect `error_max_turns` | `python3 -c "import sys,json; ..."` |
|
||||
| 4. Follow-up | WITH `--resume <sid>` | `claude -p --resume <sid> "$(cat $QFILE)" --output-format json --model claude-opus-4-8 --max-turns 30 || true` |
|
||||
| 5. Done | Process exits. Session auto-expires in ~5h. | `rm $QFILE` (trap on EXIT) |
|
||||
| 1. Write question | Local temp file | `write_file("/tmp/ask-claude-q.txt", content=question)` |
|
||||
| 2. Copy to remote | scp to 10.0.0.28 | `terminal("scp /tmp/ask-claude-q.txt n8n@10.0.0.28:/tmp/ask-claude-q.txt")` |
|
||||
| 3. Run via ask.sh | Fresh session (no --resume) | `terminal("ssh n8n@10.0.0.28 '~/claude/hermes_support/ask.sh --qfile /tmp/ask-claude-q.txt'")` |
|
||||
| 4. Follow-up | WITH `--resume <sid>` | `terminal("ssh n8n@10.0.0.28 '~/claude/hermes_support/ask.sh --resume SID --qfile /tmp/ask-claude-q.txt'")` |
|
||||
| 5. Done | Process exits. Session auto-expires in ~5h. | Cleanup temp files if desired. |
|
||||
|
||||
**Critical:** Always use `--output-format json`, `--model claude-opus-4-8`, and `--max-turns 30`. Pin the model explicitly — profile defaults drift silently. Use `|| true` after the claude command to survive non-zero exits (errors still produce parseable JSON). Use full path `/home/n8n/.local/bin/claude` on non-interactive SSH. Wrap with `timeout 280` to prevent runaway hangs.
|
||||
**Legacy method (mktemp + heredoc) — ONLY for simple questions with zero shell metacharacters.** Breaks on `"`, `$`, `(`, `)`, backticks. The scp method above is preferred for all real questions.
|
||||
|
||||
**Critical:** ask.sh handles `--output-format json`, `--model claude-opus-4-8`, `--max-turns 30`, `timeout 280`, and `|| true` internally. Pin the model explicitly — profile defaults drift silently.
|
||||
|
||||
## Decision Point: `--resume` or Fresh Session?
|
||||
|
||||
@@ -55,15 +59,35 @@ Is there an active Claude session_id from a previous turn in THIS Hermes session
|
||||
- Second opinion on architecture, security, or design
|
||||
- Research that benefits from SearXNG + Opus reasoning
|
||||
- Multi-turn discussion where Claude may ask clarifying questions
|
||||
- **Adversarial consulting** — ask for advice, then ask Claude to push back on its own advice with web searches. See `references/adversarial-consulting.md` for the pattern.
|
||||
|
||||
## Local Hermes Peer (Alternative)
|
||||
## Critical Evaluation — You Are the Final Authority
|
||||
|
||||
When Claude is unreachable, overkill, or the user wants a free local peer, use the `ask-hermes` skill (planned, not yet built — see `/home/n8n/workspace/dev/ask_hermes_plan.md`). It starts a persistent peer Hermes agent on a different profile via plain `hermes` (interactive REPL, no flags). Three commands: `start ask` / `ask hermes <instructions>` / `stop ask`. The peer is a full delegated worker with all tools — not just a Q&A box. See the plan for the full mechanism, prompt composition rules, and turn budget (max 20, soft limit).
|
||||
**Claude's advice is input, not a decision.** You are the engineer — you own the code, the system, and the final call. Claude is a consultant who cannot see your files, run your commands, or verify its own claims. Treat every recommendation as a hypothesis to be tested, not a conclusion to be implemented.
|
||||
|
||||
**Build-attempt findings (2026-06-30):**
|
||||
- `hermes -p general` works — `-p` flag exists despite not appearing in top-level `--help`.
|
||||
- REPL needs `pty=true` to stay alive; without it, prompt_toolkit exits immediately ("Input is not a terminal").
|
||||
- **Blocker: TUI output is unreadable via `process(poll)`.** The interactive REPL's ANSI redraw codes make the actual response text unparseable. Input works, output doesn't. This is the open problem — the plan's mechanism (plain `hermes`, interactive REPL, no flags) can't currently read peer responses cleanly. Possible paths: `--cli` flag for classic REPL, or headless `-z` one-shot with `--resume` for continuity.
|
||||
**Mandatory before acting on Claude's advice:**
|
||||
|
||||
1. **Do your own reasoning first.** Before relaying Claude's answer, ask yourself: does this make sense? Is there a simpler way? Would this actually work in our environment?
|
||||
|
||||
2. **Push back when Claude is wrong.** If Claude suggests something that doesn't fit — wrong tool, overengineered solution, unverified claim — tell it so explicitly in the next turn. "That won't work because X. Do a web search for alternatives." Claude corrects itself well when challenged; it doubles down when unchallenged.
|
||||
|
||||
3. **Do your own web searches to validate.** Claude's SearXNG searches are real but its interpretation may be wrong. For any claim about API behavior, version compatibility, or tool capabilities, run your own `mcp_searxng_searxng_web_search` and compare. Claude has been wrong about Qdrant endpoints, Python behavior, and shell patterns in this exact session.
|
||||
|
||||
4. **Involve the user when there's genuine ambiguity.** If Claude and you disagree after a round of pushback, or if the right approach depends on constraints only the user knows, surface the disagreement with both positions stated clearly. Don't silently pick Claude's answer over your own judgment.
|
||||
|
||||
5. **Watch for overcomplication.** Claude's default mode is thoroughness, which can become overengineering. If Claude proposes a multi-step protocol with atomic renames, CAS semantics, and lease expiration — ask: "Is there a simpler way? Can we solve this with a 50-line Python script instead?" The best fix is usually the simplest one that actually works.
|
||||
|
||||
## Claude's Limitations
|
||||
|
||||
**Claude cannot read your local files.** Claude runs on 10.0.0.28 and has no access to `/home/n8n/workspace` or any other path on your machine. When Claude says "I can't read that file" or "that path doesn't exist here" — **believe it.** Do not ask Claude to validate a file by path. Instead:
|
||||
|
||||
- **For code review:** Paste the relevant functions inline in your question. Claude can review pasted code accurately.
|
||||
- **For design review:** Describe the system and paste the key sections. Claude can reason from descriptions if they're specific.
|
||||
- **For "is this file correct?":** You verify it yourself. Claude cannot.
|
||||
|
||||
**Claude may fabricate plausible-sounding analysis when it can't verify.** In this session, Claude confidently claimed the deployed prompt was a "v5 classify-only rewrite" based on a session title it couldn't re-open. That claim was wrong. When Claude prefaces a claim with "from my notes" or "I recall" without being able to re-read the source, treat it as unverified.
|
||||
|
||||
**Claude's environment is not your environment.** Claude has its own tools, its own filesystem, and its own constraints. A command that works in Claude's environment may fail in yours. Always test Claude's suggestions locally before deploying.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -302,6 +326,7 @@ Sonnet is 3-5x faster and cheaper. Use for: factual lookups, simple analysis, qu
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **ask.sh can fail silently while direct `claude -p` works.** The wrapper may return `{"is_error": true, "error": "no Claude output (timeout or CLI failure)"}` even when Claude is healthy. When this happens, fall back to the direct invocation: `ssh n8n@10.0.0.28 'cd ~/claude/hermes_support && timeout 280 /home/n8n/.local/bin/claude -p "$(cat /tmp/ask-claude-q.txt)" --output-format json --model claude-opus-4-8 --max-turns 30 || true'`. The wrapper is preferred (it handles usage API + merge.py), but the direct path is the reliable fallback. If the direct path also fails, Claude is genuinely down.
|
||||
- **CRITICAL: Never inline questions in SSH commands.** Shell metacharacters (`"`, `$`, `(`, backticks) will break. Always write to a temp file via `mktemp` first, then `claude -p "$(cat $QFILE)"`.
|
||||
- **Always use `--output-format json`** — without it, you get plain text and can't extract session_id for follow-ups.
|
||||
- **Always set `--max-turns`** — prevents runaway loops. 30 is a good default for analysis tasks (20 was too tight with CLAUDE.md forcing web-search-first + .md writes).
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: ask-hermes
|
||||
description: Persistent peer Hermes agent for delegated work. `ask hermes <instructions>` delegates a task with full context via headless one-shot commands. Session persists across all turns in one Hermes session via --resume.
|
||||
version: 4.0.0
|
||||
version: 4.1.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux]
|
||||
@@ -32,6 +32,32 @@ Each `ask hermes` is a one-shot headless command that runs, returns a clean resp
|
||||
- Task needs my exact context state (use delegate_task on THIS profile)
|
||||
- You want to see the peer's chat directly (this is relay-only — full text in `hermes sessions list`)
|
||||
|
||||
## Critical Evaluation — You Are the Final Authority
|
||||
|
||||
**The peer's output is input, not a decision.** You are the engineer — you own the code, the system, and the final call. The peer is a separate agent with no memory of your conversation, no access to your context, and no ability to verify its own claims against your environment. Treat every finding as a hypothesis to be tested, not a conclusion to be implemented.
|
||||
|
||||
**Mandatory before acting on the peer's output:**
|
||||
|
||||
1. **Do your own reasoning first.** Before relaying the peer's answer, ask yourself: does this make sense? Is there a simpler way? Would this actually work in our environment? The peer may flag things as BROKEN that are actually correct, or claim PASS on things that are broken.
|
||||
|
||||
2. **Push back when the peer is wrong.** If the peer's analysis doesn't hold up, tell it so in a follow-up `--resume` turn. "Your claim about X is wrong because Y. Re-examine." The peer corrects itself when challenged; it doubles down when unchallenged. Do not silently discard a wrong finding — make the peer confront it.
|
||||
|
||||
3. **Do your own web searches to validate.** The peer has SearXNG but its interpretation may be wrong. For any claim about API behavior, version compatibility, or tool capabilities, run your own `mcp_searxng_searxng_web_search` and compare. The peer has been wrong about config keys, endpoint behavior, and system state in real sessions.
|
||||
|
||||
4. **Involve the user when there's genuine ambiguity.** If the peer and you disagree after a round of pushback, or if the right approach depends on constraints only the user knows, surface the disagreement with both positions stated clearly. Don't silently pick the peer's answer over your own judgment.
|
||||
|
||||
5. **Watch for overcomplication.** Peers, like any agent, can overengineer. If the peer proposes a multi-step protocol, a new abstraction layer, or a refactor that touches 10 files for a 1-line bug — ask: "Is there a simpler way?" The best fix is usually the simplest one that actually works.
|
||||
|
||||
## Peer Limitations
|
||||
|
||||
**The peer has no memory of your conversation.** The peer runs on the general profile with its own workspace, SOUL.md, and AGENTS.md. It knows nothing about what you've discussed, what files you've created, or what decisions you've made — unless you tell it via the prompt or it reads files by path. Every `ask hermes` must be self-contained with all relevant context.
|
||||
|
||||
**The peer may fabricate plausible-sounding analysis when it can't verify.** If the peer can't read a file, reach an endpoint, or confirm a claim, it may still produce confident-sounding analysis based on assumptions. When the peer prefaces a claim with "based on the description" or "assuming the config is..." without having actually read the source, treat it as unverified.
|
||||
|
||||
**The peer's environment is not your environment.** The peer has its own terminal session, its own working directory, and its own tool state. A command that works in the peer's session may fail in yours. Always test the peer's suggestions locally before deploying.
|
||||
|
||||
**Peer self-reports are not verified fact.** If the peer reports a file write, `read_file` the path to confirm. If it reports a test pass, re-run the test if the result matters. If it reports an endpoint returning 200, curl it yourself. The peer may claim "PASS" on things that are actually broken, or flag things as BROKEN that are correct. Concrete example: peer claimed `general` profile was BROKEN (bank_id leak), but the daemon log showed `general` was actively writing to `hermes-general` with successful retain + consolidation. The peer's logic was confused — it assumed the bank_id convention was violated when it wasn't. The daemon log is the source of truth, not the peer's reasoning.
|
||||
|
||||
## Command
|
||||
|
||||
### `ask hermes <instructions>`
|
||||
@@ -141,15 +167,14 @@ When the user says "ask hermes to fix X, then YOU validate," follow this exact s
|
||||
|
||||
1. **Profile flag required.** Use `-p general` — the sticky default may be dev, which would spawn a clone of this agent, not a peer. Always pin the profile explicitly.
|
||||
2. **--yolo is required.** The peer runs headless (one-shot, no TTY). Without `--yolo`, dangerous-command approval prompts fail closed (60s timeout → deny) and the peer can't complete tasks that trigger them. `--yolo` bypasses all approval prompts for the peer session only. The hardline blocklist (rm -rf /, fork bombs, mkfs on root, dd to block devices) still applies — no flag overrides that. This does NOT change the general profile's config; it only affects the peer session.
|
||||
3. **Peer skill_manage writes to the peer's profile, not yours.** If the peer patches a skill via skill_manage, it lands in the peer's profile (general), not the calling agent's profile (dev). The peer's `skill_manage` uses its own profile context. To sync changes back to dev, the calling agent must apply the same patches to its own copy. This is the #3 ask-hermes failure mode — assuming peer changes are global.
|
||||
4. **Do NOT use interactive REPL mode.** Running `hermes -p general` without `chat -q` (interactive REPL) was tried and failed — the TUI produces unreadable ANSI redraw noise when driven via `terminal(pty=true)`. The output is garbled and responses cannot be extracted. Always use headless `chat -q "..." -Q` instead.
|
||||
5. **Peer is headless.** Operator sees my relay, not the peer's chat. Full text in `hermes sessions list` if needed.
|
||||
6. **Peer self-reports are not verified fact.** Spot-check file writes, test passes before confirming to operator.
|
||||
7. **Peer's workspace is its own.** Files written to the peer's workspace are NOT in this agent's workspace. Give ABSOLUTE paths (e.g. `/home/n8n/workspace/dev/<path>`) if the peer should write to our workspace.
|
||||
8. **Blast radius.** Peer has ALL tools (terminal, web, MCP) — can rm, exfil, burn tokens. Bound with prompt-level constraints. State the blast radius to operator for sensitive tasks.
|
||||
9. **Turn budget is hard-capped.** `--max-turns 20` enforces it. If the peer hits 20 before finishing, it returns what it has — operator decides whether to continue.
|
||||
10. **Don't silently paraphrase.** Relay the peer's actual response. If long, chunk it. Call out unverified claims.
|
||||
11. **Peers can overstate findings.** A peer may correctly identify real gaps AND incorrectly flag things that aren't actually broken. In this session, the peer correctly identified the tool_executor.py ThreadPoolExecutor as the root cause of the Ctrl+C hang, but also flagged cli.py:9452 (account-usage fetch) as a second culprit — that one uses a `with` context manager that properly joins, so it's not a leak. Always verify each claim independently; don't assume all of a peer's findings are correct just because some are.
|
||||
2. **Do NOT use interactive REPL mode.** Running `hermes -p general` without `chat -q` (interactive REPL) was tried and failed — the TUI produces unreadable ANSI redraw noise when driven via `terminal(pty=true)`. The output is garbled and responses cannot be extracted. Always use headless `chat -q "..." -Q` instead.
|
||||
3. **Peer is headless.** Operator sees my relay, not the peer's chat. Full text in `hermes sessions list` if needed.
|
||||
4. **Peer self-reports are not verified fact.** Spot-check file writes, test passes before confirming to operator.
|
||||
5. **Peer's workspace is its own.** Files written to the peer's workspace are NOT in this agent's workspace. Give ABSOLUTE paths (e.g. `/home/n8n/workspace/dev/<path>`) if the peer should write to our workspace.
|
||||
6. **Blast radius.** Peer has ALL tools (terminal, web, MCP) — can rm, exfil, burn tokens. Bound with prompt-level constraints. State the blast radius to operator for sensitive tasks.
|
||||
7. **Turn budget is hard-capped.** `--max-turns 20` enforces it. If the peer hits 20 before finishing, it returns what it has — operator decides whether to continue.
|
||||
8. **Don't silently paraphrase.** Relay the peer's actual response. If long, chunk it. Call out unverified claims.
|
||||
10. **Peers can overstate findings.** A peer may correctly identify real gaps AND incorrectly flag things that aren't actually broken. In this session, the peer correctly identified the tool_executor.py ThreadPoolExecutor as the root cause of the Ctrl+C hang, but also flagged cli.py:9452 (account-usage fetch) as a second culprit — that one uses a `with` context manager that properly joins, so it's not a leak. Always verify each claim independently; don't assume all of a peer's findings are correct just because some are.
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
|
||||
Reference in New Issue
Block a user