diff --git a/create-plan/SKILL.md b/create-plan/SKILL.md index 6c01b83..50e1e72 100644 --- a/create-plan/SKILL.md +++ b/create-plan/SKILL.md @@ -1,21 +1,23 @@ --- name: create-plan -description: "Multi-agent plan-build pipeline: brief → research → assembly → Claude evidence-diff → URL re-verify → reconcile. Triggered by 'create plan for X', 'plan this', 'draft a plan for', 'build a plan for'." -version: 1.1.1 +description: "Multi-agent plan-build pipeline: brief → research → assembly → self-review → multi-model peer validation (Claude + general + dev) → URL re-verify → reconcile. Triggered by 'create plan for X', 'plan this', 'draft a plan for', 'build a plan for'." +version: 1.2.1 author: Hermes Agent license: MIT platforms: [linux] metadata: hermes: tags: [planning, plan, create-plan, orchestration, multi-agent, delegation] - related_skills: [ask-hermes, ask-claude, deep-research, writing-plans, subagent-driven-development] + related_skills: [ask-hermes, ask-claude, deep-research, writing-plans, subagent-driven-development, hermes-config-bulk-update] --- # create-plan — Multi-Agent Plan-Build Pipeline (Dispatcher) ## 1. Overview -The `create-plan` skill is a thin dispatcher that orchestrates a 7-phase plan-build pipeline (Phase 0 through Phase 6). The dispatcher runs Phase 0 — an interactive brief with the operator — then spawns a peer `ask-hermes` **curator** session that owns Phases 1 through 6 end-to-end. The curator dispatches `deep-research` for current evidence, runs assembly itself using `writing-plans` methodology, dispatches `ask-claude` for the adversarial evidence-diff, performs URL re-verification of the plan's specific cited claims, and reconciles the final plan. +The `create-plan` skill is a thin dispatcher that orchestrates a multi-phase plan-build pipeline (Phase 0 through Phase 6, with optional Phase 4b/4c peer rounds). The dispatcher runs Phase 0 — an interactive brief with the operator — then spawns a peer `ask-hermes` **curator** session that owns Phases 1 through 6 end-to-end. The curator dispatches `deep-research` for current evidence, runs assembly itself using `writing-plans` methodology, dispatches `ask-claude` for the first adversarial pass, dispatches `ask-hermes` (general profile) and optionally `ask-dev` (dev profile) for additional cross-model peer validation, performs URL re-verification of the plan's specific cited claims, and reconciles the final plan. + +**Why multi-model peer validation (Phases 4a/4b/4c):** Different models have different blind spots. Claude Opus on 10.0.0.28 catches structural issues and evidence-diff gaps. A peer on the general profile (minimax-m3 by default) catches implementation-level issues and applies the same "would the agent actually follow this?" check. A peer on the dev profile (deepseek-v4-pro) catches issues neither of the other two saw. The dev profile is intentionally a different model for this reason. In the operator's actual practice (4-round validation, July 2026), the dev peer caught a persistent-side-effect issue on a `hermes config set` call that neither Claude nor the general peer flagged — and was also wrong about a config-schema claim that a 5-second live test disproved. Both directions of value are real. See `references/multi-model-peer-validation.md` for the full pattern. **Output files:** `_plan.md` (actionable, validated) and `_research.md` (evidence, with appended re-verify log). The 600-turn ceiling is a safety net, not a budget — the curator is allowed to take as long as it needs. Correctness > speed. The user resumes the **curator** (not the dispatcher) to iterate, add constraints, or request re-validation. Per `ask-hermes` Turn Budget section (and §5.3 below), the create-plan curator always uses `--max-turns 600 --yolo`; the default 20 is not appropriate for evidence-based plan builds. @@ -73,8 +75,8 @@ The `create-plan` skill is a thin dispatcher that orchestrates a 7-phase plan-bu └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ -│ Phase 4: Adversarial validation [curator →] │ -│ - curator dispatches ask-claude (Opus) │ +│ Phase 4a: Adversarial validation (Claude) [curator →] │ +│ - curator dispatches ask-claude (Opus on 10.0.0.28) │ │ - curator extracts the structured-claims/citations section │ │ of _research.md (~10KB cap) and inlines BOTH the │ │ full _plan.md AND the extracted claims │ @@ -85,6 +87,28 @@ The `create-plan` skill is a thin dispatcher that orchestrates a 7-phase plan-bu └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ +│ Phase 4b: Peer validation (general model) [curator →] │ +│ - curator dispatches ask-hermes (general profile) for an │ +│ independent validation from a different model │ +│ - general peer reads the plan + researches live, can verify │ +│ things Claude structurally cannot (paths exist, config │ +│ accepted, distribution patterns) │ +│ - Output: validation report from general peer │ +│ - Default: 1 round. Skip only if Phase 4a found zero findings │ +│ AND the plan is short (<100 lines). │ +└─────────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────────┐ +│ Phase 4c: Peer validation (dev model, optional) [curator →] │ +│ - curator dispatches ask-dev (dev profile) for a third-model │ +│ blindspot check │ +│ - dev peer is a different model (deepseek-v4-pro vs. │ +│ minimax-m3) and catches different issues than the first two │ +│ - Only run if Phase 4a or 4b found release-blockers. │ +│ - Output: validation report from dev peer │ +└─────────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────────┐ │ Phase 5: URL re-verification [curator] │ │ - curator re-fetches URLs backing the SPECIFIC version │ │ numbers, commands, and setup steps the PLAN cites │ @@ -96,8 +120,10 @@ The `create-plan` skill is a thin dispatcher that orchestrates a 7-phase plan-bu ↓ ┌─────────────────────────────────────────────────────────────────┐ │ Phase 6: Final reconcile [curator] │ -│ - curator reads Claude's report + re-verify log │ -│ - applies agreed changes, ignores disagreements │ +│ - curator reads ALL validation reports (4a, 4b, 4c if run) │ +│ + re-verify log │ +│ - applies agreed changes from each, runs disagreement scan │ +│ on each │ │ - Output: _plan.md final │ │ - reports: curator_session_id for resume │ └─────────────────────────────────────────────────────────────────┘ @@ -181,7 +207,7 @@ The dispatcher surfaces up to **10** clarifying questions in **priority order**. ``` === FROZEN BRIEF BEGIN === Topic: -Target save dir: ~/workspace//plans/ +Target save dir: ~/workspace/plans/ Q1 (Scope): Q2 (Constraints): @@ -260,7 +286,7 @@ session_id: === CREATE-PLAN DISPATCH COMPLETE === curator_session_id: frozen_brief_topic: -target_save_dir: ~/workspace//plans/ +target_save_dir: ~/workspace/plans/ mode_flags: The curator now owns Phases 1–6 (research → assemble → self-review → Claude evidence-diff → URL re-verify → reconcile). @@ -298,7 +324,8 @@ The curator prompt embeds the two dispatch-template references inline; the dispa - **Resume the curator for:** continuing the brief, adding sections, requesting re-validation, asking the curator to expand a phase. - **Never resume across topics.** Same plan = resume, different plan = fresh curator session. - **deep-research sessions are NOT resumed by the curator.** Curator captures the `_research.md` output and moves on; the research session expires naturally. -- **ask-claude sessions are NOT resumed.** Claude is one-shot for the validation pass. Re-validation = fresh Claude session. +- **ask-claude sessions are NOT resumed.** Claude is one-shot for the evidence-diff pass. Re-validation = fresh Claude session. +- **ask-hermes and ask-dev peer sessions (Phase 4b/4c) are NOT resumed either.** Each is a one-shot validation round. Re-validation = fresh peer session, possibly on a different profile. - **Dispatcher is one-shot.** The dispatcher does NOT poll. Operator resumes the curator for follow-up. **Pre-build check — verify line-1 capture (per `~/workspace/general/plan_skill_plan.md` §7, pitfall #10):** @@ -309,7 +336,7 @@ Before declaring the skill buildable, the dispatcher must empirically confirm th - **`_plan.md`** — the actionable plan. Pinned filename. Date in frontmatter, not filename. Single writer: the curator. - **`_research.md`** — the research evidence. Pinned filename. Curator is the **sole writer**. Phase 5 re-verify log is appended in-place. -- **Default save location:** `~/workspace//plans/`. Override with explicit path in the brief. +- **Default save location:** `~/workspace/plans/`. Override with explicit path in the brief. - **Phase 5 validation log:** appended to `_research.md` (not a separate file). Contains: re-fetched URL, what was confirmed, any version/command drift found. **Same-topic clobber guard (per `~/workspace/general/plan_skill_plan.md` §7.5):** The curator checks for an existing `_plan.md` in the target dir. If found, the curator renames the existing file to `_plan..bak.md` and saves the new plan to the canonical `_plan.md`. The operator can recover the prior version from the .bak.md file. Prevents silent clobber under `--yolo`. @@ -329,6 +356,9 @@ Before declaring the skill buildable, the dispatcher must empirically confirm th 11. **Phase 4 (Claude evidence-diff) only sees the structured-claims extract, not the full research.md.** The curator's self-review (Phase 3) is the safety net for findings that fall outside the extract. 12. **Tool names in Phase 5 must match the actual tool registry.** The curator framing prompt's Phase 5.2 references `web_extract` and `mcp__playwright__browser_navigate` (double underscore). If these change in a future Hermes version, the curator will error out on URL re-verification. Verify tool names against the current tool registry before shipping a skill update. 13. **The turn-ceiling save must be proactive, not reactive.** The curator cannot "stop and write at turn 600" — the session terminates at the ceiling. The curator framing prompt instructs proactive save at ~590. If the turn budget changes, update both the ceiling value and the proactive-save threshold. +14. **When a plan creates a new skill, verify the real distribution pattern before proposing it.** Hermes skills ship as **real per-profile file copies, not symlinks.** Each profile has its own `~/.hermes/profiles/

/skills///SKILL.md` (separate inodes). The dispatcher MUST verify this empirically (`stat -c %i` on an existing shared skill like `searxng-smart-search` across profiles) before writing a plan that says "symlink the new skill to all profiles" — the assumption is wrong and the plan will produce a broken distribution. Reference: `hermes-config-bulk-update/references/skill-replication-across-profiles.md`. Plan template: "write the SKILL.md once to a canonical source, then `cp` to each target profile's `skills///` directory." +15. **Live `hermes config set` is authoritative; web-searched config docs are not.** When validating whether a Hermes config value, flag, or schema is valid, ALWAYS demand a live test (e.g., `hermes -p config set ` and check exit code + output) rather than relying on web-searched documentation. The mintlify / hermes-agent.nousresearch.com docs can be stale, refer to a different schema, or omit valid values. Real example (July 2026): a peer agent claimed `reasoning_effort: max` was not a valid value based on web docs, but `hermes config set agent.reasoning_effort max` accepted it with exit 0 — `max` is in fact the current value of both the general and research profile configs. **Rule:** if a peer or Claude asserts "X is invalid config" without showing a live test output, push back and demand the test. Five seconds of `hermes config set` beats five minutes of arguing about docs. +16. **Multi-model peer validation is the operator's de facto practice; do not skip Phase 4b/4c reflexively.** For non-trivial plans (anything crossing 3+ files, architectural changes, anything that touches operator-standing rules), Phase 4a (Claude) alone has repeatedly missed things the general peer and dev peer caught. Default: run 4a + 4b for every non-trivial plan. Run 4c only if 4a or 4b found release-blockers. The cost is ~2-5 minutes per extra round; the leverage is real (cross-model blindspots are NOT a theoretical concern). See `references/multi-model-peer-validation.md`. ## 9. Verification Checklist (from plan §7, verbatim) @@ -359,7 +389,9 @@ Before declaring the skill buildable, the dispatcher must empirically confirm th - `references/curator-framing-prompt.md` — the actual prompt the dispatcher sends to the curator (Phases 1-6) - `references/deep-research-dispatch-template.md` — the Phase 1 research dispatch template - `references/ask-claude-validation-template.md` — the Phase 4 evidence-diff prompt template +- `references/multi-model-peer-validation.md` — the Phase 4a/4b/4c multi-model peer validation pattern (why and when) - `writing-plans` skill — Phase 2 methodology - `subagent-driven-development` skill — for executing the plan after delivery - `ask-hermes` / `ask-claude` / `deep-research` skills — peer dispatch patterns - `references/multi-round-validation-pattern.md` (in subagent-driven-development) — for the multi-round validation protocol used in plan-build +- `hermes-config-bulk-update/references/skill-replication-across-profiles.md` — when a plan creates a new skill, the real per-profile distribution pattern (real file copies, not symlinks) diff --git a/create-plan/references/curator-framing-prompt.md b/create-plan/references/curator-framing-prompt.md index 1139659..52c4ca6 100644 --- a/create-plan/references/curator-framing-prompt.md +++ b/create-plan/references/curator-framing-prompt.md @@ -28,7 +28,7 @@ The dispatcher has already asked the operator up to 10 clarifying questions in p ``` === FROZEN BRIEF BEGIN === Topic: -Target save dir: ~/workspace//plans/ +Target save dir: ~/workspace/plans/ Q1 (Scope): Q2 (Constraints): diff --git a/create-plan/references/deep-research-dispatch-template.md b/create-plan/references/deep-research-dispatch-template.md index 3fb0ba2..037c42f 100644 --- a/create-plan/references/deep-research-dispatch-template.md +++ b/create-plan/references/deep-research-dispatch-template.md @@ -77,7 +77,7 @@ Exhaustive deep research: daily workspace backup cron job in 2026. A reliable, l **Spot-check rule:** the research agent's self-reports are not verified fact. If it claims a specific finding, spot-check the source URL. If it claims a file write, `read_file` the path to confirm. -The curator captures both and writes `_research.md` to the target save dir (`~/workspace//plans/`). The structure of `_research.md` is defined in `curator-framing-prompt.md` Step 1.4 — this template is only the dispatch step. +The curator captures both and writes `_research.md` to the target save dir (`~/workspace/plans/`). The structure of `_research.md` is defined in `curator-framing-prompt.md` Step 1.4 — this template is only the dispatch step. --- diff --git a/create-plan/references/multi-model-peer-validation.md b/create-plan/references/multi-model-peer-validation.md new file mode 100644 index 0000000..9d564ff --- /dev/null +++ b/create-plan/references/multi-model-peer-validation.md @@ -0,0 +1,139 @@ +# Multi-Model Peer Validation — `create-plan` Phases 4a/4b/4c + +> **This file captures the multi-model peer validation pattern observed in the operator's actual 4-round validation practice (July 2026).** The single-Claude pass that the original `create-plan` pipeline documented (Phase 4) is now Phase 4a; Phases 4b (general peer) and 4c (dev peer) are the additional cross-model rounds that catch different classes of issues. + +## Why three rounds with three different models + +Different models have different blind spots. **The same question, asked of three different models, produces three different non-overlapping sets of findings.** Empirically, the operator's July 2026 plan validation runs (4 different plans) showed: + +- **Claude Opus on 10.0.0.28** catches: structural issues, evidence-diff gaps (where the plan asserts something the research doesn't support), diagram/spec desync, missing TDD discipline, "this won't actually work" architectural concerns. Cannot verify URL currency, file existence, or live config state. +- **General peer (minimax-m3 by default) on the general profile** catches: implementation-level issues (will the agent actually follow these instructions?), trigger-phrase collisions with existing skills, subtle logic gaps in the workflow, sloppy wording that would mislead a future reader. Can run `read_file`, `terminal`, and SearXNG to verify state. +- **Dev peer (deepseek-v4-pro by default) on the dev profile** catches: things neither of the above saw. Specifically: persistent side-effects of dispatch-time config writes, profile-distribution concerns, and issues that come from a different reasoning chain entirely. The dev peer is intentionally a different model (deepseek vs. minimax) to maximize the cross-model delta. + +The cost is real: ~2-5 minutes per extra round, plus a few thousand tokens. The leverage is also real. **The July 2026 better-search plan validation had dev peer catch a `hermes config set` persistent-side-effect concern that nobody else flagged**, and was simultaneously wrong about a config-schema claim that a 5-second live test disproved. Both directions of value are real, and neither would have happened with a single Claude pass. + +## The pattern + +### Phase 4a — Claude evidence-diff (always run) + +The original Phase 4. Claude on 10.0.0.28 receives the full plan + the structured-claims section of research.md (~10KB cap). Returns release-blockers / medium / low findings. One-shot; do not resume. + +**Skip condition:** none. Always run on non-trivial plans. Trivial plans (single config edit) don't need `create-plan` at all. + +### Phase 4b — General peer validation (default: run) + +A fresh `ask-hermes` session on the general profile. Different model (minimax-m3 by default), different blind spots. The peer reads the plan from disk + does live web research with `mcp_searxng_searxng_web_search` to verify external claims. Can verify file paths, config keys, and trigger-phrase collisions (which Claude cannot, since Claude has no filesystem or live access to this machine). + +**Default: 1 round.** Skip only if Phase 4a found zero findings AND the plan is short (<100 lines). + +**Skip condition:** Phase 4a had zero findings AND plan length <100 lines. Otherwise run. + +### Phase 4c — Dev peer validation (run only on release-blockers) + +A fresh `ask-dev` session on the dev profile. Different model again (deepseek-v4-pro by default). Same workflow as Phase 4b but with a different model and a different workspace. + +**Skip condition:** Phase 4a AND Phase 4b both had zero release-blockers. Otherwise run. (The cost of a false-negative here — a release-blocker that gets through to implementation — is much higher than the cost of a redundant 4th round.) + +### Phase 6 — Reconcile across all rounds + +The curator reads ALL validation reports (4a + 4b + 4c if run) + the Phase 5 URL re-verify log. Runs the disagreement scan on EACH report separately (peer findings can disagree with each other, not just with the curator). Applies agreed changes from each, ignores disagreements, and produces the final plan with a recap of which findings came from which round. + +## Validator prompt template (for peer rounds) + +The peer prompt should be the same shape as the Claude prompt from `references/ask-claude-validation-template.md`, with two adjustments: + +1. **Replace the evidence-diff framing with an implementation-readiness framing.** The peer is checking: "would the agent actually follow these instructions and not break in turn 40?" Not: "does the plan contradict the research evidence?" (That's Claude's job.) +2. **Force live `mcp_searxng_searxng_web_search` use** for any external claim. The peer has SearXNG MCP and must use it. Same mandate as the ask-hermes / ask-dev skills. + +```markdown +You are validating an implementation plan. Read it in full first. + +CRITICAL CONSTRAINT: Do NOT propose new features, new architecture, or new complexity. Find flaws in what's already proposed. If you find a problem, say what's wrong and suggest the SIMPLEST fix. + +Use `mcp_searxng_searxng_web_search` for EVERY external claim and cite the source URL. Do not rely on parametric knowledge or reason about what a command 'would show.' Run the real command, run the real search. + +VALIDATE (each must be checkable from plan + skills text + web search): + +1. Implementation-readiness — will the agent actually follow these instructions? Are any tasks ambiguous, contradictory, or missing context? +2. Trigger-phrase collision check — verify against existing skill description fields on this machine. +3. Profile distribution reality check — does the plan's distribution pattern match real Hermes behavior? +4. Internal consistency — do the steps in §X actually produce the outputs claimed in §Y? +5. Bite-size of implementation tasks — are all tasks 2-5 minutes of focused work? +6. Verification completeness — does the test/verification list actually prove the system works? +7. Output contract — are file paths, names, and structures consistent across all sections? +8. Failure modes — anything missing from the listed failure scenarios? +9. Overengineering check — operator explicitly said "do not overcomplicate." Is anything more complex than it needs to be? +10. Live config validation — for any claim about Hermes config schema, flags, or values, run a real `hermes config set` and verify. + +Report format: +- Release-blockers (must fix before building) +- Medium (should fix) +- Low (nice to have) +- Pass-criteria met / not met (one row per validation dimension) +- **What the previous validators got wrong** (if any) + +Then: give 2-3 concrete suggestions for how to make this plan better. Prefer cutting to adding. + +==== ARTIFACT BEGINS ==== + + + +==== ARTIFACT ENDS ==== +``` + +## Disagreement scan across multiple validators + +When Phases 4a + 4b + 4c all return, the curator runs the disagreement scan on each, BUT ALSO runs a cross-validator scan: + +- **Validator A says X is broken, Validator B says X is fine.** Push back to the operator or run a live test to resolve. Don't apply one validator's framing wholesale. +- **Validator A and B both flag the same item.** High confidence — apply the agreed fix. +- **Validator A flags something the curator's self-review (Phase 3) missed.** That's exactly the value of an external pass. Apply unless the curator has a documented reason to skip. +- **Validator A makes a false claim** (caught by live test in round N+1). Push back via the next peer round; do not silently drop. + +## The cross-model delta: when to use it + +**Always use multi-model for:** +- Plan creates a new skill (the distribution pattern is wrong 50% of the time on first write) +- Plan touches multiple profiles, profile config, or skill distribution +- Plan involves long-running autonomous agents (50+ turns) — different models surface different drift patterns +- Plan modifies operator-standing rules or cross-cutting concerns + +**Single-Claude is fine for:** +- Single-file changes, well-understood territory +- Plans where the operator said "keep it simple" +- Plans under 100 lines + +**Skip peer rounds entirely for:** +- Trivial plans that don't need `create-plan` at all (use `writing-plans`) + +## Live config validation: the disagreement-canary technique + +When a peer (Claude or otherwise) asserts that a Hermes config value, flag, or schema is invalid, **the canary is a 5-second live test**: + +```bash +hermes -p config set +echo "exit: $?" +grep "" ~/.hermes/profiles//config.yaml +``` + +If exit is 0 and the value is set, the assertion is wrong. If exit is non-zero or the value is rejected, the assertion is right. Don't argue about docs. Don't argue about parametric knowledge. Run the test. + +**Empirical example (July 2026):** A peer agent claimed `reasoning_effort: max` was not a valid value based on web-searched mintlify docs. The live test `hermes -p research config set agent.reasoning_effort max` returned `✓ Set agent.reasoning_effort = max` with exit 0. The peer was wrong. The fix was one command, the lesson is one rule: **live `hermes config set` beats web docs**. + +## Lessons that have survived the cross-model delta + +These findings appeared in 2+ of 4 cross-model validation runs (July 2026): + +1. **Skill distribution: real per-profile file copies, NOT symlinks.** Easy to assume; empirically wrong on first write. (See pitfall #14 in `create-plan` SKILL.md.) +2. **No `--resume` for skills with per-dispatch turn/loop caps.** `--resume` re-enters a session with fresh `--max-turns` and the agent can run the cap-loop again, breaking the cap guarantee. (Cross-model pattern; multiple plans had this issue.) +3. **Disagreement-scan output is not "agreement with self-review" — it's input to a re-evaluation.** Don't apply peer findings wholesale; classify each item. +4. **Trigger phrases for new skills should be explicit.** "Look into", "find out", "search again" collide with normal English on dev/code profiles. (Caught by general peer; Claude did not flag.) +5. **A plan that creates a new skill needs to specify the cross-profile distribution pattern explicitly.** General peer caught this when Claude didn't. The ask-hermes precedent (16 real files) is the right reference; searxng-smart-search is NOT (12 of 16, lives in mcp/ not autonomous-ai-agents/). +6. **Live config validation wins disputes.** When in doubt, run the test. + +## See also + +- `references/ask-claude-validation-template.md` — the Phase 4a Claude prompt +- `references/curator-framing-prompt.md` — full Phase 4a-c mechanics +- `subagent-driven-development/references/multi-round-validation-pattern.md` — the original multi-round pattern (single-validator) +- `ask-hermes` and `ask-dev` skills — the peer dispatch patterns used in 4b/4c diff --git a/create-plan/references/structured-brief-10q.md b/create-plan/references/structured-brief-10q.md index 332f3e9..74d340a 100644 --- a/create-plan/references/structured-brief-10q.md +++ b/create-plan/references/structured-brief-10q.md @@ -38,7 +38,7 @@ After collection, pass the brief to the peer in this exact format: ``` === FROZEN BRIEF BEGIN === Topic: -Target save dir: ~/workspace//plans/ +Target save dir: ~/workspace/plans/ Q1 (Scope): Q2 (Constraints):