hermes-skills/create-plan/references/multi-model-peer-validation.md

# Multi-Model Peer Validation — `create-plan` Phases 4a/4b/4c

> **This file captures the multi-model peer validation pattern observed in the operator's actual 4-round validation practice (July 2026).** The single-Claude pass that the original `create-plan` pipeline documented (Phase 4) is now Phase 4a; Phases 4b (general peer) and 4c (dev peer) are the additional cross-model rounds that catch different classes of issues.

## Why three rounds with three different models

Different models have different blind spots. **The same question, asked of three different models, produces three different non-overlapping sets of findings.** Empirically, the operator's July 2026 plan validation runs (4 different plans) showed:

- **Claude Opus on 10.0.0.28** catches: structural issues, evidence-diff gaps (where the plan asserts something the research doesn't support), diagram/spec desync, missing TDD discipline, "this won't actually work" architectural concerns. Cannot verify URL currency, file existence, or live config state.
- **General peer (minimax-m3 by default) on the general profile** catches: implementation-level issues (will the agent actually follow these instructions?), trigger-phrase collisions with existing skills, subtle logic gaps in the workflow, sloppy wording that would mislead a future reader. Can run `read_file`, `terminal`, and SearXNG to verify state.
- **Dev peer (deepseek-v4-pro by default) on the dev profile** catches: things neither of the above saw. Specifically: persistent side-effects of dispatch-time config writes, profile-distribution concerns, and issues that come from a different reasoning chain entirely. The dev peer is intentionally a different model (deepseek vs. minimax) to maximize the cross-model delta.

The cost is real: ~2-5 minutes per extra round, plus a few thousand tokens. The leverage is also real. **The July 2026 better-search plan validation had dev peer catch a `hermes config set` persistent-side-effect concern that nobody else flagged**, and was simultaneously wrong about a config-schema claim that a 5-second live test disproved. Both directions of value are real, and neither would have happened with a single Claude pass.

## The pattern

### Phase 4a — Claude evidence-diff (always run)

The original Phase 4. Claude on 10.0.0.28 receives the full plan + the structured-claims section of research.md (~10KB cap). Returns release-blockers / medium / low findings. One-shot; do not resume.

**Skip condition:** none. Always run on non-trivial plans. Trivial plans (single config edit) don't need `create-plan` at all.

### Phase 4b — General peer validation (default: run)

A fresh `ask-hermes` session on the general profile. Different model (minimax-m3 by default), different blind spots. The peer reads the plan from disk + does live web research with `mcp_searxng_searxng_web_search` to verify external claims. Can verify file paths, config keys, and trigger-phrase collisions (which Claude cannot, since Claude has no filesystem or live access to this machine).

**Default: 1 round.** Skip only if Phase 4a found zero findings AND the plan is short (<100 lines).

**Skip condition:** Phase 4a had zero findings AND plan length <100 lines. Otherwise run.

### Phase 4c — Dev peer validation (run only on release-blockers)

A fresh `ask-dev` session on the dev profile. Different model again (deepseek-v4-pro by default). Same workflow as Phase 4b but with a different model and a different workspace.

**Skip condition:** Phase 4a AND Phase 4b both had zero release-blockers. Otherwise run. (The cost of a false-negative here — a release-blocker that gets through to implementation — is much higher than the cost of a redundant 4th round.)

### Phase 6 — Reconcile across all rounds

The curator reads ALL validation reports (4a + 4b + 4c if run) + the Phase 5 URL re-verify log. Runs the disagreement scan on EACH report separately (peer findings can disagree with each other, not just with the curator). Applies agreed changes from each, ignores disagreements, and produces the final plan with a recap of which findings came from which round.

## Validator prompt template (for peer rounds)

The peer prompt should be the same shape as the Claude prompt from `references/ask-claude-validation-template.md`, with two adjustments:

1. **Replace the evidence-diff framing with an implementation-readiness framing.** The peer is checking: "would the agent actually follow these instructions and not break in turn 40?" Not: "does the plan contradict the research evidence?" (That's Claude's job.)
2. **Force live `mcp_searxng_searxng_web_search` use** for any external claim. The peer has SearXNG MCP and must use it. Same mandate as the ask-hermes / ask-dev skills.

```markdown
You are validating an implementation plan. Read it in full first.

CRITICAL CONSTRAINT: Do NOT propose new features, new architecture, or new complexity. Find flaws in what's already proposed. If you find a problem, say what's wrong and suggest the SIMPLEST fix.

Use `mcp_searxng_searxng_web_search` for EVERY external claim and cite the source URL. Do not rely on parametric knowledge or reason about what a command 'would show.' Run the real command, run the real search.

VALIDATE (each must be checkable from plan + skills text + web search):

1. Implementation-readiness — will the agent actually follow these instructions? Are any tasks ambiguous, contradictory, or missing context?
2. Trigger-phrase collision check — verify against existing skill description fields on this machine.
3. Profile distribution reality check — does the plan's distribution pattern match real Hermes behavior?
4. Internal consistency — do the steps in §X actually produce the outputs claimed in §Y?
5. Bite-size of implementation tasks — are all tasks 2-5 minutes of focused work?
6. Verification completeness — does the test/verification list actually prove the system works?
7. Output contract — are file paths, names, and structures consistent across all sections?
8. Failure modes — anything missing from the listed failure scenarios?
9. Overengineering check — operator explicitly said "do not overcomplicate." Is anything more complex than it needs to be?
10. Live config validation — for any claim about Hermes config schema, flags, or values, run a real `hermes config set` and verify.

Report format:
- Release-blockers (must fix before building)
- Medium (should fix)
- Low (nice to have)
- Pass-criteria met / not met (one row per validation dimension)
- **What the previous validators got wrong** (if any)

Then: give 2-3 concrete suggestions for how to make this plan better. Prefer cutting to adding.

==== ARTIFACT BEGINS ====

<full plan here>

==== ARTIFACT ENDS ====
```

## Disagreement scan across multiple validators

When Phases 4a + 4b + 4c all return, the curator runs the disagreement scan on each, BUT ALSO runs a cross-validator scan:

- **Validator A says X is broken, Validator B says X is fine.** Push back to the operator or run a live test to resolve. Don't apply one validator's framing wholesale.
- **Validator A and B both flag the same item.** High confidence — apply the agreed fix.
- **Validator A flags something the curator's self-review (Phase 3) missed.** That's exactly the value of an external pass. Apply unless the curator has a documented reason to skip.
- **Validator A makes a false claim** (caught by live test in round N+1). Push back via the next peer round; do not silently drop.

## The cross-model delta: when to use it

**Always use multi-model for:**
- Plan creates a new skill (the distribution pattern is wrong 50% of the time on first write)
- Plan touches multiple profiles, profile config, or skill distribution
- Plan involves long-running autonomous agents (50+ turns) — different models surface different drift patterns
- Plan modifies operator-standing rules or cross-cutting concerns

**Single-Claude is fine for:**
- Single-file changes, well-understood territory
- Plans where the operator said "keep it simple"
- Plans under 100 lines

**Skip peer rounds entirely for:**
- Trivial plans that don't need `create-plan` at all (use `writing-plans`)

## Live config validation: the disagreement-canary technique

When a peer (Claude or otherwise) asserts that a Hermes config value, flag, or schema is invalid, **the canary is a 5-second live test**:

```bash
hermes -p <profile> config set <key> <value>
echo "exit: $?"
grep "<key>" ~/.hermes/profiles/<profile>/config.yaml
```

If exit is 0 and the value is set, the assertion is wrong. If exit is non-zero or the value is rejected, the assertion is right. Don't argue about docs. Don't argue about parametric knowledge. Run the test.

**Empirical example (July 2026):** A peer agent claimed `reasoning_effort: max` was not a valid value based on web-searched mintlify docs. The live test `hermes -p research config set agent.reasoning_effort max` returned `✓ Set agent.reasoning_effort = max` with exit 0. The peer was wrong. The fix was one command, the lesson is one rule: **live `hermes config set` beats web docs**.

## Lessons that have survived the cross-model delta

These findings appeared in 2+ of 4 cross-model validation runs (July 2026):

1. **Skill distribution: real per-profile file copies, NOT symlinks.** Easy to assume; empirically wrong on first write. (See pitfall #14 in `create-plan` SKILL.md.)
2. **No `--resume` for skills with per-dispatch turn/loop caps.** `--resume` re-enters a session with fresh `--max-turns` and the agent can run the cap-loop again, breaking the cap guarantee. (Cross-model pattern; multiple plans had this issue.)
3. **Disagreement-scan output is not "agreement with self-review" — it's input to a re-evaluation.** Don't apply peer findings wholesale; classify each item.
4. **Trigger phrases for new skills should be explicit.** "Look into", "find out", "search again" collide with normal English on dev/code profiles. (Caught by general peer; Claude did not flag.)
5. **A plan that creates a new skill needs to specify the cross-profile distribution pattern explicitly.** General peer caught this when Claude didn't. The ask-hermes precedent (16 real files) is the right reference; searxng-smart-search is NOT (12 of 16, lives in mcp/ not autonomous-ai-agents/).
6. **Live config validation wins disputes.** When in doubt, run the test.

## See also

- `references/ask-claude-validation-template.md` — the Phase 4a Claude prompt
- `references/curator-framing-prompt.md` — full Phase 4a-c mechanics
- `subagent-driven-development/references/multi-round-validation-pattern.md` — the original multi-round pattern (single-validator)
- `ask-hermes` and `ask-dev` skills — the peer dispatch patterns used in 4b/4c