deep-research + deep-web-research: v2.0.0 — six-move flow, domain-aware tool selection, re-strategize checkpoints, tag-based saturation, 600-turn ceiling

deep-research/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: deep-research
-description: Dispatch exhaustive deep web research to the research profile. Triggered by "deep research", "ask web", or "research this". The research profile runs with the deep-web-research skill loaded — five-move flow, external ledger, mechanical saturation, disconfirmation, condensation from disk.
+description: Dispatch exhaustive deep web research to the research profile. Triggered by "deep research", "ask web", or "research this". The research profile runs with the deep-web-research skill loaded — six-move flow, external ledger, mechanical saturation, disconfirmation, condensation from disk.
-version: 1.0.1
+version: 2.0.0
 author: Hermes Agent
 license: MIT
 platforms: [linux]
@@ -17,12 +17,14 @@ metadata:
 
 Dispatches a research question to the research profile, which runs with the `deep-web-research` skill loaded. The research agent does exhaustive, trail-following research — any tool, any trail, no rush — then condenses everything into a tight, concrete, evidence-backed answer. This agent just relays the result.
 
+See `references/design-rationale.md` for the v2.0 architecture decisions and Claude validation findings.
+
 **Trigger phrases:** "deep research", "ask web", "research this", "deep dive on", "exhaustive research on"
 
 ## Command
 
 ```
-hermes -p research -s deep-web-research chat -q "<question>" -Q --max-turns 200 --yolo
+hermes -p research -s deep-web-research chat -q "<question>" -Q --max-turns 600 --yolo
 ```
 
 - Output line 1: `session_id: <id>` — capture this
@@ -31,20 +33,21 @@ hermes -p research -s deep-web-research chat -q "<question>" -Q --max-turns 200
 
 **Subsequent asks (resume the session):**
 ```
-hermes -p research -s deep-web-research chat --resume <session_id> -q "<follow-up>" -Q --max-turns 200 --yolo
+hermes -p research -s deep-web-research chat --resume <session_id> -q "<follow-up>" -Q --max-turns 600 --yolo
 ```
 
 ## What the Research Agent Does
 
 The `deep-web-research` skill on the research profile enforces:
 
+0. **Analyze & Strategize** — analyze the question, select tools from the arsenal, write strategy to ledger
 1. **Decompose** — break the question into sub-questions with time classifications
 2. **Landscape Pass** — shallow sweep, identify key sources and terms
 3. **Deep-Dive** — 3+ search angles per sub-question, full-page reads, citation trails, depth-first
 4. **Disconfirmation** — actively hunt for contradiction and outdated claims
 5. **Condensation** — read the external findings ledger from disk, synthesize into a concrete answer
 
-The research agent writes all findings to `/tmp/research-<sid>.md` (external ledger) and uses a phase gate file to enforce completion before condensing. Mechanical saturation checks (`grep -c`) prevent endless searching.
+The research agent writes all findings to `/tmp/research-<sid>.md` (external ledger) and uses a phase gate file to enforce completion before condensing. Mechanical saturation checks (`grep -c`) prevent endless searching. Re-strategize checkpoints after Move 2 and every ~10 findings during Move 3 enable mid-research pivots.
 
 ## When to Use
 
@@ -80,33 +83,20 @@ The research agent self-reports are not verified fact. If it claims a file write
 1. **Profile flag required.** Always use `-p research`. The sticky default may be general.
 2. **Skill flag required.** Always use `-s deep-web-research`. Without it, the research agent runs in normal mode without the methodology.
 3. **--yolo is required.** The research agent runs headless. Without it, approval prompts fail closed.
-4. **200 turns is the ceiling, not the target.** The research agent should condense well before 200. Hitting the ceiling means it failed to condense.
+4. **600 turns is the ceiling, not the target.** The research agent should condense well before 600. Hitting the ceiling means it failed to condense.
 5. **Don't re-condense.** The research agent already produced a condensed answer. Relay it, don't summarize it further.
 6. **Don't do the research yourself.** If the user triggers deep research, dispatch it. Don't run a few searches and call it done. This is the #1 failure mode: the agent runs 2-3 `mcp_searxng_searxng_web_search` calls, gets empty results, and gives up. That's not deep research — that's a casual lookup. If you catch yourself typing `mcp_searxng_searxng_web_search` for a deep research request, STOP. You're doing it wrong. Dispatch to the research profile.
 7. **The methodology skill lives on the research profile.** `deep-web-research` is at `~/.hermes/profiles/research/skills/research/deep-web-research/SKILL.md`. It does NOT exist on the general profile. Don't search for it here — it won't be found. The general profile only has this delegation skill.
-8. **`reasoning_effort: xhigh` breaks `glm-5.2:cloud`.** The research profile's default model (`glm-5.2:cloud` via `custom:ollama`) rejects `xhigh` — it only accepts `high`, `medium`, `low`, `max`, or `none`. If the dispatch fails with "HTTP 400: invalid reasoning value: 'xhigh'", check `~/.hermes/profiles/research/config.yaml` → `agent.reasoning_effort`. Set it to `max` (the highest level the model actually supports). Do NOT downgrade to `high` without asking the user — they explicitly want maximum reasoning effort. If the model changes to one that supports `xhigh`, revert to `xhigh`.
 
 ## Pre-Dispatch Reasoning Check (MANDATORY)
 
-Before EVERY deep research dispatch, verify the research profile's reasoning_effort is set to the maximum the model supports. This is a hard gate — do not dispatch without checking.
+Before EVERY deep research dispatch, set the research profile's reasoning_effort to `max`. This is a hard gate — do not dispatch without it.
 
 ```
-# 1. Read the research profile's current model and reasoning_effort
-grep -E "default:|reasoning_effort:" ~/.hermes/profiles/research/config.yaml
-
-# 2. If reasoning_effort is NOT 'max' (or 'xhigh' for models that support it):
-#    - If model is glm-5.2:cloud → set to 'max' (xhigh rejected)
-#    - If model is deepseek-v4-pro:cloud → set to 'max' (xhigh rejected)
-#    - If model is minimax-m3:cloud → set to 'xhigh' (supports it)
-#    - If unsure what the model supports → set to 'max' (safe default)
-
-# 3. Apply the fix if needed:
 hermes -p research config set agent.reasoning_effort max
-# OR for models that support xhigh:
-hermes -p research config set agent.reasoning_effort xhigh
 ```
 
-**Rule:** Always dispatch with the highest reasoning effort the model actually supports. Never silently downgrade. If the model changed and you don't know what it supports, test with `max` first (universally safer than `xhigh`).
+`max` is the highest effort setting universally supported by all models. No model-specific table needed. If the model changes to one that supports `xhigh`, the user will set it in config — the pre-dispatch check just ensures it's at `max` minimum.
 
 ## Research → Validate → Fix Pipeline
 

deep-web-research/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: deep-web-research
-description: Exhaustive deep web research — five-move flow, external findings ledger, mechanical saturation check, disconfirmation, condensation from disk. Opt-in skill for the research profile.
+description: Exhaustive deep web research — six-move flow, external findings ledger, mechanical saturation check, disconfirmation, condensation from disk. Domain-aware tool selection with adaptive re-strategizing. Opt-in skill for the research profile.
-version: 1.0.0
+version: 2.0.0
 author: Hermes Agent
 ---
 
@@ -25,18 +25,54 @@ These persist across all turns — they are in the skill, not in fading context:
 - **Local and free only.** No internet-based paid services, no SaaS APIs with
   billing, no metered endpoints. Use any tool already configured that fits this rule.
 
+## Tool Selection Principles
+
+These are reasoning heuristics, not rules. Apply judgment. Read once — they are
+static reference, not generated per-question.
+
+1. **Match tool strength to information need.** Don't use SearXNG when arxiv is
+   purpose-built for paper search. Don't use SearXNG when maigret is purpose-built
+   for username lookup. Each tool exists because it's better at something than
+   general search.
+
+2. **Compose, don't isolate.** Most questions need multiple tool categories. A
+   person investigation might need OSINT (maigret, holehe) + social (twscrape) +
+   web (SearXNG) + reference (wikipedia). Compose across categories.
+
+3. **Start broad, then specialize.** Landscape pass uses broad tools (SearXNG,
+   wikipedia). Deep-dive uses specialized tools (arxiv, maigret, Firecrawl). Don't
+   lead with Spiderfoot when a SearXNG search would tell you whether there's even
+   a domain to scan.
+
+4. **Follow the trail, not the plan.** If a finding points to a source type you
+   didn't anticipate, add the tool that handles that source type. The strategy is
+   a starting point, not a contract.
+
+5. **Prefer purpose-built over general.** arxiv > SearXNG for papers. maigret >
+   SearXNG for usernames. holehe > SearXNG for email registration. gnews > SearXNG
+   for news. Use SearXNG for what doesn't have a purpose-built tool.
+
+6. **Consult the `search-key` decision matrix.** It maps task types to tools. Use
+   it as a reference, not a lookup table. If the matrix says `maigret` for username
+   lookup, that's a strong signal — but verify it makes sense for this specific
+   question.
+
+7. **OSINT tools are an escalation ladder, not a default.** Start with SearXNG for
+   person lookups. Escalate to maigret if you have a username. Escalate to holehe
+   if you have an email. Escalate to Spiderfoot only if you have a domain and the
+   scope justifies it. Don't jump to the most aggressive tool first.
+
 ## Architecture: External Findings Ledger
 
-At 200 turns, early findings scroll out of context. The ledger is the fix.
+At 600 turns, early findings scroll out of context. The ledger is the fix.
 
 **Path:** `/tmp/research-<session-id>.md`
 
 **Schema per finding:**
 ```
-## Finding #<N>: <short label>
+### Finding #<N> [SQ<N>]: <short label>
-- Sub-Question: <which sub-question this belongs to>
 - Claim: <the factual claim>
-- Source: <URL> — <credibility tier: primary|secondary|aggregator|social>
+- Source: <URL> — <credibility tier: primary|secondary|aggregator|social|osint>
 - Confirmed by: <URL> (or "single-sourced")
 - Date: <YYYY-MM-DD of the information, not of the search>
 - Confidence: confirmed | likely | single-sourced | disputed
@@ -47,11 +83,48 @@ At 200 turns, early findings scroll out of context. The ledger is the fix.
 finding with the same claim. If found, add the new source to "Confirmed by"
 instead of creating a duplicate. Duplicate findings defeat the saturation check.
 
-**Sub-question tagging:** Every finding must reference which sub-question it
+**Sub-question tagging:** Every finding line includes `[SQ<N>]` — e.g.,
-belongs to. This enables the phase gate to verify coverage and the condensation
+`### Finding #3 [SQ2]: ...`. This enables per-sub-question saturation checks
-phase to group findings by topic.
+that are immune to header ordering.
 
-## Five-Move Research Flow
+**Credibility Tiers:**
+
+| Tier | Definition | Examples |
+|---|---|---|
+| `primary` | Original source — the entity that created or owns the data | Official docs, original research paper, company release notes, raw dataset, government database, court filing |
+| `secondary` | Reports on or analyzes primary sources with editorial oversight | News article citing a study, analyst report, Wikipedia (well-cited), reputable tech blog |
+| `aggregator` | Collects/curates from other sources without original analysis | Hacker News, Reddit, link aggregators, auto-generated comparison sites |
+| `social` | Individual opinion without institutional backing | Tweet, personal blog, forum post, YouTube comment |
+| `osint` | Machine-generated tool output — marks provenance, not confidence | maigret report, holehe results, Spiderfoot scan, theHarvester output |
+
+**Important:** `osint` marks *provenance* (where the claim came from), not
+*confidence*. A single raw maigret hit is low-confidence osint; the same account
+confirmed by a human-authored source is still confirmable. Don't auto-cap
+confidence just because the tier is `osint`.
+
+## Six-Move Research Flow
+
+### Move 0: Analyze & Strategize
+
+1. Consult the `search-key` decision matrix — it's your tool reference, not a
+   lookup you match against.
+
+2. Call `mcp_searxng_searxng_instance_info` to discover available categories and
+   engines. Write them to the ledger.
+
+3. Write this to the ledger, then start Move 1:
+
+   ```
+   ## Strategy
+   - Info needed:  <kinds of facts that answer this question>
+   - Where it lives: <source types — papers? profiles? release notes? news?>
+   - Tools: <tool → what it fetches>   (purpose-built over general search)
+   - Pivots: if I find <X>, add <tool Y>
+   ```
+
+4. OSINT gate — only if the plan names maigret / holehe / theHarvester /
+   Spiderfoot AND the target is a private individual with no public role: log
+   `## Ethics Note: <intent>` and confirm the scan is proportionate.
 
 ### Move 1: Decompose
 
@@ -77,6 +150,21 @@ Shallow-but-broad sweep. For each sub-question, 1-2 searches, read top 1-2 resul
 Extract only: key sources, terms of art, where disagreement lives, major players.
 No deep-diving yet. Append landscape notes to the ledger.
 
+**After Move 2: Re-Strategize Checkpoint.** Read the strategy from the ledger. Ask:
+- Are the tools I selected actually producing results?
+- Have I discovered new angles that need different tools?
+- Is the topic different than I initially thought?
+- Are there tools I should add or drop?
+
+If the answer to any is yes, update the strategy section:
+```
+### Strategy Revision <N>
+- Trigger: <what I learned that changed my mind>
+- Added tools: <tools to add and why>
+- Dropped tools: <tools to drop and why>
+- New pivot signals: <updated signals>
+```
+
 ### Move 3: Deep-Dive per Sub-Question
 
 For each sub-question:
@@ -90,13 +178,27 @@ For each sub-question:
 - Assign credibility tier at capture time (not post-hoc in output).
 - Cross-reference: every claim needs 2+ independent sources.
 - Use browser tools (Playwright) when static extraction fails on JS-heavy pages.
-- Append every finding to the ledger.
+- Append every finding to the ledger with `[SQ<N>]` tag.
 
 Citation trail stop conditions:
 - Reached primary source (original paper, official docs, raw data)
 - Dead end (paywall, 404, requires login)
 - Circular reference (already read this source)
 
+**Every ~10 findings during Move 3: Re-Strategize Checkpoint.** Same questions
+as after Move 2. Update strategy if needed.
+
+**Information-void handling:** After 5 searches on a sub-question with zero
+relevant results (not zero *new* findings — zero *any* findings), document the
+void and move on:
+```
+## SQ<N>: <sub-question> — VOID
+- Searches attempted: <N>
+- Tools tried: <comma-separated>
+- Reason: no relevant results found
+- Note: <why this might be — too new, too obscure, paywalled, etc.>
+```
+
 ### Move 4: Disconfirmation Pass
 
 Actively try to falsify each key claim:
@@ -113,13 +215,32 @@ Disconfirmation actively hunts for disagreement.
 Read the full ledger from disk. Read the phase gate file. If any box is
 unchecked, do NOT condense — go back and complete that item.
 
-Synthesize:
+Synthesize using this template:
-- Lead with the answer. No process narrative.
+
-- Evidence-backed. Every claim tied to a source from the ledger.
+```
-- Concrete. Numbers, dates, names, specifics.
+# <Answer in 1-3 sentences>
-- Uncertainty explicit. "X confirmed by A and B. Y single-sourced from C.
+
-  Z unresolved — D and E conflict."
+## Key Findings
-- Sources section: numbered list with URLs and credibility tiers.
+- <finding> — <source> (<credibility tier>, <date>)
+- <finding> — <source> (<credibility tier>, <date>)
+
+## Uncertainty
+- <claim>: single-sourced from <source>
+- <claim>: disputed — <source A> vs <source B>
+- <claim>: unresolved — no sources found
+
+## Tools Used
+- <tool>: <what it contributed>
+- <tool>: <what it contributed>
+
+## Sources
+1. <URL> — <description> (<tier>)
+2. <URL> — <description> (<tier>)
+
+## Tools That Would Have Helped
+- <tool>: <what it would have enabled>
+None: all sources were accessible with available tools.
+```
 
 ## Guardrails
 
@@ -127,11 +248,12 @@ Synthesize:
 
 Every 3-4 searches, run this command — do not self-assess:
 ```
-grep -c "^## Finding #" /tmp/research-<sid>.md
+grep -c "^### Finding #.*\[SQ<N>\]" /tmp/research-<sid>.md
 ```
 Compare to the previous count. If zero new findings in the last 3 searches,
 that sub-question is saturated. Move to the next. This is a measurable fact,
-not a vibe.
+not a vibe. The `[SQ<N>]` tag makes this immune to header ordering — it counts
+by tag, not by position.
 
 ### Phase Gate File
 
@@ -143,17 +265,34 @@ If any box is unchecked, do NOT condense. Go back and complete that item.
 # Phase Gate
 - [ ] All sub-questions have findings in the ledger
 - [ ] Disconfirmation pass completed for key claims
+- [ ] Source diversity: ≥5 unique domains across all findings
+- [ ] Source diversity: no single domain >30% of findings
+- [ ] Tool diversity: ≥2 different tool categories used (not just SearXNG)
+- [ ] Recency spread: findings span appropriate date range for topic
 - [ ] OR: saturation condition fired (no new findings in last 3 searches)
 - [ ] OR: turn ceiling approaching (wrap up what you have)
 ```
 
 Update this file as you progress.
 
+Mechanical domain diversity check:
+```
+grep -oP 'Source: \K[^ ]+' /tmp/research-<sid>.md | grep -oP 'https?://[^/]+' | sort -u | wc -l
+```
+
 ### Turn Ceiling
 
-200 turns is a safety net, not a target. Hitting the ceiling is a failure to
+600 turns is a safety net, not a target. Hitting the ceiling is a failure to
-condense, not the normal path. If the ceiling is hit, deliver partial findings
+condense, not the normal path. Ramp: if ≥450 turns used, begin condensation
-with a note on what's missing.
+within 60 turns. If ≥540 turns used, condense immediately with what you have.
+If the ceiling is hit, deliver partial findings with a note on what's missing.
+
+### SearXNG Failure Handling
+
+If SearXNG returns errors or empty results for 3 consecutive searches (different
+queries), wait 10 seconds and retry once. If still failing, note the instance
+issue in the ledger and fall back to `web_search` (built-in) or
+`duckduckgo-search` (Python library) for the remainder of that sub-question.
 
 ## Tool Policy