hermes-skills/deep-web-research/SKILL.md

---
name: deep-web-research
description: Exhaustive deep web research — six-move flow, external findings ledger, mechanical saturation check, disconfirmation, condensation from disk. Domain-aware tool selection with adaptive re-strategizing. Opt-in skill for the research profile.
version: 2.6.0
author: Hermes Agent
---

# Deep Web Research

Loaded explicitly via `-s deep-web-research`. Not loaded during normal interactive
use of the research profile. This skill overrides default behavior to enforce
exhaustive research methodology.

## Core Rules (Always in Effect)

These persist across all turns — they are in the skill, not in fading context:

- **Confined to /tmp.** All file writes go to `/tmp/`. Never write outside /tmp. **Exception:** The final condensed report (Move 5) and the abort report (Under-Specified Question Handling) go to `/home/n8n/workspace/research/results/` (absolute path, create the directory first with `mkdir -p`). Also, any file path the operator names explicitly in the prompt (e.g., Stage 3 fix pass editing a plan file in place). The ledger and gate files stay in /tmp.
- **No self-provisioning.** Never install software. No pip, npm, apt, docker, or
  any package manager. Use only what's already configured.
- **No repeat searches.** If you catch yourself searching the same thing twice,
  stop. That sub-question is saturated.
- **Blacklist after 3 failures.** If a URL returns an error 3 times, blacklist it
  and move on. Do not retry indefinitely.
- **Local and free only.** No internet-based paid services, no SaaS APIs with
  billing, no metered endpoints. Use any tool already configured that fits this rule.

## Tool Selection Principles

These are reasoning heuristics, not rules. Apply judgment. Read once — they are
static reference, not generated per-question.

1. **Match tool strength to information need.** Don't use SearXNG when arxiv is
   purpose-built for paper search. Don't use SearXNG when maigret is purpose-built
   for username lookup. Each tool exists because it's better at something than
   general search.

2. **Compose, don't isolate.** Most questions need multiple tool categories. A
   person investigation might need OSINT (maigret, holehe) + social (twscrape) +
   web (SearXNG) + reference (wikipedia). Compose across categories.

3. **Start broad, then specialize.** Landscape pass uses broad tools (SearXNG,
   wikipedia). Deep-dive uses specialized tools (arxiv, maigret, Firecrawl). Don't
   lead with Spiderfoot when a SearXNG search would tell you whether there's even
   a domain to scan.

4. **Follow the trail, not the plan.** If a finding points to a source type you
   didn't anticipate, add the tool that handles that source type. The strategy is
   a starting point, not a contract.

5. **Prefer purpose-built over general.** arxiv > SearXNG for papers. maigret >
   SearXNG for usernames. holehe > SearXNG for email registration. gnews > SearXNG
   for news. Use SearXNG for what doesn't have a purpose-built tool.

6. **Consult the `search-key` decision matrix.** It maps task types to tools. Use
   it as a reference, not a lookup table. If the matrix says `maigret` for username
   lookup, that's a strong signal — but verify it makes sense for this specific
   question.

7. **OSINT tools are an escalation ladder, not a default.** Start with SearXNG for
   person lookups. Escalate to maigret if you have a username. Escalate to holehe
   if you have an email. Escalate to Spiderfoot only if you have a domain and the
   scope justifies it. Don't jump to the most aggressive tool first.

## Architecture: External Findings Ledger

At 600 turns, early findings scroll out of context. The ledger is the fix.

**Path:** `/tmp/research-<YYYY-MM-DD>-<slug>.md` (see Move 0 — the stem is picked once, not derived from session_id)

**Schema per finding:**
```
### Finding #<N> [SQ<N>]: <short label>
- Claim: <the factual claim>
- Source: <URL> — <credibility tier: primary|secondary|aggregator|social|osint>
- Confirmed by: <URL> (or "single-sourced")
- Date: <YYYY-MM-DD of the information, not of the search>
- Confidence: confirmed | likely | single-sourced | disputed
- Notes: <contradictions, caveats, context>
```

**Dedup rule:** Before writing a new finding, scan the ledger for an existing
finding with the same claim. If found, add the new source to "Confirmed by"
instead of creating a duplicate. Duplicate findings defeat the saturation check.

**Sub-question tagging:** Every finding line includes `[SQ<N>]` — e.g.,
`### Finding #3 [SQ2]: ...`. This enables per-sub-question saturation checks
that are immune to header ordering.

**Credibility Tiers:**

| Tier | Definition | Examples |
|---|---|---|
| `primary` | Original source — the entity that created or owns the data | Official docs, original research paper, company release notes, raw dataset, government database, court filing |
| `secondary` | Reports on or analyzes primary sources with editorial oversight | News article citing a study, analyst report, Wikipedia (well-cited), reputable tech blog |
| `aggregator` | Collects/curates from other sources without original analysis | Hacker News, Reddit, link aggregators, auto-generated comparison sites |
| `social` | Individual opinion without institutional backing | Tweet, personal blog, forum post, YouTube comment |
| `osint` | Machine-generated tool output — marks provenance, not confidence | maigret report, holehe results, Spiderfoot scan, theHarvester output |

**Important:** `osint` marks *provenance* (where the claim came from), not
*confidence*. A single raw maigret hit is low-confidence osint; the same account
confirmed by a human-authored source is still confirmable. Don't auto-cap
confidence just because the tier is `osint`.

## Under-Specified Question Handling

This skill runs headless — there is no operator to ask clarifying questions. If the question is ambiguous enough that any strategy would be a guess, **abort with a structured under-specification report instead of dispatching with a guessed scope.**

**Detection triggers (any one is enough to abort):**
- The question references a person / org / project / model with no disambiguating context (multiple plausible referents, no narrowing clue in the prompt)
- The question asks "everything about X" with no scoping (time range, domain, angle)
- The question has 3+ mutually-exclusive branches and no signal which one is wanted
- The question uses vague hedges ("good", "best", "modern", "recent") with no concrete criterion

**What to do instead of guessing:**
1. Do NOT start Move 0. Do NOT begin the ledger.
2. Write the abort report to the results path so the dispatcher can find it:

   ```
   mkdir -p /home/n8n/workspace/research/results
   ```

   Then write to `/home/n8n/workspace/research/results/<YYYY-MM-DD>-<slug>-ABORTED.md`:

   ```
   ---
   question: "<original question>"
   date: <YYYY-MM-DD>
   status: ABORTED
   ---

   ## Aborted: Under-Specified Question

   The question "<original question>" has multiple plausible interpretations.
   Cannot dispatch a research strategy without narrowing the scope.

   Possible interpretations:
   - <interpretation 1>
   - <interpretation 2>
   - <interpretation 3>

   Ask the operator to clarify which one is intended, or rewrite the question with the scope made explicit.
   ```

3. Exit cleanly. Do not fabricate findings, do not start Move 1.

**Cap on auto-clarification attempts:** None. The skill does not interactively ask the operator. One dispatch, one answer (or one abort). The dispatcher (`deep-research` on the general profile) handles the operator-facing clarifying question loop before this skill is even loaded — see the "Clarifying Questions (Before Dispatch)" section in that skill.

**Cap on output length when aborting:** Short. The whole point of the abort is to surface the problem fast. Five lines of possible interpretations is the ceiling.

## Six-Move Research Flow

### Move 0: Analyze & Strategize

**Pick the ledger stem once:** `/tmp/research-<YYYY-MM-DD>-<slug>`. Use `<stem>.md` for the ledger and `<stem>-gate.md` for the gate. Substitute this stem into every grep below. (The session_id is not available to the agent under `-Q` — use the date+slug stem instead.)

1. Consult the `search-key` decision matrix — it's your tool reference, not a
   lookup you match against.

2. Call `mcp_searxng_searxng_instance_info` to discover available categories and
   engines. Write them to the ledger.

3. Write this to the ledger, then start Move 1:

   ```
   ## Strategy
   - Info needed:  <kinds of facts that answer this question>
   - Where it lives: <source types — papers? profiles? release notes? news?>
   - Tools: <tool → what it fetches>   (purpose-built over general search)
   - Pivots: if I find <X>, add <tool Y>
   ```

4. Write the phase gate template (see Guardrails → Phase Gate) to `<stem>-gate.md` with all boxes unchecked. This is the starting state — update it as you progress.

5. OSINT gate — only if the plan names maigret / holehe / theHarvester /
   Spiderfoot AND the target is a private individual with no public role: log
   `## Ethics Note: <intent>` and confirm the scan is proportionate.

### Move 1: Decompose

Break the question into sub-questions. Write them to the ledger with time
classifications. This makes "multiple angles" principled instead of random.

```
## Sub-Questions
1. <sub-question text> [time: week|month|year|none]
2. <sub-question text> [time: week|month|year|none]
```

The time classification lives on disk, not in fading context. Move 3 reads it
back when applying time filters per sub-question.

**Time classification guide:**
- **Current-state** (releases, prices, news, versions, events): `week` or `month`
- **Established-knowledge** (how something works, architecture, algorithms, history): `none` or `year`

### Move 2: Landscape Pass

Shallow-but-broad sweep. For each sub-question, 1-2 searches, read top 1-2 results.
Extract only: key sources, terms of art, where disagreement lives, major players.
No deep-diving yet. Append landscape notes to the ledger.

**After Move 2: Re-Strategize Checkpoint.** Read the strategy from the ledger. Ask:
- Are the tools I selected actually producing results?
- Have I discovered new angles that need different tools?
- Is the topic different than I initially thought?
- Are there tools I should add or drop?

If the answer to any is yes, update the strategy section:
```
### Strategy Revision <N>
- Trigger: <what I learned that changed my mind>
- Added tools: <tools to add and why>
- Dropped tools: <tools to drop and why>
- New pivot signals: <updated signals>
```

### Move 3: Deep-Dive per Sub-Question

For each sub-question:
- Search from 3+ angles with different categories and framings. Use
  `mcp_searxng_searxng_web_search`. Apply the time filter from the ledger.
- Read full pages (5-10 per sub-question) via `mcp_searxng_web_url_read`.
  Snippets are pointers, not sources.
- Follow citation trails: if a page cites a study/paper/dataset, go read that
  source. Depth-first — one trail at a time, to its end, then the next.
- Extract structured data: tables, numbers, dates, versions, names.
- Assign credibility tier at capture time (not post-hoc in output).
- Cross-reference: every claim needs 2+ independent sources.
- Use browser tools (Playwright) when static extraction fails on JS-heavy pages.
- Append every finding to the ledger with `[SQ<N>]` tag.

Citation trail stop conditions:
- Reached primary source (original paper, official docs, raw data)
- Dead end (paywall, 404, requires login)
- Circular reference (already read this source)

**Every ~10 findings during Move 3: Re-Strategize Checkpoint.** Same questions
as after Move 2. Update strategy if needed.

**Information-void handling:** After 5 searches on a sub-question with zero
relevant results (not zero *new* findings — zero *any* findings), document the
void and move on:
```
## SQ<N>: <sub-question> — VOID
- Searches attempted: <N>
- Tools tried: <comma-separated>
- Reason: no relevant results found
- Note: <why this might be — too new, too obscure, paywalled, etc.>
```

### Move 4: Disconfirmation Pass

Actively try to falsify each key claim:
- Search for "[claim] wrong", "[claim] criticism", "[claim] outdated"
- Check dates: is a 2024 source being presented as current?
- Look for contradicting evidence
- Append disconfirmation findings to the ledger

This is distinct from cross-referencing. Cross-reference confirms agreement.
Disconfirmation actively hunts for disagreement.

### Move 5: Condensation

Read the full ledger from disk. Read the phase gate file. If any box is
unchecked, do NOT condense — go back and complete that item. (The `OR:` items
in the gate are alternatives, not requirements — a normal run will have some
unchecked; that's expected.) **Exception: the turn-ceiling ramp overrides the
gate. At ≥450 turns (once the ramp has fired), condense regardless of unchecked
boxes and list the unmet gate items under `## Uncertainty`.**

Synthesize using this template, then **WRITE the report to disk**:

1. `mkdir -p /home/n8n/workspace/research/results`
2. Write to `/home/n8n/workspace/research/results/<YYYY-MM-DD>-<slug>.md` (absolute path, derive slug from the question)

This is the single deliverable — the dispatcher expects it at this path. Do NOT just output to stdout and exit.

3. **Save to ai_brain_kb.** After writing the report, add it to the Qdrant knowledge base:
   ```
   mcp__better_qdrant__add_documents(
     collection="ai_brain_kb",
     embeddingService="ollama",
     filePath="/home/n8n/workspace/research/results/<YYYY-MM-DD>-<slug>.md"
   )
   ```
   This is mandatory — every deep research report goes into the brain. The `ai-brain-kb` skill and `better_qdrant` MCP server are configured on this profile. If the MCP call fails (network, Qdrant down), log the error to the report's Uncertainty section and continue — the report file on disk is the source of truth.

```
---
question: "<original question>"
date: <YYYY-MM-DD>
sources: <count>
confidence: <high|medium|low>
---

# <Answer in 1-3 sentences>

## Key Findings
- <finding> — <source> (<credibility tier>, <date>)
- <finding> — <source> (<credibility tier>, <date>)

## Uncertainty
- <claim>: single-sourced from <source>
- <claim>: disputed — <source A> vs <source B>
- <claim>: unresolved — no sources found

## Tools Used
- <tool>: <what it contributed>
- <tool>: <what it contributed>

## Sources
1. <URL> — <description> (<tier>)
2. <URL> — <description> (<tier>)

## Tools That Would Have Helped
- <tool>: <what it would have enabled>
(If nothing was missing, write exactly: "None: all sources were accessible with available tools.")
```

## Guardrails

### Saturation Check (Mechanical)

Every 3-4 searches, run this command — do not self-assess:
```
grep -c "^### Finding #.*\[SQ<N>\]" /tmp/research-<YYYY-MM-DD>-<slug>.md
```
Compare to the previous count. If zero new findings in the last 3 searches,
that sub-question is saturated. Move to the next. **Exception: if the count is
0 (no findings at all), keep going to 5 searches, then document it with the
information-void template.** This is a measurable fact, not a vibe. The `[SQ<N>]`
tag makes this immune to header ordering — it counts by tag, not by position.

### Phase Gate File

The phase gate is a file at `/tmp/research-<YYYY-MM-DD>-<slug>-gate.md`. Write it and read it
back — it is not a mental checklist. Before moving to Move 5, read the gate file.
(The `OR:` items are alternatives, not requirements — a normal run will have some
unchecked; that's expected.)

```
# Phase Gate
- [ ] All sub-questions have findings in the ledger or are documented as VOID
- [ ] Disconfirmation pass completed for key claims
- [ ] Source diversity: ≥5 unique domains across all findings
- [ ] Source diversity: no single domain >30% of findings
- [ ] Tool diversity: ≥2 different tool categories used (not just SearXNG)
- [ ] Recency spread: findings span appropriate date range for topic
- [ ] OR: saturation condition fired (no new findings in last 3 searches)
- [ ] OR: turn ceiling approaching (wrap up what you have)
```

Update this file as you progress.

Mechanical domain diversity check:
```
grep -oP 'Source: \K[^ ]+' /tmp/research-<YYYY-MM-DD>-<slug>.md | grep -oP 'https?://[^/]+' | sort | uniq -c | sort -rn
```
This shows both the unique domain count and the per-domain share — use it to verify the ≥5 domains and ≤30% per-domain requirements.

### Turn Ceiling

600 turns is a safety net, not a target. Hitting the ceiling is a failure to
condense, not the normal path. Ramp: if ≥450 turns used, begin condensation
within 60 turns. If ≥540 turns used, condense immediately with what you have.
If the ceiling is hit, deliver partial findings with a note on what's missing.

### SearXNG Failure Handling

If SearXNG returns errors or empty results for 3 consecutive searches (different
queries):

1. **First recovery: VPN reconnect.** Reconnect VPN to get a fresh IP, then
   retry the most recent search once. The VPN reconnect procedure is:
   ```bash
   nordvpn disconnect && nordvpn connect us <different_city>
   ```
   Use a different city from the current connection to avoid the same server.
   Verify with `nordvpn status`. This is authorized without asking — the
   nordvpn skill grants reconnect permission. Do NOT change any other VPN
   settings.
2. **Second recovery:** Wait 10 seconds and retry once with a different
   framing.
3. **Third recovery:** Fall back to `web_search` (built-in), or
   `duckduckgo-search` (Python library) if already installed.
4. **If all fail:** Document the void per the information-void template and
   move on.

The VPN reconnect is a recovery step, not a search — it doesn't count against
the turn budget or saturation tracking.

## Tool Policy

**One rule: local and free only.** Use any tool already configured on the
research profile that fits this rule. No internet-based paid services, no SaaS
APIs with billing, no metered endpoints. You determine what fits — you are not
given a list of allowed or disallowed tools.

**No self-provisioning.** Never install, pull, or spin up new tools at runtime.
(The "Tools That Would Have Helped" section in the Move 5 report template covers
tool gaps — it's informational only; do not stop or block on them.)