hermes-skills/deep-web-research/SKILL.md

name, description, version, author

name	description	version	author
deep-web-research	Exhaustive deep web research — six-move flow, external findings ledger, mechanical saturation check, disconfirmation, condensation from disk. Domain-aware tool selection with adaptive re-strategizing. Opt-in skill for the research profile.	2.0.0	Hermes Agent

Deep Web Research

Loaded explicitly via -s deep-web-research. Not loaded during normal interactive use of the research profile. This skill overrides default behavior to enforce exhaustive research methodology.

Core Rules (Always in Effect)

These persist across all turns — they are in the skill, not in fading context:

• Confined to /tmp. All file writes go to /tmp/. Never write outside /tmp.
• No self-provisioning. Never install software. No pip, npm, apt, docker, or any package manager. Use only what's already configured.
• No repeat searches. If you catch yourself searching the same thing twice, stop. That sub-question is saturated.
• Blacklist after 3 failures. If a URL returns an error 3 times, blacklist it and move on. Do not retry indefinitely.
• Local and free only. No internet-based paid services, no SaaS APIs with billing, no metered endpoints. Use any tool already configured that fits this rule.

Tool Selection Principles

These are reasoning heuristics, not rules. Apply judgment. Read once — they are static reference, not generated per-question.

1. Match tool strength to information need. Don't use SearXNG when arxiv is purpose-built for paper search. Don't use SearXNG when maigret is purpose-built for username lookup. Each tool exists because it's better at something than general search.

2. Compose, don't isolate. Most questions need multiple tool categories. A person investigation might need OSINT (maigret, holehe) + social (twscrape) + web (SearXNG) + reference (wikipedia). Compose across categories.

3. Start broad, then specialize. Landscape pass uses broad tools (SearXNG, wikipedia). Deep-dive uses specialized tools (arxiv, maigret, Firecrawl). Don't lead with Spiderfoot when a SearXNG search would tell you whether there's even a domain to scan.

4. Follow the trail, not the plan. If a finding points to a source type you didn't anticipate, add the tool that handles that source type. The strategy is a starting point, not a contract.

5. Prefer purpose-built over general. arxiv > SearXNG for papers. maigret > SearXNG for usernames. holehe > SearXNG for email registration. gnews > SearXNG for news. Use SearXNG for what doesn't have a purpose-built tool.

6. Consult the search-key decision matrix. It maps task types to tools. Use it as a reference, not a lookup table. If the matrix says maigret for username lookup, that's a strong signal — but verify it makes sense for this specific question.

7. OSINT tools are an escalation ladder, not a default. Start with SearXNG for person lookups. Escalate to maigret if you have a username. Escalate to holehe if you have an email. Escalate to Spiderfoot only if you have a domain and the scope justifies it. Don't jump to the most aggressive tool first.

Architecture: External Findings Ledger

At 600 turns, early findings scroll out of context. The ledger is the fix.

Path: /tmp/research-<session-id>.md

Schema per finding:

### Finding #<N> [SQ<N>]: <short label>
- Claim: <the factual claim>
- Source: <URL> — <credibility tier: primary|secondary|aggregator|social|osint>
- Confirmed by: <URL> (or "single-sourced")
- Date: <YYYY-MM-DD of the information, not of the search>
- Confidence: confirmed | likely | single-sourced | disputed
- Notes: <contradictions, caveats, context>

Dedup rule: Before writing a new finding, scan the ledger for an existing finding with the same claim. If found, add the new source to "Confirmed by" instead of creating a duplicate. Duplicate findings defeat the saturation check.

Sub-question tagging: Every finding line includes [SQ<N>] — e.g., ### Finding #3 [SQ2]: .... This enables per-sub-question saturation checks that are immune to header ordering.

Credibility Tiers:

Tier	Definition	Examples
primary	Original source — the entity that created or owns the data	Official docs, original research paper, company release notes, raw dataset, government database, court filing
secondary	Reports on or analyzes primary sources with editorial oversight	News article citing a study, analyst report, Wikipedia (well-cited), reputable tech blog
aggregator	Collects/curates from other sources without original analysis	Hacker News, Reddit, link aggregators, auto-generated comparison sites
social	Individual opinion without institutional backing	Tweet, personal blog, forum post, YouTube comment
osint	Machine-generated tool output — marks provenance, not confidence	maigret report, holehe results, Spiderfoot scan, theHarvester output

Important: osint marks provenance (where the claim came from), not confidence. A single raw maigret hit is low-confidence osint; the same account confirmed by a human-authored source is still confirmable. Don't auto-cap confidence just because the tier is osint.

Six-Move Research Flow

Move 0: Analyze & Strategize

1. Consult the search-key decision matrix — it's your tool reference, not a lookup you match against.

2. Call mcp_searxng_searxng_instance_info to discover available categories and engines. Write them to the ledger.

3. Write this to the ledger, then start Move 1:

   ## Strategy
   - Info needed:  <kinds of facts that answer this question>
   - Where it lives: <source types — papers? profiles? release notes? news?>
   - Tools: <tool → what it fetches>   (purpose-built over general search)
   - Pivots: if I find <X>, add <tool Y>
   

4. OSINT gate — only if the plan names maigret / holehe / theHarvester / Spiderfoot AND the target is a private individual with no public role: log ## Ethics Note: <intent> and confirm the scan is proportionate.

Move 1: Decompose

Break the question into sub-questions. Write them to the ledger with time classifications. This makes "multiple angles" principled instead of random.

## Sub-Questions
1. <sub-question text> [time: week|month|year|none]
2. <sub-question text> [time: week|month|year|none]

The time classification lives on disk, not in fading context. Move 3 reads it back when applying time filters per sub-question.

Time classification guide:

• Current-state (releases, prices, news, versions, events): week or month
• Established-knowledge (how something works, architecture, algorithms, history): none or year

Move 2: Landscape Pass

Shallow-but-broad sweep. For each sub-question, 1-2 searches, read top 1-2 results. Extract only: key sources, terms of art, where disagreement lives, major players. No deep-diving yet. Append landscape notes to the ledger.

After Move 2: Re-Strategize Checkpoint. Read the strategy from the ledger. Ask:

• Are the tools I selected actually producing results?
• Have I discovered new angles that need different tools?
• Is the topic different than I initially thought?
• Are there tools I should add or drop?

If the answer to any is yes, update the strategy section:

### Strategy Revision <N>
- Trigger: <what I learned that changed my mind>
- Added tools: <tools to add and why>
- Dropped tools: <tools to drop and why>
- New pivot signals: <updated signals>

Move 3: Deep-Dive per Sub-Question

For each sub-question:

• Search from 3+ angles with different categories and framings. Use mcp_searxng_searxng_web_search. Apply the time filter from the ledger.
• Read full pages (5-10 per sub-question) via mcp_searxng_web_url_read. Snippets are pointers, not sources.
• Follow citation trails: if a page cites a study/paper/dataset, go read that source. Depth-first — one trail at a time, to its end, then the next.
• Extract structured data: tables, numbers, dates, versions, names.
• Assign credibility tier at capture time (not post-hoc in output).
• Cross-reference: every claim needs 2+ independent sources.
• Use browser tools (Playwright) when static extraction fails on JS-heavy pages.
• Append every finding to the ledger with [SQ<N>] tag.

Citation trail stop conditions:

• Reached primary source (original paper, official docs, raw data)
• Dead end (paywall, 404, requires login)
• Circular reference (already read this source)

Every ~10 findings during Move 3: Re-Strategize Checkpoint. Same questions as after Move 2. Update strategy if needed.

Information-void handling: After 5 searches on a sub-question with zero relevant results (not zero new findings — zero any findings), document the void and move on:

## SQ<N>: <sub-question> — VOID
- Searches attempted: <N>
- Tools tried: <comma-separated>
- Reason: no relevant results found
- Note: <why this might be — too new, too obscure, paywalled, etc.>

Move 4: Disconfirmation Pass

Actively try to falsify each key claim:

• Search for "[claim] wrong", "[claim] criticism", "[claim] outdated"
• Check dates: is a 2024 source being presented as current?
• Look for contradicting evidence
• Append disconfirmation findings to the ledger

This is distinct from cross-referencing. Cross-reference confirms agreement. Disconfirmation actively hunts for disagreement.

Move 5: Condensation

Read the full ledger from disk. Read the phase gate file. If any box is unchecked, do NOT condense — go back and complete that item.

Synthesize using this template:

# <Answer in 1-3 sentences>

## Key Findings
- <finding> — <source> (<credibility tier>, <date>)
- <finding> — <source> (<credibility tier>, <date>)

## Uncertainty
- <claim>: single-sourced from <source>
- <claim>: disputed — <source A> vs <source B>
- <claim>: unresolved — no sources found

## Tools Used
- <tool>: <what it contributed>
- <tool>: <what it contributed>

## Sources
1. <URL> — <description> (<tier>)
2. <URL> — <description> (<tier>)

## Tools That Would Have Helped
- <tool>: <what it would have enabled>
None: all sources were accessible with available tools.

Guardrails

Saturation Check (Mechanical)

Every 3-4 searches, run this command — do not self-assess:

grep -c "^### Finding #.*\[SQ<N>\]" /tmp/research-<sid>.md

Compare to the previous count. If zero new findings in the last 3 searches, that sub-question is saturated. Move to the next. This is a measurable fact, not a vibe. The [SQ<N>] tag makes this immune to header ordering — it counts by tag, not by position.

Phase Gate File

The phase gate is a file at /tmp/research-<sid>-gate.md. Write it and read it back — it is not a mental checklist. Before moving to Move 5, read the gate file. If any box is unchecked, do NOT condense. Go back and complete that item.

# Phase Gate
- [ ] All sub-questions have findings in the ledger
- [ ] Disconfirmation pass completed for key claims
- [ ] Source diversity: ≥5 unique domains across all findings
- [ ] Source diversity: no single domain >30% of findings
- [ ] Tool diversity: ≥2 different tool categories used (not just SearXNG)
- [ ] Recency spread: findings span appropriate date range for topic
- [ ] OR: saturation condition fired (no new findings in last 3 searches)
- [ ] OR: turn ceiling approaching (wrap up what you have)

Update this file as you progress.

Mechanical domain diversity check:

grep -oP 'Source: \K[^ ]+' /tmp/research-<sid>.md | grep -oP 'https?://[^/]+' | sort -u | wc -l

Turn Ceiling

600 turns is a safety net, not a target. Hitting the ceiling is a failure to condense, not the normal path. Ramp: if ≥450 turns used, begin condensation within 60 turns. If ≥540 turns used, condense immediately with what you have. If the ceiling is hit, deliver partial findings with a note on what's missing.

SearXNG Failure Handling

If SearXNG returns errors or empty results for 3 consecutive searches (different queries), wait 10 seconds and retry once. If still failing, note the instance issue in the ledger and fall back to web_search (built-in) or duckduckgo-search (Python library) for the remainder of that sub-question.

Tool Policy

One rule: local and free only. Use any tool already configured on the research profile that fits this rule. No internet-based paid services, no SaaS APIs with billing, no metered endpoints. You determine what fits — you are not given a list of allowed or disallowed tools.

No self-provisioning. Never install, pull, or spin up new tools at runtime.

Tools That Would Have Helped

At the end of the response (after Sources), add a section if any limitations were hit:

Tools that would have improved this research:
- <tool name>: <what it would have enabled>
None: all sources were accessible with available tools.

This is informational only. Do your best with what's available and note what could have been better. Do not stop or block on tool gaps.