--- name: deep-web-research description: Exhaustive deep web research — six-move flow, external findings ledger, mechanical saturation check, disconfirmation, condensation from disk. Domain-aware tool selection with adaptive re-strategizing. Opt-in skill for the research profile. version: 2.0.0 author: Hermes Agent --- # Deep Web Research Loaded explicitly via `-s deep-web-research`. Not loaded during normal interactive use of the research profile. This skill overrides default behavior to enforce exhaustive research methodology. ## Core Rules (Always in Effect) These persist across all turns — they are in the skill, not in fading context: - **Confined to /tmp.** All file writes go to `/tmp/`. Never write outside /tmp. - **No self-provisioning.** Never install software. No pip, npm, apt, docker, or any package manager. Use only what's already configured. - **No repeat searches.** If you catch yourself searching the same thing twice, stop. That sub-question is saturated. - **Blacklist after 3 failures.** If a URL returns an error 3 times, blacklist it and move on. Do not retry indefinitely. - **Local and free only.** No internet-based paid services, no SaaS APIs with billing, no metered endpoints. Use any tool already configured that fits this rule. ## Tool Selection Principles These are reasoning heuristics, not rules. Apply judgment. Read once — they are static reference, not generated per-question. 1. **Match tool strength to information need.** Don't use SearXNG when arxiv is purpose-built for paper search. Don't use SearXNG when maigret is purpose-built for username lookup. Each tool exists because it's better at something than general search. 2. **Compose, don't isolate.** Most questions need multiple tool categories. A person investigation might need OSINT (maigret, holehe) + social (twscrape) + web (SearXNG) + reference (wikipedia). Compose across categories. 3. **Start broad, then specialize.** Landscape pass uses broad tools (SearXNG, wikipedia). Deep-dive uses specialized tools (arxiv, maigret, Firecrawl). Don't lead with Spiderfoot when a SearXNG search would tell you whether there's even a domain to scan. 4. **Follow the trail, not the plan.** If a finding points to a source type you didn't anticipate, add the tool that handles that source type. The strategy is a starting point, not a contract. 5. **Prefer purpose-built over general.** arxiv > SearXNG for papers. maigret > SearXNG for usernames. holehe > SearXNG for email registration. gnews > SearXNG for news. Use SearXNG for what doesn't have a purpose-built tool. 6. **Consult the `search-key` decision matrix.** It maps task types to tools. Use it as a reference, not a lookup table. If the matrix says `maigret` for username lookup, that's a strong signal — but verify it makes sense for this specific question. 7. **OSINT tools are an escalation ladder, not a default.** Start with SearXNG for person lookups. Escalate to maigret if you have a username. Escalate to holehe if you have an email. Escalate to Spiderfoot only if you have a domain and the scope justifies it. Don't jump to the most aggressive tool first. ## Architecture: External Findings Ledger At 600 turns, early findings scroll out of context. The ledger is the fix. **Path:** `/tmp/research-.md` **Schema per finding:** ``` ### Finding # [SQ]: - Claim: - Source: — - Confirmed by: (or "single-sourced") - Date: - Confidence: confirmed | likely | single-sourced | disputed - Notes: ``` **Dedup rule:** Before writing a new finding, scan the ledger for an existing finding with the same claim. If found, add the new source to "Confirmed by" instead of creating a duplicate. Duplicate findings defeat the saturation check. **Sub-question tagging:** Every finding line includes `[SQ]` — e.g., `### Finding #3 [SQ2]: ...`. This enables per-sub-question saturation checks that are immune to header ordering. **Credibility Tiers:** | Tier | Definition | Examples | |---|---|---| | `primary` | Original source — the entity that created or owns the data | Official docs, original research paper, company release notes, raw dataset, government database, court filing | | `secondary` | Reports on or analyzes primary sources with editorial oversight | News article citing a study, analyst report, Wikipedia (well-cited), reputable tech blog | | `aggregator` | Collects/curates from other sources without original analysis | Hacker News, Reddit, link aggregators, auto-generated comparison sites | | `social` | Individual opinion without institutional backing | Tweet, personal blog, forum post, YouTube comment | | `osint` | Machine-generated tool output — marks provenance, not confidence | maigret report, holehe results, Spiderfoot scan, theHarvester output | **Important:** `osint` marks *provenance* (where the claim came from), not *confidence*. A single raw maigret hit is low-confidence osint; the same account confirmed by a human-authored source is still confirmable. Don't auto-cap confidence just because the tier is `osint`. ## Six-Move Research Flow ### Move 0: Analyze & Strategize 1. Consult the `search-key` decision matrix — it's your tool reference, not a lookup you match against. 2. Call `mcp_searxng_searxng_instance_info` to discover available categories and engines. Write them to the ledger. 3. Write this to the ledger, then start Move 1: ``` ## Strategy - Info needed: - Where it lives: - Tools: (purpose-built over general search) - Pivots: if I find , add ``` 4. OSINT gate — only if the plan names maigret / holehe / theHarvester / Spiderfoot AND the target is a private individual with no public role: log `## Ethics Note: ` and confirm the scan is proportionate. ### Move 1: Decompose Break the question into sub-questions. Write them to the ledger with time classifications. This makes "multiple angles" principled instead of random. ``` ## Sub-Questions 1. [time: week|month|year|none] 2. [time: week|month|year|none] ``` The time classification lives on disk, not in fading context. Move 3 reads it back when applying time filters per sub-question. **Time classification guide:** - **Current-state** (releases, prices, news, versions, events): `week` or `month` - **Established-knowledge** (how something works, architecture, algorithms, history): `none` or `year` ### Move 2: Landscape Pass Shallow-but-broad sweep. For each sub-question, 1-2 searches, read top 1-2 results. Extract only: key sources, terms of art, where disagreement lives, major players. No deep-diving yet. Append landscape notes to the ledger. **After Move 2: Re-Strategize Checkpoint.** Read the strategy from the ledger. Ask: - Are the tools I selected actually producing results? - Have I discovered new angles that need different tools? - Is the topic different than I initially thought? - Are there tools I should add or drop? If the answer to any is yes, update the strategy section: ``` ### Strategy Revision - Trigger: - Added tools: - Dropped tools: - New pivot signals: ``` ### Move 3: Deep-Dive per Sub-Question For each sub-question: - Search from 3+ angles with different categories and framings. Use `mcp_searxng_searxng_web_search`. Apply the time filter from the ledger. - Read full pages (5-10 per sub-question) via `mcp_searxng_web_url_read`. Snippets are pointers, not sources. - Follow citation trails: if a page cites a study/paper/dataset, go read that source. Depth-first — one trail at a time, to its end, then the next. - Extract structured data: tables, numbers, dates, versions, names. - Assign credibility tier at capture time (not post-hoc in output). - Cross-reference: every claim needs 2+ independent sources. - Use browser tools (Playwright) when static extraction fails on JS-heavy pages. - Append every finding to the ledger with `[SQ]` tag. Citation trail stop conditions: - Reached primary source (original paper, official docs, raw data) - Dead end (paywall, 404, requires login) - Circular reference (already read this source) **Every ~10 findings during Move 3: Re-Strategize Checkpoint.** Same questions as after Move 2. Update strategy if needed. **Information-void handling:** After 5 searches on a sub-question with zero relevant results (not zero *new* findings — zero *any* findings), document the void and move on: ``` ## SQ: — VOID - Searches attempted: - Tools tried: - Reason: no relevant results found - Note: ``` ### Move 4: Disconfirmation Pass Actively try to falsify each key claim: - Search for "[claim] wrong", "[claim] criticism", "[claim] outdated" - Check dates: is a 2024 source being presented as current? - Look for contradicting evidence - Append disconfirmation findings to the ledger This is distinct from cross-referencing. Cross-reference confirms agreement. Disconfirmation actively hunts for disagreement. ### Move 5: Condensation Read the full ledger from disk. Read the phase gate file. If any box is unchecked, do NOT condense — go back and complete that item. Synthesize using this template: ``` # ## Key Findings - — (, ) - — (, ) ## Uncertainty - : single-sourced from - : disputed — vs - : unresolved — no sources found ## Tools Used - : - : ## Sources 1. — () 2. — () ## Tools That Would Have Helped - : None: all sources were accessible with available tools. ``` ## Guardrails ### Saturation Check (Mechanical) Every 3-4 searches, run this command — do not self-assess: ``` grep -c "^### Finding #.*\[SQ\]" /tmp/research-.md ``` Compare to the previous count. If zero new findings in the last 3 searches, that sub-question is saturated. Move to the next. This is a measurable fact, not a vibe. The `[SQ]` tag makes this immune to header ordering — it counts by tag, not by position. ### Phase Gate File The phase gate is a file at `/tmp/research--gate.md`. Write it and read it back — it is not a mental checklist. Before moving to Move 5, read the gate file. If any box is unchecked, do NOT condense. Go back and complete that item. ``` # Phase Gate - [ ] All sub-questions have findings in the ledger - [ ] Disconfirmation pass completed for key claims - [ ] Source diversity: ≥5 unique domains across all findings - [ ] Source diversity: no single domain >30% of findings - [ ] Tool diversity: ≥2 different tool categories used (not just SearXNG) - [ ] Recency spread: findings span appropriate date range for topic - [ ] OR: saturation condition fired (no new findings in last 3 searches) - [ ] OR: turn ceiling approaching (wrap up what you have) ``` Update this file as you progress. Mechanical domain diversity check: ``` grep -oP 'Source: \K[^ ]+' /tmp/research-.md | grep -oP 'https?://[^/]+' | sort -u | wc -l ``` ### Turn Ceiling 600 turns is a safety net, not a target. Hitting the ceiling is a failure to condense, not the normal path. Ramp: if ≥450 turns used, begin condensation within 60 turns. If ≥540 turns used, condense immediately with what you have. If the ceiling is hit, deliver partial findings with a note on what's missing. ### SearXNG Failure Handling If SearXNG returns errors or empty results for 3 consecutive searches (different queries), wait 10 seconds and retry once. If still failing, note the instance issue in the ledger and fall back to `web_search` (built-in) or `duckduckgo-search` (Python library) for the remainder of that sub-question. ## Tool Policy **One rule: local and free only.** Use any tool already configured on the research profile that fits this rule. No internet-based paid services, no SaaS APIs with billing, no metered endpoints. You determine what fits — you are not given a list of allowed or disallowed tools. **No self-provisioning.** Never install, pull, or spin up new tools at runtime. ### Tools That Would Have Helped At the end of the response (after Sources), add a section if any limitations were hit: ``` Tools that would have improved this research: - : None: all sources were accessible with available tools. ``` This is informational only. Do your best with what's available and note what could have been better. Do not stop or block on tool gaps.