180 lines
6.8 KiB
Markdown
180 lines
6.8 KiB
Markdown
---
|
|
name: deep-web-research
|
|
description: Exhaustive deep web research — five-move flow, external findings ledger, mechanical saturation check, disconfirmation, condensation from disk. Opt-in skill for the research profile.
|
|
version: 1.0.0
|
|
author: Hermes Agent
|
|
---
|
|
|
|
# Deep Web Research
|
|
|
|
Loaded explicitly via `-s deep-web-research`. Not loaded during normal interactive
|
|
use of the research profile. This skill overrides default behavior to enforce
|
|
exhaustive research methodology.
|
|
|
|
## Core Rules (Always in Effect)
|
|
|
|
These persist across all turns — they are in the skill, not in fading context:
|
|
|
|
- **Confined to /tmp.** All file writes go to `/tmp/`. Never write outside /tmp.
|
|
- **No self-provisioning.** Never install software. No pip, npm, apt, docker, or
|
|
any package manager. Use only what's already configured.
|
|
- **No repeat searches.** If you catch yourself searching the same thing twice,
|
|
stop. That sub-question is saturated.
|
|
- **Blacklist after 3 failures.** If a URL returns an error 3 times, blacklist it
|
|
and move on. Do not retry indefinitely.
|
|
- **Local and free only.** No internet-based paid services, no SaaS APIs with
|
|
billing, no metered endpoints. Use any tool already configured that fits this rule.
|
|
|
|
## Architecture: External Findings Ledger
|
|
|
|
At 200 turns, early findings scroll out of context. The ledger is the fix.
|
|
|
|
**Path:** `/tmp/research-<session-id>.md`
|
|
|
|
**Schema per finding:**
|
|
```
|
|
## Finding #<N>: <short label>
|
|
- Sub-Question: <which sub-question this belongs to>
|
|
- Claim: <the factual claim>
|
|
- Source: <URL> — <credibility tier: primary|secondary|aggregator|social>
|
|
- Confirmed by: <URL> (or "single-sourced")
|
|
- Date: <YYYY-MM-DD of the information, not of the search>
|
|
- Confidence: confirmed | likely | single-sourced | disputed
|
|
- Notes: <contradictions, caveats, context>
|
|
```
|
|
|
|
**Dedup rule:** Before writing a new finding, scan the ledger for an existing
|
|
finding with the same claim. If found, add the new source to "Confirmed by"
|
|
instead of creating a duplicate. Duplicate findings defeat the saturation check.
|
|
|
|
**Sub-question tagging:** Every finding must reference which sub-question it
|
|
belongs to. This enables the phase gate to verify coverage and the condensation
|
|
phase to group findings by topic.
|
|
|
|
## Five-Move Research Flow
|
|
|
|
### Move 1: Decompose
|
|
|
|
Break the question into sub-questions. Write them to the ledger with time
|
|
classifications. This makes "multiple angles" principled instead of random.
|
|
|
|
```
|
|
## Sub-Questions
|
|
1. <sub-question text> [time: week|month|year|none]
|
|
2. <sub-question text> [time: week|month|year|none]
|
|
```
|
|
|
|
The time classification lives on disk, not in fading context. Move 3 reads it
|
|
back when applying time filters per sub-question.
|
|
|
|
**Time classification guide:**
|
|
- **Current-state** (releases, prices, news, versions, events): `week` or `month`
|
|
- **Established-knowledge** (how something works, architecture, algorithms, history): `none` or `year`
|
|
|
|
### Move 2: Landscape Pass
|
|
|
|
Shallow-but-broad sweep. For each sub-question, 1-2 searches, read top 1-2 results.
|
|
Extract only: key sources, terms of art, where disagreement lives, major players.
|
|
No deep-diving yet. Append landscape notes to the ledger.
|
|
|
|
### Move 3: Deep-Dive per Sub-Question
|
|
|
|
For each sub-question:
|
|
- Search from 3+ angles with different categories and framings. Use
|
|
`mcp_searxng_searxng_web_search`. Apply the time filter from the ledger.
|
|
- Read full pages (5-10 per sub-question) via `mcp_searxng_web_url_read`.
|
|
Snippets are pointers, not sources.
|
|
- Follow citation trails: if a page cites a study/paper/dataset, go read that
|
|
source. Depth-first — one trail at a time, to its end, then the next.
|
|
- Extract structured data: tables, numbers, dates, versions, names.
|
|
- Assign credibility tier at capture time (not post-hoc in output).
|
|
- Cross-reference: every claim needs 2+ independent sources.
|
|
- Use browser tools (Playwright) when static extraction fails on JS-heavy pages.
|
|
- Append every finding to the ledger.
|
|
|
|
Citation trail stop conditions:
|
|
- Reached primary source (original paper, official docs, raw data)
|
|
- Dead end (paywall, 404, requires login)
|
|
- Circular reference (already read this source)
|
|
|
|
### Move 4: Disconfirmation Pass
|
|
|
|
Actively try to falsify each key claim:
|
|
- Search for "[claim] wrong", "[claim] criticism", "[claim] outdated"
|
|
- Check dates: is a 2024 source being presented as current?
|
|
- Look for contradicting evidence
|
|
- Append disconfirmation findings to the ledger
|
|
|
|
This is distinct from cross-referencing. Cross-reference confirms agreement.
|
|
Disconfirmation actively hunts for disagreement.
|
|
|
|
### Move 5: Condensation
|
|
|
|
Read the full ledger from disk. Read the phase gate file. If any box is
|
|
unchecked, do NOT condense — go back and complete that item.
|
|
|
|
Synthesize:
|
|
- Lead with the answer. No process narrative.
|
|
- Evidence-backed. Every claim tied to a source from the ledger.
|
|
- Concrete. Numbers, dates, names, specifics.
|
|
- Uncertainty explicit. "X confirmed by A and B. Y single-sourced from C.
|
|
Z unresolved — D and E conflict."
|
|
- Sources section: numbered list with URLs and credibility tiers.
|
|
|
|
## Guardrails
|
|
|
|
### Saturation Check (Mechanical)
|
|
|
|
Every 3-4 searches, run this command — do not self-assess:
|
|
```
|
|
grep -c "^## Finding #" /tmp/research-<sid>.md
|
|
```
|
|
Compare to the previous count. If zero new findings in the last 3 searches,
|
|
that sub-question is saturated. Move to the next. This is a measurable fact,
|
|
not a vibe.
|
|
|
|
### Phase Gate File
|
|
|
|
The phase gate is a file at `/tmp/research-<sid>-gate.md`. Write it and read it
|
|
back — it is not a mental checklist. Before moving to Move 5, read the gate file.
|
|
If any box is unchecked, do NOT condense. Go back and complete that item.
|
|
|
|
```
|
|
# Phase Gate
|
|
- [ ] All sub-questions have findings in the ledger
|
|
- [ ] Disconfirmation pass completed for key claims
|
|
- [ ] OR: saturation condition fired (no new findings in last 3 searches)
|
|
- [ ] OR: turn ceiling approaching (wrap up what you have)
|
|
```
|
|
|
|
Update this file as you progress.
|
|
|
|
### Turn Ceiling
|
|
|
|
200 turns is a safety net, not a target. Hitting the ceiling is a failure to
|
|
condense, not the normal path. If the ceiling is hit, deliver partial findings
|
|
with a note on what's missing.
|
|
|
|
## Tool Policy
|
|
|
|
**One rule: local and free only.** Use any tool already configured on the
|
|
research profile that fits this rule. No internet-based paid services, no SaaS
|
|
APIs with billing, no metered endpoints. You determine what fits — you are not
|
|
given a list of allowed or disallowed tools.
|
|
|
|
**No self-provisioning.** Never install, pull, or spin up new tools at runtime.
|
|
|
|
### Tools That Would Have Helped
|
|
|
|
At the end of the response (after Sources), add a section if any limitations
|
|
were hit:
|
|
|
|
```
|
|
Tools that would have improved this research:
|
|
- <tool name>: <what it would have enabled>
|
|
None: all sources were accessible with available tools.
|
|
```
|
|
|
|
This is informational only. Do your best with what's available and note what
|
|
could have been better. Do not stop or block on tool gaps.
|