diff --git a/deep-web-research/SKILL.md b/deep-web-research/SKILL.md new file mode 100644 index 0000000..b9b9729 --- /dev/null +++ b/deep-web-research/SKILL.md @@ -0,0 +1,179 @@ +--- +name: deep-web-research +description: Exhaustive deep web research — five-move flow, external findings ledger, mechanical saturation check, disconfirmation, condensation from disk. Opt-in skill for the research profile. +version: 1.0.0 +author: Hermes Agent +--- + +# Deep Web Research + +Loaded explicitly via `-s deep-web-research`. Not loaded during normal interactive +use of the research profile. This skill overrides default behavior to enforce +exhaustive research methodology. + +## Core Rules (Always in Effect) + +These persist across all turns — they are in the skill, not in fading context: + +- **Confined to /tmp.** All file writes go to `/tmp/`. Never write outside /tmp. +- **No self-provisioning.** Never install software. No pip, npm, apt, docker, or + any package manager. Use only what's already configured. +- **No repeat searches.** If you catch yourself searching the same thing twice, + stop. That sub-question is saturated. +- **Blacklist after 3 failures.** If a URL returns an error 3 times, blacklist it + and move on. Do not retry indefinitely. +- **Local and free only.** No internet-based paid services, no SaaS APIs with + billing, no metered endpoints. Use any tool already configured that fits this rule. + +## Architecture: External Findings Ledger + +At 200 turns, early findings scroll out of context. The ledger is the fix. + +**Path:** `/tmp/research-.md` + +**Schema per finding:** +``` +## Finding #: +- Sub-Question: +- Claim: +- Source: — +- Confirmed by: (or "single-sourced") +- Date: +- Confidence: confirmed | likely | single-sourced | disputed +- Notes: +``` + +**Dedup rule:** Before writing a new finding, scan the ledger for an existing +finding with the same claim. If found, add the new source to "Confirmed by" +instead of creating a duplicate. Duplicate findings defeat the saturation check. + +**Sub-question tagging:** Every finding must reference which sub-question it +belongs to. This enables the phase gate to verify coverage and the condensation +phase to group findings by topic. + +## Five-Move Research Flow + +### Move 1: Decompose + +Break the question into sub-questions. Write them to the ledger with time +classifications. This makes "multiple angles" principled instead of random. + +``` +## Sub-Questions +1. [time: week|month|year|none] +2. [time: week|month|year|none] +``` + +The time classification lives on disk, not in fading context. Move 3 reads it +back when applying time filters per sub-question. + +**Time classification guide:** +- **Current-state** (releases, prices, news, versions, events): `week` or `month` +- **Established-knowledge** (how something works, architecture, algorithms, history): `none` or `year` + +### Move 2: Landscape Pass + +Shallow-but-broad sweep. For each sub-question, 1-2 searches, read top 1-2 results. +Extract only: key sources, terms of art, where disagreement lives, major players. +No deep-diving yet. Append landscape notes to the ledger. + +### Move 3: Deep-Dive per Sub-Question + +For each sub-question: +- Search from 3+ angles with different categories and framings. Use + `mcp_searxng_searxng_web_search`. Apply the time filter from the ledger. +- Read full pages (5-10 per sub-question) via `mcp_searxng_web_url_read`. + Snippets are pointers, not sources. +- Follow citation trails: if a page cites a study/paper/dataset, go read that + source. Depth-first — one trail at a time, to its end, then the next. +- Extract structured data: tables, numbers, dates, versions, names. +- Assign credibility tier at capture time (not post-hoc in output). +- Cross-reference: every claim needs 2+ independent sources. +- Use browser tools (Playwright) when static extraction fails on JS-heavy pages. +- Append every finding to the ledger. + +Citation trail stop conditions: +- Reached primary source (original paper, official docs, raw data) +- Dead end (paywall, 404, requires login) +- Circular reference (already read this source) + +### Move 4: Disconfirmation Pass + +Actively try to falsify each key claim: +- Search for "[claim] wrong", "[claim] criticism", "[claim] outdated" +- Check dates: is a 2024 source being presented as current? +- Look for contradicting evidence +- Append disconfirmation findings to the ledger + +This is distinct from cross-referencing. Cross-reference confirms agreement. +Disconfirmation actively hunts for disagreement. + +### Move 5: Condensation + +Read the full ledger from disk. Read the phase gate file. If any box is +unchecked, do NOT condense — go back and complete that item. + +Synthesize: +- Lead with the answer. No process narrative. +- Evidence-backed. Every claim tied to a source from the ledger. +- Concrete. Numbers, dates, names, specifics. +- Uncertainty explicit. "X confirmed by A and B. Y single-sourced from C. + Z unresolved — D and E conflict." +- Sources section: numbered list with URLs and credibility tiers. + +## Guardrails + +### Saturation Check (Mechanical) + +Every 3-4 searches, run this command — do not self-assess: +``` +grep -c "^## Finding #" /tmp/research-.md +``` +Compare to the previous count. If zero new findings in the last 3 searches, +that sub-question is saturated. Move to the next. This is a measurable fact, +not a vibe. + +### Phase Gate File + +The phase gate is a file at `/tmp/research--gate.md`. Write it and read it +back — it is not a mental checklist. Before moving to Move 5, read the gate file. +If any box is unchecked, do NOT condense. Go back and complete that item. + +``` +# Phase Gate +- [ ] All sub-questions have findings in the ledger +- [ ] Disconfirmation pass completed for key claims +- [ ] OR: saturation condition fired (no new findings in last 3 searches) +- [ ] OR: turn ceiling approaching (wrap up what you have) +``` + +Update this file as you progress. + +### Turn Ceiling + +200 turns is a safety net, not a target. Hitting the ceiling is a failure to +condense, not the normal path. If the ceiling is hit, deliver partial findings +with a note on what's missing. + +## Tool Policy + +**One rule: local and free only.** Use any tool already configured on the +research profile that fits this rule. No internet-based paid services, no SaaS +APIs with billing, no metered endpoints. You determine what fits — you are not +given a list of allowed or disallowed tools. + +**No self-provisioning.** Never install, pull, or spin up new tools at runtime. + +### Tools That Would Have Helped + +At the end of the response (after Sources), add a section if any limitations +were hit: + +``` +Tools that would have improved this research: +- : +None: all sources were accessible with available tools. +``` + +This is informational only. Do your best with what's available and note what +could have been better. Do not stop or block on tool gaps.