Files

130 lines
9.0 KiB
Markdown
Raw Permalink Normal View History

---
name: core-search
description: "Default search method for profiles that want multi-source corroboration. Runs 4 parallel SearXNG calls with engine diversity and a counter-query, then cross-references results inline. ~5 turns. Replaces single-search for evaluative queries."
version: 1.0.0
author: Hermes Agent
metadata:
hermes:
tags: [search, corroboration, bias-reduction, multi-source, inline]
related_skills: [searxng-smart-search, better-search, deep-research]
---
# core-search — Multi-Source Corroboration Search
Replaces the default single-search behavior. Runs 4 parallel SearXNG calls with engine diversity and a counter-query, then cross-references results inline. More thorough than a single search, faster than better-search.
## Research Ladder
| Tier | Skill | Calls | Turns | Use case |
|------|-------|-------|-------|----------|
| 1 | searxng-smart-search (inline MCP) | 1 | 1 | Single factual lookup |
| **1.5** | **core-search (this skill)** | **4** | **~5** | **Default search — evaluative claims, bias-sensitive topics** |
| 2 | better-search (dispatched) | 3 loops | ~50 | Medium-depth research |
| 3 | deep-research (dispatched) | many | 200-600 | Exhaustive investigation |
## When This Skill Is Active
When loaded, this skill replaces the default search behavior. Every web search uses the 4-call pattern below. The bypass gate (Step 1) ensures simple factual queries still get a single search.
## Methodology
### Step 0 — Pre-Flight Engine Health Check
Before the 4 calls, run `searxng_instance_info` to confirm engine health. If <3 distinct engine families are healthy (e.g., only Bing working), skip the 4-call pattern and run a single general search with a note: "Engine diversity degraded — results may be biased toward a single index."
### Step 1 — Bypass Gate
Skip the 4-call pattern for queries where cross-checking adds no value. Bypass if the query is a settled fact with no reasonable dispute: definitions, unit conversions, translations, current time/weather, basic math, or single-fact lookups where the answer is universally agreed.
**Do NOT bypass based on string matching alone.** Queries like "what is the best way to deploy Kubernetes" match `what is` but are evaluative. If the query contains evaluative language (best, better, should, vs, safe, risk, cause, impact, worth, recommend), do NOT bypass even if it matches a factual pattern. Gate on contestability, not apparent simplicity.
### Step 2 — Formulate Counter-Query
Pick the right template based on query type:
| Query type | Counter template |
|------------|-----------------|
| Evaluative ("is X good/better/safe") | "criticisms of X" or "X problems drawbacks" |
| Comparative ("X vs Y") | "advantages of Y over X" or "why not X" |
| Tool/technique ("should I use X") | "X alternatives" or "X failure cases" |
| Claim ("does X cause Y") | "evidence against X causing Y" or "X debunked" |
| Policy/tech ("impact of X") | "X risks" or "X unintended consequences" |
| Trend/growth ("X growing/adoption") | "X decline" or "X plateau" or "X overhyped" |
| Procedural ("how to X") | "X alternatives" or "why not X" or "X drawbacks" |
| Superlative ("X is the best") | "X vs competitors" or "X alternatives comparison" |
| **Fallback (no template matches)** | "criticisms of X" or "X problems" |
### Step 3 — 4 Parallel SearXNG Calls (One Turn)
| # | Category | Engines | Query | Time | Limit |
|---|----------|---------|-------|------|-------|
| 1 | `general` | `bing,google,duckduckgo` | Q_neutral | adaptive (see below) | 5 |
| 2 | `news` | (default) | Q_neutral | `week` | 5 |
| 3 | `science` or `it` (routed) | (default) | Q_neutral | adaptive | 5 |
| 4 | `general` | `mojeek,wiby` | Q_counter | adaptive | 5 |
**Time-range adaptation:** Default to `year` for general queries. Drop the time restriction entirely (no `time_range` param) when the query is about: historical events, canonical references (RFCs, standards, foundational papers), or anything where the best sources are older than 12 months. Use `week` for breaking/current-event queries on all calls, not just news.
**News call gating:** Skip Call 2 (news) entirely if the query has no recency/event component — e.g., "how does X work," "history of Y," "best practices for Z." The news call is only valuable when timeliness matters.
**Engine resilience:** Call 1 includes `duckduckgo` as a fallback — if bing/google are rate-limited, duckduckgo still returns results. Call 4's mojeek/wiby are independent indexes but have weaker recall; if they return empty, note it as "independent-index gap" rather than "no counter-evidence found."
**Engine diversity:** Calls 1 and 4 both use `general` category but different engines. Bing/Google are mainstream Western indexes with link-graph ranking. Mojeek has its own independent index; Wiby targets non-commercial/small-web pages. Different indexes, different ranking algorithms, genuinely different results.
**Category routing for call 3:** Papers/theory/research → `science`. Libraries/tools/errors/code → `it`. When ambiguous, default to `science`.
### Step 4 — Deduplicate
1. Normalize URLs: strip scheme, `www.`, trailing slash, tracking params, fragments.
2. Deduplicate by exact normalized URL.
3. **Syndication check:** if 2+ results have >80% title similarity and similar snippet text, treat as one source. Prevents counting the same article on 5 domains as "corroborated."
### Step 5 — Read & Cross-Reference
- Read snippets from all unique results first.
- Full-page read (`mcp_searxng_web_url_read`) only for 2-3 load-bearing sources where the snippet is ambiguous.
- Cross-reference claims across sources.
### Step 6 — Structured Output
```
Claim: [main finding]
Supporting evidence:
- [source] — [key point]
Contradicting evidence:
- [source] — [key point] (or "none found")
Confidence: corroborated | single-source | contested | low-coverage
```
**Confidence labels:**
- **corroborated** — ≥2 sources from different engine groups agree, AND the sources trace to different primary origins (not the same press release, Wikipedia paragraph, or syndicated wire story). Domain diversity alone is not enough — check whether the sources are genuinely independent or just reprinting the same upstream fact.
- **single-source** — one source, or multiple from same engine group/syndication, or all agreeing sources trace to a single origin
- **contested** — sources disagree on the main claim. Present both sides with their evidence. Explicitly suggest escalating to better-search for resolution.
- **low-coverage** — <3 unique domains after dedup, or all calls returned thin results. Flag and suggest escalating to better-search.
### Step 7 — Quality Floor
- **Distinguish error-empty from results-empty.** If a call returns 0 results, check whether the engines were unresponsive (check `unresponsive_engines` in the response) vs genuinely finding nothing. An engine outage is NOT the same as "no evidence." If engines 429'd or timed out, note it explicitly: "Call N degraded — [engine] unavailable, results may be incomplete." Never report an engine failure as an epistemic finding.
- If any call returns 0 results, note it.
- If after dedup there are <3 unique domains, mark `low-coverage`.
- If counter-query results are all low-quality (content farms, SEO spam), de-weight them.
- **No conditional follow-up search.** The 4-call pattern is the contract. If inconclusive, suggest better-search.
## Pitfalls
- **Engine health:** Brave and Qwant are rate-limited on the current instance. Yahoo, fastbot, fireball, infospace, searchtoday, zapmeta, encyclosearch are permanently broken. Mojeek and Wiby are working. Check `searxng_instance_info` periodically.
- **Counter-query quality:** "X problems drawbacks" can surface SEO spam for polarizing topics. The Q_counter templates help, but evaluate source quality.
- **Syndication illusion:** URL dedup alone misses the same story on different domains. Title-similarity check is the mitigation.
- **Not one turn:** The 4 searches are one turn, but dedup + reading + synthesis add 3-4 more. This is ~5 turns total.
- **Engine degradation:** If the SearXNG instance is degraded (only one engine working), all 4 calls collapse into the same engine pool. Bias reduction becomes cosmetic.
- **Structured data / database-backed queries:** This methodology is designed for evaluative claims with web-accessible evidence (articles, docs, discussions). It fails for queries where the answer lives in structured databases behind JavaScript paywalls — financial performance data, sports statistics, live pricing, API-backed dashboards. If the query is about numbers that live in a database (fund returns, stock prices, sports scores), general web search is the wrong tool. Flag early: "This data lives in structured databases, not web pages. A direct financial/sports/API data source would be more reliable." Then run a single search for context and suggest the user use a dedicated data source.
## Dependencies
- `searxng-smart-search` skill (for category routing table, min_score defaults, engine health)
- `mcp_searxng_searxng_web_search` tool
- `mcp_searxng_web_url_read` tool