hermes-skills/ecosystem-surveillance/SKILL.md

name, description, version, category, author

name	description	version	category	author
ecosystem-surveillance	Multi-platform ecosystem research and surveillance using searxng as primary search, supplemented by GitHub API, site-specific APIs, and direct page fetches. Deduplicates against Qdrant vectors and produces structured markdown research artifacts.	1.1.0	research	hermes

Ecosystem Surveillance Research

Trigger: user issues "fringe research", "deep research", "deep dive", or any request for exhaustive, multi-source investigation (phone numbers, people, companies, technologies, markets, etc.). They expect comprehensive coverage — multiple search angles, trail-following, scraping of result pages, no quitting on empty results. Deduplication against existing Qdrant vectors and a structured markdown research artifact as output.

"Deep research" is NOT a casual lookup. It means: try every angle, scrape result pages, follow links, cross-reference, and persist through empty results. The user does not care how long it takes. Never stop after 2-3 searches — that's a casual lookup, not deep research.

Core Search Strategy

SearXNG is the ONLY web search source. NEVER use the web_search tool (API provider). This is a hard rule, not a preference. If SearXNG fails, notify the user immediately — do not fall back to the web_search tool. Only use direct DuckDuckGo/Brave/Google as a last resort if the entire SearXNG ecosystem is down.

Always validate search paths before modifying or claiming knowledge. Confirm with a live search that the information is current — don't rely solely on internal knowledge or the long-term memory store.

All global rules are GLOBAL. When the user says "make global" or "prominent," the rule applies across all profiles and sessions, not just this one.

Known searxng instances (try in order)

Instance	URL	Notes
search.sapti.me	https://search.sapti.me	Rate-limits after a few queries; add delays
search.nousresearch.com	https://search.nousresearch.com	Internal — may be down
search.nadeau.lu	https://search.nadeau.lu	Luxembourg instance
search.mascot.xyz	https://search.mascot.xyz	Switzerland
searx.be	https://searx.be	Belgium; often overloaded
searx.fmac.xyz	https://searx.fmac.xyz	Community instance
searxng.site	https://searxng.site	Public

Query format

curl -s --max-time 15 \
  "https://<INSTANCE>/search?q=<URL_ENCODED_QUERY>&format=json" \
  | jq -r '.results[:10] | .[].url'

Example:

curl -s --max-time 15 \
  "https://search.sapti.me/search?q=AI+assisted+GPU+mining+RTX+4090&format=json" \
  | jq -r '.results[:5] | .[].url'

Fallback: direct search engine

When searxng is entirely unreachable, use DuckDuckGo direct:

curl -s "https://lite.duckduckgo.com/lite/?q=<QUERY>" \
  -A "Mozilla/5.0" \
  | grep -oE 'https?://[^"<>]+' \
  | grep -v duckduckgo

Or fetch known authoritative pages directly (whattomine.com, GitHub, etc.) without searching.

Platform Coverage Pattern

For "fringe research" class queries, cover in this order:

1. Searxng — broad web search for recent articles, forum posts, blog announcements
2. GitHub — code repos, open-source projects in the space (github.com/search?q=<query>&type=repositories)
3. Twitter/X — via xurl CLI or searxng with site:twitter.com or site:x.com
4. Reddit — via searxng with site:reddit.com
5. arXiv — academic papers on the topic (see arxiv skill)
6. Direct authoritative sources — project docs, official calculators, API endpoints

Deduplication

Before adding results to the final artifact:

• Check against existing Qdrant vectors in the ecosystem_research collection
• Normalize URLs (drop tracking params, ?utm_*, #fragments)
• Group by source domain; flag if same story appears across multiple platforms
• De-duplicate by content hash if fetching full pages

Qdrant Configuration

• Instance: http://10.0.0.22:6333
• Collection: ecosystem_research (1024-dim Cosine, created for this skill)
• Embedding model: snowflake-arctic-embed2:latest via Ollama at http://localhost:11434
• qdrant-client: already installed (v1.18.0)

Store research findings as vectors for future deduplication:

from qdrant_client import QdrantClient
client = QdrantClient(url="http://10.0.0.22:6333")
# Upsert with snowflake-arctic-embed2 embeddings (1024-dim)

Research Artifact Format

Produce structured markdown:

# Research: <Topic>
Date: <ISO date>
Sources: <count>

## Executive Summary
<2-3 sentences>

## Key Findings
### <Finding 1>
- Source: <URL>
- Date: <date>
- Summary: <brief>

### <Finding 2>
...

## Code / Projects
| Repo | Stars | Lang | Description |
|------|-------|------|-------------|
| ...  | ...   | ...  | ...         |

## Data Points
| Metric | Value | Source |
|--------|-------|--------|
| ...    | ...   | ...    |

## Gaps / Uncertainties
- <what we don't know yet>

Pitfalls

1. Searxng rate-limiting: Most public instances limit to ~1 req/sec. Add sleep 2 between queries or you get 429/Too Many Requests.
2. Don't search for things the user already told you: If the user says "we have an RTX 4090," don't search "does RTX 4090 exist?" Search for the thing they need about it (profitability, tools, config).
3. Instance cycling: If one searxng instance is down, try another immediately — do not give up after the first failure.
4. Narrow queries beat broad ones: "RTX 4090 mining profitability 2025" returns better results than "GPU crypto".
5. Browser tools are slow: For plain-text endpoints (.md, .txt, .json, raw GitHub), prefer curl in terminal. Use browser_navigate only for interactive JS-heavy pages.
6. Verify from multiple sources: Mining profitability numbers, prices, and difficulty change constantly. Cross-reference whattomine.com, 2cryptocalc.com, and nicehash.com before presenting a figure as current.
7. Empty results are NOT a stop signal: When a search returns nothing, that's the START of deep research, not the end. Try: different query formulations, different search engines, phone lookup sites, reverse directories, social media searches, WHOIS, cached pages, archive.org. The user said "deep research" — they expect persistence through empty results. Stopping after 2-3 searches is a casual lookup, not deep research.
8. Recognize the trigger immediately: "deep research", "deep dive", "fringe research" — these are NOT casual lookups. Load this skill on first sight of those phrases. Do not treat them as one-off web searches.

References

• references/search-rules.md — Complete SearXNG web search rules: hard constraints, instance details, validation pattern, and internal-knowledge vs web-search separation.
• references/search-instance-health.md — Known searxng instances and their uptime/status from recent sessions