hermes-skills/ecosystem-surveillance/SKILL.md

---
name: ecosystem-surveillance
description: Multi-platform ecosystem research and surveillance using searxng as primary search, supplemented by GitHub API, site-specific APIs, and direct page fetches. Deduplicates against Qdrant vectors and produces structured markdown research artifacts.
version: 1.1.0
category: research
author: hermes
---

# Ecosystem Surveillance Research

Trigger: user issues "fringe research", "deep research", "deep dive", or any request for exhaustive, multi-source investigation (phone numbers, people, companies, technologies, markets, etc.). They expect comprehensive coverage — multiple search angles, trail-following, scraping of result pages, no quitting on empty results. Deduplication against existing Qdrant vectors and a structured markdown research artifact as output.

**"Deep research" is NOT a casual lookup.** It means: try every angle, scrape result pages, follow links, cross-reference, and persist through empty results. The user does not care how long it takes. Never stop after 2-3 searches — that's a casual lookup, not deep research.

## Core Search Strategy

**SearXNG is the ONLY web search source. NEVER use the web_search tool (API provider).** This is a hard rule, not a preference. If SearXNG fails, notify the user immediately — do not fall back to the web_search tool. Only use direct DuckDuckGo/Brave/Google as a last resort if the entire SearXNG ecosystem is down.

**Always validate search paths before modifying or claiming knowledge.** Confirm with a live search that the information is current — don't rely solely on internal knowledge or the long-term memory store.

**All global rules are GLOBAL.** When the user says "make global" or "prominent," the rule applies across all profiles and sessions, not just this one.

### Known searxng instances (try in order)

| Instance | URL | Notes |
|----------|-----|-------|
| search.sapti.me | `https://search.sapti.me` | Rate-limits after a few queries; add delays |
| search.nousresearch.com | `https://search.nousresearch.com` | Internal — may be down |
| search.nadeau.lu | `https://search.nadeau.lu` | Luxembourg instance |
| search.mascot.xyz | `https://search.mascot.xyz` | Switzerland |
| searx.be | `https://searx.be` | Belgium; often overloaded |
| searx.fmac.xyz | `https://searx.fmac.xyz` | Community instance |
| searxng.site | `https://searxng.site` | Public |

### Query format

```bash
curl -s --max-time 15 \
  "https://<INSTANCE>/search?q=<URL_ENCODED_QUERY>&format=json" \
  | jq -r '.results[:10] | .[].url'
```

Example:
```bash
curl -s --max-time 15 \
  "https://search.sapti.me/search?q=AI+assisted+GPU+mining+RTX+4090&format=json" \
  | jq -r '.results[:5] | .[].url'
```

### Fallback: direct search engine

When searxng is entirely unreachable, use DuckDuckGo direct:
```bash
curl -s "https://lite.duckduckgo.com/lite/?q=<QUERY>" \
  -A "Mozilla/5.0" \
  | grep -oE 'https?://[^"<>]+' \
  | grep -v duckduckgo
```

Or fetch known authoritative pages directly (whattomine.com, GitHub, etc.) without searching.

## Platform Coverage Pattern

For "fringe research" class queries, cover in this order:

1. **Searxng** — broad web search for recent articles, forum posts, blog announcements
2. **GitHub** — code repos, open-source projects in the space (`github.com/search?q=<query>&type=repositories`)
3. **Twitter/X** — via `xurl` CLI or searxng with `site:twitter.com` or `site:x.com`
4. **Reddit** — via searxng with `site:reddit.com`
5. **arXiv** — academic papers on the topic (see `arxiv` skill)
6. **Direct authoritative sources** — project docs, official calculators, API endpoints

## Deduplication

Before adding results to the final artifact:
- Check against existing Qdrant vectors in the `ecosystem_research` collection
- Normalize URLs (drop tracking params, `?utm_*`, `#fragments`)
- Group by source domain; flag if same story appears across multiple platforms
- De-duplicate by content hash if fetching full pages

### Qdrant Configuration

- **Instance**: `http://10.0.0.22:6333`
- **Collection**: `ecosystem_research` (1024-dim Cosine, created for this skill)
- **Embedding model**: `snowflake-arctic-embed2:latest` via Ollama at `http://localhost:11434`
- **qdrant-client**: already installed (v1.18.0)

Store research findings as vectors for future deduplication:
```python
from qdrant_client import QdrantClient
client = QdrantClient(url="http://10.0.0.22:6333")
# Upsert with snowflake-arctic-embed2 embeddings (1024-dim)
```

## Research Artifact Format

Produce structured markdown:

```markdown
# Research: <Topic>
Date: <ISO date>
Sources: <count>

## Executive Summary
<2-3 sentences>

## Key Findings
### <Finding 1>
- Source: <URL>
- Date: <date>
- Summary: <brief>

### <Finding 2>
...

## Code / Projects
| Repo | Stars | Lang | Description |
|------|-------|------|-------------|
| ...  | ...   | ...  | ...         |

## Data Points
| Metric | Value | Source |
|--------|-------|--------|
| ...    | ...   | ...    |

## Gaps / Uncertainties
- <what we don't know yet>
```

## Pitfalls

1. **Searxng rate-limiting**: Most public instances limit to ~1 req/sec. Add `sleep 2` between queries or you get 429/Too Many Requests.
2. **Don't search for things the user already told you**: If the user says "we have an RTX 4090," don't search "does RTX 4090 exist?" Search for the thing they need *about* it (profitability, tools, config).
3. **Instance cycling**: If one searxng instance is down, try another immediately — do not give up after the first failure.
4. **Narrow queries beat broad ones**: `"RTX 4090 mining profitability 2025"` returns better results than `"GPU crypto"`.
5. **Browser tools are slow**: For plain-text endpoints (`.md`, `.txt`, `.json`, raw GitHub), prefer `curl` in terminal. Use `browser_navigate` only for interactive JS-heavy pages.
6. **Verify from multiple sources**: Mining profitability numbers, prices, and difficulty change constantly. Cross-reference whattomine.com, 2cryptocalc.com, and nicehash.com before presenting a figure as current.
7. **Empty results are NOT a stop signal**: When a search returns nothing, that's the START of deep research, not the end. Try: different query formulations, different search engines, phone lookup sites, reverse directories, social media searches, WHOIS, cached pages, archive.org. The user said "deep research" — they expect persistence through empty results. Stopping after 2-3 searches is a casual lookup, not deep research.
8. **Recognize the trigger immediately**: "deep research", "deep dive", "fringe research" — these are NOT casual lookups. Load this skill on first sight of those phrases. Do not treat them as one-off web searches.

## References

- `references/search-rules.md` — Complete SearXNG web search rules: hard constraints, instance details, validation pattern, and internal-knowledge vs web-search separation.
- `references/search-instance-health.md` — Known searxng instances and their uptime/status from recent sessions