hermes-skills/research-tooling/SKILL.md

---
name: research-tooling
description: Use when setting up, discovering, or installing research tools (Python libs, Docker services, MCP servers) for the Hermes research profile. Covers the curated tool inventory, batch-install patterns, and stack compatibility (Ollama + SearXNG + Qdrant).
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [research, tooling, setup, pip, docker, mcp]
    related_skills: [arxiv, blogwatcher, llm-wiki, political-research, polymarket]
---

# Research Tooling

## Overview

The Hermes research profile runs on a specific stack: Ollama (local LLMs), SearXNG (200+ search engines via MCP), Qdrant (vector DB), Firecrawl (web crawling), Spiderfoot (OSINT), and Playwright MCP (browser automation). This skill covers the Python, Docker, and MCP tools that extend this stack — what to install, what to skip, and how to batch-install efficiently.

The full curated inventory lives in `references/tools-inventory.md` — 42 tools installed, zero API keys, zero external services. Every cut has rationale.

## When to Use

- User asks "what research tools do we have" or "what can we install"
- Setting up a new research environment or restoring after a wipe
- Discovering tools for a specific research task (papers, scraping, OSINT, RAG, NLP)
- User wants to know what's already installed vs. what's missing

Don't use for: running individual tools (load the domain-specific skill instead — `arxiv`, `polymarket`, etc.), or for one-off `pip install` of a single package.

## Stack Baseline (already present)

| Component | Role |
|-----------|------|
| Ollama (11 models) | Local LLM inference |
| SearXNG MCP (10.0.0.8:8888) | Meta search: arxiv, pubmed, semantic scholar, google scholar, crossref, wikipedia, wolframalpha, hackernews, reddit, github, 200+ more |
| Playwright MCP | Browser automation for JS-heavy sites |
| Firecrawl (Docker) | Web crawling at scale, markdown output |
| Spiderfoot (Docker) | OSINT automation, 100+ data sources |
| Qdrant (qdrant-client) | Vector database |
| Hindsight (local Qdrant + vLLM) | Persistent memory (canonical since 2026-06-29) |
| Hermes Agent | Agent framework, 91 skills |

## Tool Categories

See `references/tools-inventory.md` for the full curated list with install commands, keep/drop rationale, and version pins. Quick reference:

| Category | Count | Key Tools |
|----------|-------|-----------|
| Deep Research Agents | 1 | local-deep-research |
| Academic Papers | 6 | arxiv, semanticscholar, scholarly, pymed, unpywall, crossrefapi |
| Web Scraping | 7 | crawl4ai, trafilatura, newspaper4k, markdownify, scrapy, feedparser, duckduckgo-search |
| Knowledge/RAG | 8 | docling, marker-pdf, pymupdf, pdfplumber, llama-index, graphrag, txtai, sentence-transformers |
| Data Analysis | 10 | pandas, plotly, networkx, pyvis, spacy, scikit-learn, bertopic, keybert, umap-learn, sumy |
| OSINT | 3 | maigret, theHarvester, holehe |
| News | 1 | gnews |
| Media | 3 | faster-whisper, youtube-transcript-api, yt-dlp |
| Reference | 3 | wikipedia, datasets, waybackpy |

## Batch Install Pattern

Install in parallel by category to minimize round-trips. Each batch is one `pip install --quiet` call with 5-8 packages:

```bash
# Batch 1: Deep Research Agent
pip install --quiet local-deep-research

# Batch 2: Academic Papers
pip install --quiet arxiv semanticscholar scholarly pymed unpywall crossrefapi

# Batch 3: Web Scraping
pip install --quiet crawl4ai trafilatura newspaper4k markdownify scrapy feedparser duckduckgo-search

# Batch 4: Document Processing & RAG
pip install --quiet docling marker-pdf pymupdf pdfplumber llama-index graphrag txtai sentence-transformers

# Batch 5: Data Analysis & Viz
pip install --quiet pandas plotly networkx pyvis spacy scikit-learn bertopic keybert umap-learn sumy

# Batch 6: OSINT
pip install --quiet maigret theHarvester holehe

# Batch 7: News, Media, Reference
pip install --quiet gnews yt-dlp wikipedia datasets
```

Run batches 1-4 in parallel (independent), then 5-7 in parallel. Total wall time ~2-3 minutes.

### Handling Metadata Errors

If a batch fails with `Encountered error while generating package metadata` for a package that's already installed (e.g., scikit-learn), it's a pip quirk — the package exists but pip's metadata resolver trips. Workaround: install the remaining packages in that batch individually, skipping the already-installed one.

### Version Incompatibility Warnings

`pip install` may warn about version conflicts (e.g., lxml, openai, requests). These are non-fatal — the packages install and function. Only act on them if a specific tool fails at runtime.

## Non-Pip Tools (NOT installed — user preference: no web services)

These require Docker, npx, or external service connections. The user has explicitly rejected these. Do NOT offer to install them unless the user specifically asks for RSS monitoring, MCP paper search, or changes their mind about web services.

| Tool | Command | Why skipped |
|------|---------|-------------|
| RSSHub | `docker run -d -p 1200:1200 diygod/rsshub` | Docker web service |
| Miniflux | `docker pull miniflux/miniflux` + Postgres | Docker web service |
| Fetch MCP | `npx @anthropic-ai/mcp-fetch` | External service |
| Paper Search MCP | `git clone` + MCP config | External service |

## Checking What's Installed

```bash
pip3 list 2>/dev/null | grep -iE '^(local-deep-research|gpt-researcher|arxiv|semanticscholar|scholarly|pymed|unpywall|crossrefapi|crawl4ai|trafilatura|newspaper4k|markdownify|scrapy|feedparser|duckduckgo-search|docling|marker-pdf|pymupdf|pdfplumber|llama-index|graphrag|txtai|sentence-transformers|pandas|plotly|networkx|pyvis|spacy|scikit-learn|bertopic|keybert|umap-learn|sumy|maigret|theHarvester|holehe|waybackpy|shodan|gnews|faster-whisper|youtube-transcript-api|yt-dlp|pyzotero|wikipedia|datasets) ' | sort
```

Compare against `references/tools-inventory.md` to identify gaps.

## Common Pitfalls

1. **Installing tools that need API keys.** The user wants zero API keys. shodan and pyzotero were removed for this reason. Before suggesting any new tool, verify it works without registration or API keys.

2. **Installing tools that need web services.** The user rejected Docker containers (RSSHub, Miniflux) and npx services (Fetch MCP, Paper Search MCP). Only pip-installable, fully local tools.

3. **Installing tools that overlap with SearXNG.** SearXNG already indexes arxiv, pubmed, semantic scholar, google scholar, crossref, and wikipedia. Don't install MCP wrappers that just re-expose these same sources — use SearXNG directly.

4. **Installing redundant tools.** The inventory explicitly drops 36 tools. Before installing anything not in the KEPT list, check the DROPPED rationale in `references/tools-inventory.md`.

5. **Worrying about pip version warnings.** `lxml`, `openai`, `requests` version conflicts are cosmetic. The tools work. Don't downgrade packages to silence warnings — that breaks other things.

6. **Forgetting the stack is already rich.** Before reaching for a new tool, check if SearXNG, Firecrawl, Spiderfoot, or Playwright MCP already cover the need.

7. **gpt-researcher was removed.** It defaults to OpenAI + Tavily (both need API keys). Reconfiguring it for Ollama + SearXNG is non-trivial. local-deep-research is the only deep research agent — it works out of the box with this stack.

8. **local-deep-research needs model configuration.** It defaults to Ollama provider but has NO model set. It will error until you pick one. Configure via web UI (Settings) or env var `LDR_LLM_MODEL=<model_name>`. See `references/local-deep-research-config.md` for full setup.

9. **SearXNG URL override needed.** LDR defaults SearXNG to `localhost:8080`. This environment's SearXNG is at `10.0.0.8:8888`. Set via `LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL=http://10.0.0.8:8888` or in the web UI.

10. **Model quality bottleneck.** Only qwen3:8b and granite4.1:3b are truly local models. Cloud-routed Ollama models (kimi-k2.7-code:cloud, deepseek-v4-pro:cloud, etc.) work with LDR but hit external APIs. For good research synthesis, use a cloud-routed model or add a stronger local model.

11. **Hermes toolsets that require API keys.** Several built-in Hermes toolsets need third-party API keys and are disabled in the research profile (`agent.disabled_toolsets` + removed from `platform_toolsets.cli`):

| Toolset | Required API key(s) | Why disabled |
|---------|---------------------|--------------|
| `web` | firecrawl/tavily/exa/parallel (for `web_extract`) | SearXNG is search-only; `web_extract` needs a paid extract backend. Use `searxng_web_url_read` (MCP) instead. |
| `image_gen` | `FAL_KEY` | Image generation via FAL.ai |
| `moa` | `OPENROUTER_API_KEY` | Mixture-of-agents synthesis |
| `rl` | `TINKER_API_KEY` + `WANDB_API_KEY` | RL training pipeline |

If the user ever adds API keys, re-enable via `hermes config set agent.disabled_toolsets` and `hermes config set platform_toolsets.cli`. Until then, these are dead weight — they appear in the tool list but fail at runtime.

## Verification Checklist

- [ ] `pip3 list | grep` shows expected tools installed
- [ ] No Docker containers started without explicit user request
- [ ] New installs don't duplicate SearXNG-covered sources
- [ ] `references/tools-inventory.md` is up to date if new tools were added/dropped
- [ ] For local-deep-research setup, see `references/local-deep-research-config.md`