156 lines
8.5 KiB
Markdown
156 lines
8.5 KiB
Markdown
---
|
|
name: searxng-smart-search
|
|
description: "Smart SearXNG MCP search with auto-category routing, score filtering, and time-range defaults."
|
|
version: 1.3.0
|
|
author: Hermes Agent
|
|
metadata:
|
|
hermes:
|
|
related_skills: [core-search, better-search, deep-research]
|
|
---
|
|
|
|
# SearXNG Smart Search
|
|
|
|
**USER PREFERENCE — CRITICAL:** ALL web searches MUST use `mcp_searxng_searxng_web_search`. The built-in `web_search` tool is NOT to be used under any circumstances. This is a hard preference — the user explicitly requires SearXNG MCP for all factual verification and research.
|
|
|
|
Use `mcp_searxng_searxng_web_search` with these defaults per query type. Never use `web_search`, `web_extract`, or curl — always route through MCP.
|
|
|
|
## Category Routing
|
|
|
|
| Query type | `categories` | `time_range` | `min_score` |
|
|
|---|---|---|---|
|
|
| Tech/code/ML | `it,science` | `month` | `0.2` |
|
|
| News/current | `news` | `week` | `0.3` |
|
|
| General/research | `general` | `year` | `0.2` |
|
|
| Academic/papers | `science` | `year` | `0.1` |
|
|
| Social/Reddit | `social media` | `month` | `0.2` |
|
|
| Images | `images` | — | — |
|
|
| Video | `videos` | — | — |
|
|
|
|
## Defaults
|
|
|
|
- `num_results`: 5 (general), 10 (research), 3 (quick lookup)
|
|
- `language`: `en` unless user specifies otherwise
|
|
- `safesearch`: `1` (moderate)
|
|
- `response_format`: `json` for programmatic, `text` for human reading
|
|
- **Global fallback**: `time_range=year`, `min_score=0.2` — apply to any query that doesn't match a specific category above
|
|
|
|
## Post-search
|
|
|
|
After search, use `mcp_searxng_web_url_read` on top 1-2 results for full content. Use `section` param to extract specific headings.
|
|
|
|
## Instance check
|
|
|
|
Run `mcp_searxng_searxng_instance_info` once per session to confirm engines are healthy. Flag any `unresponsive_engines` in results. Full engine inventory and known-broken engines documented in `references/searxng-instance.md`.
|
|
|
|
**PITFALL — Default engines may all be broken:** SearXNG uses server-side defaults. If all are broken, every search returns zero results with no error. Diagnose by hitting the SearXNG API directly:
|
|
|
|
```bash
|
|
# Check which engines actually work
|
|
curl -s "http://localhost:8888/search?q=test&format=json&engines=bing,google,duckduckgo,brave,startpage" | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'Results: {len(d.get(\"results\",[]))}'); print(f'Unresponsive: {d.get(\"unresponsive_engines\",[])}')"
|
|
```
|
|
|
|
If only Bing works, pass `engines=bing` in every `mcp_searxng_searxng_web_search` call until the server-side defaults are fixed. The MCP tool does NOT specify engines by default — it relies entirely on SearXNG's server-side defaults.
|
|
|
|
### `engines=` Parameter — Engine Diversity
|
|
|
|
The `mcp_searxng_searxng_web_search` tool accepts an `engines` parameter (comma-separated engine names). This is the primary bias-reduction mechanism — different search engines have different indexes and ranking algorithms. Use it to get genuinely different results:
|
|
|
|
```bash
|
|
# Mainstream Western indexes
|
|
engines=bing,google
|
|
|
|
# Independent crawlers (different indexes, different ranking)
|
|
engines=mojeek,wiby
|
|
|
|
# Privacy-focused
|
|
engines=duckduckgo,startpage
|
|
```
|
|
|
|
**Engine health varies.** On the local Docker instance (localhost:8888, 279 engines), commonly broken engines include: brave (rate-limited), qwant (access denied), yahoo (HTTP error), sogou (CAPTCHA), fastbot/fireball (suspended). These show as `unresponsive_engines` in results — SearXNG handles them gracefully. Check `searxng_instance_info` periodically for current health.
|
|
|
|
**Known working general-search engines (2026-07-15):** bing, google, duckduckgo, startpage, mojeek, wiby, yandex, mwmbl, yacy, yep, naver, baidu, seznam, gmx, crowdview, gabanza, searchmysite, tusksearch, vuhuv, searchch, reloado, resulthunter, boardreader, bpb, dogpile, quark, 360search, presearch.
|
|
|
|
## Reference files
|
|
|
|
- `references/instance-config.md` — full SearXNG instance snapshot (engines, categories, plugins, locales)
|
|
- `references/mcp-server-config.md` — MCP server setup, tool list, verification commands
|
|
- `references/searxng-instance.md` — engine inventory and known-broken engines
|
|
- `references/searxng-docker-setup.md` — Docker infrastructure, engine health, MCP bridge config, management commands
|
|
- `references/telegram-bot-api-2026.md` — Telegram Bot API features relevant to Hermes integration (overlaps with `telegram-integration/references/bot-api-2026.md` — curator will consolidate)
|
|
|
|
## Disabling the built-in `web` toolset (SearXNG-only mode)
|
|
|
|
To force ALL web searches through SearXNG MCP and prevent the agent from ever using the built-in `web_search`/`web_extract` tools, two config changes are required:
|
|
|
|
**1. Disable the `web` toolset globally:**
|
|
```yaml
|
|
# ~/.hermes/config.yaml
|
|
agent:
|
|
disabled_toolsets: ["web"]
|
|
```
|
|
|
|
**2. Remove `web` from platform toolsets (CLI and any other active platforms):**
|
|
```yaml
|
|
# ~/.hermes/config.yaml
|
|
platform_toolsets:
|
|
cli:
|
|
- browser
|
|
- clarify
|
|
- code_execution
|
|
# ... all others ...
|
|
- vision
|
|
# web REMOVED from this list
|
|
```
|
|
|
|
**3. Verify SearXNG is the configured backend (already set if you followed the MCP setup):**
|
|
```yaml
|
|
web:
|
|
backend: searxng
|
|
search_backend: searxng
|
|
extract_backend: searxng # Use SearXNG for page extraction too (consistent with MCP tools)
|
|
searxng_url: http://localhost:8888
|
|
```
|
|
|
|
With both `disabled_toolsets` and `platform_toolsets` updated, the `web_search` and `web_extract` tools are completely unavailable. The agent must use `mcp_searxng_searxng_web_search` and `mcp_searxng_web_url_read` for all web access.
|
|
|
|
Full reference config snippet: `references/disable-builtin-web.md`
|
|
|
|
## Setup / Troubleshooting
|
|
|
|
### `SEARXNG_URL` — Two Separate Requirements
|
|
|
|
The built-in `web_search` tool and the MCP SearXNG tools read `SEARXNG_URL` from DIFFERENT sources. Both must be set independently:
|
|
|
|
| Tool | Reads from | Config location |
|
|
|------|-----------|----------------|
|
|
| Built-in `web_search` | `os.environ` → `~/.hermes/.env` | `SEARXNG_URL=http://localhost:8888` in `.env` |
|
|
| MCP `searxng__*` tools | MCP subprocess env | `mcp_servers.searxng.env.SEARXNG_URL` in `config.yaml` |
|
|
|
|
**Critical pitfall — `config.yaml` `web.searxng_url` does NOT feed the built-in tool:** The built-in `web_search` tool's availability check (`_has_env("SEARXNG_URL")` in `tools/web_tools.py`) calls `get_env_value()` which reads `os.environ` first, then `~/.hermes/.env`. It does NOT read `web.searxng_url` from `config.yaml`. If `SEARXNG_URL` is only in `config.yaml` and not in `.env`, the built-in `web_search` returns empty results with no error — the tool silently falls through to "no provider configured."
|
|
|
|
**Diagnostic:** `web_search` returning empty `{"data": {"web": []}}` with no error message = `SEARXNG_URL` missing from `.env`. The MCP tools working fine at the same time confirms the instance is healthy — the gap is only in the built-in tool's env source.
|
|
|
|
**Fix:** Add `SEARXNG_URL=http://localhost:8888` to `~/.hermes/.env` AND all profile `.env` files. The MCP server config in `config.yaml` already passes `SEARXNG_URL` via its `env` block:
|
|
```yaml
|
|
mcp_servers:
|
|
searxng:
|
|
env:
|
|
SEARXNG_URL: http://localhost:8888
|
|
```
|
|
But this only covers the MCP subprocess — the main Hermes process still needs it in `.env` for the built-in tool.
|
|
|
|
**Verification after fix:** Start a new session (`/reset`) — env vars are read at startup. Then `web_search("test")` should return results. The MCP tools should continue working as before.
|
|
|
|
**Bulk `.env` update:** When adding `SEARXNG_URL` to `.env`, update ALL profile `.env` files (base + all 15 profiles). Use `hermes-config-bulk-update` skill pattern — same 16-location rule applies to `.env` as to `config.yaml`. Check with: `for p in ~/.hermes/profiles/*/; do grep -l SEARXNG_URL "$p/.env" 2>/dev/null || echo "MISSING: $(basename $p)"; done`
|
|
|
|
- DuckDuckGo and Startpage are working on the local Docker instance (localhost:8888). Brave and Qwant are rate-limited. Check `searxng_instance_info` for current health.
|
|
- `time_range` only works with `news` and `general` categories
|
|
- `min_score` above 0.5 often returns zero results
|
|
- `categories` must be comma-separated, no spaces: `it,science` not `it, science`
|
|
- `web_extract` (Hermes built-in) does NOT work with SearXNG backend — it needs firecrawl/tavily/exa. Use `mcp_searxng_web_url_read` instead for URL content extraction. When the built-in `web` toolset is disabled, this is enforced automatically.
|
|
|
|
## References
|
|
|
|
- `references/mcp-server-config.md` — full MCP server setup, tool list, verification commands
|
|
- `references/searxng-mcp-only-policy.md` — USER PREFERENCE: ALL web searches must use SearXNG MCP, never built-in web_search
|