hermes-skills/social-search/SKILL.md

---
name: social-search
description: Build or extend the multi-source topic-search pipeline at /home/n8n/workspace/social/. Use when adding a new social source (Mastodon, Bluesky, Lemmy, PeerTube, etc.), wiring it into sources.yaml, debugging failed scrapers, or running a one-shot topic search across all configured sources.
version: 1.0.0
---

# social-search pipeline

A multi-source topic search at `/home/n8n/workspace/social/`. Each invocation is independent (no cron, no daemon). One .md per run lands in `results/`.

## Trigger

- Adding a new source to the pipeline
- Debugging a source that returns 403/timeout/etc.
- Running a manual search
- Discussing cron-later design (the system was built to be cron-safe)

## Layout

```
bin/social-search          the umbrella command (executable, venv shebang)
social/twitter.py          wraps twscrape CLI, formats tweets as Markdown
social/hackernews.py       Algolia HN search API → Markdown
social/lobsters.py         /hottest.json + client-side keyword filter → Markdown
social/rss.py              feedparser over feeds.yaml → Markdown
social/reddit.py           on disk but NOT in sources.yaml (403 from datacenter IPs)
sources.yaml               declarative source list — add a block, no code change
feeds.yaml                 RSS feed list (all commented by default)
results/                   one <datetime>.md per run
```

## How to add a source

1. Add an entry to `sources.yaml`. Required fields:
   ```yaml
   - name: mysource
     description: "human label"
     command: ["/home/n8n/workspace/social/.venv/bin/python", "-m", "social.mysource", "{query}"]
     timeout: 30
     limit: 25
   ```
   `{query}` is URL-encoded; `{query_raw}` is not.

2. If the source needs Python logic, create `social/mysource.py` that:
   - Reads topic from `sys.argv[1]`
   - Returns Markdown on stdout
   - Exits 0 on success, non-zero on failure
   - Uses `urllib.request` + standard library only (or imports from the venv site-packages)

3. The umbrella command runs sources in declared order. If any one fails, the run aborts and writes a partial .md with an ABORTED footer.

## How the umbrella works

`bin/social-search` is a single Python file. Key behaviors:

- Path-anchored: derives `WORKSPACE` from its own `__file__`, so it works regardless of cwd.
- Venv bootstrap: prepends `/home/n8n/workspace/social/.venv/lib/python3.13/site-packages` to `sys.path` so `social.X` modules resolve without activating the venv.
- Source invocation: `subprocess.run(cmd, capture_output=True, text=True, timeout=...)`. Any non-zero exit or timeout raises and aborts the run.
- Output filename: `YYYY-MM-DDTHH-MM-SS-<microseconds>-<tzoffset>.md` in local time. Microseconds prevent collisions when two runs land in the same second.

## Dependencies

Venv at `.venv/` has no `pip`/`ensurepip`. Install into it with:

```bash
/home/n8n/.local/bin/pip install --target=/home/n8n/workspace/social/.venv/lib/python3.13/site-packages <pkg>
```

Current packages: `requests`, `feedparser`, `pyyaml` (and their deps).

## Known dead ends (do not retry)

- **Reddit** JSON endpoint: returns 403 from datacenter IPs regardless of headers in 2025-2026. Do not retry with different UAs/proxies; the gate is at the network level. Either use a paid proxy service, a Playwright + real-cookie session, or skip Reddit entirely.
- **Lobsters** search endpoint: requires whitelisted params, rejects arbitrary query strings (HTTP 400). Use `https://lobste.rs/hottest.json` and filter client-side. Already implemented in `social/lobsters.py`.

## Commands

```bash
/home/n8n/workspace/social/bin/social-search "topic"           # all sources
/home/n8n/workspace/social/bin/social-search --source X "topic" # just one
/home/n8n/workspace/social/bin/social-search --list           # show configured
```

Exit codes: 0 = full success, 1 = source failure (partial file written), 2 = bad invocation (no topic, bad --source).

## Cron-later design notes

The scripts are cron-safe:
- Absolute paths everywhere
- No prompting, no input
- All output captured in the .md file (cron only sees stdout summary)
- Venv's python used via shebang — no `source activate` needed
- Per-source timeouts prevent hung runs

To cron: `*/30 * * * * /home/n8n/workspace/social/bin/social-search "topic-of-the-hour" >> /home/n8n/workspace/social/cron.log 2>&1` (pick the topic strategy separately).

## Communication style (user preference, enforced 2026-06-21)

When working on this pipeline or any task under this profile:

- **Concise always.** No long multi-section explanations. Trim to what's needed. The user will ask for detail if they want it.
- **Never show code unless explicitly requested.** Describe what changed, not the diff. The user can open the file.
- **Never assume or pick a default.** When offering choices, present them and wait. Picking for the user — even when a default seems obvious — triggers a strong negative correction. Treat this as a hard rule.

## Cross-model validation technique

After building or significantly changing the pipeline, validate with a different model. The minimax-m3 build was reviewed by deepseek-v4-pro, which caught seven issues (six bugs + one design gap) that the original model missed. The pattern:

1. Build with one model.
2. Hand all files to a second model with a skeptical review prompt ("find real bugs, no praise, verdict at end").
3. Fix everything the second model finds before declaring done.

This catches blind spots that a single model cannot see in its own output.

The user made explicit decisions in conversation that shaped this pipeline. They are preferences, not arbitrary choices — preserve them unless the user revises them.

- **No master on/off switch.** Every search invocation runs every source. The switch was discussed at length and dropped: the user wants "all on/working when search is done." Do not add a switch file or CLI without being asked.
- **Manual-only.** Nothing runs between invocations. No cron, no daemon, no idle poller, no background watcher. Scripts sit dormant until the user types `social-search "..."`.
- **Abort on any source error.** A single failed source stops the whole run. The partial .md is written with an ABORTED footer so the user knows exactly where it stopped. Do not change to "skip and continue" — the user explicitly preferred abort.
- **Local time in filename, UTC in header.** Filename is filename-safe (colons → dashes) + microseconds to prevent same-second collisions.
- **All files live under /home/n8n/workspace/social/.** No config in ~/.hermes/profiles/, no results elsewhere. Keeps the profile clean and the workspace self-contained.
- **Sources picked by "free + local + no API key + autonomously addable."** The user said pick what I can wire up without interviewing them; only sources needing credentials or content choices (Mastodon instance, YouTube channel list, etc.) were deferred.

## Pitfalls

- **Never re-test Reddit's JSON endpoint.** It returns 403 from datacenter IPs in 2025-2026 regardless of UA/headers/proxies. The reddit.py module is on disk but NOT in sources.yaml. Do not waste cycles trying.
- **Venv has no pip.** `/home/n8n/workspace/social/.venv/bin/` only contains `python`, `python3`, `python3.13`. No pip, no ensurepip. Always install via `/home/n8n/.local/bin/pip install --target=.../site-packages`.
- **Always run sources with absolute paths.** Cron and non-activated shells don't see `python` or `twscrape` from PATH. Hardcoded absolute paths in sources.yaml are deliberate.
- **Output filename collisions.** Two `social-search` invocations in the same second collide on the filename. Microsecond suffix prevents this — don't remove it.
- **Lobsters' /hottest.json returns ~50 stories max.** Don't promise deeper search; the implementation already filters client-side and notes this in its empty-state output.
- **X/Twitter cookies expire 2026-07-25.** The `accounts.db` cookie auth will silently break after that date. Re-extract from browser and run `twscrape add_cookie` to refresh.

## References

- `references/source-api-status.md` — tested API endpoints for each candidate source (Reddit, Lobsters, HN, RSS, twitter/twscrape) with HTTP status and notes on what's free/auth-free. Consult before adding a new source.