hermes-skills/social-media-scraping/SKILL.md

---
name: social-media-scraping
description: "Free, local social media scraping — X/Twitter via twscrape (cookie auth), Reddit, etc. Zero API fees, fully self-hosted."
version: 1.0.0
category: social-media
author: Hermes Agent
tags:
  - twitter
  - x
  - scraping
  - twscrape
  - free
  - local
  - social-media
  - cookies
---

# Social Media Scraping (Free & Local)

Read-only social media access without paid APIs. Uses browser cookies to authenticate against platforms' internal GraphQL endpoints — the same APIs the web apps use. Zero cost, fully local, no cloud dependencies.

**This skill covers free scraping tools. For the official paid X API, see the `xurl` skill.**

## When to Use

- Reading X/Twitter profiles, timelines, search results without paying for API access
- Monitoring accounts, collecting data, research
- Any read-only social media task where paid API keys are unacceptable

**Not for:** posting, liking, following, or any write operations. These tools are read-only by design.

## Primary Tool: twscrape (X/Twitter)

`twscrape` is an async Python library + CLI for X/Twitter's internal GraphQL API. Uses your own browser cookies (`auth_token` + `ct0`), stores sessions in local SQLite, and returns structured data.

- **Repo:** https://github.com/vladkens/twscrape
- **Install:** `pip install twscrape`
- **Version used:** 0.19.0 (June 2026)
- **Cost:** Free
- **Privacy:** Cookies stored locally in SQLite DB. No data leaves your machine.
- **GPU:** None — pure HTTP requests, zero inference.

### Setup

1. Install: `pip install twscrape`
2. Extract cookies from your browser while logged into x.com:
   - Open x.com → F12 → Application → Cookies
   - Copy `auth_token` and `ct0` values
3. Add account: `twscrape add_cookie my_account "auth_token=xxx; ct0=yyy"`
4. Verify: `twscrape user_by_login XDevelopers`

### Key Commands

| Action | Command |
|--------|---------|
| User lookup | `twscrape user_by_login <handle>` |
| User tweets | `twscrape user_tweets <user_id> --limit=20` |
| Search | `twscrape search "query" --limit=20` |
| Tweet details | `twscrape tweet_details <tweet_id>` |
| Tweet replies | `twscrape tweet_replies <tweet_id> --limit=20` |
| Following | `twscrape following <user_id> --limit=20` |
| Followers | `twscrape followers <user_id> --limit=20` |
| Trends | `twscrape trends` |
| List accounts | `twscrape accounts` |

All output is JSON. Use `--limit` to control result count.

### Python API

```python
import asyncio
from twscrape import API, gather

async def main():
    api = API()
    await api.pool.add_account_cookies("my_account", "auth_token=xxx; ct0=yyy")

    # User profile
    user = await api.user_by_login("XDevelopers")
    print(user.username, user.followersCount)

    # Search tweets
    tweets = await gather(api.search("python lang:en", limit=20))
    for t in tweets:
        print(t.id, t.rawContent)

asyncio.run(main())
```

## Workspace & Profile Convention

Each social media scraping project gets a dedicated Hermes profile + workspace:

- **Profile:** `~/.hermes/profiles/social/config.yaml` — model pinned to user's choice, `workspace: /workspace/social`
- **Workspace:** `/home/n8n/workspace/social/` — AGENTS.md with model, tool docs, read-only convention
- **Hindsight:** shared `~/.hermes/hindsight/config.json` (no per-profile config)

This keeps social media work isolated from other profiles and ensures the right model is used.

## Pitfalls

- **Cookie portability risk:** Cookies extracted from a laptop browser may trigger X security checks when used from a different IP (the LXC). This can cause CAPTCHA challenges, email verification, or forced logout. Works often for read-only, but no guarantee. Safest: use a dedicated X account for scraping.
- **Cookie expiration:** `auth_token` and `ct0` expire. If twscrape returns auth errors, re-extract fresh cookies from the browser.
- **Rate limiting:** X rate-limits GraphQL endpoints. twscrape handles this with account rotation (add multiple accounts to the pool). For single-account use, space out requests.
- **Read-only only:** twscrape cannot post, like, follow, or perform any write action. For writes, the paid X API (`xurl` skill) is the only official path.
- **Not a replacement for xurl:** `xurl` is for the official paid API with full read+write access. twscrape is for free read-only scraping. They serve different needs.
- **`add_cookie` is idempotent:** Re-adding the same account name just logs a warning — cookies are already stored. Use `twscrape del_accounts <name>` first if you need to replace credentials.

## Free & Local Mandate

This skill exists because the user requires free, local, self-hosted solutions. The official X API requires paid credits (minimum $5). twscrape is the free alternative. Always flag paid services before suggesting them — see `user-response-style` skill for the full rule.