hermes-skills/social-media-scraping/SKILL.md

name, description, version, category, author, tags

name	description	version	category	author	tags
social-media-scraping	Free, local social media scraping — X/Twitter via twscrape (cookie auth), Reddit, etc. Zero API fees, fully self-hosted.	1.0.0	social-media	Hermes Agent	twitter x scraping twscrape free local social-media cookies

Social Media Scraping (Free & Local)

Read-only social media access without paid APIs. Uses browser cookies to authenticate against platforms' internal GraphQL endpoints — the same APIs the web apps use. Zero cost, fully local, no cloud dependencies.

This skill covers free scraping tools. For the official paid X API, see the xurl skill.

When to Use

• Reading X/Twitter profiles, timelines, search results without paying for API access
• Monitoring accounts, collecting data, research
• Any read-only social media task where paid API keys are unacceptable

Not for: posting, liking, following, or any write operations. These tools are read-only by design.

Primary Tool: twscrape (X/Twitter)

twscrape is an async Python library + CLI for X/Twitter's internal GraphQL API. Uses your own browser cookies (auth_token + ct0), stores sessions in local SQLite, and returns structured data.

• Repo: https://github.com/vladkens/twscrape
• Install: pip install twscrape
• Version used: 0.19.0 (June 2026)
• Cost: Free
• Privacy: Cookies stored locally in SQLite DB. No data leaves your machine.
• GPU: None — pure HTTP requests, zero inference.

Setup

1. Install: pip install twscrape
2. Extract cookies from your browser while logged into x.com:
   • Open x.com → F12 → Application → Cookies
   • Copy auth_token and ct0 values
3. Add account: twscrape add_cookie my_account "auth_token=xxx; ct0=yyy"
4. Verify: twscrape user_by_login XDevelopers

Key Commands

Action	Command
User lookup	twscrape user_by_login <handle>
User tweets	twscrape user_tweets <user_id> --limit=20
Search	twscrape search "query" --limit=20
Tweet details	twscrape tweet_details <tweet_id>
Tweet replies	twscrape tweet_replies <tweet_id> --limit=20
Following	twscrape following <user_id> --limit=20
Followers	twscrape followers <user_id> --limit=20
Trends	twscrape trends
List accounts	twscrape accounts

All output is JSON. Use --limit to control result count.

Python API

import asyncio
from twscrape import API, gather

async def main():
    api = API()
    await api.pool.add_account_cookies("my_account", "auth_token=xxx; ct0=yyy")

    # User profile
    user = await api.user_by_login("XDevelopers")
    print(user.username, user.followersCount)

    # Search tweets
    tweets = await gather(api.search("python lang:en", limit=20))
    for t in tweets:
        print(t.id, t.rawContent)

asyncio.run(main())

Workspace & Profile Convention

Each social media scraping project gets a dedicated Hermes profile + workspace:

• Profile: ~/.hermes/profiles/social/config.yaml — model pinned to user's choice, workspace: /workspace/social
• Workspace: /home/n8n/workspace/social/ — AGENTS.md with model, tool docs, read-only convention
• Hindsight: shared ~/.hermes/hindsight/config.json (no per-profile config)

This keeps social media work isolated from other profiles and ensures the right model is used.

Pitfalls

• Cookie portability risk: Cookies extracted from a laptop browser may trigger X security checks when used from a different IP (the LXC). This can cause CAPTCHA challenges, email verification, or forced logout. Works often for read-only, but no guarantee. Safest: use a dedicated X account for scraping.
• Cookie expiration: auth_token and ct0 expire. If twscrape returns auth errors, re-extract fresh cookies from the browser.
• Rate limiting: X rate-limits GraphQL endpoints. twscrape handles this with account rotation (add multiple accounts to the pool). For single-account use, space out requests.
• Read-only only: twscrape cannot post, like, follow, or perform any write action. For writes, the paid X API (xurl skill) is the only official path.
• Not a replacement for xurl: xurl is for the official paid API with full read+write access. twscrape is for free read-only scraping. They serve different needs.
• add_cookie is idempotent: Re-adding the same account name just logs a warning — cookies are already stored. Use twscrape del_accounts <name> first if you need to replace credentials.

Free & Local Mandate

This skill exists because the user requires free, local, self-hosted solutions. The official X API requires paid credits (minimum $5). twscrape is the free alternative. Always flag paid services before suggesting them — see user-response-style skill for the full rule.