Add all custom Hermes skills from general profile (33 skills)
agent-workflows: workspace-context-organization autonomous-ai-agents: hermes-agent computer-use devops: hermes-config-bulk-update, hermes-profile-management, holographic-memory, telegram-integration, webhook-subscriptions email: himalaya mcp: native-mcp, searxng-smart-search media: voice-systems, youtube-content mlops: local-vector-memory, qdrant-collection-management productivity: maps, notion, ocr-and-documents project-knowledge-base research: arxiv, blogwatcher, ecosystem-surveillance save-agents-md social-media: social-media-scraping, xurl software-development: agent-self-audit, simplify-code, spike, subagent-driven-development, systematic-debugging, test-driven-development, writing-plans user-response-style
This commit is contained in:
Hermes Agent
111
social-media-scraping/SKILL.md
Normal file
111
social-media-scraping/SKILL.md
Normal file
@@ -0,0 +1,111 @@
|
||||
---
|
||||
name: social-media-scraping
|
||||
description: "Free, local social media scraping — X/Twitter via twscrape (cookie auth), Reddit, etc. Zero API fees, fully self-hosted."
|
||||
version: 1.0.0
|
||||
category: social-media
|
||||
author: Hermes Agent
|
||||
tags:
|
||||
- twitter
|
||||
- x
|
||||
- scraping
|
||||
- twscrape
|
||||
- free
|
||||
- local
|
||||
- social-media
|
||||
- cookies
|
||||
---
|
||||
|
||||
# Social Media Scraping (Free & Local)
|
||||
|
||||
Read-only social media access without paid APIs. Uses browser cookies to authenticate against platforms' internal GraphQL endpoints — the same APIs the web apps use. Zero cost, fully local, no cloud dependencies.
|
||||
|
||||
**This skill covers free scraping tools. For the official paid X API, see the `xurl` skill.**
|
||||
|
||||
## When to Use
|
||||
|
||||
- Reading X/Twitter profiles, timelines, search results without paying for API access
|
||||
- Monitoring accounts, collecting data, research
|
||||
- Any read-only social media task where paid API keys are unacceptable
|
||||
|
||||
**Not for:** posting, liking, following, or any write operations. These tools are read-only by design.
|
||||
|
||||
## Primary Tool: twscrape (X/Twitter)
|
||||
|
||||
`twscrape` is an async Python library + CLI for X/Twitter's internal GraphQL API. Uses your own browser cookies (`auth_token` + `ct0`), stores sessions in local SQLite, and returns structured data.
|
||||
|
||||
- **Repo:** https://github.com/vladkens/twscrape
|
||||
- **Install:** `pip install twscrape`
|
||||
- **Version used:** 0.19.0 (June 2026)
|
||||
- **Cost:** Free
|
||||
- **Privacy:** Cookies stored locally in SQLite DB. No data leaves your machine.
|
||||
- **GPU:** None — pure HTTP requests, zero inference.
|
||||
|
||||
### Setup
|
||||
|
||||
1. Install: `pip install twscrape`
|
||||
2. Extract cookies from your browser while logged into x.com:
|
||||
- Open x.com → F12 → Application → Cookies
|
||||
- Copy `auth_token` and `ct0` values
|
||||
3. Add account: `twscrape add_cookie my_account "auth_token=xxx; ct0=yyy"`
|
||||
4. Verify: `twscrape user_by_login XDevelopers`
|
||||
|
||||
### Key Commands
|
||||
|
||||
| Action | Command |
|
||||
|--------|---------|
|
||||
| User lookup | `twscrape user_by_login <handle>` |
|
||||
| User tweets | `twscrape user_tweets <user_id> --limit=20` |
|
||||
| Search | `twscrape search "query" --limit=20` |
|
||||
| Tweet details | `twscrape tweet_details <tweet_id>` |
|
||||
| Tweet replies | `twscrape tweet_replies <tweet_id> --limit=20` |
|
||||
| Following | `twscrape following <user_id> --limit=20` |
|
||||
| Followers | `twscrape followers <user_id> --limit=20` |
|
||||
| Trends | `twscrape trends` |
|
||||
| List accounts | `twscrape accounts` |
|
||||
|
||||
All output is JSON. Use `--limit` to control result count.
|
||||
|
||||
### Python API
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from twscrape import API, gather
|
||||
|
||||
async def main():
|
||||
api = API()
|
||||
await api.pool.add_account_cookies("my_account", "auth_token=xxx; ct0=yyy")
|
||||
|
||||
# User profile
|
||||
user = await api.user_by_login("XDevelopers")
|
||||
print(user.username, user.followersCount)
|
||||
|
||||
# Search tweets
|
||||
tweets = await gather(api.search("python lang:en", limit=20))
|
||||
for t in tweets:
|
||||
print(t.id, t.rawContent)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Workspace & Profile Convention
|
||||
|
||||
Each social media scraping project gets a dedicated Hermes profile + workspace:
|
||||
|
||||
- **Profile:** `~/.hermes/profiles/social/config.yaml` — model pinned to user's choice, `workspace: /workspace/social`
|
||||
- **Workspace:** `/home/n8n/workspace/social/` — AGENTS.md with model, tool docs, read-only convention
|
||||
- **Hindsight:** shared `~/.hermes/hindsight/config.json` (no per-profile config)
|
||||
|
||||
This keeps social media work isolated from other profiles and ensures the right model is used.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Cookie portability risk:** Cookies extracted from a laptop browser may trigger X security checks when used from a different IP (the LXC). This can cause CAPTCHA challenges, email verification, or forced logout. Works often for read-only, but no guarantee. Safest: use a dedicated X account for scraping.
|
||||
- **Cookie expiration:** `auth_token` and `ct0` expire. If twscrape returns auth errors, re-extract fresh cookies from the browser.
|
||||
- **Rate limiting:** X rate-limits GraphQL endpoints. twscrape handles this with account rotation (add multiple accounts to the pool). For single-account use, space out requests.
|
||||
- **Read-only only:** twscrape cannot post, like, follow, or perform any write action. For writes, the paid X API (`xurl` skill) is the only official path.
|
||||
- **Not a replacement for xurl:** `xurl` is for the official paid API with full read+write access. twscrape is for free read-only scraping. They serve different needs.
|
||||
- **`add_cookie` is idempotent:** Re-adding the same account name just logs a warning — cookies are already stored. Use `twscrape del_accounts <name>` first if you need to replace credentials.
|
||||
|
||||
## Free & Local Mandate
|
||||
|
||||
This skill exists because the user requires free, local, self-hosted solutions. The official X API requires paid credits (minimum $5). twscrape is the free alternative. Always flag paid services before suggesting them — see `user-response-style` skill for the full rule.
|
||||
Reference in New Issue
Block a user