hermes-skills/local-ai-media-generation/SKILL.md

name, description, version, author, metadata

name	description	version	author	metadata
local-ai-media-generation	Plan and evaluate local AI media generation pipelines (video, talking-head, voice) on consumer GPUs — research the landscape, validate tool claims against primary sources, map to hardware, and write per-software-type plans. Carries the 2026 verified landscape and license-landmine reference.	1.0.0	Hermes Agent	hermes tags related_skills mlops ai-video tts voice-cloning talking-head local-gpu planning better-search ask-claude deep-research

Local AI Media Generation — Pipeline Planning

Plan and evaluate open-source AI media generation pipelines that run locally on consumer GPUs. Covers three layers: video generation (T2V/I2V), talking-head / lip-sync (image+audio→video), and voice (TTS + zero-shot cloning). Use when the user wants to generate AI video clips, talking-head videos, or cloned-voice audio locally — especially the "funny AI celebrity clips" genre seen on X/Twitter.

§1 When to Use / When NOT to Use

Use when:

• User wants to make AI-generated video clips (text-to-video, image-to-video, talking-head) locally
• User wants to clone a voice or generate speech locally
• User asks "how do they make those AI videos on X" and wants to reproduce the pipeline
• Evaluating whether a given video/voice model fits specific GPU hardware

Do NOT use when:

• Cloud/API video generation (Sora, Runway, Kling cloud) — this skill is local-only
• Video editing / transcoding (ffmpeg workflows) — not generation
• Hermes's own voice interaction (TTS/STT for the agent itself) — use voice-systems

§2 Workflow (research → validate → map → plan)

1. Dispatch 3-layer research via better-search (one dispatch per layer):
   • Video generation (T2V + I2V): Wan, HunyuanVideo, LTX, CogVideoX, Mochi, SVD, AnimateDiff
   • Talking-head / lip-sync: EchoMimic, Hallo, Sonic, SadTalker, LatentSync, LivePortrait
   • Voice / TTS: F5-TTS, CosyVoice, Chatterbox, GPT-SoVITS, Kokoro, XTTS-v2, Fish-Speech Run in parallel (background terminal + notify_on_complete). Collect result files from ~/workspace/research/results/.
2. Adversarial review via ask-claude — send the digest to Claude Opus with the constraint: "Do NOT propose new features or architecture. Find FLAWS in what's proposed. Cite source URLs." Claude catches stale claims and missed models.
3. Verify Claude's load-bearing claims against primary sources (HF model cards, GitHub LICENSE files, discussion threads). Claude is a consultant, not a verifier — its claims are hypotheses to test. See ask-claude skill's disagreement-scan protocol.
4. Map to hardware — per-job VRAM ceiling = single largest card. Multi-GPU = data-parallel throughput (N jobs on N cards), NOT tensor-parallel single-job sharding. You cannot pool 2×16GB into 32GB effective for one job.
5. Write one plan per software type in plans/<date>-<slug>.md (AGENTS.md template). Max 4 plans unless user says otherwise.

§3 License landmines (verify before commercial use)

These recur in this problem space. Always check the weights license, not just the code repo.

• 🔴 F5-TTS weights = CC-BY-NC-4.0 — the GitHub code is MIT (switched Oct 2024), but the HF model card frontmatter is license: cc-by-nc-4.0 because the Emilia training dataset is CC-BY-NC. Maintainer confirmed this taints the weights and survives fine-tuning (Discussion #129). Do NOT use F5-TTS commercially. Use Chatterbox (MIT) as the commercial-safe default voice cloner.
• 🔴 Sonic = CC-BY-NC-SA 4.0 — non-commercial. Best face-only quality but kills commercial use.
• 🟠 HunyuanVideo — custom Tencent license excludes EU, UK, South Korea from licensed territory; >100M-MAU gate. Irrelevant for US-only personal use, fatal for distributed products in those regions.
• 🟠 LTX-2 weights — "LTX-2 Community License": free under $10M ARR, paid above. Code is Apache-2.0. Fine now; note the ceiling.
• 🟠 CosyVoice — repo is Apache-2.0, but open maintainer threads (issues #598, #853) about whether weights carry the same terms. Treat as "probably OK, verify before shipping," not certain.
• 🟠 Fish-Speech / MaskGCT / XTTS-v2 — all commercially restricted or unmaintained. XTTS-v2: Coqui shut down Jan 2024, community forks only.
• ✅ Clean for commercial: Chatterbox (MIT), Wan 2.2 incl. S2V/Animate/TI2V (Apache-2.0), EchoMimic V1/V2/V3 (Apache-2.0), Kokoro (Apache-2.0), LatentSync (Apache-2.0), CogVideoX code (Apache).
• ⚠️ Right-of-publicity / deepfake statutes — separate from model licenses. Generating a real celebrity's face + cloned voice for distributed content is a legal exposure independent of any Apache-2.0 weights license. The model license doesn't grant you the person's likeness. Flag this to the user explicitly.

§4 Hardware reality (consumer multi-GPU)

• Multi-GPU = data-parallel throughput (N GPUs → N concurrent jobs, linear, no NVLink required). All major video models support this.
• Multi-GPU ≠ tensor-parallel single-job sharding for consumer GPUs. xDiT/USP can sequence-parallel a single Wan/Hunyuan job across GPUs but it shards compute for speed, not VRAM (weights replicate unless FSDP), and without NVLink the PCIe traffic is a bad trade. You cannot split one 720p Wan 2.2 generation across two 16GB cards to reach 32GB effective.
• Practical per-job ceiling = single largest card. To run top-tier models (Wan 2.2/HunyuanVideo at full 720p+) you need one card with ≥60-80GB VRAM, not multiple consumer cards.
• Consumer escape hatches: FP8/Q8 quantization + CPU text-encoder offload trade speed for VRAM on one card.

§5 Verified 2026 landscape (condensed)

See references/ai-video-voice-landscape-2026.md for the full per-model table with VRAM, license, fit, and source URLs. Quick picks:

• Fast silent I2V (fits 12-16GB): LTX-Video 0.9.5 — 12GB native, ~90s/5s clip on 4090, built-in I2V. LTX-2.3 (Mar 2026) adds native 4K@50fps up to 20s + synchronized audio in one pass (first open model to do so), FP8 floor 16-24GB.
• Best silent quality on 24GB: Wan 2.2 14B (FP8 + CPU offload, ~4min20s/5s clip, 720p 81 frames). Apache-2.0. Largest LoRA ecosystem for celebrity likeness. Cannot fit 16GB.
• Wan2.2-TI2V-5B — T2V+I2V in one 5B model, runs on 4090, 720P@24fps. The consumer-friendly Wan.
• Talking-head (image+audio→video): EchoMimic V1 (face, 8-16GB, Apache-2.0), V2 (semi-body, 16-24GB), V3 (full-body 1.3B, 24GB default / 12GB tuned, AAAI 2026). All Apache-2.0, actively maintained.
• Audio-driven cinematic (one model, no separate lip-sync): Wan2.2-S2V-14B (Apache-2.0, Aug 2025) — image+audio+optional prompt+pose → talking video, 480P/720P, 80GB native / multi-GPU FSDP / 24GB with offload. Beats chaining TTS+lip-sync for talking clips.
• Voice cloning (commercial-safe): Chatterbox (MIT, 5s ref, 23+ langs, emotion control, 6GB Turbo). GPT-SoVITS (MIT, best few-shot from 1min audio). CosyVoice (Apache, 9 langs + 18 dialects, cross-lingual, but verify weights license before commercial ship).

§6 Pitfalls

1. Conflating code license with weights license. F5-TTS, LTX-2, CosyVoice all have code-vs-weights license splits. Always check the HF model card frontmatter license: field, not just the GitHub LICENSE.
2. Trusting research summaries over primary sources. The better-search digest said F5-TTS was MIT. It was wrong — the code is MIT, the weights are CC-BY-NC. Claude caught it; I verified against the HF card. Always verify license claims against the actual model card before putting a model in a commercial path.
3. Assuming multi-GPU pools VRAM. It doesn't (see §4). A 5×16GB box is a throughput farm for ≤16GB jobs, not a way to run 80GB jobs.
4. Post-dubbing generated video with LatentSync. LatentSync needs a clear, front-facing, stable mouth in every frame. If the T2V model generated a wide/side/moving shot, LatentSync has nothing to sync. For talking clips, use Wan2.2-S2V (audio-driven gen) instead of generate-then-dub.
5. Likeness LoRA breaking motion priors. Overtrained celebrity LoRAs on Wan produce stiff/frozen faces. And 4090 OOM on 720p/81-frame at FP8 is common — drop frames or resolution first.
6. Forgetting right-of-publicity. Apache-2.0 weights don't grant the person's likeness. Flag this to the user before they build a celebrity-clip pipeline.

§7 See Also

• better-search — dispatch the 3-layer research (one question per dispatch, parallel background runs)
• ask-claude — adversarial review of the digest; follow its disagreement-scan protocol and verify claims against primary sources
• references/ai-video-voice-landscape-2026.md — full per-model table with VRAM, license, fit, source URLs
• references/command-gotchas-verified.md — exact CLI flag corrections, diffusers class names, EchoMimic script names, Blackwell cu128 requirement (verified against primary sources during plan review)