90 lines
8.9 KiB
Markdown
90 lines
8.9 KiB
Markdown
---
|
||
name: local-ai-media-generation
|
||
description: Plan and evaluate local AI media generation pipelines (video, talking-head, voice) on consumer GPUs — research the landscape, validate tool claims against primary sources, map to hardware, and write per-software-type plans. Carries the 2026 verified landscape and license-landmine reference.
|
||
version: 1.0.0
|
||
author: Hermes Agent
|
||
metadata:
|
||
hermes:
|
||
tags: [mlops, ai-video, tts, voice-cloning, talking-head, local-gpu, planning]
|
||
related_skills: [better-search, ask-claude, deep-research]
|
||
---
|
||
|
||
# Local AI Media Generation — Pipeline Planning
|
||
|
||
Plan and evaluate open-source AI media generation pipelines that run locally on
|
||
consumer GPUs. Covers three layers: **video generation** (T2V/I2V), **talking-head /
|
||
lip-sync** (image+audio→video), and **voice** (TTS + zero-shot cloning). Use when the
|
||
user wants to generate AI video clips, talking-head videos, or cloned-voice audio
|
||
locally — especially the "funny AI celebrity clips" genre seen on X/Twitter.
|
||
|
||
## §1 When to Use / When NOT to Use
|
||
|
||
**Use when:**
|
||
- User wants to make AI-generated video clips (text-to-video, image-to-video, talking-head) locally
|
||
- User wants to clone a voice or generate speech locally
|
||
- User asks "how do they make those AI videos on X" and wants to reproduce the pipeline
|
||
- Evaluating whether a given video/voice model fits specific GPU hardware
|
||
|
||
**Do NOT use when:**
|
||
- Cloud/API video generation (Sora, Runway, Kling cloud) — this skill is local-only
|
||
- Video editing / transcoding (ffmpeg workflows) — not generation
|
||
- Hermes's own voice interaction (TTS/STT for the agent itself) — use `voice-systems`
|
||
|
||
## §2 Workflow (research → validate → map → plan)
|
||
|
||
1. **Dispatch 3-layer research** via `better-search` (one dispatch per layer):
|
||
- Video generation (T2V + I2V): Wan, HunyuanVideo, LTX, CogVideoX, Mochi, SVD, AnimateDiff
|
||
- Talking-head / lip-sync: EchoMimic, Hallo, Sonic, SadTalker, LatentSync, LivePortrait
|
||
- Voice / TTS: F5-TTS, CosyVoice, Chatterbox, GPT-SoVITS, Kokoro, XTTS-v2, Fish-Speech
|
||
Run in parallel (background terminal + notify_on_complete). Collect result files from `~/workspace/research/results/`.
|
||
2. **Adversarial review via `ask-claude`** — send the digest to Claude Opus with the constraint: "Do NOT propose new features or architecture. Find FLAWS in what's proposed. Cite source URLs." Claude catches stale claims and missed models.
|
||
3. **Verify Claude's load-bearing claims against primary sources** (HF model cards, GitHub LICENSE files, discussion threads). Claude is a consultant, not a verifier — its claims are hypotheses to test. See `ask-claude` skill's disagreement-scan protocol.
|
||
4. **Map to hardware** — per-job VRAM ceiling = single largest card. Multi-GPU = data-parallel throughput (N jobs on N cards), NOT tensor-parallel single-job sharding. You cannot pool 2×16GB into 32GB effective for one job.
|
||
5. **Write one plan per software type** in `plans/<date>-<slug>.md` (AGENTS.md template). Max 4 plans unless user says otherwise.
|
||
|
||
## §3 License landmines (verify before commercial use)
|
||
|
||
These recur in this problem space. Always check the *weights* license, not just the code repo.
|
||
|
||
- 🔴 **F5-TTS weights = CC-BY-NC-4.0** — the GitHub *code* is MIT (switched Oct 2024), but the HF model card frontmatter is `license: cc-by-nc-4.0` because the Emilia training dataset is CC-BY-NC. Maintainer confirmed this taints the weights and survives fine-tuning (Discussion #129). Do NOT use F5-TTS commercially. Use **Chatterbox** (MIT) as the commercial-safe default voice cloner.
|
||
- 🔴 **Sonic = CC-BY-NC-SA 4.0** — non-commercial. Best face-only quality but kills commercial use.
|
||
- 🟠 **HunyuanVideo** — custom Tencent license excludes EU, UK, South Korea from licensed territory; >100M-MAU gate. Irrelevant for US-only personal use, fatal for distributed products in those regions.
|
||
- 🟠 **LTX-2 weights** — "LTX-2 Community License": free under $10M ARR, paid above. Code is Apache-2.0. Fine now; note the ceiling.
|
||
- 🟠 **CosyVoice** — repo is Apache-2.0, but open maintainer threads (issues #598, #853) about whether weights carry the same terms. Treat as "probably OK, verify before shipping," not certain.
|
||
- 🟠 **Fish-Speech / MaskGCT / XTTS-v2** — all commercially restricted or unmaintained. XTTS-v2: Coqui shut down Jan 2024, community forks only.
|
||
- ✅ **Clean for commercial:** Chatterbox (MIT), Wan 2.2 incl. S2V/Animate/TI2V (Apache-2.0), EchoMimic V1/V2/V3 (Apache-2.0), Kokoro (Apache-2.0), LatentSync (Apache-2.0), CogVideoX code (Apache).
|
||
- ⚠️ **Right-of-publicity / deepfake statutes** — separate from model licenses. Generating a real celebrity's face + cloned voice for distributed content is a legal exposure independent of any Apache-2.0 weights license. The model license doesn't grant you the person's likeness. Flag this to the user explicitly.
|
||
|
||
## §4 Hardware reality (consumer multi-GPU)
|
||
|
||
- **Multi-GPU = data-parallel throughput** (N GPUs → N concurrent jobs, linear, no NVLink required). All major video models support this.
|
||
- **Multi-GPU ≠ tensor-parallel single-job sharding** for consumer GPUs. xDiT/USP can sequence-parallel a single Wan/Hunyuan job across GPUs but it shards *compute for speed, not VRAM* (weights replicate unless FSDP), and without NVLink the PCIe traffic is a bad trade. You cannot split one 720p Wan 2.2 generation across two 16GB cards to reach 32GB effective.
|
||
- **Practical per-job ceiling = single largest card.** To run top-tier models (Wan 2.2/HunyuanVideo at full 720p+) you need one card with ≥60-80GB VRAM, not multiple consumer cards.
|
||
- **Consumer escape hatches:** FP8/Q8 quantization + CPU text-encoder offload trade speed for VRAM on one card.
|
||
|
||
## §5 Verified 2026 landscape (condensed)
|
||
|
||
See `references/ai-video-voice-landscape-2026.md` for the full per-model table with VRAM, license, fit, and source URLs. Quick picks:
|
||
|
||
- **Fast silent I2V (fits 12-16GB):** LTX-Video 0.9.5 — 12GB native, ~90s/5s clip on 4090, built-in I2V. LTX-2.3 (Mar 2026) adds native 4K@50fps up to 20s + synchronized audio in one pass (first open model to do so), FP8 floor 16-24GB.
|
||
- **Best silent quality on 24GB:** Wan 2.2 14B (FP8 + CPU offload, ~4min20s/5s clip, 720p 81 frames). Apache-2.0. Largest LoRA ecosystem for celebrity likeness. Cannot fit 16GB.
|
||
- **Wan2.2-TI2V-5B** — T2V+I2V in one 5B model, runs on 4090, 720P@24fps. The consumer-friendly Wan.
|
||
- **Talking-head (image+audio→video):** EchoMimic V1 (face, 8-16GB, Apache-2.0), V2 (semi-body, 16-24GB), V3 (full-body 1.3B, 24GB default / 12GB tuned, AAAI 2026). All Apache-2.0, actively maintained.
|
||
- **Audio-driven cinematic (one model, no separate lip-sync):** Wan2.2-S2V-14B (Apache-2.0, Aug 2025) — image+audio+optional prompt+pose → talking video, 480P/720P, 80GB native / multi-GPU FSDP / 24GB with offload. Beats chaining TTS+lip-sync for talking clips.
|
||
- **Voice cloning (commercial-safe):** Chatterbox (MIT, 5s ref, 23+ langs, emotion control, 6GB Turbo). GPT-SoVITS (MIT, best few-shot from 1min audio). CosyVoice (Apache, 9 langs + 18 dialects, cross-lingual, but verify weights license before commercial ship).
|
||
|
||
## §6 Pitfalls
|
||
|
||
1. **Conflating code license with weights license.** F5-TTS, LTX-2, CosyVoice all have code-vs-weights license splits. Always check the HF model card frontmatter `license:` field, not just the GitHub LICENSE.
|
||
2. **Trusting research summaries over primary sources.** The `better-search` digest said F5-TTS was MIT. It was wrong — the *code* is MIT, the *weights* are CC-BY-NC. Claude caught it; I verified against the HF card. Always verify license claims against the actual model card before putting a model in a commercial path.
|
||
3. **Assuming multi-GPU pools VRAM.** It doesn't (see §4). A 5×16GB box is a throughput farm for ≤16GB jobs, not a way to run 80GB jobs.
|
||
4. **Post-dubbing generated video with LatentSync.** LatentSync needs a clear, front-facing, stable mouth in every frame. If the T2V model generated a wide/side/moving shot, LatentSync has nothing to sync. For talking clips, use Wan2.2-S2V (audio-driven gen) instead of generate-then-dub.
|
||
5. **Likeness LoRA breaking motion priors.** Overtrained celebrity LoRAs on Wan produce stiff/frozen faces. And 4090 OOM on 720p/81-frame at FP8 is common — drop frames or resolution first.
|
||
6. **Forgetting right-of-publicity.** Apache-2.0 weights don't grant the person's likeness. Flag this to the user before they build a celebrity-clip pipeline.
|
||
|
||
## §7 See Also
|
||
|
||
- `better-search` — dispatch the 3-layer research (one question per dispatch, parallel background runs)
|
||
- `ask-claude` — adversarial review of the digest; follow its disagreement-scan protocol and verify claims against primary sources
|
||
- `references/ai-video-voice-landscape-2026.md` — full per-model table with VRAM, license, fit, source URLs
|
||
- `references/command-gotchas-verified.md` — exact CLI flag corrections, diffusers class names, EchoMimic script names, Blackwell cu128 requirement (verified against primary sources during plan review) |