SpeedyFoxAi/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tools-update-cron: sync 2026-08-09 — 37 skill(s) updated

Browse Source
This commit is contained in:
Hermes Agent
2026-08-09 01:01:38 -05:00
parent 94ac068425
commit 004feddf0a
37 changed files with 5386 additions and 760 deletions
Download Patch File Download Diff File Expand all files Collapse all files
local-ai-media-generation/SKILL.md
+99 -13
View File
@@ -1,12 +1,12 @@
---
name: local-ai-media-generation
description: Plan and evaluate local AI media generation pipelines (video, talking-head, voice) on consumer GPUs — research the landscape, validate tool claims against primary sources, map to hardware, and write per-software-type plans. Carries the 2026 verified landscape and license-landmine reference.
version: 1.0.0
version: 1.1.0
author: Hermes Agent
metadata:
hermes:
tags: [mlops, ai-video, tts, voice-cloning, talking-head, local-gpu, planning]
related_skills: [better-search, ask-claude, deep-research]
related_skills: [better-search, ask-claude, deep-research, dgx-spark-comfyui]
---
# Local AI Media Generation — Pipeline Planning
@@ -32,15 +32,27 @@ locally — especially the "funny AI celebrity clips" genre seen on X/Twitter.
## §2 Workflow (research → validate → map → plan)
1. **Dispatch 3-layer research** via `better-search` (one dispatch per layer):
- Video generation (T2V + I2V): Wan, HunyuanVideo, LTX, CogVideoX, Mochi, SVD, AnimateDiff
- Talking-head / lip-sync: EchoMimic, Hallo, Sonic, SadTalker, LatentSync, LivePortrait
- Voice / TTS: F5-TTS, CosyVoice, Chatterbox, GPT-SoVITS, Kokoro, XTTS-v2, Fish-Speech
Run in parallel (background terminal + notify_on_complete). Collect result files from `~/workspace/research/results/`.
2. **Adversarial review via `ask-claude`** — send the digest to Claude Opus with the constraint: "Do NOT propose new features or architecture. Find FLAWS in what's proposed. Cite source URLs." Claude catches stale claims and missed models.
3. **Verify Claude's load-bearing claims against primary sources** (HF model cards, GitHub LICENSE files, discussion threads). Claude is a consultant, not a verifier — its claims are hypotheses to test. See `ask-claude` skill's disagreement-scan protocol.
4. **Map to hardware** — per-job VRAM ceiling = single largest card. Multi-GPU = data-parallel throughput (N jobs on N cards), NOT tensor-parallel single-job sharding. You cannot pool 2×16GB into 32GB effective for one job.
5. **Write one plan per software type** in `plans/<date>-<slug>.md` (AGENTS.md template). Max 4 plans unless user says otherwise.
**Decision point — does the user already have hardware and a plan?**
- **Yes (hardware exists, plan exists but is stale):** Skip the 3-layer research dispatch. Instead: (1) SSH-probe the hardware for current state (GPU, VRAM free, disk, installed models/nodes), (2) cross-reference the plan's model recommendations against the probe, (3) render ONE 5-second shot as a verification step before building anything larger. The per-shot VRAM peak and wall-clock time from that single render IS the plan's foundation. Do not dispatch more research — the landscape hasn't changed (verified July 2026: Wan 2.7 / LTX-2.3 / HunyuanVideo 1.5 still current, no new models).
- **No (greenfield, no hardware, no plan):** Dispatch 3-layer research via `better-search` (one dispatch per layer), run in parallel background, collect result files, then proceed to adversarial review.
**CRITICAL — Confirm intent before building:** Before writing any plan or dispatching any build piece, explicitly confirm what kind of video the user wants. The default assumption from "AI video pipeline" is often T2V (text-to-video: "a mountain lake at golden hour") — but the user's real goal is almost always story-driven narrative video with consistent characters acting out scenes. These are fundamentally different pipelines. T2V is a one-shot prompt → video; story-driven is I2V with keyframes, character LoRA, control signals, scene stitching, and audio. Confirming intent at the start prevents building the wrong pipeline. Ask: "Story-driven with consistent characters, or single-shot T2V?" before any research or build dispatch.
**For either path, follow the TWO-PHASE LOOP (plan → build):**
**PHASE 1 — Plan loop (run until plan is 100%):**
1. **Research first, build second.** Before writing any plan, get full details from the web. Use `mcp_searxng_searxng_web_search` for model specs, version compatibility, known issues, and workflow patterns. Do not reason from parametric knowledge — verify every claim against live sources.
2. **Start small, scale up.** Write the plan for the smallest viable output first (e.g., 5-second clip at example-workflow defaults). Prove the pipeline works at that scale before planning longer outputs. The user will tell you the target (60s, 5min) — plan the incremental path, not the final destination.
3. **Adversarial review via `ask-claude`** — send the digest to Claude Opus with the constraint: "Do NOT propose new features or architecture. Find FLAWS in what's proposed. Cite source URLs." Claude catches stale claims and missed models.
4. **Verify Claude's load-bearing claims against primary sources** (HF model cards, GitHub LICENSE files, discussion threads). Claude is a consultant, not a verifier — its claims are hypotheses to test. See `ask-claude` skill's disagreement-scan protocol.
5. **Map to hardware** — per-job VRAM ceiling = single largest card. Multi-GPU = data-parallel throughput (N jobs on N cards), NOT tensor-parallel single-job sharding. You cannot pool 2×16GB into 32GB effective for one job.
6. **Write one plan per software type** in `plans/<date>-<slug>.md` (AGENTS.md template). Max 4 plans unless user says otherwise.
7. **Validate the plan iteratively** — dispatch to a peer (ask-dev, ask-kimi-c) for validation, apply fixes, re-validate. Loop until clean. Do not accept validations blindly — apply your own judgment. Only implement accurate changes. After 7+ rounds with only low/medium findings remaining, the plan is build-ready. **Do not proceed to Phase 2 until the plan is 100%.**
**PHASE 2 — Build loop (run until software is ready):**
8. **Build piece by piece, not all at once** — break the build into numbered pieces. Dispatch one piece at a time. Verify each piece's output before dispatching the next. The coordinator (you) owns the sequence; the peer owns each piece's execution. See `research-dispatch-pitfalls` Pitfall 8 for the full pattern. Each piece gets its own temp file with exact commands and verify steps. A failed piece doesn't waste prior work.
9. **Validate after each piece** — after a piece succeeds, dispatch ask-dev to validate. Review findings — only apply accurate fixes, push back on wrong ones. Have the build peer apply fixes. Repeat until ask-dev says CLEAN. Then dispatch ask-claude for final adversarial review.
10. **Scale incrementally** — prove the pipeline at the smallest scale first (5s clip), then scale to the user's target (60s, 5min). Each scale-up is a new piece. Do not jump from 5s to 5min in one step — the intermediate step catches OOM, dimension, and context-window issues before they waste hours.
## §3 License landmines (verify before commercial use)
@@ -67,7 +79,7 @@ These recur in this problem space. Always check the *weights* license, not just
See `references/ai-video-voice-landscape-2026.md` for the full per-model table with VRAM, license, fit, and source URLs. Quick picks:
- **Fast silent I2V (fits 12-16GB):** LTX-Video 0.9.5 — 12GB native, ~90s/5s clip on 4090, built-in I2V. LTX-2.3 (Mar 2026) adds native 4K@50fps up to 20s + synchronized audio in one pass (first open model to do so), FP8 floor 16-24GB.
- **Best silent quality on 24GB:** Wan 2.2 14B (FP8 + CPU offload, ~4min20s/5s clip, 720p 81 frames). Apache-2.0. Largest LoRA ecosystem for celebrity likeness. Cannot fit 16GB.
- **Best silent quality on 24GB (July 2026):** Wan 2.2 I2V-14B GGUF (Q4_K_S, ~4-15 min/5s clip, 720p). Apache-2.0. Confirmed available on HuggingFace `QuantStack/Wan2.2-I2V-A14B-GGUF` (322K downloads/month). Uses MoE architecture — requires BOTH HighNoise and LowNoise GGUF files. Kijai wrapper + ComfyUI-GGUF loads pre-quantized files. Official unquantized model needs 80GB VRAM; GGUF Q4_K_S fits 24GB. **Wan 2.7 is NOT available as of July 2026** — not on HuggingFace (zero API results), GitHub (repo 404), or ModelScope (placeholder). See `references/wan-27-availability-july-2026.md` for the full due-diligence trace. When Wan 2.7 drops, same custom nodes, same workflow, same VRAM profile.
- **Wan2.2-TI2V-5B** — T2V+I2V in one 5B model, runs on 4090, 720P@24fps. The consumer-friendly Wan.
- **Talking-head (image+audio→video):** EchoMimic V1 (face, 8-16GB, Apache-2.0), V2 (semi-body, 16-24GB), V3 (full-body 1.3B, 24GB default / 12GB tuned, AAAI 2026). All Apache-2.0, actively maintained.
- **Audio-driven cinematic (one model, no separate lip-sync):** Wan2.2-S2V-14B (Apache-2.0, Aug 2025) — image+audio+optional prompt+pose → talking video, 480P/720P, 80GB native / multi-GPU FSDP / 24GB with offload. Beats chaining TTS+lip-sync for talking clips.
@@ -81,10 +93,84 @@ See `references/ai-video-voice-landscape-2026.md` for the full per-model table w
4. **Post-dubbing generated video with LatentSync.** LatentSync needs a clear, front-facing, stable mouth in every frame. If the T2V model generated a wide/side/moving shot, LatentSync has nothing to sync. For talking clips, use Wan2.2-S2V (audio-driven gen) instead of generate-then-dub.
5. **Likeness LoRA breaking motion priors.** Overtrained celebrity LoRAs on Wan produce stiff/frozen faces. And 4090 OOM on 720p/81-frame at FP8 is common — drop frames or resolution first.
6. **Forgetting right-of-publicity.** Apache-2.0 weights don't grant the person's likeness. Flag this to the user before they build a celebrity-clip pipeline.
7. **Over-collecting research when the landscape is stable.** A daily cron that scrapes SEO listicles into Qdrant produces noise, not signal. As of July 2026, the open AI video landscape (Wan 2.7 / LTX-2.3 / HunyuanVideo 1.5) has been stable for months. Throttle research crons to weekly. The real unknowns (does model X fit in Y GB? how long per shot? does the LoRA hold?) can only be answered by running the hardware — not by more web searches.
8. **Planning without a hardware probe.** Before writing any setup plan, SSH into the target box and run: `nvidia-smi`, `free -h`, `df -h`, `ls ~/ComfyUI/models/` (if ComfyUI exists), `python3 --version`, `which ffmpeg`. The probe IS the first section of the plan. Without it, you're guessing about free VRAM, disk space, and installed tools. AI video models are 15-50GB each — disk space is the #1 blocker on fresh LXCs (default 20GB allocation is nowhere near enough; 200GB minimum, 500GB for full pipeline).
9. **I2V models cannot do T2V via empty embeds.** The I2V-A14B model has `in_dim=36` (image latents) — WanVideoEmptyEmbeds produces 16-channel T2V input, causing a channel mismatch at the patch embedding layer: `expected input to have 36 channels, but got 16 channels instead`. For T2V, use a TI2V model (e.g., Wan2.2-TI2V-5B, `in_dim=48`, `model_type=ti2v`) which accepts both image and empty embeds. Check `config.json` in the model directory for `in_dim` and `model_type` before building a workflow.
10. **ComfyUI /prompt endpoint requires API format, NOT UI format.** The UI saves workflows as `type` + `widgets_values` arrays. The `/prompt` endpoint rejects this — it needs `class_type` + `inputs` dict with named keys matching the node's INPUT_TYPES exactly. Always query `http://localhost:8188/object_info` for the live node definitions before building an API prompt. Key gotchas: `WanVideoBlockSwap` requires `blocks_to_swap` (not `blocks`), `WanVideoSampler` requires `riflex_freq_index` (set to 0 to disable) and `scheduler` must be from the allowed list, `WanVideoModelLoader` model names include the subdirectory path. See `references/comfyui-api-prompt-format.md` for the full reference.
11. **Build-piece dispatch: write the task file, don't paste into the prompt.** When dispatching a build piece to a peer (kimi-c), write the full task to a temp file (`/tmp/kimi-build-N.txt`) with exact commands, expected outputs, and verification steps. The peer reads the file — never paste multi-step build instructions inline. This avoids shell escaping issues, token bloat, and transcription errors. Each piece file should be self-contained: target host, SSH credentials, exact commands, verify steps, and a "report results and stop" directive.
12. **Sharded safetensors models must be merged before ComfyUI can load them.** `load_torch_file` loads a single safetensors file — it does NOT auto-resolve shards from an index.json. The Wan 2.2 TI2V-5B downloads as 3 shards + index.json. Merge with: `python3 -c "from safetensors.torch import load_file, save_file; import json; index = json.load(open('index.json')); merged = {}; [merged.update(load_file(s)) for s in sorted(set(index['weight_map'].values()))]; save_file(merged, 'merged.safetensors')"`. The merged file then appears in the WanVideoModelLoader dropdown.
13. **WanVideoBlockSwap crashes on missing `vace_blocks_to_swap`.** The node's internal comparison `vace_blocks_to_swap > 0` raises `TypeError: '>' not supported between instances of 'NoneType' and 'int'` when the optional input is omitted. Always include `"vace_blocks_to_swap": 0` in the API prompt for WanVideoBlockSwap, even though it's listed as optional.
14. **Start renders at 832×480, not 1280×720 — and verify ÷32.** The TI2V-5B example workflow uses 832×480 for a reason. Wan 2.2 models require both width and height to be divisible by 32. 1280×720 fails because 720/32=22.5 (not integer) — the correct near-720p resolution is 1280×704 (704/32=22). The tensor dimension mismatch error (`tensor a (45) must match tensor b (44) at non-singleton dimension 3`) is a ÷32 boundary issue, not a VAE tiling bug. Disable VAE tiling for the first render, use the example workflow's default resolution, then scale up to verified ÷32 resolutions once the pipeline is proven.
15. **Don't make the peer monitor long renders — poll directly from the coordinator.** When a render takes 30+ minutes, the peer's turn budget (even at 30) is consumed by the polling loop. The peer hits the turn limit mid-render and returns incomplete results. Instead: have the peer submit the job and return the prompt ID, then the coordinator runs a background shell script (`terminal(background=true)`) that polls VRAM and status directly on the target host. The peer's job is to get the prompt submitted and validated — the coordinator owns the wait.
16. **Use WanVideoContextOptions for long videos — don't manually chain clips.** The Kijai wrapper includes `WanVideoContextOptions` which breaks long videos into sliding context windows with overlap and blends them automatically. One generation, one output file. No manual ffmpeg concatenation needed. Key parameters: `context_schedule=uniform_standard`, `context_frames=81` (window size), `context_stride=4`, `context_overlap=16`, `fuse_method=pyramid` (better blending than linear). The Kijai repo README shows 1025 frames generated this way. Connect `context_options` output → `WanVideoSampler` `context_options` input.
17. **WanVideoEmptyEmbeds `num_frames` is pixel frames, NOT latent frames.** Despite the Wan docs saying latent space has 4 pixel frames per 1 latent frame, the `num_frames` parameter on `WanVideoEmptyEmbeds` maps directly to output pixel frames. Setting `num_frames=361` produces 361 output frames (~15s at 24fps), not 1441. For 5s@24fps use `num_frames=121`, for 60s@24fps use `num_frames=1441`. Always verify with ffprobe after the first render — the formula `1+4*N` is for the underlying model architecture, not this node's parameter. The `frame_rate` on `VHS_VideoCombine` controls playback speed; the model generates at its native temporal rate and VHS maps it to the declared fps.
18. **All videos must have sound — plan audio from the start.** The user's requirement is that every generated video includes synchronized audio. Do not build a silent video pipeline and add audio later — audio is a first-class requirement. The research-backed approach for RTX 4090 24GB is a 3-pass pipeline: (1) video generation with Wan 2.2 I2V-A14B GGUF + character LoRA + SVI 2.0 Pro, (2) audio generation with F5-TTS for dialogue + MMAudio for ambient sound, (3) ffmpeg mux. Ovi (video+audio in one pass) is limited to 10-second clips and not suitable for 5-minute videos. Wan 2.2 S2V needs 80GB VRAM — not viable on 24GB. See `references/5min-video-audio-characters-pipeline.md` for the full research summary.
20. **T2V test renders are NOT the deliverable — don't get sidetracked.** When the user's goal is story-driven narrative video with consistent characters, T2V renders ("a mountain lake at golden hour") are infrastructure verification, not progress toward the goal. The real pipeline is I2V with keyframes, character LoRA, control signals, and audio. After proving the infrastructure works with one T2V shot, immediately pivot to the story-driven pieces. Do not spend hours optimizing T2V parameters (resolution, frame rate, duration) — those are for the I2V story pipeline, not the T2V test. The user will correct you on this (as they did in this session: "I thought I made that clear when we originally created the plan").
21. **Don't stop renders mid-run.** When a render is in progress and the user sends a correction or new instruction, do NOT kill the render process. The render represents real GPU time invested. Let it finish while you research or plan the next step in parallel. The user explicitly said "dont stop render" when a correction arrived mid-render. Only kill a render if the user explicitly says "stop" or "kill it."
22. **LAN access is required — bind to 0.0.0.0, not 127.0.0.1.** ComfyUI must be reachable from other machines on the local network. Use `--listen 0.0.0.0 --port 8188`. Claude's adversarial review may flag this as a security risk (unauth admin API on LAN) — acknowledge the risk but the user's requirement is LAN access. The user said: "I can run it manually if you set it all up. But I need lan access."
23. **24fps minimum frame rate.** The user's requirement is 24fps minimum for all generated video. Wan example workflows default to 16fps — override to 24fps. The `frame_rate` on `VHS_VideoCombine` controls playback speed. For 5 seconds at 24fps, use `num_frames=121` on `WanVideoEmptyEmbeds` (pixel frames, not latent). For 60 seconds, use `num_frames=1441`.
24. **Iterative plan validation loop — Claude → research → fix → repeat until CLEAN.** Plans for AI video pipelines contain claims about model compatibility, VRAM, training times, and tool support that are frequently wrong when sourced from parametric knowledge or single-pass research. The validated pattern: (a) write the plan, (b) dispatch Claude for adversarial review with web search mandate, (c) for every disputed claim Claude flags, dispatch a focused deep-research pass to verify against primary sources (HF model cards, GitHub READMEs, community reports), (d) update the plan with verified corrections, (e) re-dispatch Claude for re-validation, (f) repeat until Claude says CLEAN. In this session: 3 Claude rounds + 2 deep-research correction passes were needed. Claims that were wrong in the initial plan: Fun Control is a lightweight ControlNet (actually a full 64GB 14B model), LoRA training takes 4 hours on RTX 4090 (actually 15-20 hours), Phantom works with Wan 2.2 (Wan 2.1 only), LightX2V is T2V-only (I2V LoRA exists since Aug 2025). Each wrong claim would have wasted hours of build time. The loop cost ~$3 in Claude tokens and prevented days of failed builds.
25. **Wan 2.2 I2V-A14B GGUF dual-expert workflow.** The A14B uses MoE architecture with two separate 14B transformers. Both HighNoise and LowNoise GGUF files must be loaded (via UnetLoaderGGUF or WanVideoModelLoader). Expert switching is handled by two WanVideoSampler nodes with different step ranges — HighNoise handles early denoising steps, LowNoise handles late steps. With Kijai Lightning LoRA (LightX2V I2V distillation): 2 high-steps + 2 low-steps = 4 total. VRAM: ~14-16GB peak (one expert loaded at a time with offloading). The Kijai example workflow `wanvideo_2_2_I2V_A14B_example_WIP.json` shows the exact node wiring. SVI 2.0 Pro (a LoRA pair, ~1.2GB each) is confirmed compatible with GGUF models. Fun Control is a separate 64GB model — cannot run alongside I2V on 24GB. See `references/story-pipeline-verified-2026.md` for the full verified findings from 3 deep-research passes (50+ sources).
26. **WanVideoImageToVideoEncode: `fun_or_fl2v_model` must be False for standard I2V GGUF.** The node defaults to `true`, which produces 68-channel Fun-model embeds. Standard I2V-A14B GGUF expects 36 channels. The mismatch causes: `RuntimeError: Given groups=1, weight of size [5120, 36, 1, 2, 2], expected input[1, 68, 21, 45, 80] to have 36 channels, but got 68 channels instead`. Always include `"fun_or_fl2v_model": False` in the API prompt for WanVideoImageToVideoEncode when using standard (non-Fun) I2V models. Also include all required inputs: `noise_aug_strength`, `start_latent_strength`, `end_latent_strength`, `force_offload` — the node lists them as required even though the UI may hide them.
27. **Use Wan 2.1 VAE for I2V-A14B GGUF, not Wan 2.2 VAE.** The Wan 2.2 VAE produces 48-channel latents but the I2V-A14B model expects 36-channel input at the patch embedding layer. The Wan 2.1 VAE (`Wan2.1_VAE.pth`, 485MB) produces the correct channel count. Both VAEs are compatible with Wan 2.2 models for decoding — the channel mismatch only matters at the encode→model boundary. Keep both VAEs on disk: Wan 2.2 VAE for TI2V-5B, Wan 2.1 VAE for I2V-A14B GGUF.
28. **GGUF is inference-only — LoRA training needs FP8 safetensors.** The GGUF Q4_K_S files (~8.2GB each) are for inference. LoRA training requires the full FP8 safetensors (~28GB for both experts, from `Comfy-Org/Wan_2.2_ComfyUI_Repackaged` on HuggingFace). The plan's model download table must include both. GGUF files cannot be used for training — they're pre-quantized for inference only.
29. **Lightning LoRA WanVideoSampler: `start_step`/`end_step` are exclusive, official scheduler is `euler`.** The official LightX2V I2V workflow uses `euler` scheduler (not `flowmatch_distill` — that has an open color-shift bug, Issue #782). For 4-step Lightning: HighNoise `start_step=0, end_step=2` (steps 0,1) + LowNoise `start_step=2, end_step=4` (steps 2,3). `end_step` is exclusive — `0→1` gives only 1 step, not 2. Official workflow also uses `shift=6, cfg=1` (not shift=5, cfg=6). The `WanVideoLoraSelect` input field is `lora` (not `lora_name`), and the path prefix is `lightning/LoRAs/...` (not `LoRAs/...`). `merge_loras` must be `False` for GGUF — the code raises ValueError otherwise. See `references/lightning-lora-gguf-config-verified.md` for the full research findings.
30. **SageAttention install requires ComfyUI restart.** SageAttention detection in `attention.py` runs at import time (ComfyUI startup). Installing it while ComfyUI is running will succeed (`pip install` + `import sageattention` both work) but the running process won't pick it up — renders will fail with `ImportError: Selected attention mode not available`. Always restart ComfyUI after installing SageAttention. Verify with `grep -i "sage\|attention" /tmp/comfyui.log` — look for `Using pytorch attention` (not installed) vs SageAttention being loaded.
31. **Render hang diagnosis: VRAM stuck at idle means the job never started.** If `nvidia-smi` shows VRAM at ~1490 MiB (idle) for 9+ minutes after submission, the render isn't executing. Check: (a) `curl -s http://localhost:8188/queue` — if `queue_running` is empty, the job errored immediately, (b) `curl -s http://localhost:8188/history/$PROMPT_ID` — check `status.status_str` for `error` and inspect `messages` for the traceback, (c) `tail -50 /tmp/comfyui.log | grep -i error` — the ImportError or node error will be there. Common causes: missing Python package (SageAttention not installed or not restarted), wrong node input field name, model file not found.
32. **Claude adversarial review: verify scheduler claims against official workflows.** Claude may recommend `flowmatch_distill` scheduler for Lightning LoRA based on community guides, but the official LightX2V published workflows use `euler`. Always dispatch a `better-search` to verify any scheduler/model/config claim Claude makes before applying it. The disagreement-scan pattern from `ask-claude` applies: Claude's findings are hypotheses to test, not conclusions. In this session, Claude was right about SageAttention, blocks_to_swap, and the LoRA control render — but wrong about the scheduler.
33. **Optimize before scaling: fix attention, block swap, and scheduler before the production batch.** A 12-scene 60s render at 402s/scene = 80 minutes. With SageAttention + `blocks_to_swap=0` + correct scheduler, that drops to ~25-40 min. The optimization pass (install SageAttention, restart ComfyUI, set `blocks_to_swap=0`, match official workflow params) is 10 minutes of work that saves 40+ minutes of render time. Do it before Piece 11 (scene generation), not after.
34. **F5-TTS synthetic voices speak at ~2.5× natural speed — verify audio duration before scene generation.** The default F5-TTS inference produces audio that runs much faster than natural speech. A 60s script may produce only 24s of audio. Fixes in priority order: (a) use `speed=0.4` parameter in F5-TTS API call, (b) fall back to `edge-tts` with `--rate=-40%` for reliable speed control (`pip install edge-tts`), (c) pad with 1-2s silence between lines via `ffmpeg -f lavfi -i anullsrc`. For edge-tts, use `en-US-GuyNeural`/`en-US-EricNeural` for male voices and `en-US-AriaNeural`/`en-US-JennyNeural` for female. Always measure each audio file with `ffprobe` and verify total duration BEFORE dispatching scene generation — rebuilding 16 scenes because audio was too short wastes ~40 min of GPU time. The audio timing determines scene frame counts (duration × 16fps, rounded to 4n+1), so audio MUST be correct first.
35. **When the user says "continue until done" or "don't stop", keep dispatching pieces without waiting.** Do not pause between build pieces to report status and wait for confirmation. The user is not at the PC and wants the pipeline built end-to-end. Dispatch the next piece immediately when the current one completes. Only stop if a piece fails and needs user intervention, or if the user explicitly says "stop." Reporting status is fine — waiting for a reply before continuing is not.
36. **Multi-scene LTX renders can OOM the GPU even when individual scenes fit.** VRAM fragmentation across sequential generations, or accumulation of intermediate tensors between scenes, can crash the GPU partway through a batch. On RTX 4090 24GB, LTX-2.3 at 512×512 with IA2V + ID LoRA + Distilled LoRA + native audio is tight — a 6-scene × 10s batch pushed it over. Mitigations: (a) run scenes one at a time with explicit VRAM cleanup between renders, (b) reduce resolution to 384×384 for multi-scene batches, (c) use the distilled transformer-only fp8 model instead of full fp8, (d) monitor VRAM with `nvidia-smi` during the first scene and abort if peak exceeds 22GB. The user will tell you when you OOM'd — don't make them repeat it.
37. **LTX Director 2.0 timeline_data JSON: segments need `"type": "image"` and `"imageFile"` (NOT `"image"`).** The Director node reads `seg.get("imageFile")` from ComfyUI's `input/` directory — it ignores any field called `"image"`. Without `"type": "image"` and a valid `"imageFile"` filename, the Director finds zero valid segments and falls back to the `global_prompt` for the entire render. Result: every batch generates the same content regardless of what scenes you defined. Fix: copy start frames to `~/comfy-ui/input/`, use `"imageFile": "filename.png"` (just the basename), and include `"type": "image"` on every segment. Audio segments use `"file": "/absolute/path/to/audio.mp3"`. Verified against the source code at `ltx_director.py:394-403`.
38. **TrueNAS file transfer requires CIFS mount, not just smbclient.** The `smbclient` command is often not installed in minimal LXCs. To copy rendered videos to TrueNAS: (a) Check if smbclient exists first, (b) If not, mount the CIFS share on the Hermes host using `sudo mount -t cifs //10.0.0.117/share/path /mnt/point -o user=n8n,pass=passw0rd,vers=3.0`, (c) Copy files to the mount point. Alternative: use `scp` to transfer to the Hermes host first, then mount from there. The mount requires sudo privileges and cannot be done from within an unprivileged container.
39. **LTX render submission via API: use curl to `/prompt` endpoint, poll `/history/$PROMPT_ID`.** The ComfyUI API workflow: (1) POST JSON prompt to `http://<host>:8188/prompt` with `Content-Type: application/json`, (2) Extract `prompt_id` from response, (3) Poll `http://<host>:8188/history/$PROMPT_ID` until `status.status_str` is "success" or "error", (4) Check `outputs` section for generated file names. The prompt JSON uses node IDs as keys with `class_type` and `inputs` structure. Example: `curl -s -X POST http://10.0.0.202:8188/prompt -H "Content-Type: application/json" -d '{"prompt": {...}}'`.
40. **VHS VideoCombine produces two output files: `_00001.mp4` (video-only) and `_00001-audio.mp4` (video+audio).** The non-suffixed `.mp4` has no audio stream. Always ship the `-audio.mp4` variant to the user. Verify with `ffprobe` before declaring a render complete — a video-only file looks correct but is silent.
40. **6-fix baseline validated for LTX-2.3 on RTX 4090 24GB.** Confirmed working settings (2026-07-22): fp8 distilled transformer, 18 steps single-stage euler sampler, 768×512 resolution, CFG 1.0, guide_strength 1.0, NO ID LoRA, TenStrip cond-safe rank-72 LoRA at 1.0 strength, audio disabled for test renders. Prompts: 50-80 words, 2-3 actions max. This configuration produces clean renders without artifacts. `euler_ancestral_cfg_pp` FAILS with fp8 distilled model — use plain `euler` only.
41. **LTX 2 LoRAs are NOT compatible with LTX 2.3.** Civitai lists both under "LTX" searches. Always verify `modelVersions[].baseModel` is `LTXV 2.3` before downloading — the top-level `baseModel` field is often empty. Accept: `LTXV 2.3`. Reject: `LTXV2`, `LTX2`, `LTXV 2`. See `references/civitai-lora-download.md` for the full search→verify→download→upload workflow.
42. **IC-LoRA Ingredients two-stage wiring (verified July 2026).** The official two-stage IC-LoRA pattern (from `LTX-2.3_ICLoRA_Outpaint_Two_Stage_Distilled.json`) requires 3 nodes inserted into an existing two-stage sampler workflow. See `references/ic-lora-two-stage-wiring.md` for the full node-level wiring diagram, link map, and 5 common failure modes. Key rules: (a) IC-LoRA Loader goes after distilled LoRA, before SageAttention — MODEL output fans to shared bus, NOT directly to CFGGuiders, (b) GuideAdvanced sits between Conditioning and Pass 1 CFGGuider — its latent output goes through ConcatAV (preserve audio path), NOT directly to sampler, (c) CropGuides sits between Pass 1 and Pass 2 — takes SeparateAV.video_latent (5D tensor, NOT raw AV sampler output), (d) GuideAdvanced pos/neg → CFGGuider(Pass1) AND CropGuides → CFGGuider(Pass2), (e) IC strength 1.0 for two-stage (same as single-stage). Reference image: multi-panel character sheet on black background, looped to static video. The official Ingredients workflow is single-stage only — two-stage wiring must be ported from Outpaint/Inpaint/Lipdub variants.
## §7 See Also
- `better-search` — dispatch the 3-layer research (one question per dispatch, parallel background runs)
- `ask-claude` — adversarial review of the digest; follow its disagreement-scan protocol and verify claims against primary sources
- `references/dgx-spark-comfyui-setup.md` — DGX Spark (GB10, aarch64) ComfyUI deployment: current state, Manager frontend invisibility root cause + fix, diagnosis checklist, torch-safe update procedure, bind mounts, launch flags, cross-box management (verified 2026-07-28)
- `references/5min-video-audio-characters-pipeline.md` — research summary: 3-pass pipeline for 5-min video + audio + consistent characters on RTX 4090 24GB (28 sources, July 2026)
- `references/ai-video-voice-landscape-2026.md` — full per-model table with VRAM, license, fit, source URLs
- `references/command-gotchas-verified.md` — exact CLI flag corrections, diffusers class names, EchoMimic script names, Blackwell cu128 requirement (verified against primary sources during plan review)
- `references/command-gotchas-verified.md` — exact CLI flag corrections, diffusers class names, EchoMimic script names, Blackwell cu128 requirement (verified against primary sources during plan review)
- `references/ltx-director-timeline-format.md` — LTX Director 2.0 timeline_data JSON format: segment field names, common failure mode, VRAM notes (verified against source code)
- `scripts/ltx_test_render.sh` — automated 5s baseline test render script for LTX-2.3 on RTX 4090, submits via ComfyUI API and polls for completion