tools-update-cron: sync 2026-08-09 — 37 skill(s) updated
This commit is contained in:
Hermes Agent
+382
-111
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: ltx-video-pipeline
|
||||
description: "LTX Video pipeline on 10.0.0.202 — model chain wiring, render settings, concat, stock management, and Transition LoRA multi-scene workflows."
|
||||
version: 2.0.0
|
||||
version: 2.2.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux]
|
||||
@@ -50,33 +50,98 @@ See `references/character-descriptions.md` for the canonical character descripti
|
||||
| GPU | RTX 4090 24GB VRAM |
|
||||
| ffmpeg | /usr/bin/ffmpeg (7.1.5) |
|
||||
| State file | `~/workspace/general/ltx-pipeline-state.md` |
|
||||
| Continue file | `~/workspace/general/ltx-video-pipeline-continue.md` |
|
||||
| Continue file | `~/workspace/general/ltx-pipeline-continue.md` |
|
||||
| Plan file | `~/workspace/general/ltx-pipeline-plan.md` |
|
||||
|
||||
## Resolution Limits
|
||||
|
||||
### .202 (RTX 4090 24GB, fp8 UNET, spatial x2 upscaler)
|
||||
|
||||
| Resolution | Latent | VRAM Est. | Feasible |
|
||||
|---|---|---|---|
|
||||
| 1280×720 (720p) | 640×360 | ~20GB | Yes — same ballpark as 1280×736 |
|
||||
| 1408×768 | 704×384 | ~21-22GB | Probably — tight but fp8 helps |
|
||||
| 1536×864 | 768×432 | ~22-23GB | Maybe — right at the edge |
|
||||
| 1920×1080 (1080p) | 960×540 | OOM | No — 24GB not enough |
|
||||
|
||||
Higher res = longer renders (roughly linear with pixel count). 1408×768 is ~15% more pixels than 1280×736, adds ~3-4 min per scene. The spatial upscaler x2 means the model works at half res — that's the real bottleneck, not output pixels.
|
||||
|
||||
### DGX (10.0.0.6, 128GB unified, bf16 UNET 42GB)
|
||||
|
||||
Can push to 1920×1080 or higher. bf16 UNET + tile 1408 already in the DGX max quality workflow. See `references/dgx-spark-comfyui.md`.
|
||||
|
||||
## DGX Workflow Porting (.202 → DGX)
|
||||
|
||||
When porting workflow modifications from .202 to DGX, the changes are parameter-level — nothing structural depends on fp8 vs bf16 or single vs multi-GPU:
|
||||
|
||||
**What ports (same changes, different workflow file):**
|
||||
- Variable durations (EmptyLTXVLatentVideo length values)
|
||||
- Rewritten prompts (text node strings)
|
||||
- Per-scene save_output=true (VHS VideoCombine)
|
||||
- Brand names stripped, static wardrobe removed
|
||||
|
||||
**What stays DGX-specific (don't touch):**
|
||||
- bf16 UNET + bf16 Gemma model loaders
|
||||
- LTXVSequenceParallelMultiGPUPatcher
|
||||
- tile 1408 resolution
|
||||
- Different VAE/CLIP loader nodes
|
||||
|
||||
**Process:** Node types are the same (LTXVConditioning, VHS_VideoCombine, EmptyLTXVLatentVideo) — just wired to different model files. The DGX workflow has different node ID numbering, so changes must be mapped to the correct node IDs. Load the DGX workflow in ComfyUI UI, identify the right nodes, apply the same edits.
|
||||
|
||||
**⚠️ CORRECTED 2026-07-30: DGX and .202 workflows have IDENTICAL structure.** Both use the same single-render pattern: 1 EmptyLatent, 1 LTXVConditioning, 8 LoadImage, 9 VHS nodes. The initial assumption of a structural mismatch was wrong — confirmed by byte-level node inventory diff. Porting changes from .202 to DGX is parameter-only: PROMPT text, LENGTH values, VHS save_output/prefix. No restructuring needed. The 3 DGX quality differences (bf16 UNET, bf16 Gemma CLIP, MultiGPU patcher) are the ONLY structural differences. See `plans/2026-07-30-dgx-fullhqhours.md` for the corrected plan.
|
||||
|
||||
## Related Skills & References
|
||||
|
||||
- **Official ComfyUI skill (v5.1.0)** — enabled in general profile at `~/.hermes/profiles/general/skills/creative/comfyui/`. 36 files: 10 scripts, 7 workflows, 4 references, 7 tests. The execution scripts (`run_workflow.py`, `run_batch.py`, `extract_schema.py`, `check_deps.py`, `ws_monitor.py`) are directly useful for LTX pipeline work — they handle param injection, submission, monitoring, and output download. Use `--host http://10.0.0.202:8188` for remote execution (scripts default to 127.0.0.1:8188). **`fetch_logs.py` is essential for debugging failed renders** — retrieves full Python tracebacks from the ComfyUI history API. See §Debugging Failed Renders. The `comfy-cli` lifecycle commands (install, launch, stop) are NOT needed — we use systemd. Hub install blocked by security scanner (25 findings, community source) — manually copied from bundled location instead.
|
||||
- **Consolidated ComfyUI reference** — `~/workspace/general/comfy_skill.md` (11 sections, 51 chunks in ai_brain_kb). Covers all ComfyUI knowledge: environment, update procedure, API format, Open WebUI integration, LTX pipeline, WanVideo, SageAttention, fact_store, AI brain, and the official skill. Search ai_brain_kb with natural language queries for any ComfyUI topic.
|
||||
- **`ai-brain-kb` skill** — manage the `ai_brain_kb` Qdrant collection. After every deep research run or significant session, add result files to the brain for cross-session semantic search.
|
||||
- **`holographic-memory` skill** — fact_store for real-time working memory. Save pipeline discoveries, bug fixes, and configuration details proactively during sessions.
|
||||
- **LTX Director 2.0 full reference** — `references/ltx-director-2.0-how-it-works.md` — timeline_data schema, keyframe loading from source code, end frames feature (native FLF2V), two-stage pipeline architecture, model list, example workflows, storybook pipeline integration pattern. From README + source code + hotfix workflow JSON (2026-07-29).
|
||||
- **LTX Director best practices (July 2026 research)** — `references/ltx-director-best-practices-2026-07-29.md` — 18-source deep research: prompting rules, keyframe best practices, timeline standards, render settings, current model recommendations, known bugs. Supersedes older prompting and settings assumptions.
|
||||
- **Qwen Image prompting (July 2026 research)** — `references/qwen-image-prompting-2026-07-29.md` — 18-source deep research: txt2img structured prompt format, img2img natural language instructions, character consistency workflow, Next-Scene LoRA usage, Lightning 4-step trade-offs, common pitfalls. For generating photorealistic keyframes for LTX video stories.
|
||||
- **DGX Spark ComfyUI** — `references/dgx-spark-comfyui.md` — full infrastructure reference for the DGX Spark (10.0.0.6): access, Docker stack, Manager versioning pitfall (V3.41 trap), models, custom nodes, torch update constraints, and fleet context. Check DGX before pulling models from HuggingFace — LAN copies are faster.
|
||||
|
||||
## Model Chain
|
||||
|
||||
### DEFAULT: Single-Stage I2V (fp8 + TenStrip cond-safe or Distilled, NO ID LoRA) — USE THIS FOR ALL SCENES
|
||||
### DEFAULT: Single-Stage I2V (dev-fp8 + TenStrip cond-safe, NO ID LoRA) — CONFIRMED CLEAN 2026-07-22
|
||||
|
||||
**The 6 fixes from deep research (2026-07-22, 15 sources, v2 correction) are the baseline.** The 2-clip test confirmed: single-stage 18 steps, guide_strength 1.0, no ID LoRA, 768×512, simplified prompts — clean 241-frame output, no Director wiring bug.
|
||||
**✅ USER-CONFIRMED BASELINE (Fix 1b, 2026-07-22):** dev-fp8 + TenStrip cond-safe @ 1.0, LTXDirector chain, guide_strength 0.80, euler/simple 18 steps, CFG 1.0, 768×512, 121 frames, no NAG (ConditioningZeroOut). Single-clip test produced clean output. 2-clip test (boss + woman, frame-B transition) also passed — both clips clean, 242-frame concat, user-approved. This is the production baseline.
|
||||
|
||||
**⚠️ Fix #5 (Distilled LoRA 1.0) was CORRECTED by follow-up deep research (2026-07-22, 15 sources).** Community consensus: Distilled LoRA at 1.0 causes quality degradation for I2V. The correct range is 0.5-0.7. The confusion was conflating two different parameters: I2V conditioning strength (guide_strength, should be 1.0) vs Distilled LoRA strength (should be 0.5-0.7). Sources: aistudynow.com ("Do not set the strength to 1.0"), official ComfyUI guide (uses 0.5), TenStrip experiments ("The official rank 384 LoRA can actively work against conditioned inputs").
|
||||
**⚠️ CRITICAL: The TenStrip cond-safe LoRA is a DISTILLATION LoRA — it MUST be paired with the DEV base, NOT the distilled base.** Running cond-safe on the already-distilled transformer causes double-distillation: structure loss, degraded conditioning fidelity, and increased hallucination (closet doors, second characters, frozen objects). From the model page: "Applying them on top of an already-distilled model double-distills and degrades quality." See `references/double-distillation-artifacts.md` for the full root cause analysis (2026-07-22, Grok diagnosis).
|
||||
|
||||
**PREFERRED: TenStrip cond-safe LoRA (rank-72, 662 MB) at strength 1.0.** This is purpose-built for I2V — zeroes out cross-attention bridges, adaln/scale-shift tables, gate logits, and prompt scale-shift that fight I2V conditioning. "This is technically what an official I2V distilled lora should have had." Download from huggingface.co/TenStrip/LTX2.3_Distilled_Lora_1.1_Experiments — file: `ltx-2.3-22b-distilled-lora-1.1_fro90_ceil72_condsafe.safetensors`. Safe at 1.0 on first pass I2V. Upscale pass at 0.4.
|
||||
**HARD RULES (from failures — do not violate):**
|
||||
1. **LTXDirector chain ONLY** — SamplerCustomAdvanced/Dev_FullSteps incompatible with fp8 UNET (produces blur of colors, no video)
|
||||
2. **NAG (LTX2_NAG) crashes with MultimodalGuider** (GitHub #535, tensor 1024 vs 6144) — use ConditioningZeroOut for negative prompts
|
||||
3. **euler_ancestral_cfg_pp FAILS with fp8** — use plain euler
|
||||
4. **dev-fp8 + cond-safe is the correct pairing** (NOT distilled + cond-safe)
|
||||
5. **Always debug failed renders with ComfyUI history API:** `curl HOST:8188/history/<prompt_id>` — returns full Python traceback with node ID, exception type, and error message. See §Debugging Failed Renders.
|
||||
|
||||
**FALLBACK: Official Distilled LoRA (rank-384, 7.1 GB) at strength 0.5-0.7.** Use only if TenStrip cond-safe is not available. The official LoRA's cross-attention bridges actively fight I2V conditioning — do NOT use at 1.0.
|
||||
**PREFERRED: TenStrip cond-safe LoRA (rank-72, 662 MB) at strength 1.0, paired with dev-fp8 UNET.** This is purpose-built for I2V — zeroes out cross-attention bridges, adaln/scale-shift tables, gate logits, and prompt scale-shift that fight I2V conditioning. "This is technically what an official I2V distilled lora should have had." Download from huggingface.co/TenStrip/LTX2.3_Distilled_Lora_1.1_Experiments — file: `ltx-2.3-22b-distilled-lora-1.1_fro90_ceil72_condsafe.safetensors`. Safe at 1.0 on first pass I2V. Upscale pass at 0.4.
|
||||
|
||||
**FALLBACK: Official Distilled LoRA (rank-384, 7.1 GB) at strength 0.5-0.7, paired with distilled fp8 UNET.** Use only if TenStrip cond-safe is not available. The official LoRA's cross-attention bridges actively fight I2V conditioning — do NOT use at 1.0.
|
||||
|
||||
**ONLY Architecture: LTXDirector (our proven chain)**
|
||||
```
|
||||
UNETLoader (fp8 distilled transformer)
|
||||
UNETLoader (dev-fp8 transformer) ← MUST be dev, not distilled
|
||||
→ LTX2LoraLoaderAdvanced (TenStrip cond-safe rank-72, strength 1.0) ← PREFERRED
|
||||
→ LTXDirector (single-stage, 16-20 steps)
|
||||
```
|
||||
Uses `guide_strength` parameter (0.80 confirmed). Has timeline_data segments, Director Guide, Director Crop Guides. Produces 121 frames (5.04s) or 241 frames (10s). Frame count verification: 249 = wiring bug.
|
||||
|
||||
**⚠️ Dev_FullSteps native workflow (SamplerCustomAdvanced) is INCOMPATIBLE with fp8 UNET-only.** Confirmed 2026-07-22: Fix 1 test used Dev_FullSteps (SamplerCustomAdvanced, BasicScheduler, KSamplerSelect, CFGGuider, LTX2_NAG) with dev-fp8 UNET — output was a blur of colors, no actual video. The native workflow requires a full checkpoint (46GB). Our fp8 UNET-only setup (23GB) MUST use the LTXDirector chain. Do NOT use SamplerCustomAdvanced, BasicScheduler, or KSamplerSelect with fp8 UNET-only. The NAG+MultimodalGuider incompatibility (GitHub #535) was a secondary issue — even with CFGGuider workaround, the native workflow produces garbage with fp8 UNET-only.
|
||||
|
||||
**ALTERNATE (no cond-safe available):**
|
||||
```
|
||||
UNETLoader (distilled fp8 transformer)
|
||||
→ LTX2LoraLoaderAdvanced (distilled-lora-384-1.1, strength 0.5-0.7) ← FALLBACK
|
||||
→ LTXDirector (single-stage, 16-20 steps)
|
||||
```
|
||||
|
||||
**Why this chain:**
|
||||
- **TenStrip cond-safe at 1.0** — purpose-built for I2V, zeroes out conditioning-fighting layers. Safe at full strength.
|
||||
- **dev-fp8 + TenStrip cond-safe at 1.0** — correct pairing. Cond-safe is a distillation LoRA designed for the dev base. Together they produce the cleanest I2V conditioning.
|
||||
- **NO ID LoRA** — TalkVid-3K is trained for talking-head footage, not action scenes or camera pans. It adds frontal-face bias with no upside. Drop it for ALL scenes unless doing a dedicated talking-head video.
|
||||
- **Single-stage (16-20 steps)** — the two-stage refiner (4 steps at denoise 0.42) may be smearing, not refining. Single-stage at 18 steps produced clean output in the 2-clip test.
|
||||
- **guide_strength 1.0** — I2V conditioning at 1.0 anchors the start frame properly. The old 0.5 was too weak, causing identity drift. The old "DO NOT use 1.0" rule was based on v4 failures where ID LoRA was ALSO at 1.0 — the duplication was from stacked LoRA interference, not from guide_strength alone. With ID LoRA dropped, guide_strength 1.0 is clean.
|
||||
- **guide_strength 0.75-0.85** — community-recommended range for multi-clip pans. The old 1.0 value was tested on the broken double-distillation setup; the correct value for dev-fp8 + cond-safe is being validated (2026-07-22 test in progress at 0.80).
|
||||
|
||||
**ID LoRA drop rule (when keeping it for talking-head only):** Rewire node 131's `model` input from `["201", 0]` to `["200", 0]` and delete node 201.
|
||||
|
||||
@@ -133,6 +198,10 @@ In every scene JSON, ensure:
|
||||
|
||||
Claude Opus SSH'd into 10.0.0.202, read the workflow JSON, read the LTXDirectorGuide source code (`ltx_director_guide.py`), checked ffprobe frame counts, and compared against the shipped reference workflow. Full diagnosis in `references/claude-wiring-diagnosis-2026-07-21.md`.
|
||||
|
||||
### Single-Clip I2V (OLD BASELINE — dev-fp8 + cond-safe, single-stage)
|
||||
|
||||
These settings are for standalone single-clip I2V renders using the dev-fp8 + TenStrip cond-safe chain. For multi-scene Director timelines, see §Config Split below.
|
||||
|
||||
| Setting | Value | Notes |
|
||||
|---------|-------|-------|
|
||||
| Resolution | 768×512 | LTX trained for widescreen; 512×512 is suboptimal |
|
||||
@@ -142,51 +211,173 @@ Claude Opus SSH'd into 10.0.0.202, read the workflow JSON, read the LTXDirectorG
|
||||
| Scheduler | **simple** | NOT linear_quadratic — distilled model is fragile with non-standard schedules |
|
||||
| Steps | 16-20 single-stage | No refiner. Two-stage (8+4) may smear, not refine |
|
||||
| CFG | 1.0 | Distilled model — do NOT raise for standard scenes |
|
||||
| guide_strength | **1.0** | Confirmed clean in 2-clip test (2026-07-22). The old "DO NOT use 1.0" rule was from v4 where ID LoRA was also at 1.0 — the duplication was stacked LoRA interference, not guide_strength alone. With ID LoRA dropped, 1.0 is clean. |
|
||||
| guide_strength | **0.80** (confirmed clean) | 2-clip cyberpunk test 2026-07-22: 0.80 with dev-fp8 + cond-safe produced clean output, no artifacts, smooth pan continuity. Range 0.75-0.85 is safe. The old 1.0 value was tested on the BROKEN double-distillation setup (distilled + cond-safe) — the artifacts were attributed to guide_strength but the root cause was the model/LoRA mismatch. |
|
||||
| Distilled LoRA | **TenStrip cond-safe 1.0** (preferred) or **official 0.5-0.7** (fallback) | Official LoRA at 1.0 degrades I2V quality — use TenStrip cond-safe instead |
|
||||
| ID LoRA | **Dropped** | TalkVid-3K is talking-head only; drop for all non-talking scenes |
|
||||
| Peak VRAM | ~23.5GB / 24GB | |
|
||||
|
||||
### Sampler — plain `euler` ONLY (RESOLVED 2026-07-22)
|
||||
### Multi-Scene Director (PRODUCTION — distilled-1.1 fp8, two-stage hotfix workflow)
|
||||
|
||||
**`euler_ancestral_cfg_pp` FAILS with fp8 distilled model.** Confirmed 2026-07-22: 2-clip test produced catastrophic artifacts — hallucinated wrong man, woman falls face-first, legs turn backwards. Community consensus (RuneXX) was correct: `euler_ancestral_cfg_pp` is for the full dev model (non-distilled, CFG>1), not our fp8 distilled pipeline. The official Lightricks recommendation applies to dev model, not distilled — the conflict was a model mismatch, not a contradiction.
|
||||
These are the settings from the WhatDreamsCost hotfix workflow and the MShahzadAbdulmajeed 11-segment community example. Use these for storybook/multi-scene Director timelines.
|
||||
|
||||
**Use plain `euler` for ALL renders. No exceptions.** Scheduler: `simple`. Steps: 18. CFG: 1.0.
|
||||
| Setting | Value | Notes |
|
||||
|---------|-------|-------|
|
||||
| Resolution | 768×512 | Must be divisible by 32 |
|
||||
| FPS | 24 | Standard cinematic |
|
||||
| Segment duration | ~72 frames / 3s | Community standard from 11-segment example |
|
||||
| Stage 1 steps | **10** | euler, linear_quadratic, denoise 1.0 |
|
||||
| Stage 2 steps | **10** | euler, linear_quadratic, denoise 0.42, spatial x2 upscale |
|
||||
| CFG | 1.0 | Distilled model |
|
||||
| guide_strength | **1.00** | Used across all 11 segments in the community-proven example. Lower = more freedom, higher = tighter anchor. |
|
||||
| UNET | **distilled-1.1 int8_convrot** (NEW Jul 2026) | Faster, less VRAM than fp8_scaled. Fallback: distilled-1.1 fp8_scaled |
|
||||
| Distilled LoRA | Kijai compressed rank-111 (2.74GB) or official 384-1.1 (7.61GB) | |
|
||||
| Spatial Upscaler | **x2-1.1** (MANDATORY) | v1.0 has logo corruption and flickering |
|
||||
| Peak VRAM | ~23.5GB / 24GB | |
|
||||
|
||||
### Sampler — `euler` for distilled, `euler_ancestral` for dev (UPDATED 2026-07-22)
|
||||
|
||||
**`euler_ancestral_cfg_pp` FAILS with fp8 distilled model.** Confirmed 2026-07-22: 2-clip test produced catastrophic artifacts — hallucinated wrong man, woman falls face-first, legs turn backwards. Community consensus (RuneXX) was correct: `euler_ancestral_cfg_pp` is for the full dev model (non-distilled, CFG>1), not our fp8 distilled pipeline.
|
||||
|
||||
**With dev-fp8 + cond-safe (correct pairing):** `euler_ancestral` may work. The Dev_FullSteps workflow uses `euler_ancestral` for pass 1 and `euler` for pass 2. Test `euler_ancestral` on the dev-fp8 + cond-safe setup — if it produces clean output, prefer it for better temporal coherence. If artifacts appear, fall back to plain `euler`.
|
||||
|
||||
**For distilled fp8 (fallback path):** Use plain `euler` ONLY. Scheduler: `simple`. Steps: 8. CFG: 1.0.
|
||||
|
||||
### Duration — 5s (121 frames) Sweet Spot (VALIDATED 2026-07-22)
|
||||
|
||||
Community sweet spot is 5-6s (121f). 10s risks slow-motion. Frame count must be 8n+1 — 121 works (8×15+1). Change `duration_frames: 240` → `121`, `end_frame: 240` → `121`, `duration_seconds: 10.0` → `5.04`. Keep steps=18, scheduler=simple, denoise=1. Confirmed clean 121-frame output in 2-clip test.
|
||||
Community sweet spot is 5-6s (121f requested). 10s risks slow-motion. Frame count must be 8n+1 — 121 works (8×15+1). Change `duration_frames: 240` → `121`, `end_frame: 240` → `121`, `duration_seconds: 10.0` → `5.04`. Keep steps=18, scheduler=simple, denoise=1.
|
||||
|
||||
## LTX Director 2.0 (installed on .202, v2.0.2)
|
||||
**Actual output is 121 frames at 5.04s (24fps).** With 121 requested frames and the Fix 1b single-stage LTXDirector chain (dev-fp8 + cond-safe, euler/simple 18 steps), ffprobe confirms exactly 121 frames, 5.04s. The 8n+1 rounding quirk (producing 129 frames) was observed on the older two-stage workflow — the single-stage chain produces the exact requested frame count. For 10s renders (241 requested), output is 241 frames. Confirmed in 2-clip cyberpunk test 2026-07-22: 121 frames per clip, 242 total after concat, 10.08s, clean output with TenStrip cond-safe + euler.
|
||||
|
||||
## Debugging Failed Renders
|
||||
|
||||
When a render produces garbage output (blur of colors, no video, wrong content), use the ComfyUI history API to get the full traceback. The bundled `comfyui` skill (v5.1.0) provides `fetch_logs.py` for this.
|
||||
|
||||
### Quick diagnosis (from Hermes host)
|
||||
|
||||
```bash
|
||||
# 1. Get recent history entries
|
||||
sshpass -p 'passw0rd' ssh [email protected] "curl -s http://localhost:8188/history | python3 -c \"
|
||||
import json, sys
|
||||
d = json.load(sys.stdin)
|
||||
for pid, entry in sorted(d.items(), key=lambda x: x[1].get('timestamp',0) if isinstance(x[1],dict) else 0, reverse=True)[:5]:
|
||||
if not isinstance(entry, dict): continue
|
||||
status = entry.get('status', {})
|
||||
print(f'{pid}: status={status.get(\\\"status_str\\\",\\\"?\\\")} completed={status.get(\\\"completed\\\",\\\"?\\\")}')
|
||||
\""
|
||||
|
||||
# 2. Get full traceback for a failed prompt_id
|
||||
sshpass -p 'passw0rd' ssh [email protected] "curl -s http://localhost:8188/history/<prompt_id> | python3 -c \"
|
||||
import json, sys
|
||||
d = json.load(sys.stdin)
|
||||
entry = d.get('<prompt_id>', {})
|
||||
status = entry.get('status', {})
|
||||
for msg in status.get('messages', []):
|
||||
if isinstance(msg, list) and msg[0] == 'execution_error':
|
||||
err = msg[1]
|
||||
print(f\\\"Node: {err.get('node_id')} ({err.get('node_type')})\\\")
|
||||
print(f\\\"Error: {err.get('exception_message')}\\\")
|
||||
print(f\\\"Type: {err.get('exception_type')}\\\")
|
||||
\""
|
||||
```
|
||||
|
||||
### Using the bundled comfyui skill's fetch_logs.py
|
||||
|
||||
The official comfyui skill at `~/.hermes/profiles/general/skills/creative/comfyui/scripts/fetch_logs.py` provides a cleaner interface:
|
||||
|
||||
```bash
|
||||
sshpass -p 'passw0rd' ssh [email protected] "cd ~/comfy-ui && python3 /path/to/fetch_logs.py <prompt_id>"
|
||||
```
|
||||
|
||||
### Common failure signatures
|
||||
|
||||
| Symptom | Likely cause | Check |
|
||||
|---------|-------------|-------|
|
||||
| Blur of colors, no video | SamplerCustomAdvanced with fp8 UNET-only | Use LTXDirector chain instead |
|
||||
| `tensor a (1024) must match tensor b (6144)` | NAG + MultimodalGuider incompatibility (#535) | Use CFGGuider with cfg=1.0 |
|
||||
| 249 frames instead of 241 | Director wiring bug (guide frames leaked) | Node 132 latent must be ["55", 2] |
|
||||
| Hallucinated second character | Double-distillation (cond-safe on distilled) | Switch UNET to dev-fp8 |
|
||||
| Frozen objects, static props | guide_strength too high | Drop to 0.75-0.85 |
|
||||
|
||||
Director 2.0 is already on .202 (commit 0dfa657). We use the LTXDirector node — 2.0 is the same node with new features. Key additions relevant to our pipeline:
|
||||
|
||||
### NAG (Negative-Aware Guidance)
|
||||
### NAG (Negative-Aware Guidance) — INCOMPATIBLE, DO NOT USE
|
||||
|
||||
Enables negative prompts at CFG=1 for distilled model. Wiring:
|
||||
- Checkpoint → LTX2 NAG (model input)
|
||||
- LTX2 NAG model output → LoRA chain
|
||||
- Gemma API Text Encode → `nag_cond_video` + `nag_cond_audio`
|
||||
- Gemma API Text Encode → LTXConditioning `negative` input
|
||||
- Remove ConditioningZeroOut node
|
||||
**NAG is incompatible with our fp8 setup.** Confirmed 2026-07-22: LTX2_NAG crashes with MultimodalGuider (GitHub #535, tensor size mismatch 1024 vs 6144). Even with CFGGuider workaround, the Dev_FullSteps native workflow that NAG requires is itself incompatible with fp8 UNET-only (produces blur of colors).
|
||||
|
||||
Known pitfall: last-frame burn if Director Guide positive/negative wiring is wrong (GitHub #153). NAG is the highest-value Director 2.0 feature for our pipeline — evaluate before IC-LoRA.
|
||||
**Use ConditioningZeroOut for negative prompts instead.** This is the confirmed production approach — simple, stable, no compatibility issues.
|
||||
|
||||
### IC-LoRA Ingredients (Character Consistency)
|
||||
NAG may be re-evaluated if we ever switch to a full checkpoint (46GB) that supports the native workflow, but for fp8 UNET-only it's a dead end.
|
||||
|
||||
LTX-2.3-22b-IC-LoRA-Ingredients model for character consistency via reference style sheet. NOT on .202 yet — needs download from HuggingFace (gated, requires "Agree and Access"). LoRA stack: Distilled @ 0.6 + IC-LoRA-Ingredients @ 1.4 + VBVR @ 0.7. Prompt format: `reference: [describe style sheet]` then `Generated video: [describe action]`. Forces AI to match face, clothing, props from reference image. Full details in `references/ic-lora-ingredients.md`.
|
||||
### IC-LoRA Ingredients (Character Consistency) — CONFIRMED WORKING 2026-07-22
|
||||
|
||||
**Stock integration note:** The character stock on TrueNAS (`SHEETS2_00005_Boss.png`, `cyberpunk_woman_neon_*.jpg`) is the input format IC-LoRA Ingredients expects. The 2-clip test uses Flux-generated panoramas — stock materials are NOT wired into the standard I2V pipeline. To use stock as character anchors, IC-LoRA Ingredients is required.
|
||||
LTX-2.3-22b-IC-LoRA-Ingredients model for character consistency via reference style sheet. Downloaded on .202 (1.31GB). **CONFIRMED WORKING with fp8 UNET-only + LTXDirector.** The `LTXICLoRALoaderModelOnly` node accepts standard MODEL input from UNETLoader — no full checkpoint needed. Grok confirmed compatibility; single-clip boss test rendered successfully 2026-07-22.
|
||||
|
||||
**Model chain with IC-LoRA:**
|
||||
```
|
||||
UNETLoader (dev-fp8)
|
||||
→ LTX2LoraLoaderAdvanced (TenStrip cond-safe rank-72, strength 1.0)
|
||||
→ LTXICLoRALoaderModelOnly (Ingredients, strength 1.0)
|
||||
→ LTXDirector
|
||||
```
|
||||
|
||||
**IC-LoRA settings (hybrid, from Grok):**
|
||||
| Setting | Value | Notes |
|
||||
|---------|-------|-------|
|
||||
| Steps | 24 | Up from baseline 18 |
|
||||
| CFG | 2.5 | Up from baseline 1.0 |
|
||||
| IC-LoRA strength | 1.0 | Start here, raise to 1.2-1.3 if identity weak |
|
||||
| cond-safe strength | 1.0 | Unchanged |
|
||||
| guide_strength | 0.80 | Unchanged |
|
||||
| Resolution | 768×448 | IC-LoRA trained bucket (768×512 works but expect mild drift) |
|
||||
| Sampler | euler, simple scheduler | Unchanged |
|
||||
|
||||
**Reference sheet requirements:**
|
||||
- Multi-panel composite on black background: face close-up + body turnaround + key props
|
||||
- Resize to 768×448, loop to 121-frame static video at 24fps
|
||||
- Save to `~/comfy-ui/input/` on .202
|
||||
- Boss sheet: `SHEETS2_00005_Boss.png` (3328×2432) on TrueNAS → resize → `ic_ref_boss_768x448_121f.mp4`
|
||||
|
||||
**IC-LoRA prompt format:**
|
||||
```
|
||||
Reference sheet: <character description — face, body, clothing, key features>. <panel layout description>. Generated video: <standard LTX prompt — shot type, camera, action, lighting, environment>
|
||||
```
|
||||
|
||||
**CRITICAL: IC-LoRA adds ~28 reference frames to the latent.** Output is always 249 frames (not 121). Trim frames 28-148 post-render:
|
||||
```bash
|
||||
ffmpeg -y -i input.mp4 -vf 'select=between(n\,28\,148),setpts=N/FRAME_RATE/TB' -r 24 -c:v libx264 output.mp4
|
||||
```
|
||||
|
||||
**Working wiring (confirmed 2026-07-22):** See `references/ic-lora-wiring.md` for the exact node connections, workflow JSON template, and submission pattern. Key connections:
|
||||
- CFGGuider(17): positive/negative from IC-LoRA Guide (302,0/302,1)
|
||||
- CropGuides(55): positive/negative from IC-LoRA Guide (302,0/302,1)
|
||||
- Concat(18): video_latent from DirectorGuide (133,2) — NOT IC-LoRA Guide
|
||||
- IC-LoRA Guide(302): positive/negative from DirectorGuide (133,0/133,1), latent from DirectorGuide (133,2)
|
||||
|
||||
**IC-LoRA v5 workflow bug (session 2026-07-22):** The shipped `/tmp/api_ic_lora_v5.json` had node 18 video_latent pointing to 302,2 (IC-LoRA Guide latent) instead of 133,2. This leaked the reference sheet into the output (char sheet visible at end of video). Fix with one-time python edit before every IC-LoRA submit:
|
||||
```bash
|
||||
sshpass -p 'passw0rd' ssh [email protected] "
|
||||
python3 -c '
|
||||
import json
|
||||
with open(\"/tmp/api_ic_lora_v5.json\") as f: w = json.load(f)
|
||||
w[\"18\"][\"inputs\"][\"video_latent\"] = [\"133\", 2]
|
||||
with open(\"/tmp/api_ic_lora_v5.json\", \"w\") as f: json.dump(w, f, indent=2)
|
||||
print(\"Fixed node 18 to 133,2\")
|
||||
'
|
||||
"
|
||||
```
|
||||
Always verify the connection after edit. The background render monitor (terminal background=true + notify_on_complete) is the recommended pattern for IC-LoRA renders (24 steps, longer runtime).
|
||||
|
||||
**Stock integration:** The character stock on TrueNAS (`SHEETS2_00005_Boss.png`, `cyberpunk_woman_neon_*.jpg`) is the input format IC-LoRA Ingredients expects. Convert stock images to reference sheet videos for IC-LoRA. For standard I2V without IC-LoRA, use Flux-generated panoramas.
|
||||
|
||||
### IC-LoRA Control (Canny/Depth/Pose)
|
||||
|
||||
Motion transfer from reference video. Not relevant for our I2V pipeline — we control motion via prompts and start frames.
|
||||
|
||||
### guide_strength — RESOLVED (2026-07-22)
|
||||
### guide_strength — CORRECTED (2026-07-22)
|
||||
|
||||
**Use 1.0.** The 2-clip test confirmed: guide_strength 1.0 is clean with ID LoRA dropped. The old "DO NOT use 1.0" rule was based on v4 failures where ID LoRA was ALSO at 1.0 — the duplication was from stacked LoRA interference, not from guide_strength alone. With ID LoRA dropped, 1.0 is the correct value for proper I2V start-frame anchoring.
|
||||
**Use 0.80 (CONFIRMED CLEAN).** The 2-clip cyberpunk test (2026-07-22) with dev-fp8 + cond-safe at guide_strength 0.80 produced clean output on both clips — no artifacts, smooth pan continuity, user-approved. Range 0.75-0.85 is safe; 0.80 is the confirmed production value.
|
||||
|
||||
From the LTX Director GitHub issue #258: "Hard pin = velocity discontinuity. When a pin lands where motion is active, the static frame overrides motion mid-stream → snap." This applies when guide_strength is combined with other strong conditioning (ID LoRA, Transition LoRA). With a clean single-LoRA chain, 1.0 is safe.
|
||||
From the LTX Director GitHub issue #258: "Hard pin = velocity discontinuity. When a pin lands where motion is active, the static frame overrides motion mid-stream → snap." This applies when guide_strength is combined with other strong conditioning (ID LoRA, Transition LoRA). With a clean single-LoRA chain, 0.80 is the recommended starting point.
|
||||
|
||||
**Verification:** After rendering, check the actual guide_strength used by extracting metadata:
|
||||
```bash
|
||||
@@ -204,6 +395,22 @@ ffprobe -v quiet -show_entries format_tags=prompt output.mp4 | grep -oP 'guide_s
|
||||
|
||||
LTX-2.3 needs detailed, structured prompts. Short 1-2 sentence prompts produce bad output: subject duplication, wrong scenes entirely, static camera, identity drift, unnatural motion, flickering. The user confirmed: "I watched the video, it worked but it's all messed up. I think the issue is the prompts."
|
||||
|
||||
### Director-Specific Prompting (from research, July 2026)
|
||||
|
||||
When using LTX Director for multi-segment timelines, the Prompt Relay system (which Director is built on) has specific rules:
|
||||
|
||||
- **Segment 1: Static description only.** No motion. No action. Nothing not present in the input image. Describe only the visible state.
|
||||
- **Segments 2+: Motion changes only.** Describe only what changes or moves during that period. Do NOT re-establish what segment 1 already described.
|
||||
- **Global prompt:** Camera behavior, style, lighting — applied across ALL segments. Example: "static tripod, no pan/zoom/handheld motion, consistent daylight"
|
||||
- **Per-segment prompts:** 1-3 sentences each. Overly long prompts dilute attention.
|
||||
- **Never repeat global-prompt content in per-segment prompts** — causes the model to over-weight static descriptions, reducing motion.
|
||||
- **Prompt Relay syntax:** Pick inline (pipe-separated) OR block (newline headers). Never mix both in one prompt.
|
||||
- **Word count:** No strict limit, but community pattern is concise. The MShahzadAbdulmajeed 11-segment example uses short action-focused prompts per segment.
|
||||
|
||||
Sources: Kijai's PromptRelay README, MShahzadAbdulmajeed 11-segment workflow (July 2026).
|
||||
|
||||
### Standalone I2V Prompting (original rules, still valid for single-clip renders)
|
||||
|
||||
**Two confirmed failure modes from our renders:**
|
||||
|
||||
1. **Subject duplication** — "The man getting out of bed split into two men. One got out of bed and he was still in bed at the same time." Root cause: `guide_strength: 1.0` (hard pin) + no camera direction + no motion detail.
|
||||
@@ -265,16 +472,17 @@ Shot + Camera → Subject + Action → Lighting + Environment → Audio + Mood
|
||||
|
||||
The 7 fixes below supersede the old 6-agent review (2026-07-21). These are the new defaults.
|
||||
|
||||
### The 6 Fixes (ranked by impact, v2 corrected 2026-07-22)
|
||||
### The 7 Fixes (ranked by impact, v3 corrected 2026-07-22)
|
||||
|
||||
| # | Fix | From | To | Why |
|
||||
|---|-----|------|----|-----|
|
||||
| 1 | ID LoRA | TalkVid-3K (talking-head) | Drop entirely | Trained for static faces, fights against action motion |
|
||||
| 2 | I2V conditioning | 0.5 | 1.0 | Too weak to anchor start frame — causes identity drift |
|
||||
| 3 | Stages | Two-stage (8+4 refiner) | Single-stage (16-20 steps) | Refiner may be smearing, not refining |
|
||||
| 4 | Resolution | 512×512 | 768×512 | LTX trained for widescreen, square is suboptimal |
|
||||
| 5 | Distilled LoRA | Official 384 @ 1.0 | **TenStrip cond-safe @ 1.0** (preferred) or official @ 0.5-0.7 (fallback) | Official LoRA fights I2V conditioning at high strength. TenStrip cond-safe zeroes out those layers. |
|
||||
| 6 | Prompts | 150-200 words | 2-3 actions, 50-80 words | LTX can only execute 2-3 simultaneous actions |
|
||||
| 1 | Model/LoRA pairing | distilled + cond-safe (DOUBLE-DISTILLATION) | **dev-fp8 + cond-safe** (correct pairing) | Cond-safe is a distillation LoRA — must pair with dev base. Double-distillation causes hallucinated environments, characters, and frozen objects. See `references/double-distillation-artifacts.md`. |
|
||||
| 2 | ID LoRA | TalkVid-3K (talking-head) | Drop entirely | Trained for static faces, fights against action motion |
|
||||
| 3 | guide_strength | 1.0 (tested on broken setup) | **0.80 (CONFIRMED)** | 2-clip test passed clean at 0.80 with dev-fp8 + cond-safe. Range 0.75-0.85 is safe. |
|
||||
| 4 | Stages | Two-stage (8+4 refiner) | Single-stage (16-20 steps) | Refiner may be smearing, not refining |
|
||||
| 5 | Resolution | 512×512 | 768×512 | LTX trained for widescreen, square is suboptimal |
|
||||
| 6 | Distilled LoRA | Official 384 @ 1.0 | **TenStrip cond-safe @ 1.0** (preferred, with dev-fp8) or official @ 0.5-0.7 (fallback, with distilled) | Official LoRA fights I2V conditioning at high strength. TenStrip cond-safe zeroes out those layers — but MUST be paired with dev-fp8. |
|
||||
| 7 | Prompts | 150-200 words | 2-3 actions, 50-80 words | LTX can only execute 2-3 simultaneous actions |
|
||||
|
||||
**⚠️ Fix #5 was CORRECTED by follow-up deep research (2026-07-22, 15 sources).** The original claim (Distilled LoRA 0.7→1.0) was wrong. Community consensus: official Distilled LoRA at 1.0 causes quality degradation for I2V. The correct range is 0.5-0.7. TenStrip cond-safe is the solution — purpose-built for I2V at 1.0.
|
||||
|
||||
@@ -292,13 +500,14 @@ See `references/artifact-deep-research-template.md` for the question template us
|
||||
|
||||
### Fix Application Order (test after each)
|
||||
|
||||
1. Drop ID LoRA — zero-cost, biggest single impact
|
||||
2. Switch to single-stage 16-20 steps — eliminates refiner smear risk
|
||||
3. Download TenStrip cond-safe LoRA — purpose-built for I2V at 1.0
|
||||
4. Set guide_strength to 1.0 — proper I2V anchoring (safe with ID LoRA dropped)
|
||||
5. Switch to 768×512 — widescreen training distribution
|
||||
6. Simplify prompts to 50-80 words — 2-3 actions max
|
||||
7. Git pull deps (with caution — see Pitfalls: may land incompatible native workflows)
|
||||
1. Fix model/LoRA pairing — switch UNET to dev-fp8 + keep cond-safe @ 1.0. HIGHEST impact, fixes double-distillation root cause. ✅ DONE.
|
||||
2. Lower guide_strength to 0.80 — confirmed clean in 2-clip test. ✅ DONE.
|
||||
3. ~~Enable NAG with strong negatives~~ — NAG is INCOMPATIBLE with fp8 UNET-only. Use ConditioningZeroOut instead. ✅ DONE.
|
||||
4. Drop ID LoRA — zero-cost, eliminates talking-head bias on action scenes. ✅ DONE.
|
||||
5. Switch to single-stage 16-20 steps — eliminates refiner smear risk. ✅ DONE.
|
||||
6. Switch to 768×512 — widescreen training distribution. ✅ DONE.
|
||||
7. Simplify prompts to 50-80 words — 2-3 actions max.
|
||||
8. Git pull deps (with caution — see Pitfalls: may land incompatible native workflows)
|
||||
|
||||
### Negative Prompt (updated)
|
||||
|
||||
@@ -339,86 +548,83 @@ Added the text/watermark line — LTX-2.3's upscaler v1.0 was trained on data co
|
||||
5. Vary lighting and atmosphere per scene to match the setting
|
||||
6. Test one scene first before rendering all 6
|
||||
7. **MANDATORY: Validate every prompt against the 10-point checklist in `references/prompt-validation-checklist.md` BEFORE submitting to render queue.** This is a FIRM quality gate — do not skip. The user's standing rule: "ensure prompts match ltx standard. ALWAYS. This should be FIRM in memory and validate before EVERY run." If any check fails, fix the prompt and re-validate. Do not submit a workflow with a failing prompt.
|
||||
8. Verify guide_strength is 1.0 in every scene JSON before submitting
|
||||
8. Verify guide_strength is 0.80 in every scene JSON before submitting
|
||||
|
||||
## Skill Boundary: Which ComfyUI Skill to Use
|
||||
## Storybook Pipeline (NEW — 2026-07-29, peer-reviewed by Grok + Claude)
|
||||
|
||||
Two ComfyUI skills exist in this profile. Use the RIGHT one for the task — they serve different purposes and using the wrong one wastes time.
|
||||
Repeatable template system for story-driven multi-scene video using LTX Director. Full plan: `~/workspace/general/ltx23_plan.md` (also in ltx23_kb).
|
||||
|
||||
| Task | Use This Skill | Why |
|
||||
|------|---------------|-----|
|
||||
| LTX video renders (any kind) | **`media/ltx-video-pipeline`** (this skill) | Has the 6-fix baseline, prompt standards, pitfalls, Director wiring fix, TrueNAS workflow |
|
||||
| Prompt validation for LTX | **`media/ltx-video-pipeline`** | 10-point checklist, character descriptions, zhuanchang rules |
|
||||
| Workflow JSON building for LTX | **`media/ltx-video-pipeline`** | Model chain, node wiring, segment format, duration control |
|
||||
| Story structure / scene planning | **`media/ltx-video-pipeline`** | Story templates, causal chain rules, 2-clip test pattern |
|
||||
| TrueNAS uploads of LTX output | **`media/ltx-video-pipeline`** | Naming convention, smbclient pattern, stock directory layout |
|
||||
| ComfyUI health check | **`creative/comfyui`** (official v5.1.0) | `health_check.py` script, server reachability, model inventory |
|
||||
| Missing node/model install | **`creative/comfyui`** | `check_deps.py` + `auto_fix_deps.py` |
|
||||
| Generic image generation (SD, Flux) | **`creative/comfyui`** | `run_workflow.py`, batch sweeps, parameter injection |
|
||||
| Workflow schema extraction | **`creative/comfyui`** | `extract_schema.py` |
|
||||
| WebSocket monitoring | **`creative/comfyui`** | `ws_monitor.py` |
|
||||
| Non-LTX video (Wan, AnimateDiff) | **`creative/comfyui`** | Example workflows, video generation support |
|
||||
### How Director Is Actually Used (User's Workflow)
|
||||
|
||||
**Rule of thumb:** If the task involves LTX on .202, use this skill. If it's general ComfyUI management or non-LTX generation, use `creative/comfyui`.
|
||||
Director is a **visual timeline editor** inside ComfyUI — not a programmatic pipeline. The user loads the Director workflow, drops images on tracks, pastes prompts, and hits render. Director handles everything internally: joining scenes, audio generation, upscaling.
|
||||
|
||||
## ComfyUI Skill Integration (2026-07-22)
|
||||
**Incremental build pattern** (the correct workflow, not monolithic timeline):
|
||||
1. Generate first clip (up to 20s) with a start frame + prompt
|
||||
2. Load that clip back into Director, extend duration (e.g., 20s → 40s)
|
||||
3. Provide a prompt for the NEW section only
|
||||
4. Director generates the extension and joins it seamlessly to the existing clip
|
||||
5. Repeat: 40s → 60s, 60s → 80s, etc.
|
||||
|
||||
**Status: INSTALLED (v5.1.0).** The official Hermes ComfyUI skill is installed at `~/.hermes/profiles/general/skills/creative/comfyui/`. It provides agentic workflow automation — parameter injection, batch runs, real-time monitoring, node/model management — without manual UI interaction.
|
||||
**Keyframe placement**: Images can go at start, middle, OR end of any segment. This means you can anchor the beginning of a scene, the midpoint, or the destination.
|
||||
|
||||
**How it was installed (2026-07-22):**
|
||||
```bash
|
||||
# The skill lives in the Hermes bundle (~/.hermes/skills/creative/comfyui/) but is
|
||||
# NOT auto-seeded into profiles. `hermes update` updates the bundle, not the profile.
|
||||
# `hermes skills install comfyui` finds community/hub matches but NOT the bundled one.
|
||||
# Correct method: copy from bundle to profile.
|
||||
cp -r ~/.hermes/skills/creative/comfyui ~/.hermes/profiles/general/skills/creative/
|
||||
**Why this matters**: Each segment is rendered and reviewed before extending. If segment 3 fails, you only re-render that extension — not the whole timeline. This is the correct pattern; do NOT design for monolithic all-at-once rendering.
|
||||
|
||||
**What the agent produces:**
|
||||
1. Global prompt (camera/style/lighting for all scenes)
|
||||
2. Per-scene table: keyframe prompt (for Qwen Image) + Director prompt
|
||||
3. Build plan: incremental order, keyframe placement, durations
|
||||
4. **Generated keyframe images** — agent submits Qwen T2I/I2I jobs via ComfyUI API on .202, downloads outputs, uploads to TrueNAS keyframes/ directory
|
||||
5. User loads keyframes into Director, pastes prompts, renders manually
|
||||
|
||||
**Pre-generation validation:** Before generating any keyframes, cross-reference the story against the ltx23_kb Qdrant collection for prompting rule compliance. Search for "LTX Director prompting rules", "prompt format", "common mistakes", and "keyframe best practices". Flag any violations (e.g., Segment 1 containing motion, global-prompt content repeated in per-segment, missing FLF2V where appropriate).
|
||||
|
||||
### Core Design (from peer review)
|
||||
|
||||
1. **Don't generate workflow JSON — patch frozen golden graphs.** Workflow topology bugs (wiring, crop guides, IC-LoRA) are the #1 failure mode. Keep 2-3 golden graph files, patch only whitelisted fields: `timeline_data` string, image filenames, seed, dims, frame count. Both Grok and Claude independently flagged this.
|
||||
2. **Per-segment rendering, not monolithic timeline.** A 6-scene timeline that dies at scene 5 costs the whole run. Render one segment per job, validate each (`frame count == duration*fps+1`), keep it, move on. Enables cheap single-shot re-rolls.
|
||||
3. **Character consistency is won at the keyframe stage.** Prompt-only drifts by scene 3. Lock a character reference sheet and inject it into every keyframe generation prompt. IC-LoRA helps within LTX but can't fix drift Flux already introduced.
|
||||
4. **Silent audio for v1.** LTX 2.3 does joint AV generation natively. Per-scene TTS inside ComfyUI fights that. Render silent or with LTX native ambience, mix narration in the ffmpeg assembly step.
|
||||
5. **Start simple — one opinionated path.** Flux|Qwen × start-only|FLF2V × IC-LoRA|prompt-only × TTS|silent = 16 paths. Ship one end-to-end first (Flux keyframes, start-only, prompt-only, silent), then parameterize.
|
||||
|
||||
### Pipeline Stages
|
||||
```
|
||||
story.json → keyframe_prompts.json → [validate against ltx23_kb] → [agent generates keyframes via Qwen on .202] → keyframes on TrueNAS → [user loads into Director] → [user renders on .202] → assembly (ffmpeg concat + TrueNAS upload)
|
||||
```
|
||||
|
||||
**Post-install configuration for remote ComfyUI (10.0.0.202):**
|
||||
```bash
|
||||
# Set default host (or pass --host to every command)
|
||||
export COMFYUI_HOST="http://10.0.0.202:8188"
|
||||
**Pre-generation validation (MANDATORY):** Before generating any keyframes, cross-reference the story against the ltx23_kb Qdrant collection. Search for "LTX Director prompting rules", "prompt format", "common mistakes", and "keyframe best practices". Flag violations: Segment 1 containing motion, global-prompt content repeated in per-segment, missing FLF2V where appropriate, keyframe density below community standard (~3s per keyframe). Fix violations before generating images.
|
||||
|
||||
# Verify (health_check.py defaults to localhost — must pass --host)
|
||||
python3 ~/.hermes/profiles/general/skills/creative/comfyui/scripts/health_check.py \
|
||||
--host http://10.0.0.202:8188
|
||||
### Config Split (Clarified 2026-07-29)
|
||||
|
||||
# Smoke test
|
||||
python3 ~/.hermes/profiles/general/skills/creative/comfyui/scripts/run_workflow.py \
|
||||
--workflow ~/.hermes/profiles/general/skills/creative/comfyui/workflows/sd15_txt2img.json \
|
||||
--args '{"prompt": "test", "steps": 4}' \
|
||||
--host http://10.0.0.202:8188 \
|
||||
--output-dir /tmp/comfyui-test
|
||||
```
|
||||
The skill previously mixed two different chains. They are separate:
|
||||
|
||||
**Scripts included:**
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `hardware_check.py` | Probe GPU/VRAM → recommend local vs Comfy Cloud |
|
||||
| `extract_schema.py` | List controllable params + model deps from workflow |
|
||||
| `check_deps.py` | Check workflow against running server for missing nodes/models |
|
||||
| `auto_fix_deps.py` | Auto-install missing nodes/models |
|
||||
| `run_workflow.py` | Inject params, submit, monitor, download outputs |
|
||||
| `run_batch.py` | Submit workflow N times with sweeps, parallel up to tier limit |
|
||||
| `ws_monitor.py` | Real-time WebSocket viewer for executing jobs |
|
||||
| `health_check.py` | Verification checklist: comfy-cli + server + models + smoke test |
|
||||
| `fetch_logs.py` | Pull traceback/status messages for a given prompt_id |
|
||||
| Use Case | Chain | Steps | Scheduler | Model |
|
||||
|----------|-------|-------|-----------|-------|
|
||||
| Single-clip I2V (old baseline) | dev-fp8 + cond-safe, single-stage LTXDirector | 18 | simple | dev-fp8 |
|
||||
| Multi-scene Director (hotfix workflow) | distilled-1.1 fp8, two-stage (8+4) | 8+4 | linear_quadratic | distilled-1.1 fp8 |
|
||||
|
||||
**Known issues with bundled workflows (discovered 2026-07-22):**
|
||||
The Director hotfix workflow from WhatDreamsCost uses distilled-1.1 fp8 with the two-stage pipeline (Stage 1: 8 steps denoise 1.0, Stage 2: 4 steps denoise 0.42, spatial x2 upscale). This is the **production Director chain**. The old single-stage dev-fp8 + cond-safe chain is for standalone I2V clips, not multi-scene Director timelines.
|
||||
|
||||
1. **`_comment` string field crashes ComfyUI v0.27.0.** The bundled `sd15_txt2img.json` has a top-level `"_comment"` string field. ComfyUI v0.27.0's `execution.py:1121` expects `_meta` to be a dict and crashes with `AttributeError: 'str' object has no attribute 'get'` when it encounters a string. Fix: strip `_comment` before submitting. The `run_workflow.py` script does NOT strip this automatically — pre-process the workflow JSON first:
|
||||
```python
|
||||
import json
|
||||
with open('workflow.json') as f: wf = json.load(f)
|
||||
wf.pop('_comment', None) # Remove the string field
|
||||
with open('workflow_fixed.json', 'w') as f: json.dump(wf, f)
|
||||
```
|
||||
### Storybook Pitfalls (from "Burn the Helix" session, 2026-07-29)
|
||||
|
||||
2. **Checkpoint name mismatch.** The bundled `sd15_txt2img.json` expects `v1-5-pruned-emaonly.safetensors` but the .202 server has `v1-5-pruned.safetensors`. Always verify checkpoint names against `comfy model list` or `/api/experiment/models/checkpoints` before submitting.
|
||||
1. **Qwen I2I official template has placeholder images.** The Comfy-Org `qwen_image_edit_2511.json` template ships with `LoadImage` nodes pointing to `leather_sofa.png` and `texture_fur.png` — these don't exist on disk. Before submitting, change both LoadImage nodes: node 41 → character portrait, node 83 → empty environment T2I output. Otherwise you get 5 node errors on submit.
|
||||
|
||||
3. **Health check defaults to localhost.** `health_check.py` defaults to `http://127.0.0.1:8188`. For remote ComfyUI, always pass `--host http://10.0.0.202:8188`.
|
||||
2. **FLF2V is a pixel morph, not an animation.** LTX Director's FLF2V interpolates pixels between start and end keyframes — it does NOT animate the prompt's action. Result: empty alley fades to a still image of Kael kneeling, with no walk-in, no kneel, no motion. **For scenes where action matters more than destination** (walk-ins, climbs, runs, reveals): use start-only I2V (one keyframe at start, no end keyframe). **For scenes where destination matters more** (specific pose, chip falling, exact framing): use FLF2V. This is the single most important keyframe strategy decision per scene.
|
||||
|
||||
---
|
||||
3. **Start+mid keyframes cause early character bleed.** When using start (empty) + mid (character) keyframes, Director blends toward the mid keyframe — the character appears in frame well before the midpoint. Avoid start+mid entirely. Use either start-only I2V (action scenes) or FLF2V (destination scenes).
|
||||
|
||||
4. **Gaze direction drifts downward.** LTX over-interprets "walking" as looking at the ground. End keyframes with "gaze forward hard" still produce downward gaze. Fix: add "eyes locked on distant [target], chin up" to end keyframe prompts. The explicit upward cue counteracts the walking-downward association.
|
||||
|
||||
5. **One scene at a time.** Do not present all scenes' prompts at once. The user works incrementally — generate images for one scene, render, review, then move to the next. Only show the current scene's Director prompt, total time, and keyframe placement. Before giving Director prompts for a scene, confirm the user has generated the keyframe images for that scene — don't assume they exist.
|
||||
|
||||
6. **Compounding timeline is what the user enters.** Director's "extend timeline" field takes the TOTAL time, not the new scene's duration. Show it as: "Scene 2 — Dead Drop Alley (12s) — Total: 22s". Keep a running table: 10s → 22s → 32s → 44s → 58s → 68s → 80s.
|
||||
|
||||
7. **Keyframe naming convention.** sN-1 = scene N start frame (empty environment, T2I). sN-2 = scene N end frame (character in scene, I2I with portrait as image1). Plus one base character portrait (T2I, used as image1 reference for all I2I generations).
|
||||
|
||||
8. **Qwen I2I official template has model subdirectory mismatch.** The Comfy-Org template expects models in the root of `models/loras/` and `models/diffusion_models/`, but they may be in subdirectories (e.g., `Qwen-Image/Edit/`). ComfyUI only scans the root of each model folder. Fix: create symlinks from the root to the actual files. Example: `ln -sf Qwen-Image/Edit/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors ~/comfy-ui/models/loras/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors`. Same pattern for diffusion_models. Refresh ComfyUI (F5) after creating symlinks — it rescans on reload.
|
||||
|
||||
### LTX Director Reference
|
||||
|
||||
Full reference: `references/ltx-director-2.0-how-it-works.md` — timeline_data schema, keyframe loading from source code, end frames feature (native FLF2V), two-stage pipeline architecture, model list, example workflows. From README + source code + hotfix workflow JSON (2026-07-29).
|
||||
|
||||
## Story Structure (CRITICAL — session 2026-07-21)
|
||||
|
||||
@@ -460,7 +666,7 @@ The user initially asked for simpler stories, then reversed: "go back to your or
|
||||
6. Update state file
|
||||
7. Only then start the next story
|
||||
|
||||
**Before scaling to a full 6-scene story, test with 2 clips first.** A 2-clip test (shared background, camera pan between subjects) validates the model chain, prompt quality, and transition smoothness at minimal cost before committing to a full render. See `references/claude-live-ssh-inspection.md` for the pattern of having Claude SSH into .202 to inspect live state and build the test plan.
|
||||
**CRITICAL: 2-clip tests ONLY until everything is 100%.** Do NOT scale to 6 scenes until: transition is smooth, character is consistent, no artifacts. Fix one thing at a time on 2 clips, validate with user, then next fix. User directive 2026-07-22: "Stick with 2 scenes until everything is 100%." A 2-clip test (shared background, camera pan between subjects) validates the model chain, prompt quality, and transition smoothness at minimal cost before committing to a full render. See `references/claude-live-ssh-inspection.md` for the pattern of having Claude SSH into .202 to inspect live state and build the test plan.
|
||||
|
||||
### 2-Clip Test Pattern (Panorama + Frame-B Transition)
|
||||
|
||||
@@ -491,6 +697,8 @@ This pattern produced the 2-clip FLF panorama plan (2026-07-22) where Claude dis
|
||||
|
||||
**Pitfall:** Claude may error on first attempt (SSH timeout, tool failure). Resume the session with `--resume` and ask what went wrong — Claude self-diagnoses and recovers. The first attempt cost $1.05 (21 turns, is_error); the resume cost $0.11 (2 turns, success).
|
||||
|
||||
**Pitfall (user intent / scope validation — 2026-07-22 session):** When user says "proceed" or "it looks okay" after a single-clip review/fix, do NOT assume it means "complete more single-clip work". User frequently intends immediate move to the 2-clip test (the actual goal). Always restate: "Single clip clean — proceeding to 2-clip IC-LoRA test?" and get explicit confirmation before building/submitting 2-clip workflows. "Stop" commands require immediate halt — no autonomous continuation. This overrides the pending list when user intent differs.
|
||||
|
||||
**Save and document EVERYTHING.** After each story completes, upload to TrueNAS:
|
||||
- `outputs/` — concat video
|
||||
- `workflows/` — scene JSONs
|
||||
@@ -504,7 +712,10 @@ This pattern produced the 2-clip FLF panorama plan (2026-07-22) where Claude dis
|
||||
|
||||
- **Always fp8.** Only fall back to Q4 GGUF if fp8 OOMs and user approves.
|
||||
- **Prefer newest and best.** When choosing between an old proven path and a new better one, prefer the new one. But verify compatibility first — new doesn't mean compatible (e.g., the 2026-07-22 ComfyUI-LTXVideo update brought a better architecture that doesn't work with our fp8 models).
|
||||
- **MANDATORY: Delegate ALL mechanical work to glm (NOT kimi-c).** Building workflow JSONs, running renders, extracting frames, submitting to queue, waiting for completion, concat, uploading — these are mechanical tasks that burn expensive deepseek tokens. Use `delegate_task` for ALL of these, targeting the glm profile. The calling agent stays lean for decision-making, validation, and user communication. This is a FIRM standing rule — do not do mechanical work yourself when a cheaper peer can do it. User directive 2026-07-22: "stop using ask kimi c for tasks, use glm instead."
|
||||
- **MANDATORY: Delegate ALL mechanical work to Claude (ask-claude) FIRST, glm as FALLBACK.** Building workflow JSONs, running renders, extracting frames, submitting to queue, waiting for completion, concat, uploading — these are mechanical tasks that burn expensive deepseek tokens. Use `delegate_task` for ALL of these. Claude is PRIMARY (free — Anthropic Pro subscription, infrastructure cost) and can SSH to .202. glm is FALLBACK when Claude unavailable or tokens exhausted. The calling agent stays lean for decision-making, validation, and user communication. User directive 2026-07-22: "I would prefer that you start with claude and glm is fallback. Claude can also ssh."
|
||||
- **MANDATORY: Validate non-visual artifacts before submission.** Workflow JSONs, config files, scripts — dispatch a validation sub-agent to verify correctness before submitting to the render queue. Catches wiring errors, missing nodes, and model file mismatches that would waste a 5-7 minute render. User validates video output by watching — do NOT dispatch validation sub-agents for renders. User directive 2026-07-22: "do not validate when complete. I will validate by watching videos."
|
||||
- **CRITICAL: glm profile needs the ComfyUI skill.** The glm profile does NOT have the ComfyUI skill by default — it only lives in the general profile. Before dispatching LTX work to glm, verify the skill is available: `hermes -p glm skills list | grep comfyui`. If missing, copy it: `cp -r ~/.hermes/profiles/general/skills/creative/comfyui ~/.hermes/profiles/glm/skills/creative/comfyui`. Then include "FIRST: Load the ComfyUI skill (skill_view(name='comfyui'))" in the glm prompt. The skill provides `run_workflow.py`, `ws_monitor.py`, and `extract_schema.py` — use `--host http://10.0.0.202:8188` for remote execution. Confirmed 2026-07-22: first glm dispatch was missing the skill and had to be killed + re-dispatched.
|
||||
- **CRITICAL: Dispatch glm in background mode for LTX renders.** LTX renders take 5-7 minutes per clip plus workflow building and concat — easily 15+ minutes total. The foreground 600s timeout will kill the dispatch mid-render. Use `terminal(background=true, notify_on_complete=true)` for all glm LTX dispatches. Confirmed 2026-07-22: 2-clip cyberpunk render completed successfully in background mode.
|
||||
- **File-based API submission.** Do NOT inline complex JSON in curl commands — Hermes' command parser blocks them. Write the workflow to a file (e.g., `/tmp/submit_ltx_render.sh`), then execute with `bash /tmp/submit_ltx_render.sh`. See `references/api-submission-pattern.md` for the complete pattern including polling, verification, and TrueNAS upload.
|
||||
- **ALWAYS validate with glm after creating or updating anything.** Workflow JSONs, config changes, file writes — dispatch glm via `delegate_task` to read the file and verify correctness before submitting to the render queue. Glm catches wiring errors, missing nodes, and model file mismatches that would waste a 5-7 minute render. This is a quality gate, not optional.
|
||||
- **Test one scene before rendering all 6.**
|
||||
@@ -523,7 +734,7 @@ Models on .202:
|
||||
- CLIP: `t5xxl_fp8_e4m3fn.safetensors` + `clip_l.safetensors` in `models/text_encoders/`
|
||||
- VAE: `ae.safetensors` in `models/vae/`
|
||||
|
||||
Working node graph (512×512 txt2img):
|
||||
Working node graph (768×512 txt2img — preferred resolution for LTX widescreen):
|
||||
```json
|
||||
{
|
||||
"5": {"class_type": "CLIPTextEncode", "inputs": {"text": "<PROMPT>", "clip": ["11", 0]}},
|
||||
@@ -535,10 +746,12 @@ Working node graph (512×512 txt2img):
|
||||
"11": {"class_type": "DualCLIPLoader", "inputs": {"clip_name1": "t5xxl_fp8_e4m3fn.safetensors", "clip_name2": "clip_l.safetensors", "type": "flux"}},
|
||||
"16": {"class_type": "UNETLoader", "inputs": {"unet_name": "flux1-dev-fp8-e4m3fn.safetensors", "weight_dtype": "fp8_e4m3fn"}},
|
||||
"3": {"class_type": "KSampler", "inputs": {"seed": 42, "steps": 20, "cfg": 1.0, "sampler_name": "euler", "scheduler": "simple", "denoise": 1.0, "model": ["16", 0], "positive": ["7", 0], "negative": ["6", 0], "latent_image": ["27", 0]}},
|
||||
"27": {"class_type": "EmptySD3LatentImage", "inputs": {"width": 512, "height": 512, "batch_size": 1}}
|
||||
"27": {"class_type": "EmptySD3LatentImage", "inputs": {"width": 768, "height": 512, "batch_size": 1}}
|
||||
}
|
||||
```
|
||||
|
||||
Full template at `templates/flux-768x512-txt2img.json`. The old 512×512 template (`templates/flux-512-txt2img.json`) is kept for reference but 768×512 is the production default.
|
||||
|
||||
**Submission:** MUST wrap in `{"prompt": <workflow>}` envelope. The ComfyUI API rejects bare workflow JSON with `"no_prompt"` error.
|
||||
|
||||
```bash
|
||||
@@ -564,6 +777,7 @@ curl -s -X POST http://localhost:8188/prompt -H "Content-Type: application/json"
|
||||
- Save to `/tmp/api_scene_XX.json`
|
||||
- **To modify and re-render existing workflows** (change sampler, duration, etc. without rebuilding): see `references/modify-and-re-render-recipe.md` — proven pattern with Python in-place editing, verification, and submission.
|
||||
- **To programmatically build multiple clip workflows from templates** (common modifications across C1/C2 with per-clip overrides): see `references/programmatic-workflow-modification.md` — scp a Python script to .202, avoids shell heredoc quoting issues.
|
||||
- **To recover a lost workflow from a successful render's output file:** see `references/workflow-extraction-from-output.md` — extract the full workflow JSON from the output MP4's metadata (ffprobe → prompt tag). Use this when the original `/tmp/` JSON was deleted or you need to audit what actually ran.
|
||||
|
||||
### 4. Submit to Queue
|
||||
```bash
|
||||
@@ -659,11 +873,55 @@ All stock materials (start frames, audio, character refs, workflows, outputs) ar
|
||||
|
||||
**Using stock as I2V start frames:** See `references/stock-as-i2v-start-frame.md` — pattern for using character stock images as direct I2V start frames without IC-LoRA Ingredients. Simpler, works with existing 6-fix baseline. For multi-clip character consistency, use IC-LoRA Ingredients instead.
|
||||
|
||||
## Freshness Protocol — Version-Aware Research (FIRM — user directive 2026-07-30)
|
||||
|
||||
**The AI tools space moves weekly.** APIs break, nodes get renamed, workflows from last month don't run today. A 4-month-old ComfyUI tutorial is archaeology. The agent MUST NOT find an old blog post and confidently apply it without verification.
|
||||
|
||||
### Mandatory Before Any Fix or Recommendation
|
||||
|
||||
1. **Check installed version first.** Before searching "how to do X", determine what version is actually running on the target system. The fix for ComfyUI nightly is different from the fix for 3.12.0 stable.
|
||||
```bash
|
||||
# ComfyUI version
|
||||
ssh [email protected] "cd ~/comfy-ui && git log -1 --format='%h %s %ai'"
|
||||
# Custom nodes
|
||||
ssh [email protected] "cd ~/comfy-ui/custom_nodes/ComfyUI-LTXDirector && git log -1 --format='%h %s %ai'"
|
||||
```
|
||||
|
||||
2. **Check release notes before random fixes.** When something breaks, check the project's GitHub releases/changelog FIRST — the breakage is often documented as an intentional change. Do NOT jump to searching error messages.
|
||||
|
||||
3. **Date-gate all searches.** For fast-moving projects, use time filters: `after:2026-06-01` or "past month". A blog post from February titled "How to use ComfyUI MultiGPU" is almost certainly wrong today.
|
||||
|
||||
4. **Prefer official docs over tutorials.** The project's own README, docs site, or GitHub releases page is more likely current than a third-party tutorial.
|
||||
|
||||
5. **Test one atomic step before building the whole plan.** Verify step 1 actually works before writing step 2. A 10-step workflow where step 7 fails because step 1's assumptions were wrong wastes hours.
|
||||
|
||||
6. **Cross-reference with ltx23_kb first.** Search the knowledge base for what we already know about the current version before searching the web.
|
||||
|
||||
### Search Query Template for Fast-Moving Projects
|
||||
|
||||
```
|
||||
site:github.com/{owner}/{repo} (release OR changelog OR breaking) after:{3_months_ago}
|
||||
site:github.com/{owner}/{repo}/issues (error OR fix OR workaround) updated:>={last_week}
|
||||
"{project} {current_version}" (fix OR error OR workaround OR update)
|
||||
```
|
||||
|
||||
### Red Flags (abort and re-verify)
|
||||
|
||||
- Tutorial/blog post older than 3 months with no version mentioned
|
||||
- GitHub issue marked "closed" but the fix references a different version
|
||||
- Reddit thread where the top comment says "this is outdated, use X instead"
|
||||
- Any source that says "just do Y" without mentioning a specific version
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **CRITICAL: Double-distillation — TenStrip cond-safe LoRA MUST be paired with dev-fp8 UNET, NOT distilled.** The cond-safe LoRA is a distillation LoRA. Running it on the already-distilled transformer damages conditioning paths, causing hallucinated environments (closet doors), hallucinated characters (second girl), and object persistence artifacts (frozen cup on tricep). From the model page: "Applying them on top of an already-distilled model double-distills and degrades quality." Fix: switch UNET to dev-fp8 (already on disk at `~/comfy-ui/models/diffusion_models/ltx-2.3-22b-dev_transformer_only_fp8_scaled.safetensors`, 22GB) OR drop cond-safe and use pure distilled. See `references/double-distillation-artifacts.md` for full root cause analysis with Grok diagnosis, secondary causes, and fix priority order. Confirmed 2026-07-22: 2-clip cyberpunk test produced major artifacts from this exact misconfiguration.
|
||||
- **CRITICAL: True-extend (extract last frame → use as next clip start) carries latent memory.** LTX remembers subjects from prior clips even when the start frame shows empty space. Confirmed 2026-07-22: Clip 2 hallucinated a second woman stepping up from the couch where the man was in Clip 1. Fix: use the pre-rendered empty mid-room frame B as Clip 2's start frame instead of the extracted last frame. Clean start, no latent memory. See §2-Clip Test Pattern.
|
||||
- **ffmpeg last-frame extraction:** `ffmpeg -sseof -3 -i input.mp4 -vsync vfr -q:v 2 output.png` may fail. Working alternative: `ffmpeg -sseof -3 -i input.mp4 -update 1 -frames:v 1 -q:v 2 output.png`. The `-update 1` flag is needed for single-frame PNG output from video input.
|
||||
- **CRITICAL: git pull on ComfyUI-LTXVideo may land incompatible native workflows.** The 2026-07-22 update (30K lines) brought a new native single-stage architecture (CheckpointLoaderSimple, LTXVScheduler, SamplerCustomAdvanced, GemmaAPITextEncode) that uses full checkpoints (46 GB), not our fp8 UNET-only transformer (23 GB). The new workflow drops LTXDirector entirely. Our fp8 chain (UNETLoader → LTX2LoraLoaderAdvanced → LTXDirector) is incompatible with the new architecture. **Before git pulling, snapshot the working state.** If the pull lands new example workflows, do NOT assume they work with our fp8 models — verify compatibility first. The new architecture requires a full checkpoint that won't fit 24 GB VRAM.
|
||||
- **CRITICAL: Start frame resolution determines output resolution.** LTXDirector doubles the start frame dimensions. 256×256 start frame → 512×512 output. 768×512 start frame → 768×512 output. The `custom_width`/`custom_height` inputs on LTXDirector do NOT override — resolution is derived from the start frame. Generate Flux start frames at the TARGET output resolution. Confirmed 2026-07-22: Story 1 v1 rendered at 512×512 because start frames were 256×256.
|
||||
- **CRITICAL: Sub-agent validation mandate.** After any glm subagent completes work (renders, file ops, JSON builds), re-dispatch a separate validation subagent to independently verify every output — ffprobe resolution/frame counts, file existence, concat integrity. Never trust subagent output without independent verification. User directive 2026-07-22: "After glm does the work, re-dispatch to validate. Every time."
|
||||
- **Subagent timeout: was 600s, now 1200s (20 min).** `delegation.child_timeout_seconds` and `terminal.timeout` both set to 1200 across general, grok45, and grok43 profiles (2026-07-22). This covers most single-clip renders (~3-4 min) plus workflow building. For multi-scene renders exceeding 20 min, still use `terminal(background=true, notify_on_complete=true)`. Confirmed 2026-07-22: Story 1 delegation timed out at 600s with 57 API calls — renders completed but subagent couldn't report. IC-LoRA delegation also timed out at 600s mid-render.
|
||||
- **CRITICAL: Dev_FullSteps native workflow (SamplerCustomAdvanced) produces garbage with fp8 UNET-only.** Confirmed 2026-07-22: Fix 1 test used Dev_FullSteps (SamplerCustomAdvanced, BasicScheduler, KSamplerSelect, CFGGuider) with dev-fp8 UNET — output was a blur of colors, no actual video. The native workflow requires a full checkpoint (46GB). Our fp8 UNET-only setup (23GB) MUST use the LTXDirector chain (UNETLoader → LTX2LoraLoaderAdvanced → LTXDirector). Do NOT use SamplerCustomAdvanced, BasicScheduler, or KSamplerSelect with fp8 UNET-only.
|
||||
- **CRITICAL: Upload to TrueNAS after EVERY render.** The user can only view videos on TrueNAS (10.0.0.117), not in LXC containers. scp from .202 to Hermes host, then smbclient to TrueNAS. Do NOT skip this — the user has no other way to review output. (Learned 2026-07-22: user said "I can only view videos on truenas, not in lxc's.")
|
||||
- **CRITICAL: Do NOT interrupt a running Claude session.** If Claude is mid-build (background process running), do not scp new question files or send follow-ups unless the user explicitly says "tell Claude now." Wait for the session to complete. Mid-build interruptions can cause the session to error out. (Learned 2026-07-22: sent a fact_store instruction mid-build; user corrected: "I said when claude is complete.")
|
||||
- **CRITICAL: Claude's intermediate errors are normal self-correction.** Claude will make mistakes, hit errors, then try a different approach. `is_error: true` on intermediate turns does NOT mean the session failed. Wait for the final turn output. Only diagnose after the background process completes. Jumping to conclusions mid-session wastes turns and money. (Learned 2026-07-22: killed a working session twice, $1.78 wasted. User: "you just need to wait for the final turn output. Not jump to conclusions.")
|
||||
@@ -682,3 +940,16 @@ All stock materials (start frames, audio, character refs, workflows, outputs) ar
|
||||
- **Duration control: set `segment[\"length\"]`, NOT `duration_seconds` or `segment[\"end\"]`.** The LTXDirector ignores node-level `duration_seconds` and segment `end` — it uses `segment[\"length\"]` as the frame count. For 5s@24fps: `td[\"segments\"][0][\"length\"] = 120`. Setting only `duration_seconds` or `end` silently produces the full 10s output. Confirmed 2026-07-21: two failed attempts before finding the correct field.
|
||||
- **Story must have a causal chain.** Standalone vignettes (wake up → kitchen → porch → forest) produce visual inconsistencies because the model has no narrative thread. Each scene must follow from the previous one. See `references/story-structure-guide.md` for the full 4-story templates and design rules.
|
||||
- **User prefers detailed stories over simplified ones.** When the user said "keep it simple" and then "go back to your original," the original detailed templates were preferred. Don't over-simplify — the 4-story cyberpunk templates with loglines, scene tables, and character descriptions are the right level of detail.
|
||||
- **CRITICAL: Do NOT copy ComfyUI built-in blueprints as workflow files.** The blueprints in `~/comfy-ui/blueprints/` use subgraph wrappers (single node of type `fd6ee5f8-...`) that are NOT complete workflows. They only work when loaded through ComfyUI's template browser. For direct workflow files, download the official expanded templates from Comfy-Org GitHub: `https://raw.githubusercontent.com/Comfy-Org/workflow_templates/refs/heads/main/templates/image_qwen_Image_2512.json` (T2I) and `https://raw.githubusercontent.com/Comfy-Org/workflow_templates/refs/heads/main/templates/image_qwen_image_edit_2511.json` (I2I). These have full node graphs (4-6 nodes with links and groups). (Learned 2026-07-29: copied blueprints as workflows; user said "YOU DID NOT SET THE QWEN WORKFLOWS UP CORRECTLy" — blueprints are incomplete subgraph wrappers.)
|
||||
- **CRITICAL: Prompt wardrobe/appearance audit before EVERY render.** Mismatched wardrobe terms between prompts and reference frames cost a 35-minute re-render (RapVidFinal, 2026-07-30). Before queueing any multi-scene workflow, grep all PROMPT nodes for wardrobe/appearance terms (colors, clothing items, accessories) and verify they match the reference frame spec. A 30-second grep saves a full re-render. Example: `grep -i 'tracksuit\|jacket\|shirt\|pants\|chains\|color' workflow.json | sort | uniq -c`. If any prompt says "black" and frames are "red", fix before queueing. This is a mandatory pre-queue step.
|
||||
- **I2V Prompt Engineering (RapVidFinal lessons, 2026-07-30):** See `references/ltx23-i2v-prompt-engineering.md` — proven patterns from 3 rounds of Claude+Grok dual-agent validation: brand name stripping, I2V static wardrobe rule, action thinning (2-3 max), physical aggression cues, lightning through-line, single camera/lighting logic per scene. All 8 RapVidFinalHours prompts passed final validation at 60-93 words each.
|
||||
- **CRITICAL: Do NOT overcomplicate. Do exactly what the user asks — nothing more.** One command that succeeds = task complete. STOP. No follow-up research, no validation, no alternative downloads, no format checks, no "let me also..." additions. The user will tell you if something is wrong. This is the #1 user frustration pattern — agent completes the task then keeps going with unnecessary extras. (Learned 2026-07-29: user said "STOP", "Why do you keep making things so complicated???", "stop. Stop doing overkill. JUST DO simply what I ask.") The pattern `ssh ... python3 -c "import json; open('/tmp/file.json','w').write(escaped_json)"` fails because shell escaping corrupts the JSON (exit code 2, no file created). The working pattern: write the JSON locally → `sshpass -p 'passw0rd' scp local.json n8n@10.0.0.202:/tmp/remote.json` → then curl from .202. Confirmed 2026-07-22: 6 LTX scene JSONs failed to write via inline Python; scp succeeded on all 6.plates from Comfy-Org GitHub: `https://raw.githubusercontent.com/Comfy-Org/workflow_templates/refs/heads/main/templates/image_qwen_Image_2512.json` (T2I) and `https://raw.githubusercontent.com/Comfy-Org/workflow_templates/refs/heads/main/templates/image_qwen_image_edit_2511.json` (I2I). These have full node graphs (4-6 nodes with links and groups). (Learned 2026-07-29: copied blueprints as workflows; user said "YOU DID NOT SET THE QWEN WORKFLOWS UP CORRECTLy" — blueprints are incomplete subgraph wrappers.)
|
||||
- **CRITICAL: Prompt wardrobe/appearance audit before EVERY render.** Mismatched wardrobe terms between prompts and reference frames cost a 35-minute re-render (RapVidFinal, 2026-07-30). Before queueing any multi-scene workflow, grep all PROMPT nodes for wardrobe/appearance terms (colors, clothing items, accessories) and verify they match the reference frame spec. A 30-second grep saves a full re-render. Example: `grep -i 'tracksuit\|jacket\|shirt\|pants\|chains\|color' workflow.json | sort | uniq -c`. If any prompt says "black" and frames are "red", fix before queueing. This is a mandatory pre-queue step.
|
||||
- **I2V Prompt Engineering (RapVidFinal lessons, 2026-07-30):** See `references/ltx23-i2v-prompt-engineering.md` — proven patterns from 3 rounds of Claude+Grok dual-agent validation: brand name stripping, I2V static wardrobe rule, action thinning (2-3 max), physical aggression cues, lightning through-line, single camera/lighting logic per scene. All 8 RapVidFinalHours prompts passed final validation at 60-93 words each.
|
||||
- **CRITICAL: Do NOT overcomplicate. Do exactly what the user asks — nothing more.** One command that succeeds = task complete. STOP. No follow-up research, no validation, no alternative downloads, no format checks, no "let me also..." additions. The user will tell you if something is wrong. This is the #1 user frustration pattern — agent completes the task then keeps going with unnecessary extras. (Learned 2026-07-29: user said "STOP", "Why do you keep making things so complicated???", "stop. Stop doing overkill. JUST DO simply what I ask.") The pattern `ssh ... python3 -c "import json; open('/tmp/file.json','w').write(escaped_json)"` fails because shell escaping corrupts the JSON (exit code 2, no file created). The working pattern: write the JSON locally → `sshpass -p 'passw0rd' scp local.json n8n@10.0.0.202:/tmp/remote.json` → then curl from .202. Confirmed 2026-07-22: 6 LTX scene JSONs failed to write via inline Python; scp succeeded on all 6.
|
||||
- **CRITICAL: IC-LoRA adds ~28 reference frames to the latent — always trim post-render.** Output is 249 frames regardless of requested duration. Trim frames 28-148 with ffmpeg: `ffmpeg -y -i input.mp4 -vf 'select=between(n\,28\,148),setpts=N/FRAME_RATE/TB' -r 24 -c:v libx264 output.mp4`. Do NOT try to fix this in the workflow — the extra frames are inherent to how LTXAddVideoICLoRAGuide injects reference conditioning. Confirmed 2026-07-22: 4 renders, all produced 249 frames.
|
||||
- **CRITICAL: Subagent delegation timeout (600s) is too short for IC-LoRA renders.** IC-LoRA renders take longer (24 steps vs 18, higher CFG) plus workflow building and reference sheet conversion. The subagent timed out at 600s mid-render. Use `delegate_task` with the understanding that the subagent may time out — check the live transcript to see how far it got, then finish the work yourself. The subagent did the heavy lifting (reference sheet conversion, workflow building, first render) — the orchestrator just needed to fix wiring and resubmit. Confirmed 2026-07-22: Claude subagent timed out at 600s with 25 API calls, but the reference video and workflow JSON were correct.
|
||||
- **CRITICAL: Do NOT copy ComfyUI built-in blueprints as workflow files.** The blueprints in `~/comfy-ui/blueprints/` use subgraph wrappers (single node of type `fd6ee5f8-...`) that are NOT complete workflows. They only work when loaded through ComfyUI's template browser. For direct workflow files, download the official expanded templates from Comfy-Org GitHub: `https://raw.githubusercontent.com/Comfy-Org/workflow_templates/refs/heads/main/templates/image_qwen_Image_2512.json` (T2I) and `https://raw.githubusercontent.com/Comfy-Org/workflow_templates/refs/heads/main/templates/image_qwen_image_edit_2511.json` (I2I). These have full node graphs (4-6 nodes with links and groups). (Learned 2026-07-29: copied blueprints as workflows; user said "YOU DID NOT SET THE QWEN WORKFLOWS UP CORRECTLy" — blueprints are incomplete subgraph wrappers.)
|
||||
- **CRITICAL: Prompt wardrobe/appearance audit before EVERY render.** Mismatched wardrobe terms between prompts and reference frames cost a 35-minute re-render (RapVidFinal, 2026-07-30). Before queueing any multi-scene workflow, grep all PROMPT nodes for wardrobe/appearance terms (colors, clothing items, accessories) and verify they match the reference frame spec. A 30-second grep saves a full re-render. Example: `grep -i 'tracksuit\|jacket\|shirt\|pants\|chains\|color' workflow.json | sort | uniq -c`. If any prompt says "black" and frames are "red", fix before queueing. This is a mandatory pre-queue step.
|
||||
- **I2V Prompt Engineering (RapVidFinal lessons, 2026-07-30):** See `references/ltx23-i2v-prompt-engineering.md` — proven patterns from 3 rounds of Claude+Grok dual-agent validation: brand name stripping, I2V static wardrobe rule, action thinning (2-3 max), physical aggression cues, lightning through-line, single camera/lighting logic per scene. All 8 RapVidFinalHours prompts passed final validation at 60-93 words each.
|
||||
- **CRITICAL: Do NOT overcomplicate. Do exactly what the user asks — nothing more.** One command that succeeds = task complete. STOP. No follow-up research, no validation, no alternative downloads, no format checks, no "let me also..." additions. The user will tell you if something is wrong. This is the #1 user frustration pattern — agent completes the task then keeps going with unnecessary extras. (Learned 2026-07-29: user said "STOP", "Why do you keep making things so complicated???", "stop. Stop doing overkill. JUST DO simply what I ask.") The pattern `ssh ... python3 -c "import json; open('/tmp/file.json','w').write(escaped_json)"` fails because shell escaping corrupts the JSON (exit code 2, no file created). The working pattern: write the JSON locally → `sshpass -p 'passw0rd' scp local.json n8n@10.0.0.202:/tmp/remote.json` → then curl from .202. Confirmed 2026-07-22: 6 LTX scene JSONs failed to write via inline Python; scp succeeded on all 6.
|
||||
|
||||
Reference in New Issue
Block a user