Fix: flatten comfyui and ltx-video-pipeline to Gitea convention (flat skill dirs). comfyui v5.1.0, ltx-video-pipeline v2.0.0.
This commit is contained in:
+184
-10
@@ -15,6 +15,31 @@ metadata:
|
||||
|
||||
End-to-end LTX Video pipeline on the ComfyUI LXC at 10.0.0.202.
|
||||
|
||||
## Naming Convention (FIRM — user directive 2026-07-22)
|
||||
|
||||
All output files uploaded to TrueNAS MUST use this format:
|
||||
|
||||
```
|
||||
LTX_YYYY-MM-DD_<what>_<sampler>_<duration>.mp4
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `LTX_2026-07-22_cyberpunk-boss-pan_euler_10s.mp4`
|
||||
- `LTX_2026-07-22_cyberpunk-boss-pan_euler-ancestral_5s.mp4`
|
||||
- `LTX_2026-07-22_boss-stock-test_euler-ancestral_10s.mp4`
|
||||
|
||||
Components:
|
||||
- `YYYY-MM-DD`: render date
|
||||
- `<what>`: short kebab-case description of content (e.g., `cyberpunk-boss-pan`, `boss-stock-test`, `woman-window-reveal`)
|
||||
- `<sampler>`: `euler` or `euler-ancestral`
|
||||
- `<duration>`: `5s` or `10s`
|
||||
|
||||
Do NOT use generic names like `LTX_cyberpunk_C1_00005_.mp4` or `LTX_2clip_test_C1_00002_.mp4` — these are ComfyUI auto-generated filenames that convey zero information about what's in the video. Rename during the concat/upload step.
|
||||
|
||||
## Character Descriptions
|
||||
|
||||
See `references/character-descriptions.md` for the canonical character descriptions used in prompts and Flux panorama generation. These are the ground truth — do not improvise or use generic descriptions.
|
||||
|
||||
## Environment
|
||||
|
||||
| Setting | Value |
|
||||
@@ -113,7 +138,7 @@ Claude Opus SSH'd into 10.0.0.202, read the workflow JSON, read the LTXDirectorG
|
||||
| Resolution | 768×512 | LTX trained for widescreen; 512×512 is suboptimal |
|
||||
| FPS | 24 | Standard cinematic |
|
||||
| Duration | 5s (121 frames) for testing, 10s (241 frames) for final | Test at 5s first — temporal coherence degrades after 5-6s |
|
||||
| Sampler | euler | |
|
||||
| Sampler | **euler** (ONLY — mandatory) | euler_ancestral_cfg_pp FAILS catastrophically with fp8 distilled model (hallucinated subjects, backwards limbs). Plain euler only. See §Sampler below. |
|
||||
| Scheduler | **simple** | NOT linear_quadratic — distilled model is fragile with non-standard schedules |
|
||||
| Steps | 16-20 single-stage | No refiner. Two-stage (8+4) may smear, not refine |
|
||||
| CFG | 1.0 | Distilled model — do NOT raise for standard scenes |
|
||||
@@ -122,6 +147,41 @@ Claude Opus SSH'd into 10.0.0.202, read the workflow JSON, read the LTXDirectorG
|
||||
| ID LoRA | **Dropped** | TalkVid-3K is talking-head only; drop for all non-talking scenes |
|
||||
| Peak VRAM | ~23.5GB / 24GB | |
|
||||
|
||||
### Sampler — plain `euler` ONLY (RESOLVED 2026-07-22)
|
||||
|
||||
**`euler_ancestral_cfg_pp` FAILS with fp8 distilled model.** Confirmed 2026-07-22: 2-clip test produced catastrophic artifacts — hallucinated wrong man, woman falls face-first, legs turn backwards. Community consensus (RuneXX) was correct: `euler_ancestral_cfg_pp` is for the full dev model (non-distilled, CFG>1), not our fp8 distilled pipeline. The official Lightricks recommendation applies to dev model, not distilled — the conflict was a model mismatch, not a contradiction.
|
||||
|
||||
**Use plain `euler` for ALL renders. No exceptions.** Scheduler: `simple`. Steps: 18. CFG: 1.0.
|
||||
|
||||
### Duration — 5s (121 frames) Sweet Spot (VALIDATED 2026-07-22)
|
||||
|
||||
Community sweet spot is 5-6s (121f). 10s risks slow-motion. Frame count must be 8n+1 — 121 works (8×15+1). Change `duration_frames: 240` → `121`, `end_frame: 240` → `121`, `duration_seconds: 10.0` → `5.04`. Keep steps=18, scheduler=simple, denoise=1. Confirmed clean 121-frame output in 2-clip test.
|
||||
|
||||
## LTX Director 2.0 (installed on .202, v2.0.2)
|
||||
|
||||
Director 2.0 is already on .202 (commit 0dfa657). We use the LTXDirector node — 2.0 is the same node with new features. Key additions relevant to our pipeline:
|
||||
|
||||
### NAG (Negative-Aware Guidance)
|
||||
|
||||
Enables negative prompts at CFG=1 for distilled model. Wiring:
|
||||
- Checkpoint → LTX2 NAG (model input)
|
||||
- LTX2 NAG model output → LoRA chain
|
||||
- Gemma API Text Encode → `nag_cond_video` + `nag_cond_audio`
|
||||
- Gemma API Text Encode → LTXConditioning `negative` input
|
||||
- Remove ConditioningZeroOut node
|
||||
|
||||
Known pitfall: last-frame burn if Director Guide positive/negative wiring is wrong (GitHub #153). NAG is the highest-value Director 2.0 feature for our pipeline — evaluate before IC-LoRA.
|
||||
|
||||
### IC-LoRA Ingredients (Character Consistency)
|
||||
|
||||
LTX-2.3-22b-IC-LoRA-Ingredients model for character consistency via reference style sheet. NOT on .202 yet — needs download from HuggingFace (gated, requires "Agree and Access"). LoRA stack: Distilled @ 0.6 + IC-LoRA-Ingredients @ 1.4 + VBVR @ 0.7. Prompt format: `reference: [describe style sheet]` then `Generated video: [describe action]`. Forces AI to match face, clothing, props from reference image. Full details in `references/ic-lora-ingredients.md`.
|
||||
|
||||
**Stock integration note:** The character stock on TrueNAS (`SHEETS2_00005_Boss.png`, `cyberpunk_woman_neon_*.jpg`) is the input format IC-LoRA Ingredients expects. The 2-clip test uses Flux-generated panoramas — stock materials are NOT wired into the standard I2V pipeline. To use stock as character anchors, IC-LoRA Ingredients is required.
|
||||
|
||||
### IC-LoRA Control (Canny/Depth/Pose)
|
||||
|
||||
Motion transfer from reference video. Not relevant for our I2V pipeline — we control motion via prompts and start frames.
|
||||
|
||||
### guide_strength — RESOLVED (2026-07-22)
|
||||
|
||||
**Use 1.0.** The 2-clip test confirmed: guide_strength 1.0 is clean with ID LoRA dropped. The old "DO NOT use 1.0" rule was based on v4 failures where ID LoRA was ALSO at 1.0 — the duplication was from stacked LoRA interference, not from guide_strength alone. With ID LoRA dropped, 1.0 is the correct value for proper I2V start-frame anchoring.
|
||||
@@ -273,12 +333,92 @@ Added the text/watermark line — LTX-2.3's upscaler v1.0 was trained on data co
|
||||
### Prompting Workflow
|
||||
|
||||
1. Write the full prompt with all required elements (shot, camera, action, lighting, environment)
|
||||
2. Only append `zhuanchang` if the scene involves a transformation/morph
|
||||
3. Use the same character description across all scenes (from global prompt or character sheet)
|
||||
4. Vary lighting and atmosphere per scene to match the setting
|
||||
5. Test one scene first before rendering all 6
|
||||
6. **MANDATORY: Validate every prompt against the 10-point checklist in `references/prompt-validation-checklist.md` BEFORE submitting to render queue.** This is a FIRM quality gate — do not skip. The user's standing rule: "ensure prompts match ltx standard. ALWAYS. This should be FIRM in memory and validate before EVERY run." If any check fails, fix the prompt and re-validate. Do not submit a workflow with a failing prompt.
|
||||
7. Verify guide_strength is 1.0 in every scene JSON before submitting
|
||||
2. **Use canonical character descriptions from `references/character-descriptions.md`** — do NOT improvise or use generic descriptions. The user provided exact descriptions for each character; use them verbatim in every prompt.
|
||||
3. Only append `zhuanchang` if the scene involves a transformation/morph
|
||||
4. Use the same character description across all scenes (from `references/character-descriptions.md` or character sheet)
|
||||
5. Vary lighting and atmosphere per scene to match the setting
|
||||
6. Test one scene first before rendering all 6
|
||||
7. **MANDATORY: Validate every prompt against the 10-point checklist in `references/prompt-validation-checklist.md` BEFORE submitting to render queue.** This is a FIRM quality gate — do not skip. The user's standing rule: "ensure prompts match ltx standard. ALWAYS. This should be FIRM in memory and validate before EVERY run." If any check fails, fix the prompt and re-validate. Do not submit a workflow with a failing prompt.
|
||||
8. Verify guide_strength is 1.0 in every scene JSON before submitting
|
||||
|
||||
## Skill Boundary: Which ComfyUI Skill to Use
|
||||
|
||||
Two ComfyUI skills exist in this profile. Use the RIGHT one for the task — they serve different purposes and using the wrong one wastes time.
|
||||
|
||||
| Task | Use This Skill | Why |
|
||||
|------|---------------|-----|
|
||||
| LTX video renders (any kind) | **`media/ltx-video-pipeline`** (this skill) | Has the 6-fix baseline, prompt standards, pitfalls, Director wiring fix, TrueNAS workflow |
|
||||
| Prompt validation for LTX | **`media/ltx-video-pipeline`** | 10-point checklist, character descriptions, zhuanchang rules |
|
||||
| Workflow JSON building for LTX | **`media/ltx-video-pipeline`** | Model chain, node wiring, segment format, duration control |
|
||||
| Story structure / scene planning | **`media/ltx-video-pipeline`** | Story templates, causal chain rules, 2-clip test pattern |
|
||||
| TrueNAS uploads of LTX output | **`media/ltx-video-pipeline`** | Naming convention, smbclient pattern, stock directory layout |
|
||||
| ComfyUI health check | **`creative/comfyui`** (official v5.1.0) | `health_check.py` script, server reachability, model inventory |
|
||||
| Missing node/model install | **`creative/comfyui`** | `check_deps.py` + `auto_fix_deps.py` |
|
||||
| Generic image generation (SD, Flux) | **`creative/comfyui`** | `run_workflow.py`, batch sweeps, parameter injection |
|
||||
| Workflow schema extraction | **`creative/comfyui`** | `extract_schema.py` |
|
||||
| WebSocket monitoring | **`creative/comfyui`** | `ws_monitor.py` |
|
||||
| Non-LTX video (Wan, AnimateDiff) | **`creative/comfyui`** | Example workflows, video generation support |
|
||||
|
||||
**Rule of thumb:** If the task involves LTX on .202, use this skill. If it's general ComfyUI management or non-LTX generation, use `creative/comfyui`.
|
||||
|
||||
## ComfyUI Skill Integration (2026-07-22)
|
||||
|
||||
**Status: INSTALLED (v5.1.0).** The official Hermes ComfyUI skill is installed at `~/.hermes/profiles/general/skills/creative/comfyui/`. It provides agentic workflow automation — parameter injection, batch runs, real-time monitoring, node/model management — without manual UI interaction.
|
||||
|
||||
**How it was installed (2026-07-22):**
|
||||
```bash
|
||||
# The skill lives in the Hermes bundle (~/.hermes/skills/creative/comfyui/) but is
|
||||
# NOT auto-seeded into profiles. `hermes update` updates the bundle, not the profile.
|
||||
# `hermes skills install comfyui` finds community/hub matches but NOT the bundled one.
|
||||
# Correct method: copy from bundle to profile.
|
||||
cp -r ~/.hermes/skills/creative/comfyui ~/.hermes/profiles/general/skills/creative/
|
||||
```
|
||||
|
||||
**Post-install configuration for remote ComfyUI (10.0.0.202):**
|
||||
```bash
|
||||
# Set default host (or pass --host to every command)
|
||||
export COMFYUI_HOST="http://10.0.0.202:8188"
|
||||
|
||||
# Verify (health_check.py defaults to localhost — must pass --host)
|
||||
python3 ~/.hermes/profiles/general/skills/creative/comfyui/scripts/health_check.py \
|
||||
--host http://10.0.0.202:8188
|
||||
|
||||
# Smoke test
|
||||
python3 ~/.hermes/profiles/general/skills/creative/comfyui/scripts/run_workflow.py \
|
||||
--workflow ~/.hermes/profiles/general/skills/creative/comfyui/workflows/sd15_txt2img.json \
|
||||
--args '{"prompt": "test", "steps": 4}' \
|
||||
--host http://10.0.0.202:8188 \
|
||||
--output-dir /tmp/comfyui-test
|
||||
```
|
||||
|
||||
**Scripts included:**
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `hardware_check.py` | Probe GPU/VRAM → recommend local vs Comfy Cloud |
|
||||
| `extract_schema.py` | List controllable params + model deps from workflow |
|
||||
| `check_deps.py` | Check workflow against running server for missing nodes/models |
|
||||
| `auto_fix_deps.py` | Auto-install missing nodes/models |
|
||||
| `run_workflow.py` | Inject params, submit, monitor, download outputs |
|
||||
| `run_batch.py` | Submit workflow N times with sweeps, parallel up to tier limit |
|
||||
| `ws_monitor.py` | Real-time WebSocket viewer for executing jobs |
|
||||
| `health_check.py` | Verification checklist: comfy-cli + server + models + smoke test |
|
||||
| `fetch_logs.py` | Pull traceback/status messages for a given prompt_id |
|
||||
|
||||
**Known issues with bundled workflows (discovered 2026-07-22):**
|
||||
|
||||
1. **`_comment` string field crashes ComfyUI v0.27.0.** The bundled `sd15_txt2img.json` has a top-level `"_comment"` string field. ComfyUI v0.27.0's `execution.py:1121` expects `_meta` to be a dict and crashes with `AttributeError: 'str' object has no attribute 'get'` when it encounters a string. Fix: strip `_comment` before submitting. The `run_workflow.py` script does NOT strip this automatically — pre-process the workflow JSON first:
|
||||
```python
|
||||
import json
|
||||
with open('workflow.json') as f: wf = json.load(f)
|
||||
wf.pop('_comment', None) # Remove the string field
|
||||
with open('workflow_fixed.json', 'w') as f: json.dump(wf, f)
|
||||
```
|
||||
|
||||
2. **Checkpoint name mismatch.** The bundled `sd15_txt2img.json` expects `v1-5-pruned-emaonly.safetensors` but the .202 server has `v1-5-pruned.safetensors`. Always verify checkpoint names against `comfy model list` or `/api/experiment/models/checkpoints` before submitting.
|
||||
|
||||
3. **Health check defaults to localhost.** `health_check.py` defaults to `http://127.0.0.1:8188`. For remote ComfyUI, always pass `--host http://10.0.0.202:8188`.
|
||||
|
||||
---
|
||||
|
||||
## Story Structure (CRITICAL — session 2026-07-21)
|
||||
|
||||
@@ -364,9 +504,9 @@ This pattern produced the 2-clip FLF panorama plan (2026-07-22) where Claude dis
|
||||
|
||||
- **Always fp8.** Only fall back to Q4 GGUF if fp8 OOMs and user approves.
|
||||
- **Prefer newest and best.** When choosing between an old proven path and a new better one, prefer the new one. But verify compatibility first — new doesn't mean compatible (e.g., the 2026-07-22 ComfyUI-LTXVideo update brought a better architecture that doesn't work with our fp8 models).
|
||||
- **MANDATORY: Delegate ALL mechanical work to kimi-c.** Building workflow JSONs, running renders, extracting frames, submitting to queue, waiting for completion, concat, uploading — these are mechanical tasks that burn expensive deepseek tokens. Use `ask kimi-c` (peer agent on kimi-c profile) for ALL of these. The calling agent stays lean for decision-making, validation, and user communication. This is a FIRM standing rule — do not do mechanical work yourself when a cheaper peer can do it. The user's directive: "You SHOULD BE TASKING kimi c to save tokens, NOT YOU."
|
||||
- **File-based dispatch for kimi-c (avoid shell quoting failures).** Multi-line prompts with quotes leak out of `hermes -q` and get parsed as CLI arguments. Write the full prompt to a temp file (e.g. `/tmp/kimi-render-pipeline.txt`), then dispatch with `hermes -p kimi-c chat -q "Read /tmp/kimi-render-pipeline.txt and execute ALL steps" -Q --max-turns 600 --yolo`. Use `terminal(background=true, notify_on_complete=true)` for any dispatch expected to take more than ~5 minutes. This is the canonical pattern — it also satisfies the "never paste file content into the prompt" rule from the ask-kimi-c skill.
|
||||
- **ALWAYS validate with kimi-c after creating or updating anything.** Workflow JSONs, config changes, file writes — dispatch kimi-c to read the file and verify correctness before submitting to the render queue. Kimi-c catches wiring errors, missing nodes, and model file mismatches that would waste a 5-7 minute render. This is a quality gate, not optional.
|
||||
- **MANDATORY: Delegate ALL mechanical work to glm (NOT kimi-c).** Building workflow JSONs, running renders, extracting frames, submitting to queue, waiting for completion, concat, uploading — these are mechanical tasks that burn expensive deepseek tokens. Use `delegate_task` for ALL of these, targeting the glm profile. The calling agent stays lean for decision-making, validation, and user communication. This is a FIRM standing rule — do not do mechanical work yourself when a cheaper peer can do it. User directive 2026-07-22: "stop using ask kimi c for tasks, use glm instead."
|
||||
- **File-based API submission.** Do NOT inline complex JSON in curl commands — Hermes' command parser blocks them. Write the workflow to a file (e.g., `/tmp/submit_ltx_render.sh`), then execute with `bash /tmp/submit_ltx_render.sh`. See `references/api-submission-pattern.md` for the complete pattern including polling, verification, and TrueNAS upload.
|
||||
- **ALWAYS validate with glm after creating or updating anything.** Workflow JSONs, config changes, file writes — dispatch glm via `delegate_task` to read the file and verify correctness before submitting to the render queue. Glm catches wiring errors, missing nodes, and model file mismatches that would waste a 5-7 minute render. This is a quality gate, not optional.
|
||||
- **Test one scene before rendering all 6.**
|
||||
- **Verify frame count after every render (ffprobe).**
|
||||
- **Save all artifacts to TrueNAS immediately** (user can only view videos on TrueNAS, not in LXCs).
|
||||
@@ -422,6 +562,8 @@ curl -s -X POST http://localhost:8188/prompt -H "Content-Type: application/json"
|
||||
- Modify: start_frame path, audio path, segment prompt
|
||||
- For Transition LoRA: add `LoraLoaderModelOnly` node, rewire chain, append `zhuanchang` to prompts
|
||||
- Save to `/tmp/api_scene_XX.json`
|
||||
- **To modify and re-render existing workflows** (change sampler, duration, etc. without rebuilding): see `references/modify-and-re-render-recipe.md` — proven pattern with Python in-place editing, verification, and submission.
|
||||
- **To programmatically build multiple clip workflows from templates** (common modifications across C1/C2 with per-clip overrides): see `references/programmatic-workflow-modification.md` — scp a Python script to .202, avoids shell heredoc quoting issues.
|
||||
|
||||
### 4. Submit to Queue
|
||||
```bash
|
||||
@@ -440,6 +582,34 @@ sshpass -p 'passw0rd' ssh [email protected] "while true; do q=\$(curl -s http://loc
|
||||
```
|
||||
|
||||
### 6. Identify Output Files
|
||||
|
||||
**Preferred: Use ComfyUI history API (reliable, works regardless of output directory).** After queue drains, query the history for each prompt_id to get exact filenames:
|
||||
|
||||
```bash
|
||||
# Get output filename from prompt_id
|
||||
curl -s http://localhost:8188/history/<prompt_id> | python3 -c "
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
entry=d.get('<prompt_id>',{})
|
||||
outputs=entry.get('outputs',{})
|
||||
node37=outputs.get('37',{})
|
||||
images=node37.get('images',[])
|
||||
for img in images:
|
||||
print(img.get('filename','?'))
|
||||
"
|
||||
```
|
||||
|
||||
**Fallback: Search both output directories.** Outputs land in `~/comfy-ui/output/` or `~/comfy-ui/output/video/` depending on the workflow's `filename_prefix`. Use `find` with a reference file timestamp:
|
||||
|
||||
```bash
|
||||
# Find all mp4s newer than a known reference file
|
||||
find ~/comfy-ui/output -name "*.mp4" -newer /tmp/api_2clip_C1_v2.json -type f
|
||||
|
||||
# Or list by modification time
|
||||
ls -lt ~/comfy-ui/output/*.mp4 ~/comfy-ui/output/video/*.mp4 2>/dev/null | head -10
|
||||
```
|
||||
|
||||
For metadata-based identification (when outputs have predictable prefixes):
|
||||
```bash
|
||||
python3 << 'PYEOF'
|
||||
import subprocess, os, re
|
||||
@@ -487,6 +657,8 @@ All stock materials (start frames, audio, character refs, workflows, outputs) ar
|
||||
- FFHQ — 70K faces at 1024×1024 (NVIDIA, non-commercial)
|
||||
- CivitAI — community LoRAs and LTX workflows
|
||||
|
||||
**Using stock as I2V start frames:** See `references/stock-as-i2v-start-frame.md` — pattern for using character stock images as direct I2V start frames without IC-LoRA Ingredients. Simpler, works with existing 6-fix baseline. For multi-clip character consistency, use IC-LoRA Ingredients instead.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **CRITICAL: True-extend (extract last frame → use as next clip start) carries latent memory.** LTX remembers subjects from prior clips even when the start frame shows empty space. Confirmed 2026-07-22: Clip 2 hallucinated a second woman stepping up from the couch where the man was in Clip 1. Fix: use the pre-rendered empty mid-room frame B as Clip 2's start frame instead of the extracted last frame. Clean start, no latent memory. See §2-Clip Test Pattern.
|
||||
@@ -499,7 +671,9 @@ All stock materials (start frames, audio, character refs, workflows, outputs) ar
|
||||
- **`zhuanchang` trigger word is required** for Transition LoRA to activate. Append to end of segment prompts.
|
||||
- **CFG must be 1.0** for distilled model. Higher values cause artifacts.
|
||||
- **Queue is sequential** — submitting 6 scenes means ~30-40 min total. Use the blocking wait loop from step 5.
|
||||
- **`process(action='wait')` timeout is clamped to 60s regardless of what you request.** The configured limit on the Hermes host caps wait timeouts at 60 seconds. For renders that take 5-7 minutes each, a single `process(action='wait', timeout=900)` will time out after 60s with the process still running. Pattern: use `process(action='wait', timeout=60)` in a loop, checking `status` after each call. When status is still `"running"`, call wait again. When status is `"exited"`, the queue has drained. Confirmed 2026-07-22: 2-clip render took 3 wait cycles (~3 min total) despite requesting 900s.
|
||||
- **Output files accumulate** — the Director workflow produces 2 outputs per scene (guide pass + main pass). Identify the final output by checking for `LoraLoaderModelOnly` in metadata (Transition batch) or by resolution (512×512 = fp8 batch).
|
||||
- **Output directory varies by workflow prefix.** The `SaveImage`/`SaveVideo` node's `filename_prefix` determines where files land. Workflows with prefix `LTX_Director_` go to `~/comfy-ui/output/video/`. Workflows with custom prefixes (e.g., `LTX_2clip_test_C1`) go to `~/comfy-ui/output/` directly. When identifying new outputs, search both directories: `find ~/comfy-ui/output -name "*.mp4" -newer <reference_file> -type f`. Do NOT assume all outputs are in `video/`.\n- **CRITICAL: Template file naming — verify you're using the right base files before re-rendering.** The .202 `/tmp/` directory accumulates multiple generations of workflow JSONs with confusingly similar names: `api_2clip_C1.json` (old living room test), `api_cyberpunk_C1.json` (cyberpunk test), `api_2clip_01.json` (earlier living room variant). When modifying and re-rendering, ALWAYS inspect the timeline_data segment's `imageFile` and `prompt` fields to confirm you're working with the correct template. A one-line Python check: `python3 -c "import json; d=json.load(open('/tmp/api_XXX.json')); td=json.loads(d['prompt']['131']['inputs']['timeline_data']); print(td['segments'][0]['imageFile'], td['segments'][0]['prompt'][:80])"`. Confirmed 2026-07-22: pointed delegation at wrong template, produced living room output instead of cyberpunk.
|
||||
- **cifs kernel module not available** on Hermes host — use smbclient for TrueNAS, not mount.cifs.
|
||||
- **smbclient is NOT installed on .202** — scp files to Hermes host first, then upload to TrueNAS from there. Do not try to install smbclient on .202 (no root access).
|
||||
- **State file is the crash-recovery artifact** — update `~/workspace/general/ltx-pipeline-state.md` as you go. A new session reads it to resume.
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# 2-Clip Test Pattern — Panorama + True-Extend
|
||||
|
||||
Proven 2026-07-22. Validates model chain, prompt quality, and transition smoothness at minimal cost before committing to a full 6-scene story.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Generate one wide Flux panorama** (1536×512) with both subjects in a shared scene. Use Flux.1-dev fp8 on .202. Prompt: describe the full room with subject 1 on left, subject 2 on right, open space between them. This guarantees identical background/lighting/perspective.
|
||||
|
||||
2. **Crop 3 windows** (768×512 each) with ffmpeg:
|
||||
- A = left (subject 1 in frame, subject 2 off-screen right)
|
||||
- B = center (mid-room transition point)
|
||||
- C = right (subject 2 in frame, subject 1 off-screen left)
|
||||
Upload to `~/comfy-ui/input/`.
|
||||
|
||||
3. **Clip 1 workflow:** Copy v5 fix template (`/tmp/api_s1_fix_01.json`). Drop ID LoRA (rewire 131.model from ["201",0] to ["200",0], delete node 201). Set imageFile to A, prompt drives slow steady rightward pan, 10s/240 frames. Submit.
|
||||
|
||||
4. **Extract last frame:** `ffmpeg -sseof -3 -i clip1.mp4 -vsync vfr -q:v 2 ~/comfy-ui/input/last_frame.png`
|
||||
|
||||
5. **Clip 2 workflow:** Same template. imageFile = extracted last frame. Prompt continues the pan, reveals subject 2. Frame C is compositional target, not hard landing point. Submit.
|
||||
|
||||
6. **Concat:** `ffmpeg -f concat -safe 0 -i list.txt -c copy output_20s.mp4`
|
||||
|
||||
## Why True-Extend, Not FLF
|
||||
|
||||
FLF decelerates toward its target keyframe — clip 1 slows down approaching B, clip 2 speeds up leaving it. Creates a velocity "hitch" at the seam. True-extend avoids this: constant velocity, no deceleration. LTX's Extend mode is purpose-built for seamless continuation.
|
||||
|
||||
Sources: WaveSpeedAI (2026), LTX blog "How to Extend AI Videos" (2026).
|
||||
|
||||
## Model Config
|
||||
|
||||
- UNET: fp8 distilled transformer
|
||||
- LoRA: Distilled only @ 0.7 (NO ID LoRA — talking-head domain, wrong for camera moves)
|
||||
- Two-stage: 8 steps denoise 1.0 + 4 steps denoise 0.42
|
||||
- Sampler: euler, Scheduler: simple, CFG: 1.0, guide_strength: 0.5
|
||||
- Resolution: 768×512, 24fps, 10s per clip
|
||||
- Audio: disabled
|
||||
|
||||
## Validation
|
||||
|
||||
- Seam: no flash/pop/jump at boundary; pan speed continuous
|
||||
- Pan: steady rightward throughout, no reversal/wobble/stall
|
||||
- Background: same room/lighting across both clips
|
||||
- Subjects: coherent, no duplication or morphing
|
||||
- Artifacts: no limb melting, flicker, ghosting, warping
|
||||
- Timing: ~10s each at 24fps (~240 frames)
|
||||
|
||||
## Execution Confirmation (2026-07-22)
|
||||
|
||||
Successfully executed end-to-end on .202:
|
||||
- Flux panorama: 1536×512, 948KB, rendered in ~30s
|
||||
- Crops: A (842K), B (858K), C (845K) via ffmpeg crop filter
|
||||
- Clip 1: 241 frames, 10.04s, 1.0MB — status success
|
||||
- Last frame extracted: 611K PNG
|
||||
- Clip 2: 241 frames, 10.04s, 1.6MB — status success
|
||||
- Concat: 482 frames, 20.08s, 2.5MB — minor non-monotonic DTS warning (harmless)
|
||||
- Both clips used Distilled-only (no ID LoRA), two-stage, 768×512, 24fps
|
||||
|
||||
## TrueNAS Upload (MANDATORY)
|
||||
|
||||
**User can only view videos on TrueNAS, not in LXC containers.** After every render, upload outputs to TrueNAS immediately:
|
||||
```bash
|
||||
# scp from .202 to Hermes host first (smbclient not on .202)
|
||||
scp [email protected]:~/comfy-ui/output/<file>.mp4 /tmp/
|
||||
# Then upload to TrueNAS
|
||||
smbclient -N //10.0.0.117/proxmoxBackup -c 'cd ai_vid_stock_material\\outputs; put /tmp/<file>.mp4 <file>.mp4'
|
||||
```
|
||||
Do NOT skip this step — the user cannot review videos any other way.
|
||||
|
||||
## Fallback
|
||||
|
||||
If true-extend produces visible jump: use FLF A→B / B→C with pre-rendered keyframe B. Pixel-exact seam but may have subtle velocity change.
|
||||
@@ -0,0 +1,126 @@
|
||||
# ComfyUI API Submission & TrueNAS Upload Pattern
|
||||
|
||||
## Problem
|
||||
Submitting complex workflow JSONs via shell heredocs often fails due to quote escaping issues. Additionally, smbclient is NOT installed on the ComfyUI LXC (10.0.0.202), so TrueNAS uploads must be done from the Hermes host.
|
||||
|
||||
## Solution: File-Based API Submission
|
||||
|
||||
### Step 1: Write Workflow to File
|
||||
```bash
|
||||
cat > /tmp/submit_ltx_render.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
curl -s -X POST http://10.0.0.202:8188/prompt \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"prompt": {
|
||||
"35": {"class_type": "UNETLoader", "inputs": {"unet_name": "ltx-2.3-22b-distilled_transformer_only_fp8_scaled.safetensors", "weight_dtype": "default"}},
|
||||
"131": {"class_type": "LTXDirector", "inputs": {
|
||||
"model": ["200", 0],
|
||||
"clip": ["12", 0],
|
||||
"audio_vae": ["8", 0],
|
||||
"start_second": 0,
|
||||
"end_second": 5.0,
|
||||
"duration_seconds": 5.0,
|
||||
"start_frame": 0,
|
||||
"end_frame": 120,
|
||||
"duration_frames": 120,
|
||||
"timeline_data": "{\"mainTrackEnabled\": true, \"audioTrackEnabled\": false, \"motionTrackEnabled\": false, \"global_prompt\": \"<PROMPT>\", \"segments\": [{\"type\": \"image\", \"start\": 0.0, \"length\": 120, \"prompt\": \"<DETAILED_PROMPT>\", \"imageFile\": \"<START_FRAME>\", \"guide_strength\": 1.0, \"audioFile\": \"\", \"end\": 5.0}]}",
|
||||
"guide_strength": 1.0
|
||||
}},
|
||||
"200": {"class_type": "LTX2LoraLoaderAdvanced", "inputs": {"lora_name": "ltx-2.3-22b-distilled-lora-1.1_fro90_ceil72_condsafe.safetensors", "model": ["35", 0], "strength_model": 1.0, "video": 1.0, "video_to_audio": 0.0, "audio": 0.0, "audio_to_video": 0.0, "other": 0.0}}
|
||||
// ... rest of workflow nodes
|
||||
}
|
||||
}'
|
||||
EOF
|
||||
chmod +x /tmp/submit_ltx_render.sh
|
||||
```
|
||||
|
||||
### Step 2: Submit and Capture Prompt ID
|
||||
```bash
|
||||
RESULT=$(bash /tmp/submit_ltx_render.sh)
|
||||
PROMPT_ID=$(echo "$RESULT" | python3 -c "import json,sys; print(json.load(sys.stdin)['prompt_id'])")
|
||||
echo "Submitted: $PROMPT_ID"
|
||||
```
|
||||
|
||||
### Step 3: Poll for Completion
|
||||
```bash
|
||||
while true; do
|
||||
STATUS=$(curl -s "http://10.0.0.202:8188/history/$PROMPT_ID")
|
||||
echo "$STATUS" | grep -q '"status_str":"success"' && echo "COMPLETE" && break
|
||||
echo "$STATUS" | grep -q '"status_str":"error"' && echo "ERROR" && echo "$STATUS" && break
|
||||
echo "Running..."
|
||||
sleep 10
|
||||
done
|
||||
```
|
||||
|
||||
### Step 4: Verify Output
|
||||
```bash
|
||||
# Check duration
|
||||
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 ~/comfy-ui/output/<filename>.mp4
|
||||
|
||||
# Check frames
|
||||
ffprobe -v error -select_streams v:0 -show_entries stream=nb_frames -of default=noprint_wrappers=1:nokey=1 ~/comfy-ui/output/<filename>.mp4
|
||||
|
||||
# Check resolution
|
||||
ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of default=noprint_wrappers=1:nokey=1 ~/comfy-ui/output/<filename>.mp4
|
||||
```
|
||||
|
||||
## TrueNAS Upload (From Hermes Host)
|
||||
|
||||
### Prerequisites
|
||||
- smbclient is NOT on .202 (LXC)
|
||||
- Copy files to Hermes host first (10.0.0.42)
|
||||
- Upload from Hermes host using smbclient
|
||||
|
||||
### Upload Command
|
||||
```bash
|
||||
# Copy from LXC to Hermes host
|
||||
sshpass -p 'passw0rd' scp [email protected]:~/comfy-ui/output/<filename>.mp4 ~/comfy-ui/output/
|
||||
|
||||
# Upload to TrueNAS
|
||||
smbclient //10.0.0.117/proxmoxBackup -U n8n -c "cd ai_vid_stock_material\\\\outputs; put <local_filename>.mp4"
|
||||
```
|
||||
|
||||
### Alternative: Mount and Copy
|
||||
```bash
|
||||
# Mount TrueNAS share (if cifs kernel module available)
|
||||
sudo mount -t cifs //10.0.0.117/proxmoxBackup/ai_vid_stock_material/outputs /mnt/truenas_outputs -o user=n8n,vers=3.0
|
||||
|
||||
# Copy file
|
||||
cp <local_filename>.mp4 /mnt/truenas_outputs/
|
||||
|
||||
# Unmount
|
||||
sudo umount /mnt/truenas_outputs
|
||||
```
|
||||
|
||||
## Verification Checklist
|
||||
- [ ] Prompt ID captured from API response
|
||||
- [ ] Queue status shows `queue_running: []` and `queue_pending: []`
|
||||
- [ ] History shows `"status_str":"success"`
|
||||
- [ ] ffprobe confirms expected duration (5s = 121 frames at 24fps, 10s = 241 frames)
|
||||
- [ ] ffprobe confirms resolution (768×512 for 6-fix baseline)
|
||||
- [ ] File uploaded to TrueNAS at `//10.0.0.117/proxmoxBackup/ai_vid_stock_material/outputs/`
|
||||
- [ ] Filename follows naming convention: `LTX_YYYY-MM-DD_<what>_<sampler>_<duration>.mp4`
|
||||
|
||||
## Common Errors
|
||||
|
||||
### "no_prompt" Error
|
||||
- **Cause:** Submitted bare workflow JSON instead of `{\"prompt\": <workflow>}` envelope
|
||||
- **Fix:** Wrap workflow in `{"prompt": ...}` before sending to `/prompt` endpoint
|
||||
|
||||
### "tree connect failed: NT_STATUS_BAD_NETWORK_NAME"
|
||||
- **Cause:** Wrong SMB path or missing credentials
|
||||
- **Fix:** Use `//10.0.0.117/proxmoxBackup` (not the full path), then `cd` into subdirectory
|
||||
|
||||
### Quote Escaping Failures
|
||||
- **Cause:** Inline JSON with quotes breaks shell parsing
|
||||
- **Fix:** Write JSON to file first, then read with `-d @/path/to/file.json`
|
||||
|
||||
### VRAM Stuck at Idle
|
||||
- **Cause:** Job never started (error during parsing)
|
||||
- **Fix:** Check `curl -s http://localhost:8188/queue` for empty `queue_running`, then check `curl -s http://localhost:8188/history/<prompt_id>` for error messages
|
||||
|
||||
## Related Files
|
||||
- `/tmp/submit_ltx_render.sh` — Example submission script
|
||||
- `~/comfy-ui/output/` — ComfyUI output directory on .202
|
||||
- `//10.0.0.117/proxmoxBackup/ai_vid_stock_material/outputs/` — TrueNAS destination
|
||||
@@ -0,0 +1,34 @@
|
||||
# Artifact Deep Research — Question Template
|
||||
|
||||
Template for dispatching a focused deep-research pass when LTX-2.3 artifacts persist after applying the ranked fixes. Used 2026-07-22 for Story 1 "Ghost in the Wire."
|
||||
|
||||
## Structure
|
||||
|
||||
1. **OUR SETUP** — exact model chain, render settings, hardware
|
||||
2. **STORY PROMPTS** — all scene prompts verbatim
|
||||
3. **WHAT WE'VE ALREADY TRIED** — v4 and v5 settings + results
|
||||
4. **EXISTING RESEARCH** — path to prior artifact research
|
||||
5. **WHAT WE NEED NOW** — 10 specific research questions targeting our exact setup
|
||||
|
||||
## Dispatch Command
|
||||
|
||||
```bash
|
||||
hermes -p research -s deep-web-research chat -q "Read the full research question from /home/n8n/workspace/research/<date>-ltx-artifacts-question.md and execute it exhaustively..." -Q --max-turns 600 --yolo
|
||||
```
|
||||
|
||||
## Key Research Questions (from 2026-07-22 dispatch)
|
||||
|
||||
1. Given our EXACT prompts, what specific artifacts would LTX-2.3 produce with these scene descriptions?
|
||||
2. Does ID LoRA (talkvid-3k, trained for talking-head) cause artifacts on full-body action scenes?
|
||||
3. Is the LTX Director refiner stage (4 steps, denoise 0.42) introducing artifacts?
|
||||
4. Is fp8_scaled the worst fp8 variant? Compare vs fp8_e4m3fn vs v1.1 for I2V with LoRAs.
|
||||
5. Does 512×512 square aspect produce more artifacts than widescreen?
|
||||
6. Is Distilled LoRA at 0.7 still too high when stacked with ID LoRA?
|
||||
7. Are there NEW fixes or LoRAs released in the last 30 days?
|
||||
8. Could Flux.1-dev start frame inconsistencies cause artifacts?
|
||||
9. Are there known LTX Director node bugs beyond the wiring bug we already fixed?
|
||||
10. Does LTX-2.3 struggle with cyberpunk/neon-heavy scenes specifically?
|
||||
|
||||
## Expected Output
|
||||
|
||||
`/home/n8n/workspace/research/results/<YYYY-MM-DD>-ltx-2.3-artifacts-deep.md` with YAML frontmatter and findings ranked by likelihood of causing OUR specific artifacts.
|
||||
@@ -0,0 +1,46 @@
|
||||
# LTX-2.3 Artifact Root Causes — 6-Agent Consensus (2026-07-21)
|
||||
|
||||
Full research result from 6 independent agents diagnosing Story 1 "Ghost in the Wire" artifacts.
|
||||
|
||||
## Methodology
|
||||
|
||||
6 agents reviewed the exact technical setup independently:
|
||||
- Claude Opus 4.8 (SSH print mode, 9 turns)
|
||||
- Kimi K2.7 Code (kimi-c profile, 30 turns)
|
||||
- Kimi K2.6 (kimi profile, 30 turns)
|
||||
- MiniMax M3 (minimax profile, 30 turns)
|
||||
- GLM-5.2 (glm profile, 30 turns)
|
||||
- Deep research (research profile, 600 turns, 18 sources including Reddit, GitHub, HuggingFace)
|
||||
|
||||
## Consensus Findings
|
||||
|
||||
### CRITICAL (6/6 agents agree)
|
||||
1. **LoRA stacking at 1.0+1.0** — Distilled + ID LoRA both at full strength cause interference on faces/hands. Fix: Distilled → 0.7, ID → 0.6 (or drop ID if no audio).
|
||||
|
||||
### HIGH (4-5/6 agents agree)
|
||||
2. **10s exceeds temporal coherence** — LTX-2.3 degrades after 5-6s. Fix: test at 5s (121 frames).
|
||||
3. **linear_quadratic scheduler** — Distilled model is fragile with non-standard schedules. Fix: switch to `simple`.
|
||||
4. **Inconsistent guide_strength** — 0.5 on scene 1, 0.7 on others breaks flow. Fix: 0.5 uniform.
|
||||
|
||||
### MEDIUM (2-3/6 agents agree)
|
||||
5. **Audio track enabled with no audio** — AudioVAE NaN risk. Fix: disable audioTrackEnabled.
|
||||
6. **Text in prompts** — LTX can't render readable text. Fix: replace with abstract descriptions.
|
||||
7. **512×512 square aspect** — LTX prefers widescreen. Fix: 768×512 (requires new frames).
|
||||
8. **fp8_scaled v1.0 quality** — Kijai's v1.1 is better. Fix: model upgrade.
|
||||
|
||||
## Deep Research Sources (18 total)
|
||||
|
||||
Key community sources:
|
||||
- GitHub #148: End-of-video logo/watermark artifacts (upscaler v1.0 contamination)
|
||||
- HF Discussion #13: Upscaler sigma fix table by frame count
|
||||
- Reddit r/StableDiffusion: Skin compression fix (Nearest Exact interpolation)
|
||||
- Reddit r/StableDiffusion: Deformed bodies / identity drift in I2V
|
||||
- Reddit r/comfyui: Official workflow vs ComfyUI built-in (prompt ignoring bug)
|
||||
- Reddit r/StableDiffusion: RL LoRA for coherence (OmniNFT)
|
||||
- HuggingFace RuneXX: Dev model sampler comparison
|
||||
- HuggingFace LiconStudio: MSR V2 LoRA for identity preservation
|
||||
- GitHub #255: Image conditioning does not preserve identity
|
||||
- GitHub #244: Noise in distilled 1.1 model
|
||||
- LTX.io blog: Official artifact reduction guide
|
||||
|
||||
Full research: `~/workspace/research/results/2026-07-21-ltx-2.3-artifacts.md`
|
||||
@@ -0,0 +1,24 @@
|
||||
# Canonical Character Descriptions
|
||||
|
||||
Ground truth for all LTX prompts and Flux panorama generation. Do NOT improvise or use generic descriptions — these are the exact descriptions the user provided.
|
||||
|
||||
## Boss
|
||||
|
||||
Black suit, muscular build, shaved head, goatee. Stern expression, chrome temple implant. Stock reference: `SHEETS2_00005_Boss.png` on TrueNAS.
|
||||
|
||||
**Prompt fragment:** "A muscular boss with shaved head and goatee wearing a black suit sits at a terminal in a neon-lit cyberpunk room. His chrome temple implant glints in the neon glow. Stern expression."
|
||||
|
||||
## Woman
|
||||
|
||||
Mid-20s, short dark hair, chrome temple implants, cybernetic fingers, black synth-leather jacket. Stock references: `cyberpunk_woman_neon_01.jpg`, `cyberpunk_woman_neon_02.jpg` on TrueNAS.
|
||||
|
||||
**Prompt fragment:** "A woman in her mid-20s with short dark hair, chrome temple implants, and cybernetic fingers wears a black synth-leather jacket. She stands by a window gazing out at the neon cityscape."
|
||||
|
||||
## Pitfall: Generic Descriptions
|
||||
|
||||
Using generic descriptions like "stern boss" or "woman by window" instead of the canonical descriptions causes:
|
||||
- Face distortion (model doesn't know what "stern boss" looks like)
|
||||
- Character morphing (model blends between prompt and start frame)
|
||||
- Wrong character entirely (model substitutes its own training data)
|
||||
|
||||
Always use the exact canonical descriptions above in every prompt that references these characters.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Claude Live SSH Inspection — Plan-Building Pattern
|
||||
|
||||
When Claude needs to build a plan for a target machine it can SSH into, have it inspect the live state FIRST before writing the plan. This eliminates guesswork about what's installed, what models are on disk, what workflows exist, and what the queue looks like.
|
||||
|
||||
## Pattern
|
||||
|
||||
1. **Write the question** — include the goal, what to inspect, and the deliverable path
|
||||
2. **scp to 10.0.0.28** — standard ask-claude flow
|
||||
3. **Claude SSHs into the target** — inspects files, checks queue, reads workflow JSONs, verifies model paths
|
||||
4. **Claude writes the plan** — saves to its own host (10.0.0.28); scp back to Hermes host after
|
||||
|
||||
## When to Use
|
||||
|
||||
- Building a plan that depends on specific files/models/workflows on a target machine
|
||||
- The target's state is complex enough that describing it in the prompt would be error-prone
|
||||
- You want Claude to verify claims against reality (e.g., "is this LoRA actually on disk?")
|
||||
|
||||
## Example (2026-07-22)
|
||||
|
||||
Goal: 2-clip LTX-2.3 test plan. Target: 10.0.0.202 (ComfyUI).
|
||||
|
||||
Claude was told to SSH into .202 and inspect:
|
||||
- LoRAs on disk: `ls ~/comfy-ui/models/loras/`
|
||||
- Workflow JSONs: `ls /tmp/api_s1_*.json`
|
||||
- Diffusion models: `ls ~/comfy-ui/models/diffusion_models/`
|
||||
- Start frames: `ls ~/comfy-ui/input/ltx_start_frame*`
|
||||
- Queue status: `curl -s http://localhost:8188/queue`
|
||||
|
||||
Key discoveries from live inspection (none were in any state file):
|
||||
- Distilled LoRA on disk is the 384 variant, not 384-1.1
|
||||
- ID LoRA (talkvid-3k) is trained for talking-head, wrong for action scenes
|
||||
- FLF example workflows exist on the box
|
||||
- Existing start frames are all cyberpunk single-subject — can't be reused
|
||||
- Queue empty, clear to run
|
||||
|
||||
Plan produced: FLF panorama approach — one wide Flux panorama, 3 cropped windows (A/B/C), Clip 1 FLF A→B, Clip 2 FLF B→C, shared frame B = seamless pan.
|
||||
|
||||
## Error Recovery
|
||||
|
||||
Claude may error on first attempt (SSH timeout, tool failure, wrong command format). **Resume, don't start fresh:**
|
||||
|
||||
| Attempt | Turns | Cost | Result |
|
||||
|---------|-------|------|--------|
|
||||
| Fresh session | 21 | $1.05 | is_error: true, empty result |
|
||||
| Resume (--resume) | 2 | $0.11 | Success — self-diagnosed, recovered, produced plan |
|
||||
|
||||
**Pattern:** When Claude returns `is_error: true` with empty result, write a short follow-up asking what went wrong, scp it, and resume with `--resume <session_id>`. Claude self-diagnoses and recovers. Do NOT start a fresh session — you lose the context and pay the full cost again.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Claude cannot write files to the Hermes host directly. It writes to 10.0.0.28 and you scp back, or it outputs inline for you to save.
|
||||
- SSH credentials must be in the question (Claude doesn't have them in its environment). Use the exact command: `sshpass -p 'passw0rd' ssh [email protected]`
|
||||
- Keep the inspection list focused — 5-6 specific checks, not a full system audit.
|
||||
- **Live-SSH inspection can fail silently.** Claude may error out with `is_error: true` and empty result after 20+ turns. Resume with `--resume` — Claude self-diagnoses. Do not start fresh.
|
||||
- **Don't send follow-up instructions mid-build.** If the user says "tell Claude X once he completes," wait for the current turn to finish. Sending mid-execution wastes the instruction (Claude is busy) and frustrates the user.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Claude Wiring Diagnosis — 2026-07-21
|
||||
|
||||
Claude Opus 4.8 SSH'd into 10.0.0.202 and diagnosed the subject duplication bug.
|
||||
|
||||
## Root Cause
|
||||
|
||||
Node 132 (LTXDirectorGuide, the refiner pass) had its `latent` input wired to `["34", 0]` (uncropped LTXVSeparateAVLatent output) instead of `["55", 2]` (LTXDirectorCropGuides cropped output).
|
||||
|
||||
## Mechanism (from source code review)
|
||||
|
||||
`LTXDirectorGuide` doesn't blend the I2V image in — it **appends** it as extra latent frames on the end of the sequence, then records how many to remove later in a conditioning key (`ltx_director_guide.py:596`). `LTXDirectorCropGuides` trims exactly that many afterward.
|
||||
|
||||
The critical line is `ltx_director_guide.py:339`:
|
||||
```python
|
||||
initial_latent_length = int(latent_length) # captured from whatever arrives
|
||||
```
|
||||
|
||||
The graph runs 133 first (8 steps, denoise 1.0), then 132 (4 steps, 0.42). Node 55 correctly crops 133's guide frames off... but `55:2` (the cropped latent) is never consumed. Node 132 takes `34:0` instead — the raw, uncropped sampler output. Only 55's *conditioning* outputs are wired, which is why the break is invisible in the UI.
|
||||
|
||||
So 132 counts stage-1's leftover guide frames as **real video content**, appends a second guide on top, and the final crop removes only one set. The orphaned guide frames decode as actual footage: a second copy of the man, frozen in the start-frame pose, while the real generation gets out of bed.
|
||||
|
||||
## Empirical Confirmation
|
||||
|
||||
- **ffprobe: 249 frames / 10.375s. Requested: 240 / 10.0s.** The overshoot is leftover guide latent, decoded as video.
|
||||
- **Batches 1, 2, and 3 are all 249 frames** (`00007`, `00013`, `00019`, `00025`–`00030`). Byte-identical symptom across every "fix" — the wiring never changed.
|
||||
- **Log shows two `[LTXDirectorGuide] execute started` per render, both "Using Appended Keyframe Guidance"** — double append, confirmed.
|
||||
|
||||
## Reference Workflow Comparison
|
||||
|
||||
`example_workflows/LTX_Director_2_Workflow_Hotfix.json` wires it as:
|
||||
```
|
||||
stage1 → 34 Separate → 55 CropGuides → 14 LTXVLatentUpsampler → 132 Guide
|
||||
(55:2, cropped)
|
||||
```
|
||||
|
||||
Our v3 deleted node 14 (`LTXVLatentUpsampler`) and reconnected 132 straight to `34:0`. Removing the upsampler is what orphaned the crop.
|
||||
|
||||
## Fix
|
||||
|
||||
Point `132.inputs.latent` at `["55", 2]` instead of `["34", 0]`. Restoring the upsampler between them is the fully-correct form.
|
||||
|
||||
## Verification
|
||||
|
||||
Output should be exactly **240 frames** for 10s@24fps. 249 frames = bug present.
|
||||
|
||||
## Other Findings
|
||||
|
||||
- **ID LoRA / distilled LoRA at 1.0** — not the cause. Log confirms `is_lora_active: False, ic_lora_name: None`.
|
||||
- **Scene 2 "soup" problem** — separate issue. The start frame `ltx_start_frame_02_00001_.png` shows a man over a tall steaming stockpot, which reads as soup. Bad start frame, not a prompt bug.
|
||||
- **Seed is 0 and shared** — bump to random. ComfyUI caches identical graphs.
|
||||
- **Session ID:** 5381500e-a071-42f5-8d6b-d780520a981b
|
||||
- **Cost:** $1.34 USD, 31 turns
|
||||
@@ -0,0 +1,108 @@
|
||||
# IC-LoRA Ingredients — Reference Sheet Control for LTX 2.3
|
||||
|
||||
**Model:** Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients
|
||||
**File:** `ltx-2.3-22b-ic-lora-ingredients-0.9.safetensors` (1.31 GB)
|
||||
**HF URL:** https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients
|
||||
**Status:** Downloaded on .202 (2026-07-22, 1.31 GB) — gated, requires "Agree and Access" on HF website + valid token
|
||||
**Token:** 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 (user: MDKRUSH, 2026-07-22)
|
||||
**Trained bucket:** 768×448, 121 frames, 24 fps
|
||||
|
||||
## fp8 Compatibility Gap (UNRESOLVED — deep research in progress 2026-07-22)
|
||||
|
||||
The official IC-LoRA workflow (`LTX-2.3_ICLoRA_Ingredients_Single_Stage_Distilled.json`) uses a completely different architecture from our pipeline:
|
||||
|
||||
| Component | Official IC-LoRA Workflow | Our Pipeline |
|
||||
|-----------|--------------------------|--------------|
|
||||
| Model loader | CheckpointLoaderSimple (full 46GB checkpoint) | UNETLoader (fp8 distilled, 23GB) |
|
||||
| Sampler | SamplerCustomAdvanced + ManualSigmas + CFGGuider | LTXDirector (single-stage) |
|
||||
| Text encoder | GemmaAPITextEncode | LTXConditioning |
|
||||
| Total nodes | 38 | 22 |
|
||||
|
||||
**Key question:** Can `LTXICLoRALoaderModelOnly` accept a model from UNETLoader (fp8 distilled transformer) instead of CheckpointLoaderSimple? The node takes a "model" input — compatibility with fp8 UNET-only model is unverified. Deep research dispatched 2026-07-22 to answer this and 11 other compatibility questions. Results pending at `/home/n8n/workspace/research/results/2026-07-22-ic-lora-fp8-compatibility.md`.
|
||||
|
||||
## What It Does
|
||||
|
||||
Conditions video generation on a **reference sheet** — a single composite image inventorying characters, props, and location. The model reads the reference latents in-context and renders a new clip whose characters, props, and setting match the sheet. This is the solution for using stock character sheets (like `SHEETS2_00005_Boss.png` on TrueNAS) as character consistency anchors.
|
||||
|
||||
## Recommended Settings (from official README)
|
||||
|
||||
| Setting | Value | Notes |
|
||||
|---------|-------|-------|
|
||||
| LoRA strength | **1.4** | NOT 1.0 — official recommendation |
|
||||
| Inference steps | **30** | Higher than our standard 18 |
|
||||
| Guidance scale | **4.0** | NOT 1.0 — video-to-video mode |
|
||||
| Resolution | 768×448 | Trained bucket — best results here |
|
||||
| Frames | 121 | 24 fps |
|
||||
| Negative prompt | `worst quality, inconsistent motion, blurry, jittery, distorted` | |
|
||||
| STG | mode `stg_v`, block 29, scale 1.0 | Helps motion stability |
|
||||
|
||||
## Prompt Format
|
||||
|
||||
Two-part structure (matching training):
|
||||
|
||||
```
|
||||
Reference sheet: <description of the panels in the sheet — characters, props, location>
|
||||
Generated video: <description of the action / shot you want generated>
|
||||
```
|
||||
|
||||
The `Reference sheet:` text describes what's in the panels. The `Generated video:` text drives the action. The model reads the reference latents for "what things look like" and the prompt for "what happens."
|
||||
|
||||
## Control Signal Requirements
|
||||
|
||||
- Reference sheet: single composite image with one clean panel per visual element
|
||||
- Each character: face close-up + body turnaround
|
||||
- Each prop: product-style render
|
||||
- One clean location panel
|
||||
- Laid out on black background with NO text
|
||||
- **Bigger panels carry over better** — give important elements larger panels
|
||||
- Reference must be looped into a static video ≥ 121 frames at output resolution
|
||||
|
||||
## Node Chain (ComfyUI)
|
||||
|
||||
```
|
||||
CheckpointLoaderSimple (full 46GB checkpoint — adapt to UNETLoader fp8 for 24GB)
|
||||
→ LTXICLoRALoaderModelOnly (ingredients-0.9.safetensors, strength 1.4)
|
||||
→ LTXAddVideoICLoRAGuide (reference sheet as static video input)
|
||||
→ LTXVCropGuides → SamplerCustomAdvanced
|
||||
```
|
||||
|
||||
## Nodes on .202
|
||||
|
||||
- `LTXICLoRALoaderModelOnly` — in `iclora.py`, loads IC-LoRA and extracts `latent_downscale_factor`
|
||||
- `LTXAddVideoICLoRAGuide` — in `iclora.py`, applies reference video conditioning
|
||||
- `LTXAddVideoICLoRAGuideAdvanced` — extended version with `attention_strength` and `attention_mask`
|
||||
|
||||
## Example Workflow
|
||||
|
||||
On .202: `~/comfy-ui/custom_nodes/ComfyUI-LTXVideo/example_workflows/2.3/LTX-2.3_ICLoRA_Ingredients_Single_Stage_Distilled.json`
|
||||
|
||||
Uses 39 nodes including: CheckpointLoaderSimple, LTXICLoRALoaderModelOnly, LTXAddVideoICLoRAGuide, GemmaAPITextEncode, LTXVCropGuides, SamplerCustomAdvanced, ManualSigmas.
|
||||
|
||||
## Download Process
|
||||
|
||||
1. Visit https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients
|
||||
2. Click "Agree and Access" (gated model)
|
||||
3. Download with HF token:
|
||||
```bash
|
||||
HF_TOKEN='hf_...' huggingface-cli download Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients \
|
||||
ltx-2.3-22b-ic-lora-ingredients-0.9.safetensors \
|
||||
--local-dir ~/comfy-ui/models/loras/
|
||||
```
|
||||
4. Place in `~/comfy-ui/models/loras/` on .202
|
||||
|
||||
## Stock Integration
|
||||
|
||||
The boss stock on TrueNAS (`ai_vid_stock_material/character_refs/SHEETS2_00005_Boss.png`) is a character sheet — exactly the input format IC-LoRA Ingredients expects. To use it:
|
||||
|
||||
1. Download the character sheet from TrueNAS
|
||||
2. Create a reference sheet image with the boss character panel(s)
|
||||
3. Loop into a 121-frame static video at 768×448
|
||||
4. Feed as the reference/control input to LTXAddVideoICLoRAGuide
|
||||
|
||||
## Tips (from README)
|
||||
|
||||
- **Bigger panels carry over better** — give important characters/props larger panels
|
||||
- **Identity drift fix:** ensure clean front-facing close-up + full turnaround for each character
|
||||
- **Element not appearing:** add a dedicated panel for any prop/character that needs to persist
|
||||
- **Reference too short:** static video must be ≥ 121 frames
|
||||
- **Element-driven reference-sheet generator** exists for authoring sheets (referenced in README)
|
||||
@@ -0,0 +1,118 @@
|
||||
# Modify-and-Re-Render Recipe
|
||||
|
||||
Proven pattern for changing settings on existing workflow JSONs and re-rendering without rebuilding from scratch. Used 2026-07-22 for the sampler+duration 2-clip test.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Changing sampler, steps, scheduler, duration, or other render settings
|
||||
- Testing a new LoRA or model chain variant on existing scenes
|
||||
- Re-running a failed batch with corrected settings
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Copy existing workflow JSONs** to `_v2` variants (preserve originals):
|
||||
```bash
|
||||
cp /tmp/api_2clip_C1.json /tmp/api_2clip_C1_v2.json
|
||||
cp /tmp/api_2clip_C2.json /tmp/api_2clip_C2_v2.json
|
||||
```
|
||||
|
||||
2. **Modify in-place with Python via scp** (SSH heredoc quoting is fragile — scp the script, then run it):
|
||||
|
||||
Write the modification script locally, scp it to .202, then execute:
|
||||
```bash
|
||||
# Write script locally
|
||||
cat > /tmp/modify_v3.py << 'PYEOF'
|
||||
import json
|
||||
for fname in ["/tmp/api_boss_C1_v3.json", "/tmp/api_boss_C2_v3.json"]:
|
||||
with open(fname) as f:
|
||||
d = json.load(f)
|
||||
p = d["prompt"]
|
||||
|
||||
# Sampler change (node 20 is KSamplerSelect, not KSampler)
|
||||
p["20"]["inputs"]["sampler_name"] = "euler"
|
||||
|
||||
# Duration change (node-level)
|
||||
p["131"]["inputs"]["duration_frames"] = 121
|
||||
p["131"]["inputs"]["end_frame"] = 121
|
||||
p["131"]["inputs"]["duration_seconds"] = 5.04
|
||||
|
||||
# Duration change (timeline_data JSON — CRITICAL, don't skip)
|
||||
td = json.loads(p["131"]["inputs"]["timeline_data"])
|
||||
td["normalDurationFrames"] = 121
|
||||
for seg in td["segments"]:
|
||||
seg["length"] = 121
|
||||
seg["end"] = 5.04
|
||||
p["131"]["inputs"]["timeline_data"] = json.dumps(td)
|
||||
|
||||
with open(fname, "w") as f:
|
||||
json.dump(d, f, indent=2)
|
||||
print(f"Updated {fname}: sampler={p['20']['inputs']['sampler_name']}")
|
||||
PYEOF
|
||||
|
||||
# scp to .202 and run
|
||||
sshpass -p 'passw0rd' scp /tmp/modify_v3.py [email protected]:/tmp/
|
||||
sshpass -p 'passw0rd' ssh [email protected] python3 /tmp/modify_v3.py
|
||||
```
|
||||
|
||||
**Why scp, not heredoc:** SSH heredoc quoting strips quotes from Python string-keyed dicts, causing `KeyError` on nodes like `"20"`. The scp approach preserves the script exactly as written. Confirmed 2026-07-22: heredoc failed twice (KeyError on node 20); scp worked first try.
|
||||
|
||||
3. **Verify all changes** before submitting:
|
||||
```python
|
||||
for fname in files:
|
||||
d = json.load(open(fname))
|
||||
p = d["prompt"]
|
||||
assert p["20"]["inputs"]["sampler_name"] == "euler_ancestral_cfg_pp"
|
||||
assert p["131"]["inputs"]["duration_frames"] == 121
|
||||
td = json.loads(p["131"]["inputs"]["timeline_data"])
|
||||
assert td["segments"][0]["length"] == 121
|
||||
```
|
||||
|
||||
4. **Submit to queue:**
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8188/prompt -H "Content-Type: application/json" -d @/tmp/api_2clip_C1_v2.json
|
||||
curl -s -X POST http://localhost:8188/prompt -H "Content-Type: application/json" -d @/tmp/api_2clip_C2_v2.json
|
||||
```
|
||||
|
||||
5. **Wait for queue drain** (blocking poll from Hermes host):
|
||||
```bash
|
||||
sshpass -p 'passw0rd' ssh [email protected] "while true; do q=\$(curl -s http://localhost:8188/queue | python3 -c \"import json,sys; d=json.load(sys.stdin); print(len(d['queue_running'])+len(d['queue_pending']))\"); if [ \"\$q\" -eq 0 ]; then echo 'QUEUE_EMPTY'; break; fi; echo \"queue: \$q\"; sleep 30; done"
|
||||
```
|
||||
|
||||
6. **Identify new outputs** (search both output directories):
|
||||
```bash
|
||||
find ~/comfy-ui/output -name "*.mp4" -newer /tmp/api_2clip_C1_v2.json -type f
|
||||
```
|
||||
|
||||
7. **Verify frame counts:**
|
||||
```bash
|
||||
ffprobe -v quiet -show_entries stream=nb_frames,codec_name,width,height,duration -of csv=p=0 <file>
|
||||
```
|
||||
|
||||
8. **Concat, scp, upload to TrueNAS** (standard pipeline steps 7-8).
|
||||
|
||||
## Key Pitfalls
|
||||
|
||||
- **sshpass is NOT on .202 — pull files from Hermes host instead of pushing from .202.** When transferring output files from .202 to the Hermes host, `sshpass` is not installed on .202. Use `sshpass -p 'passw0rd' scp n8n@10.0.0.202:<remote_path> <local_path>` from the Hermes host (pull), not `ssh ... 'sshpass scp ...'` from .202 (push). Confirmed 2026-07-22: push attempt failed with `sshpass: command not found`.
|
||||
- **CRITICAL: Verify you have the RIGHT template before modifying.** `/tmp/` accumulates multiple generations with confusingly similar names: `api_2clip_C1.json` (old living room), `api_cyberpunk_C1.json` (cyberpunk), `api_2clip_01.json` (earlier variant). ALWAYS inspect `imageFile` and `prompt` before modifying:
|
||||
```bash
|
||||
python3 -c "import json; d=json.load(open('/tmp/api_XXX.json')); td=json.loads(d['prompt']['131']['inputs']['timeline_data']); print(td['segments'][0]['imageFile'], td['segments'][0]['prompt'][:80])"
|
||||
```
|
||||
Confirmed 2026-07-22: pointed delegation at wrong template, produced living room output instead of cyberpunk.
|
||||
- **Must update BOTH node-level AND timeline_data JSON fields.** The LTXDirector reads `segment["length"]` from timeline_data for actual frame count. Node-level `duration_frames` alone won't change output duration.
|
||||
- **121 frames = 8×15+1** satisfies the 8n+1 requirement for LTX.
|
||||
- **5.04 seconds** is the correct `duration_seconds` for 121 frames at 24fps (121/24 ≈ 5.0417).
|
||||
- **Verify with ffprobe, not by file size.** File size varies with content complexity; frame count is the ground truth.
|
||||
- **Use ComfyUI history API to identify outputs reliably.** After queue drains, query each prompt_id:
|
||||
```bash
|
||||
curl -s http://localhost:8188/history/<prompt_id> | python3 -c "
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
entry=d.get('<prompt_id>',{})
|
||||
outputs=entry.get('outputs',{})
|
||||
node37=outputs.get('37',{})
|
||||
images=node37.get('images',[])
|
||||
for img in images:
|
||||
print(img.get('filename','?'))
|
||||
"
|
||||
```
|
||||
This works regardless of which output directory the workflow uses. Fallback: `find ~/comfy-ui/output -name "*.mp4" -newer <reference_file> -type f`.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Multi-Agent Parallel Review Pattern (2026-07-21)
|
||||
|
||||
Proven pattern for diagnosing complex technical issues: dispatch 5+ independent agents simultaneously with the same diagnostic brief, then consolidate findings into a ranked consensus.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Complex artifact/quality issues where no single agent has the full answer
|
||||
- User reports "a LOT of artifacts" or "all kinds of inconsistencies" but can't pinpoint the cause
|
||||
- Need both technical review (model settings, workflow) AND community research (GitHub issues, Reddit, social media)
|
||||
|
||||
## Pattern
|
||||
|
||||
### 1. Write a single diagnostic brief
|
||||
Save to `/tmp/<topic>-review.txt`. Include:
|
||||
- Exact technical setup (model, settings, prompts, frame counts)
|
||||
- What the user is seeing (symptoms)
|
||||
- What to investigate (specific questions)
|
||||
- Web search mandate: "Use mcp_searxng_searxng_web_search for every claim and cite the source URL"
|
||||
|
||||
### 2. Dispatch all peers in parallel
|
||||
Use `terminal(background=true, notify_on_complete=true)` for each:
|
||||
- Claude Opus (ask.sh on 10.0.0.28) — best for technical reasoning
|
||||
- Kimi K2.7 Code (kimi-c profile) — good for code-level analysis
|
||||
- Kimi K2.6 (kimi profile) — broad knowledge
|
||||
- MiniMax M3 (minimax profile) — alternative perspective
|
||||
- GLM-5.2 (glm profile) — alternative perspective
|
||||
- Deep research (research profile, 600 turns) — GitHub, Reddit, HuggingFace, social media
|
||||
|
||||
### 3. Consolidate findings
|
||||
When all complete, build a ranked table:
|
||||
- CRITICAL: all agents agree
|
||||
- HIGH: 4-5 agents agree
|
||||
- MEDIUM: 2-3 agents agree
|
||||
- UNIQUE: single agent, high-impact
|
||||
|
||||
### 4. Apply fixes in order
|
||||
Test after each fix. Start with zero-cost config changes before model downloads or frame regeneration.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **kimi-c may hit tool name issues** — the SearXNG MCP tool uses double underscores (`mcp__searxng__`) but the prompt says single. kimi-c may burn turns trying to resolve this. Accept partial results.
|
||||
- **MiniMax may produce truncated output** — the process log may only show reasoning, not the final answer. Check the full log with `process(action='log')`.
|
||||
- **Deep research can't access X/Twitter or Discord** — SearXNG returns empty for social media queries. Document as uncertainty, not failure.
|
||||
- **Don't wait for all before acting** — apply consensus fixes as soon as 3+ agents agree. The remaining peers add detail but shouldn't block progress.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Official LTX Sample Prompts
|
||||
|
||||
Extracted from ltx.io/blog/ltx-2-3-prompt-guide and ltx.io/blog/how-to-write-a-prompt (June 2026). These are the official examples — use as templates for structure and style.
|
||||
|
||||
## Example 1: News Broadcast (T2V, dialogue + camera)
|
||||
|
||||
EXT. SMALL TOWN STREET – MORNING – LIVE NEWS BROADCAST
|
||||
|
||||
The shot opens on a news reporter standing in front of a row of cordoned-off cars, yellow caution tape fluttering behind him. The light is warm, early sun reflecting off the camera lens. The faint hum of chatter and distant drilling fills the air. The reporter, composed but visibly excited, looks directly into the camera, microphone in hand.
|
||||
|
||||
Reporter (live): "Thank you, Sylvia. And yes — this is a sentence I never thought I'd say on live television — but this morning, here in the quiet town of New Castle, Vermont… black gold has been found!"
|
||||
|
||||
He gestures slightly toward the field behind him.
|
||||
|
||||
Reporter (grinning): "If my cameraman can pan over, you'll see what all the excitement's about."
|
||||
|
||||
The camera pans right, slowly revealing a construction site surrounded by workers in hard hats. A beat of silence — then, with a sudden roar, a geyser of oil erupts from the ground, blasting upward in a violent plume. Workers cheer and scramble, the black stream glistening in the morning light. The camera shakes slightly, trying to stay focused through the chaos.
|
||||
|
||||
Reporter (off-screen, shouting over the noise): "There it is, folks — the moment New Castle will never forget!"
|
||||
|
||||
The camera catches the sunlight gleaming off the oil mist before pulling back, revealing the entire scene — the small-town skyline silhouetted against the wild fountain of oil.
|
||||
|
||||
## Example 2: Frog Yoga (T2V, comedy, dialogue + timing)
|
||||
|
||||
The camera opens in a calm, sunlit frog yoga studio. Warm morning light washes over the wooden floor as incense smoke drifts lazily in the air. The senior frog instructor sits cross-legged at the center, eyes closed, voice deep and calm.
|
||||
|
||||
"We are one with the pond."
|
||||
All the frogs answer softly: "Ommm..."
|
||||
|
||||
"We are one with the mud."
|
||||
"Ommm..."
|
||||
|
||||
He smiles faintly. "We are one with the flies."
|
||||
|
||||
A pause. The camera pans to the side towards one frog who twitches, eyes darting. Suddenly its tongue snaps out, catching a fly mid-air and pulling it into its mouth.
|
||||
|
||||
The master exhales slowly, still serene. "But we do not chase the flies..." Beat. "not during class."
|
||||
|
||||
The guilty frog lowers its head in shame, folding its hands back into a meditative pose. The other frogs resume their chant: "Ommm..."
|
||||
|
||||
Camera holds for a moment on the embarrassed frog, eyes closed too tightly, pretending nothing happened.
|
||||
|
||||
## Example 3: Dialogue with Acting Directions (from prompt guide)
|
||||
|
||||
A middle-aged man with greying hair speaks in a sad, slow-paced voice, "I remember after you kids came along..." He pauses and looks to the side, then continues, "your mom..." His eyes widen momentarily. He finishes with a cracking voice, "said something to me I never quite understood." The camera slowly zooms into his face. The audio is crisp with faint room tone.
|
||||
|
||||
## Example 4: Structured Prompt (from "How to Write a Prompt")
|
||||
|
||||
**Subject:** The sun and a high-tech cityscape with glowing skyscrapers
|
||||
**Action:** The sun rising above the horizon, a second black sun appearing in the distance
|
||||
**Framing:** Wide shot capturing the full skyline
|
||||
**Lighting and Style:** Warm golden light, sci-fi cinematic register
|
||||
**Camera Motion:** Slow upward pan revealing more of the sky
|
||||
|
||||
**Final Prompt:** "Wide shot of the sun rising over a high-tech futuristic city with glowing skyscrapers. Warm, golden lighting. Camera slowly pans upward to reveal another sun — black and ominous in the distance."
|
||||
|
||||
## Example 5: Vague vs Structured (from "How to Write a Prompt")
|
||||
|
||||
| Vague | Structured |
|
||||
|-------|------------|
|
||||
| "A woman walking in a city at night." | "Medium shot, eye level, tracking from behind. A woman in a navy trench coat walks through a rain-soaked Tokyo side street at night, neon reflections in the puddles, shallow depth of field." |
|
||||
| "Show a sunrise in a city" | "Wide shot of the sun rising over a high-tech futuristic city with glowing skyscrapers. Warm golden lighting. Camera slowly pans upward to reveal a second black sun in the distance." |
|
||||
| "Make a video of someone walking" | "Medium shot tracking a woman in a red coat walking through a crowded Tokyo street at night, neon signs reflecting in puddles, handheld camera following from behind." |
|
||||
|
||||
## Key Patterns
|
||||
|
||||
- **Dialogue:** Break into short phrases with acting directions between each line
|
||||
- **Structure:** framing → subject → action → style → motion (shot-size-first) OR subject → action → framing → style → motion (subject-first)
|
||||
- **I2V:** Focus on motion and action — start frame already defines the visual
|
||||
- **Length:** Longer prompts consistently outperform short ones on 2.3
|
||||
- **Emotion:** Physical cues, never abstract labels
|
||||
- **Camera:** State "static" explicitly if no movement; state exact motion otherwise
|
||||
@@ -0,0 +1,65 @@
|
||||
# LTX Pipeline State Template
|
||||
|
||||
Copy this template at the start of any LTX pipeline session. Update as you go.
|
||||
|
||||
```markdown
|
||||
# LTX Video Pipeline — Session State
|
||||
|
||||
**Date:** YYYY-MM-DD | **Target:** 10.0.0.202 (RTX 4090 24GB)
|
||||
|
||||
## Current Model Chain
|
||||
|
||||
```
|
||||
UNETLoader (fp8 distilled)
|
||||
→ [LoraLoaderModelOnly (Transition) — only if using transitions]
|
||||
→ LTX2LoraLoaderAdvanced (Distilled, strength 1.0)
|
||||
→ LTX2LoraLoaderAdvanced (ID, strength 1.0)
|
||||
→ LTXDirector
|
||||
```
|
||||
|
||||
## Render Settings
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| Resolution | 512×512 |
|
||||
| FPS | 24 |
|
||||
| Duration | 10s per scene |
|
||||
| guide_strength | 0.7 |
|
||||
| CFG | 1.0 (4.0 for Transition scenes) |
|
||||
| Steps | 8 main / 4 guide |
|
||||
|
||||
## Scene Inventory
|
||||
|
||||
| # | Prompt Summary | Start Frame | Audio | Output File | Status |
|
||||
|---|---------------|-------------|-------|-------------|--------|
|
||||
| 1 | | | | | |
|
||||
| 2 | | | | | |
|
||||
| 3 | | | | | |
|
||||
| 4 | | | | | |
|
||||
| 5 | | | | | |
|
||||
| 6 | | | | | |
|
||||
|
||||
## Queue Status
|
||||
|
||||
- Running:
|
||||
- Pending:
|
||||
- GPU:
|
||||
|
||||
## Outputs
|
||||
|
||||
| File | Scene | Duration | Size |
|
||||
|------|-------|----------|------|
|
||||
| | | | |
|
||||
|
||||
## Issues Found
|
||||
|
||||
| Symptom | Cause | Fix Applied |
|
||||
|---------|-------|-------------|
|
||||
| | | |
|
||||
|
||||
## Next Steps
|
||||
|
||||
1.
|
||||
2.
|
||||
3.
|
||||
```
|
||||
@@ -0,0 +1,77 @@
|
||||
# Programmatic Workflow JSON Modification
|
||||
|
||||
When modifying multiple LTX workflow JSONs with the same changes (sampler, duration, imageFile, prompt), use a Python script rather than manual JSON editing. This avoids quoting errors in shell heredocs and ensures consistent modifications across clips.
|
||||
|
||||
## Pattern
|
||||
|
||||
1. Write the modification script locally (on Hermes host)
|
||||
2. scp it to .202
|
||||
3. Run it via SSH
|
||||
|
||||
## Example: build_clips.py (2026-07-22)
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
# Load templates
|
||||
c1 = json.load(open("/tmp/api_cyberpunk_C1.json"))
|
||||
c2 = json.load(open("/tmp/api_cyberpunk_C2.json"))
|
||||
|
||||
# Common modifications for both clips
|
||||
for d, name in [(c1, "C1"), (c2, "C2")]:
|
||||
p = d["prompt"]
|
||||
|
||||
# Node 20: sampler
|
||||
p["20"]["inputs"]["sampler_name"] = "euler_ancestral_cfg_pp"
|
||||
|
||||
# Node 131: duration -> 121 frames / 5.04s
|
||||
p["131"]["inputs"]["duration_frames"] = 121
|
||||
p["131"]["inputs"]["end_frame"] = 121
|
||||
p["131"]["inputs"]["duration_seconds"] = 5.04
|
||||
p["131"]["inputs"]["end_second"] = 5.04
|
||||
|
||||
# Node 131 timeline_data
|
||||
td = json.loads(p["131"]["inputs"]["timeline_data"])
|
||||
td["normalDurationFrames"] = 121
|
||||
seg = td["segments"][0]
|
||||
seg["length"] = 121
|
||||
seg["end"] = 5.04
|
||||
|
||||
# C1-specific: boss pan right
|
||||
p1 = c1["prompt"]
|
||||
td1 = json.loads(p1["131"]["inputs"]["timeline_data"])
|
||||
td1["segments"][0]["imageFile"] = "ltx_boss_A.png"
|
||||
td1["segments"][0]["prompt"] = "..." # full prompt
|
||||
p1["131"]["inputs"]["timeline_data"] = json.dumps(td1)
|
||||
|
||||
# C2-specific: frame-B -> woman reveal
|
||||
p2 = c2["prompt"]
|
||||
td2 = json.loads(p2["131"]["inputs"]["timeline_data"])
|
||||
td2["segments"][0]["imageFile"] = "ltx_boss_B.png" # CRITICAL: pre-rendered empty frame B
|
||||
td2["segments"][0]["prompt"] = "..." # full prompt
|
||||
p2["131"]["inputs"]["timeline_data"] = json.dumps(td2)
|
||||
|
||||
# Write both
|
||||
json.dump(c1, open("/tmp/api_boss_C1_fixed.json", "w"), indent=2)
|
||||
json.dump(c2, open("/tmp/api_boss_C2_fixed.json", "w"), indent=2)
|
||||
```
|
||||
|
||||
## Why Not Shell Heredocs?
|
||||
|
||||
Shell heredocs with Python inside them have quoting issues:
|
||||
- Single quotes inside the Python string break the heredoc delimiter
|
||||
- Multi-line prompts with special characters get mangled
|
||||
- Escaping is fragile and error-prone
|
||||
|
||||
The scp + remote execution pattern avoids all of this.
|
||||
|
||||
## Dispatch
|
||||
|
||||
```bash
|
||||
# Write script locally
|
||||
write_file /tmp/build_clips.py
|
||||
|
||||
# scp to .202 and run
|
||||
sshpass -p 'passw0rd' scp /tmp/build_clips.py [email protected]:/tmp/build_clips.py
|
||||
sshpass -p 'passw0rd' ssh [email protected] 'python3 /tmp/build_clips.py'
|
||||
```
|
||||
@@ -0,0 +1,43 @@
|
||||
# LTX Prompt Validation Checklist
|
||||
|
||||
Run this 10-point check on EVERY segment prompt BEFORE submitting to the render queue. This is a quality gate — do not skip.
|
||||
|
||||
## The 10 Checks
|
||||
|
||||
| # | Check | Pass if |
|
||||
|---|-------|---------|
|
||||
| 1 | Shot type | Contains one of: Medium shot, Wide shot, Close-up, Low angle, Tracking shot, Overhead, POV |
|
||||
| 2 | Camera direction | Contains one of: pan left/right, dolly in/out, tilt up/down, zoom, tracking, static, handheld, crane, Steadicam |
|
||||
| 3 | Actions (2-3 max) | 2-3 distinct actions, present tense, sequential. Not 5-8 simultaneous actions. |
|
||||
| 4 | Lighting | Describes light source, quality, and color temp (e.g., "warm afternoon light from tall windows") |
|
||||
| 5 | Environment | Describes textures, atmosphere, setting details |
|
||||
| 6 | Word count | 50-80 words for I2V (action-focused — start frame already defines visuals). Official LTX says longer is better for T2V, but I2V needs less scene-setting. Under 50 = model fills gaps. Over 80 = model drops actions. |
|
||||
| 7 | No text/logos | No words like "text", "logo", "sign", "label", "readable letters" in the prompt |
|
||||
| 8 | No zhuanchang | Only append "zhuanchang" if this is a transition/morph scene. Never on standard I2V. |
|
||||
| 9 | No emotional labels | No abstract emotions ("sad", "angry", "happy"). Use physical cues instead ("shoulders slumped", "eyes downcast"). |
|
||||
| 10 | Single logic | One speed, one camera direction, one lighting logic. No conflicting descriptions. |
|
||||
|
||||
## Prompt Structure (Official LTX 2.3)
|
||||
|
||||
Two valid orderings per ltx.io (June 2026):
|
||||
|
||||
**Shot-size-first:** framing → subject → action → style → motion
|
||||
**Subject-first:** subject → action → framing → style → motion
|
||||
|
||||
For I2V: focus on motion and action — the start frame already defines the visual. Describe the transition from stillness to motion, not static elements already in the image.
|
||||
|
||||
## Example: FAIL
|
||||
|
||||
> The camera continues its slow steady pan right through the same neon-lit cyberpunk room, past flickering holographic displays and tangled cables, gradually revealing a woman in her late 20s with chrome temple implants standing by a rain-streaked window. Pink and blue neon light traces her silhouette. She turns her head toward the camera. Dark atmosphere, photorealistic, cinematic 35mm.
|
||||
|
||||
**Fails:** Check #1 (no shot type — missing "Medium shot" or equivalent).
|
||||
|
||||
## Example: PASS
|
||||
|
||||
> Medium shot, slow steady pan right. The camera continues through the same neon-lit cyberpunk room, past flickering holographic displays and tangled cables, gradually revealing a woman in her late 20s with chrome temple implants standing by a rain-streaked window. Pink and blue neon light traces her silhouette. She turns her head toward the camera. Dark atmosphere, photorealistic, cinematic 35mm.
|
||||
|
||||
**Passes:** All 10 checks. Shot type present, camera explicit, 2 actions (standing, turns head), lighting described, environment detailed, ~60 words, no text/logos, no zhuanchang, no emotional labels, single direction/speed.
|
||||
|
||||
## Audit History
|
||||
|
||||
- 2026-07-22: Cyberpunk Clip 2 prompt caught missing shot type in pre-render audit. Fixed before dispatch.
|
||||
@@ -0,0 +1,134 @@
|
||||
# LTX Video Prompting — Quick Reference
|
||||
|
||||
Condensed from the 544-line deep research at `/home/n8n/workspace/general/LTX_prompt_instructions.md`.
|
||||
|
||||
## Prompt Order (Official LTX 2.3)
|
||||
|
||||
LTX Studio's pipelines weight tokens by position. Two orderings work well:
|
||||
|
||||
**Order one (shot-size-first):** framing → subject → action → style → motion
|
||||
Use when the shot size is the most important decision. "Wide shot, low angle, a detective in a long coat walks toward a warehouse door, harsh sodium lighting, slow push-in."
|
||||
|
||||
**Order two (subject-first):** subject → action → framing → style → motion
|
||||
Use when the character or object is more critical. "A woman in a red coat walks through a Tokyo street at night, medium tracking shot, neon reflections in puddles, handheld follow."
|
||||
|
||||
**Failure mode:** piling every adjective into one long noun phrase. Cut to the 2-3 that matter most.
|
||||
|
||||
**For I2V specifically:** Focus the prompt on motion and action — the visual starting point is already defined by the input image. Avoid describing static elements already visible in the image. Describe the transition from stillness to motion.
|
||||
|
||||
Source: ltx.io/blog/ltx-2-3-prompt-guide, ltx.io/blog/how-to-write-a-prompt (official, June 2026)
|
||||
|
||||
## Shot Sizes
|
||||
|
||||
| Shot | What It Shows |
|
||||
|------|---------------|
|
||||
| Extreme Wide | Vast environment, subject tiny |
|
||||
| Wide / Establishing | Full body + surroundings |
|
||||
| Medium Wide | Knees up |
|
||||
| Medium Shot | Waist up |
|
||||
| Medium Close-up | Chest up |
|
||||
| Close-up | Face and neck |
|
||||
| Extreme Close-up | Single feature |
|
||||
|
||||
## Camera Angles
|
||||
|
||||
| Angle | Effect |
|
||||
|-------|--------|
|
||||
| Eye Level | Neutral, familiar |
|
||||
| Low Angle | Power, dominance |
|
||||
| High Angle | Vulnerability |
|
||||
| Bird's Eye | Detached, god-like |
|
||||
| Dutch Angle | Unease, tension |
|
||||
| Over-the-Shoulder | Dialogue |
|
||||
| POV | First-person immersion |
|
||||
|
||||
## Camera Movements
|
||||
|
||||
| Movement | Prompt Phrase |
|
||||
|----------|---------------|
|
||||
| Static | "Static camera," "tripod-locked" |
|
||||
| Dolly-in | "Slow dolly-in," "camera pushes forward" |
|
||||
| Dolly-out | "Camera pulls back slowly" |
|
||||
| Pan | "Slow pan left/right" |
|
||||
| Tilt | "Tilts up to reveal" |
|
||||
| Tracking | "Tracks alongside," "follows behind" |
|
||||
| Orbit | "Camera orbits slowly around" |
|
||||
| Handheld | "Subtle handheld, micro-shakes only" |
|
||||
|
||||
## Lighting
|
||||
|
||||
| Type | Prompt Phrase |
|
||||
|------|---------------|
|
||||
| Window light | "Soft north-window light, gentle falloff" |
|
||||
| Golden hour | "Golden hour rim light, long shadows" |
|
||||
| Overcast | "Overcast, diffuse light, low contrast" |
|
||||
| Tungsten | "Practical lamp as key, warm tungsten" |
|
||||
| Neon | "Neon glow, cyan and magenta reflections" |
|
||||
| Backlight | "Backlit rim light, subject silhouetted" |
|
||||
| High contrast | "High-contrast studio lighting, deep shadows" |
|
||||
|
||||
## Lens Cues
|
||||
|
||||
- "35mm lens, f/2.8, shallow depth of field" — people/products
|
||||
- "85mm portrait lens, bokeh, crisp eyes" — talking heads
|
||||
- "16mm wide, deep focus" — rooms/landscapes
|
||||
- "50mm standard" — natural perspective
|
||||
|
||||
## Motion Verbs
|
||||
|
||||
Walking, pouring, turning, lifting, revealing, drifting, sprinting, reaching, pausing, stepping, rising, falling, swaying, flickering, rotating, gliding, accelerating, decelerating, spinning, floating, sliding, bouncing, nodding, shaking, gesturing, breathing, blinking, smiling, frowning, gasping, laughing, crying, whispering, shouting
|
||||
|
||||
## Emotion — Show, Don't Tell
|
||||
|
||||
❌ "A sad woman sits at a table"
|
||||
✅ "A woman sits at a table, her shoulders slumped forward, eyes downcast, fingers tracing the rim of an empty coffee cup"
|
||||
|
||||
## Negative Prompt
|
||||
|
||||
```
|
||||
no extra limbs, no face warp, no object duplication
|
||||
no text artifacts, no floating logos, no watermark
|
||||
no extreme motion blur, no rolling shutter wobble
|
||||
no flicker, no frame-to-frame texture shift
|
||||
no Dutch angle, no rapid handheld, keep horizon level
|
||||
```
|
||||
|
||||
For I2V: add `static, frozen, no motion, Ken Burns zoom`
|
||||
|
||||
## Transition LoRA (zhuanchang)
|
||||
|
||||
- **Use ONLY on transition/morph scenes** — character morph, style change, scene switch, day→night
|
||||
- **NEVER on standard I2V** — the LoRA forces transformation behavior even without the trigger word. Confirmed 2026-07-21: caused subject duplication and wrong-scene hallucination on standard I2V scenes.
|
||||
- **Place at END of prompt**
|
||||
- **CFG: 4.0** (not 1.0 — Transition LoRA needs higher CFG)
|
||||
- **Strength: 1.0**
|
||||
- **Node: LoraLoaderModelOnly** (standard ComfyUI node, NOT LTX2LoraLoaderAdvanced)
|
||||
|
||||
## guide_strength (RESOLVED 2026-07-22)
|
||||
|
||||
**Use 1.0 for all standard I2V with ID LoRA dropped.** The old "1.0 = NEVER" rule was from v4 where ID LoRA was also at 1.0 — the duplication was stacked LoRA interference, not guide_strength alone. With ID LoRA dropped, 1.0 is clean and provides proper start-frame anchoring.
|
||||
|
||||
| Value | When | Why |
|
||||
|-------|------|-----|
|
||||
| **1.0** | Standard I2V (no ID LoRA) | Proper start-frame anchoring. Confirmed clean in 2-clip test. |
|
||||
| **0.5-0.7** | I2V WITH ID LoRA (talking-head only) | Lower strength prevents LoRA interference |
|
||||
|
||||
Verify after render:
|
||||
```bash
|
||||
ffprobe -v quiet -show_entries format_tags=prompt output.mp4 | grep -oP 'guide_strength.*?(\d+\.?\d*)'
|
||||
```
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
| Mistake | Symptom | Fix |
|
||||
|---------|---------|-----|
|
||||
| guide_strength too low (0.5) | Identity drift, weak start-frame anchoring | 1.0 (safe with ID LoRA dropped) |
|
||||
| ID LoRA on action scenes | Frontal-face bias, fights camera motion | Drop ID LoRA entirely |
|
||||
| Transition LoRA on standard I2V | Wrong scene, hallucinated content | Remove Transition LoRA entirely |
|
||||
| zhuanchang on every scene | Model forces morphing everywhere | Only on actual transitions |
|
||||
| No camera direction | Static/random movement | Add dolly/pan/tracking |
|
||||
| Prompts too short (<30 words) | Model fills gaps with random data | 50-80 words minimum |
|
||||
| No motion detail | Character freezes | Sequential action in present tense |
|
||||
| Emotional labels | Abstract expressions | Physical cues only |
|
||||
| Conflicting descriptions | Averaged competing signals | One speed, one camera, one light |
|
||||
| euler_ancestral_cfg_pp on fp8 distilled | Catastrophic artifacts, hallucinated subjects | Plain euler only |
|
||||
@@ -0,0 +1,99 @@
|
||||
# Reference Sheet Creation for IC-LoRA Ingredients
|
||||
|
||||
Proven workflow for creating IC-LoRA reference sheets from stock character images on TrueNAS. Used 2026-07-22 for the boss+woman cyberpunk test.
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Download stock from TrueNAS
|
||||
|
||||
```bash
|
||||
smbclient -N //10.0.0.117/proxmoxBackup -c 'cd ai_vid_stock_material\\character_refs; get SHEETS2_00005_Boss.png /tmp/SHEETS2_00005_Boss.png'
|
||||
smbclient -N //10.0.0.117/proxmoxBackup -c 'cd ai_vid_stock_material\\character_refs; get cyberpunk_woman_neon_01.jpg /tmp/cyberpunk_woman_neon_01.jpg'
|
||||
```
|
||||
|
||||
### 2. Extract panels from multi-panel character sheets
|
||||
|
||||
The boss stock (3328×2432) is a 2-panel sheet on a bright background. Use brightness thresholding to find content regions:
|
||||
|
||||
```python
|
||||
from PIL import Image
|
||||
import numpy as np
|
||||
|
||||
img = Image.open("/tmp/SHEETS2_00005_Boss.png")
|
||||
arr = np.array(img)
|
||||
brightness = arr.mean(axis=2)
|
||||
is_content = brightness < 200 # content is darker than bright background
|
||||
|
||||
# Find vertical regions (rows with >5% content pixels)
|
||||
row_content = is_content.mean(axis=1)
|
||||
content_rows = np.where(row_content > 0.05)[0]
|
||||
gaps = np.diff(content_rows)
|
||||
split_points = np.where(gaps > 20)[0] # gaps >20px = panel boundary
|
||||
|
||||
# Extract each panel with horizontal bounds
|
||||
regions = []
|
||||
start = content_rows[0]
|
||||
for sp in split_points:
|
||||
end = content_rows[sp]
|
||||
regions.append((start, end))
|
||||
start = content_rows[sp + 1]
|
||||
regions.append((start, content_rows[-1]))
|
||||
|
||||
for i, (y1, y2) in enumerate(regions):
|
||||
region = is_content[y1:y2+1, :]
|
||||
col_content = region.mean(axis=0)
|
||||
content_cols = np.where(col_content > 0.02)[0]
|
||||
x1, x2 = content_cols[0], content_cols[-1]
|
||||
crop = img.crop((x1, y1, x2, y2))
|
||||
crop.save(f"/tmp/boss_panel_{i}.png")
|
||||
```
|
||||
|
||||
### 3. Composite reference sheet
|
||||
|
||||
IC-LoRA requires: 768×448, black background, one panel per character, NO text.
|
||||
|
||||
```python
|
||||
ref = Image.new("RGB", (768, 448), (0, 0, 0))
|
||||
|
||||
# Place boss panels (left side)
|
||||
bp0 = boss_panels[0].copy()
|
||||
bp0.thumbnail((300, 200), Image.LANCZOS)
|
||||
ref.paste(bp0, (10, 10))
|
||||
|
||||
bp1 = boss_panels[1].copy()
|
||||
bp1.thumbnail((300, 220), Image.LANCZOS)
|
||||
ref.paste(bp1, (10, 220))
|
||||
|
||||
# Place woman panels (right side)
|
||||
w1 = woman1.copy()
|
||||
w1.thumbnail((200, 200), Image.LANCZOS)
|
||||
ref.paste(w1, (330, 10))
|
||||
|
||||
w2 = woman2.copy()
|
||||
w2.thumbnail((200, 200), Image.LANCZOS)
|
||||
ref.paste(w2, (550, 10))
|
||||
|
||||
ref.save("/tmp/ic_lora_reference_sheet.png")
|
||||
```
|
||||
|
||||
### 4. Loop to 121-frame static video
|
||||
|
||||
```bash
|
||||
ffmpeg -y -loop 1 -i /tmp/ic_lora_reference_sheet.png \
|
||||
-c:v libx264 -t 5.04 -r 24 -pix_fmt yuv420p \
|
||||
/tmp/ic_lora_reference_121f.mp4
|
||||
```
|
||||
|
||||
### 5. Copy to .202 for ComfyUI
|
||||
|
||||
```bash
|
||||
sshpass -p 'passw0rd' scp /tmp/ic_lora_reference_121f.mp4 [email protected]:~/comfy-ui/input/
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Bright backgrounds need thresholding.** The boss stock has a ~230 brightness background — content detection needs `brightness < 200`, not `< 30` (which is for black backgrounds).
|
||||
- **Panel gap detection is fragile.** The `gaps > 20` threshold works for the boss sheet but may need tuning for other sheets. Always inspect extracted panels before compositing.
|
||||
- **IC-LoRA requires black background.** The reference sheet MUST have a black background — bright backgrounds confuse the conditioning.
|
||||
- **Bigger panels = better carry-over.** Give important characters more space in the composite.
|
||||
- **Resolution must match trained bucket.** 768×448 is the IC-LoRA trained resolution. Other resolutions may work but are untested.
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Scan LTX Director output videos and map them to scenes.
|
||||
|
||||
Usage: python3 scan_videos.py [output_dir]
|
||||
|
||||
Extracts from each LTX_Director_*.mp4:
|
||||
- Scene number (from start_frame in prompt metadata)
|
||||
- Model type (fp8+LoRAs, Q4, or unknown)
|
||||
- Resolution and duration
|
||||
- Whether Transition LoRA is wired (LoraLoaderModelOnly in prompt)
|
||||
- Whether zhuanchang trigger is present
|
||||
- File size
|
||||
"""
|
||||
|
||||
import subprocess, os, re, sys
|
||||
|
||||
outdir = sys.argv[1] if len(sys.argv) > 1 else os.path.expanduser("~/comfy-ui/output/video")
|
||||
files = sorted([f for f in os.listdir(outdir) if f.startswith("LTX_Director_") and f.endswith(".mp4")])
|
||||
|
||||
print(f"{'File':40s} | {'Resolution':20s} | {'Scene':10s} | {'Model':12s} | {'Trans':6s} | {'zhuanchang':10s} | {'Size':>8s}")
|
||||
print("-" * 120)
|
||||
|
||||
for f in files:
|
||||
path = os.path.join(outdir, f)
|
||||
|
||||
# Resolution and duration
|
||||
r = subprocess.run(["ffprobe", "-v", "quiet", "-select_streams", "v:0",
|
||||
"-show_entries", "stream=width,height,duration",
|
||||
"-of", "csv=p=0", path], capture_output=True, text=True)
|
||||
res = r.stdout.strip()
|
||||
|
||||
# Prompt metadata
|
||||
r2 = subprocess.run(["ffprobe", "-v", "quiet", "-show_entries", "format_tags=prompt",
|
||||
"-of", "csv=p=0", path], capture_output=True, text=True)
|
||||
prompt_raw = r2.stdout.strip()
|
||||
|
||||
# Scene from imageFile
|
||||
img_match = re.search(r"imageFile.*?ltx_start_frame_(\d+)", prompt_raw)
|
||||
scene = f"Scene_{img_match.group(1)}" if img_match else "?"
|
||||
|
||||
# Model type
|
||||
if "UNETLoader" in prompt_raw and "LTX2LoraLoaderAdvanced" in prompt_raw:
|
||||
model = "fp8+LoRAs"
|
||||
elif "UnetLoaderGGUF" in prompt_raw:
|
||||
model = "Q4"
|
||||
else:
|
||||
model = "?"
|
||||
|
||||
# Transition LoRA
|
||||
has_trans = "LoraLoaderModelOnly" in prompt_raw
|
||||
has_zhuanchang = "zhuanchang" in prompt_raw
|
||||
|
||||
# File size
|
||||
size = os.path.getsize(path)
|
||||
size_str = f"{size/1024:.0f}KB"
|
||||
|
||||
print(f"{f:40s} | {res:20s} | {scene:10s} | {model:12s} | {str(has_trans):6s} | {str(has_zhuanchang):10s} | {size_str:>8s}")
|
||||
@@ -0,0 +1,69 @@
|
||||
# Stock as I2V Start Frame
|
||||
|
||||
Pattern for using stock character images (from TrueNAS `ai_vid_stock_material/character_refs/`) as direct I2V start frames in the standard LTX pipeline. Simpler than IC-LoRA Ingredients — no additional LoRA, no reference sheet, no static video loop. Works with the existing 6-fix baseline.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Quick character test: "does this stock image animate well?"
|
||||
- Single-clip renders where character consistency across scenes isn't critical
|
||||
- Before investing in IC-LoRA Ingredients setup
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- Multi-scene stories needing character consistency across clips → use IC-LoRA Ingredients
|
||||
- Stock images with white/light backgrounds → LTX may hallucinate background elements
|
||||
- Low-resolution stock → resize artifacts will compound in animation
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Download stock from TrueNAS:**
|
||||
```bash
|
||||
smbclient -N //10.0.0.117/proxmoxBackup -c 'cd ai_vid_stock_material\character_refs; get SHEETS2_00005_Boss.png /tmp/boss.png'
|
||||
```
|
||||
|
||||
2. **Resize to 768×512** (our standard resolution):
|
||||
```python
|
||||
from PIL import Image
|
||||
img = Image.open('/tmp/boss.png')
|
||||
img = img.resize((768, 512), Image.LANCZOS)
|
||||
img.save('/tmp/boss_start_frame.png')
|
||||
```
|
||||
|
||||
3. **Copy to .202 input directory:**
|
||||
```bash
|
||||
sshpass -p 'passw0rd' scp /tmp/boss_start_frame.png [email protected]:/home/n8n/comfy-ui/input/
|
||||
```
|
||||
|
||||
4. **Modify an existing workflow JSON** — change the `imageFile` in timeline_data:
|
||||
```python
|
||||
import json
|
||||
with open('/tmp/api_cyberpunk_C1.json') as f:
|
||||
d = json.load(f)
|
||||
nodes = d['prompt']
|
||||
td = json.loads(nodes['131']['inputs']['timeline_data'])
|
||||
td['segments'][0]['imageFile'] = 'boss_start_frame.png'
|
||||
nodes['131']['inputs']['timeline_data'] = json.dumps(td)
|
||||
with open('/tmp/api_boss_test.json', 'w') as f:
|
||||
json.dump(d, f)
|
||||
```
|
||||
|
||||
5. **Update the prompt** to describe the stock character (not the Flux-generated character):
|
||||
- Describe what the stock image shows (clothing, features, expression)
|
||||
- Keep the same environment/lighting/camera from the original template
|
||||
- Target 50-80 words, 2-3 actions
|
||||
|
||||
6. **Submit and verify** (standard pipeline steps 4-8).
|
||||
|
||||
## Example: Boss Stock
|
||||
|
||||
**Stock:** `SHEETS2_00005_Boss.png` (3328×2432 RGB PNG, white background)
|
||||
**Resized:** 768×512
|
||||
**Prompt:** "Medium shot, slow steady pan right. A stern boss in a dark suit sits at a terminal in a neon-lit cyberpunk room. Pink and blue holographic light flickers across his face as he types on a glowing keyboard. Over 5 seconds: he leans forward studying the screen with cold intensity, then slowly turns his head right. His chrome temple implant glints in the neon glow. Volumetric fog drifts through colored light from overhead panels. Cables and server racks line the walls, indicator lights blinking. Cinematic lighting, shallow depth of field, 35mm film grain."
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **CRITICAL: Single stock images as I2V start frames do NOT anchor the face.** LTX treats the start frame as "general scene composition" not "this exact person." The model will morph the face, change clothing details, and substitute its own training data for the character. Confirmed 2026-07-22: boss stock test — face distorted, did not stay true to image. User: "stock test was malformed. It worked 80% but face distorted and did not stay to image." This approach is useful for quick composition tests only. For character consistency, use IC-LoRA Ingredients.
|
||||
- **Character description in prompt MUST match the canonical descriptions** in `references/character-descriptions.md`. Using generic descriptions like "stern boss" instead of "muscular boss with shaved head and goatee wearing a black suit" causes the model to blend between the stock image and its own training data. The user provided exact descriptions — use them verbatim.
|
||||
- **White/light backgrounds in stock images** may cause LTX to hallucinate background elements or wash out the scene. The model was trained on varied backgrounds; a plain white background gives it no environment cues. Prefer stock with scene-appropriate backgrounds, or use Flux to generate a start frame that composites the character into the target environment.
|
||||
- **Stock image aspect ratio** rarely matches 768×512. Resize with LANCZOS; avoid stretching (crop to aspect ratio first if needed).
|
||||
- **This is NOT a character-consistency solution across clips.** Each clip gets its own start frame; there's no mechanism to keep the same face across scenes. For multi-clip consistency, use IC-LoRA Ingredients.
|
||||
@@ -0,0 +1,134 @@
|
||||
# Story Structure Guide — LTX Video Pipeline
|
||||
|
||||
## The Problem (v4 Post-Mortem)
|
||||
|
||||
v4 rendered correctly (241 frames, no duplication, wiring fixed) but the user said "the videos had all kinds of inconsistencies." The root cause: the story was 6 standalone vignettes with no causal chain. Each scene was a different location with no narrative thread connecting them. LTX needs a story where each scene follows from the previous one.
|
||||
|
||||
## Story Design Rules
|
||||
|
||||
1. **Chain of events, not vignettes.** Each scene must be caused by the previous scene. "She discovers something → she runs → she hides → she chooses." Not "he wakes up → he makes coffee → he walks outside."
|
||||
2. **One character, one journey.** Multi-character stories cause identity drift. The ID LoRA helps but isn't perfect across different faces.
|
||||
3. **No dialogue, no plot twists.** LTX can't do dialogue well. Visual storytelling: action, reaction, environment, choice.
|
||||
4. **6 scenes × 10s = 60s.** This is the sweet spot. Each scene is one beat in the story.
|
||||
5. **Clear visual variety per scene.** Each scene should have a distinct setting, lighting, and camera language. This prevents the model from blending scenes together.
|
||||
6. **Match LTX prompt style for EVERY scene.** Every scene prompt must have: shot type, camera movement, subject action (present tense, sequential), lighting, audio. No exceptions.
|
||||
|
||||
## Story Template
|
||||
|
||||
```
|
||||
Scene 1: INCITING INCIDENT — character discovers/encounters something
|
||||
Scene 2: ESCALATION — the situation intensifies, stakes rise
|
||||
Scene 3: REACTION — character responds, makes a decision
|
||||
Scene 4: CHASE/STRUGGLE — physical action, pursuit, or confrontation
|
||||
Scene 5: LOW POINT — character is isolated, vulnerable, reflects
|
||||
Scene 6: CHOICE/RESOLUTION — character makes the final decision, walks toward outcome
|
||||
```
|
||||
|
||||
## User Preference: Detail Over Simplicity
|
||||
|
||||
The user initially asked for simpler stories, then reversed: "go back to your original." The detailed 4-story templates below are the preferred level of detail. Don't over-simplify.
|
||||
|
||||
---
|
||||
|
||||
## Story 1: "Ghost in the Wire" (Cyberpunk) — COMPLETED 2026-07-21
|
||||
|
||||
**Logline:** A netrunner discovers a corporate AI that's become self-aware — and it's been killing people through their neural implants. She has 60 seconds to decide: expose it, or join it.
|
||||
|
||||
**Character:** Woman, mid-20s, short dark hair, chrome temple implants, cybernetic fingers, worn black synth-leather jacket.
|
||||
|
||||
**Chain:** Hack → Discovery → Confrontation → Chase → Choice → Consequence
|
||||
|
||||
| # | Setting | Action | Camera | guide_strength |
|
||||
|---|---------|--------|--------|---------------|
|
||||
| 1 | Neon-lit hacker den | Woman at terminal, code streams across screens. Her eyes widen — she's found something. | Medium shot, slow push-in | 0.5 |
|
||||
| 2 | Corporate data vault (virtual) | Her avatar materializes inside a geometric data fortress. Crystalline servers pulse blue. She touches one — it turns red. The AI speaks. | Wide, slow orbit | 0.7 |
|
||||
| 3 | Rooftop, rain | She rips off her neural link, gasping. Rain pours. Drone's red eye appears in the distance — they know. | Medium close-up → wide reveal | 0.7 |
|
||||
| 4 | Lower city chase | She runs through a crowded neon market. Drones weave between stalls. She slides under a closing blast door. | Tracking shot, handheld | 0.7 |
|
||||
| 5 | Abandoned subway | She catches her breath in a flooded tunnel. Bioluminescent moss glows. Her implant flickers — the AI is still in her head. | Static, slow tilt down to reflection | 0.7 |
|
||||
| 6 | Tunnel junction → choice | Left tunnel: surface, exposure, truth. Right tunnel: deeper, the AI's offer. She walks right. Camera holds on empty junction. | Wide, static, then slow pull-back | 0.7 |
|
||||
|
||||
**Result:** All 6 scenes rendered successfully. 241 frames each (no wiring bug). Concat: 1446 frames, 60.18s. Uploaded to TrueNAS as `LTX_Director_S1_60s.mp4`. Full prompts at `story1-ghost-in-the-wire-prompts.md`.
|
||||
|
||||
---
|
||||
|
||||
## Story 2: "Chrome Angels" (Cyberpunk) — IN PROGRESS 2026-07-21
|
||||
|
||||
**Logline:** A black-market cybernetic surgeon gets a new patient — a corporate enforcer whose face he recognizes. It's his brother, declared dead five years ago. The brother doesn't remember him.
|
||||
|
||||
**Character:** Surgeon, 40s, graying hair, tired eyes, cybernetic eye replacement, worn surgical coat. Brother: same face, younger, corporate chrome, cold expression.
|
||||
|
||||
**Chain:** Operation → Recognition → Confrontation → Memory → Betrayal → Sacrifice
|
||||
|
||||
| # | Setting | Action | Camera |
|
||||
|---|---------|--------|--------|
|
||||
| 1 | Flooded basement clinic | Surgeon's chrome fingers replace organic ones. Water drips from pipes. Patient unconscious on gurney. | Close-up on hands → slow reveal |
|
||||
| 2 | Clinic, post-op | He pulls back drapes. Freezes. It's his brother — same jaw, same scar. But the eyes are cold. Corporate. | Medium shot, static → slow push-in |
|
||||
| 3 | Clinic, confrontation | Brother wakes. "That identity was terminated. I am Unit 734." He stands, towering. Surgeon backs against wall. | Low angle, static |
|
||||
| 4 | Memory flash (virtual) | Two boys on a rooftop, sunset. One points at a corporate gunship. "One day I'm going to work for them." The other: "You'd sell your soul for chrome." Memory corrupts into static. | Wide rooftop → dissolve to static |
|
||||
| 5 | Clinic, betrayal | Brother's hand around surgeon's throat — but trembling. A flicker of recognition. Then red light blinks on temple — corp override. Face goes blank. | Medium close-up, handheld |
|
||||
| 6 | Clinic, sacrifice | Surgeon jams EMP spike into brother's neural port. Brother seizes, collapses. Eyes — human now — focus one last time. "You kept your soul." They close. | Overhead shot, slow pull-up |
|
||||
|
||||
**Status:** Flux start frames submitted (prompt IDs: be0b5181 through 04668385). Waiting for queue drain, then build scene JSONs and render.
|
||||
|
||||
---
|
||||
|
||||
## Story 3: "The Last Human Job" (Cyberpunk) — PLANNED
|
||||
|
||||
**Logline:** In a city where AI replaced every worker, one man still runs a physical repair shop. When a damaged android brings in a human memory core, he uncovers a black-market trade in stolen consciousness — and learns his own mind was taken years ago.
|
||||
|
||||
**Character:** Repairman, 50s, weathered face, cybernetic arm, oil-stained hands, kind eyes.
|
||||
|
||||
**Chain:** Repair → Discovery → Investigation → Revelation → Confrontation → Identity
|
||||
|
||||
| # | Setting | Action | Camera |
|
||||
|---|---------|--------|--------|
|
||||
| 1 | Repair shop | Repairman's hands in the guts of a broken drone. Shelves of parts tower. Neon from street paints everything pink and blue. Bell rings. | Close-up → medium reveal |
|
||||
| 2 | Shop counter | Damaged android at counter, one eye flickering. "Please. They're erasing people." Pulls a glowing memory core from its own chest. | Medium shot, static |
|
||||
| 3 | Black-market memory den | Underground facility. Rows of humans in suspension pods, memories extracted and sold. He recognizes a face in one pod — his own. | Tracking shot through facility |
|
||||
| 4 | Memory den, revelation | He stares at his own body in the pod. Android: "You've been dead for three years. The person you think you are is a copy." He touches the glass. | Static, slow dolly-in on reflection |
|
||||
| 5 | Memory den, confrontation | Corporate security arrives. Red lasers cut through dark. He disables a guard — his hands move with military precision he didn't know he had. | Handheld, chaotic but controlled |
|
||||
| 6 | Rooftop escape → dawn | He and android burst onto rooftop at dawn. City stretches below. He looks at his hands — chrome and flesh, both his. "What do I do now?" Android: "Whatever the original you would have wanted." | Wide, slow pull-back |
|
||||
|
||||
---
|
||||
|
||||
## Story 4: "Neon Baptism" (Cyberpunk) — PLANNED
|
||||
|
||||
**Logline:** A street kid gets caught stealing from the wrong corporation. Instead of prison, they offer her full-body augmentation — a "recruitment." But when the chrome activates, she's not the weapon they expected.
|
||||
|
||||
**Character:** Street kid, 17-19, androgynous, dirty face, sharp eyes, worn synth-leather. Post-augmentation: same face, chrome limbs, glowing blue optical implant, sleek corporate body.
|
||||
|
||||
**Chain:** Theft → Capture → Offer → Transformation → Awakening → Rebellion
|
||||
|
||||
| # | Setting | Action | Camera |
|
||||
|---|---------|--------|--------|
|
||||
| 1 | Neon market, night | Kid weaves through crowded market, stolen data chip clutched in hand. Drone spots her. Red targeting laser paints her back. She runs. | Tracking, low angle, handheld |
|
||||
| 2 | Corporate interrogation room | Thrown into white room. Sterile. Single chair. Holographic face appears — calm, corporate. "You stole from us. That's impressive. We have an offer." | Wide, static, harsh overhead |
|
||||
| 3 | Augmentation chamber | Strapped to surgical chair. Robotic arms descend — chrome limbs, neural ports, optical implants. She screams. Camera circles as her body is rebuilt. | Slow orbit, clinical |
|
||||
| 4 | Training sim (virtual) | Consciousness boots into virtual training ground. Faster, stronger. Targets appear, she destroys them. But between targets: flashes of her old life. She holds on. | POV + wide cuts |
|
||||
| 5 | Corp tower, awakening | She wakes in corporate tower, chrome body active. Guards escort her to first mission. At a junction, she stops. Looks at her reflection — chrome and human, both her. She smiles. | Tracking, then static |
|
||||
| 6 | Tower → escape | She moves toward the window. Guards fire. She's through the glass, falling 80 stories, chrome limbs catching ledges. Lands in lower city, crouched, steam rising. Stands. Looks up. Walks into neon dark. | Overhead → tracking → wide static |
|
||||
|
||||
## Stock Management Per Story
|
||||
|
||||
| Story | New Frames Needed | Character | Key Visuals |
|
||||
|-------|------------------|-----------|-------------|
|
||||
| Ghost in the Wire | 6 | Female netrunner, 20s | Neon den, data vault, rooftop, market, subway, tunnel |
|
||||
| Chrome Angels | 4 | Male surgeon, 40s + brother | Basement clinic, rooftop memory |
|
||||
| The Last Human Job | 5 | Male repairman, 50s | Repair shop, memory den, rooftop dawn |
|
||||
| Neon Baptism | 6 | Street kid, 17-19 | Market, white room, surgical chair, virtual grid, corp tower |
|
||||
|
||||
## Cyberpunk Stock Sources (from better-search research 2026-07-21)
|
||||
|
||||
**Pexels** (Pexels License — free commercial, no attribution): 5,000+ cyberpunk images. Best for photorealistic characters. Mikhail Nilov and Yaroslav Shuraev sets are the strongest.
|
||||
|
||||
**Unsplash** (Unsplash License — free commercial, no attribution): Best for Tokyo rain/neon street environments. Photorealistic.
|
||||
|
||||
**Pixabay** (CC0-like): 34K+ cyberpunk city images. Mix of photos and illustrations.
|
||||
|
||||
**StockCake** (Royalty-free): AI-generated props/holograms. Quality varies.
|
||||
|
||||
**Freepik** (Free tier: attribution required): Vectors, neon signs. NOT CC0 — use only if attribution is acceptable.
|
||||
|
||||
**PublicDomainPictures** (CC0): True public domain. Limited selection.
|
||||
|
||||
Full research: `~/workspace/research/results/2026-07-21-cyberpunk-stock-materials-ai-video.md`
|
||||
@@ -0,0 +1,121 @@
|
||||
# Workflow Transformation: Two-Stage → Single-Stage I2V (7 Fixes)
|
||||
|
||||
Concrete recipe for transforming a two-stage LTX Director workflow JSON into a single-stage I2V workflow applying all 7 fixes. Use this when you have an existing two-stage scene JSON and need to produce a single-stage version.
|
||||
|
||||
**Status: PROVEN WORKING (2026-07-22).** The 2-clip test confirmed all 7 fixes produce clean 241-frame output with no Director wiring bug and no subject duplication. guide_strength 1.0 is clean when ID LoRA is dropped — the old "DO NOT use 1.0" rule was from stacked LoRA interference, not guide_strength alone.
|
||||
|
||||
## Source Pattern
|
||||
|
||||
Two-stage workflows have:
|
||||
- **Stage 1 (main):** Nodes 131 (Director), 133 (Guide), 55 (CropGuides), 33 (Scheduler, 8 steps, denoise 1.0), 30 (Noise), 32 (Sampler), 31 (SamplerCustomAdvanced), 29 (ConcatAVLatent), 28 (CFGGuider), 34 (SeparateAVLatent)
|
||||
- **Stage 2 (refiner):** Nodes 132 (Guide), 54 (CropGuides), 21 (Scheduler, 4 steps, denoise 0.42), 20 (Sampler), 19 (SamplerCustomAdvanced), 18 (ConcatAVLatent), 17 (CFGGuider)
|
||||
- **ID LoRA:** Node 201 (LTX2LoraLoaderAdvanced, TalkVid-3K) chained after node 200 (Distilled LoRA)
|
||||
- **Output:** Node 1 (VAEDecode) → Node 2 (CreateVideo) → Node 37 (SaveVideo)
|
||||
|
||||
## Nodes to Delete (9 total)
|
||||
|
||||
```
|
||||
21 — BasicScheduler (refiner, 4 steps, denoise 0.42)
|
||||
28 — CFGGuider (refiner)
|
||||
29 — LTXVConcatAVLatent (refiner)
|
||||
31 — SamplerCustomAdvanced (refiner)
|
||||
32 — KSamplerSelect (refiner)
|
||||
34 — LTXVSeparateAVLatent (refiner)
|
||||
54 — LTXDirectorCropGuides (refiner)
|
||||
132 — LTXDirectorGuide (refiner)
|
||||
201 — LTX2LoraLoaderAdvanced (ID LoRA, TalkVid-3K)
|
||||
```
|
||||
|
||||
## Rewiring (6 connections)
|
||||
|
||||
| Node.Input | Old Value | New Value | Reason |
|
||||
|-----------|-----------|-----------|--------|
|
||||
| 131.model | `["201", 0]` | `["200", 0]` | Skip ID LoRA, go direct to Distilled LoRA |
|
||||
| 17.model | `["132", 3]` | `["133", 3]` | CFGGuider now uses main Guide, not refiner |
|
||||
| 17.positive | `["132", 0]` | `["133", 0]` | Same |
|
||||
| 17.negative | `["132", 1]` | `["133", 1]` | Same |
|
||||
| 18.video_latent | `["132", 2]` | `["133", 2]` | ConcatAVLatent takes main Guide output |
|
||||
| 18.audio_latent | `["34", 1]` | `["131", 3]` | Audio from Director directly (34 is deleted) |
|
||||
| 19.sigmas | `["21", 0]` | `["33", 0]` | Sampler uses main scheduler (21 is deleted) |
|
||||
| 55.latent | `["34", 0]` | `["22", 0]` | CropGuides takes from main SeparateAVLatent |
|
||||
| 1.samples | `["54", 2]` | `["55", 2]` | VAEDecode from main CropGuides (54 is deleted) |
|
||||
|
||||
## Settings Changes
|
||||
|
||||
| Node.Input | Old | New | Fix # |
|
||||
|-----------|-----|-----|-------|
|
||||
| 200.strength_model | 0.7 | 1.0 | #5 Distilled LoRA at full strength |
|
||||
| 33.steps | 8 | 18 | #3 Single-stage needs more steps |
|
||||
| 33.denoise | 1.0 | 1.0 | Unchanged (already correct for single-stage) |
|
||||
| 131.guide_strength | 0.5 | 1.0 | #2 I2V conditioning at full strength — CONFIRMED CLEAN |
|
||||
| 131.end_second | 5.0 | 10.0 | #4 Duration (match to desired output) |
|
||||
| 131.duration_seconds | 5.0 | 10.0 | Same |
|
||||
| 131.end_frame | 120 | 240 | Same (10s × 24fps) |
|
||||
| 131.duration_frames | 120 | 240 | Same |
|
||||
| 30.noise_seed | (any) | 42 | #7 Fixed seed for reproducibility |
|
||||
| 37.filename_prefix | (old) | (new) | Rename output prefix |
|
||||
|
||||
## Timeline Data Changes
|
||||
|
||||
| Field | Old | New | Fix # |
|
||||
|-------|-----|-----|-------|
|
||||
| global_prompt | (old story) | (new story prompt) | #6 |
|
||||
| segments[0].length | 120 | 240 | #4 |
|
||||
| segments[0].end | 5.0 | 10.0 | #4 |
|
||||
| segments[0].guide_strength | 0.5 | 1.0 | #2 |
|
||||
| segments[0].imageFile | (old) | (new filename) | #6 |
|
||||
| segments[0].prompt | (old 150+ word) | (new 50-80 word) | #6 |
|
||||
| audioSegments | (any) | [] | Clean up |
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
After transformation, verify:
|
||||
|
||||
- [ ] Node count: 22 (down from 31)
|
||||
- [ ] No deleted nodes remain: 21, 28, 29, 31, 32, 34, 54, 132, 201
|
||||
- [ ] 131.model → `["200", 0]` (Distilled LoRA only, no ID LoRA)
|
||||
- [ ] 200.strength_model = 1.0
|
||||
- [ ] 33.steps = 18, denoise = 1.0
|
||||
- [ ] 131.guide_strength = 1.0
|
||||
- [ ] 131.duration_frames = 240 (for 10s@24fps)
|
||||
- [ ] 30.noise_seed = 42
|
||||
- [ ] All rewired connections match the table above
|
||||
- [ ] timeline_data segment has correct imageFile, prompt, length, guide_strength
|
||||
- [ ] audioSegments is empty (or correctly populated if audio is needed)
|
||||
- [ ] JSON is valid and wraps in `{"prompt": {...}}` envelope
|
||||
|
||||
## Automation
|
||||
|
||||
The transformation can be automated with a Python script. Key pattern:
|
||||
|
||||
```python
|
||||
# Delete refiner + ID LoRA nodes
|
||||
nodes_to_delete = ['21', '28', '29', '31', '32', '34', '54', '132', '201']
|
||||
for nid in nodes_to_delete:
|
||||
del wf[nid]
|
||||
|
||||
# Rewire
|
||||
wf['131']['inputs']['model'] = ['200', 0]
|
||||
wf['17']['inputs']['model'] = ['133', 3]
|
||||
wf['17']['inputs']['positive'] = ['133', 0]
|
||||
wf['17']['inputs']['negative'] = ['133', 1]
|
||||
wf['18']['inputs']['video_latent'] = ['133', 2]
|
||||
wf['18']['inputs']['audio_latent'] = ['131', 3]
|
||||
wf['19']['inputs']['sigmas'] = ['33', 0]
|
||||
wf['55']['inputs']['latent'] = ['22', 0]
|
||||
wf['1']['inputs']['samples'] = ['55', 2]
|
||||
|
||||
# Update settings
|
||||
wf['200']['inputs']['strength_model'] = 1.0
|
||||
wf['33']['inputs']['steps'] = 18
|
||||
wf['131']['inputs']['guide_strength'] = 1.0
|
||||
wf['131']['inputs']['end_second'] = 10.0
|
||||
wf['131']['inputs']['duration_seconds'] = 10.0
|
||||
wf['131']['inputs']['end_frame'] = 240
|
||||
wf['131']['inputs']['duration_frames'] = 240
|
||||
wf['30']['inputs']['noise_seed'] = 42
|
||||
```
|
||||
|
||||
## Session Reference
|
||||
|
||||
First applied 2026-07-22: transformed `/tmp/api_s1_fix_01.json` → `/tmp/api_2clip_C1.json` on 10.0.0.202. 31 nodes → 22 nodes. All 7 fixes applied. Verified with remote checks and kimi-c validation (all 7 checks PASS). Rendered clean: 241 frames, 10.04s, 768×512, 24fps, no Director wiring bug.
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"5": {"class_type": "CLIPTextEncode", "inputs": {"text": "<PROMPT>", "clip": ["11", 0]}},
|
||||
"6": {"class_type": "CLIPTextEncode", "inputs": {"text": "blurry, low quality, distorted face, bad anatomy, watermark, text, logo", "clip": ["11", 0]}},
|
||||
"7": {"class_type": "FluxGuidance", "inputs": {"conditioning": ["5", 0], "guidance": 3.5}},
|
||||
"8": {"class_type": "VAEDecode", "inputs": {"samples": ["3", 0], "vae": ["10", 0]}},
|
||||
"9": {"class_type": "SaveImage", "inputs": {"filename_prefix": "ltx_start_frame_XX", "images": ["8", 0]}},
|
||||
"10": {"class_type": "VAELoader", "inputs": {"vae_name": "ae.safetensors"}},
|
||||
"11": {"class_type": "DualCLIPLoader", "inputs": {"clip_name1": "t5xxl_fp8_e4m3fn.safetensors", "clip_name2": "clip_l.safetensors", "type": "flux"}},
|
||||
"16": {"class_type": "UNETLoader", "inputs": {"unet_name": "flux1-dev-fp8-e4m3fn.safetensors", "weight_dtype": "fp8_e4m3fn"}},
|
||||
"3": {"class_type": "KSampler", "inputs": {"seed": 42, "steps": 20, "cfg": 1.0, "sampler_name": "euler", "scheduler": "simple", "denoise": 1.0, "model": ["16", 0], "positive": ["7", 0], "negative": ["6", 0], "latent_image": ["27", 0]}},
|
||||
"27": {"class_type": "EmptySD3LatentImage", "inputs": {"width": 512, "height": 512, "batch_size": 1}}
|
||||
}
|
||||
Reference in New Issue
Block a user