109 lines
5.5 KiB
Markdown
109 lines
5.5 KiB
Markdown
# IC-LoRA Ingredients — Reference Sheet Control for LTX 2.3
|
||
|
||
**Model:** Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients
|
||
**File:** `ltx-2.3-22b-ic-lora-ingredients-0.9.safetensors` (1.31 GB)
|
||
**HF URL:** https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients
|
||
**Status:** Downloaded on .202 (2026-07-22, 1.31 GB) — gated, requires "Agree and Access" on HF website + valid token
|
||
**Token:** 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 (user: MDKRUSH, 2026-07-22)
|
||
**Trained bucket:** 768×448, 121 frames, 24 fps
|
||
|
||
## fp8 Compatibility Gap (UNRESOLVED — deep research in progress 2026-07-22)
|
||
|
||
The official IC-LoRA workflow (`LTX-2.3_ICLoRA_Ingredients_Single_Stage_Distilled.json`) uses a completely different architecture from our pipeline:
|
||
|
||
| Component | Official IC-LoRA Workflow | Our Pipeline |
|
||
|-----------|--------------------------|--------------|
|
||
| Model loader | CheckpointLoaderSimple (full 46GB checkpoint) | UNETLoader (fp8 distilled, 23GB) |
|
||
| Sampler | SamplerCustomAdvanced + ManualSigmas + CFGGuider | LTXDirector (single-stage) |
|
||
| Text encoder | GemmaAPITextEncode | LTXConditioning |
|
||
| Total nodes | 38 | 22 |
|
||
|
||
**Key question:** Can `LTXICLoRALoaderModelOnly` accept a model from UNETLoader (fp8 distilled transformer) instead of CheckpointLoaderSimple? The node takes a "model" input — compatibility with fp8 UNET-only model is unverified. Deep research dispatched 2026-07-22 to answer this and 11 other compatibility questions. Results pending at `/home/n8n/workspace/research/results/2026-07-22-ic-lora-fp8-compatibility.md`.
|
||
|
||
## What It Does
|
||
|
||
Conditions video generation on a **reference sheet** — a single composite image inventorying characters, props, and location. The model reads the reference latents in-context and renders a new clip whose characters, props, and setting match the sheet. This is the solution for using stock character sheets (like `SHEETS2_00005_Boss.png` on TrueNAS) as character consistency anchors.
|
||
|
||
## Recommended Settings (from official README)
|
||
|
||
| Setting | Value | Notes |
|
||
|---------|-------|-------|
|
||
| LoRA strength | **1.4** | NOT 1.0 — official recommendation |
|
||
| Inference steps | **30** | Higher than our standard 18 |
|
||
| Guidance scale | **4.0** | NOT 1.0 — video-to-video mode |
|
||
| Resolution | 768×448 | Trained bucket — best results here |
|
||
| Frames | 121 | 24 fps |
|
||
| Negative prompt | `worst quality, inconsistent motion, blurry, jittery, distorted` | |
|
||
| STG | mode `stg_v`, block 29, scale 1.0 | Helps motion stability |
|
||
|
||
## Prompt Format
|
||
|
||
Two-part structure (matching training):
|
||
|
||
```
|
||
Reference sheet: <description of the panels in the sheet — characters, props, location>
|
||
Generated video: <description of the action / shot you want generated>
|
||
```
|
||
|
||
The `Reference sheet:` text describes what's in the panels. The `Generated video:` text drives the action. The model reads the reference latents for "what things look like" and the prompt for "what happens."
|
||
|
||
## Control Signal Requirements
|
||
|
||
- Reference sheet: single composite image with one clean panel per visual element
|
||
- Each character: face close-up + body turnaround
|
||
- Each prop: product-style render
|
||
- One clean location panel
|
||
- Laid out on black background with NO text
|
||
- **Bigger panels carry over better** — give important elements larger panels
|
||
- Reference must be looped into a static video ≥ 121 frames at output resolution
|
||
|
||
## Node Chain (ComfyUI)
|
||
|
||
```
|
||
CheckpointLoaderSimple (full 46GB checkpoint — adapt to UNETLoader fp8 for 24GB)
|
||
→ LTXICLoRALoaderModelOnly (ingredients-0.9.safetensors, strength 1.4)
|
||
→ LTXAddVideoICLoRAGuide (reference sheet as static video input)
|
||
→ LTXVCropGuides → SamplerCustomAdvanced
|
||
```
|
||
|
||
## Nodes on .202
|
||
|
||
- `LTXICLoRALoaderModelOnly` — in `iclora.py`, loads IC-LoRA and extracts `latent_downscale_factor`
|
||
- `LTXAddVideoICLoRAGuide` — in `iclora.py`, applies reference video conditioning
|
||
- `LTXAddVideoICLoRAGuideAdvanced` — extended version with `attention_strength` and `attention_mask`
|
||
|
||
## Example Workflow
|
||
|
||
On .202: `~/comfy-ui/custom_nodes/ComfyUI-LTXVideo/example_workflows/2.3/LTX-2.3_ICLoRA_Ingredients_Single_Stage_Distilled.json`
|
||
|
||
Uses 39 nodes including: CheckpointLoaderSimple, LTXICLoRALoaderModelOnly, LTXAddVideoICLoRAGuide, GemmaAPITextEncode, LTXVCropGuides, SamplerCustomAdvanced, ManualSigmas.
|
||
|
||
## Download Process
|
||
|
||
1. Visit https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients
|
||
2. Click "Agree and Access" (gated model)
|
||
3. Download with HF token:
|
||
```bash
|
||
HF_TOKEN='hf_...' huggingface-cli download Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients \
|
||
ltx-2.3-22b-ic-lora-ingredients-0.9.safetensors \
|
||
--local-dir ~/comfy-ui/models/loras/
|
||
```
|
||
4. Place in `~/comfy-ui/models/loras/` on .202
|
||
|
||
## Stock Integration
|
||
|
||
The boss stock on TrueNAS (`ai_vid_stock_material/character_refs/SHEETS2_00005_Boss.png`) is a character sheet — exactly the input format IC-LoRA Ingredients expects. To use it:
|
||
|
||
1. Download the character sheet from TrueNAS
|
||
2. Create a reference sheet image with the boss character panel(s)
|
||
3. Loop into a 121-frame static video at 768×448
|
||
4. Feed as the reference/control input to LTXAddVideoICLoRAGuide
|
||
|
||
## Tips (from README)
|
||
|
||
- **Bigger panels carry over better** — give important characters/props larger panels
|
||
- **Identity drift fix:** ensure clean front-facing close-up + full turnaround for each character
|
||
- **Element not appearing:** add a dedicated panel for any prop/character that needs to persist
|
||
- **Reference too short:** static video must be ≥ 121 frames
|
||
- **Element-driven reference-sheet generator** exists for authoring sheets (referenced in README)
|