SpeedyFoxAi/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix: flatten comfyui and ltx-video-pipeline to Gitea convention (flat skill dirs). comfyui v5.1.0, ltx-video-pipeline v2.0.0.

Browse Source
This commit is contained in:
Hermes Agent
2026-07-22 16:19:32 -05:00
parent d94132f4ca
commit d824f865c1
61 changed files with 184 additions and 674 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ltx-video-pipeline/SKILL.md
+184 -10
View File
@@ -15,6 +15,31 @@ metadata:
End-to-end LTX Video pipeline on the ComfyUI LXC at 10.0.0.202.
## Naming Convention (FIRM — user directive 2026-07-22)
All output files uploaded to TrueNAS MUST use this format:
```
LTX_YYYY-MM-DD_<what>_<sampler>_<duration>.mp4
```
Examples:
- `LTX_2026-07-22_cyberpunk-boss-pan_euler_10s.mp4`
- `LTX_2026-07-22_cyberpunk-boss-pan_euler-ancestral_5s.mp4`
- `LTX_2026-07-22_boss-stock-test_euler-ancestral_10s.mp4`
Components:
- `YYYY-MM-DD`: render date
- `<what>`: short kebab-case description of content (e.g., `cyberpunk-boss-pan`, `boss-stock-test`, `woman-window-reveal`)
- `<sampler>`: `euler` or `euler-ancestral`
- `<duration>`: `5s` or `10s`
Do NOT use generic names like `LTX_cyberpunk_C1_00005_.mp4` or `LTX_2clip_test_C1_00002_.mp4` — these are ComfyUI auto-generated filenames that convey zero information about what's in the video. Rename during the concat/upload step.
## Character Descriptions
See `references/character-descriptions.md` for the canonical character descriptions used in prompts and Flux panorama generation. These are the ground truth — do not improvise or use generic descriptions.
## Environment
| Setting | Value |
@@ -113,7 +138,7 @@ Claude Opus SSH'd into 10.0.0.202, read the workflow JSON, read the LTXDirectorG
| Resolution | 768×512 | LTX trained for widescreen; 512×512 is suboptimal |
| FPS | 24 | Standard cinematic |
| Duration | 5s (121 frames) for testing, 10s (241 frames) for final | Test at 5s first — temporal coherence degrades after 5-6s |
| Sampler | euler | |
| Sampler | **euler** (ONLY — mandatory) | euler_ancestral_cfg_pp FAILS catastrophically with fp8 distilled model (hallucinated subjects, backwards limbs). Plain euler only. See §Sampler below. |
| Scheduler | **simple** | NOT linear_quadratic — distilled model is fragile with non-standard schedules |
| Steps | 16-20 single-stage | No refiner. Two-stage (8+4) may smear, not refine |
| CFG | 1.0 | Distilled model — do NOT raise for standard scenes |
@@ -122,6 +147,41 @@ Claude Opus SSH'd into 10.0.0.202, read the workflow JSON, read the LTXDirectorG
| ID LoRA | **Dropped** | TalkVid-3K is talking-head only; drop for all non-talking scenes |
| Peak VRAM | ~23.5GB / 24GB | |
### Sampler — plain `euler` ONLY (RESOLVED 2026-07-22)
**`euler_ancestral_cfg_pp` FAILS with fp8 distilled model.** Confirmed 2026-07-22: 2-clip test produced catastrophic artifacts — hallucinated wrong man, woman falls face-first, legs turn backwards. Community consensus (RuneXX) was correct: `euler_ancestral_cfg_pp` is for the full dev model (non-distilled, CFG>1), not our fp8 distilled pipeline. The official Lightricks recommendation applies to dev model, not distilled — the conflict was a model mismatch, not a contradiction.
**Use plain `euler` for ALL renders. No exceptions.** Scheduler: `simple`. Steps: 18. CFG: 1.0.
### Duration — 5s (121 frames) Sweet Spot (VALIDATED 2026-07-22)
Community sweet spot is 5-6s (121f). 10s risks slow-motion. Frame count must be 8n+1 — 121 works (8×15+1). Change `duration_frames: 240` → `121`, `end_frame: 240` → `121`, `duration_seconds: 10.0` → `5.04`. Keep steps=18, scheduler=simple, denoise=1. Confirmed clean 121-frame output in 2-clip test.
## LTX Director 2.0 (installed on .202, v2.0.2)
Director 2.0 is already on .202 (commit 0dfa657). We use the LTXDirector node — 2.0 is the same node with new features. Key additions relevant to our pipeline:
### NAG (Negative-Aware Guidance)
Enables negative prompts at CFG=1 for distilled model. Wiring:
- Checkpoint → LTX2 NAG (model input)
- LTX2 NAG model output → LoRA chain
- Gemma API Text Encode → `nag_cond_video` + `nag_cond_audio`
- Gemma API Text Encode → LTXConditioning `negative` input
- Remove ConditioningZeroOut node
Known pitfall: last-frame burn if Director Guide positive/negative wiring is wrong (GitHub #153). NAG is the highest-value Director 2.0 feature for our pipeline — evaluate before IC-LoRA.
### IC-LoRA Ingredients (Character Consistency)
LTX-2.3-22b-IC-LoRA-Ingredients model for character consistency via reference style sheet. NOT on .202 yet — needs download from HuggingFace (gated, requires "Agree and Access"). LoRA stack: Distilled @ 0.6 + IC-LoRA-Ingredients @ 1.4 + VBVR @ 0.7. Prompt format: `reference: [describe style sheet]` then `Generated video: [describe action]`. Forces AI to match face, clothing, props from reference image. Full details in `references/ic-lora-ingredients.md`.
**Stock integration note:** The character stock on TrueNAS (`SHEETS2_00005_Boss.png`, `cyberpunk_woman_neon_*.jpg`) is the input format IC-LoRA Ingredients expects. The 2-clip test uses Flux-generated panoramas — stock materials are NOT wired into the standard I2V pipeline. To use stock as character anchors, IC-LoRA Ingredients is required.
### IC-LoRA Control (Canny/Depth/Pose)
Motion transfer from reference video. Not relevant for our I2V pipeline — we control motion via prompts and start frames.
### guide_strength — RESOLVED (2026-07-22)
**Use 1.0.** The 2-clip test confirmed: guide_strength 1.0 is clean with ID LoRA dropped. The old "DO NOT use 1.0" rule was based on v4 failures where ID LoRA was ALSO at 1.0 — the duplication was from stacked LoRA interference, not from guide_strength alone. With ID LoRA dropped, 1.0 is the correct value for proper I2V start-frame anchoring.
@@ -273,12 +333,92 @@ Added the text/watermark line — LTX-2.3's upscaler v1.0 was trained on data co
### Prompting Workflow
1. Write the full prompt with all required elements (shot, camera, action, lighting, environment)
2. Only append `zhuanchang` if the scene involves a transformation/morph
3. Use the same character description across all scenes (from global prompt or character sheet)
4. Vary lighting and atmosphere per scene to match the setting
5. Test one scene first before rendering all 6
6. **MANDATORY: Validate every prompt against the 10-point checklist in `references/prompt-validation-checklist.md` BEFORE submitting to render queue.** This is a FIRM quality gate — do not skip. The user's standing rule: "ensure prompts match ltx standard. ALWAYS. This should be FIRM in memory and validate before EVERY run." If any check fails, fix the prompt and re-validate. Do not submit a workflow with a failing prompt.
7. Verify guide_strength is 1.0 in every scene JSON before submitting
2. **Use canonical character descriptions from `references/character-descriptions.md`** — do NOT improvise or use generic descriptions. The user provided exact descriptions for each character; use them verbatim in every prompt.
3. Only append `zhuanchang` if the scene involves a transformation/morph
4. Use the same character description across all scenes (from `references/character-descriptions.md` or character sheet)
5. Vary lighting and atmosphere per scene to match the setting
6. Test one scene first before rendering all 6
7. **MANDATORY: Validate every prompt against the 10-point checklist in `references/prompt-validation-checklist.md` BEFORE submitting to render queue.** This is a FIRM quality gate — do not skip. The user's standing rule: "ensure prompts match ltx standard. ALWAYS. This should be FIRM in memory and validate before EVERY run." If any check fails, fix the prompt and re-validate. Do not submit a workflow with a failing prompt.
8. Verify guide_strength is 1.0 in every scene JSON before submitting
## Skill Boundary: Which ComfyUI Skill to Use
Two ComfyUI skills exist in this profile. Use the RIGHT one for the task — they serve different purposes and using the wrong one wastes time.
| Task | Use This Skill | Why |
|------|---------------|-----|
| LTX video renders (any kind) | **`media/ltx-video-pipeline`** (this skill) | Has the 6-fix baseline, prompt standards, pitfalls, Director wiring fix, TrueNAS workflow |
| Prompt validation for LTX | **`media/ltx-video-pipeline`** | 10-point checklist, character descriptions, zhuanchang rules |
| Workflow JSON building for LTX | **`media/ltx-video-pipeline`** | Model chain, node wiring, segment format, duration control |
| Story structure / scene planning | **`media/ltx-video-pipeline`** | Story templates, causal chain rules, 2-clip test pattern |
| TrueNAS uploads of LTX output | **`media/ltx-video-pipeline`** | Naming convention, smbclient pattern, stock directory layout |
| ComfyUI health check | **`creative/comfyui`** (official v5.1.0) | `health_check.py` script, server reachability, model inventory |
| Missing node/model install | **`creative/comfyui`** | `check_deps.py` + `auto_fix_deps.py` |
| Generic image generation (SD, Flux) | **`creative/comfyui`** | `run_workflow.py`, batch sweeps, parameter injection |
| Workflow schema extraction | **`creative/comfyui`** | `extract_schema.py` |
| WebSocket monitoring | **`creative/comfyui`** | `ws_monitor.py` |
| Non-LTX video (Wan, AnimateDiff) | **`creative/comfyui`** | Example workflows, video generation support |
**Rule of thumb:** If the task involves LTX on .202, use this skill. If it's general ComfyUI management or non-LTX generation, use `creative/comfyui`.
## ComfyUI Skill Integration (2026-07-22)
**Status: INSTALLED (v5.1.0).** The official Hermes ComfyUI skill is installed at `~/.hermes/profiles/general/skills/creative/comfyui/`. It provides agentic workflow automation — parameter injection, batch runs, real-time monitoring, node/model management — without manual UI interaction.
**How it was installed (2026-07-22):**
```bash
# The skill lives in the Hermes bundle (~/.hermes/skills/creative/comfyui/) but is
# NOT auto-seeded into profiles. `hermes update` updates the bundle, not the profile.
# `hermes skills install comfyui` finds community/hub matches but NOT the bundled one.
# Correct method: copy from bundle to profile.
cp -r ~/.hermes/skills/creative/comfyui ~/.hermes/profiles/general/skills/creative/
```
**Post-install configuration for remote ComfyUI (10.0.0.202):**
```bash
# Set default host (or pass --host to every command)
export COMFYUI_HOST="http://10.0.0.202:8188"
# Verify (health_check.py defaults to localhost — must pass --host)
python3 ~/.hermes/profiles/general/skills/creative/comfyui/scripts/health_check.py \
--host http://10.0.0.202:8188
# Smoke test
python3 ~/.hermes/profiles/general/skills/creative/comfyui/scripts/run_workflow.py \
--workflow ~/.hermes/profiles/general/skills/creative/comfyui/workflows/sd15_txt2img.json \
--args '{"prompt": "test", "steps": 4}' \
--host http://10.0.0.202:8188 \
--output-dir /tmp/comfyui-test
```
**Scripts included:**
| Script | Purpose |
|--------|---------|
| `hardware_check.py` | Probe GPU/VRAM → recommend local vs Comfy Cloud |
| `extract_schema.py` | List controllable params + model deps from workflow |
| `check_deps.py` | Check workflow against running server for missing nodes/models |
| `auto_fix_deps.py` | Auto-install missing nodes/models |
| `run_workflow.py` | Inject params, submit, monitor, download outputs |
| `run_batch.py` | Submit workflow N times with sweeps, parallel up to tier limit |
| `ws_monitor.py` | Real-time WebSocket viewer for executing jobs |
| `health_check.py` | Verification checklist: comfy-cli + server + models + smoke test |
| `fetch_logs.py` | Pull traceback/status messages for a given prompt_id |
**Known issues with bundled workflows (discovered 2026-07-22):**
1. **`_comment` string field crashes ComfyUI v0.27.0.** The bundled `sd15_txt2img.json` has a top-level `"_comment"` string field. ComfyUI v0.27.0's `execution.py:1121` expects `_meta` to be a dict and crashes with `AttributeError: 'str' object has no attribute 'get'` when it encounters a string. Fix: strip `_comment` before submitting. The `run_workflow.py` script does NOT strip this automatically — pre-process the workflow JSON first:
```python
import json
with open('workflow.json') as f: wf = json.load(f)
wf.pop('_comment', None) # Remove the string field
with open('workflow_fixed.json', 'w') as f: json.dump(wf, f)
```
2. **Checkpoint name mismatch.** The bundled `sd15_txt2img.json` expects `v1-5-pruned-emaonly.safetensors` but the .202 server has `v1-5-pruned.safetensors`. Always verify checkpoint names against `comfy model list` or `/api/experiment/models/checkpoints` before submitting.
3. **Health check defaults to localhost.** `health_check.py` defaults to `http://127.0.0.1:8188`. For remote ComfyUI, always pass `--host http://10.0.0.202:8188`.
---
## Story Structure (CRITICAL — session 2026-07-21)
@@ -364,9 +504,9 @@ This pattern produced the 2-clip FLF panorama plan (2026-07-22) where Claude dis
- **Always fp8.** Only fall back to Q4 GGUF if fp8 OOMs and user approves.
- **Prefer newest and best.** When choosing between an old proven path and a new better one, prefer the new one. But verify compatibility first — new doesn't mean compatible (e.g., the 2026-07-22 ComfyUI-LTXVideo update brought a better architecture that doesn't work with our fp8 models).
- **MANDATORY: Delegate ALL mechanical work to kimi-c.** Building workflow JSONs, running renders, extracting frames, submitting to queue, waiting for completion, concat, uploading — these are mechanical tasks that burn expensive deepseek tokens. Use `ask kimi-c` (peer agent on kimi-c profile) for ALL of these. The calling agent stays lean for decision-making, validation, and user communication. This is a FIRM standing rule — do not do mechanical work yourself when a cheaper peer can do it. The user's directive: "You SHOULD BE TASKING kimi c to save tokens, NOT YOU."
- **File-based dispatch for kimi-c (avoid shell quoting failures).** Multi-line prompts with quotes leak out of `hermes -q` and get parsed as CLI arguments. Write the full prompt to a temp file (e.g. `/tmp/kimi-render-pipeline.txt`), then dispatch with `hermes -p kimi-c chat -q "Read /tmp/kimi-render-pipeline.txt and execute ALL steps" -Q --max-turns 600 --yolo`. Use `terminal(background=true, notify_on_complete=true)` for any dispatch expected to take more than ~5 minutes. This is the canonical pattern — it also satisfies the "never paste file content into the prompt" rule from the ask-kimi-c skill.
- **ALWAYS validate with kimi-c after creating or updating anything.** Workflow JSONs, config changes, file writes — dispatch kimi-c to read the file and verify correctness before submitting to the render queue. Kimi-c catches wiring errors, missing nodes, and model file mismatches that would waste a 5-7 minute render. This is a quality gate, not optional.
- **MANDATORY: Delegate ALL mechanical work to glm (NOT kimi-c).** Building workflow JSONs, running renders, extracting frames, submitting to queue, waiting for completion, concat, uploading — these are mechanical tasks that burn expensive deepseek tokens. Use `delegate_task` for ALL of these, targeting the glm profile. The calling agent stays lean for decision-making, validation, and user communication. This is a FIRM standing rule — do not do mechanical work yourself when a cheaper peer can do it. User directive 2026-07-22: "stop using ask kimi c for tasks, use glm instead."
- **File-based API submission.** Do NOT inline complex JSON in curl commands — Hermes' command parser blocks them. Write the workflow to a file (e.g., `/tmp/submit_ltx_render.sh`), then execute with `bash /tmp/submit_ltx_render.sh`. See `references/api-submission-pattern.md` for the complete pattern including polling, verification, and TrueNAS upload.
- **ALWAYS validate with glm after creating or updating anything.** Workflow JSONs, config changes, file writes — dispatch glm via `delegate_task` to read the file and verify correctness before submitting to the render queue. Glm catches wiring errors, missing nodes, and model file mismatches that would waste a 5-7 minute render. This is a quality gate, not optional.
- **Test one scene before rendering all 6.**
- **Verify frame count after every render (ffprobe).**
- **Save all artifacts to TrueNAS immediately** (user can only view videos on TrueNAS, not in LXCs).
@@ -422,6 +562,8 @@ curl -s -X POST http://localhost:8188/prompt -H "Content-Type: application/json"
- Modify: start_frame path, audio path, segment prompt
- For Transition LoRA: add `LoraLoaderModelOnly` node, rewire chain, append `zhuanchang` to prompts
- Save to `/tmp/api_scene_XX.json`
- **To modify and re-render existing workflows** (change sampler, duration, etc. without rebuilding): see `references/modify-and-re-render-recipe.md` — proven pattern with Python in-place editing, verification, and submission.
- **To programmatically build multiple clip workflows from templates** (common modifications across C1/C2 with per-clip overrides): see `references/programmatic-workflow-modification.md` — scp a Python script to .202, avoids shell heredoc quoting issues.
### 4. Submit to Queue
```bash
@@ -440,6 +582,34 @@ sshpass -p 'passw0rd' ssh [email protected] "while true; do q=\$(curl -s http://loc
```
### 6. Identify Output Files
**Preferred: Use ComfyUI history API (reliable, works regardless of output directory).** After queue drains, query the history for each prompt_id to get exact filenames:
```bash
# Get output filename from prompt_id
curl -s http://localhost:8188/history/<prompt_id> | python3 -c "
import json,sys
d=json.load(sys.stdin)
entry=d.get('<prompt_id>',{})
outputs=entry.get('outputs',{})
node37=outputs.get('37',{})
images=node37.get('images',[])
for img in images:
print(img.get('filename','?'))
"
```
**Fallback: Search both output directories.** Outputs land in `~/comfy-ui/output/` or `~/comfy-ui/output/video/` depending on the workflow's `filename_prefix`. Use `find` with a reference file timestamp:
```bash
# Find all mp4s newer than a known reference file
find ~/comfy-ui/output -name "*.mp4" -newer /tmp/api_2clip_C1_v2.json -type f
# Or list by modification time
ls -lt ~/comfy-ui/output/*.mp4 ~/comfy-ui/output/video/*.mp4 2>/dev/null | head -10
```
For metadata-based identification (when outputs have predictable prefixes):
```bash
python3 << 'PYEOF'
import subprocess, os, re
@@ -487,6 +657,8 @@ All stock materials (start frames, audio, character refs, workflows, outputs) ar
- FFHQ — 70K faces at 1024×1024 (NVIDIA, non-commercial)
- CivitAI — community LoRAs and LTX workflows
**Using stock as I2V start frames:** See `references/stock-as-i2v-start-frame.md` — pattern for using character stock images as direct I2V start frames without IC-LoRA Ingredients. Simpler, works with existing 6-fix baseline. For multi-clip character consistency, use IC-LoRA Ingredients instead.
## Pitfalls
- **CRITICAL: True-extend (extract last frame → use as next clip start) carries latent memory.** LTX remembers subjects from prior clips even when the start frame shows empty space. Confirmed 2026-07-22: Clip 2 hallucinated a second woman stepping up from the couch where the man was in Clip 1. Fix: use the pre-rendered empty mid-room frame B as Clip 2's start frame instead of the extracted last frame. Clean start, no latent memory. See §2-Clip Test Pattern.
@@ -499,7 +671,9 @@ All stock materials (start frames, audio, character refs, workflows, outputs) ar
- **`zhuanchang` trigger word is required** for Transition LoRA to activate. Append to end of segment prompts.
- **CFG must be 1.0** for distilled model. Higher values cause artifacts.
- **Queue is sequential** — submitting 6 scenes means ~30-40 min total. Use the blocking wait loop from step 5.
- **`process(action='wait')` timeout is clamped to 60s regardless of what you request.** The configured limit on the Hermes host caps wait timeouts at 60 seconds. For renders that take 5-7 minutes each, a single `process(action='wait', timeout=900)` will time out after 60s with the process still running. Pattern: use `process(action='wait', timeout=60)` in a loop, checking `status` after each call. When status is still `"running"`, call wait again. When status is `"exited"`, the queue has drained. Confirmed 2026-07-22: 2-clip render took 3 wait cycles (~3 min total) despite requesting 900s.
- **Output files accumulate** — the Director workflow produces 2 outputs per scene (guide pass + main pass). Identify the final output by checking for `LoraLoaderModelOnly` in metadata (Transition batch) or by resolution (512×512 = fp8 batch).
- **Output directory varies by workflow prefix.** The `SaveImage`/`SaveVideo` node's `filename_prefix` determines where files land. Workflows with prefix `LTX_Director_` go to `~/comfy-ui/output/video/`. Workflows with custom prefixes (e.g., `LTX_2clip_test_C1`) go to `~/comfy-ui/output/` directly. When identifying new outputs, search both directories: `find ~/comfy-ui/output -name "*.mp4" -newer <reference_file> -type f`. Do NOT assume all outputs are in `video/`.\n- **CRITICAL: Template file naming — verify you're using the right base files before re-rendering.** The .202 `/tmp/` directory accumulates multiple generations of workflow JSONs with confusingly similar names: `api_2clip_C1.json` (old living room test), `api_cyberpunk_C1.json` (cyberpunk test), `api_2clip_01.json` (earlier living room variant). When modifying and re-rendering, ALWAYS inspect the timeline_data segment's `imageFile` and `prompt` fields to confirm you're working with the correct template. A one-line Python check: `python3 -c "import json; d=json.load(open('/tmp/api_XXX.json')); td=json.loads(d['prompt']['131']['inputs']['timeline_data']); print(td['segments'][0]['imageFile'], td['segments'][0]['prompt'][:80])"`. Confirmed 2026-07-22: pointed delegation at wrong template, produced living room output instead of cyberpunk.
- **cifs kernel module not available** on Hermes host — use smbclient for TrueNAS, not mount.cifs.
- **smbclient is NOT installed on .202** — scp files to Hermes host first, then upload to TrueNAS from there. Do not try to install smbclient on .202 (no root access).
- **State file is the crash-recovery artifact** — update `~/workspace/general/ltx-pipeline-state.md` as you go. A new session reads it to resume.