tools-update-cron: sync 2026-08-09 — 37 skill(s) updated
This commit is contained in:
Hermes Agent
@@ -0,0 +1,180 @@
|
||||
---
|
||||
name: dgx-comfyui-setup
|
||||
description: Use when setting up or migrating ComfyUI on DGX Spark.
|
||||
version: 1.0.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
platforms: [linux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [dgx, spark, comfyui, install, migration, gb10, aarch64]
|
||||
related_skills: [deep-research, ask-claude, ai-brain-kb]
|
||||
---
|
||||
|
||||
# DGX Spark ComfyUI Setup
|
||||
|
||||
Complete workflow for installing or migrating ComfyUI on NVIDIA DGX Spark (GB10 Grace Blackwell, sm_121a, 128GB unified LPDDR5x, aarch64).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- DGX Spark running DGX OS (Ubuntu 24.04 aarch64)
|
||||
- Docker with NVIDIA Container Toolkit (ships with DGX OS)
|
||||
- SSH access: adm1n@10.0.0.6, password passw0rd, sudo passw0rd
|
||||
- ~25 GB disk before models
|
||||
|
||||
## Workflow (MANDATORY ORDER)
|
||||
|
||||
### Phase 1: Research First
|
||||
|
||||
Always dispatch deep research before touching the box. The DGX ecosystem moves fast — what was best last month may be superseded.
|
||||
|
||||
```
|
||||
deep-research: "best way to run ComfyUI on DGX Spark as of [DATE]"
|
||||
deep-research: "full install instructions, OS/firmware prerequisites, pitfall fixes"
|
||||
deep-research: "is [chosen method] still the best? any newer alternatives?"
|
||||
```
|
||||
|
||||
### Phase 2: Step-by-Step Install
|
||||
|
||||
Execute ONE step at a time. Verify each step before proceeding. Do NOT batch steps.
|
||||
|
||||
1. **Stop and remove old containers/images** — `docker stop/rm`, `docker rmi`, `docker image prune -f`
|
||||
2. **Clone spark-comfyui** — `git clone https://github.com/bjarkebolding/spark-comfyui.git`
|
||||
3. **Build image** — `./spark-comfyui.sh install` (10-30 min, background with notify)
|
||||
4. **Configure mounts** — edit `spark-mounts.conf` to point at existing workspace or use defaults
|
||||
5. **System tuning** — `./spark-comfyui.sh tune --persist` (swap off, persistence mode, clock cap if needed)
|
||||
6. **Start as service** — `./spark-comfyui.sh service` (auto-start on boot, restart unless-stopped)
|
||||
7. **Fix permissions** — `chown -R 1000:1000` on workspace if migrating from aeon-spark (was root:root)
|
||||
8. **Verify imports** — `docker logs spark-comfyui | grep "IMPORT FAILED"` — must be zero
|
||||
9. **Run doctor** — `./spark-comfyui.sh doctor` — all gates must pass
|
||||
|
||||
### Phase 3: Fix Common Pitfalls
|
||||
|
||||
These failures are predictable. Check them proactively.
|
||||
|
||||
| Pitfall | Symptom | Fix |
|
||||
|---------|---------|-----|
|
||||
| libGL.so.1 not found | LTXVideo, VideoHelperSuite, RES4LYF, Easy-Use import fail | Dockerfile needs `libgl1` (NOT `libgl1-mesa-glx` — doesn't exist on Ubuntu 24.04) |
|
||||
| interleaved_freqs_cis missing | ComfyUI-LTXVideo import fail | `git pull` LTXVideo to commit 548a393 or newer ("Support core rope change") |
|
||||
| soundfile missing | comfyui-various import fail | Create `requirements.txt` with `soundfile` in the node dir, or bake into Dockerfile |
|
||||
| No __init__.py | ComfyUI_LTX-2_VRAM_Memory_Management import fail | Remove the node — obsolete, superseded by spark-comfyui mod 10-unified-memory-free |
|
||||
| Permission denied on comfyui.db | DB init error in logs | `chown -R 1000:1000` on workspace (aeon-spark left files as root:root) |
|
||||
| --enable-manager redundant | No error, just unnecessary | Remove from entrypoint.sh — ComfyUI core loads built-in Manager |
|
||||
| bf16-vae crashes audio VAE | `Input type (float) and bias type (c10::BFloat16) should be the same` during audio encoding | Remove `--bf16-vae` from entrypoint.sh SPARK_BF16 flags. Audio VAE conv layers have float32 bias. Only `--bf16-unet --bf16-text-enc` should remain. |
|
||||
|
||||
### Phase 4: Validate with Claude
|
||||
|
||||
After install is verified (doctor green, zero import failures), dispatch Claude for adversarial validation:
|
||||
|
||||
```
|
||||
scp validation prompt to 10.0.0.28
|
||||
ask-claude: validate the install — find flaws, not propose new features
|
||||
```
|
||||
|
||||
Claude's common findings:
|
||||
- Drop `--enable-manager` (redundant)
|
||||
- Bake runtime deps into Dockerfile (libgl1, soundfile)
|
||||
- Verify models mount resolves (curl CheckpointLoaderSimple)
|
||||
- Check service isn't double-managed (Docker restart + systemd)
|
||||
|
||||
### Phase 5: Apply Fixes + Rebuild
|
||||
|
||||
Apply Claude's fixes, rebuild image, restart service:
|
||||
|
||||
```
|
||||
./spark-comfyui.sh stop
|
||||
./spark-comfyui.sh update # rebuilds with cached layers
|
||||
./spark-comfyui.sh service
|
||||
```
|
||||
|
||||
### Phase 6: Persist Knowledge
|
||||
|
||||
After everything is working:
|
||||
- **ai_brain_kb**: add full setup document with `mcp__better_qdrant__add_documents`
|
||||
- **fact_store**: add key facts (stack versions, commands, pitfalls, update procedure)
|
||||
- **fact_store**: add Dockerfile fixes as pitfall facts for future sessions
|
||||
|
||||
## Key Commands
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `./spark-comfyui.sh status --watch` | Live dashboard |
|
||||
| `./spark-comfyui.sh doctor` | Health check with live GPU gates |
|
||||
| `./spark-comfyui.sh update` | Rebuild image (cached layers) |
|
||||
| `./spark-comfyui.sh update --rollback` | Revert to previous image |
|
||||
| `./spark-comfyui.sh backup` | Backup workflows, settings, custom nodes |
|
||||
| `./spark-comfyui.sh restore FILE` | Restore from backup |
|
||||
| `./spark-comfyui.sh stop` | Stop container |
|
||||
| `./spark-comfyui.sh service --disable` | Disable auto-start |
|
||||
| `docker logs -f spark-comfyui` | Live container logs |
|
||||
|
||||
## Update Procedure
|
||||
|
||||
- `./spark-comfyui.sh update` rebuilds image on current ComfyUI master
|
||||
- Cached layers: torch + SageAttention stages stay cached, only ComfyUI + custom nodes rebuild
|
||||
- Old image saved as `spark-comfyui:previous` for instant rollback
|
||||
- Custom node requirements installed at container start from `requirements.txt` files
|
||||
- Torch pinned to cu130 aarch64 — cannot be corrupted by custom nodes
|
||||
- Entrypoint auto-repairs torch before every launch
|
||||
|
||||
## DGX Spark Access
|
||||
|
||||
- Host: 10.0.0.6, hostname: edgexpert-7d04
|
||||
- SSH: `sshpass -p 'passw0rd' ssh -o StrictHostKeyChecking=no [email protected]`
|
||||
- Sudo: `echo passw0rd | sudo -S <cmd>`
|
||||
- Repo: `/home/adm1n/spark-comfyui/`
|
||||
- Workspace: `/home/adm1n/comfy/workspace/`
|
||||
- ComfyUI: `http://10.0.0.6:8188`
|
||||
|
||||
## DGX Model Inventory
|
||||
|
||||
Full current inventory of all models on the DGX, including sizes, paths, and known gaps vs latest available: see `references/dgx-inventory.md`. Update this file after any model add/remove/upgrade session.
|
||||
|
||||
## Adding New Models or Workflow Changes
|
||||
|
||||
When adding a new model (VAE, LoRA, checkpoint, etc.) or making any workflow change on the DGX:
|
||||
|
||||
1. **Download the model** — `wget` from HuggingFace into the correct `models/` subdirectory.
|
||||
2. **Create a NEW workflow file** — copy the working workflow JSON, rename it (e.g. `DirectorWorking1_PrunaVAE.json`), and modify only the changed node/widget. Never edit the original.
|
||||
3. **Verify** — check the new workflow loads without import errors.
|
||||
|
||||
**CRITICAL: Never modify or overwrite original working saved workflows.** Preserve known-good workflows as-is. All additions, model swaps, node changes, or experiments go into a new workflow file. The original is the fallback.
|
||||
|
||||
## Don't Do
|
||||
|
||||
- Don't skip research — the ecosystem changes fast
|
||||
- Don't batch steps — one at a time, verify each
|
||||
- Don't use `libgl1-mesa-glx` — doesn't exist on Ubuntu 24.04
|
||||
- Don't leave old LTXVideo — must be at commit 548a393 or newer
|
||||
- Don't skip Claude validation — catches things you'll miss
|
||||
- Don't forget to persist knowledge to brain + fact store
|
||||
- Don't modify original working workflows — always create a copy
|
||||
- Don't assume the DGX workflow structure differs from .202 — both use the same node types and pattern (1 EmptyLatent, 1 LTXVConditioning, 8 LoadImage, 9 VHS). Cross-environment porting is parameter-only, not restructuring.
|
||||
- Don't hand-pick change categories when porting between workflows — diff the source delta first. Compute `base → modified` as a field-level widget diff, then apply that exact delta to the target. Hand-picking misses parameters (per-scene prompts, seeds, lengths, audio trims).
|
||||
- Don't treat `LTXVSequenceParallelMultiGPUPatcher` as functional on DGX — the GB10 is a single-GPU system. The node is dead weight (won't error, won't split anything). Remove it in new workflow copies.
|
||||
- Don't assume bf16 workflows complete on DGX Spark. The LTXAV model at bf16 stages 40GB, and the final VHS VideoCombine step loads all rendered frames back into memory for concatenation. On the 128GB unified DGX, this can exhaust memory and crash the container silently (no OOM error, just a restart). The individual scene renders complete fine — it's the final combine that fails. Mitigations: (a) skip the final combine VHS node and output per-scene files only, (b) reduce resolution, or (c) use fp8 models instead of bf16 for the combine step.
|
||||
- Don't upload the wrong VHS output variant. VHS VideoCombine with audio produces TWO files per render: `prefix_0000N.mp4` (video-only) and `prefix_0000N-audio.mp4` (video + audio muxed). Always use the `-audio` variant when shipping to TrueNAS. Verify with `ffprobe` before uploading.
|
||||
- Don't upload individual scene files when the user wants the final video. The VHS FINAL VIDEO node concatenates all scenes into one file. Upload only that one file (the `-audio` variant) unless the user explicitly asks for per-scene outputs.
|
||||
- Don't assume a render failed when the queue is empty. Check `docker logs | grep -c 'got prompt'` to confirm a prompt was ever submitted. An empty queue with zero prompts means the workflow was never queued — not that it crashed. The DGX ComfyUI-Manager channel warning and Crystools GPU error are non-fatal startup noise, not render failures.
|
||||
- Don't assume higher precision (bf16 vs fp8) fixes LTX-2.3 temporal issues. Lightning smearing (rapid on/off events), identity drift (bald→hair), and action sequencing delays (mic appears then rapping starts) are architectural limitations of video diffusion models, not precision artifacts. bf16 produces slightly cleaner fine detail but the same behavioral issues. IC-LoRA Ingredients helps with identity consistency. Shorter prompts with fewer simultaneous actions help with sequencing. See `references/ltx23-temporal-issues.md`.
|
||||
|
||||
For concrete examples of these pitfalls in action (prompt location, hand-picked categories, structural assumptions, dual-validation pattern, bf16 combine crash, VHS audio variants), see `references/workflow-porting-pitfalls.md`.
|
||||
|
||||
For LTX-2.3 temporal issues that higher precision does NOT fix (lightning smearing, identity drift, action sequencing) and the IC-LoRA fix, see `references/ltx23-temporal-issues.md`.
|
||||
|
||||
For a complete IC-LoRA guide (how it works, available models, character sheets, proven settings, prompt format), see `references/ic-lora-guide.md`.
|
||||
|
||||
## Cross-Environment Mirroring (.202 → DGX)
|
||||
|
||||
When the user wants the DGX to match the .202 LXC setup at full precision:
|
||||
|
||||
1. **Inventory BOTH sides first.** Most "upgrades" are already present — the DGX often already has full-precision variants. Use `find` by model type on both hosts.
|
||||
2. **Only copy what's genuinely missing.** Don't copy everything someone assumes is missing.
|
||||
3. **Verify LoRA paths.** The DGX may have LoRAs in subdirectories (`loras/ltxv/ltx2/`) while .202 has them flat (`loras/`). Use symlinks to create flat aliases — avoids duplicating 7GB+ files.
|
||||
4. **Check cond-safe LoRA presence.** Director workflows may depend on the cond-safe variant specifically.
|
||||
5. **Copy custom nodes that are genuinely missing.** Git clone into the workspace `custom_nodes/` directory. For nodes without a public repo (like SeqParallel V5), scp the .py files directly.
|
||||
6. **Copy the working workflow JSON** from .202 `user/default/workflows/` to DGX.
|
||||
7. **Rebuild Docker image** — `./spark-comfyui.sh stop && ./spark-comfyui.sh update && ./spark-comfyui.sh service`. Cached layers make this fast (~10 min).
|
||||
8. **Verify** — `docker logs spark-comfyui | grep "IMPORT FAILED"` must be zero.
|
||||
|
||||
**Known delta (2026-07-29):** IC-LoRA Ingredients, cond-safe LoRA, Cinemagraph, Transition, SeqParallel V5, MelBandRoFormer, and comfyui-ollama were on .202 but not DGX. Full-precision models (dev 43GB, Gemma 3 12B 23GB) were already present.
|
||||
Reference in New Issue
Block a user