hermes-skills/dgx-spark-comfyui/SKILL.md

---
name: dgx-spark-comfyui
description: Install, migrate, maintain ComfyUI on DGX Spark (GB10).
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [dgx-spark, comfyui, gb10, aarch64, unified-memory, sm_121a, infrastructure]
    related_skills: [local-ai-media-generation, proxmox-lxc-deployment]
---

# DGX Spark ComfyUI — Install, Migrate, Maintain

## Overview

The DGX Spark (GB10 Grace Blackwell, sm_121a, 128GB unified LPDDR5x, 3.7TB NVMe, aarch64) is a capable ComfyUI host but its unusual architecture requires specific configuration. The community has converged on **spark-comfyui** (bjarkebolding) as the standard — one script, fully containerized, self-healing, actively maintained.

**Current best option (July 2026):** [spark-comfyui](https://github.com/bjarkebolding/spark-comfyui) — 52 commits, last updated July 27, 2026. NVIDIA forum thread: 19 likes, 848 views.

**Our DGX Spark:** 10.0.0.6, hostname edgexpert-7d04, SSH `[email protected]` (passw0rd), sudo passw0rd.

## Fresh Install (spark-comfyui)

```bash
# 1. Clone
git clone https://github.com/bjarkebolding/spark-comfyui.git
cd spark-comfyui

# 2. Build image (10-30 min)
./spark-comfyui.sh install

# 3. System tuning (swap off, persistence mode, optional clock cap)
./spark-comfyui.sh tune --persist

# 4. If overcurrent crashes occur, cap GPU clock:
./spark-comfyui.sh tune --clock-cap 2100 --persist

# 5. Run as service (auto-start on boot, survives crashes)
./spark-comfyui.sh service

# 6. Verify
./spark-comfyui.sh doctor
```

Models go in `data/models/checkpoints/` (etc.). No venv, no system Python changes.

## Migration from aeon-spark

### Step-by-step

1. Stop and remove old container + images:
   ```bash
   docker stop comfyui-spark && docker rm comfyui-spark
   docker rmi ghcr.io/aeon-7/comfyui-aeon-spark:latest
   docker image prune -f
   ```

2. Clone spark-comfyui and install (see Fresh Install above).

3. Configure `spark-mounts.conf` to reuse existing workspace:
   ```
   models = /home/adm1n/comfy/workspace/models
   custom_nodes = /home/adm1n/comfy/workspace/custom_nodes
   user = /home/adm1n/comfy/workspace/user
   output = /home/adm1n/comfy/workspace/output
   input = /home/adm1n/comfy/workspace/input
   ```

4. Fix permissions — aeon-spark wrote files as root, spark-comfyui runs as UID 1000:
   ```bash
   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/user/
   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/custom_nodes/
   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/models/
   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/output/
   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/input/
   ```

5. Remove obsolete custom node (no `__init__.py`, superseded by unified memory mod):
   ```bash
   rm -rf /home/adm1n/comfy/workspace/custom_nodes/ComfyUI_LTX-2_VRAM_Memory_Management
   ```

6. Tune and start service.

**Downtime:** ~30 minutes (image build + model copy time). Models, custom nodes, workflows, and user data are all reused — only the Docker image is rebuilt.

## Dockerfile Patch for libGL
## Dockerfile Patch for libGL
The spark-comfyui Dockerfile installs `libx11-6 libxext6 libxcb1` but NOT `libgl1`. This causes import failures for LTXVideo, VideoHelperSuite, RES4LYF, and Easy-Use (`ImportError: libGL.so.1: cannot open shared object file`).

**Fix:** Add `libgl1 libglib2.0-0` to both apt-get install lines in `container/Dockerfile`. Do NOT use `libgl1-mesa-glx` — that package was removed in Ubuntu 24.04 and has no installation candidate.

```bash
sed -i 's/libx11-6 libxext6 libxcb1/libx11-6 libxext6 libxcb1 libgl1 libglib2.0-0/' container/Dockerfile
```

Then rebuild: `./spark-comfyui.sh update`

## Common Import Failures and Fixes

| Failure | Cause | Fix |
|---------|-------|-----|
| `libGL.so.1: cannot open shared object file` | Missing mesa libs in container | Patch Dockerfile to add `libgl1 libglib2.0-0`, rebuild |
| `No module named 'soundfile'` | Missing pip package, no requirements.txt in custom node | Create `requirements.txt` with `soundfile` in the node dir |
| `cannot import name 'interleaved_freqs_cis'` | LTXVideo custom node stale vs ComfyUI core | `git pull origin master` in ComfyUI-LTXVideo dir |
| `ComfyUI_LTX-2_VRAM_Memory_Management` no `__init__.py` | Obsolete node, superseded by unified memory mod | `rm -rf` the directory |
| `comfyui.db.lock` Permission denied | Root-owned files from aeon-spark | `chown -R 1000:1000` on workspace dirs |
| Impact Pack / controlnet_aux import failed | Missing dependencies | Acceptable — non-critical nodes |

## Manager Fixes

### Manager invisible in UI ("Blocked by policy")

The pip package `comfyui-manager` conflicts with the git clone at `custom_nodes/ComfyUI-Manager`. The pip package's `should_be_disabled()` returns True for any dir containing "comfyui-manager", blocking the git clone's `WEB_DIRECTORY = "js"`.

**Fix:** `pip uninstall comfyui-manager -y`. The git clone is then unblocked and Manager appears in the UI bar.

### Manager: "Invalid channel" / empty node list

The git clone may point to the OLD `ltdrdata/ComfyUI-Manager` repo. The repo moved to `Comfy-Org/ComfyUI-Manager`. Three fixes needed:

1. **Git remote:** `git remote set-url origin https://github.com/Comfy-Org/ComfyUI-Manager.git`
2. **channels.list / channels.list.template:** Replace `ltdrdata/ComfyUI-Manager` with `Comfy-Org/ComfyUI-Manager`
3. **Hardcoded DEFAULT_CHANNEL in Python:** Replace in `glob/manager_core.py` and `glob/manager_server.py`

### --enable-manager flag is redundant

ComfyUI core already loads the built-in Manager. The `--enable-manager` flag in the entrypoint is unnecessary. Remove it from `container/entrypoint.sh` DEFAULT_FLAGS.

## Dockerfile Changes (Baked into Image)

The spark-comfyui Dockerfile at `container/Dockerfile` has these local modifications:
1. `libgl1-mesa-glx` → `libgl1` (Ubuntu 24.04 compat — mesa-glx removed in 24.04)
2. `pip install soundfile` after `manager_requirements.txt`
3. `--enable-manager` removed from entrypoint.sh DEFAULT_FLAGS (redundant)

## Entrypoint Changes

The spark-comfyui entrypoint at `container/entrypoint.sh` has this local modification:
1. `--bf16-vae` removed from SPARK_BF16 default flags (2026-07-29) — the audio VAE conv layers have float32 bias that crashes with `Input type (float) and bias type (c10::BFloat16) should be the same`. Only `--bf16-unet --bf16-text-enc` remain.

These survive `./spark-comfyui.sh update` because they're in the repo's tracked files. If the upstream files change significantly, re-apply after `git pull`.

## Key Commands

| Command | Purpose |
|---------|---------|
| `./spark-comfyui.sh install` | Build image |
| `./spark-comfyui.sh run` | Foreground |
| `./spark-comfyui.sh service` | Background, auto-restart, survives reboots |
| `./spark-comfyui.sh stop` | Stop container |
| `./spark-comfyui.sh update` | Rebuild image on current ComfyUI master |
| `./spark-comfyui.sh update --rollback` | Swap back to previous image |
| `./spark-comfyui.sh doctor` | Live GPU gates in throwaway container |
| `./spark-comfyui.sh status --watch` | Live sparkline dashboard |
| `./spark-comfyui.sh tune --persist` | System tuning (swap off, persistence mode) |
| `./spark-comfyui.sh tune --clock-cap 2100 --persist` | Cap GPU clock to prevent overcurrent crashes |
| `./spark-comfyui.sh backup` | Backup workflows, settings, custom nodes |
| `./spark-comfyui.sh restore FILE` | Rebuild from backup |

## Workflow-Driven Model Download

When the user says "download all required files for workflow X," do NOT guess or search manually. Extract dependencies from the workflow JSON, inventory what's already on the DGX, then download only what's missing — in parallel.

### Step-by-step

1. **Read the workflow JSON** — `cat /home/adm1n/comfy/workspace/user/default/workflows/{name}.json`
2. **Extract all model references** — parse `widgets_values` from Loader nodes (VAELoader, DualCLIPLoader, UNETLoader, LoraLoader, LoadImage, etc.). Also check subgraph nodes (UUID-type nodes) — they embed model filenames in their widget values.
3. **Extract input images** — `LoadImage` nodes reference files in `input/`.
4. **Check what's already on the DGX** — `ls -la` each model directory for the expected filenames.
5. **Find download URLs** — workflow MarkdownNote nodes often contain HuggingFace links. If not, search `huggingface.co/{filename}` to find the repo. Use HF API (`https://huggingface.co/api/models/{org}/{repo}`) to get file sizes.
6. **Download in parallel** — use `terminal(background=true, notify_on_complete=true)` for each file. Large models (20GB+) go to `diffusion_models/`, LoRAs to `loras/`, VAEs to `vae/`, text encoders to `text_encoders/`.
7. **Copy input images** — if the workflow references images from .202, use two-hop pipe: `ssh .202 "cat /path/file" | ssh DGX "cat > /home/adm1n/comfy/workspace/input/file"`. Do NOT try direct scp from the Hermes host — the files live on .202, not locally.

### Example: qwenI2I.json + qwenT2I.json (2026-07-30)

Workflow JSONs at `user/default/workflows/qwenI2I.json` and `qwenT2I.json`.

**qwenI2I.json needs:**
| File | Type | Dir | Status |
|------|------|-----|--------|
| `qwen_image_edit_2511_fp8mixed.safetensors` | diffusion model | `diffusion_models/` | Download from Comfy-Org/Qwen-Image-Edit_ComfyUI |
| `qwen_2.5_vl_7b_fp8_scaled.safetensors` | text encoder | `text_encoders/` | Already present |
| `qwen_image_vae.safetensors` | VAE | `vae/` | Already present |
| `Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors` | LoRA | `loras/` | Already present |
| `August-2017-4643-2.jpg` | input image | `input/` | Copy from .202 |
| `Qwen-Image-2512_00007_.png` | input image | `input/` | Copy from .202 |
| `RobLTX.png` | input image | `input/` | Copy from .202 |

**qwenT2I.json needs:**
| File | Type | Dir | Status |
|------|------|-----|--------|
| `qwen_image_2512_fp8_e4m3fn.safetensors` | diffusion model | `diffusion_models/` | Download from Comfy-Org/Qwen-Image_ComfyUI |
| `qwen_2.5_vl_7b_fp8_scaled.safetensors` | text encoder | `text_encoders/` | Already present |
| `qwen_image_vae.safetensors` | VAE | `vae/` | Already present |
| `Qwen-Image-2512-Lightning-4steps-V1.0-fp32.safetensors` | LoRA | `loras/` | Download from lightx2v/Qwen-Image-2512-Lightning |

### Two-Hop File Transfer (input images from .202)

The Hermes host cannot directly scp from .202 to DGX. Pipe through the Hermes host:

```bash
sshpass -p 'n8n' ssh -o StrictHostKeyChecking=no [email protected] \
  "cat /home/n8n/comfy-ui/input/FILENAME" | \
  sshpass -p 'passw0rd' ssh -o StrictHostKeyChecking=no [email protected] \
  "cat > /home/adm1n/comfy/workspace/input/FILENAME"
```

Run each image transfer in its own background process for parallelism.

### Subgraph Nodes (UUID-type)

Workflows from Comfy-Org templates use subgraph nodes with UUID type identifiers (e.g. `cdb2cf24-c432-439b-b5c8-5f69838580c9`). These embed model filenames directly in their `widgets_values` array — parse them the same way as regular loader nodes. The subgraph's internal nodes are stored in the workflow's `extra` field.

### Max Precision Policy

**Always check HuggingFace for higher-precision versions of every model referenced in a workflow.** Download the max-precision variant to disk alongside the workflow's fp8 reference. Do NOT modify the workflow JSON — keep it pointing at the fp8 filename. The user can switch to max precision later by editing the workflow themselves.

For each model in a workflow, check the same HF repo for bf16/fp16/fp32 variants:
- `qwen_image_edit_2511_fp8mixed` → also download `qwen_image_edit_2511_bf16` (39GB)
- `qwen_image_2512_fp8_e4m3fn` → also download `qwen_image_2512_bf16` (41GB)
- `qwen_2.5_vl_7b_fp8_scaled` → also download `qwen_2.5_vl_7b.safetensors` (16.6GB)
- Lightning LoRAs are already fp32 — no higher precision exists
- VAE has only one version (254MB) — no higher precision exists

Use the HF API to discover available files: `https://huggingface.co/api/models/{org}/{repo}` → check `siblings` for bf16/fp16/fp32 variants. Download in parallel with the fp8 versions.

## Cross-Environment Model Parity

When someone says "make the DGX match the .202 setup," verify before acting. The DGX (Docker, aarch64, unified memory) and .202 (bare-metal LXC, x86_64, discrete VRAM) have different environments. A .202 workflow JSON won't load on DGX without path adjustments — node IDs, model paths, and custom node versions differ.

**Checklist before copying models:**
1. Inventory BOTH sides first (`find` by model type)
2. Most "upgrades" are already present — the DGX often already has full-precision variants
3. Only copy what's genuinely missing, not what someone assumes is missing
4. Verify LoRA paths match what the workflow JSON expects (subdirectories like `loras/ltxv/ltx2/` vs `loras/`)
5. Check cond-safe LoRA presence — Director workflows may depend on it

**Full mirroring procedure (when user explicitly wants .202 parity at full precision):**
1. Inventory both sides: `find models/ -iname '*ltx*'` and `ls custom_nodes/` on both hosts
2. Copy missing LoRAs via scp (two-hop: .202 → local → DGX)
3. Fix LoRA path mismatches with symlinks (e.g. `ln -s loras/ltxv/ltx2/file.safetensors loras/file.safetensors`) — avoids duplicating 7GB+ files
4. Git clone missing custom nodes into workspace `custom_nodes/`; for nodes without public repos (like SeqParallel V5), scp the .py files directly
5. Copy the working Director workflow JSON from .202 `user/default/workflows/` to DGX
6. Rebuild Docker: `./spark-comfyui.sh stop && ./spark-comfyui.sh update && ./spark-comfyui.sh service`
7. Verify: `docker logs spark-comfyui | grep "IMPORT FAILED"` must be zero

**Known delta (2026-07-29):** IC-LoRA Ingredients, cond-safe LoRA, Cinemagraph, Transition, SeqParallel V5, MelBandRoFormer, and comfyui-ollama were on .202 but not DGX. Full-precision models (dev 43GB, Gemma 3 12B 23GB) were already present.

**Model inventory:** See `references/dgx-model-inventory.md` for the full current snapshot.

**Workflow port + validation checklist:** See `references/workflow-port-validation.md` for the step-by-step procedure to copy a workflow from .202 to DGX with byte-for-byte verification, input file transfer, and independent validation.

**Multi-agent validation pattern:** See `references/multi-agent-validation.md` for dispatching Claude + Grok + self in parallel to validate a workflow before queuing.

**Max-quality upgrade pattern:** See `references/rapvidfinal-quality-analysis.md` for the ranked bang-for-buck analysis pattern — dispatch multiple agents to analyze a workflow, rank quality improvements, apply top 2-3 as a low-risk package.

**Hours-capable workflow pattern:** See `references/hours-capable-workflow.md` for converting multi-scene workflows from single-concat to per-scene file output, removing the ImageBatchMulti RAM wall for arbitrarily long videos.

**Diff-based workflow port pattern:** See `references/diff-based-workflow-port.md` for porting parameter changes between workflows with field-level widget diffs — prevents missing changes and editing wrong nodes. Validated by Claude + Grok on the RapVidFinal → FullHqHours port (2026-07-30).

## Pitfall: Don't Pull Files When Dispatching Validators

When the user says "ask X to SSH in and validate workflow Y," do NOT pull the workflow files locally first. The agents (Grok, Claude) should SSH directly into the DGX and read the files themselves. Pulling files locally wastes time and defeats the purpose of independent validation — the agents need to verify the files exist at the expected paths on the actual host.

**Wrong:** Pull JSONs to /tmp, scp to agent hosts, paste inline.
**Right:** Write a prompt telling the agent to `sshpass -p 'passw0rd' ssh adm1n@10.0.0.6` and `cat` the files directly.

## Multi-Scene OOM on DGX Spark — Two Failure Modes

The DGX Spark (128GB unified memory) can OOM during multi-scene LTX video workflows. There are TWO distinct failure modes — knowing which one hit determines the fix.

### Failure Mode 1: Mid-Render OOM (full-precision models, 6+ scenes)

**Symptoms:** Container is OOM-killed during scene rendering. Sampling progress bars show completions but NO per-scene output files (`DGXmaxq_Scene*.mp4`) exist — only generic `LTX_00001.*` temp files. Journal shows `python invoked oom-killer` with 110GB+ anon-rss.

**Root cause:** Full-precision stack (43GB dev + 23GB Gemma = 66GB base) leaves only ~62GB for decoded frames across all scenes. With 8 scenes at 768×448×121 frames each, memory exhausts before all VHS combine nodes run. The `save_output=True` fix does NOT help here — scenes never reach their VHS combine.

**Fix — switch to fp8 distilled:** The distilled-1.1 fp8 checkpoint (23GB) + fp4 Gemma (12GB) = 35GB base, leaving ~93GB for frames. This is enough for 8+ scenes. The quality difference at CFG 1.0 is negligible for most workflows.

**Fix — reduce scenes per run:** If full precision is required, split into 4-scene batches. Run batch 1 (scenes 1-4), then batch 2 (scenes 5-8), stitch with ffmpeg.

**PITFALL — `save_output=True` does NOT protect against Mode 1:** The agent may incorrectly claim that `save_output=True` on per-scene VHS nodes will save intermediate files even if the container OOMs. This is FALSE for Mode 1. `save_output` only works when the VHS node actually executes and writes — but in Mode 1, the OOM kills the process during sampling, before any VHS combine runs. Do NOT tell the user that `save_output` will prevent data loss in this scenario. Only Mode 2 (final combine OOM) benefits from `save_output`.

### Failure Mode 2: Final Combine OOM (all scenes rendered, combine crashes)

**Symptoms:** All per-scene VHS nodes complete and write `DGXmaxq_Scene*.mp4` files to `output/`. The FINAL VIDEO VHS_VideoCombine node then loads all decoded scene frames simultaneously for concatenation and OOMs.

**Root cause:** Even with fp8 models, holding 8 decoded scenes in memory for the final concat can exhaust 128GB.

**Fix:** Set `save_output=True` on all per-scene VHS nodes with unique prefixes. Each scene writes to `output/` immediately after rendering. If the final combine still crashes, the scene files survive and can be ffmpeg-concatenated manually:

```bash
# After crash, stitch surviving scene files:
for f in output/DGXmaxq_Scene*.mp4; do
  echo "file '$f'" >> concat.txt
done
ffmpeg -f concat -safe 0 -i concat.txt -i audio.mp3 \
  -map 0:v -map 1:a -c:v copy -c:a aac -b:a 256k -shortest final.mp4
```

**Important:** The `-audio.mp4` variant from VHS has both video+audio streams. The non-suffixed `.mp4` is video-only. Always use the `-audio.mp4` files.

### Diagnosing Which Mode Hit

1. Check if per-scene files exist: `ls -lth output/ | grep DGXmaxq_Scene`
2. Check journal for OOM kills: `sudo journalctl --since '<time>' --no-pager | grep -i 'oom.*kill'`
3. Count completed sampling runs in docker logs: `docker logs spark-comfyui | grep -c '100%|██████████| 8/8'`
4. Check container restart count: `docker inspect spark-comfyui --format='{{.RestartCount}}'`
5. If sampling runs completed but no scene files → Mode 1 (mid-render OOM)
6. If scene files exist but no final file → Mode 2 (final combine OOM)

**Real example (2026-07-30):** RapVidFinal_DGXmaxq workflow — 16 sampling runs completed (8 scenes × 2 stages), zero `DGXmaxq_Scene*` files, OOM kill at 112GB anon-rss. Mode 1. See `references/dgx-oom-investigation-2026-07-31.md` for full transcript.

### VAEDecodeTiled Parameter Pitfall — Tiled in Name Only

A workflow may already use `VAEDecodeTiled` but with parameters that defeat tiling entirely. This was discovered by Claude during the RapVidFinal_DGXmaxq investigation (2026-07-31):

| Parameter | Workflow Value | Effect |
|-----------|---------------|--------|
| `tile_size` | 1408 | ≥ base frame width (1280) → single spatial tile = no tiling |
| `temporal_size` | 4096 | ≫ 121 frames → no temporal tiling, whole clip decodes in one allocation |

The node is "tiled" in name only — it behaves like a plain `VAEDecode` for the memory peak. This is the proximate trigger for Mode 1 OOM: the single non-tiled decode allocation on top of 63GB resident models crosses 128GB.

**Fix:** Retune the `VAEDecodeTiled` widget values to actually tile:
- `tile_size`: 768 (must be < frame width to create multiple spatial tiles)
- `overlap`: 64
- `temporal_size`: 32 (must be < frame count to create temporal tiles)
- `temporal_overlap`: 8

This caps decode-time peak at zero quality cost and preserves full-precision models. Do NOT switch to fp8 distilled as a first resort — check VAEDecodeTiled parameters first.

**How to check:** Parse the workflow JSON for `VAEDecodeTiled` nodes and inspect `widgets_values`. If `tile_size` ≥ frame width or `temporal_size` ≥ frame count, tiling is defeated.

**PITFALL — Do NOT change model checkpoints without explicit permission:** When the user asks for a specific change (e.g. "disconnect the final merge"), do ONLY that change. Do not also switch UNETLoader checkpoints, disable LoRAs, or make other "while I'm here" optimizations. The user will tell you when they want model changes. Overstepping wastes the user's time reverting unwanted changes and erodes trust.

## CRITICAL: Ask Before Restarting

**NEVER restart the ComfyUI container without explicit user permission.** The container runs production workflows. Restarting kills in-progress generations. Always ask first, state the reason, and wait for approval.

This applies to: `./spark-comfyui.sh stop`, `./spark-comfyui.sh service` (restarts), `docker restart spark-comfyui`, and `./spark-comfyui.sh update` (which stops + rebuilds + restarts).

## Max-Quality LTX Workflow on DGX

When the user wants maximum quality (not speed) on the DGX, use the full-precision dev checkpoint workflow at `user/default/workflows/DirectorMaxQuality.json`:

| Setting | Distilled (Speed) | Max Quality (Dev) |
|---------|-------------------|-------------------|
| Checkpoint | distilled-1.1 fp8 | ltx-2.3-22b-dev.safetensors (43GB BF16) |
| Text encoder | Gemma 3 12B fp4 | Gemma 3 12B full (23GB) |
| Stage 1 steps | 8 | 40 |
| Stage 2 steps | 4 | 20 |
| CFG (both stages) | 1.0 | 3.0 |
| Guide strength | 0.5 | 1.0 |
| Sampler | euler | euler |
| Scheduler | linear_quadratic | linear_quadratic |

**Key differences from distilled:**
- No distilled LoRAs — the dev checkpoint doesn't need them
- CFG > 1 enables negative prompting (inactive at CFG=1)
- 60 total steps vs 12 — much longer render times
- 66GB base VRAM (43GB + 23GB) — DGX 128GB unified handles this
- UNETLoader must point to the full checkpoint in `diffusion_models/`, not transformer-only
- DualCLIPLoader must use full-precision Gemma, not fp4
- The full checkpoint needs a symlink: `diffusion_models/ltx-2.3-22b-dev.safetensors → checkpoints/ltx-2.3-22b-dev.safetensors`

## Docker Symlink Pitfall

**Absolute host paths in symlinks dangle inside the Docker container.** The bind mount maps `/home/adm1n/comfy/workspace/models` → `/opt/ComfyUI/models`, so a symlink targeting `/home/adm1n/comfy/workspace/models/checkpoints/file.safetensors` resolves on the host but NOT inside the container (where that path doesn't exist).

**Fix:** Always use relative symlinks so they survive the bind-mount namespace:
```bash
# WRONG — absolute host path, dangles in container:
ln -s /home/adm1n/comfy/workspace/models/checkpoints/ltx-2.3-22b-dev.safetensors diffusion_models/ltx-2.3-22b-dev.safetensors

# RIGHT — relative path, resolves on both host and container:
ln -s ../checkpoints/ltx-2.3-22b-dev.safetensors diffusion_models/ltx-2.3-22b-dev.safetensors
```

This applies to any symlink under the bind-mounted model directories (checkpoints, loras, vae, diffusion_models, etc.). Verify with `docker exec spark-comfyui ls -lh /opt/ComfyUI/models/<path>` — a dangling symlink shows `l????????? ?` with no size.

## SPARK_BF16 Default

spark-comfyui sets `SPARK_BF16=1` by default, which applies `--bf16-unet --bf16-vae --bf16-text-enc` globally. This is correct for LTX 2.3 (all bf16 models) but would corrupt any fp16-native model loaded later. For fp16 workflows, set `SPARK_BF16=0` at run time.

## Duplicate Node Dirs

Both `ComfyUI-Ollama`/`comfyui-ollama` and `comfyui_tensor_parallel_v2`/`v3` may be present in custom_nodes. Duplicate NODE_CLASS_MAPPINGS cause last-loaded-wins registration ambiguity. Logs may show no import failure, but behavior is undefined. Keep one of each pair.

### WhatDreamsCost LTX Director not in ComfyUI Manager

The LTX Director node (WhatDreamsCost/WhatDreamsCost-ComfyUI, 1,821 stars) is NOT registered in ComfyUI Manager's `custom-node-list.json`. It will never appear in the Manager UI. Install manually: `git clone https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI` into `custom_nodes/`. Author: Jonathan Watkins (@WhatDreamsCost). This is a critical LTX pipeline tool — timeline-based multi-scene video generation with IC-LoRA track, audio, keyframes, and Retake mode. Monitor GitHub releases directly for updates since Manager won't notify you.

### Windows-Style Path Mismatches in Copied Workflows

Workflows copied from .202 (x86_64 LXC) to DGX (Docker, aarch64) may contain Windows-style backslash paths in model references (e.g. `LTX\\v2\\file.safetensors`, `vae_approx\\taeltx2_3.safetensors`). TWO fixes are needed — JSON string replace AND directory symlinks. One without the other won't work.

**Fix A — JSON backslash replacement (MANDATORY, do first):** ComfyUI's `folder_paths` on Linux uses forward slashes. Backslash paths in the JSON will fail even if symlinks exist. Replace all `\` with `/` in widget values:
```python
import json
with open("workflow.json") as f:
    wf = json.load(f)
nodes = wf if isinstance(wf, list) else wf.get("nodes", [])
for n in nodes:
    for i, v in enumerate(n.get("widgets_values", [])):
        if isinstance(v, str) and "\\" in v:
            n["widgets_values"][i] = v.replace("\\", "/")
with open("workflow.json", "w") as f:
    json.dump(wf, f, indent=2)
```

**Fix B — Directory symlinks (do second):** Create the subdirectory structure the workflow expects:
```bash
# LoRA: workflow expects loras/LTX/v2/file.safetensors, file is at loras/file.safetensors
mkdir -p /home/adm1n/comfy/workspace/models/loras/LTX/v2
cd /home/adm1n/comfy/workspace/models/loras/LTX/v2
ln -sf ../../ltx-2.3-22b-distilled-1.1_lora-dynamic_fro09_avg_rank_111_bf16.safetensors .

# VAE: workflow expects vae_approx/file.safetensors, file is at vae/file.safetensors
mkdir -p /home/adm1n/comfy/workspace/models/vae_approx
cd /home/adm1n/comfy/workspace/models/vae_approx
ln -sf ../vae/taeltx2_3.safetensors .
```

Always verify symlinks resolve inside the Docker container: `docker exec spark-comfyui ls -lh /opt/ComfyUI/models/<path>`. A dangling symlink shows `l????????? ?` with no size.

**Pitfall:** Symlinks alone won't fix backslash paths. The UI will still show "Missing Models" because ComfyUI's `folder_paths` can't resolve `LTX\v2\` on Linux. Always do Fix A first.

## Clock Cap Not Verified by Doctor

`./spark-comfyui.sh doctor` does NOT check whether `--clock-cap 2100` is applied. The GPU may boost to ~3003 MHz uncapped, risking silent overcurrent hard-reboots under heavy video generation. After `tune --clock-cap 2100 --persist`, verify manually:
```bash
nvidia-smi -q -d CLOCK | grep -A5 "Max Clocks"
```

## Current State (2026-07-29)

1. **SageAttention 3 causes mosaic artifacts on DGX Spark** (upstream issue #321). spark-comfyui uses Sage 2.2 with native sm_121 kernel and live verification at every startup. Do NOT upgrade to Sage 3.

2. **CUDA 13.1+ ptxas may be rejected by r580 driver.** spark-comfyui pins to CUDA 13.0.3. Do not bump the CUDA base image without field testing.

3. **SPARK_BF16=1 is ON by default** — `--bf16-unet --bf16-text-enc` applied globally (correct for LTX 2.3 bf16 models). `--bf16-vae` was REMOVED (2026-07-29) because the audio VAE conv layers have float32 bias that crashes with `Input type (float) and bias type (c10::BFloat16) should be the same`. The video VAE handles bf16 fine; the audio VAE does not. Set `SPARK_BF16=0` for fp16 workflows.

4. **Overcurrent crashes:** System hard-reboots during video generation. Fix: `--clock-cap 2100`. Applied via `nvidia-smi -lgc 0,2100` and persisted in `comfyui-tune.service` (2026-07-29). Doctor does NOT verify this — check manually with `nvidia-smi --query-gpu=clocks.current.graphics --format=csv,noheader` (should be ≤2100).

5. **Swap thrashing:** System freezes near memory limit. Fix: `swapoff -a` (handled by `tune --persist`).

6. **Double memory from safetensors mmap:** On unified memory, mmap creates two resident copies. spark-comfyui patches `get_free_memory()` to use `psutil` instead of `cudaMemGetInfo`.

7. **x86_64 PyTorch contamination:** `pip install --upgrade -r requirements.txt` naively pulls x86_64 wheels. spark-comfyui's entrypoint verifies and auto-repairs torch before every launch.

8. **Duplicate node dirs:** `ComfyUI-Ollama`/`comfyui-ollama` and `tensor_parallel_v2`/`v3` both present — last-loaded-wins ambiguity. Keep one of each pair.

9. **Full-precision LTX stack loaded:** ltx-2.3-22b-dev (43GB BF16), Gemma 3 12B full (23GB), all LoRAs from .202, DirectorMaxQuality workflow at CFG 3.0 / 60 steps. See `references/dgx-model-inventory.md`.

## Research Reports

Full research reports at `/home/n8n/workspace/research/results/`:
- `2026-07-28-dgx-comfyui.md` — Best-practice survey (15 sources)
- `2026-07-28-dgx-comfyui-install.md` — Complete step-by-step install guide (18 sources, 596 lines)
- `2026-07-28-dgx-comfyui-best-check.md` — Confirmation spark-comfyui is still best (22 sources)

## Related

- `local-ai-media-generation` — Plan and evaluate local AI media generation pipelines
- `proxmox-lxc-deployment` — Deploy containerized services on Proxmox LXCs
- NVIDIA forum thread: https://forums.developer.nvidia.com/t/spark-comfyui-a-self-healing-comfyui-setup-for-dgx-spark/376542