hermes-skills/dgx-spark-comfyui/SKILL.md

name, description, version, author, license, platforms, metadata

name	description	version	author	license	platforms	metadata
dgx-spark-comfyui	Install, migrate, maintain ComfyUI on DGX Spark (GB10).	1.0.0	Hermes Agent	MIT	linux	hermes tags related_skills dgx-spark comfyui gb10 aarch64 unified-memory sm_121a infrastructure local-ai-media-generation proxmox-lxc-deployment

DGX Spark ComfyUI — Install, Migrate, Maintain

Overview

The DGX Spark (GB10 Grace Blackwell, sm_121a, 128GB unified LPDDR5x, 3.7TB NVMe, aarch64) is a capable ComfyUI host but its unusual architecture requires specific configuration. The community has converged on spark-comfyui (bjarkebolding) as the standard — one script, fully containerized, self-healing, actively maintained.

Current best option (July 2026): spark-comfyui — 52 commits, last updated July 27, 2026. NVIDIA forum thread: 19 likes, 848 views.

Our DGX Spark: 10.0.0.6, hostname edgexpert-7d04, SSH [email protected] (passw0rd), sudo passw0rd.

Fresh Install (spark-comfyui)

# 1. Clone
git clone https://github.com/bjarkebolding/spark-comfyui.git
cd spark-comfyui

# 2. Build image (10-30 min)
./spark-comfyui.sh install

# 3. System tuning (swap off, persistence mode, optional clock cap)
./spark-comfyui.sh tune --persist

# 4. If overcurrent crashes occur, cap GPU clock:
./spark-comfyui.sh tune --clock-cap 2100 --persist

# 5. Run as service (auto-start on boot, survives crashes)
./spark-comfyui.sh service

# 6. Verify
./spark-comfyui.sh doctor

Models go in data/models/checkpoints/ (etc.). No venv, no system Python changes.

Migration from aeon-spark

Step-by-step

1. Stop and remove old container + images:

   docker stop comfyui-spark && docker rm comfyui-spark
   docker rmi ghcr.io/aeon-7/comfyui-aeon-spark:latest
   docker image prune -f
   

2. Clone spark-comfyui and install (see Fresh Install above).

3. Configure spark-mounts.conf to reuse existing workspace:

   models = /home/adm1n/comfy/workspace/models
   custom_nodes = /home/adm1n/comfy/workspace/custom_nodes
   user = /home/adm1n/comfy/workspace/user
   output = /home/adm1n/comfy/workspace/output
   input = /home/adm1n/comfy/workspace/input
   

4. Fix permissions — aeon-spark wrote files as root, spark-comfyui runs as UID 1000:

   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/user/
   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/custom_nodes/
   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/models/
   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/output/
   echo passw0rd | sudo -S chown -R 1000:1000 /home/adm1n/comfy/workspace/input/
   

5. Remove obsolete custom node (no __init__.py, superseded by unified memory mod):

   rm -rf /home/adm1n/comfy/workspace/custom_nodes/ComfyUI_LTX-2_VRAM_Memory_Management
   

6. Tune and start service.

Downtime: ~30 minutes (image build + model copy time). Models, custom nodes, workflows, and user data are all reused — only the Docker image is rebuilt.

Dockerfile Patch for libGL

Dockerfile Patch for libGL

The spark-comfyui Dockerfile installs libx11-6 libxext6 libxcb1 but NOT libgl1. This causes import failures for LTXVideo, VideoHelperSuite, RES4LYF, and Easy-Use (ImportError: libGL.so.1: cannot open shared object file).

Fix: Add libgl1 libglib2.0-0 to both apt-get install lines in container/Dockerfile. Do NOT use libgl1-mesa-glx — that package was removed in Ubuntu 24.04 and has no installation candidate.

sed -i 's/libx11-6 libxext6 libxcb1/libx11-6 libxext6 libxcb1 libgl1 libglib2.0-0/' container/Dockerfile

Then rebuild: ./spark-comfyui.sh update

Common Import Failures and Fixes

Failure	Cause	Fix
libGL.so.1: cannot open shared object file	Missing mesa libs in container	Patch Dockerfile to add libgl1 libglib2.0-0, rebuild
No module named 'soundfile'	Missing pip package, no requirements.txt in custom node	Create requirements.txt with soundfile in the node dir
cannot import name 'interleaved_freqs_cis'	LTXVideo custom node stale vs ComfyUI core	git pull origin master in ComfyUI-LTXVideo dir
ComfyUI_LTX-2_VRAM_Memory_Management no __init__.py	Obsolete node, superseded by unified memory mod	rm -rf the directory
comfyui.db.lock Permission denied	Root-owned files from aeon-spark	chown -R 1000:1000 on workspace dirs
Impact Pack / controlnet_aux import failed	Missing dependencies	Acceptable — non-critical nodes

Manager Fixes

Manager invisible in UI ("Blocked by policy")

The pip package comfyui-manager conflicts with the git clone at custom_nodes/ComfyUI-Manager. The pip package's should_be_disabled() returns True for any dir containing "comfyui-manager", blocking the git clone's WEB_DIRECTORY = "js".

Fix: pip uninstall comfyui-manager -y. The git clone is then unblocked and Manager appears in the UI bar.

Manager: "Invalid channel" / empty node list

The git clone may point to the OLD ltdrdata/ComfyUI-Manager repo. The repo moved to Comfy-Org/ComfyUI-Manager. Three fixes needed:

1. Git remote: git remote set-url origin https://github.com/Comfy-Org/ComfyUI-Manager.git
2. channels.list / channels.list.template: Replace ltdrdata/ComfyUI-Manager with Comfy-Org/ComfyUI-Manager
3. Hardcoded DEFAULT_CHANNEL in Python: Replace in glob/manager_core.py and glob/manager_server.py

--enable-manager flag is redundant

ComfyUI core already loads the built-in Manager. The --enable-manager flag in the entrypoint is unnecessary. Remove it from container/entrypoint.sh DEFAULT_FLAGS.

Dockerfile Changes (Baked into Image)

The spark-comfyui Dockerfile at container/Dockerfile has these local modifications:

1. libgl1-mesa-glx → libgl1 (Ubuntu 24.04 compat — mesa-glx removed in 24.04)
2. pip install soundfile after manager_requirements.txt
3. --enable-manager removed from entrypoint.sh DEFAULT_FLAGS (redundant)

Entrypoint Changes

The spark-comfyui entrypoint at container/entrypoint.sh has this local modification:

1. --bf16-vae removed from SPARK_BF16 default flags (2026-07-29) — the audio VAE conv layers have float32 bias that crashes with Input type (float) and bias type (c10::BFloat16) should be the same. Only --bf16-unet --bf16-text-enc remain.

These survive ./spark-comfyui.sh update because they're in the repo's tracked files. If the upstream files change significantly, re-apply after git pull.

Key Commands

Command	Purpose
./spark-comfyui.sh install	Build image
./spark-comfyui.sh run	Foreground
./spark-comfyui.sh service	Background, auto-restart, survives reboots
./spark-comfyui.sh stop	Stop container
./spark-comfyui.sh update	Rebuild image on current ComfyUI master
./spark-comfyui.sh update --rollback	Swap back to previous image
./spark-comfyui.sh doctor	Live GPU gates in throwaway container
./spark-comfyui.sh status --watch	Live sparkline dashboard
./spark-comfyui.sh tune --persist	System tuning (swap off, persistence mode)
./spark-comfyui.sh tune --clock-cap 2100 --persist	Cap GPU clock to prevent overcurrent crashes
./spark-comfyui.sh backup	Backup workflows, settings, custom nodes
./spark-comfyui.sh restore FILE	Rebuild from backup

Workflow-Driven Model Download

When the user says "download all required files for workflow X," do NOT guess or search manually. Extract dependencies from the workflow JSON, inventory what's already on the DGX, then download only what's missing — in parallel.

Step-by-step

1. Read the workflow JSON — cat /home/adm1n/comfy/workspace/user/default/workflows/{name}.json
2. Extract all model references — parse widgets_values from Loader nodes (VAELoader, DualCLIPLoader, UNETLoader, LoraLoader, LoadImage, etc.). Also check subgraph nodes (UUID-type nodes) — they embed model filenames in their widget values.
3. Extract input images — LoadImage nodes reference files in input/.
4. Check what's already on the DGX — ls -la each model directory for the expected filenames.
5. Find download URLs — workflow MarkdownNote nodes often contain HuggingFace links. If not, search huggingface.co/{filename} to find the repo. Use HF API (https://huggingface.co/api/models/{org}/{repo}) to get file sizes.
6. Download in parallel — use terminal(background=true, notify_on_complete=true) for each file. Large models (20GB+) go to diffusion_models/, LoRAs to loras/, VAEs to vae/, text encoders to text_encoders/.
7. Copy input images — if the workflow references images from .202, use two-hop pipe: ssh .202 "cat /path/file" | ssh DGX "cat > /home/adm1n/comfy/workspace/input/file". Do NOT try direct scp from the Hermes host — the files live on .202, not locally.

Example: qwenI2I.json + qwenT2I.json (2026-07-30)

Workflow JSONs at user/default/workflows/qwenI2I.json and qwenT2I.json.

qwenI2I.json needs:

File	Type	Dir	Status
qwen_image_edit_2511_fp8mixed.safetensors	diffusion model	diffusion_models/	Download from Comfy-Org/Qwen-Image-Edit_ComfyUI
qwen_2.5_vl_7b_fp8_scaled.safetensors	text encoder	text_encoders/	Already present
qwen_image_vae.safetensors	VAE	vae/	Already present
Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors	LoRA	loras/	Already present
August-2017-4643-2.jpg	input image	input/	Copy from .202
Qwen-Image-2512_00007_.png	input image	input/	Copy from .202
RobLTX.png	input image	input/	Copy from .202

qwenT2I.json needs:

File	Type	Dir	Status
qwen_image_2512_fp8_e4m3fn.safetensors	diffusion model	diffusion_models/	Download from Comfy-Org/Qwen-Image_ComfyUI
qwen_2.5_vl_7b_fp8_scaled.safetensors	text encoder	text_encoders/	Already present
qwen_image_vae.safetensors	VAE	vae/	Already present
Qwen-Image-2512-Lightning-4steps-V1.0-fp32.safetensors	LoRA	loras/	Download from lightx2v/Qwen-Image-2512-Lightning

Two-Hop File Transfer (input images from .202)

The Hermes host cannot directly scp from .202 to DGX. Pipe through the Hermes host:

sshpass -p 'n8n' ssh -o StrictHostKeyChecking=no [email protected] \
  "cat /home/n8n/comfy-ui/input/FILENAME" | \
  sshpass -p 'passw0rd' ssh -o StrictHostKeyChecking=no [email protected] \
  "cat > /home/adm1n/comfy/workspace/input/FILENAME"

Run each image transfer in its own background process for parallelism.

Subgraph Nodes (UUID-type)

Workflows from Comfy-Org templates use subgraph nodes with UUID type identifiers (e.g. cdb2cf24-c432-439b-b5c8-5f69838580c9). These embed model filenames directly in their widgets_values array — parse them the same way as regular loader nodes. The subgraph's internal nodes are stored in the workflow's extra field.

Max Precision Policy

Always check HuggingFace for higher-precision versions of every model referenced in a workflow. Download the max-precision variant to disk alongside the workflow's fp8 reference. Do NOT modify the workflow JSON — keep it pointing at the fp8 filename. The user can switch to max precision later by editing the workflow themselves.

For each model in a workflow, check the same HF repo for bf16/fp16/fp32 variants:

• qwen_image_edit_2511_fp8mixed → also download qwen_image_edit_2511_bf16 (39GB)
• qwen_image_2512_fp8_e4m3fn → also download qwen_image_2512_bf16 (41GB)
• qwen_2.5_vl_7b_fp8_scaled → also download qwen_2.5_vl_7b.safetensors (16.6GB)
• Lightning LoRAs are already fp32 — no higher precision exists
• VAE has only one version (254MB) — no higher precision exists

Use the HF API to discover available files: https://huggingface.co/api/models/{org}/{repo} → check siblings for bf16/fp16/fp32 variants. Download in parallel with the fp8 versions.

Cross-Environment Model Parity

When someone says "make the DGX match the .202 setup," verify before acting. The DGX (Docker, aarch64, unified memory) and .202 (bare-metal LXC, x86_64, discrete VRAM) have different environments. A .202 workflow JSON won't load on DGX without path adjustments — node IDs, model paths, and custom node versions differ.

Checklist before copying models:

1. Inventory BOTH sides first (find by model type)
2. Most "upgrades" are already present — the DGX often already has full-precision variants
3. Only copy what's genuinely missing, not what someone assumes is missing
4. Verify LoRA paths match what the workflow JSON expects (subdirectories like loras/ltxv/ltx2/ vs loras/)
5. Check cond-safe LoRA presence — Director workflows may depend on it

Full mirroring procedure (when user explicitly wants .202 parity at full precision):

1. Inventory both sides: find models/ -iname '*ltx*' and ls custom_nodes/ on both hosts
2. Copy missing LoRAs via scp (two-hop: .202 → local → DGX)
3. Fix LoRA path mismatches with symlinks (e.g. ln -s loras/ltxv/ltx2/file.safetensors loras/file.safetensors) — avoids duplicating 7GB+ files
4. Git clone missing custom nodes into workspace custom_nodes/; for nodes without public repos (like SeqParallel V5), scp the .py files directly
5. Copy the working Director workflow JSON from .202 user/default/workflows/ to DGX
6. Rebuild Docker: ./spark-comfyui.sh stop && ./spark-comfyui.sh update && ./spark-comfyui.sh service
7. Verify: docker logs spark-comfyui | grep "IMPORT FAILED" must be zero

Known delta (2026-07-29): IC-LoRA Ingredients, cond-safe LoRA, Cinemagraph, Transition, SeqParallel V5, MelBandRoFormer, and comfyui-ollama were on .202 but not DGX. Full-precision models (dev 43GB, Gemma 3 12B 23GB) were already present.

Model inventory: See references/dgx-model-inventory.md for the full current snapshot.

Workflow port + validation checklist: See references/workflow-port-validation.md for the step-by-step procedure to copy a workflow from .202 to DGX with byte-for-byte verification, input file transfer, and independent validation.

Multi-agent validation pattern: See references/multi-agent-validation.md for dispatching Claude + Grok + self in parallel to validate a workflow before queuing.

Max-quality upgrade pattern: See references/rapvidfinal-quality-analysis.md for the ranked bang-for-buck analysis pattern — dispatch multiple agents to analyze a workflow, rank quality improvements, apply top 2-3 as a low-risk package.

Hours-capable workflow pattern: See references/hours-capable-workflow.md for converting multi-scene workflows from single-concat to per-scene file output, removing the ImageBatchMulti RAM wall for arbitrarily long videos.

Diff-based workflow port pattern: See references/diff-based-workflow-port.md for porting parameter changes between workflows with field-level widget diffs — prevents missing changes and editing wrong nodes. Validated by Claude + Grok on the RapVidFinal → FullHqHours port (2026-07-30).

Pitfall: Don't Pull Files When Dispatching Validators

When the user says "ask X to SSH in and validate workflow Y," do NOT pull the workflow files locally first. The agents (Grok, Claude) should SSH directly into the DGX and read the files themselves. Pulling files locally wastes time and defeats the purpose of independent validation — the agents need to verify the files exist at the expected paths on the actual host.

Wrong: Pull JSONs to /tmp, scp to agent hosts, paste inline. Right: Write a prompt telling the agent to sshpass -p 'passw0rd' ssh [email protected] and cat the files directly.

Multi-Scene OOM on DGX Spark — Two Failure Modes

The DGX Spark (128GB unified memory) can OOM during multi-scene LTX video workflows. There are TWO distinct failure modes — knowing which one hit determines the fix.

Failure Mode 1: Mid-Render OOM (full-precision models, 6+ scenes)

Symptoms: Container is OOM-killed during scene rendering. Sampling progress bars show completions but NO per-scene output files (DGXmaxq_Scene*.mp4) exist — only generic LTX_00001.* temp files. Journal shows python invoked oom-killer with 110GB+ anon-rss.

Root cause: Full-precision stack (43GB dev + 23GB Gemma = 66GB base) leaves only ~62GB for decoded frames across all scenes. With 8 scenes at 768×448×121 frames each, memory exhausts before all VHS combine nodes run. The save_output=True fix does NOT help here — scenes never reach their VHS combine.

Fix — switch to fp8 distilled: The distilled-1.1 fp8 checkpoint (23GB) + fp4 Gemma (12GB) = 35GB base, leaving ~93GB for frames. This is enough for 8+ scenes. The quality difference at CFG 1.0 is negligible for most workflows.

Fix — reduce scenes per run: If full precision is required, split into 4-scene batches. Run batch 1 (scenes 1-4), then batch 2 (scenes 5-8), stitch with ffmpeg.

PITFALL — save_output=True does NOT protect against Mode 1: The agent may incorrectly claim that save_output=True on per-scene VHS nodes will save intermediate files even if the container OOMs. This is FALSE for Mode 1. save_output only works when the VHS node actually executes and writes — but in Mode 1, the OOM kills the process during sampling, before any VHS combine runs. Do NOT tell the user that save_output will prevent data loss in this scenario. Only Mode 2 (final combine OOM) benefits from save_output.

Failure Mode 2: Final Combine OOM (all scenes rendered, combine crashes)

Symptoms: All per-scene VHS nodes complete and write DGXmaxq_Scene*.mp4 files to output/. The FINAL VIDEO VHS_VideoCombine node then loads all decoded scene frames simultaneously for concatenation and OOMs.

Root cause: Even with fp8 models, holding 8 decoded scenes in memory for the final concat can exhaust 128GB.

Fix: Set save_output=True on all per-scene VHS nodes with unique prefixes. Each scene writes to output/ immediately after rendering. If the final combine still crashes, the scene files survive and can be ffmpeg-concatenated manually:

# After crash, stitch surviving scene files:
for f in output/DGXmaxq_Scene*.mp4; do
  echo "file '$f'" >> concat.txt
done
ffmpeg -f concat -safe 0 -i concat.txt -i audio.mp3 \
  -map 0:v -map 1:a -c:v copy -c:a aac -b:a 256k -shortest final.mp4

Important: The -audio.mp4 variant from VHS has both video+audio streams. The non-suffixed .mp4 is video-only. Always use the -audio.mp4 files.

Diagnosing Which Mode Hit

1. Check if per-scene files exist: ls -lth output/ | grep DGXmaxq_Scene
2. Check journal for OOM kills: sudo journalctl --since '<time>' --no-pager | grep -i 'oom.*kill'
3. Count completed sampling runs in docker logs: docker logs spark-comfyui | grep -c '100%|██████████| 8/8'
4. Check container restart count: docker inspect spark-comfyui --format='{{.RestartCount}}'
5. If sampling runs completed but no scene files → Mode 1 (mid-render OOM)
6. If scene files exist but no final file → Mode 2 (final combine OOM)

Real example (2026-07-30): RapVidFinal_DGXmaxq workflow — 16 sampling runs completed (8 scenes × 2 stages), zero DGXmaxq_Scene* files, OOM kill at 112GB anon-rss. Mode 1. See references/dgx-oom-investigation-2026-07-31.md for full transcript.

VAEDecodeTiled Parameter Pitfall — Tiled in Name Only

A workflow may already use VAEDecodeTiled but with parameters that defeat tiling entirely. This was discovered by Claude during the RapVidFinal_DGXmaxq investigation (2026-07-31):

Parameter	Workflow Value	Effect
tile_size	1408	≥ base frame width (1280) → single spatial tile = no tiling
temporal_size	4096	≫ 121 frames → no temporal tiling, whole clip decodes in one allocation

The node is "tiled" in name only — it behaves like a plain VAEDecode for the memory peak. This is the proximate trigger for Mode 1 OOM: the single non-tiled decode allocation on top of 63GB resident models crosses 128GB.

Fix: Retune the VAEDecodeTiled widget values to actually tile:

• tile_size: 768 (must be < frame width to create multiple spatial tiles)
• overlap: 64
• temporal_size: 32 (must be < frame count to create temporal tiles)
• temporal_overlap: 8

This caps decode-time peak at zero quality cost and preserves full-precision models. Do NOT switch to fp8 distilled as a first resort — check VAEDecodeTiled parameters first.

How to check: Parse the workflow JSON for VAEDecodeTiled nodes and inspect widgets_values. If tile_size ≥ frame width or temporal_size ≥ frame count, tiling is defeated.

PITFALL — Do NOT change model checkpoints without explicit permission: When the user asks for a specific change (e.g. "disconnect the final merge"), do ONLY that change. Do not also switch UNETLoader checkpoints, disable LoRAs, or make other "while I'm here" optimizations. The user will tell you when they want model changes. Overstepping wastes the user's time reverting unwanted changes and erodes trust.

CRITICAL: Ask Before Restarting

NEVER restart the ComfyUI container without explicit user permission. The container runs production workflows. Restarting kills in-progress generations. Always ask first, state the reason, and wait for approval.

This applies to: ./spark-comfyui.sh stop, ./spark-comfyui.sh service (restarts), docker restart spark-comfyui, and ./spark-comfyui.sh update (which stops + rebuilds + restarts).

Max-Quality LTX Workflow on DGX

When the user wants maximum quality (not speed) on the DGX, use the full-precision dev checkpoint workflow at user/default/workflows/DirectorMaxQuality.json:

Setting	Distilled (Speed)	Max Quality (Dev)
Checkpoint	distilled-1.1 fp8	ltx-2.3-22b-dev.safetensors (43GB BF16)
Text encoder	Gemma 3 12B fp4	Gemma 3 12B full (23GB)
Stage 1 steps	8	40
Stage 2 steps	4	20
CFG (both stages)	1.0	3.0
Guide strength	0.5	1.0
Sampler	euler	euler
Scheduler	linear_quadratic	linear_quadratic

Key differences from distilled:

• No distilled LoRAs — the dev checkpoint doesn't need them
• CFG > 1 enables negative prompting (inactive at CFG=1)
• 60 total steps vs 12 — much longer render times
• 66GB base VRAM (43GB + 23GB) — DGX 128GB unified handles this
• UNETLoader must point to the full checkpoint in diffusion_models/, not transformer-only
• DualCLIPLoader must use full-precision Gemma, not fp4
• The full checkpoint needs a symlink: diffusion_models/ltx-2.3-22b-dev.safetensors → checkpoints/ltx-2.3-22b-dev.safetensors

Docker Symlink Pitfall

Absolute host paths in symlinks dangle inside the Docker container. The bind mount maps /home/adm1n/comfy/workspace/models → /opt/ComfyUI/models, so a symlink targeting /home/adm1n/comfy/workspace/models/checkpoints/file.safetensors resolves on the host but NOT inside the container (where that path doesn't exist).

Fix: Always use relative symlinks so they survive the bind-mount namespace:

# WRONG — absolute host path, dangles in container:
ln -s /home/adm1n/comfy/workspace/models/checkpoints/ltx-2.3-22b-dev.safetensors diffusion_models/ltx-2.3-22b-dev.safetensors

# RIGHT — relative path, resolves on both host and container:
ln -s ../checkpoints/ltx-2.3-22b-dev.safetensors diffusion_models/ltx-2.3-22b-dev.safetensors

This applies to any symlink under the bind-mounted model directories (checkpoints, loras, vae, diffusion_models, etc.). Verify with docker exec spark-comfyui ls -lh /opt/ComfyUI/models/<path> — a dangling symlink shows l????????? ? with no size.

SPARK_BF16 Default

spark-comfyui sets SPARK_BF16=1 by default, which applies --bf16-unet --bf16-vae --bf16-text-enc globally. This is correct for LTX 2.3 (all bf16 models) but would corrupt any fp16-native model loaded later. For fp16 workflows, set SPARK_BF16=0 at run time.

Duplicate Node Dirs

Both ComfyUI-Ollama/comfyui-ollama and comfyui_tensor_parallel_v2/v3 may be present in custom_nodes. Duplicate NODE_CLASS_MAPPINGS cause last-loaded-wins registration ambiguity. Logs may show no import failure, but behavior is undefined. Keep one of each pair.

WhatDreamsCost LTX Director not in ComfyUI Manager

The LTX Director node (WhatDreamsCost/WhatDreamsCost-ComfyUI, 1,821 stars) is NOT registered in ComfyUI Manager's custom-node-list.json. It will never appear in the Manager UI. Install manually: git clone https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI into custom_nodes/. Author: Jonathan Watkins (@WhatDreamsCost). This is a critical LTX pipeline tool — timeline-based multi-scene video generation with IC-LoRA track, audio, keyframes, and Retake mode. Monitor GitHub releases directly for updates since Manager won't notify you.

Windows-Style Path Mismatches in Copied Workflows

Workflows copied from .202 (x86_64 LXC) to DGX (Docker, aarch64) may contain Windows-style backslash paths in model references (e.g. LTX\\v2\\file.safetensors, vae_approx\\taeltx2_3.safetensors). TWO fixes are needed — JSON string replace AND directory symlinks. One without the other won't work.

Fix A — JSON backslash replacement (MANDATORY, do first): ComfyUI's folder_paths on Linux uses forward slashes. Backslash paths in the JSON will fail even if symlinks exist. Replace all \ with / in widget values:

import json
with open("workflow.json") as f:
    wf = json.load(f)
nodes = wf if isinstance(wf, list) else wf.get("nodes", [])
for n in nodes:
    for i, v in enumerate(n.get("widgets_values", [])):
        if isinstance(v, str) and "\\" in v:
            n["widgets_values"][i] = v.replace("\\", "/")
with open("workflow.json", "w") as f:
    json.dump(wf, f, indent=2)

Fix B — Directory symlinks (do second): Create the subdirectory structure the workflow expects:

# LoRA: workflow expects loras/LTX/v2/file.safetensors, file is at loras/file.safetensors
mkdir -p /home/adm1n/comfy/workspace/models/loras/LTX/v2
cd /home/adm1n/comfy/workspace/models/loras/LTX/v2
ln -sf ../../ltx-2.3-22b-distilled-1.1_lora-dynamic_fro09_avg_rank_111_bf16.safetensors .

# VAE: workflow expects vae_approx/file.safetensors, file is at vae/file.safetensors
mkdir -p /home/adm1n/comfy/workspace/models/vae_approx
cd /home/adm1n/comfy/workspace/models/vae_approx
ln -sf ../vae/taeltx2_3.safetensors .

Always verify symlinks resolve inside the Docker container: docker exec spark-comfyui ls -lh /opt/ComfyUI/models/<path>. A dangling symlink shows l????????? ? with no size.

Pitfall: Symlinks alone won't fix backslash paths. The UI will still show "Missing Models" because ComfyUI's folder_paths can't resolve LTX\v2\ on Linux. Always do Fix A first.

Clock Cap Not Verified by Doctor

./spark-comfyui.sh doctor does NOT check whether --clock-cap 2100 is applied. The GPU may boost to ~3003 MHz uncapped, risking silent overcurrent hard-reboots under heavy video generation. After tune --clock-cap 2100 --persist, verify manually:

nvidia-smi -q -d CLOCK | grep -A5 "Max Clocks"

Current State (2026-07-29)

1. SageAttention 3 causes mosaic artifacts on DGX Spark (upstream issue #321). spark-comfyui uses Sage 2.2 with native sm_121 kernel and live verification at every startup. Do NOT upgrade to Sage 3.

2. CUDA 13.1+ ptxas may be rejected by r580 driver. spark-comfyui pins to CUDA 13.0.3. Do not bump the CUDA base image without field testing.

3. SPARK_BF16=1 is ON by default — --bf16-unet --bf16-text-enc applied globally (correct for LTX 2.3 bf16 models). --bf16-vae was REMOVED (2026-07-29) because the audio VAE conv layers have float32 bias that crashes with Input type (float) and bias type (c10::BFloat16) should be the same. The video VAE handles bf16 fine; the audio VAE does not. Set SPARK_BF16=0 for fp16 workflows.

4. Overcurrent crashes: System hard-reboots during video generation. Fix: --clock-cap 2100. Applied via nvidia-smi -lgc 0,2100 and persisted in comfyui-tune.service (2026-07-29). Doctor does NOT verify this — check manually with nvidia-smi --query-gpu=clocks.current.graphics --format=csv,noheader (should be ≤2100).

5. Swap thrashing: System freezes near memory limit. Fix: swapoff -a (handled by tune --persist).

6. Double memory from safetensors mmap: On unified memory, mmap creates two resident copies. spark-comfyui patches get_free_memory() to use psutil instead of cudaMemGetInfo.

7. x86_64 PyTorch contamination: pip install --upgrade -r requirements.txt naively pulls x86_64 wheels. spark-comfyui's entrypoint verifies and auto-repairs torch before every launch.

8. Duplicate node dirs: ComfyUI-Ollama/comfyui-ollama and tensor_parallel_v2/v3 both present — last-loaded-wins ambiguity. Keep one of each pair.

9. Full-precision LTX stack loaded: ltx-2.3-22b-dev (43GB BF16), Gemma 3 12B full (23GB), all LoRAs from .202, DirectorMaxQuality workflow at CFG 3.0 / 60 steps. See references/dgx-model-inventory.md.

Research Reports

Full research reports at /home/n8n/workspace/research/results/:

• 2026-07-28-dgx-comfyui.md — Best-practice survey (15 sources)
• 2026-07-28-dgx-comfyui-install.md — Complete step-by-step install guide (18 sources, 596 lines)
• 2026-07-28-dgx-comfyui-best-check.md — Confirmation spark-comfyui is still best (22 sources)

Related

• local-ai-media-generation — Plan and evaluate local AI media generation pipelines
• proxmox-lxc-deployment — Deploy containerized services on Proxmox LXCs
• NVIDIA forum thread: https://forums.developer.nvidia.com/t/spark-comfyui-a-self-healing-comfyui-setup-for-dgx-spark/376542