hermes-skills/comfyui/references/remote-connectivity.md

# Remote ComfyUI Server — Connectivity & Diagnostics

This skill assumes ComfyUI is running locally on `127.0.0.1:8188` by default.
When the ComfyUI server lives on another machine (LAN, headless GPU box,
Proxmox LXC with passthrough, etc.), follow this guide.

## Quick Checklist

1. **ComfyUI running with `--listen 0.0.0.0`** on the remote host
2. **Firewall open** for port 8188 (or whatever custom port)
3. **Hermes can reach it** via `curl http://REMOTE:8188/system_stats`
4. **Use `--host http://REMOTE:PORT`** on every script invocation (or set `$COMFYUI_HOST`)
5. If curl fails, **SSH into the remote host to diagnose and restart** (see [SSH-based diagnostics](#ssh-based-remote-diagnostics-and-remediation))

## SSH-based Remote Diagnostics and Remediation

When the remote ComfyUI server is unreachable on its HTTP port but you have
SSH access to the host, SSH directly in to diagnose and fix. This is
common when the user manages headless GPU boxes (Proxmox LXCs, dedicated
render servers, etc.) and wants Hermes to handle restarts.

### Prerequisites in the Hermes container
- `sshpass` is needed for non-interactive password auth. If missing:
  ```bash
  # This Hermes container may not have apt-write access
  pip install --user sshpass 2>/dev/null || echo "Must install sshpass on the host manually"
  ```
- If `sshpass` cannot be installed, use Python `pexpect` instead:
  ```bash
  pip install pexpect
  ```
  ...or ask the user for the correct username if `root`/`n8n` both fail.

### One-shot SSH diagnostic (safe for terminal tool — no PTY tricks needed)
```bash
# Requires sshpass installed
sshpass -p 'PASSWORD' ssh -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null \
  '[email protected]' \
  'n8n' \
  'echo ===CONNECTED===; nvidia-smi; echo ===DONE==='
```

**If password is rejected, STOP.** Do not loop. Ask the user:
> "SSH password rejected for user `X` on `10.0.0.202`. What is the correct username and password?"

### What to check/fix once SSH'd in
| Command | What it tells you |
|---------|-----------------|
| `nvidia-smi` | GPU + driver status |
| `ps aux \| grep -i comfy` | Is ComfyUI running? |
| `curl -s http://127.0.0.1:8188/system_stats` | Is it responding locally? |
| `ss -tlnp \| grep 8188` | Is it listening on 0.0.0.0 or just 127.0.0.1? |
| `ufw status \| grep 8188` or `iptables -L \| grep 8188` | Is the firewall blocking? |

### Common fixes
```bash
# ComfyUI not running → start it with LAN binding
comfy launch -- --listen 0.0.0.0 --port 8188 &

# Or if launched manually, relaunch:
cd ~/comfy/ComfyUI  # or wherever it's installed
python main.py --listen 0.0.0.0 --port 8188 &

# Firewall blocking → open it
ufw allow 8188/tcp
# or
iptables -A INPUT -p tcp --dport 8188 -j ACCEPT

# GPU driver issue (e.g. after host kernel update)
nvidia-smi  # check for driver mismatch
# If missing, reinstall driver on the host, NOT in the LXC
```

### Hermes-side SSH config (one-time setup)
Store host info so future sessions can SSH without re-configuring:
```bash
mkdir -p ~/.ssh && chmod 700 ~/.ssh
cat > ~/.ssh/config <<EOF
Host n8n
    HostName 10.0.0.202
    User USERNAME
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
    IdentitiesOnly no
EOF
chmod 600 ~/.ssh/config
```

### Important: Do not PTY-loop (terminal tool)
When `sshpass` is unavailable, do NOT write Python PTY scripts that call
`os.setsid()` inside the `terminal` tool — this is permission-denied inside
unprivileged Proxmox LXCs.

Instead use **`execute_code` with `pty.fork()`** — the sandbox allows the
fork but the `terminal` tool does not.  See
`references/remote-manual-update.md` for the reusable `ssh_cmd()` recipe.
That reference also covers:
- Updating a manual `git clone` install (no `comfy` CLI available)
- Safely restarting the server when `nohup` is rejected by `terminal`
- The model-directory pitfall (`CheckpointLoaderSimple` vs `diffusion_models`)
- The version-file staleness pitfall (`comfyui_version.py`)

If neither `sshpass`/`pexpect` nor `execute_code` + `pty.fork()` works,
fallback is asking the user for credentials or to run the command themselves.

---

### From Hermes (this container)

```bash
# Test 1: Is the host alive at all?
ping -c 1 10.0.0.202

# Test 2: Is port 8188 open?
python3 -c "import socket; s=socket.socket(); s.settimeout(3); s.connect(('10.0.0.202', 8188)); print('open')" 2>/dev/null || echo "closed/unreachable"

# Test 3: Does it speak HTTP?
curl -s --max-time 5 http://10.0.0.202:8188/system_stats | head -c 100

# Test 4: Wrong port? Scan common ComfyUI ports
for port in 8188 8080 3000 5000 7860 9000; do
  python3 -c "import socket; s=socket.socket(); s.settimeout(1); s.connect(('10.0.0.202', $port)); print('$port open')" 2>/dev/null
done
```

### Likely causes & fixes

| Symptom | Cause | Fix on remote host |
|---------|-------|--------------------|
| `Connection refused` | ComfyUI listening on `127.0.0.1` only | `comfy launch -- --listen 0.0.0.0 --port 8188` |
| `No route to host` | Firewall / network segment blocking | `ufw allow 8188/tcp` or adjust Proxmox firewall rules |
| `Connection timed out` | ComfyUI not running at all | `ps aux \| grep -i comfy` then start it |
| Wrong port open | ComfyUI launched on a non-default port | Re-launch with `--port 8188` or note the actual port |
| Works from remote shell but not Hermes | Proxmox LXC bridge / VLAN isolation | Check CT firewall rules; ensure both CTs share bridge |
| Password rejected via sshpass | Wrong username or password, or password-only SSH disabled | Clarify user/password with user; do not loop, ask once |

## Credential Ambiguity — Pitfall

When the user says something like:
> "SSH n8n - passw0rd"

They may mean any of:
- Host alias = `n8n`, password = `passw0rd`, user = default (`root`)
- Host alias = `n8n`, user = `n8n`, password = `passw0rd`
- Host IP `10.0.0.202`, user = `n8n`, password = `passw0rd`

**Best practice:** Immediately write the SSH config with the most likely
translation, then run a **minimal connectivity probe** before attempting
diagnosis:
```bash
sshpass -p 'passw0rd' ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
  n8n 'echo ok'  # or 'USER@HOST'
```

If this fails with "Permission denied", **stop and ask the user for the
correct username and password** rather than writing elaborate PTY-based
workarounds. Time spent writing Python PTY scripts is better spent
asking for clarified credentials.

## Remote Execution Examples

### Single image generation
```bash
python3 scripts/run_workflow.py \
  --workflow workflows/flux_dev_txt2img.json \
  --args '{"prompt": "cyberpunk city at night", "seed": -1, "steps": 30}' \
  --host http://10.0.0.202:8188 \
  --output-dir ./outputs
```

### Batch with parallel sweeps
```bash
python3 scripts/run_batch.py \
  --workflow workflows/sdxl.json \
  --args '{"prompt": "abstract art"}' \
  --count 8 --randomize-seed --parallel 4 \
  --host http://10.0.0.202:8188 \
  --output-dir ./outputs/batch
```

### img2img with local source image uploaded to remote
```bash
python3 scripts/run_workflow.py \
  --workflow workflows/sdxl_img2img.json \
  --input-image image=./photo.png \
  --args '{"prompt": "oil painting style", "denoise": 0.65}' \
  --host http://10.0.0.202:8188 \
  --output-dir ./outputs
```

### Real-time WebSocket progress monitoring
```bash
python3 scripts/run_workflow.py \
  --workflow workflows/wan_t2v.json \
  --args '{"prompt": "a cat dancing", "frames": 81}' \
  --host http://10.0.0.202:8188 \
  --ws \
  --output-dir ./outputs
```

## Environment Variable

Set once in your shell profile to avoid typing `--host` every time:
```bash
export COMFYUI_HOST="http://10.0.0.202:8188"
```

All scripts in this skill respect `$COMFYUI_HOST` when `--host` is not
explicitly passed.

## Proxmox LXC Specifics

If the remote ComfyUI is in an unprivileged Proxmox LXC:
- Ensure the LXC has `features: nesting=1` if running comfy-cli inside it
- GPU passthrough requires `lxc.cgroup2.devices.allow` entries for the NVIDIA/AMD card
- Bridge networking (`net0: bridge=vmbr0`) should make it LAN-reachable from
  other LXCs on the same bridge
- If Hermes is in a different LXC that cannot reach `10.0.0.202`, check:
  - Proxmox host-level firewall rules
  - Whether both CTs share the same bridge (`vmbr0` vs `vmbr1`)
  - The LXC `net0` IP assignment (DHCP vs static)