hermes-skills/proxmox-lxc-deployment/SKILL.md

---
name: proxmox-lxc-deployment
description: "Deploy containerized services (Docker, Immich, etc.) on Proxmox LXCs. Covers storage, NFS, Docker setup, and the LXC-vs-VM mental model."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [proxmox, lxc, docker, deployment, nfs, storage, immich]
---

# Proxmox LXC Deployment

Deploy containerized services on Proxmox LXC containers. The key insight: **an LXC is not a VM.** Block devices, in-guest fstab, in-guest NFS mounts, and kernel module loading all work differently (or not at all). This skill captures the patterns that work.

## When to Use

- Deploying Docker-based services (Immich, SearXNG, etc.) in a Proxmox LXC
- Adding storage to an LXC
- Mounting external NFS/SMB shares for use inside an LXC
- Planning infrastructure that involves Proxmox containers

## LXC vs VM Mental Model

| Concern | VM | LXC |
|---------|----|-----|
| Add storage | Attach virtual disk → guest sees `/dev/sdX` → mkfs + mount | `pct set CTID -mpN pool:size,mp=/path` on host |
| fstab | Guest `/etc/fstab` works normally | Guest fstab is NOT processed; use host-side mount points |
| NFS mount | `mount -t nfs` inside guest works | Fails in unprivileged LXC; mount on host, bind-mount in |
| Docker | Works out of the box | Needs `nesting=1,keyctl=1` on the CT |
| Kernel modules | Guest can load modules | Shares host kernel; modules must be loaded on host |

## Storage: Adding a Disk to an LXC

**Wrong (VM pattern):**
```bash
# Inside the LXC — this does NOT work
mkfs.ext4 /dev/sdb
mount /dev/sdb /var/data
```

**Right (LXC pattern):**
```bash
# On the Proxmox HOST
pct set 9300 -mp0 local-lvm:200,mp=/var/immich-data
```

This creates a 200GB volume on `local-lvm` storage, formats it, and bind-mounts it at `/var/immich-data` inside the container. No guest-side mkfs, no fstab entry. The container sees it immediately (no reboot needed for new mount points, though a restart may be needed if the CT was running).

To pass through an existing host directory:
```bash
pct set 9300 -mp1 /host/path,mp=/container/path
# Read-only:
pct set 9300 -mp1 /host/path,mp=/container/path,ro=1
```

## NFS: Mounting a NAS Share for LXC Use

**Wrong:** `apt install nfs-common` + `mount -t nfs` inside the LXC. Fails with "operation not permitted" in unprivileged containers. This is a kernel-level restriction — NFS and CIFS are not `FS_USERNS_MOUNT` filesystems, so the mount syscall is refused inside a user namespace regardless of capabilities. Docker volume drivers with `type: nfs` opts also fail (same userns restriction). FUSE is also unavailable (no `/dev/fuse` in unprivileged LXC).

**Right:** Mount on the Proxmox host, then bind-mount into the LXC.

```bash
# On Proxmox HOST
mkdir -p /mnt/immich_nfs
mount -t nfs 10.0.0.50:/volume1/photo /mnt/immich_nfs

# Persistent: add to HOST /etc/fstab
# Use 'hard' not 'soft' for read-write mounts — soft returns EIO on timeout
# and Immich's job queue will mark assets as failed.
# 10.0.0.50:/volume1/photo /mnt/immich_nfs nfs vers=4.1,hard,_netdev,nofail 0 0

# Pass into LXC (read-write for uploads/thumbs, read-only for external library)
pct set 9300 -mp0 /mnt/immich_nfs,mp=/mnt/nas
# Read-only variant:
pct set 9300 -mp0 /mnt/immich_nfs,mp=/mnt/nas,ro=1
```

### Synology NFS + Unprivileged LXC: UID Mapping

Unprivileged LXC maps container root (uid 0) to host uid 100000. Synology's default `root_squash` will return EACCES on every write. On the Synology DSM, set the NFS share's squash to **Map all users to admin** with `anonuid=100000,anongid=100000`. This is the single most common failure in Proxmox+Synology NFS threads.

DSM path: Control Panel → Shared Folder → select share → Edit → NFS Permissions → Create rule:
- Client: `10.0.0.177`
- Squash: Map all users to admin
- Security: sys
- Enable asynchronous: yes

### NFS Mount Options by Use Case

| Use case | Options | Why |
|----------|---------|-----|
| Read-only external library | `ro,vers=4.1,hard,_netdev,nofail` | `hard` prevents EIO on transient blips |
| Read-write uploads/thumbs | `vers=4.1,hard,_netdev,nofail` | `soft` returns EIO → Immich marks assets failed |
| Never use `soft` for any Immich data path | — | Job queue corruption on timeout |

## Docker in an LXC

### Required CT Features

Before installing Docker, enable nesting and keyctl on the container:

```bash
# On Proxmox HOST
pct set 9300 -features nesting=1,keyctl=1
pct reboot 9300
```

Without `nesting=1`, Docker fails to create containers. Without `keyctl=1`, Docker's overlay2 storage driver may fail.

### Docker Storage Location

The LXC root disk is often small (20-40GB). Docker images, layers, and logs will fill it. Point Docker's data-root at a dedicated mount point BEFORE installing Docker:

```bash
# Inside LXC, before apt install docker-ce
mkdir -p /var/immich-data/docker
cat > /etc/docker/daemon.json << 'EOF'
{"data-root":"/var/immich-data/docker"}
EOF
```

After Docker is installed, verify the storage driver:
```bash
docker info | grep "Storage Driver"  # must say overlay2, not vfs
```

If the LXC root is ZFS-backed and Docker's data-root is also on ZFS, Docker may fall back to `vfs` (which copies every layer in full — 3GB of images becomes 15GB+). Using an ext4 mount point for `data-root` avoids this.

### Docker Install (Debian)

```bash
apt update && apt install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list
apt update && apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
usermod -aG docker $USER
```

## Postgres in Docker: Don't Pre-create the Data Directory

When using a bind mount for Postgres data, do NOT pre-create and chown the directory. The Postgres container runs as uid 999 and initializes an empty PGDATA itself. Pre-creating with `chown 1000:1000` causes `initdb: could not change permissions` on first boot.

**Wrong:**
```bash
mkdir -p /var/data/postgres
chown -R 1000:1000 /var/data/postgres  # breaks initdb
```

**Right:**
```bash
# Either don't create it at all (Docker will)
# OR create it but don't chown:
mkdir -p /var/data/postgres
# Let the container handle ownership
```

## Pre-flight Checks

Before deploying a service in an LXC, verify:

```bash
# CPU features (needed by vector DBs, ML, etc.)
lscpu | grep -o 'avx2\|sse4_2'

# Docker storage driver
docker info | grep "Storage Driver"

# Mount points visible
df -h | grep -E '/var/|/mnt/'

# NFS readability (if applicable)
ls -l /mnt/photos && head -c1 /mnt/photos/some-file.jpg >/dev/null && echo "NFS OK"
```

## Pitfalls

- **Treating LXC like a VM for storage.** Use `pct set -mpN`, not guest-side mkfs/fstab.
- **NFS inside unprivileged LXC.** Always mount on host, bind-mount in. No workaround exists — NFS/CIFS are not `FS_USERNS_MOUNT` filesystems.
- **Forgetting `nesting=1,keyctl=1`.** Docker silently fails without them. If Docker is already running, both are already set — skip the check.
- **Docker data-root on small root disk.** Set `data-root` in daemon.json before first start.
- **Pre-chowning Postgres data dir.** Let the container initialize it (runs as uid 999).
- **ZFS + Docker = vfs storage driver.** Ensure Docker's data-root is on ext4/xfs, not ZFS.
- **`pct` not `qm`.** `qm` is for VMs, `pct` is for containers. Using the wrong one silently does nothing or errors confusingly.
- **Over-provisioning storage.** Not every service needs a dedicated virtual disk. For Immich: uploads/thumbs can live on NFS (widely used, supported). Only Postgres MUST be local. If root disk is tight, `pct resize CTID rootfs +32G` is one command, online, no reboot — cheaper than a new virtual disk.
- **PostgreSQL on NFS.** Hard blocker, not a performance warning. NFS hiccup during checkpoint → fsync error → Postgres PANICs or corrupts data silently. Immich docs: "Network shares are not supported for the database." Keep Postgres on local disk always.
- **Synology NFS UID mapping with unprivileged LXC.** Container root (uid 0) maps to host uid 100000. Synology's default `root_squash` returns EACCES on every write. Set squash to "Map all users to admin" with `anonuid=100000,anongid=100000`.
- **Agent cannot modify Proxmox host.** The operator's standing rule: all `pct set`, fstab edits, disk creation, and bind mounts must be done by the user manually. Plans must list these as operator steps, not agent-executed commands.

## References

- `references/immich-plan-example.md` — Full worked example: Immich on LXC with Synology NAS external library (Jul 2026, v3.0.3)