hermes-skills/proxmox-lxc-deployment/SKILL.md

name, description, version, author, license, platforms, metadata

name	description	version	author	license	platforms	metadata
proxmox-lxc-deployment	Deploy containerized services (Docker, Immich, etc.) on Proxmox LXCs. Covers storage, NFS, Docker setup, and the LXC-vs-VM mental model.	1.0.0	Hermes Agent	MIT	linux	hermes tags proxmox lxc docker deployment nfs storage immich

Proxmox LXC Deployment

Deploy containerized services on Proxmox LXC containers. The key insight: an LXC is not a VM. Block devices, in-guest fstab, in-guest NFS mounts, and kernel module loading all work differently (or not at all). This skill captures the patterns that work.

When to Use

• Deploying Docker-based services (Immich, SearXNG, etc.) in a Proxmox LXC
• Adding storage to an LXC
• Mounting external NFS/SMB shares for use inside an LXC
• Planning infrastructure that involves Proxmox containers

LXC vs VM Mental Model

Concern	VM	LXC
Add storage	Attach virtual disk → guest sees /dev/sdX → mkfs + mount	pct set CTID -mpN pool:size,mp=/path on host
fstab	Guest /etc/fstab works normally	Guest fstab is NOT processed; use host-side mount points
NFS mount	mount -t nfs inside guest works	Fails in unprivileged LXC; mount on host, bind-mount in
Docker	Works out of the box	Needs nesting=1,keyctl=1 on the CT
Kernel modules	Guest can load modules	Shares host kernel; modules must be loaded on host

Storage: Adding a Disk to an LXC

Wrong (VM pattern):

# Inside the LXC — this does NOT work
mkfs.ext4 /dev/sdb
mount /dev/sdb /var/data

Right (LXC pattern):

# On the Proxmox HOST
pct set 9300 -mp0 local-lvm:200,mp=/var/immich-data

This creates a 200GB volume on local-lvm storage, formats it, and bind-mounts it at /var/immich-data inside the container. No guest-side mkfs, no fstab entry. The container sees it immediately (no reboot needed for new mount points, though a restart may be needed if the CT was running).

To pass through an existing host directory:

pct set 9300 -mp1 /host/path,mp=/container/path
# Read-only:
pct set 9300 -mp1 /host/path,mp=/container/path,ro=1

NFS: Mounting a NAS Share for LXC Use

Wrong: apt install nfs-common + mount -t nfs inside the LXC. Fails with "operation not permitted" in unprivileged containers. This is a kernel-level restriction — NFS and CIFS are not FS_USERNS_MOUNT filesystems, so the mount syscall is refused inside a user namespace regardless of capabilities. Docker volume drivers with type: nfs opts also fail (same userns restriction). FUSE is also unavailable (no /dev/fuse in unprivileged LXC).

Right: Mount on the Proxmox host, then bind-mount into the LXC.

# On Proxmox HOST
mkdir -p /mnt/immich_nfs
mount -t nfs 10.0.0.50:/volume1/photo /mnt/immich_nfs

# Persistent: add to HOST /etc/fstab
# Use 'hard' not 'soft' for read-write mounts — soft returns EIO on timeout
# and Immich's job queue will mark assets as failed.
# 10.0.0.50:/volume1/photo /mnt/immich_nfs nfs vers=4.1,hard,_netdev,nofail 0 0

# Pass into LXC (read-write for uploads/thumbs, read-only for external library)
pct set 9300 -mp0 /mnt/immich_nfs,mp=/mnt/nas
# Read-only variant:
pct set 9300 -mp0 /mnt/immich_nfs,mp=/mnt/nas,ro=1

Synology NFS + Unprivileged LXC: UID Mapping

Unprivileged LXC maps container root (uid 0) to host uid 100000. Synology's default root_squash will return EACCES on every write. On the Synology DSM, set the NFS share's squash to Map all users to admin with anonuid=100000,anongid=100000. This is the single most common failure in Proxmox+Synology NFS threads.

DSM path: Control Panel → Shared Folder → select share → Edit → NFS Permissions → Create rule:

• Client: 10.0.0.177
• Squash: Map all users to admin
• Security: sys
• Enable asynchronous: yes

NFS Mount Options by Use Case

Use case	Options	Why
Read-only external library	ro,vers=4.1,hard,_netdev,nofail	hard prevents EIO on transient blips
Read-write uploads/thumbs	vers=4.1,hard,_netdev,nofail	soft returns EIO → Immich marks assets failed
Never use soft for any Immich data path	—	Job queue corruption on timeout

Docker in an LXC

Required CT Features

Before installing Docker, enable nesting and keyctl on the container:

# On Proxmox HOST
pct set 9300 -features nesting=1,keyctl=1
pct reboot 9300

Without nesting=1, Docker fails to create containers. Without keyctl=1, Docker's overlay2 storage driver may fail.

Docker Storage Location

The LXC root disk is often small (20-40GB). Docker images, layers, and logs will fill it. Point Docker's data-root at a dedicated mount point BEFORE installing Docker:

# Inside LXC, before apt install docker-ce
mkdir -p /var/immich-data/docker
cat > /etc/docker/daemon.json << 'EOF'
{"data-root":"/var/immich-data/docker"}
EOF

After Docker is installed, verify the storage driver:

docker info | grep "Storage Driver"  # must say overlay2, not vfs

If the LXC root is ZFS-backed and Docker's data-root is also on ZFS, Docker may fall back to vfs (which copies every layer in full — 3GB of images becomes 15GB+). Using an ext4 mount point for data-root avoids this.

Docker Install (Debian)

apt update && apt install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list
apt update && apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
usermod -aG docker $USER

Postgres in Docker: Don't Pre-create the Data Directory

When using a bind mount for Postgres data, do NOT pre-create and chown the directory. The Postgres container runs as uid 999 and initializes an empty PGDATA itself. Pre-creating with chown 1000:1000 causes initdb: could not change permissions on first boot.

Wrong:

mkdir -p /var/data/postgres
chown -R 1000:1000 /var/data/postgres  # breaks initdb

Right:

# Either don't create it at all (Docker will)
# OR create it but don't chown:
mkdir -p /var/data/postgres
# Let the container handle ownership

Pre-flight Checks

Before deploying a service in an LXC, verify:

# CPU features (needed by vector DBs, ML, etc.)
lscpu | grep -o 'avx2\|sse4_2'

# Docker storage driver
docker info | grep "Storage Driver"

# Mount points visible
df -h | grep -E '/var/|/mnt/'

# NFS readability (if applicable)
ls -l /mnt/photos && head -c1 /mnt/photos/some-file.jpg >/dev/null && echo "NFS OK"

Pitfalls

• Treating LXC like a VM for storage. Use pct set -mpN, not guest-side mkfs/fstab.
• NFS inside unprivileged LXC. Always mount on host, bind-mount in. No workaround exists — NFS/CIFS are not FS_USERNS_MOUNT filesystems.
• Forgetting nesting=1,keyctl=1. Docker silently fails without them. If Docker is already running, both are already set — skip the check.
• Docker data-root on small root disk. Set data-root in daemon.json before first start.
• Pre-chowning Postgres data dir. Let the container initialize it (runs as uid 999).
• ZFS + Docker = vfs storage driver. Ensure Docker's data-root is on ext4/xfs, not ZFS.
• pct not qm. qm is for VMs, pct is for containers. Using the wrong one silently does nothing or errors confusingly.
• Over-provisioning storage. Not every service needs a dedicated virtual disk. For Immich: uploads/thumbs can live on NFS (widely used, supported). Only Postgres MUST be local. If root disk is tight, pct resize CTID rootfs +32G is one command, online, no reboot — cheaper than a new virtual disk.
• PostgreSQL on NFS. Hard blocker, not a performance warning. NFS hiccup during checkpoint → fsync error → Postgres PANICs or corrupts data silently. Immich docs: "Network shares are not supported for the database." Keep Postgres on local disk always.
• Synology NFS UID mapping with unprivileged LXC. Container root (uid 0) maps to host uid 100000. Synology's default root_squash returns EACCES on every write. Set squash to "Map all users to admin" with anonuid=100000,anongid=100000.
• Agent cannot modify Proxmox host. The operator's standing rule: all pct set, fstab edits, disk creation, and bind mounts must be done by the user manually. Plans must list these as operator steps, not agent-executed commands.

References

• references/immich-plan-example.md — Full worked example: Immich on LXC with Synology NAS external library (Jul 2026, v3.0.3)