How a Template Is Built¶

This document explains the full sequence pve-template-builder follows when given a specification file, and how each step is actually executed underneath. For a field-by-field reference of the specification file itself, see the Specification File Reference. For the reasoning behind non-obvious choices, see Design Notes.

pve-template-builder must run as root on a Linux host: creating loop devices, mounting filesystems, and chrooting all require it. It shells out to a number of standard system tools rather than reimplementing them.

Requirements¶

On Debian/Ubuntu, install them with:

apt install debootstrap gdisk dosfstools e2fsprogs qemu-utils apt-move apt-utils

Installing pve-template-builder itself from the .deb published at Releases does this automatically, via the package's own apt dependencies — see Quick start.

This covers the default ext4 root filesystem; for xfs/btrfs, also install xfsprogs/btrfs-progs respectively. apt-move/apt-utils are only needed when builder.bootstrap.debootstrap.cache is set. Building an Ubuntu-based image from a Debian build host (or vice versa) also needs the other distribution's keyring package (ubuntu-keyring or debian-archive-keyring) — both debootstrap and the local mirror need it to verify what they download from a mirror that isn't the host's own.

Tool	Package	Used for
`debootstrap`	`debootstrap`	bootstrapping the base OS
`sgdisk`	`gdisk`	GPT partitioning
`qemu-nbd`, `modprobe`, `udevadm`	`qemu-utils`, `kmod`, `udev`	attaching the qcow2 image as an nbd device (needs the `nbd` kernel module)
`mkfs.vfat`	`dosfstools`	formatting the EFI System Partition
`mkfs.ext4`	`e2fsprogs`	formatting the root partition (default filesystem)
`mkfs.xfs`	`xfsprogs`	formatting the root partition (if `disk.filesystem: xfs`)
`mkfs.btrfs`	`btrfs-progs`	formatting the root partition (if `disk.filesystem: btrfs`)
`mount`, `umount`, `chroot`	`util-linux`, `coreutils`	mounting partitions and running build steps
`fstrim`	`util-linux`	discarding unused blocks (deallocated directly in the qcow2 file via nbd discard passthrough)
`qemu-img`	`qemu-utils`	creating the qcow2 image, and compressing it afterward if `--compress` is passed
`apt-move`	`apt-move`	sorting downloaded packages into the local mirror used by `builder.bootstrap.debootstrap.cache`
`apt-ftparchive`	`apt-utils`	generating that local mirror's package index
`debian-archive-keyring`/`ubuntu-keyring`	same	verifying a Debian/Ubuntu mirror's signature, when building for the other distribution than the build host's own

PVE API token setup¶

template.pve.api-token (or PVE_API_TOKEN) needs a PVE API token, in the full user@realm!tokenid=secret form — see template.pve. Create one:

pveum user token add root@pam pve-template-builder

A freshly created token has no permissions of its own by default (PVE calls this Privilege Separation) — you must grant it a role explicitly, or every real operation gets rejected even though the token check itself passes. See Why the ACL step is easy to miss for what that failure looks like if you skip it.

Minimal privileges¶

Privilege	Needed for	Scope
`VM.Audit`	checking whether `vmid` already exists	`/vms/<vmid>`
`VM.Allocate`	creating the VM; deleting an existing one first, if `replace` (the default); converting it into a template	`/vms/<vmid>`
`VM.Config.HWType`	`template.pve.parameters` keys like `machine`, `vga`, `scsihw`, `serialN`/`usbN`	`/vms/<vmid>`
`VM.Config.CPU`	`template.pve.parameters` keys like `cores`, `sockets`, `cpu`	`/vms/<vmid>`
`VM.Config.Memory`	the `memory` key	`/vms/<vmid>`
`VM.Config.Network`	`netN`/`ipconfigN` keys	`/vms/<vmid>`
`VM.Config.Disk`	the `boot` key	`/vms/<vmid>`
`VM.Config.Options`	`template.pve.parameters` keys like `bios`, `tags`, `agent`, and the auto-filled `description`	`/vms/<vmid>`
`Datastore.AllocateTemplate`	uploading the built image	`/storage/<import-storage>`
`Datastore.Allocate`	removing the uploaded image afterward	`/storage/<import-storage>`
`Datastore.AllocateSpace`	creating the VM's `scsi0`/`efidisk0` disks	`/storage/<storage>`

(import-storage and storage are often the same PVE storage — see template.pve — in which case it just needs every Datastore.* privilege above on that one storage. The six VM.Config.* privileges are only needed for the specific template.pve.parameters keys noted above — a spec that doesn't set a given key doesn't need the matching privilege — but PVE checks them on VM creation, not just when reconfiguring an existing VM, which is easy to miss; see How the minimal privilege set was derived for the full key-to-privilege mapping and how this was found. A spec that sets cicustom, cipassword, citype, ciuser, nameserver, searchdomain, or sshkeys directly also needs VM.Config.Cloudinit.)

One role covering every privilege above, applied at whichever of the VM and storage paths actually apply, is simplest. Create it, and grant it on the storage paths, once:

pveum role add template-builder --privs "VM.Audit VM.Allocate VM.Config.HWType VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.Disk VM.Config.Options Datastore.AllocateTemplate Datastore.Allocate Datastore.AllocateSpace"
pveum acl modify /storage/<storage> -token 'root@pam!pve-template-builder' -role template-builder
pveum acl modify /storage/<import-storage> -token 'root@pam!pve-template-builder' -role template-builder

(The two acl modify lines above are only needed once per storage — skip the second if storage and import-storage are the same one.)

The /vms/<vmid> grant is different: it needs running once per vmid, and before pve-template-builder ever runs against that spec — not after, even though the tool is the one creating the VM:

pveum acl modify /vms/<vmid> -token 'root@pam!pve-template-builder' -role template-builder

This works even though vmid doesn't exist yet — see Why the /vms/<vmid> grant must come before the first build for why this ordering matters and why it works.

If you'd rather not deal with ACLs at all, creating the token without privilege separation makes it inherit the user's own rights directly — for root@pam specifically, that means full access with no acl modify step needed:

pveum user token add root@pam pve-template-builder --privsep 0

Command-line flags¶

pve-template-builder [flags] <spec.yaml>

Flag	Effect
`--work-dir <dir>`	Build in `<dir>` instead of a fresh temporary directory.
`--keep`	Don't remove the work directory (built qcow2, mountpoints) on success. Useful for inspecting a build or reusing `--work-dir`.
`--skip-upload`	Build the image but don't talk to the PVE API at all — no upload, no VM create/delete.
`--insecure-skip-tls-verify`	Don't verify the PVE API server's TLS certificate.
`--inspect-interactive`	After the last build step, open an interactive shell in the chroot (see step 8) before any further cleanup, for manually inspecting or fixing up the target filesystem.
`--compress`	Run an extra `qemu-img convert -O qcow2 -c` pass over the built image before publishing it, trading build time for a smaller image (see step 10). Without it, the image is left as built directly on the qcow2 file — sparsified by `fstrim`, but not compressed.
`--dry-run`	Parse and validate the specification file, then exit without doing anything (no root required).
`--verbose`, `-v`	Show low-level diagnostic output — raw command invocations, PVE API request tracing — in the live log and in the failure report, instead of only in `pve-template-builder.log`. Also exports `VERBOSE=true` to `builder.steps` `run` steps and the `--inspect-interactive` shell (see step 8).

Build progress output¶

Every line — including the low-level diagnostic detail --verbose gates, whether or not it was actually passed — is always captured in full to pve-template-builder.log in the current directory (truncated fresh at the start of each run), regardless of what's shown on screen. A one-line header (PVE Template Builder — <spec.yaml>) is printed first; what follows depends on whether stdout is a terminal:

Terminal: the full task list is shown upfront, each task's/step's marker updating in place as the build progresses (pending → running → done/failed), with a scrolling log window beneath it showing recent output. On success, the log window is removed, leaving the completed task list plus a short summary (template name, VMID, node, PVE API endpoint) — unless --skip-upload was given. On failure, the task list is left in place (✗ on whatever failed) with a static report below it: the failing task's/step's name, the error, and the last 20 log lines. See Build progress output, in detail for the exact layout/scrolling mechanics.
Not a terminal (piped or redirected — the common case in CI): each task prints as <name>... then ok (or FAILED plus the same static report as above) once it completes; "Image Configuration Steps" instead prints its own name and each of its steps the same way, indented. The same success summary is printed at the end, unless --skip-upload was given.

The build sequence¶

Everything below happens inside a single work directory (a fresh os.MkdirTemp unless --work-dir is given), removed on success unless --keep is passed.

1. Create the qcow2 disk image¶

qemu-img create -f qcow2 <work-dir>/<name>.qcow2 <disk.size in bytes>

<name> is template.pve.name with anything other than letters, digits, - and _ replaced by _. The image is built directly in qcow2 format — there is no separate raw-image stage or later conversion step (unless --compress is passed; see step 10).

2. Attach it as an nbd device¶

modprobe nbd max_part=16 ensures the kernel's nbd driver is loaded with partition-scanning support, then qemu-nbd --connect=/dev/nbdN --discard=unmap --detect-zeroes=unmap -f qcow2 <image> attaches the image to a free /dev/nbdN device — tried in order starting from /dev/nbd0, since qemu-nbd (unlike losetup --find) has no way to pick a free device itself. --discard=unmap passes filesystem discards (see step 10) straight through to the qcow2 file, deallocating clusters directly instead of just zeroing them.

3. Partition it (GPT + UEFI)¶

Only disk.format: gpt-uefi is supported. Partitioning is done with sgdisk, run against the nbd device (/dev/nbdN) now that it's attached:

sgdisk -o — initialize a fresh, empty GPT.
sgdisk -n 1:1MiB:+512MiB -t 1:ef00 -c 1:EFI — a 512 MiB EFI System Partition, starting 1 MiB in for alignment. This size is currently fixed, not configurable from the spec.
sgdisk -n 2:0:0 -t 2:8300 -c 2:root — a second partition, type "Linux filesystem", using the rest of the disk.

udevadm settle plus a poll loop (up to 10s) then waits for /dev/nbdNp1 and /dev/nbdNp2 to actually appear before continuing.

4. Format the partitions¶

EFI partition: mkfs.vfat -F 32 -n EFI <partition>
Root partition: mkfs.<filesystem> -L root <partition>, where <filesystem> is disk.filesystem (default ext4; xfs and btrfs are also supported).

5. Mount the root partition¶

mount <root-partition> <work-dir>/root. Everything from here on happens under this mountpoint (referred to below as <target>).

6. Bootstrap the OS (`debootstrap`)¶

If builder.bootstrap.debootstrap.cache is set, a local package mirror is built (or refreshed) there, and debootstrap installs from file://<cache> instead of mirror — see Bootstrapping with a local package mirror for how that cache actually works.
debootstrap [flags] <suite> <target> <mirror> runs, populating <target> with the base system. If it fails, the contents of <target>/debootstrap/debootstrap.log are printed (when that file exists): debootstrap's own failure output often just points at this log's path rather than including its content, even though the log usually has the actual underlying error.
A base apt source is written into <target> (see Base apt source Signed-By logic), since debootstrap itself never sets one up.

If builder.bootstrap.debootstrap.preset is set, all of the above already happened using the preset's own suite/mirror/arch/include defaults (any of these given explicitly in the spec still win), and step 8 below runs an extra step first — see Presets.

7. Mount the EFI partition and set up the chroot¶

mount <efi-partition> <target>/boot/efi — this happens after debootstrap, since <target>/boot doesn't exist until the base system is unpacked.
Bind mounts into <target>: mount --bind /dev, mount --bind /dev/pts, mount -t proc proc, mount -t sysfs sys, for /dev, /dev/pts, /proc, /sys respectively. /dev/pts needs its own bind mount on top of /dev's, since a plain (non-recursive) bind of /dev doesn't pull in filesystems mounted inside it (the host's devpts instance) — without it, a build step that needs a pty (e.g. running ssh or sudo -i) would fail to get one.
<target>/etc/resolv.conf is overwritten with the host's own /etc/resolv.conf (removing it first, since it may be a symlink), so DNS resolution works for the apt-get/run calls that follow.

8. Run the build steps¶

Each entry in builder.steps runs in order; see the Specification File Reference for the field reference of each step type. Before running, each step is logged as step <n>/<total> (<kind>): <name>, where <name> is the step's own name if set, otherwise a kind-specific default (see builder.steps); the same (<kind>: <name>) shows up in the error if the step fails. (An include step never reaches this stage at all: it's expanded — replaced by the steps it names, recursively — back when the spec file is loaded and validated, long before the build itself starts; see include.) How each of the remaining kinds is actually executed:

copy: file (resolved relative to the spec file's directory, unless already absolute) is copied to <target>/<to>, recursively if it's a directory; or url is downloaded (plain HTTP GET) to <target>/<to>; or (when neither is set) contents is written to <target>/<to> as a single file, defaulting to mode 0644 instead of inheriting a source file's mode. If unpack is true, the file/url source is instead extracted into <target>/<to> as an archive (tar+gzip, tar+bzip2, or zip, detected from its content). owner/group (name or numeric id, resolved against <target>'s own /etc/passwd//etc/group) are then applied recursively to every entry copied/extracted/written. mode (octal) is applied recursively too, but for file/url its x bit is computed smartly per owner/group/other category rather than taken literally: a category only gets x if mode grants it r and the corresponding source entry already had some execute bit set. contents, having no source entry, gets mode applied literally.
apt-source: if gpg-key is set at all, its file/contents/url (copied, written literally, or fetched with a plain HTTP GET, respectively) is written to <target>/etc/apt/keyrings/<gpg-key.filename>. The source/sources content is written as-is to <target>/etc/apt/sources.list.d/<name>.sources (deb822 format).
package: runs apt-get update followed by apt-get install -y <packages...> inside the chroot.
run: with contents (or the plain-string form), the command string is run as-is inside the chroot. With file/url instead, the named (or downloaded) executable is copied into <target>/tmp/, made executable, run directly by path (not through a shell), and removed again afterwards regardless of whether it succeeded.

All of copy, apt-source, package and run's contents/plain-string form execute via:

chroot <target> /bin/sh -e -c '<command>'

-e means the shell exits as soon as any command in a multi-line run step fails, rather than only the last command's exit status determining whether the step succeeded. A run step's file/url form instead execs the copied-in file directly, with no shell involved.

Every command run inside the chroot — this and file/url's exec alike — gets a fixed environment, not the environment pve-template-builder itself was started with: HOME=/root, PATH=/usr/sbin:/usr/bin:/sbin:/bin, LANG=C, DEBIAN_FRONTEND=noninteractive, and nothing else, except VERBOSE=true when --verbose/-v was passed — the one deliberate leak-through, there so a run step can opt into noisier output of its own (e.g. apt-get without -q) when the outer build is already running verbose. None of the build host's own environment variables (proxies, credentials, whatever happens to be set in the invoking shell) carry over, so a step's behavior only ever depends on what the spec itself sets up and this one flag. --inspect-interactive's shell (see below) gets the same fixed environment, plus the host's own TERM (otherwise the terminal wouldn't work).

(copy itself is done directly on the host filesystem, not inside the chroot — only its ownership/mode application needs the target's /etc/passwd//etc/group, which it reads directly rather than executing anything in the chroot.)

If --inspect-interactive was given, an interactive shell (bash if the target has it, otherwise /bin/sh) is opened in the chroot right here, connected to the terminal's own stdin/stdout/stderr. This happens whether or not a build step failed, so the target filesystem can be inspected in whatever state a failing step left it in. Exiting the shell with status 0 lets the build continue as normal (or, if a step had already failed, still fail with that step's error). Exiting with any non-zero status aborts the build right there, even if every step had otherwise succeeded — the only way an interactive session can signal "this build shouldn't continue". If stdin or stdout isn't a terminal, there's no way to actually interact with a shell, so this is skipped (a warning is logged) and the build continues as if the flag hadn't been given.

9. Clean apt caches¶

As a final, implicit step — not something any builder.steps entry needs to ask for — apt's own temporary and cache files inside the target are removed, since they're pure build-time overhead that would otherwise ship in the image:

apt-get clean && rm -rf /var/lib/apt/lists/* && mkdir -p /var/lib/apt/lists/partial

10. Trim, unmount, and detach the image¶

Before anything is unmounted, fstrim <target> discards unused blocks on the root filesystem. Since the nbd device was attached with --discard=unmap (step 2), these discards pass straight through and deallocate the corresponding clusters in the qcow2 file directly, skipping space freed by package removals, deleted apt caches, and the like. A failure here (e.g. the nbd device not supporting discard) is logged as a warning but doesn't fail the build — it only affects the output image's size, not its correctness.

Before unmounting anything, every process still running with its root inside the chroot — something a run step left behind, e.g. a backgrounded daemon or anything still attached to a pty via the /dev/pts bind mount — is killed (SIGKILL, found by scanning /proc/*/root), since a process still holding files open there would otherwise make the unmounts that follow fail with "target is busy". This is best-effort (a slow-to-die process gets a brief grace period, then the unmounts proceed regardless); the chroot bind mounts, the EFI partition, and the root partition are then unmounted (in reverse order), and the nbd device is detached (qemu-nbd --disconnect). The qcow2 file at <work-dir>/<name>.qcow2 is now the finished image — there is no separate conversion step.

If --compress was passed, one more pass runs at this point:

qemu-img convert -O qcow2 -c <name>.qcow2 <name>-compressed.qcow2

(-c compresses the output; this rewrites the whole image, the same cost the old raw→qcow2 conversion always paid, in exchange for a smaller file.) Everything downstream (--skip-upload, publishing) then uses <name>-compressed.qcow2 instead.

If --skip-upload was given, the build stops here and the resulting qcow2 path is logged.

11. Publish to Proxmox VE¶

See template.pve for the spec fields involved. In order:

Check the API token: the token's format is checked locally first (user@realm!tokenid=secret, not just the secret half), then against GET /access/permissions. Done before anything else — see Why the token is checked before anything expensive for why.
Check for an existing VM: GET /nodes/{node}/qemu/{vmid}/config. A non-2xx response is treated as "doesn't exist".
Delete it if present: if replace is true (the default), DELETE /nodes/{node}/qemu/{vmid}?purge=1&destroy-unreferenced-disks=1, waiting for the resulting task to finish. If replace is false and the VM exists, the build fails here instead.
Upload the image: a multipart POST to /nodes/{node}/storage/{storage}/upload with content=import and the qcow2 file, where {storage} is import-storage. The task is awaited, and the resulting volume id is assumed to be <import-storage>:import/<filename>. The request always carries an explicit Content-Length instead of chunked transfer encoding, which PVE's pveproxy rejects for this endpoint.
Create the VM: POST /nodes/{node}/qemu with vmid, name, every key/value from parameters (booleans become 1/0), plus two automatically-added disks, both on storage (deliberately not import-storage — why):
- scsi0=<storage>:0,import-from=<volid>[,<scsi-disk options>] — the built image, imported from the volume just uploaded to import-storage, plus every key/value in scsi-disk appended after it (sorted by key, booleans as 1/0).
- efidisk0=<storage>:1,efitype=4m,pre-enrolled-keys=0[,<efi-disk overrides>] — see Why efidisk0 defaults to pre-enrolled-keys=0 for why pre-enrolled-keys=0 is the default.
The task is awaited before continuing.
Convert it into a template: POST /nodes/{node}/qemu/{vmid}/template — see Why converting to a template is a separate API call for why this is a separate, required call, not implied by parameters.template: true. The task (if the API returns one — this call may also complete synchronously) is awaited before the build reports success.
Remove the uploaded image: DELETE /nodes/{node}/storage/{import-storage}/content/{volid}, where {volid} is the same volume id step 4 uploaded and step 5 imported from — dead weight once step 5 has copied its content into the VM's own disk. Best-effort: a failure here is only logged as a warning, not reported as a failed build.

All PVE API calls target template.pve.api-endpoint, or the PVE_URL environment variable if set (which takes precedence), and send Authorization: PVEAPIToken=<token> where <token> is template.pve.api-token, or the PVE_API_TOKEN environment variable if set (which also takes precedence).