Design Notes

The "why" behind decisions in How a Template Is Built and the Specification File Reference — verified findings, rejected alternatives, and the reasoning behind non-obvious choices. Neither of those two documents assumes you've read this one; this one assumes you've read (or are about to read) them, and exists so their own text can stay focused on what the tool does rather than why.

PVE token permissions

Why the ACL step is easy to miss

PVE API tokens are created with Privilege Separation enabled by default, meaning the token starts out with no permissions of its own — even a token for root@pam does not inherit root's rights automatically. Without an explicit role grant, the token still authenticates fine (so the token check passes), but every actual operation gets rejected. The image upload in particular fails in a very confusing way when the token lacks the needed rights: instead of a clean 403, it can come back from pveproxy as a bogus "No space left on device" once the file is more than a few hundred KB (a real PVE-side bug in how it aborts an in-progress multipart upload, not anything this tool does).

How the minimal privilege set was derived

Most of the privileges in PVE API token setup's table were read directly out of each API call's own "Permissions" entry in PVE's API viewer (its apidoc.js data file specifically, since the viewer itself doesn't render without JavaScript). That's accurate for the calls it documents at all (VM.Audit, VM.Allocate, the Datastore.* privileges), but it turns out to be incomplete for VM creation itself: the API viewer's "Permissions" entry for POST /nodes/{node}/qemu only mentions VM.Allocate, but qemu-server's actual handler for that call (PVE::API2::Qemu's create_vm) additionally runs every key in the request body through the same per-parameter permission check used by the PUT (reconfigure) endpoint, $check_vm_modify_config_perm — this applies just as much to a brand-new VM's initial config as to editing an existing one. This was found the hard way, from a real 403 on VM creation (Permission check failed (/vms/<vmid>, VM.Config.HWType)), not predicted from the API viewer.

That check buckets each config key into one of several privileges, all scoped to /vms/<vmid> same as VM.Allocate:

Privilege	Keys
VM.Config.HWType	acpi, hotplug, kvm, machine, scsihw, smbios1, tablet, vga, watchdog, audio0, serialN/usbN set to socket/spice
VM.Config.CPU	cores, cpu, cpulimit, cpuunits, numa, smp, sockets, vcpus
VM.Config.Memory	memory, balloon, shares
VM.Config.Network	netN, ipconfigN
VM.Config.Disk	boot, bootdisk, vmstatestorage
VM.Config.Options	agent, autostart, bios, description, keyboard, localtime, migrate_downtime, migrate_speed, name, onboot, ostype, protection, reboot, startdate, tdf, template, tags (startup needs Sys.Modify instead)
VM.Config.Cloudinit (or VM.Config.Network)	cicustom, cipassword, citype, ciuser, nameserver, searchdomain, sshkeys

Actual disk assignments (scsi0, efidisk0, and any other template.pve.parameters disk key) aren't in any of these buckets — they go through check_storage_access instead, which is what the Datastore.* privileges below actually cover. Which of the table above a given spec needs depends entirely on which keys template.pve.parameters sets; in practice every one of the example specs touches VM.Config.HWType, VM.Config.CPU, VM.Config.Memory, VM.Config.Network, VM.Config.Disk, and VM.Config.Options at once (machine/vga/scsihw/serial0, cores/cpu, memory, net0, boot, and bios/tags/agent/the auto-filled description, respectively), so the role below just grants all of them — see One role covering every privilege for why that's harmless even where a given spec doesn't need one of them. VM.Config.Cloudinit is the only one worth leaving out by default: it's only needed if template.pve.parameters sets cicustom, cipassword, citype, ciuser, nameserver, searchdomain, or sshkeys directly (ipconfigN alone doesn't need it — that one's covered by VM.Config.Network already).

If net0's bridge is managed through PVE's SDN subsystem specifically (not a plain, unmanaged Linux bridge like the example specs' vmbr0), the token additionally needs SDN.Use on it.

Datastore.Allocate (needed to remove the uploaded image once the VM's own disk has imported its content) is a slightly misleading name here: PVE reuses the same privilege for creating/removing a storage definition itself. Scoped to /storage/<import-storage> rather than /storage, it only ever means the former in this tool's case.

One role covering every privilege is simplest to set up: PVE only ever checks the privileges relevant to what's actually being done at a given path, so a role carrying privileges that don't apply there is harmless — no need for three narrowly-tailored roles just because the privileges apply to three different kinds of path.

The example role is named template-builder, not pve-template-builder: pveum rejects role IDs starting with pve (case-insensitively) — that namespace is reserved for PVE's own built-in roles like PVEVMAdmin. The token itself (root@pam!pve-template-builder) is unaffected, since token IDs aren't under that restriction.

Why the /vms/<vmid> grant must come before the first build

This works even though vmid doesn't exist yet: PVE's ACLs are just path-string rules, entirely independent of whether a resource currently exists at that path — this is exactly how you pre-authorize a VMID before creating it, and the only way this tool can ever create one at all, since it's the one creating it.

What doesn't work is the reverse order: this tool's very first PVE API call for any build, before it creates or touches anything, is checking whether vmid already exists (GET .../qemu/{vmid}/config, needing VM.Audit on /vms/<vmid>) — so without this grant already in place, that check itself gets rejected, and the build fails immediately, before ever reaching VM.Allocate.

Scoping the role to one specific vmid, rather than the whole /vms subtree, means the token can never touch any other VM; managing more than one vmid means either one acl modify line per vmid, or granting at /vms instead to cover all of them at once.

Publish to Proxmox VE

Why the token is checked before anything expensive

Skipping this check would still fail eventually, but confusingly: an authentication failure on the actual upload request doesn't come back as a clean 401 — PVE accepts the connection and only then rejects it, but by that point the client is already sending a multi-hundred-MB body, so what actually surfaces is a low-level connection error (write: broken pipe or similar) with no indication it was ever about authentication.

Why efidisk0 defaults to pre-enrolled-keys=0

This leaves the EFI variable store without a Platform Key enrolled, which leaves Secure Boot unenforced — OVMF has nothing to verify a bootloader's signature against, so it just boots whatever's there. This matters because builder.steps produces an unsigned bootloader by default (e.g. a plain bootctl install of systemd-boot): with the old default of pre-enrolled-keys=1 (Secure Boot pre-armed with the standard Microsoft/UEFI CA keys), such a bootloader fails to load at all, with OVMF's BdsDxe reporting "Access Denied". Only set efi-disk: {pre-enrolled-keys: true} if the image's bootloader is actually signed for Secure Boot (e.g. systemd-boot-efi-signed chain-loaded via shim, not just installed directly).

Why converting to a template is a separate API call

Easy to assume this is redundant with setting template: true in parameters, but it isn't: that only sets metadata (prevents the VM from starting, shows the template icon in the UI). The disks themselves are still plain vm-<vmid>-disk-N volumes until the dedicated template API call renames them into base volumes and creates the storage-level snapshots a linked clone needs. Skip it, and cloning the "template" later fails with something like Linked clone feature is not supported for '<storage>:vm-<vmid>-disk-N' (scsi0) — the disk name in that error is the tell: a properly converted template's disk would be named base-<vmid>-disk-N.

Why import-storage and storage are separate fields

Even though they're often the same PVE storage in practice: a storage needs the import content type to receive the uploaded image, and a possibly different one needs the images content type to actually host a VM's disks — PVE storages don't necessarily support both, and using one where the other is required fails with a clean storage '<name>' does not support vm images (or a similar import counterpart) from the API.

Bootstrapping with a local package mirror

Setting cache makes the packages debootstrap needs get downloaded once and reused across builds, instead of being fetched fresh from the network every time. This is more involved than it sounds, because debootstrap fetches packages itself rather than through apt, so there's no existing package cache to just point it at — one has to be built:

1. Determine the package list: debootstrap --no-check-gpg --print-debs <suite> <scratch-dir> <mirror> prints exactly the packages a real run would install, without installing anything. --no-check-gpg is used here specifically because debootstrap verifies the Release file's signature via sqv/gpgv as part of this, and sqv (at least the version this was tested against) prints the verifying key's fingerprint to stdout on success — which would otherwise corrupt the package list being parsed. This is safe: the actual package files are still verified against the trusted keyring by the apt-get calls that follow.
2. Download those packages: an isolated apt-get update followed by apt-get download <packages...>. "Isolated" means a scratch sources.list containing only a deb [arch=<arch>] <mirror> <suite> <components> line, and Dir::Etc::sourcelist, Dir::State::lists, Dir::Cache and Dir::State::status all pointed at scratch locations under <cache>/<mirror>/.build/ (see step 3 for what <mirror> is), rather than the host's real apt configuration and state. The Dir::State::status isolation (an empty file) matters in particular: without it, apt's candidate resolution considers whatever happens to already be installed on the build host, and a package installed there at a version that doesn't quite match the target suite's (e.g. the host tracking testing/unstable) fails to resolve to any download source at all. If mirror is recognized as a Debian or Ubuntu one, that scratch line also gets signed-by=<path> pointed at the matching keyring installed on the build host (not the target, which doesn't exist yet at this point); missing entirely, this fails with a clear error up front instead of apt-get update itself failing with an unrelated-looking "Missing key ..." once it can't verify the Release file it just downloaded. An unrecognized mirror gets no signed-by at all, falling back to the host's own default apt trust store.
3. Sort the downloads into a proper mirror layout: apt-move moves the downloaded .deb files into a pool/ hierarchy matching the real Debian archive layout, rooted at <cache> joined with mirror's own host and path (e.g. <cache>/archive.ubuntu.com/ubuntu) — not <cache> itself. pool/ is shared and suite-blind by design (packages from every suite of the same origin are meant to accumulate there together, across builds), but two specs that happen to set the same cache directory for two different origins (e.g. one for Debian, one for Ubuntu) must never end up sharing one pool/ — with no separation, the index regeneration in step 4 would index both archives' packages into the same Packages file, and debootstrap's resolver can pick a same-named, same-arch package from the other distro entirely, with no version coordination between them at all. This was a real, observed bug: a Debian libc6 breaking an Ubuntu build's base-files.
4. Regenerate the package index: the Packages/Packages.gz/Release files apt-move itself produces are immediately overwritten using apt-ftparchive. This is necessary because the packaged apt-move (4.2.27 in Debian) only ever emits an MD5sum: field per package and no SHA256: section in Release, while current debootstrap requires SHA256 checksums for both the index files themselves and each individual package — without this step, every single package would fail debootstrap's download validation ("was corrupt"), even though the file itself is fine.
5. Bootstrap from the local mirror: the real debootstrap invocation uses file://<cache>/<mirror> as its mirror instead of the configured one, with --no-check-gpg added (the local mirror has no signed Release file).

The scratch state (steps 1–2) lives under <cache>/<mirror>/.build, so re-running a build against the same cache directory and the same mirror reuses the already-fetched apt package indexes rather than re-downloading them. Two different suites of the same mirror (e.g. bookworm and trixie off deb.debian.org/debian) do share this same pool/ — that's intentional, since packages common to both suites are then only ever downloaded once.

ca-certificates and the HTTPS chicken-and-egg problem

When mirror is an https:// URL, ca-certificates is automatically added to the effective --include list, if it isn't there already, to avoid a chicken-and-egg problem: the base apt source (below) also points at mirror, for later package steps to use — but a freshly bootstrapped system has no CA trust store yet, so its own apt-get can't establish a TLS connection to fetch ca-certificates, the one package that would fix that. debootstrap itself doesn't hit this problem because it fetches packages using the host's tools (already trusted), not the chroot's — so including ca-certificates in the base install sidesteps the problem entirely.

Base apt source Signed-By logic

The Signed-By: line in the automatically-written base apt source is only included if mirror is recognized as a Debian or Ubuntu one (matched by hostname) and the matching keyring file actually exists in the target — which it does, since installing it is part of the base debootstrap install for that distribution. Matching by mirror rather than just using whichever keyring happens to be present matters in principle (though the target only ever has the one distribution's keyring installed in practice, so this is mostly future-proofing): if it doesn't match, or the matching keyring isn't there, the entry falls back on the trust debootstrap already established by installing the archive's keyring into the target's /etc/apt/trusted.gpg.d, the same way a plain deb <mirror> ... line in a classic sources.list would — apt still honors that, just with a "Missing Signed-By" notice on every invocation since it can no longer tell which keyring actually vouches for this particular source.

Presets

Why ubuntu-serial targets noble, not the newest release

Verified directly against the archive: starting with 26.04 ("resolute"), Ubuntu carries both the traditional GNU implementation of several core tools (gnu-coreutils, sudo) and a Rust reimplementation (rust-coreutils/coreutils-from-uutils, sudo-rs) side by side, mid-transition, with both sides of each pair marked required/important simultaneously and no Conflicts declared between the two coreutils implementations. debootstrap's package selection is a priority-ordered walk, not a real dependency solver the way apt is, so it ends up trying to unpack both — and its raw second-stage dpkg unpack (unlike a normal apt install, which runs full Replaces-aware conflict resolution) fails outright the moment it hits a file both packages ship (e.g. both providing a touch.1.gz man page symlink, pointing at different targets).

There's no --exclude-based workaround either: debootstrap only applies --exclude to its initial package list, not to packages its own dependency resolution pulls back in afterward to satisfy some other package's Depends/Pre-Depends — which is exactly what happens here regardless. Ubuntu's own debootstrap package doesn't special-case this yet either (checked directly: identical fallback logic to Debian's for resolute).

noble predates this transition entirely, sidestepping the whole class of problem — if a future preset needs a newer release once this settles upstream, ubuntu-serial's suite/include/exclude fields are exactly what to revisit.

Why ubuntu-serial includes fdisk, initramfs-tools, and excludes usrmerge

fdisk, for the shared boot-setup step's grow-root script, which calls sfdisk. sfdisk lives in the fdisk package on both distros, but Ubuntu ships it at Priority: optional (not installed by a plain debootstrap unless asked for by name), while Debian's is Priority: important (installed by default) — verified directly against both archives' Packages indices.

initramfs-tools: Ubuntu's kernel package only Recommends it (as one half of an alternative with linux-initramfs-tool), never Depends on it, and debootstrap never installs Recommends. Debian's kernel package Depends on it outright, so debian-serial needs no such override. Without it, no initramfs ever gets built, and boot setup's own kernel-install step later fails outright with no /boot/initrd.img-$KVER to hand off — verified by bootstrapping a real, minimal Ubuntu system and checking directly: the kernel package's own postinst creates the top-level /boot/vmlinuz and /boot/initrd.img symlinks unconditionally, but they point to a target that plain debootstrap alone never actually produces.

usrmerge (excluded): it's the legacy transitional package for migrating an existing system to a merged /usr, which a system debootstrap creates fresh already is from the start, and recent Ubuntu suites have dropped it from the archive entirely — a debootstrap too old to know that still asks for it by name regardless, and fails with a confusing "Unable to locate package usrmerge" rather than a clean complaint about being out of date.

Why ubuntu-serial skips the signed systemd-boot-efi-amd64-signed package

Ubuntu dropped it from its archive. That's fine here regardless, since efidisk0's own pre-enrolled-keys=0 default already leaves Secure Boot unenforced for an unsigned bootloader either way — see why efidisk0 defaults to pre-enrolled-keys=0 above.

Build progress output, in detail

The terminal UI's exact layout mechanics, for anyone modifying internal/ui: if the header, every task/step and the (normally 8-line) scrolling log window can't all fit the terminal at once, the log window is shrunk first — down to as little as 30% of the space left after the header, never below 2 lines — and only if the task list still doesn't fit even then does it scroll too, keeping the currently running task/step visible along with the following couple of tasks/steps where there's room, so scrolling forward doesn't strand the current one right at the bottom edge with nothing upcoming shown yet — with a blue ↑/↓ in place of the first/last row, each followed by how many rows are scrolled out of view that way (e.g. "↓ 3 more tasks"). The current task/step is highlighted (bold text on a light yellow background, with a bold black indicator). The header stays visible throughout the task list, and through a short success summary or the start of a failure report — only a long failure report (with its up to 20 log lines) can eventually scroll it away, the same way any other lengthy terminal output would.