Creating Your Own ProxCP KVM Templates

This tutorial covers how to create a custom KVM template for use with ProxCP and Proxmox.

Required: at least 1 Proxmox node

Assumption: The desired operating system ISO is already downloaded onto your Proxmox node.

The commands covered here are relevant to Debian and RHEL-based operating systems however the process can be applied to any other operating system.

Create VM

• Create a new VM in Proxmox with the following settings:
  • VMID: anything
  • Name: operating system description (i.e. centos-8-amd64)
  • OS: your chosen ISO image
  • Hard Disk: use VirtIO SCSI (scsi0) with 10GB storage and no caching
  • CPU & Memory: 1 kvm64 CPU core, 1GB memory
  • Network: VirtIO (virtio0), vmbr0 bridge, no firewall
• After creation, go to the Hardware tab and delete the CDROM ISO. Add a cloud-init drive on ide2. Add the CDROM ISO back as ide0.

Base OS

• Boot the new VM and complete the base operating system installation. Generally, templates should use most default settings and minimal packages. There are a few things to keep in mind during installation:
  • Partitions: create 1 ext4 root "/" partition (no LVM, no SWAP)
    • If you accidentally create a swap partition: Deleting Linux Swap Partition on KVM
  • Networking: use a generic hostname (i.e. localhost or centos-8-amd64)
  • Packages: use minimal packages, however the OpenSSH server is required

OS Setup Stage 1

• Once the base OS installation is complete, shutdown the VM
• In Proxmox, go to Options and set the boot order to scsi0 only (1 option)
• Go to the Hardware tab and remove the ide0 CDROM. Verify the network device is net0 using vmbr0 bridge
• Power on the VM
• Login as root
• If you created a user during installation, remove that user with deluser [username] && rm -rf /home/[username]
• Complete updates apt update && apt upgrade && apt install setserial or yum update && yum install setserial

Serial Console Setup

KVM templates have to be configured to use a serial console for noVNC access.

• Edit the /etc/default/grub file:
  • Add or replace

GRUB_CMDLINE_LINUX='quiet console=tty0 console=ttyS0,115200'
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

• Save the file
• Run update-grub (debian/ubuntu) or grub2-mkconfig -o /boot/grub2/grub.cfg (centos/rhel)
• Run cat /etc/securetty | grep ttyS0 and ensure ttyS0 is in /etc/securetty. This file may not exist which is okay.

If you are creating a template for a newer operating system using "systemd":

• Run systemctl enable serial-getty@ttyS0.service && systemctl start serial-getty@ttyS0.service

Next:

• Shutdown the VM shutdown -h now
• In Proxmox go to the Hardware tab and add a Serial Port (serial0) then set the Display to Serial Terminal 0 (serial0)
• Start the VM

OS Setup Stage 2

• Login as root
• Edit /etc/ssh/sshd_config and ensure Port is 22 and PermitRootLogin is yes
  • It is best to uncomment these lines to explicitly declare these settings
• Install any other software you want to include in this template
  • Some good examples to include: htop bmon zip unzip wget nano curl ethtool net-tools
• Confirm with fdisk -l that there are no other disk partitions after root "/"

Cloud-init

• Install cloud-init:
  • Debian/Ubuntu: apt install cloud-init cloud-initramfs-growroot cloud-guest-utils xfsprogs gdisk
  • CentOS/RHEL: yum install cloud-init cloud-utils-growpart cloud-guest-utils xfsprogs libicu gdisk
• Proxmox requires cloud-init 18.2 or newer cloud-init --version
• Edit /etc/cloud/cloud.cfg as the following. Note the OS-dependent options

datasource_list: [ NoCloud, ConfigDrive ]
users:
   - default
disable_root: false
preserve_hostname: false
apt_preserve_sources_list: true Ubuntu/Debian-based OS only
cloud_init_modules:
 - migrator
 - seed_random
 - bootcmd
 - write-files
 - growpart
 - resizefs
 - disk_setup
 - mounts
 - set_hostname
 - update_hostname
 - update_etc_hosts
 - ca-certs
 - rsyslog / audit rsyslog for ubuntu/debian, audit for centos/rhel
 - users-groups
 - ssh
cloud_config_modules:
 - emit_upstart
 - ssh-import-id
 - locale
 - set-passwords
 - grub-dpkg
 - apt-pipelining Ubuntu/Debian-based OS only
 - apt-configure Ubuntu/Debian-based OS only
 - ntp
 - timezone
 - disable-ec2-metadata
 - runcmd
 - byobu
cloud_final_modules:
 - package-update-upgrade-install
 - fan
 - landscape
 - lxd Ubuntu/Debian-based OS only
 - puppet
 - chef
 - mcollective
 - salt-minion
 - rightscale_userdata
 - scripts-vendor
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - ssh-authkey-fingerprints
 - keys-to-console
 - phone-home
 - final-message
 - power-state-change
system_info:
   distro: debian / centos / ubuntu OS dependent
   default_user:
     name: debian This can be anything
     lock_passwd: True
     gecos: Debian Same as name above
     groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
     sudo: ["ALL=(ALL) NOPASSWD:ALL"]
     shell: /bin/bash
   ntp_client: auto
   paths:
      cloud_dir: /var/lib/cloud/
      templates_dir: /etc/cloud/templates/
   ssh_svcname: ssh / sshd ssh for ubuntu/debian, sshd for centos/rhel
resize_rootfs: true
growpart:
    mode: auto
    devices: ["/"]
    ignore_growroot_disabled: false

Crucially, this configuration will do the following when booting a cloned VM for the first time: set hostname, grow root partition, allow root login and change the root password, set IP/networking configuration, and generating new SSH keys.

OS Cleaning

At this point you should have a VM with a single user (root), a serial console, any packages you want included in the template, configured SSH, and configured cloud-init. Now we need to clean the OS for redistribution.

• Clean package manager apt autoremove && apt autoclean && apt clean or yum clean all
• Stop logging service rsyslog stop or service auditd stop
• Rotate log files logrotate -f /etc/logrotate.conf && logrotate -f /etc/logrotate.d/*
• Remove things:
  • rm -rf /etc/ssh/ssh_host_*
  • rm -rf /var/log/*.log.* /var/log/apt/* /var/log/btmp.* /var/log/dmesg.* /var/tmp/* /tmp/* /etc/udev/rules.d/70*
  • cat /dev/null > /var/log/btmp && cat /dev/null > /var/log/dmesg && cat /dev/null /var/log/lastlog
  • Ensure all files in /var/log are deleted or emptied (0 bytes)
• Clear network configuration:
  • Ubuntu/Debian w/ ifupdown: remove everything in /etc/network/interfaces except "lo loopback" device; remove all files in /etc/network/interfaces.d/
  • Ubuntu/Debian w/ netplan: remove all files in /etc/netplan/
  • CentOS/RHEL: remove UUID, HWADDR, NETMASK, GATEWAY, IPADDR, NAMESERVERS from /etc/sysconfig/network-scripts/ifcfg-eth0
• Set random, long root password passwd
• Remove history rm ~/.bash_history ~/anaconda*
• unset HISTFILE
• Shutdown shutdown -h now

Preparing the Template for ProxCP

• Right click the VM in Proxmox, set it as a template
• Delete any notes you have in the VM
• Go to the cloud-init tab in Proxmox, set:
  • User: root
  • Password: random, long string
  • IP: dhcp
• Note the name of the Proxmox node and the new template's VMID, then add it in ProxCP for use

Remember that KVM templates cannot currently be shared between Proxmox nodes. Therefore, the templates need to be manually copied to each Proxmox node you have. Copying can be done fairly easily with vzdump, rsync, and qmrestore tools.

Notes on Windows OS

This tutorial covers KVM template creation for Linux operating systems. At the time of writing, Windows does not officially support cloud-init. There is a community option, cloudbase-init, however it does not currently work with Proxmox.