Creating Your Own ProxCP KVM Templates

This tutorial covers how to create a custom KVM template for use with ProxCP and Proxmox.

Required: at least 1 Proxmox node

Assumption: The desired operating system ISO is already downloaded onto your Proxmox node.

The commands covered here are relevant to Debian and RHEL-based operating systems however the process can be applied to any other operating system.

Create VM

• Create a new VM in Proxmox with the following settings:
  • VMID: anything
  • Name: operating system description (i.e. centos-8-amd64)
  • OS: your chosen ISO image
  • Hard Disk: use VirtIO SCSI (scsi0) with 10GB storage and no caching
  • CPU & Memory: 1 kvm64 CPU core, 1GB memory
  • Network: VirtIO (virtio0), vmbr0 bridge, no firewall
• After creation, go to the Hardware tab and delete the CDROM ISO. Add a cloud-init drive on ide2. Add the CDROM ISO back as ide0.

Base OS

• Boot the new VM and complete the base operating system installation. Generally, templates should use most default settings and minimal packages. There are a few things to keep in mind during installation:
  • Partitions: create 1 ext4 root "/" partition (no LVM, no SWAP)
  • Networking: use a generic hostname (i.e. localhost or centos-8-amd64)
  • Packages: use minimal packages, however the OpenSSH server is required

OS Setup Stage 1

• Once the base OS installation is complete, shutdown the VM
• In Proxmox, go to Options and set the boot order to scsi0 only (1 option)
• Go to the Hardware tab and remove the ide0 CDROM. Verify the network device is net0 using vmbr0 bridge
• Power on the VM
• Login as root
• If you created a user during installation, remove that user with deluser [username] && rm -rf /home/[username]
• Complete updates apt update && apt upgrade && apt install setserial or yum update && yum install setserial

Serial Console Setup

KVM templates have to be configured to use a serial console for noVNC access.

• Edit the /etc/default/grub file:
  • Add or replace

GRUB_CMDLINE_LINUX='quiet console=tty0 console=ttyS0,115200'
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

• Save the file
• Run update-grub (debian/ubuntu) or grub2-mkconfig -o /boot/grub2/grub.cfg (centos/rhel)
• Run cat /etc/securetty | grep ttyS0 and ensure ttyS0 is in /etc/securetty. This file may not exist which is okay.

If you are creating a template for a newer operating system using "systemd":

• Run systemctl enable serial-getty@ttyS0.service && systemctl start serial-getty@ttyS0.service

Next:

• Shutdown the VM shutdown -h now
• In Proxmox go to the Hardware tab and add a Serial Port (serial0) then set the Display to Serial Terminal 0 (serial0)
• Start the VM

OS Setup Stage 2

• Login as root
• Edit /etc/ssh/sshd_config and ensure Port is 22 and PermitRootLogin is yes
  • It is best to uncomment these lines to explicitly declare these settings
• Install any other software you want to include in this template
  • Some good examples to include: htop bmon zip unzip wget nano curl ethtool net-tools
• Confirm with fdisk -l that there are no other disk partitions after root "/"

Cloud-init

• Install cloud-init:
  • Debian/Ubuntu: apt install cloud-init cloud-initramfs-growroot cloud-guest-utils xfsprogs gdisk
  • CentOS/RHEL: yum install cloud-init cloud-utils-growpart cloud-guest-utils xfsprogs libicu gdisk
• Proxmox requires cloud-init 18.2 or newer cloud-init --version
• Edit /etc/cloud/cloud.cfg as the following. Note the OS-dependent options

datasource_list: [ NoCloud, ConfigDrive ]
users:
   - default
disable_root: false
preserve_hostname: false
apt_preserve_sources_list: true Ubuntu/Debian-based OS only
cloud_init_modules:
 - migrator
 - seed_random
 - bootcmd
 - write-files
 - growpart
 - resizefs
 - disk_setup
 - mounts
 - set_hostname
 - update_hostname
 - update_etc_hosts
 - ca-certs
 - rsyslog / audit rsyslog for ubuntu/debian, audit for centos/rhel
 - users-groups
 - ssh
cloud_config_modules:
 - emit_upstart
 - ssh-import-id
 - locale
 - set-passwords
 - grub-dpkg
 - apt-pipelining Ubuntu/Debian-based OS only
 - apt-configure Ubuntu/Debian-based OS only
 - ntp
 - timezone
 - disable-ec2-metadata
 - runcmd
 - byobu
cloud_final_modules:
 - package-update-upgrade-install
 - fan
 - landscape
 - lxd Ubuntu/Debian-based OS only
 - puppet
 - chef
 - mcollective
 - salt-minion
 - rightscale_userdata
 - scripts-vendor
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - ssh-authkey-fingerprints
 - keys-to-console
 - phone-home
 - final-message
 - power-state-change
system_info:
   distro: debian / centos / ubuntu OS dependent
   default_user:
     name: debian This can be anything
     lock_passwd: True
     gecos: Debian Same as name above
     groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
     sudo: ["ALL=(ALL) NOPASSWD:ALL"]
     shell: /bin/bash
   ntp_client: auto
   paths:
      cloud_dir: /var/lib/cloud/
      templates_dir: /etc/cloud/templates/
   ssh_svcname: ssh / sshd ssh for ubuntu/debian, sshd for centos/rhel
resize_rootfs: true
growpart:
    mode: auto
    devices: ["/"]
    ignore_growroot_disabled: false

OS Cleaning

At this point you should have a VM with a single user (root), a serial console, any packages you want included in the template, configured SSH, and configured cloud-init. Now we need to clean the OS for redistribution.

• Clean package manager apt autoremove && apt autoclean && apt clean or yum clean all
• Stop logging service rsyslog stop or service auditd stop
• Rotate log files logrotate -f /etc/logrotate.conf && logrotate -f /etc/logrotate.d/*
• Remove things:
  • rm -rf /etc/ssh/ssh_host_*
  • rm -rf /var/log/*.log.* /var/log/apt/* /var/log/btmp.* /var/log/dmesg.* /var/tmp/* /tmp/* /etc/udev/rules.d/70*
  • cat /dev/null > /var/log/btmp && cat /dev/null > /var/log/dmesg && cat /dev/null /var/log/lastlog
  • Ensure all files in /var/log are deleted or emptied (0 bytes)
• Clear network configuration:
  • Ubuntu/Debian w/ ifupdown: remove everything in /etc/network/interfaces except "lo loopback" device; remove all files in /etc/network/interfaces.d/
  • Ubuntu/Debian w/ netplan: remove all files in /etc/netplan/
  • CentOS/RHEL: remove UUID, HWADDR, NETMASK, GATEWAY, IPADDR, NAMESERVERS from /etc/sysconfig/network-scripts/ifcfg-eth0
• Set random, long root password passwd
• Remove history rm ~/.bash_history ~/anaconda*
• unset HISTFILE
• Shutdown shutdown -h now