Proxmox Cluster: Major Upgrade Without the Fear

A major upgrade of a Proxmox cluster — typically from 8.x to 9.x — is the moment administrators start to sweat. Unlike a minor update on a standalone server, a major upgrade touches the Corosync stack, the kernel ABI, the storage plugin layout and, worst case, the cluster configuration database pmxcfs. Starting without a plan risks a split cluster, lost HA resources or VMs that no longer boot after the reboot.

The good news: Proxmox has shipped a very reliable upgrade tool for years — pve8to9 (or the equivalent script for your target version) — and the sequence of steps has essentially been the same since PVE 5.x. This article is our playbook from real customer projects: preparation, sequence, quorum handling, HA behaviour and — crucially — the rollback path if things go wrong.

Before the upgrade: pve8to9 and the truth about your cluster

The most important command before any major upgrade is the compatibility checker. For the path from 8.x to 9.x it is called pve8to9 and it ships with the pve-manager package. You must run it on every node individually, not just one:

# On every node:
apt update && apt dist-upgrade -y   # bring the 8.x series fully up to date first
pve8to9 --full

The checker reports items such as:

• Outdated storage configurations (e.g. LVM-thin without a discard option)
• Kernel modules removed in 9.x (e.g. legacy Broadcom drivers)
• Corosync configurations with only a single ring
• Repository entries still pointing at bullseye/bookworm
• Certificates about to expire
• Unsupported CPU flags (relevant on very old hardware)

Take the warnings seriously. In our projects we typically see two to five findings per cluster — most harmless, but occasionally a genuine show-stopper such as a Ceph cluster that has not been migrated yet or a ZFS pool with features disabled that are default in the new version.

Snapshot and backup strategy: the two layers

Before the upgrade we secure the environment on two layers:

1. Host config backup: /etc/pve, /etc/network/interfaces, /etc/corosync/corosync.conf, /etc/hosts, ZFS pool configuration and the output of pveversion -v go to external storage as a tar archive.
2. VM/CT layer: A fresh, consistent Proxmox Backup Server job across all guests. For critical VMs an additional ZFS or Ceph snapshot right before the node reboot.

If you do not yet run Proxmox Backup Server, now is the time to think about a PBS instance — for a major upgrade, “vzdump to NFS” is the absolute lower bound. A snapshot alone is not a backup, because it lives on the storage you are about to rebuild.

Order of operations: node by node, never in parallel

A Proxmox cluster is updated in a rolling fashion. Never in parallel, never big-bang. The proven sequence is:

Step	Action	Reason
1	Pick the first node (ideally the one with the fewest VMs)	Minimal migration
2	Live-migrate VMs/CTs to other nodes	Zero downtime
3	Set the node’s HA resources to “ignored” or pin them elsewhere	Prevent fence
4	Switch repositories to 9.x	sed -i 's/bookworm/trixie/g' /etc/apt/sources.list.d/*.list
5	apt update && apt dist-upgrade	Core package upgrade
6	Reboot into the new kernel	New kernel modules active
7	Check pvecm status — quorum restored?	Cluster healthy
8	Next node — same procedure

Only once all nodes are on 9.x do you enable new features such as an updated Corosync protocol or new storage options. A mixed cluster (some nodes 8.x, some 9.x) is officially supported only during the upgrade window, not as a steady state.

Quorum and HA: the critical half hour

A 3-node cluster tolerates exactly one failure. While one node is rebooting you depend on the exact two remaining nodes — if one of those two fails now, the cluster loses quorum and pmxcfs becomes read-only.

Two concrete measures dramatically reduce that risk:

• Redundant Corosync ring: We always configure two physically separated networks for Corosync (e.g. a dedicated VLAN plus the management network as fallback). If a switch reboots, Corosync survives via the second ring.
• HA groups configured properly: VMs that must stay online get an HA group with restricted=1 and at least two failover nodes. During the upgrade we shift the primary priority away from the node being upgraded.

For clusters with many HA VMs we use the maintenance mode PVE has offered since 8.1:

# Put node into maintenance mode -- HA migrates away automatically
ha-manager crm-command node-maintenance enable pve-node-01

# After the reboot, re-enable
ha-manager crm-command node-maintenance disable pve-node-01

For two-node setups (they still exist) a QDevice as external quorum witness is mandatory. Without a QDevice, every node upgrade means a complete cluster outage because quorum is lost.

The rollback path: when it does go wrong

Honestly: in nine out of ten projects the upgrade runs clean. The tenth case decides whether you go home at 10 pm or are still hanging off a console cable at 4 am.

Our rollback options — in this order:

1. Kernel downgrade via GRUB: The previous kernel remains installed after apt dist-upgrade. Pick it from the GRUB menu at boot and check whether the node runs normally again. Often resolves driver and NIC issues.
2. Package downgrade: For individual broken packages apt install <package>=<old-version> can help — provided the APT cache still holds the old versions. That is why we cache /var/cache/apt/archives externally before the upgrade.
3. PBS restore of the VMs: If a VM no longer starts after the upgrade (e.g. because of a QEMU machine version change), we restore it from the PBS backup — if necessary onto a node that has not yet been upgraded.
4. Full node reinstall: The node is removed from the cluster (pvecm delnode), reinstalled from scratch with the old 8.x version and rejoined. Thanks to pmxcfs it inherits the cluster configuration automatically.

Important: A zfs rollback of the root pool only works if you took a snapshot of rpool before the apt dist-upgrade — which we do as a matter of principle:

zfs snapshot -r rpool@pre-pve9-upgrade
# ... upgrade goes sideways ...
zfs rollback -r rpool@pre-pve9-upgrade
reboot

After the upgrade: clean up and document

Once all nodes are on 9.x we walk through again:

• pveversion -v on every node — all identical?
• pvecm status — quorum, Corosync rings, node count all correct?
• ceph -s (if Ceph is in play) — all OSDs up, PGs active+clean?
• Check old kernels with proxmox-boot-tool kernel list and remove older versions if appropriate (but always keep at least the previous one)
• Delete snapshots (rpool@pre-pve9-upgrade) after two weeks of error-free operation — not before

And document the steps you actually walked through, the maintenance windows and any oddities. At the next major upgrade — typically in two to three years — you will be grateful for every line.

---

DATAZONE supports you through the Proxmox major upgrade. We plan the maintenance window with you, run the pve8to9 check on your cluster, iron out the findings and guide the rolling upgrade — remote or on site around Neuburg. Get in touch for a no-obligation Proxmox consulting session or an upgrade quote: contact us.