Deep Dive of the Day Backing up and restoring the Supervisor control plane in VCF 9.1
If you run Supervisors, KB 457944 is a good reason to look at this now. Its recovery steps for missing Supervisor control plane VMs include "Restore the Supervisor Cluster from a Supervisor backup", which only works if that backup exists. The good news is that it isn't a separate product or appliance; it's a checkbox most people haven't ticked.
In VCF 9.1, Supervisor control plane backup rides on the vCenter file-based backup. In the VAMI on port 5480, you edit (or create) the backup schedule and select "Supervisors Control Plane". TechDocs puts it simply: "The state of all Supervisors in vCenter is backed up as part of the vCenter backups." The target is the same FTP, FTPS, HTTP, HTTPS, SFTP, NFS or SMB server you already use for vCenter, and the option covers every Supervisor that vCenter manages, not individual ones.
What goes into it is the Supervisor's own state: etcd, the infrastructure pod container images (so the control plane can be rebuilt even after a vCenter upgrade), the Kubernetes CA certificate and key, every vSphere Namespace and the Kubernetes resources behind workloads, such as deployments, pods, VMs, VKS resources and persistent volume claims. Keeping the same CA matters, because vSphere Pods and the Spherelet don't have to be taught to trust a new one.
Plenty of Supervisor configuration lives in vCenter instead, and comes back with a vCenter restore rather than a Supervisor one: namespace resource pools, control plane VM settings, VKS content libraries, network settings, storage policies, certificates, registries, identity providers, permissions and vSphere Zones. The two restores are separate workflows, and TechDocs is explicit that restoring vCenter does not restore the Supervisor control plane.
The restore itself starts from Supervisor Management > Supervisors > Restore in the vSphere Client. You point it at the backup server (or reuse the vCenter schedule's location and credentials), pick a backup file and the Supervisor, and finish the wizard. The Supervisor goes into Configuring, its control plane VMs are deleted and redeployed from the backup, and you follow progress from the Config Status column.
Rutger Blom tested this end to end in a 9.1 lab, and his numbers are useful for planning. A manual backup took about 15 minutes and roughly 4.4 GB, and the restore task took about 43 minutes, with reconciliation carrying on afterwards. vCenter deployed three new control plane VMs; the VKS node VMs were left alone, and kubectl worked against the guest cluster afterwards.
The gotchas are where it gets interesting. A namespace deleted after the recovery point isn't recreated by a Supervisor restore (TechDocs says you need the vCenter state for that), and in Rutger's test a node volume added after the backup was rolled back. In one test a VKS cluster came back with only link-local IPv6 addresses on its nodes and an unreachable API, which he couldn't pin on the restore or his lab. The restore also replaces all three control plane VMs with no undo, so it's for lost or corrupted control plane state, not a fix for a failed upgrade.
The biggest one: workload data isn't in the backup. Persistent data inside VKS clusters, vSphere Pods and data in VM Service VMs need their own protection, and TechDocs points to Velero and the Velero Plugin for vSphere for that. Rutger's view is blunt: "I would not start a planned Supervisor restore without reliable, tested backups."
For production, a few rules follow. Don't take backups while the Supervisor or a Supervisor Service is in error, or during an upgrade, because a backup can complete and still not be restorable. If vCenter may also need restoring, keep the installer for your exact vCenter build, since Rutger notes the VCSA installer and backup must match. Restore vCenter first when you need both, and Supervisors below vSphere 8 Update 3 aren't included at all.
In a nested lab this is cheap to try and worth doing once before you need it. An SFTP target on a small Linux VM is enough, a daily job with a week's retention mirrors Broadcom's recommendation for vCenter and SDDC Manager, and running a restore on a throwaway namespace will show you how long your Supervisor is out of action. After a restore, check the API, the expected namespaces, node addresses, load balancer VIPs and DNS before you call it done; as Rutger puts it, "A completed vCenter task is only the first checkpoint."
Based on Backup and Restore the Supervisor Control Plane, VCF 9.1 (TechDocs) Official KB 457944: Supervisor Control Plane VMs missing after upgrade Official Backing Up and Restoring the vSphere Supervisor Control Plane in VCF 9.1 (Rutger Blom, 5 Aug)