Deep Dive of the Day Choosing and protecting key providers for encryption in VCF
Every encryption feature you might switch on in VCF (vSAN data-at-rest encryption, VM encryption and virtual TPMs) depends on a key provider configured on vCenter, and the choice you make there decides what happens on the worst day: a full power-off and restart. There are two kinds. A Standard Key Provider talks KMIP to an external key server; a vSphere Native Key Provider is built into vCenter, generates a Key Derivation Key and pushes it to every host in the cluster, with no external server at all. Broadcom's own guidance is that you can mix them, for example a Standard Key Provider for vSAN and the Native Key Provider for virtual TPMs, and a vCenter can hold up to 32 providers.
With a Standard Key Provider, hosts cache keys in memory and lose them on reboot. vSAN hosts contact the key server directly, so their disks won't mount until it's reachable, and VMs with VM encryption or a vTPM won't start until vCenter and the key server are both up. That's why the VCF blog's first two rules are to run more than one key server and keep them away from shared failures, and why the classic trap is a key server VM sitting on the very encrypted datastore it unlocks. Watch for drift too: a Storage vMotion can quietly move a key server onto an encrypted datastore. KMIP uses TCP 5696, and with vSAN encryption every vSAN host needs to reach it, not just vCenter.
The Native Key Provider is quicker to stand up but comes with its own rules. TechDocs is blunt that "You must back up a vSphere Native Key Provider", and a new provider can't be used until you do; vCenter alarms every 24 hours until then. That backup, and every vCenter file-based backup, contains the key, so encrypt your vCenter backups and store the NKP backup and its password somewhere you can reach when vCenter is down. NKP has no KMIP or HSM support, so anything outside VMware that needs keys still needs a real key server. It doesn't need a TPM, but if you use one it must be TPM 2.0, and the "Use key provider only with TPM protected ESX hosts" option is on by default: turn it off unless every host has a TPM, or hosts without one simply won't get the key.
A few gotchas catch people in production. Key providers can't be renamed, so pick names carefully and use the same name everywhere a provider is shared. vCenter will let you delete a provider that's still in use; running VMs carry on but won't power on again until it's restored, so rekey first. Providers belong to individual vCenter instances, so for cross-vCenter vMotion or disaster recovery the same provider (or an NKP restored from backup) must exist on the target. Rotation is a shallow rekey for most compliance needs; a deep rekey rewrites data, needs the VM off with no snapshots, and adds flash wear. Key Persistence keeps keys across reboots on TPM 2.0 hosts, but it's off by default because a stolen host could then unlock its own data.
For nested labs the Native Key Provider is the obvious choice: no extra appliance, and nested hosts without a TPM work fine as long as you untick the TPM-only option. Do the backup straight away, because rebuilding a lab vCenter without it leaves encrypted VMs unreadable. Community advice also warns against stacking VM encryption on top of encrypted vSAN, which adds CPU overhead because data is encrypted twice and makes deduplication of those VMs close to useless. Finally, use the lab for what Broadcom recommends testing anyway: restore an NKP backup, power the whole cluster off and on with key servers starting as they would in a real outage, and take one key server away to see what keeps running.
Based on vSphere Native Key Provider Overview (TechDocs) Official VCF blog: Encryption Key Provider Design Official StarWind: vSphere Native Key Provider best practices