Deep Dive of the Day Lockdown mode, SSH and vpxuser: closing the side doors into ESX
This week's Check Point report has a detail that should make every vSphere admin pause: after compromising a firewall, the attackers "pivoted from the compromised gateway to a VMware ESXi host over SSH, authenticating with the vpxuser service account", then ran a wiper. The report doesn't say how they got that credential, and no VMware flaw was involved. What it does show is that a reachable SSH service on a host is a door, and that the account behind it doesn't have to be root. So for a longer Friday piece, let's walk through the controls that decide who can get a shell on ESX at all.
Start with vpxuser itself. Broadcom's KB 453134 describes it as "the service account vpxuser is used by vCenter Server to manage and communicate with ESXi hosts". You never set its password: vCenter generates it and, per KB 320773, changes it automatically every 30 days, controlled by the vCenter advanced setting VirtualCenter.VimPasswordExpirationInDays. Both KBs are written for vSphere 8.0, but the mechanism is the one your 9.x hosts inherit.
That rotation has a couple of gotchas that bite in real environments. KB 320773 is blunt that "VMware does not recommend increasing the value", and that the host's Security.PasswordMaxDays "should always be a greater value" than vCenter's rotation interval. KB 453134 lists the ways it goes wrong: a host password policy shorter than 30 days, a host left in maintenance mode longer than its maximum password age, a host unreachable for more than 30 days, or a vCenter interval set longer than the host's. The symptom is a host going "Not Responding", and the fix is usually to correct the policy and reconnect the host with root.
Now the main control: lockdown mode. The vSphere 9.0 docs describe it simply: "In lockdown mode, operations must be performed through vCenter by default." There are two levels. Normal lockdown can be switched on or off from the Direct Console User Interface (DCUI) as well as the vSphere Client; strict lockdown can be changed only from the vSphere Client, because it stops the DCUI service altogether.
The part that matters for this week's story is what happens to SSH and the ESXi Shell. According to the lockdown mode behaviour page, in both normal and strict mode, shell and SSH access (if those services are running) is limited to users in the DCUI.Access advanced setting and to Exception Users who have administrator privileges on the host: "For all other users, ESXi Shell or SSH access is deactivated." Existing sessions for anyone else are closed when the host enters lockdown. By that rule, a host in lockdown mode wouldn't accept an SSH login as vpxuser unless someone had added it to one of those lists.
Lockdown mode doesn't cut vCenter off, which is the point of it. The same page says the vSphere Web Services API stays open to vCenter (vpxuser), to Exception Users according to their permissions, and to vCloud Director (vslauser) where present, and that privileged users can still manage the host through vCenter, the vSphere Client or the SDK. It also notes that "All access is logged for both strict and normal lockdown mode", which is worth wiring into your log forwarding.
The two lists are where lockdown mode quietly gets undone. The exception users page says the Exception Users list "is meant for service accounts that perform very specific tasks, and not for administrators", such as a backup agent, and warns that "Adding administrator users to the Exception Users list defeats the purpose of lockdown mode." DCUI.Access "is meant for a catastrophic failure of vCenter"; Broadcom suggests removing root from it in favour of a named local account whose password lives in a safe, and warns that anyone on that list "can change lockdown mode settings regardless of their privileges". Only local ESX users can go on it, not Active Directory accounts.
Christian Mohn's ESXi hardening checklist on vNinja [Community] puts the wider context well: "ESXi does not support MFA, SAML, OIDC, or FIDO2 for any access path", so SSH, the DCUI and the Host Client all sit below vCenter's identity federation. He suggests normal lockdown for most environments (strict also removes the DCUI you might want for break-glass recovery), reviewing every Exception Users entry as "a documented bypass of Lockdown Mode", keeping management interfaces on a dedicated management network, and watching UserLoginSessionEvent entries for DCUI logins. He also scripts the change across clusters with ChangeLockdownMode("lockdownNormal") through the host access manager.
For production, the practical recipe is short. Leave SSH and the ESXi Shell stopped unless you're actively troubleshooting, turn on normal lockdown, keep Exception Users to genuine service accounts, give DCUI.Access one named break-glass account, and make sure only your admin network can reach host management at all, so a compromised firewall or jump box isn't one SSH hop from your hypervisors. Thursday's deep dive on vSphere Configuration Profiles is a reminder that host settings like these are better enforced centrally than set by hand.
In a nested lab the trade-offs are different, and that's fine. You'll probably want SSH on while you build and break things, and strict lockdown on a nested host you can only reach through a console window is a quick way to lock yourself out. The useful lab exercise is to turn normal lockdown on, confirm that vCenter, your automation and your backup tool still work, and see which of them complain. Whatever fails is something that's talking to the host directly, and it's much better to find that in a lab than during a production change.
Based on Configuring and Managing Lockdown Mode on ESX Hosts (vSphere 9.0) Official Lockdown Mode Behavior (vSphere 9.0) Official Specifying Accounts with Access Privileges in Lockdown Mode Official KB 453134: vpxuser password expiration Official KB 320773: vpxuser expiry settings Official ESXi Has No MFA: A Practical Hardening Checklist (vNinja) PHP2Shell Exploitation on Check Point Gateway Press