Following on from my last post on vSphere 7.0 certificate Management, I wanted to continue with another certificate related post. This one being Site Recovery Manager (SRM) 8.3. Like vSphere 7.0, this version seems simpler than previous versions I have used.
With SRM, it’s the Appliance Certificate replacement that I am going to take you through in this blog post.
Firstly log into the SRM appliance management console via https://<srm-fqdn>:5480 and select the ‘Certificates’ option on the left, followed by ‘Generate CSR’ in the top right.
Fill in the information for your certificate, then click ‘Generate and Download’. You then need to process the CSR with your certificate authority, whether thats an internal, public or lab CA.
Once you have your certificate, select the ‘Certificates’ option on the left again, this time followed by ‘Change’ in the top right.
Select the last option in the Select certificate type section; ‘CA-signed certificate generated from CSR’. Then, browse both your newly generated certificate and either you root CA certificate, or the CA chain. Click ‘Change’ once done.
This should complete the replacement of the SRM appliance certificate!
If like me you get an error complaining that the IP or Common name / SAN is missing, make sure the local host field is set to the FQDN when connecting SRM to vCenter.
vCenter 7.0 brings many new features, one of which is a much smoother certificate management experience. There are now 4 main ‘modes’ for certificate management.
These are; Fully Managed Mode, Hybrid Mode, Subordinate CA Mode and finally Full Custom Mode. There is a great article here from Bob Plankers explaining the difference between each.
As mentioned in Bob’s blog, Hybrid Mode is the recommend option, and I will show you that process here in this blog.
Firstly, in your vSphere Client, browse to Administration > Certificates. Then click Actions and select ‘Generate Certificate Signing Request (CSR)’.
Complete the required fields with your information, making sure you have at least added the common name as a Subject Alternative Name to avoid issues with modern browsers. Click Next.
Finally, copy or download your CSR to generate the certificate on the CA of your choosing. Click Finish when ready.
Once you have your certificate, return to Administration > Certificates and this time select ‘Import and Replace Certificate’.
You then need to select the second option. This may seem slightly deceiving but it effectively is the option you need when you have generated the CSR from vCenter like this.
Now browse and select both your freshly produced certificate, and the root certificate or certificate chain if you have issuing CA’s.
Hit replace, then wait for the Web Client to restart with the new certificate.
Now one final step is needed to complete Hybrid Mode. You need to download the VMCA Root certificate from https://<vCenterFQDN by clicking the ‘Download trusted root CA certificates’ option and distributing it to your vSphere admins.
Once distributed and installed on your vSphere admins client devices, they should not get certificate errors when either browsing to vCenter or the hosts it manages.
You could however, get this error due to the default certificate having a 5 year validity period and not being within the new ‘standard’ of 398 days.
If you receive this, you will want to adjust the vpxd.certmgmt.certs.daysValid value in the vCenter Advanced Settings. It defaults to 1825, making it 365 (one year) will stop this.
You can then renew the certificate on each host by clicking ‘Renew’ in the Configure > Certificates menu –
Before (5 years) –
After (1 Year)-
If you want to do this renewal via PowerCLI (because…well why wouldn’t you!?) there is a nice function here by Ankush Sethi which does a great job.
A useful thing for a home lab or VMware lab, is a certificate authority. There are Windows based CA’s as well as Linux based and many others. I wanted to take the Linux based route for my home lab to give me some administration time in Linux, being that Windows is my safe place! After a bit of googling, I settled on Easy-RSA as it looked like it would do what I needed in the lab. There are already a few guides out there for this, but this is my take on it for use in my VMware home lab.
I settled on CentOS 8 as a base OS. Why? Why not, I don’t have any Centos VM’s and I decided it would be good to use something other than Ubuntu or Photon.
Firstly, I stood up a low resource VM (1 vCPU, 1GB RAM) giving it a static IP and creating an admin account.
I then kicked off the update of all the install packages on the OS by first elevating to the root account using ‘SU’ and then running the upgrade command for the DNF package manager.
This prompts a ~600MB download after confirming you want to continue. Once the download completes it gets on with upgrading.
Once done, its time to install some additional packages starting with epel-release, easy-rsa and openssl. Lets quickly give some background to each.
epel-release (Extra Packages for Enterprise Linux) is a repository of popular packages which aren’t available by default. easy-rsa is one of the packages in this repository.
easy-rsa This is a utility for managing Public Key Infrastructure(PKI) aka Certificate Authority. Check out some info here.
openssl A widely used tool, in this case to create Certificate Signing Requests (CSR). I’ll let you read about this here.
Lets get to the install. You can run them as separate installs like this –
Now for ease of administration, create a directory in the admin users home directory and create a symbolic link so it remains updated. You also want to limit access to your admin user in my case ‘ca_admin’.
Once created, you are now ready to create the root certificate and private key by running the following command –
You will be prompted to specify a passphrase which you need to keep safe as you will need it when issuing certificates. There will then be a second prompt to provide a common name; Enter you CA’s name. eg. CA01.
This process will have now created your root certificate and the private key (keep this safe). You will find them in the following locations /home/ca_admin.easy-rsa/pki/ca.crt (root certificate) and /home/ca_admin.easy-rsa/pki/private/ca.key (private key).
If you are using a Windows device to access your HomeLab, you are going to want to add the ca.crl file to the ‘Trusted Root Certification Authorities’ store on your Windows device so that any certificates issued are trusted. You can copy the ca.crt file using a tool such as WINSCP to transfer the file to a local directory to then install. You can do the equivalent on Mac and Linux OS’s too.
You will also need this handy for any certificates that require the full chain to be included.
I won’t go into every Certificate Signing Request (CSR) scenario as there are many. I will however, show you the commands needed to produce a certificate from a CSR.
To issue a certificate from a CSR, you will need to copy the .req or .csr file to a directory such as /tmp on your CA server, again using a tool like WINSCP.
You can then run the following commands to import the certificate signing request. The Common name is often the device name or FQDN.
./easyrsa import-req /tmp/<csr_file_name>.req <CommonName or FQDN>
./easyrsa sign-req server <CommonName or FQDN>
The import command will import the .req or .csr file into /home/ca_admin.easy-rsa/pki/reqs (you can’t place the .req directly in here!) which is then processed by the sign-req command, again asking for the passphrase, leaving you with your new certificate in the /home/ca_admin.easy-rsa/pki/issued directory.
You can then use WINSCP again to transfer the file off the CA, and install it on the device or service in which you requested it from.
As always with any root CA, you don’t want it to become compromised. To help with this, keep it turned off when you’re not issuing, or administering your certificates.
I have also not included any Certificate Revocation List details as this isn’t something I need in my lab environment.
Now the VMware bit… below is the process for acquiring the CSR and installing the generated certificate on an ESXi host and a vCenter server using the methods above.
Standalone ESXi Host 6.7
First for a standalone ESXi Host, browse to – Host > Manage > Security & Users > Certificates
Select Import new certificate then select either ‘Generate FQDN signing request’ or ‘Generate IP signing request’.
You will be presented with a screen like this.
Copy this into a file with the extension .req. This can then be imported and issued using the method above.
Then, go back the the ‘Import new certificate’ wizard and import the certificate in the same format at the CSR into the box. (Open the .crt file using notepad)
Once complete close and open your browser and head back to your hosts web client and you will see you no longer have a certificate error.
vCenter Server 6.7
Log into your vCenter appliance using via SSH. Then run the following command –
Select option 1, (you will be prompted to provide your SSO credentials), followed by option 1 again.
You will then need to provide the following information for the CSR.
As you complete the wizard you will have a .csr and a .key file in /tmp which again can be issued using the process above.
If using WinSCP you may hit the following error.
You will need to change over to the bash shell.
chsh -s /bin/bash root
You could then face another error…
root@vcsa02 [ ~ ]# chsh -s /bin/bash root
You are required to change your password immediately (password aged)
chsh: PAM: Authentication token is no longer valid; new one required
This is due to the password expiring. To change the password on the account run the passwd command
Further info on both of these errors can be found at these two VMware Articles. Here and here.
Once you have issued the certificate, you need to then copy the .crt file back to the /tmp directory along with the root certificate (or chain).
Now back to the Certificate Manager. Selecting option 1 to now import the certifictes. You will be prompted to provide the path and file name of each component. The certificate you created, the .key file that was created during the CSR generation and the root or CA chain certificate. Finally you will be asked to confirm you want to replace the Machine SSL certificate, type y.
It will take a few minutes, but eventually you will get confirmation that the task is complete and you can then reload your browser to see the Web Client is now showing a valid certificate.
Hope this has been useful. I will cover vCenter 7.0 Machine SSL certificate replacement in a future post.
The use of VMware tags recently became a requirement for some of my colleagues in an environment that was inherited. They were faced with being unable to create Tags & Tag Categories or Assign and Delete them, despite ‘having admin rights’.
Upon investigation it became apparent that while the admins had been granted the Administrators Role at the vCenter Object Level, they had had not been granted sufficient rights at the Global Permissions Level, or any rights for that matter.
The following graphic shows the vSphere Inventory hierarchy.
As you can see from the graphic, assigning privileges at the Global Level is required to manage Tags and additionally, Content Libraries.
You can see below a vSphere Admin, who has the Administrator Role assigned at the vCenter Level, is not able to select the New, Edit, Delete or Add Permission options for Tags or Tag Categories.
In this scenario there were two different permission requirements, one for a vSphere Admin team, the other for a Storage team both of which could be addressed by two existing roles; Administrators and Tagging Admin. You could of course create a custom role should you have a requirement to do so. Here are the privileges assigned to the tagging admin role for reference.
The vSphere Admin Team required the Administrator Role at the Global Level (root object) so they could manage Content Libraries and Tags, while the Storage Team required the ability to Create and Assign Tags only.
You can assign permissions to a user or a group from multiple Identity Sources, in this scenario, an Active Directory source. You will need to do this from an account that already has the Administrator Role at the Global Level. By default, the firstname.lastname@example.org account has this privilege. (replace the domain as needed if you have used a different SSO domain name)
From the Menu, select ‘Administration’ and select the ‘Global Permissions’ option in on the left-hand side.
From here, select the Add Permission icon.
You can now select the domain of the user or group you wish to add from the drop-down list and begin to type the name of that user or group. It will begin to narrow your selection as you type.
Select the user.
Now select the appropriate role and select the ‘Propagate to child object’ option.
If you have multiple users, groups or roles you need to assign, repeat the process.
You will now see both permissions in the Global Permissions menu. If logged in, you will need to log out and back in for this to take effect. Both scenarios are shown.
If you now return to the Tags & Custom Attributes menu, logged in as a user from either of these groups, you will see that the New, Edit, Delete or Add Permission options for Tags or Tag Categories are now available.
Note: Providing a user or group privileges at the Global Permissions Level and selecting, ‘Propagate to child objects’ will give that user or group the privilege’s on the child objects such as vCenter, Cluster, VM and Datastore.
This can be useful if you have multiple vCenters in Enhanced Linked Mode (ELM) as you only need to apply it once.
Further information can be found in the following VMware article which explains how you can grant permissions on a Tag object, rather than at the Global or vCenter Levels to give you further granualar control.
I have been rebuilding my lab hosts a lot lately! Once because I fiddled too much with my vSAN cluster and killed it… Another more interesting occasion being the release of VCF 4.0 on VMUG and beginning the deployment of this!
I prefer to use Standard vSwitches for my management network in my labs and needed a quick and easy way to get the hosts back online with minimal effort. One thing I don’t like is seeing vSwitch0… I prefer seeing useful and descriptive naming, like I’m sure many others do!
Below are a few lines of PowerCLI to quickly and easily create a new vSwitch using a spare VMNIC (you should be using more than one physical NIC for resiliency), then migrate the Management VM Kernel adapter and original VMNIC over to it, followed by a clean up of vSwitch0.
#Variables<#ESX Host to target#> $ESXHost = "ESX102.lab.local"
<#Name of the Management Switch#> $ManagementSwitchName = "vSS_Management"
<#vmnic to be used for Management Switch#> $ManagementSwitchNIC = "vmnic1"
<#MTU size for Management Switch#> $ManagementSwitchMTU = "1500"
<#Name of the Portgroup for the VMKernel Adapter#> $ManagementVMKPortGroupName = "vSS_VMK_Management"
<#Name of the PortGroup for VM's#> $ManagementPGSwitchName = "vSS_PG_Management"<#Management VMKernal Nic to be migrated#>$vNic = "vmk0"
<#Management VMKernel assosiated pNic#>$PhysiscalNic = "vmnic0"
<#Old vSwitch#> $OldvSwitch = "vSwitch0"
#New Standard Management Switch
$NewSwitch1 = New-VirtualSwitch -VMHost $ESXHost -Name $ManagementSwitchName -Nic $ManagementSwitchNIC -mtu $ManagementSwitchMTU
$NewSwitch1 | New-VirtualPortGroup -Name $ManagementVMKPortGroupName -VLanId 0
$NewSwitch1 | New-VirtualPortGroup -Name $ManagementPGSwitchName
Once the new vSwitch is in place, the next block of code migrates the Management VM Kernel adapter and the VMNIC over to it.