Tag Archives: vSphere 6.7

Centos Based Certificate Authority For The VMware Lab

A useful thing for a home lab or VMware lab, is a certificate authority. There are Windows based CA’s as well as Linux based and many others. I wanted to take the Linux based route for my home lab to give me some administration time in Linux, being that Windows is my safe place! After a bit of googling, I settled on Easy-RSA as it looked like it would do what I needed in the lab. There are already a few guides out there for this, but this is my take on it for use in my VMware home lab.

I settled on CentOS 8 as a base OS. Why? Why not, I don’t have any Centos VM’s and I decided it would be good to use something other than Ubuntu or Photon.

Firstly, I stood up a low resource VM (1 vCPU, 1GB RAM) giving it a static IP and creating an admin account.

I then kicked off the update of all the install packages on the OS by first elevating to the root account using ‘SU’ and then running the upgrade command for the DNF package manager.

dnf upgrade

This prompts a ~600MB download after confirming you want to continue. Once the download completes it gets on with upgrading.

Once done, its time to install some additional packages starting with epel-release, easy-rsa and openssl. Lets quickly give some background to each.

epel-release (Extra Packages for Enterprise Linux) is a repository of popular packages which aren’t available by default. easy-rsa is one of the packages in this repository.

easy-rsa This is a utility for managing Public Key Infrastructure(PKI) aka Certificate Authority. Check out some info here.

openssl A widely used tool, in this case to create Certificate Signing Requests (CSR). I’ll let you read about this here.

Lets get to the install. You can run them as separate installs like this –

sudo dnf install epel-release
sudo dnf install easy-rsa
sudo dnf install openssl

or as a one liner –

sudo dnf install epel-release easy-rsa openssl

Now for ease of administration, create a directory in the admin users home directory and create a symbolic link so it remains updated. You also want to limit access to your admin user in my case ‘ca_admin’.

mkdir ~/easy-rsa
ln -s /usr/share/easy-rsa/3/* ~/easy-rsa/
chmod 700 /home/ca_admin/easy-rsa

If you’re not familiar with chmod commands, ‘chomd 700’ = Protects a file against any access from other users, while the issuing user still has full access.

Now to initialise your new PKI. Change Directory (cd) to the easy-rsa directory you created in your admin users home directory and run the initialisation command.

cd ~/easy-rsa
./easyrsa init-pki

You will get a message showing it is complete and it will state the new pki directory that has been created inside the easy-rsa directory (/home/ca_admin/easy-rsa/pki)

You now need to create and populate a file called ‘vars’ in /home/ca_admin/easy-rsa and populate it with your organisation/lab information. You can achieve this with the vi editor.

set_var EASYRSA_REQ_COUNTRY "GB"
set_var EASYRSA_REQ_PROVINCE "Labshire"
set_var EASYRSA_REQ_CITY "Lab City"
set_var EASYRSA_REQ_ORG "Home Lab"
set_var EASYRSA_REQ_EMAIL "admin@lab.local"
set_var EASYRSA_REQ_OU "Lab"
set_var EASYRSA_ALGO "ec"(or rsa if you wish...)
set_var EASYRSA_DIGEST "sha512"

Once created, you are now ready to create the root certificate and private key by running the following command –

./easyrsa build-ca

You will be prompted to specify a passphrase which you need to keep safe as you will need it when issuing certificates. There will then be a second prompt to provide a common name; Enter you CA’s name. eg. CA01.

This process will have now created your root certificate and the private key (keep this safe). You will find them in the following locations /home/ca_admin.easy-rsa/pki/ca.crt (root certificate) and /home/ca_admin.easy-rsa/pki/private/ca.key (private key).

If you are using a Windows device to access your HomeLab, you are going to want to add the ca.crl file to the ‘Trusted Root Certification Authorities’ store on your Windows device so that any certificates issued are trusted. You can copy the ca.crt file using a tool such as WINSCP to transfer the file to a local directory to then install. You can do the equivalent on Mac and Linux OS’s too.

You will also need this handy for any certificates that require the full chain to be included.

I won’t go into every Certificate Signing Request (CSR) scenario as there are many. I will however, show you the commands needed to produce a certificate from a CSR.

To issue a certificate from a CSR, you will need to copy the .req or .csr file to a directory such as /tmp on your CA server, again using a tool like WINSCP.

You can then run the following commands to import the certificate signing request. The Common name is often the device name or FQDN.

cd ~/easy-rsa
./easyrsa import-req /tmp/<csr_file_name>.req <CommonName or FQDN> 
./easyrsa sign-req server <CommonName or FQDN>

The import command will import the .req or .csr file into /home/ca_admin.easy-rsa/pki/reqs (you can’t place the .req directly in here!) which is then processed by the sign-req command, again asking for the passphrase, leaving you with your new certificate in the /home/ca_admin.easy-rsa/pki/issued directory.

You can then use WINSCP again to transfer the file off the CA, and install it on the device or service in which you requested it from.

As always with any root CA, you don’t want it to become compromised. To help with this, keep it turned off when you’re not issuing, or administering your certificates.

I have also not included any Certificate Revocation List details as this isn’t something I need in my lab environment.

Now the VMware bit… below is the process for acquiring the CSR and installing the generated certificate on an ESXi host and a vCenter server using the methods above.

Standalone ESXi Host 6.7

First for a standalone ESXi Host, browse to – Host > Manage > Security & Users > Certificates

Select Import new certificate then select either ‘Generate FQDN signing request’ or ‘Generate IP signing request’.

You will be presented with a screen like this.

Copy this into a file with the extension .req. This can then be imported and issued using the method above.

Then, go back the the ‘Import new certificate’ wizard and import the certificate in the same format at the CSR into the box. (Open the .crt file using notepad)

Once complete close and open your browser and head back to your hosts web client and you will see you no longer have a certificate error.

vCenter Server 6.7

Log into your vCenter appliance using via SSH. Then run the following command –

/usr/lib/vmware-vmca/bin/certificate-manager

Select option 1, (you will be prompted to provide your SSO credentials), followed by option 1 again.

You will then need to provide the following information for the CSR.

As you complete the wizard you will have a .csr and a .key file in /tmp which again can be issued using the process above.

If using WinSCP you may hit the following error.

You will need to change over to the bash shell.

chsh -s /bin/bash root

You could then face another error…

root@vcsa02 [ ~ ]# chsh -s /bin/bash root
You are required to change your password immediately (password aged)
chsh: PAM: Authentication token is no longer valid; new one required

This is due to the password expiring. To change the password on the account run the passwd command

Further info on both of these errors can be found at these two VMware Articles. Here and here.

Once you have issued the certificate, you need to then copy the .crt file back to the /tmp directory along with the root certificate (or chain).

Now back to the Certificate Manager. Selecting option 1 to now import the certifictes. You will be prompted to provide the path and file name of each component. The certificate you created, the .key file that was created during the CSR generation and the root or CA chain certificate. Finally you will be asked to confirm you want to replace the Machine SSL certificate, type y.

It will take a few minutes, but eventually you will get confirmation that the task is complete and you can then reload your browser to see the Web Client is now showing a valid certificate.

Hope this has been useful. I will cover vCenter 7.0 Machine SSL certificate replacement in a future post.

Thanks for reading!

Creating Virtual Distributed Port Groups Using PowerCLI

I recently needed to create a new Distributed Port Group and set a specific load balancing policy on an existing Distributed Switch. Nothing to exciting, but a task many have to complete. As this is a common repeatable task, i put together this short .ps1 to allow a repeatable way of completing this.

You can find the file here on GitHub

Just populate the Variables section with required information like so…

Save the file, then run the the .ps1 file from PowerShell prompt. (Ensure you have the PowerCLI Module installed; see my previous post)

Note you must add .\ to the beginning of the file name if you are executing the file from the directory you’re already in

Enter credentials with sufficient privilege in vCenter.

You will then see an output similar to this:

If you now take a look in the Web Client you will see the freshly created Distributed Port Group.

Creating just a single portgroup could potentially be quicker in the Web Client. What isn’t quicker, is multiple.

If you have a requirement to create multiple Distributed Port Groups on a vDS, you can use this script to do so in one go.

Just populate the Variables section with required information like above, then run the the .ps1 file from a PowerShell prompt. (Ensure you have the PowerCLI Module installed; see this post.

This uses and Array Table to build your source data, in this example, the PortGroup Name and VLAN ID for each. You can add further rows (more Port Groups) to the array by repeating the line in the red box, or add additional attributes by repeating the text from the green box on each line.

There are many ways you could modify this script to change the source of data, including ‘Get-Content’ from a .txt file for instance.

You can get the script here.

This is just one way to create Port Groups using PowerCLI, have a play around and make it work for you!

Thanks for reading.

vCenter Tag Administration Permissions

The use of VMware tags recently became a requirement for some of my colleagues in an environment that was inherited.  They were faced with being unable to create Tags & Tag Categories or Assign and Delete them, despite ‘having admin rights’.

Upon investigation it became apparent that while the admins had been granted the Administrators Role at the vCenter Object Level, they had had not been granted sufficient rights at the Global Permissions Level, or any rights for that matter.

The following graphic shows the vSphere Inventory hierarchy.

Link to VMware’s article – Here

As you can see from the graphic, assigning privileges at the Global Level is required to manage Tags and additionally, Content Libraries.

You can see below a vSphere Admin, who has the Administrator Role assigned at the vCenter Level, is not able to select the New, Edit, Delete or Add Permission options for Tags or Tag Categories.

In this scenario there were two different permission requirements, one for a vSphere Admin team, the other for a Storage team both of which could be addressed by two existing roles; Administrators and Tagging Admin.  You could of course create a custom role should you have a requirement to do so. Here are the privileges assigned to the tagging admin role for reference.

The vSphere Admin Team required the Administrator Role at the Global Level (root object) so they could manage Content Libraries and Tags, while the Storage Team required the ability to Create and Assign Tags only.

You can assign permissions to a user or a group from multiple Identity Sources, in this scenario, an Active Directory source.  You will need to do this from an account that already has the Administrator Role at the Global Level.  By default, the administrator@vsphere.local account has this privilege. (replace the domain as needed if you have used a different SSO domain name)

From the Menu, select ‘Administration’ and select the ‘Global Permissions’ option in on the left-hand side.

From here, select the Add Permission icon.

You can now select the domain of the user or group you wish to add from the drop-down list and begin to type the name of that user or group.  It will begin to narrow your selection as you type. 

Select the user.

Now select the appropriate role and select the ‘Propagate to child object’ option.

If you have multiple users, groups or roles you need to assign, repeat the process.

You will now see both permissions in the Global Permissions menu.  If logged in, you will need to log out and back in for this to take effect. Both scenarios are shown.

If you now return to the Tags & Custom Attributes menu, logged in as a user from either of these groups, you will see that the New, Edit, Delete or Add Permission options for Tags or Tag Categories are now available.

Note: Providing a user or group privileges at the Global Permissions Level and selecting, ‘Propagate to child objects’ will give that user or group the privilege’s on the child objects such as vCenter, Cluster, VM and Datastore. 

This can be useful if you have multiple vCenters in Enhanced Linked Mode (ELM) as you only need to apply it once.

vCenter –

Host –

Datastore –

Further information can be found in the following VMware article which explains how you can grant permissions on a Tag object, rather than at the Global or vCenter Levels to give you further granualar control.

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-2199584C-B422-4EEF-9340-5449E1FB7DAE.html

Inheriting any environment can be difficult and full of unknowns, hopefully this could help you if you are experiencing a similar issue!

Thanks for reading!

Deploying Custom Virtual Standard Switches for Management

I have been rebuilding my lab hosts a lot lately! Once because I fiddled too much with my vSAN cluster and killed it… Another more interesting occasion being the release of VCF 4.0 on VMUG and beginning the deployment of this!

I prefer to use Standard vSwitches for my management network in my labs and needed a quick and easy way to get the hosts back online with minimal effort. One thing I don’t like is seeing vSwitch0… I prefer seeing useful and descriptive naming, like I’m sure many others do!

Below are a few lines of PowerCLI to quickly and easily create a new vSwitch using a spare VMNIC (you should be using more than one physical NIC for resiliency), then migrate the Management VM Kernel adapter and original VMNIC over to it, followed by a clean up of vSwitch0.

#Variables
<#ESX Host to target#> $ESXHost = "ESX102.lab.local"
<#Name of the Management Switch#> $ManagementSwitchName = "vSS_Management"
<#vmnic to be used for Management Switch#> $ManagementSwitchNIC = "vmnic1"
<#MTU size for Management Switch#> $ManagementSwitchMTU = "1500"
<#Name of the Portgroup for the VMKernel Adapter#> $ManagementVMKPortGroupName = "vSS_VMK_Management"
<#Name of the PortGroup for VM's#> $ManagementPGSwitchName = "vSS_PG_Management"
 
<#Management VMKernal Nic to be migrated#>$vNic = "vmk0"
<#Management VMKernel assosiated pNic#>$PhysiscalNic = "vmnic0"
<#Old vSwitch#> $OldvSwitch = "vSwitch0"
 
#New Standard Management Switch
$NewSwitch1 = New-VirtualSwitch -VMHost $ESXHost -Name $ManagementSwitchName -Nic $ManagementSwitchNIC -mtu $ManagementSwitchMTU
$NewSwitch1 | New-VirtualPortGroup -Name $ManagementVMKPortGroupName -VLanId 0
$NewSwitch1 | New-VirtualPortGroup -Name $ManagementPGSwitchName

Once the new vSwitch is in place, the next block of code migrates the Management VM Kernel adapter and the VMNIC over to it.

#Migrate Mangement VMKernel Adapter
$mgmt_vmk = Get-VMHostNetworkAdapter -VMHost $ESXhost -Name $vNic
$pnic = Get-VMHostNetworkAdapter -VMHost $esxhost -Name $PhysiscalNic
Add-VirtualSwitchPhysicalNetworkAdapter -VirtualSwitch $NewSwitch1 -VMHostPhysicalNic $pnic -VMHostVirtualNic $mgmt_vmk -VirtualNicPortgroup $ManagementVMKPortGroupName -Confirm:$false

Now the clean up block. This removes the now redundant vSwitch0.

#Remove Original vSwitch0
Remove-VirtualSwitch -VirtualSwitch (Get-VirtualSwitch -VMHost $ESXHost  | Where-Object {$_.Name -eq $OldvSwitch}) -Confirm:$false

Note: If you have more than two VMNIC’s associated with the vSwitch, you will need to adjust this to include them.

Thanks for reading.

Home Lab – Equipment and Software: Part 1

Thanks for coming back! If you missed the first post in my Home Lab series you can find it here.

In this post I will begin drilling into the equipment and software that makes up my Home Lab and my reasoning for these choices.

I’m going to skip the original Raspberry Pi, there are enough blogs covering the use cases for them and begin at the first significant device; my MacBook Pro (late 2013). I wanted something mobile to start with so I could take it to work, use it on commutes to other offices etc. The MacBook came with an Intel i7 2.3Ghz Quad core chip, 16GB of memory and a 512GB SSD. This wasn’t going to be able to run everything, but its enough to run what I need when I’m away from home.

Before I dive into the VM’s and nested hosts, lets look at the networking configuration I used in VMware Fusion. I created four custom networks in total. One being a Management network for my ESXi Hosts and my VCSA. The second for vMotion and the other two as guest VM networks. None of these networks are NAT’d or have DHCP enabled, however I have selected the ‘Connect the host Mac to this network option in the VMware Fusion Preferences for the management network.

There are two ways to set these custom networks up. The first being the UBER Network Fuser and the second, editing the VMWare Fusion network file. In ‘/Library/Preferences/VMWare Fusion’, you will find the file called ‘networking’.

There are guides already available if you search google for either option so I won’t go into this further. This is the one I used – https://tinyurl.com/y7cjkhky.

Now onto the virtual machines. Running directly on VMware Fusion, I have a Windows Domain Controller / DNS Server, a PFSense firewall and two 6.7 ESXi hosts. They all use local storage, including the ESXi Datastores. My PFSense virtual firewall provides my layer 3 routing and eventually interfaces with my physical firewall. The Domain Controller/DNS Server is a ‘standard’ deployment, nothing special. The two ESXi hosts are the standard 6.7 image available from the VMUG Advantage subscription. Check out the last post for more on VMUG.

Then within ESXi, there’s my vCenter Appliance and the DR node of my vSphere Replication Appliances. At this point you might be wondering how I have fit this into 16GB of memory…

To start with, I built the first ESXi host with 12GB of memory and deployed my vCenter appliance on this (the tiny appliance requires 10GB). Once I had successfully deployed the appliance, I reduced the vCenter memory to 6GB and then followed this by reducing the ESXi host to 7GB.

I then created the second ESXi host, which also has has 7GB allocated. Its a tight squeeze but it allows me the basics I need when I’m not at home and there’s still enough room for some small VM’s with the nested ESXi hosts if needed.

One final thing… To ease the starting and suspension of this lab, I use the following script that I run from PowerShell on the Mac.

Lab_Start_Up

Write-Host "Starting PFSense Firewall"
vmrun start "/Users/<Path_to_VM>/PFSense.vmwarevm" nogui
Write-Host "Starting DC01"
vmrun start "/Users/<Path_to_VM>/DC01/DC01.vmwarevm" nogui
Write-Host "Starting ESX01"
vmrun start "/Users/<Path_to_VM>/ESX01/ESX01.vmwarevm" nogui
Write-Host "Starting ESX02"
vmrun start "/Users/<Path_to_VM>/ESX02/ESX02.vmwarevm" nogui

Write-Host "Start Up Complete!"

Lab_Supend

vmrun suspend "/Users/<Path_to_VM>/ESX02/ESX02.vmwarevm"
vmrun suspend "/Users/<Path_to_VM>/ESX01/ESX01.vmwarevm"
vmrun suspend "/Users/<Path_to_VM>/DC01/DC01.vmwarevm"
vmrun suspend "/Users/<Path_to_VM>/PFSense.vmwarevm"

Write-Host "Suspension Complete!"

That wraps up the high level MacBook Pro setup. Thanks for reading!

Downgrading VM Hardware Version

Having to downgrade a VM’s hardware version or compatibility level is something that comes up now and again.  There are many reasons you may need to do this; being it moving a VM to an environment with an older version of vSphere, or having issues following a planned upgrade that you now no longer have a snapshot for.

There are 3 VMware supported ways to downgrade which you can find in this VMware KB – https://kb.vmware.com/s/article/1028019

Let’s take a look at the options.

Revert to a previous snapshot.

Now this is only of use if you took a snapshot in the first place… If this is a planned upgrade, I would hope one was taken! However, you could have performed the upgrade and removed the snapshot before discovering the defect which is leading you to return to a lower hardware version.

If you do have an appropriate snapshot, reverting to it is a viable option.  You can achieve this by the usual means.  Right click the VM, hover over the Snapshot option and either select revert to snapshot, or manage snapshots if you have more than one or aren’t sure.

VMware vCenter Converter.

Another option is the VMware vCenter Converter.  This is a free tool available from VMware that can perform a number of conversion tasks.  One of these is the ability to copy the original VM to ‘another’ but allowing changes to the VM ‘hardware’ such as the hardware version, disk sizes, disk provisioning (Thick or Thin), CPU and memory changes and much more. (more info here —-). You also get the option to power off the original and power on the new as it completes.  This method reduces downtime but can be heavy going with high IO machines , in fact I wouldn’t do it with high IO machines.

Once your conversion is complete, you can then delete original. This is a great option with a simple to follow wizard, however it can take a fair amount of time to complete depending on the size of the VM.

Original VM, Hardware Version 14.

During the conversion you will need to have either renamed the original VM inventory name, or use a temporary name during the conversion, which, you can return to the original name once you delete the original.

You have the option to select any hardware version

Now power off, rename back to the original name and perform a storage vMotion to rename the files on the datastore.

Detaching and Re-Attaching the disks.

This is my personal preferred method! If I only need to modify the VM hardware version, this is my go to.  This involves creating a new VM with no hard disks and then attaching the disks from the original VM. This approach is particularly useful when your VM has large disks.  Lets go through the process.

Create a New Virtual Machine giving it a temporary name, the required hardware version, and  matching CPU, memory, SCSI controllers and network adapters.  One additional setting you will need to take care with is that you match the Firmware option.  Don’t worry about adding any hard disks. NB: If you have any dependencies for the original MAC addresses, if you take this route, you will need to note them and manually assign them to the new adapaters.

Now, ensure you have a backup, as you should do before any work, and take a snapshot the original VM to give you a fast rollback should you need it (quiesce the memory if you so wish). Power off the VM and rename it, <name>_old for instance.

Edit the VM settings and note the location of the disk(s).

Detach the hard disk(s), ensuring NOT to tick the option to delete from datastore (that will ruin your day…) 

Now edit the new VM settings and attach an existing disk

Browse to the disk(s) location and attach the disks starting with the OS disk first.

Power on and you have now downgraded the hardware version. If you are using static IP addresses, be prepeared to reassign these.

So we now have a bit of cleaning up to do… The inventory name is incorrect and the disks aren’t stored with the VM.  

You now need to power the VM off, rename it to the original name and perform a storage vMotion to rename and move all the VM files and disks into a single folder on the datastore.

When conducting any change, always ensure you have a backup and a valid rollback should something not go as planned!

Thanks for reading.

Installing PowerCLI using Install-Module

I was asked recently ‘Do we have PowerCLI downloaded?’.  Yes, we may, but it could be anywhere and it is likely an outdated version.

There is no need to download the installer! You can install PowerCLI using the Install-Module cmdlet in Windows PowerShell. (Providing you have an internet connection!) Below we will look at the steps required to install the latest version of PowerCLI on your system.

From an elevated PowerShell prompt run the following –

Install-Module VMware.PowerCLI.

If you don’t already have it installed, you will be prompted to install the NuGet Provider. Type ‘y’ and enter to continue.

You will get a further prompt to confirm you are happy to install a module from the ‘PSGallery’. Again, ‘y’ and enter to continue.

The PowerCLI Module will then begin to install. It will cycle through installing multiple dependent packages which will take a few minutes. Sit back and wait…

Once returned to the prompt, you can confirm the installation by running –

Get-Module VMware.PowerCLI -List Available | FL

You have now installed PowerCLI version 12.0.0.15947286. You will likely end up installing a later version.

Last step, load the module for use

Import-Module VMware.PowerCLI

You’re ready to go! But…

Not every system you need to use this module on will have internet access. In this, case the ‘Save-Module’ cmdlet is your friend.

Save-Module -Name VMware.PowerCLI -Path <Path to directory>

The module will then proceed to be downloaded into the directory you have specified and will look like this –

On your target server, you will need to confirm your module paths. You can do this by using the following command. You may have more than one path.

$env:PSModulePath

Now copy the directory that contains the module you have saved, to a module path on the target server. Likely ‘C:\Programfiles\WindowsPowerShell\Modules’ on a Windows System.

Now the Module is on your system, all that’s left is to import the module as above –

Import-Module VMware.PowerCLI

Thanks for reading! Hope this has been of use and catch you in the next post.