I recently did the first NSX upgrade in my Homelab! My Lab is running older hardware, HPE DL380 Gen8’s to be exact and this lead to an upgrade issue when attempting to upgrade my Edge node!
Here is the error I was faced with:
Edge 3.2.0.0.0.19067356/Edge/nub/VMware-NSX-edge-3.2.0.0.0.19067362.nub switch OS task failed on edge TransportNode c7aa6c6e-1a49-423a-a5c2-87cfa0ba9e1c: clientType EDGE , target edge fabric node id c7aa6c6e-1a49-423a-a5c2-87cfa0ba9e1c, return status switch_os execution failed with msg: An unexpected exception occurred: CommandFailedError: Command ['chroot', '/os_bak', '/opt/vmware/nsx-edge/bin/config.py', '--update-only'] returned non-zero code 1: b"lspci: Unable to load libkmod resources: error -12\nlspci: Unable to load libkmod resources: error -12\nlspci: Unable to load libkmod resources: error -12\nlspci: Unable to load libkmod resources: error -12\nlspci: Unable to load libkmod resources: error -12\nSystem has not been booted with systemd as init system (PID 1). Can't operate.\nERROR: Unable to get maintenance mode information\nNsxRpcClient encountered an error: [Errno 2] No such file or directory\nWARNING: Exception reading InbandMgmtInterfaceMsg from nestdb, Command '['/opt/vmware/nsx-nestdb/bin/nestdb-cli', '--json', '--cmd', 'get', 'InbandMgmtInterfaceMsg']' returned non-zero exit status 1.\nINFO: Corelist count is rounded to 2 from 3\nERROR: NSX Edge configuration has failed. 1G hugepage support required\n" .
This thread on VMTN provided the answer and an explanation, so I won’t repeat that here but I did want to share this.
In order to work around this issue, you need to power off the Edge node VM, add the following advanced setting and power back on.
featMask.vm.cpuid.pdpe1gb = Val:1
Once back online, you can continue with the upgrade!
Hopefully this will help any homelabbers out there running older hardware!
As always, thanks for reading!
If you like my content, consider following me on Twitter so you don’t miss out!
As part of my latest homelab setup, I run my vCenter Server VM and Domain Controller VM on VMware Workstation Pro 16 and not on an ESXi host. This is mainly so I don’t have to fire up a noisy rack mount server when I only need my vCenter Server for something.
This did leave me with the problem of a lack of physical ports in my Windows PC to be able to bridge my VM’s with, so out came the trusty USB Ethernet adapter!
By default, this USB device is connected to the host machine. As I would always forget to connect this to my domain controller VM(which is the VM I bridged with this interface), I would end up having to reboot it once I had connected it as I would get all sorts of problems, including DNS issues.
To address this, there is a line that can be added to the vmx file of the VM to auto connect a specific device to a VM. It took me a while to find the information that was pertinent to specifically version 16 of VMware Workstation so I thought I would write a quick article on it.
Firstly, you need to grab the VID and PID values from the registry by looking through the devices in the following path, or searching if you have an idea on what its friendly name is:
Once you have these values, open up the VMX file of the VM you want to auto connect the device to in a text editor, making sure the VM is in a powered off state first.
Add the following line to the file, replacing the VID and PID values with the ones you located in the registry earlier.
Now, power on the VM and you will see the device is automatically connected to the VM and not the Host!
I hope this has been useful. I aim to get some more blog posts up on my lab setup this year, so keep an eye out if you are looking for idea’s for your own home lab.
A useful thing for a home lab or VMware lab, is a certificate authority. There are Windows based CA’s as well as Linux based and many others. I wanted to take the Linux based route for my home lab to give me some administration time in Linux, being that Windows is my safe place! After a bit of googling, I settled on Easy-RSA as it looked like it would do what I needed in the lab. There are already a few guides out there for this, but this is my take on it for use in my VMware home lab.
I settled on CentOS 8 as a base OS. Why? Why not, I don’t have any Centos VM’s and I decided it would be good to use something other than Ubuntu or Photon.
Firstly, I stood up a low resource VM (1 vCPU, 1GB RAM) giving it a static IP and creating an admin account.
I then kicked off the update of all the install packages on the OS by first elevating to the root account using ‘SU’ and then running the upgrade command for the DNF package manager.
dnf upgrade
This prompts a ~600MB download after confirming you want to continue. Once the download completes it gets on with upgrading.
Once done, its time to install some additional packages starting with epel-release, easy-rsa and openssl. Lets quickly give some background to each.
epel-release (Extra Packages for Enterprise Linux) is a repository of popular packages which aren’t available by default. easy-rsa is one of the packages in this repository.
easy-rsa This is a utility for managing Public Key Infrastructure(PKI) aka Certificate Authority. Check out some info here.
openssl A widely used tool, in this case to create Certificate Signing Requests (CSR). I’ll let you read about this here.
Lets get to the install. You can run them as separate installs like this –
Now for ease of administration, create a directory in the admin users home directory and create a symbolic link so it remains updated. You also want to limit access to your admin user in my case ‘ca_admin’.
If you’re not familiar with chmod commands, ‘chomd 700’ = Protects a file against any access from other users, while the issuing user still has full access.
Now to initialise your new PKI. Change Directory (cd) to the easy-rsa directory you created in your admin users home directory and run the initialisation command.
cd ~/easy-rsa
./easyrsa init-pki
You will get a message showing it is complete and it will state the new pki directory that has been created inside the easy-rsa directory (/home/ca_admin/easy-rsa/pki)
You now need to create and populate a file called ‘vars’ in /home/ca_admin/easy-rsa and populate it with your organisation/lab information. You can achieve this with the vi editor.
Once created, you are now ready to create the root certificate and private key by running the following command –
./easyrsa build-ca
You will be prompted to specify a passphrase which you need to keep safe as you will need it when issuing certificates. There will then be a second prompt to provide a common name; Enter you CA’s name. eg. CA01.
This process will have now created your root certificate and the private key (keep this safe). You will find them in the following locations /home/ca_admin.easy-rsa/pki/ca.crt (root certificate) and /home/ca_admin.easy-rsa/pki/private/ca.key (private key).
If you are using a Windows device to access your HomeLab, you are going to want to add the ca.crl file to the ‘Trusted Root Certification Authorities’ store on your Windows device so that any certificates issued are trusted. You can copy the ca.crt file using a tool such as WINSCP to transfer the file to a local directory to then install. You can do the equivalent on Mac and Linux OS’s too.
You will also need this handy for any certificates that require the full chain to be included.
I won’t go into every Certificate Signing Request (CSR) scenario as there are many. I will however, show you the commands needed to produce a certificate from a CSR.
To issue a certificate from a CSR, you will need to copy the .req or .csr file to a directory such as /tmp on your CA server, again using a tool like WINSCP.
You can then run the following commands to import the certificate signing request. The Common name is often the device name or FQDN.
cd ~/easy-rsa
./easyrsa import-req /tmp/<csr_file_name>.req <CommonName or FQDN>
./easyrsa sign-req server <CommonName or FQDN>
The import command will import the .req or .csr file into /home/ca_admin.easy-rsa/pki/reqs (you can’t place the .req directly in here!) which is then processed by the sign-req command, again asking for the passphrase, leaving you with your new certificate in the /home/ca_admin.easy-rsa/pki/issued directory.
You can then use WINSCP again to transfer the file off the CA, and install it on the device or service in which you requested it from.
As always with any root CA, you don’t want it to become compromised. To help with this, keep it turned off when you’re not issuing, or administering your certificates.
I have also not included any Certificate Revocation List details as this isn’t something I need in my lab environment.
Now the VMware bit… below is the process for acquiring the CSR and installing the generated certificate on an ESXi host and a vCenter server using the methods above.
Standalone ESXi Host 6.7
First for a standalone ESXi Host, browse to – Host > Manage > Security & Users > Certificates
Select Import new certificate then select either ‘Generate FQDN signing request’ or ‘Generate IP signing request’.
You will be presented with a screen like this.
Copy this into a file with the extension .req. This can then be imported and issued using the method above.
Then, go back the the ‘Import new certificate’ wizard and import the certificate in the same format at the CSR into the box. (Open the .crt file using notepad)
Once complete close and open your browser and head back to your hosts web client and you will see you no longer have a certificate error.
vCenter Server 6.7
Log into your vCenter appliance using via SSH. Then run the following command –
/usr/lib/vmware-vmca/bin/certificate-manager
Select option 1, (you will be prompted to provide your SSO credentials), followed by option 1 again.
You will then need to provide the following information for the CSR.
As you complete the wizard you will have a .csr and a .key file in /tmp which again can be issued using the process above.
If using WinSCP you may hit the following error.
You will need to change over to the bash shell.
chsh -s /bin/bash root
You could then face another error…
root@vcsa02 [ ~ ]# chsh -s /bin/bash root
You are required to change your password immediately (password aged)
chsh: PAM: Authentication token is no longer valid; new one required
This is due to the password expiring. To change the password on the account run the passwd command
Further info on both of these errors can be found at these two VMware Articles. Here and here.
Once you have issued the certificate, you need to then copy the .crt file back to the /tmp directory along with the root certificate (or chain).
Now back to the Certificate Manager. Selecting option 1 to now import the certifictes. You will be prompted to provide the path and file name of each component. The certificate you created, the .key file that was created during the CSR generation and the root or CA chain certificate. Finally you will be asked to confirm you want to replace the Machine SSL certificate, type y.
It will take a few minutes, but eventually you will get confirmation that the task is complete and you can then reload your browser to see the Web Client is now showing a valid certificate.
Hope this has been useful. I will cover vCenter 7.0 Machine SSL certificate replacement in a future post.
Thanks for coming back! If you missed the first post in my Home Lab series you can find it here.
In this post I will begin drilling into the equipment and software that makes up my Home Lab and my reasoning for these choices.
I’m going to skip the original Raspberry Pi, there are enough blogs covering the use cases for them and begin at the first significant device; my MacBook Pro (late 2013). I wanted something mobile to start with so I could take it to work, use it on commutes to other offices etc. The MacBook came with an Intel i7 2.3Ghz Quad core chip, 16GB of memory and a 512GB SSD. This wasn’t going to be able to run everything, but its enough to run what I need when I’m away from home.
Before I dive into the VM’s and nested hosts, lets look at the networking configuration I used in VMware Fusion. I created four custom networks in total. One being a Management network for my ESXi Hosts and my VCSA. The second for vMotion and the other two as guest VM networks. None of these networks are NAT’d or have DHCP enabled, however I have selected the ‘Connect the host Mac to this network option in the VMware Fusion Preferences for the management network.
There are two ways to set these custom networks up. The first being the UBER Network Fuser and the second, editing the VMWare Fusion network file. In ‘/Library/Preferences/VMWare Fusion’, you will find the file called ‘networking’.
There are guides already available if you search google for either option so I won’t go into this further. This is the one I used – https://tinyurl.com/y7cjkhky.
Now onto the virtual machines. Running directly on VMware Fusion, I have a Windows Domain Controller / DNS Server, a PFSense firewall and two 6.7 ESXi hosts. They all use local storage, including the ESXi Datastores. My PFSense virtual firewall provides my layer 3 routing and eventually interfaces with my physical firewall. The Domain Controller/DNS Server is a ‘standard’ deployment, nothing special. The two ESXi hosts are the standard 6.7 image available from the VMUG Advantage subscription. Check out the last post for more on VMUG.
Then within ESXi, there’s my vCenter Appliance and the DR node of my vSphere Replication Appliances. At this point you might be wondering how I have fit this into 16GB of memory…
To start with, I built the first ESXi host with 12GB of memory and deployed my vCenter appliance on this (the tiny appliance requires 10GB). Once I had successfully deployed the appliance, I reduced the vCenter memory to 6GB and then followed this by reducing the ESXi host to 7GB.
I then created the second ESXi host, which also has has 7GB allocated. Its a tight squeeze but it allows me the basics I need when I’m not at home and there’s still enough room for some small VM’s with the nested ESXi hosts if needed.
One final thing… To ease the starting and suspension of this lab, I use the following script that I run from PowerShell on the Mac.
Home Labs are a fantastic way to do all manor of things from the comfort of your home. Ranging from home automation, file storage, home media streaming, learning how to build a website, learning a new operating system or exploring new technologies. My lab is mainly an VMware testing and learning lab.
Home Labs can come in many forms and sizes. From Raspberry PI’s to a full rack setup with a horrible electricity bill! Home Labs are the sort of thing that start small, but grow and evolve over time as your thirst for knowledge does.
In this Home lab blog I will be sharing my Home Lab journey in the hope it inspires others to begin their own journeys.
Like many others, my humble lab began with an Original Raspberry Pi many years ago until recently, in the last 6 months, I have invested in various bits of additional kit. The hardware has ranged from a late 2013 MacBook Pro that I use when I’m mobile (not so much at the minute!) to rack mount servers with both physical and virtual firewalls and switches, to tie it all together.
There are endless combinations of hardware, software, resources and subscriptions out there to suit each individuals goals.
Over the course of a series of posts in the Home Lab blog, I will share with you my Home Lab including hardware, software, subscriptions and learning resources I have used to to get me to the point I am at today.
I’d like to end this introduction post sharing with you, possibly, my most valued tool: My VMUG Advantage Subscription. This subscription provides you with access to online events and communities, money off training/exams and VMWorld tickets but, most importantly, access to non-production VMware licences!
As a VMUG Advantage member you get access to loads of VMware products to use in your Home Lab. Among the titles are vSphere, vCenter, vSAN, NSX, VCF and also the latest vSphere 7! This does how ever come at a cost… $200.
There is often a prompt for 10% off (code ADVNOW) which meant my subscription cost roughly £140 for 365 days of labbing! Now I appreciate not everyone may be able to afford this, in which case it is worth trying to ask your employer if they can cover the costs as a professional subscription?
This, without a doubt, has been one of the BEST resources I have had access to. Whether you are just wanting to get familiar with a new technology, dig deeper into a technology you already use, or prepare for a VMware certification, this is really an invaluable resource to have! Do take note, this cannot be used in a production environment!
Interested? Head over to the VMUG membership page to learn more!
Thanks for reading, I hope this has been useful. In the next post I will cover the equipment and software that makes up my Home Lab. See you then!
Stephan McTighe 