Category Archives: NSX

NSX-T Edge Node 3.2 Upgrade Issue – Hugepage Support

I recently did the first NSX upgrade in my Homelab! My Lab is running older hardware, HPE DL380 Gen8’s to be exact and this lead to an upgrade issue when attempting to upgrade my Edge node!

Here is the error I was faced with:

Edge 3.2.0.0.0.19067356/Edge/nub/VMware-NSX-edge-3.2.0.0.0.19067362.nub switch OS task failed on edge TransportNode c7aa6c6e-1a49-423a-a5c2-87cfa0ba9e1c: clientType EDGE , target edge fabric node id c7aa6c6e-1a49-423a-a5c2-87cfa0ba9e1c, return status switch_os execution failed with msg: An unexpected exception occurred: CommandFailedError: Command ['chroot', '/os_bak', '/opt/vmware/nsx-edge/bin/config.py', '--update-only'] returned non-zero code 1: b"lspci: Unable to load libkmod resources: error -12\nlspci: Unable to load libkmod resources: error -12\nlspci: Unable to load libkmod resources: error -12\nlspci: Unable to load libkmod resources: error -12\nlspci: Unable to load libkmod resources: error -12\nSystem has not been booted with systemd as init system (PID 1). Can't operate.\nERROR: Unable to get maintenance mode information\nNsxRpcClient encountered an error: [Errno 2] No such file or directory\nWARNING: Exception reading InbandMgmtInterfaceMsg from nestdb, Command '['/opt/vmware/nsx-nestdb/bin/nestdb-cli', '--json', '--cmd', 'get', 'InbandMgmtInterfaceMsg']' returned non-zero exit status 1.\nINFO: Corelist count is rounded to 2 from 3\nERROR: NSX Edge configuration has failed. 1G hugepage support required\n" .

This thread on VMTN provided the answer and an explanation, so I won’t repeat that here but I did want to share this.

In order to work around this issue, you need to power off the Edge node VM, add the following advanced setting and power back on.

featMask.vm.cpuid.pdpe1gb = Val:1

Once back online, you can continue with the upgrade!

Hopefully this will help any homelabbers out there running older hardware!

As always, thanks for reading!

If you like my content, consider following me on Twitter so you don’t miss out!

https://platform.twitter.com/widgets.js

NSX-T Manager Certificate Replacement

I decided it was time to add VMware NSX-T to my HomeLab. I had been putting it off for a while but I couldn’t avoid it any longer!

Once I had fired up my NSX Manager Nodes and Cluster (I am using version 3.1), I looked to installing certificates. I choose to use a single certificate for all 3 of the NSX managers and the cluster using Subject Alternative Names (SAN’s) to simplify the process and this means I don’t need to renew 4 certificates each time.

As this is a different process to other VMware products I have put together a quick run through on how to achieve this.

Firstly, we need to generate the CSR from one of the NSX Manager nodes using openssl. SSH to one of your nodes and run the following command to create a new file called ‘ssl.conf’:

vim ssl.conf

Then populate this file with the below text, changing the values to suit your environment. I have left my values in to help with reading the file. If you are using a single NSX manager in your lab, you can remove the lines for DNS.3, DNS.4, IP.3 and IP.4.

[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[ req_distinguished_name ]
countryName = GB
stateOrProvinceName = Labshire
localityName = Lab City
organizationName = SMT-Lab
organizationalUnitName = SMT-Lab
commonName = vm-nsx-00.smt-lab.local

[ req_ext ]
subjectAltName = @alt_names

[alt_names]
DNS.1 = vm-nsx-00.smt-lab.local
DNS.2 = vm-nsx-01.smt-lab.local
DNS.3 = vm-nsx-02.smt-lab.local
DNS.4 = vm-nsx-03.smt-lab.local
IP.1 = 10.200.15.34
IP.2 = 10.200.15.35
IP.3 = 10.200.15.36
IP.4 = 10.200.15.37

Now to generate the CSR, run the following, but replacing the files names to suit:

openssl req -out vm-nsx-00.smt-lab.local.csr -newkey rsa:2048 -nodes -keyout vm-nsx-00.smt-lab.local.key -config ssl.conf -sha256

This will generate 2 files in the current working directory. You will have your CSR and the private key. Using something like WinSCP, copy the files off the NSX manager to a location of your choice.

Head off to your CA and issue the certificate using the CSR.

Now you need to copy the root and issuing (if you have an issuing CA) certificate to certificate you just created. This will complete the chain. Also have the private key handy as you are going to need it.

We are now ready to import import the certificate. Head to System > Certificates > Import and select Import Certificate.

Give it a name, browse to the certificate file that now includes the certificate chain, followed by browsing the the private key file. Be sure to change the ‘Service Certificate’ slider to ‘No’ and then click Import.

Once imported you can select it and see that it includes the certificates in the chain.

Now to assign them! Firstly, click on the identifier next to the name and copy the value. This is what will be used to target the certificate in the next steps.

To validate and replace the certificates in NSX we need to use API’s. Using a tool like Postman, validate then replace the certificate on the NSX Manager Cluster by running the following as a GET request. Note you need to provide credentials for the NSX managers on the Authorization tab.

https://vm-nsx-00.smt-lab.local/api/v1/trust-management/certificates/82c80092-3571-40cd-8960-3189594ec0f1?action=validate 

The result of ‘”status” : “ok”‘ is what we are looking for here.

Now its confirmed valid, lets replace the certificate by running the following POST request:

https://vm-nsx-00.smt-lab.local/api/v1/cluster/api-certificate?action=set_cluster_certificate&certificate_id=82c80092-3571-40cd-8960-3189594ec0f1

Then its time to apply to all nodes by running each line below:

https://vm-nsx-01.smt-lab.local/api/v1/node/services/http?action=apply_certificate&certificate_id=a94b3600-696b-43bf-a2df-c1e8e2180c3a

https://vm-nsx-02.smt-lab.local/api/v1/node/services/http?action=apply_certificate&certificate_id=a94b3600-696b-43bf-a2df-c1e8e2180c3a

https://vm-nsx-03.smt-lab.local/api/v1/node/services/http?action=apply_certificate&certificate_id=a94b3600-696b-43bf-a2df-c1e8e2180c3a

And that completes the replacement. If you browse to either your cluster address or individual nodes, you will see your new certificate in place.

You can find the full VMWare documentation on this here.

Thanks for reading!