Following the recent PowerCLI 12.3 release, I’m sure many will be rushing to make use of some of the great new additions which you can find information on here.
I thought I would quickly show the process of upgrading to the latest version whilst uninstalling any older versions you might still have installed. Partly because I’m playing with creating video content at the moment! Credit to @colinwestwater for his article on how to clean up old versions, blog post here.
Some time back I wrote about setting up and enabling a HyTrust Key Management setup for vSphere to make use of VM and vSAN encryption. Following the release of vSphere 7.0 Update 2, VMware have introduced native key management capabilities! This is a great feature as you no longer require a potentially expensive separate key management solution to make use of vSphere’s encryption offerings.
Lets take a look at this new capability by heading over to the Key Providers menu on your vCenter object, and selecting ‘Add Native Key Provider’:
Give your provider a name:
It then needs backing up! There is an option to do this next to the ‘Add’ option, or in the flow graphic at the bottom:
It is recommended to protect this with a password, make sure you keep this safe along with the key itself, after it downloads when you hit ‘Back Up Key Provider’. You won’t be able to restore the provider without it should you have a need to. Without the provider, any VM’s or data encrypted with it will be lost.
Once its backed up and safely stored you will have an active KMS! You can choose to set it to default if you have more than one key provider if you wish. Any VM’s that are encrypted from the point of changing the default, will be with the new provider, any already encrypted VM’s will continue to be encrypted with the original key.
If you head over to vSAN services, you will now have your native key provider available and can enable Data-At-Rest encryption as well as Data-In-Transit encryption:
Likewise, if you edit the settings of a VM via the VM Options tab you will be able to enable VM encryption:
There you have it, a native Key Management capability, in built with vSphere 7.0 Update 2.
Having recently had to do some work with RDM perennial reservations I looked into ways to make this less of a manual headache. There are plenty of examples out there for doing this, which I took as a basis to make a PowerShell function. If anything it was a great way to refresh my PowerShell skills and an opportunity to learn some new skills.
Note: Although this has been tested in my environment, please make sure you test it appropriately before running against a production environment!
Lets take a look…
Get-PerennialReservation
This function targets a vSphere cluster, gets all RDM disks that are connected to VM’s and then queries each host in the cluster to check if the disk/storage device is perennially reserved or not.
There are multiple ways to use it, whether that is by specifying the target cluster using the -Cluster parameter or by piping it from Get-Cluster. You can also specify a specific canonical name or a comma separated string of them, if you just want the status of a single/select disk(s) using the -CanonicalName parameter. There is also an Export flag to export the results to CSV, if you wish to make use of the data outside of PowerShell. You can get the full usage information by running the following command once you have loaded the function into your PowerShell session:
This function again targets a vSphere cluster, gets all RDM disks that are connected to VM’s and sets the IsPerenniallyReserved flag too ‘True’ on all hosts.
There are multiple ways to use it like the Get function; specifying the target cluster using the -Cluster paramater or by piping it from Get-Cluster. You can still specify a specific canonical name or a comma separated string of them, if you just want to set the flag of a single/select disk(s) using the -CanonicalName parameter. There is still an Export function that will provide you an output to CSV. You can get the full usage information by running the following command once you have loaded the function into your PowerShell session:
To complete the set there is a Remove function. This function again targets a vSphere cluster, but this time you need to pass in the canonical name you wish to set the IsPerenniallyReserved flag too ‘False’ for.
To use this one, you need to specify the target cluster using the -Cluster paramater and specify a specific canonical name or a comma separated string of them, using the -CanonicalName parameter. There is still an Export function that will provide you an output to CSV. You can get the full usage information by running the following command once you have loaded the function into your PowerShell session:
There are times as a vSphere admin, you are going to want to run ESXCLI commands against multiple ESXi Hosts from a central location. This could be for configuration / administration, reporting, patching or a number of other things.
Recently I have been testing different values in the /DataMover/MaxHWTransferSize advanced setting. To make life easier, I wanted a way to change multiple hosts quickly and easily. To do this, I customised a script that Luc Dekens posted as a solution to a problem someone was having that can be used to send ESXCLI commands to multiple hosts using PowerCLI and plink.exe. This slightly modified version uses a CSV file as a source containing my hosts FQDN and the username and password I will be connecting with.
Plink, which is part of the PuTTy suite, can can be found here.
When using this script, you need to either run the script from a directory containing the plink executable, copy it to where you want to run the script, or adjust the script to include the path to the plink executable… whichever takes your fancy.
Disclaimer: Always complete your own testing in an appropriate environment and refer to the vendors official documentation!
$Hosts = Import-Csv C:\ESXiHosts.csv
$Commad = 'esxcfg-advcfg -s 16384 /DataMover/MaxHWTransferSize'
Foreach ($H in $Hosts) {
#Starting the SSH Service if not already started
$SSHService = Get-VMHostService -VMHost $H.HostName | where {$_.Key -eq 'TSM-SSH'}
if ($SSHService.Running -eq 'True') {
Write-Host "****************************" -ForegroundColor Blue
Write-Host "WARNING: SSH already enabled, this will be stopped on completion of this script" -ForegroundColor Yellow
}
Else {
Write-Host "Starting SSH Service on Host $($H.HostName)" -ForegroundColor Green
Start-VMHostService -HostService $SSHService -Confirm:$false > $null
}
#Running the defined ESXCLI Command(s)
Write-host "Running remote SSH commands on $($H.HostName)." -ForegroundColor Green
Echo Y | ./plink.exe $H.HostName -pw $H.Password -l $H.UserName $Commad
#Stopping the SSH Service
$SSHService = Get-VMHostService -VMHost $H.HostName | where {$_.Key -eq 'TSM-SSH'}
if ($SSHService.Running) {
Write-Host "Stopping SSH Service on Host $($H.HostName)" -ForegroundColor Green
Stop-VMHostService -HostService $SSHService -Confirm:$false > $null
Write-Host "****************************" -ForegroundColor Blue
}
}
Write-Host "Complete $(Get-Date)" -ForegroundColor Green
You can run as many commands as you need by declaring another ‘Command’ variable at the beginning of the script and adding another line to the ‘Running the defined ESXCLI Command(s)’ section.
When run, it will then cycle through each of the ESXi hosts from your CSV file, enable SSH (if its not already enabled), accept the host key, run the commands you have specified and finally turn the SSH service off.
Here you can see it has set the MaxHWTransferSize to 16384 on each host.
You will see the Recent Task pane show the SSH Service starts and stops.
The commands passed in can be anything you need. All you need to do is change the commands that are defined in the variables section. For example, restarting the management agents –
Recently I decided it was time to add a second vCenter 7.0 Appliance to my main lab environment after the lab containing my SRM and vSphere Replication installation ceased to exist…
I thought I would take the CLI route as its been a while, and thought I’d share!
To begin, you need to decide what you are deploying. There are four deployment options available to you, which you can see listed below. To see the options, mount the vCenter ISO image, browse to vcsa-cli-installer\templates\install, and you will find 4 templates;
Embedded on ESXi
Embedded on VC
Embedded replication on ESXi
Embedded replication on VC.
Note there is not a distributed option here anymore as this is depreciated in 7.0.
For my lab I will be using the 3rd option; ‘Embedded replication on ESXi’. Firstly because I’m deploying to a standalone host and not to an existing vCenter. Secondly as I already have an existing VCSA and SSO Domain. This new VCSA will be added, or linked to the existing VCSA for my ‘Recovery’ site, in my Site Recovery Manager (SRM) setup.
If you are looking to deploy your first VCSA, onto a standalone host, you will want to use the ‘Embedded on ESXi’ template.
Once you have decided on the template that suits your scenario, you are going to add some details to this template, such as the ESXi host information you are deploying to, networking information, NTP and in my case SSO details as I will be adding it to an existing SSO Domain. One important value is the deployment size (deployment_option in the example below).
A useful command that can be run to help you decide what size appliance is suitable for your needs is:
vcsa-deploy --supported-deployment-sizes
This outputs the vCenter sizing to assist you. It shows you the resource requirements as well as the amount of hosts and VM’s each can support.
For my lab, ‘tiny’ covers my needs.
Here is the json file I used for the deployment in my lab. I have excluded the passwords for obvious reason, but it can be ran like this, and will prompt you for the passwords in the terminal.
{
"__version": "2.13.0",
"__comments": "Sample template to deploy a vCenter Server Appliance with an embedded Platform Services Controller as a replication partner to another embedded vCenter Server Appliance, on an ESXi host.",
"new_vcsa": {
"esxi": {
"hostname": "smt-lab-esx-04.smt-lab.local",
"username": "root",
"password": "",
"deployment_network": "vSS_PG_Management",
"datastore": "smt-lab-vmfs-02a"
},
"appliance": {
"__comments": [
"You must provide the 'deployment_option' key with a value, which will affect the VCSA's configuration parameters, such as the VCSA's number of vCPUs, the memory size, the storage size, and the maximum numbers of ESXi hosts and VMs which can be managed. For a list of acceptable values, run the supported deployment sizes help, i.e. vcsa-deploy --supported-deployment-sizes"
],
"thin_disk_mode": true,
"deployment_option": "tiny",
"name": "smt-lab-vcsa-02"
},
"network": {
"ip_family": "ipv4",
"mode": "static",
"system_name": "smt-lab-vcsa-02.smt-lab.local",
"ip": "10.200.15.249",
"prefix": "24",
"gateway": "10.200.15.254",
"dns_servers": [
"10.200.15.10"
]
},
"os": {
"password": "",
"ntp_servers": "0.uk.pool.ntp.org",
"ssh_enable": true
},
"sso": {
"password": "",
"domain_name": "vsphere.local",
"first_instance": false,
"replication_partner_hostname": "smt-lab-vcsa-01.smt-lab.local",
"sso_port": 443
}
},
"ceip": {
"description": {
"__comments": [
"++++VMware Customer Experience Improvement Program (CEIP)++++",
"VMware's Customer Experience Improvement Program (CEIP) ",
"provides VMware with information that enables VMware to ",
"improve its products and services, to fix problems, ",
"and to advise you on how best to deploy and use our ",
"products. As part of CEIP, VMware collects technical ",
"information about your organization's use of VMware ",
"products and services on a regular basis in association ",
"with your organization's VMware license key(s). This ",
"information does not personally identify any individual. ",
"",
"Additional information regarding the data collected ",
"through CEIP and the purposes for which it is used by ",
"VMware is set forth in the Trust & Assurance Center at ",
"http://www.vmware.com/trustvmware/ceip.html . If you ",
"prefer not to participate in VMware's CEIP for this ",
"product, you should disable CEIP by setting ",
"'ceip_enabled': false. You may join or leave VMware's ",
"CEIP for this product at any time. Please confirm your ",
"acknowledgement by passing in the parameter ",
"--acknowledge-ceip in the command line.",
"++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"
]
},
"settings": {
"ceip_enabled": true
}
}
}
Once you have prepared your file, there are a couple of commands you can run from a PowerShell prompt to validate your configuration before deploying, saving you some time should mistakes have been made. The first being:
.\vcsa-deploy.exe install --accept-eula --acknowledge-ceip --verify-template-only <Path to json File>
This completes some basic checks to ensure your json file is correct, here is a successful output:
Secondly:
.\vcsa-deploy.exe install --accept-eula --acknowledge-ceip --precheck-only <Path to json File>
This will perform a more in depth validation, checking things like the credentials for your SSO domain, DNS or whether the IP or name you plan to use for your VCSA is in use already.
Note: Make sure you have your DNS setup correctly and is resolving the appliance FQDN!
It will also provide warnings if it thinks you might not be using an appropriate template. I originally specified a host what was already managed by vCenter, so it warned me like so:
You will get a similar output to the first command, should you pass all the tests. If not you will need to look at resolving them to ensure you get a successful deployment.
The Install!
Once you are confident you have everything in place, including DNS, and your configuration files are correct, you are ready to install:
.\vcsa-deploy.exe install --accept-eula --acknowledge-ceip --no-ssl-certificate-verification <Path to json File>
Here is a cut down version of the output you will see during the deployment:
====== [START] Start executing Task: To validate CLI options at 12:46:25 ======
Command line arguments verfied.
[SUCCEEDED] Successfully executed Task 'CLIOptionsValidationTask: Executing CLI
optionsValidation task' in TaskFlow 'template_validation' at 12:46:26
[START] Start executing Task: To validate the syntax of the template. at
12:46:27
Template syntax validation for template
'M:\Software\VMware\vCenter\embedded_vCSA_replication_on_ESXi.json' succeeded.
Syntax validation for all templates succeeded.
====== [START] Start executing Task: Perform precheck tasks. at 12:46:39 ======
[START] Start executing Task: Verify that the provided credentials for the
target ESXi/VC are valid at 12:46:45
The certificate of server 'smt-lab-esx-04.smt-lab.local' will not be verified
because you have provided either the '--no-ssl-certificate-verification' or
'--no-esx-ssl-verify' command parameter, which disables verification for all
certificates. Remove this parameter from the command line if you want server
certificates to be verified.
================== [START] Start executing Task: at 12:47:47 ==================
= [SUCCEEDED] Successfully executed Task '' in TaskFlow 'install' at 12:47:47 =
[START] Start executing Task: Check whether the datastore's free space
accommodate the VCSA's deployment option at 12:47:51
[SUCCEEDED] Successfully executed Task 'Running precheck: TargetDsFreespace' in
TaskFlow 'install' at 12:47:51
==========VCSA Deployment Progress Report========== Task: Install
required RPMs for the appliance.(RUNNING 5/100) - Setting up storage
VCSA Deployment is still running
==========VCSA Deployment Progress Report========== Task: Install
required RPMs for the appliance.(SUCCEEDED 100/100) - Task has completed
successfully. Task: Run firstboot scripts.(SUCCEEDED 100/100) - Task has
completed successfully.
Successfully completed VCSA deployment. VCSA Deployment Start Time:
2020-12-28T13:19:19.291Z VCSA Deployment End Time: 2020-12-28T14:18:27.103Z
[SUCCEEDED] Successfully executed Task 'MonitorDeploymentTask: Monitoring
Deployment' in TaskFlow 'embedded_vCSA_replication_on_ESXi' at 14:18:45
Monitoring VCSA Deploy task completed
The certificate of server 'smt-lab-vcsa-02.smt-lab.local' will not be verified
because you have provided either the '--no-ssl-certificate-verification' or
'--no-esx-ssl-verify' command parameter, which disables verification for all
certificates. Remove this parameter from the command line if you want server
certificates to be verified.
== [START] Start executing Task: Join active domain if necessary at 14:18:59 ==
Domain join task not applicable, skipping task
[SUCCEEDED] Successfully executed Task 'Running deployment: Domain Join' in
TaskFlow 'embedded_vCSA_replication_on_ESXi' at 14:18:59
[START] Start executing Task: Provide the login information about new
appliance. at 14:19:10
Appliance Name: smt-lab-vcsa-02
System Name: smt-lab-vcsa-02.smt-lab.local
System IP: 10.200.15.249
Log in as: Administrator@vsphere.local
[SUCCEEDED] Successfully executed Task 'ApplianceLoginSummaryTask: Provide
appliance login information.' in TaskFlow 'embedded_vCSA_replication_on_ESXi' at
14:19:10
=================================== 14:19:16 ===================================
Once complete, you will now have a second vCenter appliance deployed in Linked mode with the original. Here it is once I had configured a datacenter and cluster with two hosts.
Tags are a really useful component in VMware. They can be used for all manor of things, whether it’s for storage policies, backups, identifying a group of objects or in the case of this post, managing permissions.
Having a method of easily assigning permissions to singular or multiple objects in vCenter can be a great benefit to a vSphere Admin as it’s gives them greater control over the environment they manage.
Lets take a look at what is needed to get this setup:
Script
Tag Category & Tags for each support role.
AD Security Groups
AD Service Account
vCenter Roles (one for the service account, then one for each of the support roles)
PowerCLI VICredentials
Scheduled Task
In this example I will use 4 common support teams that could be used, DBA, EUC, Operations and Storage. These can be anything you have a requirement for.
Script
Here is the script that applies the permissions based on the assigned tags. It can also be found here on GitHub. Save this on your management server of choice, or wherever you intend to run the scheduled task as a .PS1 file. In this example it’s saved on a management server in C:\Scripts\VI_Permissions.ps1.
Now onto Tag Categories and Tags in vCenter. Create a Tag category called ‘Support _Teams’ (Or something of your choosing, just make sure you are consistent throughout):
Or using PowerShell – New-TagCategory -Name Support_Teams -Cardinality Multiple -EntityType All
You can select as many object types as you wish and you will also want to allow multiple tags per object.
Now create a tag for each of the support teams in the tag category you just created:
Now for some corresponding AD Security Group for each role you wish to have:
Service Account (AD User)
Now to create an AD user account that will be used to apply the permissions within vCenter. This will be the account that will be used to run the scheduled task, connect to vCenter and will have the appropriate permissions to assign permissions for the support roles.
Support Team Roles
Now we need to create a suitable role for each team. In this example I have copied the Virtual Machine Power User role, but these roles can contain which ever privilege’s you require.
Under ‘Administration > Roles’ you will see the options to either create a new Role or copy an existing. From here you will be able to assign it a name and specify the privilege’s you require.
You will be referencing these Role names in the script so make sure you continue to match the names thought the process.
Permissioning Role
As mentioned in the service account section, the account (tag_permissions) running the scheduled task will need permissions in vCenter through a role. The privileges this role will hold, needs to include all the privilege’s that are referenced in all of your Support Team Roles in order for it to have the right to assign the permissions. For example, if all your support roles are a copy of the ‘Virtual Machine power user’ role, your tagging permissions role will need to contain the same privileges.
Depending on how broad the scope of your support team roles, you may want to use the ‘Administrator’ or the ‘No cryptography administrator’ role. This is entirely up to you and how you manage your estate.
For this example in my lab, I will use the predefined ‘Administrator’ role to grant the ‘tag_permissions’ AD account permissions at the Global Root, ensuring you have selected the ‘Propagate to children’ option.
You could create a copy of the ‘Administrator’ role and name it something like ‘VI Permissions Service’ for instance, to give you flexibility to modify it in the future as well as making it easy to identify. With any high privileged account, ensure you secure it appropriately.
Create VI Credential Item
Now to create an encrypted credentials file that the service account running the scheduled task can import and then use to connect to vCenter without any intervention.
The AD account that is used to run the scheduled task, must be the account that also creates the credentials file as this is the only user that can use it. It will require permissions to run PowerShell and have access to a folder location to store the credentials file on your chosen management server.
To begin, start a PowerShell session in the context of the service account:
Note: Ensure the server that you are running this scheduled task from has PowerCLI installed. Installing PowerCLI.
Then run the following, entering your vCenter FQDN and the user and password that you created:
Ensure you are storing the file somewhere with appropriate access to allow this but, also to restrict any unnecessary access. The credentials file can have the password read if the user account that created it is compromised and gains access to the file using those windows credentials.
Scheduled Task
Now for the last component, the scheduled task. On a management server or a server of your choosing, create a scheduled task:
Assign an appropriate schedule that suits the level of change and size of your environment:
Now configure the trigger to execute the script:
Now thats everything you need to set this up, so lets give it a run though!
Assigning Tags and Permissions
Lets take a look at my demo VM permissions before we begin assigning permissions:
Lets check the VM permissions before having any tags assigned:
Now you can either manually run the scheduled task or wait until its next scheduled run time. Once the job has run, you can now check the tags match the permissions assigned by running the following:
Finally, if you want to know which objects are supported by a specific team and have access you can check this by running:
Get-TagAssignment | Where {$_.Tag -like "Support_Teams/DBA_Team"}
You now have a way of assigning and removing permissions from vCenter objects using Tags. In this example I have used virtual machine object, but depending on your requirements, and the scope you set on the tag category, you could use this for other vCenter objects.
IT security, accountability and auditability are critical today. Securing vCenter Server using auditable identities, for instance via an Active Directory identity source, is likely common for most vCenter consumers. This ensures individual access can be used to audit actions back to an admin as well as provide higher security through strong password policies and the absence of a shared account credentials, such as the SSO administrator.
Something that can be overlooked is the security of the ESXi hosts themselves.
There are multiple options for securing your ESXi hosts, one being the use of lockdown modes and limited user access, restricted management network access, or a combination of them both.
In this blog post I want to show you a simple way to configure your hosts to use an Active Directory group to control user access to an ESXi host.
This is achieved by editing the value associated with an advanced setting – ‘Config.HostAgent.plugins.hostsvc.esxAdminsGroup’.
One prerequisite to achieving this is that the hosts must be domain joined. Information on how to do this can be found here.
Editing this on a handful of servers is easy enough if you are doing it manually, but who wants to be manually typing or copying & pasting this?
I have put together a PowerShell Function that can simplify the editing of this setting for single hosts, or on mass to all ESXi hosts that you have connected your PowerCLI session to.
Set-ESXiHostAdminGroup -Target single -Entity esxi01 -Group "infrastructure_admins"
Set-ESXiHostAdminGroup -Target all -Group "infrastructure_admins"
For the purposes of this demo, I will update all ESXi Hosts within my vCenter instance using the second example. You could edit the function to target clusters if you wish.
Setting before –
Set-ESXiHostAdminGroup -Target all -Group "infrastructure_admins"
Function Output
After –
You will need to wait the length of time defined in this setting until being able to log in –
Now all users that are in the ‘infrastructure_admins’ Active Directory group will be able to log into the ESXi host with administrator permissions using their AD credentials.
Following on from my last post on vSphere 7.0 certificate Management, I wanted to continue with another certificate related post. This one being Site Recovery Manager (SRM) 8.3. Like vSphere 7.0, this version seems simpler than previous versions I have used.
With SRM, it’s the Appliance Certificate replacement that I am going to take you through in this blog post.
Firstly log into the SRM appliance management console via https://<srm-fqdn>:5480 and select the ‘Certificates’ option on the left, followed by ‘Generate CSR’ in the top right.
Fill in the information for your certificate, then click ‘Generate and Download’. You then need to process the CSR with your certificate authority, whether thats an internal, public or lab CA.
Once you have your certificate, select the ‘Certificates’ option on the left again, this time followed by ‘Change’ in the top right.
Select the last option in the Select certificate type section; ‘CA-signed certificate generated from CSR’. Then, browse both your newly generated certificate and either you root CA certificate, or the CA chain. Click ‘Change’ once done.
This should complete the replacement of the SRM appliance certificate!
If like me you get an error complaining that the IP or Common name / SAN is missing, make sure the local host field is set to the FQDN when connecting SRM to vCenter.
vCenter 7.0 brings many new features, one of which is a much smoother certificate management experience. There are now 4 main ‘modes’ for certificate management.
These are; Fully Managed Mode, Hybrid Mode, Subordinate CA Mode and finally Full Custom Mode. There is a great article here from Bob Plankers explaining the difference between each.
As mentioned in Bob’s blog, Hybrid Mode is the recommend option, and I will show you that process here in this blog.
Firstly, in your vSphere Client, browse to Administration > Certificates. Then click Actions and select ‘Generate Certificate Signing Request (CSR)’.
Complete the required fields with your information, making sure you have at least added the common name as a Subject Alternative Name to avoid issues with modern browsers. Click Next.
Finally, copy or download your CSR to generate the certificate on the CA of your choosing. Click Finish when ready.
Once you have your certificate, return to Administration > Certificates and this time select ‘Import and Replace Certificate’.
You then need to select the second option. This may seem slightly deceiving but it effectively is the option you need when you have generated the CSR from vCenter like this.
Now browse and select both your freshly produced certificate, and the root certificate or certificate chain if you have issuing CA’s.
Hit replace, then wait for the Web Client to restart with the new certificate.
Now one final step is needed to complete Hybrid Mode. You need to download the VMCA Root certificate from https://<vCenterFQDN by clicking the ‘Download trusted root CA certificates’ option and distributing it to your vSphere admins.
Once distributed and installed on your vSphere admins client devices, they should not get certificate errors when either browsing to vCenter or the hosts it manages.
You could however, get this error due to the default certificate having a 5 year validity period and not being within the new ‘standard’ of 398 days.
NET::ERR_CERT_VALIDITY_TOO_LONG
If you receive this, you will want to adjust the vpxd.certmgmt.certs.daysValid value in the vCenter Advanced Settings. It defaults to 1825, making it 365 (one year) will stop this.
You can then renew the certificate on each host by clicking ‘Renew’ in the Configure > Certificates menu –
Before (5 years) –
After (1 Year)-
If you want to do this renewal via PowerCLI (because…well why wouldn’t you!?) there is a nice function here by Ankush Sethi which does a great job.
I recently assisted a friend who had an issue with DFS Namespaces following an Active Directory Upgrade from 2008R2 to 2012R2. They were faced with not being able to access the NameSpace following the demotion of the last 2008R2 controller and promotion of the final 2012R2 controller.
Upon opening the DFS NameSpace management console, the following error was displayed when selecting the required NameSpace – “The namespace cannot be queried. Element not found.”
After looking in the FRS (File Replication Service) and DFSR (Distributed File System Replication) event logs, I came to realise that the forest was using FRS for replication! This isn’t supported after 2008R2. Ideally, you would have completed the migration from FRS to DFSR before upgrading the domain controllers.
Note: Always make sure you have a backup, snapshot or other reliable rollback method in place before doing anything in your live environment. This worked for me, it doesn’t guarantee it will work for you!
With FRS being the likely cause, I needed to confirm this. I ran the following command to confirm the status –
Dfsrmig /getglobalstate
It returned the following result confirming that FRS was still in fact being used.
Current DFSR global state: 'Start'
Succeeded.
Before being able to look at the DFS NameSpace issue, this needed addressing. Luckily you can still remediate this after upgrading the domain controllers. I would still advise confirming all the prerequisites are in place BEFORE upgrading!
Now onto the migration from FRS to DFSR.
Firstly, run the following command to move the state to the second of the four states. The four states being; Start, Prepared, Redirected and Eliminated.
Dfsrmig /setglobalstate 1
You will then want to run a directory sync to speed things up, especially if you have a large replication interval!
Run the following RepAdmin command to get things moving.
Repadmin /syncall /AdeP
You can then monitor the progress by running –
Dfsrmig /getmigrationstate
You will then see any remaining domain controllers that are yet to have synchronized the new state. Eventually you will see that all domain controllers have migrated to the second state; Prepared.
Now time to move to the Redirected state. Same process as the previous set but this time specifying ‘setglobalstate 2’
Again run the RepAdmin to get replication moving and monitor using the ‘getmigrationstate’ command. As in the previous step, you will eventually see that all domain controllers have migrated to the third state; Redirected.
Last one! Same as before, but this time you want to use ‘setglobalstate 3 –
Once complete you will get confirmation that you have reached the final state; Eliminated.
You will now be able to run the ‘net share’ command to see that the SYSVOL share has been moved to ‘C:\Windows\SYSVOL_DFSR\sysvol’ and that the FRS Windows service is stopped and set to disabled.
Output of the ‘net share’ command
File Replication Service (FRS) service
This should now give you a correctly functioning directory again! You will want to now check the Directory Services, File Replication and DFSR Logs in Windows Event Viewer to ensure you have no further errors.
Now onto repairing the NameSpace. I read a few different blogs and guides for this, some included deleting the NameSpace via ADSI Edit others didn’t.
I found I didn’t need to delete anything, bonus.
The get the NameSpace accessible again I found that right clicking the NameSpace and removing it, followed by recreating it using the ‘New NameSpace Wizard’ did the trick.
Upon recreating it, all of the folders reappeared and were accessible again with no additional configuration required. (these screenshots are of my lab, not the live environment as it was not appropriate)