VMware ESX NIC Teaming

VMware ESX ships with the ability to connect multiple NICs to a single vSwitch–right out of the box.  No additional drivers or configuration needed.  You just need to click a few buttons to add the additional NICs.  You end up with something like this:

There are several advanced configuration options available, but for the most part it “just works” and can be set up very quickly.  This is one of the many great benefits of virtualizing your server hardware–individual virtual machines benefit from enhanced network redundancy without any additional configuration in the guest OS.  Plus, it works for all guests, regardless of OS vendor.

Hyper-V NIC Teaming

How does Hyper-V handle NIC teaming?  They punt and leave it up to the hardware (server and/or NIC) manufacturers.  This puts the system administrator in an unpleasant situation.  Take, for example, this thread on Mark Minasi’s Forum.  That is not an isolated incident, as you can see.

Since this feature is up to the server vendor, let’s see what one of them has to say about it.  According to this HP integration note, dated January 2009:

IMPORTANT
Windows Server 2008 Hyper-V does not support the Network Configuration Utility (NIC Teaming). Deselect this component before installing the PSP components.

Ouch.  But wait–HP just published a HOWTO paper on the topic.  Perhaps things are changing.  Evidently, the installation order of the teaming drivers and Hyper-V is critical:

NOTE
If the teaming software is installed first, the network adapters may cease passing traffic. The resolution to this issue is to uninstall both the HP teaming software and Hyper-V, reboot the server and then reinstall Hyper-V and the teaming software.

But this is my favorite part (Server Core fans, take note):

Uninstalling teaming software
If HP ProLiant Network Teaming Software is already installed on a server, it must be removed (uninstalled) prior to installing and enabling Hyper-V.

NOTE
There is no known method of uninstalling HP ProLiant Teaming software on machines running Windows Server 2008 Server Core. A tool for removing the teaming software is under development and will be available on the HP website in the future. HP recommends that any machine designated to run Server Core, Hyper-V, and teaming be configured as such during the initial installation, using the specified order of installation described above.

Even if Hyper-V is “free” and the HP NIC teaming drivers are “free,” your time is valuable.  Do you want to spend it configuring your hypervisor host OS, or getting real work done for your customers?

UPDATE You can read more about NIC teaming straight from Microsoft in this new post, via Cheng’s Random Thoughts.  Wow — don’t configure VLANs in SCVMM.  Where’s the single pane of glass?

Please note, the purpose of this article is to highlight the state-of-the-art in hypervisor NIC teaming –  not knock HP.  I am very satisfied with the HP Proliant servers that I use every day.  They run ESX like a champ!

Related posts:

1. Single Pane of Glass — Hyper-V Edition
2. It’s not Hyper-V — it’s the drivers!
3. Let them eat cake!
4. Thanks for all the port groups!

There are three different affected areas and you can fix each one manually or all in one shot with a VI Toolkit (for Windows) PowerShell script that I wrote for just this purpose.

1.  SCVMM Port Groups

First, let’s address the new port groups that SCVMM created when managing your VMware ESX VMs.  I covered this in more detail in a previous article.  Essentially, the SCVMM interface presents vSwitches, not port groups.  It then creates one or more new port groups for you without warning — one per vSwitch, per VLAN, on each ESX host.

To recover from this mess, you will need to reconfigure any VMs that are currently using these port groups and then delete the port groups from the vSwitches.  If you have more than a couple VMs this task will be easier with the script.

2.  Custom Attribute: ClusterInvariantVMMId

I bet you did not know that SCVMM would add a new strangely-named custom attribute to vCenter and tag every single VM in your inventory with a GUID.  I’ve heard puzzled VMware customers asking about this.  Initially, the explanation for this phenomenon was not clear — until I evaluated the SCVMM product and VMware vCenter integration.  If you are done with SCVMM, it is safe to remove this attribute with VI Client (on the Administration menu).

3.  Self-Service Role

Evidently the Self-Service Portal feature of SCVMM will allow users to provision and manage VMs on ESX hosts in addition to Hyper-V.  I tried this and it did not work, but I probably wasn’t persistent enough.  When you are done using SCVMM you won’t be needing that role anymore — go ahead and delete it.

PowerShell Cleanup Script

If you would like to clean up all three of these issues in one step, use the following VI Toolkit (for Windows):

# SCVMM-Cleanup.ps1
#
# When SCVMM connects to vCenter, it makes several changes and doesn't clean up
# after itself.  This script returns everything back to normal.
 
# Version 1.0  January 5, 2009
# Eric Gray
 
# modify this line accordingly
Get-VC vchostname -user xxx -Password yyy
 
# VMs using SCVMM-created port groups will be reconfigured to use this instead
$portGroup = "VM Network"
 
# Move all NICs from the SCVMM.vSwitchX.Y port groups to $portGroup
get-vm | Get-NetworkAdapter | where {$_.NetworkName -like "SCVMM*"} `
    | Set-NetworkAdapter -NetworkName $portGroup -Confirm:$false
 
# Delete those SCVMM port groups after they are no longer in use
foreach ($esx in Get-VMHost) {
    Get-VirtualPortGroup -VMHost $esx.Name | where { $_.Name -like "SCVMM*" } `
    | Remove-VirtualPortGroup  -Confirm:$false
}
 
# Clean up the custom attribute
Get-Datacenter | Remove-CustomField -Name ClusterInvariantVMMId -Confirm:$false
 
# Remove the SCVMM role
$authMgr = Get-View AuthorizationManager
foreach($role in $authMgr.RoleList | where {$_.Name -eq "SCVMMSelfServiceUser"}){
    $authMgr.RemoveAuthorizationRole($role.roleid, 1)
}

If you or one of your coworkers accidentally polluted your vCenter environment, this should help you get everything back in order.

Thanks go to Carter Shanklin for reviewing the script.

Related posts:

1. IGT Part 6: Cluster invari-what ID?
2. Managing VI3 with SCVMM considered harmful
3. Thanks for all the port groups!
4. VMware vCenter Server 4 task and event retention

VMware ESX virtual machine networking is pretty straightforward.  A virtual switch (vSwitch) is created on each host by default and is associated with one or more physical NICs.  When multiple physical NICs are used, virtual machines benefit from added redundancy and load balancing by simply connecting to such a vSwitch.

However, VMs don’t connect directly to vSwitches, they connect to port groups on a vSwitch. Port groups offer a means of configuring different settings, like VLAN and security, on a single vSwitch.  ESX virtual network administrators perform the infrequent task of creating port groups on hosts.  Later, virtual machine administrators simply select appropriate port groups when configuring VMs, seen as friendly names like this in the VI Client:

VM administrators do not necessarily know or care about VLAN or security settings, so this model abstracts some of the complexity that is a result of this network configuration flexibility.  Here is what a vSwitch with two port groups and two physical NICs looks like:

Unfortunately, if you use Microsoft System Center Virtual Machine Manager (SCVMM) to configure your VMware ESX virtual machines, you are only offered vSwitches in the configuration interfaces, not port groups:

When creating a new VM, or modifying an existing  one, SCVMM simply introduces new port groups on the fly into your environment.  Well, that’s one way to avoid having to figure out which existing port group to use, but it might have been better if the SCVMM interface actually presented the correct objects in the first place.

After making such a configuration change, the VM is then connected to a cryptically-named network like this:

And new port groups are added to the vSwitches on all ESX hosts, something the virtual network administrator might not appreciate:

Are you thinking about managing VMware ESX with SCVMM?  Are you also the virtual network administrator?  If not, you may have some explaining to do.

Related posts:

1. Clean up vCenter with PowerShell after SCVMM leaves
2. Yes, NIC teaming is not unsupported
3. Single Pane of Glass — Hyper-V Edition
4. Managing VI3 with SCVMM considered harmful