This is not an article to convince you to use one over the other; to see what experts have to say about the matter, take a look at a recent article from Duncan Epping on whether to go pure Distributed or hybrid.

Whenever this topic comes up for debate, it’s clear that the major concern about vDS is the inability to manage virtual networks if vCenter Server goes down.  Thus, the impetus for considering a hybrid environment with management interfaces on Standard vSwitches and only VM networking benefiting from vDS.  However, there is a configuration alternative that may just boost confidence in a pure vDS network.

Distributed Switch Port Group Bindings

Distributed Switches, just like Standard vSwitches, use port groups to configure various network capabilities, VLANs, etc.  One difference is that vDS port groups have three different binding options: static, dynamic, and ephemeral.  For an overview of these options, check out KB Article 1010593.

The key point to note is that port groups using ephemeral bindings behave very much like a Standard vSwitch — even with vCenter Server powered off, administrators have the ability to connect directly to an ESXi host and reconfigure VM networking.

In fact, it is even possible to create a new VM from scratch directly on a host while vCenter is offline, as seen here:

Behind the scenes, with vCenter unavailable, a temporary port is created on the host for the vNIC with the ID “h-1″:

After vCenter Server comes back online, everything syncs up and the VM is automatically updated with a numeric port ID from the vDS:

Leveraging Ephemeral Port Groups

If the inability to quickly provision a new VM or to reconnect a vNIC while vCenter Server is unavailable has kept you from considering a pure vDS network architecture, ephemeral port groups may be a suitable safety net.  You would not even need to use ephemeral port groups for production virtual networks — simply create a few to have as backups for accessing the most critical VLANs.

In reality, if vCenter is down that’s probably the first issue to tackle.  But, in order to recover from a minor catastrophe, it may necessary to manually register that VM and get it online — an idle ephemeral port group could save you the trouble of having to temporarily create a new vSwitch directly on a host.

Related posts:

1. Thanks for all the port groups!
2. NIC Teaming Update: Hyper-V Still Cloudy as Ever
3. Clean up vCenter with PowerShell after SCVMM leaves
4. SCVMM Damage Control

Two different vSwitches are provided in vSphere: Standard and Distributed.  Both offer NIC teaming for load balancing and fault tolerance, intuitive VLAN support, and Cisco Discovery Protocol (CDP) for easy lights out datacenter management.  Be sure to check out Jason Boche’s great overview of CDP.

vSphere Distributed Switch — Simple Network Management

The Distributed Switch adds advanced networking features to your virtual infrastructure, such as load-based teaming and private VLANs, and offers centralized port group management — eliminating the need to configure vSwitches and port groups individually on each host.  vSphere administrators can also choose to go with a hybrid model, maintaining a Standard vSwitch on each host — typically for management — and leveraging a Distributed Switch for virtual machine traffic.

Here you can see a Distributed Switch that spans four ESX hosts, utilizing two physical NICs per host :

By connecting these physical NICs to multiple trunk ports, virtual machines benefit from network redundancy and load balancing while making it trivial to create port groups for any VLAN required.  Configurations are instantly propagated across the cluster, boosting efficiency and minimizing human error.

Also shown above is the CDP information for one of the physical NICs, handy for identifying connected physical switch ports — no need to convince a coworker to visit the datacenter and help trace cables!

Hyper-V NIC Teaming Update

It’s been a while since I last wrote about the state of Hyper-V NIC teaming, some readers may be curious to know if anything has changed.  Let’s start our investigation by reviewing the latest official Microsoft support policy:

Since Network Adapter Teaming is only provided by Hardware Vendors, Microsoft does not provide any support for this technology…

Since Microsoft Hyper-V Virtualization is a new technology, we recommend that you thoroughly test your teaming solution in a test environment prior to deploying into Production.

Looks like it’s up to the end user to put this puzzle together.  Fortunately, enthusiastic bloggers have stepped up to provide some of the missing guidance to help decide which combination has the best chance of success.  Hyper-V expert Hans Vredevoort begrudgingly admits:

Now that Windows Server 2008 R2 SP1 was published almost a month ago, I have attempted to evaluate the consequences for this lovely combination.

There are just a few main elements to consider before attempting to bring robust private cloud networking capabilities to the Windows hypervisor:

• Windows Server service pack and hotfixes
• Third-party network drivers
• Third-party network teaming drivers
• VLAN requirements, and at what level in the stack to implement

[Math quiz: calculate the number of possible combinations administrators must consider.]

What’s all the fuss?  Just download the latest versions of everything and go.  Not so fast…

Experts Offered No Immunity

Stu Fox, Microsoft virtualization expert and VCritical reader, recently shared his experience with this very issue — encountering an outage after a simple driver update:

Hope for the Future?

It’s hard to say if Microsoft will ever integrate NIC teaming capabilities directly into the Windows platform; the next version is expected sometime next year.  For now Hyper-V users will just have to rely on the upcoming Virtual Machine Manager 2012 release, which will no doubt introduce complexity abstraction to VM networking in an effort to bring it closer to the simplicity vSphere administrators have enjoyed for years.

Is it any wonder Gartner concludes that Hyper-V is under-performing?

Related posts:

1. Yes, NIC teaming is not unsupported
2. The Secret of Ephemeral Port Groups
3. Thanks for all the port groups!
4. Single Pane of Glass — Hyper-V Edition

Layer 2 Networking

If you are a vSphere administrator, it may come as a surprise to find out that some public clouds do not permit layer 2 connectivity between virtual machines.  Actually, you may be even more surprised to learn that it is very typical for your VMs to be instantiated on completely different subnets.  That means that all data must flow through another device on the network — an Ethernet bridge or IP router, filtering traffic — even if two virtual machines are on the same IP subnet.

The OpSource Cloud, just like vCloud Director clouds, offers true layer 2 connectivity.  If two of your VMs are on the same network, they communicate directly, not through an intermediary security bridge or router.  Here you see that the VM I am currently logged into has the MAC address of another VM to which it connects:

Lower Latency

The above discussion on layer 2 connectivity may seem esoteric, but it ultimately manifests as very low latency between workloads running in the cloud — essential for multi-tier applications.  OpSource actually guarantees sub-millisecond response time, and in my experience they deliver:

Plenty of Bandwidth

For a quick test of bandwidth between two VMs, I used iperf with default options and attained well over 900Mb/s — plenty for most applications that would be deployed in this type of public cloud scenario.

Conclusion

Because OpSource Cloud is built on top of vSphere, it works like a vSphere administrator would expect:  VMs are deployed on known subnets with layer 2 connectivity, offering the best potential performance while making it much easier to monitor and diagnose any network-related issues that could arise.

Related posts:

1. OpSource Cloud Experience — Introduction
2. OpSource Cloud Experience — Connecting
3. OpSource Cloud Experience — Import/Export
4. Thanks for all the port groups!

VMware ESX NIC Teaming

VMware ESX ships with the ability to connect multiple NICs to a single vSwitch–right out of the box.  No additional drivers or configuration needed.  You just need to click a few buttons to add the additional NICs.  You end up with something like this:

There are several advanced configuration options available, but for the most part it “just works” and can be set up very quickly.  This is one of the many great benefits of virtualizing your server hardware–individual virtual machines benefit from enhanced network redundancy without any additional configuration in the guest OS.  Plus, it works for all guests, regardless of OS vendor.

Hyper-V NIC Teaming

How does Hyper-V handle NIC teaming?  They punt and leave it up to the hardware (server and/or NIC) manufacturers.  This puts the system administrator in an unpleasant situation.  Take, for example, this thread on Mark Minasi’s Forum.  That is not an isolated incident, as you can see.

Since this feature is up to the server vendor, let’s see what one of them has to say about it.  According to this HP integration note, dated January 2009:

IMPORTANT
Windows Server 2008 Hyper-V does not support the Network Configuration Utility (NIC Teaming). Deselect this component before installing the PSP components.

Ouch.  But wait–HP just published a HOWTO paper on the topic.  Perhaps things are changing.  Evidently, the installation order of the teaming drivers and Hyper-V is critical:

NOTE
If the teaming software is installed first, the network adapters may cease passing traffic. The resolution to this issue is to uninstall both the HP teaming software and Hyper-V, reboot the server and then reinstall Hyper-V and the teaming software.

But this is my favorite part (Server Core fans, take note):

Uninstalling teaming software
If HP ProLiant Network Teaming Software is already installed on a server, it must be removed (uninstalled) prior to installing and enabling Hyper-V.

NOTE
There is no known method of uninstalling HP ProLiant Teaming software on machines running Windows Server 2008 Server Core. A tool for removing the teaming software is under development and will be available on the HP website in the future. HP recommends that any machine designated to run Server Core, Hyper-V, and teaming be configured as such during the initial installation, using the specified order of installation described above.

Even if Hyper-V is “free” and the HP NIC teaming drivers are “free,” your time is valuable.  Do you want to spend it configuring your hypervisor host OS, or getting real work done for your customers?

UPDATE You can read more about NIC teaming straight from Microsoft in this new post, via Cheng’s Random Thoughts.  Wow — don’t configure VLANs in SCVMM.  Where’s the single pane of glass?

Please note, the purpose of this article is to highlight the state-of-the-art in hypervisor NIC teaming –  not knock HP.  I am very satisfied with the HP Proliant servers that I use every day.  They run ESX like a champ!

Related posts:

1. NIC Teaming Update: Hyper-V Still Cloudy as Ever
2. Single Pane of Glass — Hyper-V Edition
3. Getting eth0 back in a SLES for VMware clone
4. Thanks for all the port groups!

There are three different affected areas and you can fix each one manually or all in one shot with a VI Toolkit (for Windows) PowerShell script that I wrote for just this purpose.

1.  SCVMM Port Groups

First, let’s address the new port groups that SCVMM created when managing your VMware ESX VMs.  I covered this in more detail in a previous article.  Essentially, the SCVMM interface presents vSwitches, not port groups.  It then creates one or more new port groups for you without warning — one per vSwitch, per VLAN, on each ESX host.

To recover from this mess, you will need to reconfigure any VMs that are currently using these port groups and then delete the port groups from the vSwitches.  If you have more than a couple VMs this task will be easier with the script.

2.  Custom Attribute: ClusterInvariantVMMId

I bet you did not know that SCVMM would add a new strangely-named custom attribute to vCenter and tag every single VM in your inventory with a GUID.  I’ve heard puzzled VMware customers asking about this.  Initially, the explanation for this phenomenon was not clear — until I evaluated the SCVMM product and VMware vCenter integration.  If you are done with SCVMM, it is safe to remove this attribute with VI Client (on the Administration menu).

3.  Self-Service Role

Evidently the Self-Service Portal feature of SCVMM will allow users to provision and manage VMs on ESX hosts in addition to Hyper-V.  I tried this and it did not work, but I probably wasn’t persistent enough.  When you are done using SCVMM you won’t be needing that role anymore — go ahead and delete it.

PowerShell Cleanup Script

If you would like to clean up all three of these issues in one step, use the following VI Toolkit (for Windows):

# SCVMM-Cleanup.ps1
#
# When SCVMM connects to vCenter, it makes several changes and doesn't clean up
# after itself.  This script returns everything back to normal.
 
# Version 1.0  January 5, 2009
# Eric Gray
 
# modify this line accordingly
Get-VC vchostname -user xxx -Password yyy
 
# VMs using SCVMM-created port groups will be reconfigured to use this instead
$portGroup = "VM Network"
 
# Move all NICs from the SCVMM.vSwitchX.Y port groups to $portGroup
get-vm | Get-NetworkAdapter | where {$_.NetworkName -like "SCVMM*"} `
    | Set-NetworkAdapter -NetworkName $portGroup -Confirm:$false
 
# Delete those SCVMM port groups after they are no longer in use
foreach ($esx in Get-VMHost) {
    Get-VirtualPortGroup -VMHost $esx.Name | where { $_.Name -like "SCVMM*" } `
    | Remove-VirtualPortGroup  -Confirm:$false
}
 
# Clean up the custom attribute
Get-Datacenter | Remove-CustomField -Name ClusterInvariantVMMId -Confirm:$false
 
# Remove the SCVMM role
$authMgr = Get-View AuthorizationManager
foreach($role in $authMgr.RoleList | where {$_.Name -eq "SCVMMSelfServiceUser"}){
    $authMgr.RemoveAuthorizationRole($role.roleid, 1)
}

If you or one of your coworkers accidentally polluted your vCenter environment, this should help you get everything back in order.

Thanks go to Carter Shanklin for reviewing the script.

Related posts:

1. IGT Part 6: Cluster invari-what ID?
2. Managing VI3 with SCVMM considered harmful
3. Thanks for all the port groups!
4. VMware vCenter Server 4 task and event retention

VMware ESX virtual machine networking is pretty straightforward.  A virtual switch (vSwitch) is created on each host by default and is associated with one or more physical NICs.  When multiple physical NICs are used, virtual machines benefit from added redundancy and load balancing by simply connecting to such a vSwitch.

However, VMs don’t connect directly to vSwitches, they connect to port groups on a vSwitch. Port groups offer a means of configuring different settings, like VLAN and security, on a single vSwitch.  ESX virtual network administrators perform the infrequent task of creating port groups on hosts.  Later, virtual machine administrators simply select appropriate port groups when configuring VMs, seen as friendly names like this in the VI Client:

VM administrators do not necessarily know or care about VLAN or security settings, so this model abstracts some of the complexity that is a result of this network configuration flexibility.  Here is what a vSwitch with two port groups and two physical NICs looks like:

Unfortunately, if you use Microsoft System Center Virtual Machine Manager (SCVMM) to configure your VMware ESX virtual machines, you are only offered vSwitches in the configuration interfaces, not port groups:

When creating a new VM, or modifying an existing  one, SCVMM simply introduces new port groups on the fly into your environment.  Well, that’s one way to avoid having to figure out which existing port group to use, but it might have been better if the SCVMM interface actually presented the correct objects in the first place.

After making such a configuration change, the VM is then connected to a cryptically-named network like this:

And new port groups are added to the vSwitches on all ESX hosts, something the virtual network administrator might not appreciate:

Are you thinking about managing VMware ESX with SCVMM?  Are you also the virtual network administrator?  If not, you may have some explaining to do.

Related posts:

1. The Secret of Ephemeral Port Groups
2. Clean up vCenter with PowerShell after SCVMM leaves
3. NIC Teaming Update: Hyper-V Still Cloudy as Ever
4. Managing VI3 with SCVMM considered harmful