• Full accounting for all provisioned storage space and complete monitoring
• Automatic response to a nearly-full datastore through vCenter Alarms, PowerShell, and Storage VMotion

You may be nearly convinced to start using thin provisioning, but still wondering…

What happens if a datastore fills up?

When a datastore runs out of space, thin-provisioned virtual disks can no longer dynamically grow to accommodate additional storage demand.  When VMware ESX detects this condition, virtual machines in need of additional storage are instantly paused to prevent guest operating systems from failing. Conversely, VMs that that read and write to existing allocated storage blocks will continue running without issue — not all virtual machines will be paused just because a datastore is out of space.

If you ever find yourself in this situation, it’s not hard to fix.  Here is one simple approach, step-by-step:

1. Free up some space by deleting or moving files — ISO images or powered-off VMs would be perfect
2. Resume one of the paused VMs
3. Use Storage VMotion to move the disks for that VM to another datastore
4. Resume the remaining VMs

Watch the procedure in action:

Depending on the size and storage demand of each VM, additional migrations may be needed.  An alternative resolution would be to add additional space to the SAN LUN and grow the VMFS volume.

The Experiment

To simulate a sudden storage demand by the thin-provisioned VMs in the above video, I simply copied a large file from a network share to each Windows Server 2003 VM simultaneously.

For the curious, below is a PowerShell script for the task.  Run it from anywhere — it uses Sysinternals psexec to remotely initiate a file copy on each VM from a network share.

# VM naming convention combines this string with 2-digit number
$vmPrefix = "VM2003e-"
 
# UNC path to a large file that will be copied into each VM
$iso="\\fileserver\ISO\large.iso"
 
# credentials to download above file from \\fileserver
$user="domain\username"
$pass="pa55word"
 
# pass this function a list of numbers
function createVmList ($series) {
    $vmList = @()
 
    foreach ($id in $series) {
        $vmList +=  %{"$vmPrefix{0:00}" -f $id}
    }
    $vmList
}
 
# make sure Sysinternals psexec is in your path
function psexecOnVm ($cmd, $vmList) {
    $vmList | % {Invoke-Expression "psexec.exe \\$_ -d -u $user -p $pass $cmd"}
}
 
psexecOnVM "cmd /c copy /Y $iso c:\" (createVMlist (31..39))

VMware ESX is Resilient

You may have been surprised at how easy it is to recover from a full datastore — without so much as a guest OS reboot.  It’s a testament to the rock-solid architecture behind VMware ESX and VMFS.  No other virtualization platform comes close.  Try for yourself.  See what happens if a group of thin-provisioned Hyper-V virtual machines suddenly run out of storage — it’s not going to be pretty.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. PowerShell Prevents Datastore Emergencies
3. vSphere Thin-Provisioned Disk Performance
4. Finding thin-provisioned virtual disks with PowerShell

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
 
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
 
if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Related posts:

1. Easy recovery from a full VMware ESX datastore
2. Register a LUN full of VMs with PowerCLI
3. Responsible Thin Provisioning in VMware vSphere
4. VMware vSphere 4 has a Snapshot Alarm

There are still lots of folks trying to figure out why their VMware vCenter virtual machines are all polluted with a new custom attribute called clusterinvariantvmmid.  I know this because of the steady flow of incoming Google traffic with this keyword.  It’s kind of odd–Google for that term and you will get about four results, the top two being TechNet Forums and VCritical.  I would have thought that by now Microsoft would have formally documented this aspect of the vCenter/SCVMM integration, but evidently they have not.

Not to worry.  If you have landed here after searching Google for custerinvariantvmmid, the VCritical article on how to clean up after SCVMM leaves is still as applicable as ever–and it’s all automated with PowerShell.  Some say ironic.  I say use the best tool for the job.

Related posts:

1. IGT Part 7: Virtual VMware ESX 4 in high demand
2. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off
3. IGT Part 3: One-VM-per-LUN doubters
4. IGT Part 4: Failover frustration

If that looks interesting, here is what you do:

• Download Console2 (current version is 2.00 Beta, build 142); extract on your system somewhere
• Grab this file with icons and a replacement configuration file (console.xml); extract into the same location
• Make a shortcut to console.exe
  • Multiple tabs can be opened during launch by adding -t to your shortcut target (e.g., -t Console2 -t “VI Toolkit”)
• Optional: fix your prompts

My configuration file has a few changes to the mouse behavior:

• Left – select + copy automatically
• Right – paste
• Middle – pop-up menu

If you prefer to not use my entire configuration file, just copy the applicable portions from the <tabs> element.

I hope you  have enjoyed this slight diversion from the regular VCritical programming lineup.  Thanks for reading.

Related posts:

1. VI Toolkit (for Windows) 1.5 and the PowerShell prompt
2. PowerShell Prevents Datastore Emergencies
3. Automating VI Client login
4. VI Toolkit (for Windows) 1.5 released

I tend to use a two-row prompt in all of my shells (cmd.exe, Bash, PowerShell) because long commands get confusing when they wrap to another line. Some people think this is a terrible idea — if you are one of them, feel free to stop reading right now.

Here is the default VITK 1.5 console, along with a typical command:

You can see the customization added by VITK 1.5. Note that VITK 1.0 did not specify a prompt at all — it simply used whatever PowerShell was configured to use. The PowerShell prompt can be configured in a variety of ways, one of which is to create the file: …\My Documents\WindowsPowerShell\profile.ps1 containing:

function global:prompt {(get-location).path + "`n>;"}

That stopped working because VITK 1.5 now specifically defines a prompt function in C:\Program Files\VMware\Infrastructure\VIToolkitForWindows\Scripts\Initialize-VIToolkitEnvironment.ps1

To get a two-row prompt, simply open that file and add `n (that symbol is a backtick — above the Tab key on most keyboards) right before the > on line 81 (the last line shown in the snippet below):

# change prompt text
Write-Host "[" -NoNewLine
Write-Host "VI Toolkit" -NoNewLine -foregroundcolor Green
Write-Host "] " -NoNewLine
Write-Host ((Get-location).Path + "`n>") -NoNewLine

You will end up with this:

I don’t think this change causes any harm to the VITK installation — please correct me if I’m wrong.

In related news…
A quick way to set up the standard Windows Command Prompt with two rows is to just load this .reg file:

Windows Registry Editor Version 5.00
 
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment]
"PROMPT"="$P\\$_$G"

Related posts:

1. Better console for PowerShell and VITK
2. Finding thin-provisioned virtual disks with PowerShell
3. VI Toolkit (for Windows) 1.5 released
4. Clean up vCenter with PowerShell after SCVMM leaves

Let’s use PowerShell to find out if Microsoft made their release date:

$RtmDate = Get-Date "October 21, 2008"
$MgtPackReleaseDate = Get-Date "January 28, 2009"
$MgtPackDueDate = $RtmDate.AddDays(60)
 
$DaysTaken = $MgtPackReleaseDate.Subtract($RtmDate).Days
$DaysLate = $MgtPackReleaseDate.Subtract($MgtPackDueDate).Days
 
Write-Host "Due date:`t" $MgtPackDueDate.ToShortDateString()
Write-Host "Release date:`t" $MgtPackReleaseDate.ToShortDateString()
Write-Host "Days taken:`t" $DaysTaken
Write-Host "Days late:`t" $DaysLate

Results:

Due date:        12/20/2008
Release date:    1/28/2009
Days taken:      99
Days late:       39

Related posts:

1. Wild SCVMM 2008 R2 rumors and speculation
2. Hello, SCVMM 2008 R2
3. Misleading documentation or none at all? I’ll take the latter, thanks.
4. Is SCVMM 2008 R2 really banned from VMworld?

After launching the new version I noticed a couple of really nice additions right away.  Two new helpful commands:

Get-VIToolkitDocumentation will pop open the local help file

Get-VIToolkitCommunity launches your web browser to the official Community site

Carter Shanklin also posted a series of videos on Vimeo.  I was glad to hear about the fixes to Move-VM:

Related posts:

1. Better console for PowerShell and VITK
2. VMware ESXi 4: SD Flash in BL460c G6
3. Clean up vCenter with PowerShell after SCVMM leaves
4. Upgrading VMs to vSphere virtual hardware

I was poking around with the System Center Virtual Machine Manager 2008 (SCVMM) PowerShell functionality and came across an interesting cmdlet that can be used to reveal which guest operating systems are customizable:

PS C:\Windows\System32> Get-OperatingSystem |
>> where { $_.IsCustomizationAllowed -eq $true } | sort | ft Name

Which yields the following list:

Name
----
64-bit edition of Windows Server 2008 Datacenter
64-bit edition of Windows Server 2008 Enterprise
64-bit edition of Windows Server 2008 Standard
64-bit edition of Windows Vista
Windows 2000 Advanced Server
Windows 2000 Server
Windows Server 2003 Datacenter Edition (32-bit x86)
Windows Server 2003 Datacenter x64 Edition
Windows Server 2003 Enterprise Edition (32-bit x86)
Windows Server 2003 Enterprise x64 Edition
Windows Server 2003 Standard Edition (32-bit x86)
Windows Server 2003 Standard x64 Edition
Windows Server 2003 Web Edition
Windows Server 2008 Datacenter 32-Bit
Windows Server 2008 Enterprise 32-Bit
Windows Server 2008 Standard 32-Bit
Windows Small Business Server 2003
Windows Vista
Windows Web Server 2008
Windows XP 64-Bit Edition
Windows XP Professional

Interesting.  No Linux?  I thought SCVMM and Hyper-V were embracing at least one Linux distribution.  Let’s try the converse of that command:

PS C:\Windows\System32> Get-OperatingSystem |
>> where { $_.IsCustomizationAllowed -eq $false } | sort | ft Name

Name
----
None
Novell NetWare 5.1
Novell NetWare 6.x
Open Enterprise Server
Other (32 bit)
Other (64 bit)
Other Linux (32 bit)
Other Linux (64 bit)
Red Hat Enterprise Linux 2
Red Hat Enterprise Linux 3
Red Hat Enterprise Linux 3 (64 bit)
Red Hat Enterprise Linux 4
Red Hat Enterprise Linux 4 (64 bit)
Red Hat Enterprise Linux 5
Red Hat Enterprise Linux 5 (64 bit)
Sun Solaris 10 (32 bit)
Sun Solaris 10 (64 bit)
Suse Linux Enterprise Server 10 (32 bit)
Suse Linux Enterprise Server 10 (64 bit)
Suse Linux Enterprise Server 9 (32 bit)
Suse Linux Enterprise Server 9 (64 bit)
Ubuntu Linux (32 bit)
Ubuntu Linux (64 bit)
Windows NT Server 4.0

Whoa! Why is SLES 10 is on that list?  That is a supported guest OS.

Sometimes supported doesn’t mean fully functional. For a virtualization platform that treats Linux workloads like first-class citizens, choose VMware ESX with vCenter Server:

Related posts:

1. Hyper-V Linux Smokescreen
2. What is Red Hat Enterprise Virtualization?
3. RHEV Manager — It’s not just a clever name
4. Dear Red Hat…

The VI Client interface will not let administrators create new virtual machines with thin-provisioned disks.  Even so, it is entirely possible that your environment contains such disks.  Let’s take a look at a couple of ways to create these disks and then use a VI Toolkit (for Windows) PowerShell script to see if any exist.

Manual VMDK Creation

One way that thin disks may be introduced into an environment is by manually creating a VMDK file and then attaching that file to a virtual machine.  This can be accomplished by the vmkfstools command with the -d thin option.  This procedure is not for beginners, so if you have used it you presumably evaluated the risks of overcommitting your storage.  But what if one of your colleagues did this without telling you?

Compact Clones

Another source of thin disks requires a little background first:

There is a prominent option in the VI Client interface to convert a virtual machine to template (and vice versa).  But if you right-click a VM in the inventory, you are presented with a variation of that feature:  Clone to Template.

The wizard that launches in response to that option allows an administrator to optionally choose the Compact disk format — this is a thin-provisioned disk.  If your template has a 50GB virtual disk but only 8GB are actually used, this feature can provide significant storage savings.

So far so good.  Now, what do you suppose happens if you decide to convert that template to a VM?  Nothing happens — it just works.  You are free to use that VM just like any other in your inventory.  But this VM keeps its thin-provisioned disk, a fact that is not discernible through the user interface.

Find the Thin-Provisioned Disks

I wrote a small VI Toolkit (for Windows) PowerShell script that identifies latent thin disks in both VMs and templates.  I hope you find it useful:

# FindThinDisks.ps1
#
# Identifies VMs and templates that are using thin-provisioned
# virtual disks.
 
# Version 1.0  January 14, 2009
# Eric Gray
 
$vmtp = Get-VM
$vmtp += Get-Template
 
foreach($vm in $vmtp | Get-View){
  foreach($dev in $vm.Config.Hardware.Device){
    if(($dev.GetType()).Name -eq "VirtualDisk"){
      if($dev.Backing.ThinProvisioned -eq $true) {
        $vm.Name + "`t" + $dev.Backing.FileName
      }
    }
  }
}

Thin-provisioning is a great tool to have in your toolbox, when used properly.

Script inspired by this thread.

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Responsible Thin Provisioning in VMware vSphere
3. Easy recovery from a full VMware ESX datastore
4. PowerShell Prevents Datastore Emergencies

There are three different affected areas and you can fix each one manually or all in one shot with a VI Toolkit (for Windows) PowerShell script that I wrote for just this purpose.

1.  SCVMM Port Groups

First, let’s address the new port groups that SCVMM created when managing your VMware ESX VMs.  I covered this in more detail in a previous article.  Essentially, the SCVMM interface presents vSwitches, not port groups.  It then creates one or more new port groups for you without warning — one per vSwitch, per VLAN, on each ESX host.

To recover from this mess, you will need to reconfigure any VMs that are currently using these port groups and then delete the port groups from the vSwitches.  If you have more than a couple VMs this task will be easier with the script.

2.  Custom Attribute: ClusterInvariantVMMId

I bet you did not know that SCVMM would add a new strangely-named custom attribute to vCenter and tag every single VM in your inventory with a GUID.  I’ve heard puzzled VMware customers asking about this.  Initially, the explanation for this phenomenon was not clear — until I evaluated the SCVMM product and VMware vCenter integration.  If you are done with SCVMM, it is safe to remove this attribute with VI Client (on the Administration menu).

3.  Self-Service Role

Evidently the Self-Service Portal feature of SCVMM will allow users to provision and manage VMs on ESX hosts in addition to Hyper-V.  I tried this and it did not work, but I probably wasn’t persistent enough.  When you are done using SCVMM you won’t be needing that role anymore — go ahead and delete it.

PowerShell Cleanup Script

If you would like to clean up all three of these issues in one step, use the following VI Toolkit (for Windows):

# SCVMM-Cleanup.ps1
#
# When SCVMM connects to vCenter, it makes several changes and doesn't clean up
# after itself.  This script returns everything back to normal.
 
# Version 1.0  January 5, 2009
# Eric Gray
 
# modify this line accordingly
Get-VC vchostname -user xxx -Password yyy
 
# VMs using SCVMM-created port groups will be reconfigured to use this instead
$portGroup = "VM Network"
 
# Move all NICs from the SCVMM.vSwitchX.Y port groups to $portGroup
get-vm | Get-NetworkAdapter | where {$_.NetworkName -like "SCVMM*"} `
    | Set-NetworkAdapter -NetworkName $portGroup -Confirm:$false
 
# Delete those SCVMM port groups after they are no longer in use
foreach ($esx in Get-VMHost) {
    Get-VirtualPortGroup -VMHost $esx.Name | where { $_.Name -like "SCVMM*" } `
    | Remove-VirtualPortGroup  -Confirm:$false
}
 
# Clean up the custom attribute
Get-Datacenter | Remove-CustomField -Name ClusterInvariantVMMId -Confirm:$false
 
# Remove the SCVMM role
$authMgr = Get-View AuthorizationManager
foreach($role in $authMgr.RoleList | where {$_.Name -eq "SCVMMSelfServiceUser"}){
    $authMgr.RemoveAuthorizationRole($role.roleid, 1)
}

If you or one of your coworkers accidentally polluted your vCenter environment, this should help you get everything back in order.

Thanks go to Carter Shanklin for reviewing the script.

Related posts:

1. IGT Part 6: Cluster invari-what ID?
2. Managing VI3 with SCVMM considered harmful
3. Thanks for all the port groups!
4. VMware vCenter Server 4 task and event retention