When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
 
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
 
if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Related posts:

1. Easy recovery from a full VMware ESX datastore
2. Responsible Thin Provisioning in VMware vSphere
3. VI Toolkit (for Windows) 1.5 and the PowerShell prompt
4. VMware vSphere 4 has a Snapshot Alarm

If that looks interesting, here is what you do:

• Download Console2 (current version is 2.00 Beta, build 142); extract on your system somewhere
• Grab this file with icons and a replacement configuration file (console.xml); extract into the same location
• Make a shortcut to console.exe
  • Multiple tabs can be opened during launch by adding -t to your shortcut target (e.g., -t Console2 -t “VI Toolkit”)
• Optional: fix your prompts

My configuration file has a few changes to the mouse behavior:

• Left – select + copy automatically
• Right – paste
• Middle – pop-up menu

If you prefer to not use my entire configuration file, just copy the applicable portions from the <tabs> element.

I hope you  have enjoyed this slight diversion from the regular VCritical programming lineup.  Thanks for reading.

Related posts:

1. VI Toolkit (for Windows) 1.5 and the PowerShell prompt
2. Finding thin-provisioned virtual disks with PowerShell
3. PowerShell Prevents Datastore Emergencies
4. Clean up vCenter with PowerShell after SCVMM leaves

I tend to use a two-row prompt in all of my shells (cmd.exe, Bash, PowerShell) because long commands get confusing when they wrap to another line. Some people think this is a terrible idea — if you are one of them, feel free to stop reading right now.

Here is the default VITK 1.5 console, along with a typical command:

You can see the customization added by VITK 1.5. Note that VITK 1.0 did not specify a prompt at all — it simply used whatever PowerShell was configured to use. The PowerShell prompt can be configured in a variety of ways, one of which is to create the file: …\My Documents\WindowsPowerShell\profile.ps1 containing:

function global:prompt {(get-location).path + "`n>;"}

That stopped working because VITK 1.5 now specifically defines a prompt function in C:\Program Files\VMware\Infrastructure\VIToolkitForWindows\Scripts\Initialize-VIToolkitEnvironment.ps1

To get a two-row prompt, simply open that file and add `n (that symbol is a backtick — above the Tab key on most keyboards) right before the > on line 81 (the last line shown in the snippet below):

# change prompt text
Write-Host "[" -NoNewLine
Write-Host "VI Toolkit" -NoNewLine -foregroundcolor Green
Write-Host "] " -NoNewLine
Write-Host ((Get-location).Path + "`n>") -NoNewLine

You will end up with this:

I don’t think this change causes any harm to the VITK installation — please correct me if I’m wrong.

In related news…
A quick way to set up the standard Windows Command Prompt with two rows is to just load this .reg file:

Windows Registry Editor Version 5.00
 
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment]
"PROMPT"="$P\\$_$G"

Related posts:

1. Better console for PowerShell and VITK
2. Finding thin-provisioned virtual disks with PowerShell
3. VI Toolkit (for Windows) 1.5 released
4. Clean up vCenter with PowerShell after SCVMM leaves

After launching the new version I noticed a couple of really nice additions right away.  Two new helpful commands:

Get-VIToolkitDocumentation will pop open the local help file

Get-VIToolkitCommunity launches your web browser to the official Community site

Carter Shanklin also posted a series of videos on Vimeo.  I was glad to hear about the fixes to Move-VM:

Related posts:

1. Upgrading VMs to vSphere virtual hardware
2. VMware ESXi 4: SD Flash in BL460c G6
3. Better console for PowerShell and VITK
4. Clean up vCenter with PowerShell after SCVMM leaves

The VI Client interface will not let administrators create new virtual machines with thin-provisioned disks.  Even so, it is entirely possible that your environment contains such disks.  Let’s take a look at a couple of ways to create these disks and then use a VI Toolkit (for Windows) PowerShell script to see if any exist.

Manual VMDK Creation

One way that thin disks may be introduced into an environment is by manually creating a VMDK file and then attaching that file to a virtual machine.  This can be accomplished by the vmkfstools command with the -d thin option.  This procedure is not for beginners, so if you have used it you presumably evaluated the risks of overcommitting your storage.  But what if one of your colleagues did this without telling you?

Compact Clones

Another source of thin disks requires a little background first:

There is a prominent option in the VI Client interface to convert a virtual machine to template (and vice versa).  But if you right-click a VM in the inventory, you are presented with a variation of that feature:  Clone to Template.

The wizard that launches in response to that option allows an administrator to optionally choose the Compact disk format — this is a thin-provisioned disk.  If your template has a 50GB virtual disk but only 8GB are actually used, this feature can provide significant storage savings.

So far so good.  Now, what do you suppose happens if you decide to convert that template to a VM?  Nothing happens — it just works.  You are free to use that VM just like any other in your inventory.  But this VM keeps its thin-provisioned disk, a fact that is not discernible through the user interface.

Find the Thin-Provisioned Disks

I wrote a small VI Toolkit (for Windows) PowerShell script that identifies latent thin disks in both VMs and templates.  I hope you find it useful:

# FindThinDisks.ps1
#
# Identifies VMs and templates that are using thin-provisioned
# virtual disks.
 
# Version 1.0  January 14, 2009
# Eric Gray
 
$vmtp = Get-VM
$vmtp += Get-Template
 
foreach($vm in $vmtp | Get-View){
  foreach($dev in $vm.Config.Hardware.Device){
    if(($dev.GetType()).Name -eq "VirtualDisk"){
      if($dev.Backing.ThinProvisioned -eq $true) {
        $vm.Name + "`t" + $dev.Backing.FileName
      }
    }
  }
}

Thin-provisioning is a great tool to have in your toolbox, when used properly.

Script inspired by this thread.

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Responsible Thin Provisioning in VMware vSphere
3. PowerShell Prevents Datastore Emergencies
4. Easy recovery from a full VMware ESX datastore

There are three different affected areas and you can fix each one manually or all in one shot with a VI Toolkit (for Windows) PowerShell script that I wrote for just this purpose.

1.  SCVMM Port Groups

First, let’s address the new port groups that SCVMM created when managing your VMware ESX VMs.  I covered this in more detail in a previous article.  Essentially, the SCVMM interface presents vSwitches, not port groups.  It then creates one or more new port groups for you without warning — one per vSwitch, per VLAN, on each ESX host.

To recover from this mess, you will need to reconfigure any VMs that are currently using these port groups and then delete the port groups from the vSwitches.  If you have more than a couple VMs this task will be easier with the script.

2.  Custom Attribute: ClusterInvariantVMMId

I bet you did not know that SCVMM would add a new strangely-named custom attribute to vCenter and tag every single VM in your inventory with a GUID.  I’ve heard puzzled VMware customers asking about this.  Initially, the explanation for this phenomenon was not clear — until I evaluated the SCVMM product and VMware vCenter integration.  If you are done with SCVMM, it is safe to remove this attribute with VI Client (on the Administration menu).

3.  Self-Service Role

Evidently the Self-Service Portal feature of SCVMM will allow users to provision and manage VMs on ESX hosts in addition to Hyper-V.  I tried this and it did not work, but I probably wasn’t persistent enough.  When you are done using SCVMM you won’t be needing that role anymore — go ahead and delete it.

PowerShell Cleanup Script

If you would like to clean up all three of these issues in one step, use the following VI Toolkit (for Windows):

# SCVMM-Cleanup.ps1
#
# When SCVMM connects to vCenter, it makes several changes and doesn't clean up
# after itself.  This script returns everything back to normal.
 
# Version 1.0  January 5, 2009
# Eric Gray
 
# modify this line accordingly
Get-VC vchostname -user xxx -Password yyy
 
# VMs using SCVMM-created port groups will be reconfigured to use this instead
$portGroup = "VM Network"
 
# Move all NICs from the SCVMM.vSwitchX.Y port groups to $portGroup
get-vm | Get-NetworkAdapter | where {$_.NetworkName -like "SCVMM*"} `
    | Set-NetworkAdapter -NetworkName $portGroup -Confirm:$false
 
# Delete those SCVMM port groups after they are no longer in use
foreach ($esx in Get-VMHost) {
    Get-VirtualPortGroup -VMHost $esx.Name | where { $_.Name -like "SCVMM*" } `
    | Remove-VirtualPortGroup  -Confirm:$false
}
 
# Clean up the custom attribute
Get-Datacenter | Remove-CustomField -Name ClusterInvariantVMMId -Confirm:$false
 
# Remove the SCVMM role
$authMgr = Get-View AuthorizationManager
foreach($role in $authMgr.RoleList | where {$_.Name -eq "SCVMMSelfServiceUser"}){
    $authMgr.RemoveAuthorizationRole($role.roleid, 1)
}

If you or one of your coworkers accidentally polluted your vCenter environment, this should help you get everything back in order.

Thanks go to Carter Shanklin for reviewing the script.

Related posts:

1. IGT Part 6: Cluster invari-what ID?
2. Managing VI3 with SCVMM considered harmful
3. Thanks for all the port groups!
4. VMware vCenter Server 4 task and event retention