PowerShell Prevents Datastore Emergencies

In my previous post on VMware vSphere thin provisioning, I pointed out the new datastore alarm feature.  You can take advantage of this feature to respond to a sudden storage demand and automatically take action before end users notice.

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

Datastore Alarm

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Datastore alarm running a PowerShell script

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Storage VMotion in progress

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

Datastore emergency events

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Tags: , , , , , , , ,

17 comments

  1. Alan Renouf’s avatar

    That is amazing, very cool. Great use of PowerCLI and Alarms!

  2. Eric Gray’s avatar

    Thanks, Alan! Coming from you that means a lot.

    Eric

  3. Scott Sauer’s avatar

    Eric,
    Great write up, I was just discussing my fears of going thin with a fellow co-worker. This addresses our fears of a “storage emergency” situation. Going to give it a shot once I get through all of our upgrades.
    Thanks!
    Scott

    1. Eric Gray’s avatar

      Scott,

      That’s good news. If you devise a more robust VM selection/migration script, please consider sharing.

      Eric

    2. Scott Sauer’s avatar

      Eric,
      Here is an article I wrote that I would like to share with anyone that comes across the thread.
      http://www.virtualinsanity.com/index.php/2009/10/12/get-thin-provisioning-working-for-you-in-vsphere/

      Thanks!
      Scott

    3. Adam’s avatar

      I’ve gone through the process of trying to figure out a better VM selection process.. I pulled the gut of the VM Disk usage from another source but the rest of tathered together by me. The script will search suspect datastore for the smallest VM based on disk utilization and then check if there are any snapshots.. if there are snapshots for that particular VM they get removed the the VM gets moved to an evacuation datastore of your choosing.

      Add-PSSnapin VMware.Vimautomation.Core
      Connect-VIServer localhost

      $vmToMove = Get-View -ViewType VirtualMachine | %{
      New-Object PSObject -Property @{
      Name = $_.Name
      Host = (Get-View $_.Summary.Runtime.Host).Name
      Datastore = [system.String]::Join(“,”,($_.Storage.PerDatastoreUsage | %{Get-View $_.Datastore} | %{$_.Name}))
      Size = ($_.Storage.PerDatastoreUsage | Measure-Object -Property Committed -Sum).Sum
      }
      } | Sort Size | ?{$_.DataStore -eq $ENV:VMWARE_ALARM_TARGET_NAME} | Select Name, Datastore -first 1

      $destDS = Get-Datastore yourevacuationdatastore
      $snapshot = get-vm $vmtomove.name | get-snapshot
      If ($snapshot -eq $NULL)
      {
      write-host “No Snapshots Present”}
      else { write-host “Snapshots Found and Removing”
      get-vm $vmtomove.name | get-snapshot | remove-snapshot -Confirm:$false }

      if ($destDS) {
      move-vm -VM $vmToMove.Name -Datastore $destDS -RunAsync
      }
      Send-MailMessage -To “whereveryouwanttogo” -From “somewhereitcamefrom” -Subject “VM Successfully Moved to Evacuation Datastore” -Body “$vmToMove has been moved to $destDS from $env:VMWARE_ALARM_TARGET_NAME” -SMTPServer “yoursmptserver”

      This is working for me right now in POC.

      Thank you,

      Adam Savage

      1. Eric Gray’s avatar

        Awesome, Adam! Thanks for sharing the enhancement.

        1. Adam’s avatar

          No problem at all.. I also had some difficulties with getting the alarm to launch the script correctly.. Turned out that virtualcenter was launching powershell from C:\Windows\SysWOW64\WindowsPowershell\v1.0\powershell.exe instead of the normal c:\windows\system32.. So i had the execution policy set on the x86 version but not the x64 version.. food for thought.

          Thanks!

          Adam Savage

        2. ap’s avatar

          Is there any to create an alarm for datacenter using VI API ?

          If yes, please help.

        3. boraq’s avatar

          Hi

          Could you please tell me what’s this ‘$env:VMWARE_ALARM_TARGET_NAME’??

Comments are now closed.