Comments on: Easy recovery from a full VMware ESX datastore

By: Amal

Amal — Sun, 23 Jan 2011 05:21:22 +0000

I’ve not tried this in 4.1 but does ESX perform the same pause function if the datastore goes offline completely? If not, it should. A couple years ago I had a catastrophic loss of service with one of my iSCSI storage arrays. Recovery was simple, but the VM guests had all hard-crashed. It would be nice if those machines could have been put into instantaneous suspension (halt and suspend all CPU and storage calls) until the storage array was able to be brought back online.

By: Objective 6.4 : Troubleshoot Storage Performance and Connectivity

Objective 6.4 : Troubleshoot Storage Performance and Connectivity — Thu, 19 Aug 2010 10:17:34 +0000

[...] http://www.vcritical.com/2009/10/easy-recovery-from-a-full-vmware-esx-datastore/ [...]

By: Eric Gray

Eric Gray — Fri, 09 Oct 2009 14:22:20 +0000

That is good to know, thanks for the details. So all of the VMs on a LUN get paused at the same time whether or not they need additional storage — then it is a race against the 200MB clock. For the VMs that don’t make it… unpredictable results.

Monitoring virtualization storage is crucial, which is why vSphere accounts not only for space currently used but for space allocated by thin provisioning — critical for managing risk. As far as I can tell, SCOM doesn’t address this aspect.

By: Stu Fox

Stu Fox — Fri, 09 Oct 2009 00:31:12 +0000

You’ll get warned in the event log at 2GB free, and at 200MB free the VM’s will get paused on Hyper-V. That would be the worst case – that you ignored the warnings and didn’t take action. Of course with OpsMgr you would get warned earlier.

By: Eric Gray

Eric Gray — Thu, 08 Oct 2009 20:02:59 +0000

Stu,

That would not be the worst case — more like best case. If it ever happens to you, let me know.

Eric

By: Stu Fox

Stu Fox — Thu, 08 Oct 2009 00:33:01 +0000

By not pretty you mean worst case when the machines get paused just like on VMware?

Of course you’d be monitoring the host volume with OpsMgr so it would proactively alert you anyway and you’d have planned for that. Hell, even perfmon would probably do that for you.

By: Eric Gray

Eric Gray — Wed, 07 Oct 2009 17:24:02 +0000

Jason, thanks for pointing out those additional recovery choices.

Ben, interesting idea. If you come up with something let me know. Nice blog, by the way.

By: Ben Thomas

Ben Thomas — Wed, 07 Oct 2009 00:44:33 +0000

Cool post! I would be interested to try that with some preventative alarms to automatically move off powered off VMs with powerCLI. Seems to make a case for an “emergency” empty LUN for situations like this.

By: Jason Boche

Jason Boche — Tue, 06 Oct 2009 23:45:31 +0000

Another option for the back pocket: Find a VM that can be powered off. Chances are (and by default) it has no memory reservation configured. The net result is that when the VM is powered off, a VMkernel swap file equal to the size of assigned RAM can safely be removed when the VM is powered down.

Alternatively, creating a reservation equal to assigned memory on the fly will zero out the swap file but the zeroing won’t actually happen until the next power operation of the VM.

By: Tweets that mention How to recover when a VMware ESX datastore runs out of space | VCritical -- Topsy.com

Tue, 06 Oct 2009 22:26:50 +0000

[...] This post was mentioned on Twitter by VMware Planet V12n. VMware Planet V12n said: Easy recovery from a full VMware ESX datastore (VCritical) http://bit.ly/DXVGX [...]