In this latest post by TechHead guest contributor James Pearce he covers a topic near and dear to many of us – how to get VMware ESX/ESXi and its VMs to shut down gracefully upon power failure to the host. Tighter integration between a UPS and a VMware ESX/ESXi host is no doubt something that will become more mature over time though for now it can be an issue for many administrators especially those running the free version of ESXi. So read on to find out how James overcame this issue in his virtualization lab.
A Nearly Free UPS
I recently acquired an APC Smart UPS that was being chucked out from work (having never worked) for my home lab along with an ancient AP9606 management card. With the batteries changed the UPS burst into life – but after some messing about getting the right firmware on it, I was disappointed to find no easy way to get it to shutdown my ESXi box when it needed to.
VMware License Restriction
The VMware management appliance (vima) can shutdown only paid-for installations of ESXi (using apcupsd and VMware community member lamw’s scripts) – the necessary interfaces on the free version have been made read-only since ESXi v3.5 U3.
Burp Suite
I’ve been finding a lot of use recently for network sniffers, so thought I’d have a look at how the VMware vSphere Client works, as obviously that can shut down the host. As luck would have it, the client is nothing more than a glorified web browser with the slight complication that it’s talking over SSL – but that’s no problem for PortSwigger’s Burp suite in its transparent proxy mode.
The traffic captures revealed that only three frames would be needed to perform the shutdown (hello, authenticate, and shutdown). A little manipulation is needed to get the session keys in, but that is basically it. ESXi’s startup and shutdown policy will do the work suspending or shutting down individual VMs, as configured through the vSphere Client.
The Script – shutdown.bat
Using this newly found knowledge I’ve created a Windows batch file (with a few supporting text files which are basically HTTP requests) that takes the hostname, username and password as parameters and will then shut down the host cleanly. The script needs something to launch it –APC PowerChute Network Shutdown in my case – and a utility to send the commands over SSL, for which I’ve used Nmap ncat (which just needs to be installed).
I have put all the necessary script files into a single convenient zipped file which you can download from here – the scripts are fairly well commented so you should be able to follow what is happening.
APC PowerChute
A potential issue is that APC’s PowerChute Network Shutdown utility will always shut down the Windows machine it’s running on. I’ve therefore used a separate Windows management VM to host PowerChute and my script, since I wanted everything else just suspended.
PowerChute has an option to ‘run this command’ but it’s limited to 8.3 paths and won’t accept command line parameters. A separate batch file is needed (poweroff.bat) that runs the shutdown script with the parameters – but that could shut down other ESXi boxes as well if required. Also the PowerChute service needs to be run as local Administrator as the default Local System account doesn’t have sufficient rights.
Testing the Scripts
Download the ZIP and extract the files – I’ve assumed the package will be extracted to c:scriptsesxi; update the path in poweroff.bat otherwise. Also the hostname, username and password also need to be specified in poweroff.bat.
Next install and configure PowerChute (in particular change the service user account) and enter the script in the ‘run this command’ box – I also increased the time allowed, but in practice it runs in a few seconds.
Some waiting around can be avoided when testing by setting the UPS low-battery duration as high as it will go – just remember to change it back.
Next open up vSphere Client from a real machine, pull the UPS plug and once the battery get’s down to the specified number of minutes remaining, the script should run and the tasks will appear in vSphere Client. Shortly afterwards the VM used to launch the script will itself shutdown under the control of PowerChute!
In Summary
The complete set of files can be downloaded here, and nMap ncat installation for Windows from here. Then a UPS management application is needed, for APC Smart UPSs use PowerChute for Windows.
The shutdown script includes logging and should report most errors. Bear in mind though that once a host is shutdown, it probably won’t be restarted when utility power is restored.
Burp Suite is a handy utility to bypass device limitations by enabling the scripting of management tasks that are only usually available through a web interface. I’ve used it to build scripts to regularly reboot home-spec routers every couple of weeks to keep them stable, and to set the time on the APC AP9606 management card daily since it doesn’t support NTP – and here to build a UPS shutdown script for ESXi; functionality that should really be built into ESXi in the first place.
