Needless to say that a Server suddenly being turned off or losing power is a roll of the dice. Especially sensitive to abrupt shut downs are SQL, Exchange and Hyper-V Servers. For this reason, it’s extremely important to have an unattended graceful shutdown software for power outage events.
What is Needed:
- APC Smart UPS
- APC Network Monitoring Card (APC-NMC)
- Windows Server
- APC PCNS Software
Note: There are several types of UPS’, cables and software you can use to accomplish a graceful shutdown however in this post I will cover only the APC Smart UPS with NMC and PCNS because it’s a system I have been using successfully to gracefully shut down multiple servers from a single UPS, under specific conditions.
To begin, install your NMC on the SMART UPS and configure an IP address. Access the NMC using a Web browser (or Telnet if you miss the good ‘ol days).
Log in to the user interface and set up E-Mail or SNMP alerts. It’s generally a good idea to be alerted when the UPS is going to shut down the servers. This can be done from the administration –> notification menu.
Set up the type of alerts you want from the event actions, then set up either SMTP or SNMP for alert notifications.
To begin configuring shutdown, check how much runtime you have by clicking on the UPS –> Overview menu. Runtime is how much time your UPS will be able to power your systems during a power outage.
As you can see, the image above indicates that we have approximately 1 hour and 8 minutes after the power goes out, before our UPS batteries are completely drained. In contrast, the image below shows a different UPS with only 26 minutes of runtime. Our shutdown policies should be based on how much runtime we have, which is different for every situation. For this reason, we cannot simply use ‘default’ settings, we must study and understand how shutdown works and what variables are required, based on several important factors.
Go to the UPS Tab, this is where the important settings are entered. These settings are not as self explanatory as they seem and they are very, very important. For this reason, I urge you to read about, and fully understand these settings before proceeding. You can get detailed information about these parameters in this blog by Steve Jenkins.
UPS –> Control
The UPS control is used for a user initiated shutdown sequence. This is useful for testing or for manually initiating a shutdown sequence where the UPS signals the servers to shut themselves down gracefully.
UPS –> Configuration –> Shutdown
1. Low Battery Duration: The point at which the UPS sends a signal to gracefully shut down all the servers. You should set this threshold to give your servers plenty of time to gracefully shut down.
2. Shutdown Delay: This is how long the UPS will stay on after all the servers have successfully shut down.
3. Maximum Required Delay: This value is calculated by the NMC after it queries the PCNS clients. After you add/remove PCNS clients, this value will change. It’s based on how much time it thinks it will take to gracefully shut down all your operating systems.
Note: Basic Signaling Shutdown is for serial cable communication, leave it unchecked. This does not apply to what we are doing here!
4. Duration of Shutdown Sleep Time: How long the UPS will stay off when you initiate a manual shutdown sequence manually from the control menu.
5. Minimum Battery Capacity: This is the minimum charge level the NMC will require before it turns the power back on. The capacity should be enough to sustain another shutdown sequence if the power cuts off again.
6. Return Delay: How long the UPS will stay off (and recharge) after power has been restored. This is useful because as we all know, power disruptions can be intermittent and it’s best to wait for power to be steadily restores before restating your servers.
PowerChute Clients Setup
PowerChute Clients – add the IP addresses of the servers you are going to gracefully shut down.
Now that we have configure the network monitoring card, it’s time to configure the servers.
Download and install PCNS on the physical servers you plan to gracefully shut down in the event of a power outage. Do not install PCNS on virtual servers.
Once the software is installed, it will open a browser and begin the configuration utility.
Select your preferred networking protocol, SCVMM support and UPS configuration. You will need the NMC user name, password and authentication phrase. Unless you have changed them, the defaults are:
User name: apc
Password: apc
Auth Phrase: admin user phrase
Add the IP address of your Network Monitoring Card (NMC) and click next…
The wizard will confirm the settings, then click on the apply button.
If you are using Hyper-V, set the duration for Hyper-V shutdown. This should be how long it takes to gracefully shut down the virtual machines. It should provide ample time for VM shutdown before the physical machine is shut down. If the example below, we set that threshold for 10 minutes (600 seconds) to give an old server plenty of time to shut down its VMs. If our low battery duration is 10 minutes, that leaves us no time to gracefully shut down our physical servers! So make sure you set this value high enough to allow graceful shutdown of your VM’s while still allowing plenty of time for your physical servers to shut down before your UPS shuts off! If you set your VMs shutdown duration for 10 minutes, then your low battery duration should be set to at least 12 minutes.
After the VM shutdown duration time has been exceeded, the PowerChute software will start to shut down the physical machine.
You do not need to turn off the UPS, this action will be performed by the network monitoring card based on its settings.
Once the wizard connects successfully, click finish.
Please note, it’s useful to test your configuration in a lab environment before configuring these solutions in a production environment. A lab test will give you ideas of how to tweak the settings to best fit your needs.