Needless to say that a Server suddenly being turned off or losing power is a roll of the dice. Especially sensitive to abrupt shut downs are SQL, Exchange and Hyper-V Servers. For this reason, it’s extremely important to have an unattended graceful shutdown software for power outage events.
What is Needed:
Note: There are several types of UPS’, cables and software you can use to accomplish a graceful shutdown however in this post I will cover only the APC Smart UPS with NMC and PCNS because it’s a system I have been using successfully to gracefully shut down multiple servers from a single UPS, under specific conditions.
To begin, install your NMC on the SMART UPS and configure an IP address. Access the NMC using a Web browser (or Telnet if it’s Throwback Thursday).
Log in to the user interface and set up E-Mail or SNMP alerts. It’s generally a good idea to be alerted when the UPS is going to shut down the servers. This can be done from the administration –> notification menu.
Set up the type of alerts you want from the event actions, then set up either SMTP or SNMP for alert notifications.
To begin configuring shutdown, check how much runtime you have by clicking on the UPS –> Overview menu. Runtime is how much time your UPS will be able to power your systems during a power outage.
As you can see, the image above indicates that we have approximately 1 hour and 8 minutes after the power goes out, before our UPS batteries are completely drained. In contrast, the image below shows a different UPS with only 26 minutes of runtime. Our shutdown policies should be based on how much runtime we have, which is different for every situation. For this reason, we cannot simply use ‘default’ settings, we must study and understand how shutdown works and what variables are required, based on several important factors.
Go to the UPS Tab, this is where the important settings are entered. These settings are not as self explanatory as they seem and they are very, very important. For this reason, I urge you to read about, and fully understand these settings before proceeding. You can get detailed information about these parameters in this blog by Steve Jenkins.
The UPS control is used for a user initiated shutdown sequence. This is useful for testing or for manually initiating a shutdown sequence where the UPS signals the servers to shut themselves down gracefully.
1. Low Battery Duration: The point at which the UPS sends a signal to gracefully shut down all the servers. You should set this threshold to give your servers plenty of time to gracefully shut down.
2. Shutdown Delay: This is how long the UPS will stay on after all the servers have successfully shut down.
3. Maximum Required Delay: This value is calculated by the NMC after it queries the PCNS clients. After you add/remove PCNS clients, this value will change. It’s based on how much time it thinks it will take to gracefully shut down all your operating systems.
Note: Basic Signaling Shutdown is for serial cable communication, leave it unchecked. This does not apply to what we are doing here!
4. Duration of Shutdown Sleep Time: How long the UPS will stay off when you initiate a manual shutdown sequence manually from the control menu.
5. Minimum Battery Capacity: This is the minimum charge level the NMC will require before it turns the power back on. The capacity should be enough to sustain another shutdown sequence if the power cuts off again.
6. Return Delay: How long the UPS will stay off (and recharge) after power has been restored. This is useful because as we all know, power disruptions can be intermittent and it’s best to wait for power to be steadily restores before restating your servers.
First Officer: Captain, we’ve lost main power! Captain: Blimey, how long will auxiliary power hold us in orbit? First Officer: Approximately [runtime] minutes, captain. Captain: We have plenty of time. It takes us [maximum required delay] to evacuate the ship, so there is no need to panic! Let’s wait until we reach the [low battery duration] threshold. Maybe by then we will get our main power restored. (Pressing intercom) Engineering, this is the captain! I need power… the lives… of our crew… depend on it! Engineering Officer: I’m giving it all I’ve got, captain!
First Officer (profusely sweating): Captain, we have only [low battery duration] minutes left on auxiliary and the ship’s main power is still off-line. If we don’t evacuate now, we won’t get the entire crew out on time. Should I send the evacuation signal? Captain: Make it so… and may God have mercy on our souls! Hopefully that gives you a better understanding of how the shutdown process work!
PowerChute Clients – add the IP addresses of the servers you are going to gracefully shut down.
Now that we have configure the network monitoring card, it’s time to configure the servers.
Download and install PCNS on the physical servers you plan to gracefully shut down in the event of a power outage. Do not install PCNS on virtual servers.
Once the software is installed, it will open a browser and begin the configuration utility.
Select your preferred networking protocol, SCVMM support and UPS configuration. You will need the NMC user name, password and authentication phrase. Unless you have changed them, the defaults are:
User name: apc Password: apc Auth Phrase: admin user phrase
Add the IP address of your Network Monitoring Card (NMC) and click next…
The wizard will confirm the settings, then click on the apply button.
If you are using Hyper-V, set the duration for Hyper-V shutdown. This should be how long it takes to gracefully shut down the virtual machines. It should provide ample time for VM shutdown before the physical machine is shut down. If the example below, we set that threshold for 10 minutes (600 seconds) to give an old server plenty of time to shut down its VMs. If our low battery duration is 10 minutes, that leaves us no time to gracefully shut down our physical servers! So make sure you set this value high enough to allow graceful shutdown of your VM’s while still allowing plenty of time for your physical servers to shut down before your UPS shuts off! If you set your VMs shutdown duration for 10 minutes, then your low battery duration should be set to at least 12 minutes.
After the VM shutdown duration time has been exceeded, the PowerChute software will start to shut down the physical machine.
You do not need to turn off the UPS, this action will be performed by the network monitoring card based on its settings.
Once the wizard connects successfully, click finish.
Please note, it’s useful to test your configuration in a lab environment before configuring these solutions in a production environment. A lab test will give you ideas of how to tweak the settings to best fit your needs.
What is SNMP: Simple Network Management Protocol (SNMP) is a protocol that is used to configure and collect information about network devices such as servers, printers, routers and switches. In this article, we will use SNMP to collect important health data from Dell servers, Microsoft operating systems and Sonicwall routers. RAID degradation, Windows performance problems, low disk space, replication errors, account privilege changes and software installation notifications are just some of the events we are going to monitor using the techniques below.
SNMP TRAP – This the Software that will collect data from all the network devices, store it in an SQL database, and send you e-mail alerts for critical events. The Software, Dell OMSE, is free to install on a Dell server. SNMP Agent – Software that collects data from the hardware it’s installed on and passes it on to the TRAP server.
The TRAP server is a server that will collect and store SNMP data from agents.
You will need a server to act as the TRAP server, it must have a static IP and SNMP Ports 161 & 162 (UDP) open to the internal LAN, since a variety of devices will send SNMP data to the TRAP server.
Afte you select a TRAP server, install the Windows SNMP Service on it..
From the command prompt, type services.msc to open the services console.
1. Open the SNMP Service
2. Locate the TRAP tab
3. Select a community name. Use a friendly, descriptive name with no spaces or special characters. This name will be used throughout the process of configuring the SNMP agents later on.
4. Add the Trap server’s own IP address to the trap destination field
Once finished, look for the security tab.
1. Click on the security tab
2. Uncheck the authentication trap
3. Add the community name as READ ONLY. Selecting read/write poses a security risk, since SNMP commands can then be sent to the server to modify its settings by anybody inside the network that knows the community name. For this reason, I recommend using READ ONLY settings for all SNMP enabled devices.
4. Add localhost to the accepted packets field.
Restart the SNMP service so that changes take effect.
Download Dell Open Manage Essentials
Prior to installation, disable UAC.
Extract and install Dell Open Manage Essentials. OMSE has several prerequisites (.net 3.5, silverlight, etc.) which need to be installed, but that is beyond the scope of this article. You can install them by clicking on their respective links prior to installing OMSE.
Once all of the pre-requisites have been installed, proceed with the installation.
Open the Dell OpenManage Essentials application. There will be a wizard that explains the process of installing SNMP agents. Click next as you read the instructions or just finish since we will discuss that here in detail.
Configure the device discovery by adding your network’s internal IP address range in the discovery scope. OMSE will use this to scan your network and inventory your devices.
The next step is to select the type of agents to monitor. Although OMSE can use monitoring agents such as WMI and WS-MAN, we are going to focus on SNMP.
Enter your community name in the GET community field.
OMSE will begin a network discovery process. If you want to monitor your workstations and other DHCP enabled devices, allow the network discovery to proceed. You may however, not want to do this! DELL OMSE will ping devices based on a pre-determined schedule and if your users turn off their desktops and printers at night, you will receive system down notifications. You can opt to disable alerts during specific hours but this is not a good option. If a server’s RAID subsystem becomes degraded in the night, you will probably want to know about it right away.
I prefer to monitor network and infrastructure devices such as servers, networked printers, switches and routers, which are usually outside of the DHCP scope. There are other ways to get around the above mentioned issues, but for the sake of making things simple, I am going to exclude my DHCP scope and monitor only devices with static IP addresses.
Right click on the discovery and select STOP!
Create an exclusion range as shown below and enter your DHCP scope.
Click on discover schedule and select a date/time for discovery to be performed. In the name resolution section, use NetBIOS resolution if you don’t mind having the extra traffic. NetBIOS will find more devices on your network since it’s a broadcast protocol.
When finished with the discovery schedule, select date/time for the inventory schedule.
Finally select status schedule and add a pre-defined time to poll the devices.
I like to configure polling to every 20 minutes, this means it will take up to 20 minutes for the system to detect a node down and send you an alert. You can increase or decrease the polling time interval to compensate for network traffic vs. alert speed.
Click on the ALERTS menu and select as shown below to create a new email alert.
Give your alert a friendly name.
In the next screen, you can customize your alert message.
Click on the email settings tab to configure an SMTP server.
Select the type of notifications you want to receive.
Select the categories…
and the device types.
From the discovery and inventory menu, select the LAN inventory scope, right click on it and perform a discovery and inventory.
When completed, your monitored devices will be shown as below.
Now it’s time to install the agents on the client devices.
Installing SNMP Agents
There are two types of SNMP agents we will install to monitor our network devices. These agents will poll their devices for health and report back to the TRAP server via SNMP.
Windows SNMP Agent Event Viewer
This agent will collect data about Windows operating system and installed application. You can select which alerts will be sent to you by choosing Windows EVENT ID’s, or by category.
To begin, log in to a server that you want to monitor and install the SNMP service.
In the trap tab, add the community name and the IP address of the SNMP TRAP server.
In the security tab, disable authentication trap, add the READ ONLY community name and accept SNMP from localhost.
When finished, restart the SNMP service so that the changes take effect.
Once this is done, you can add alerts two ways. One way is to manually add the alerts you want. To do this, open a command prompt and run the command evntwin.exe.
In the example below, we are going to add Windows Server backup alerts. When Windows server backup does not complete successfully, the event will trigger an SNMP alert, which will be sent to the TRAP server, logged and finally e-mailed to you.
Click custom, and locate Windows backup from the applications folder. Highlight the Windows Backup events that you want to monitor and click on the add button.
Categorize the events by severity, highlight them and click add to add critical events and warnings in one simple step.
Once you have finished adding your custom events, highlight them and select settings to throttle the events. This will prevent to many events from filling your inbox in a short period of time.
Another way (and a better way) is to download our custom events script and run the script based on the type of server that you have. The script will import the most important events for you with the single click of a button. There are tens of thousands of events, so having a quick script will save you lots of time and trouble.
Download the zip file and extract the contents to c:\snmp folder.
Find the batch file for the type of server you are installing:
AutoImportExch – Exchange 2013, 2016 servers AutoImportDC – Windows server 2008, 2012 and 2016 Domain Controllers AutoImportServer – Windows Server 2008, 2012 and 2016 AutoImportSharePoint- SharePoint Server 2013, 2016
To install, simply double click on the server script and select run as administrator.
The script will begin installing the event alerts with periodic pauses at different categories.
When the script is finished, it will restart the SNMP service and log you out of Windows.
Log back in and execute the command evntwin.exe and you will see that thousands of critical events have been imported in to the event trap translator. It may take a while to load as it parses through thousands of events.
Don’t forget to highlight ALL the events, select settings, then apply a throttle.
I suggest no more than 2 of the same events in an 8 hour period… for sanity’s sake.
Now follow the above steps for all your Windows servers and you will be alerted whenever a critical alert takes place within your server environment. Sometimes, the errors can be hard to understand due to the large amount of information that is passed on.
If you have trouble, look for the Error Event ID (see example above)and a quick Web search will tell you more about the problem.
Conclusion: Rather than spending boring hours sifting through monotonous Windows logs, specific Windows event errors will trigger SNMP events, OMSE then sends you email notifications which allow you can take immediate action. This will no doubt free up valuable time so that you can concentrate on more important tasks:
Dell Open Manage Administrator is a collector of Dell hardware specific events. It can monitor the status of your RAID array, temperature of the CPU as well as memory and power supply redundancy. It will take critical events and forward them to the TRAP server who will log the event and send you a notification.
Dell OMSA should be installed on bare metal systems. Do not install it on virtual machines because virtual machines do not have hardware to monitor. OMSA is for monitoring physical machines only.
To begin, download Dell Open Manage Administrator. Extract the contents and install the software using the setup program. Log in to the UI and select alert management –> alert actions as shown below.
Click on each system event and enable the broadcast message option. Enable the system events you want to monitor.
If the server has a RAID controller, you will find the RAID alerts at the bottom of the page. Be sure to enable ALL the RAID events.
Enable the platform filter events.
Make sure that the community string and trap destinations are configured…
and finally decide on the verbosity level you want for the alert conditions.
The OMSA will now send SNMP alerts to the trap server and you will receive email alerts whenever and important event is triggered.
Adding Other Devices to Monitor using SNMP
With SNMP, it’s not just Windows and Dell servers you can manage! You can manage printers, routers, switches, et. al.
All you need is to enable SNMP on the device, set the community string and tell the device where to send the SNMP events to (the TRAP server).
In the example below, see how easy it is to configure SNMP alerts on a Sonicwall router?
All you need to enter is the community name and the IP address of the TRAP server! Then, go to Logs –> Categories and select the categories you want monitored.
Even if your devices do not support SNMP, you can still monitor whether or not they are on or off. Dell OMSE will send you an alert if a device fails to respond to a ping.
Navigate to public folders and click on the public folder name.
Select the subfolder you want to modify to highlight it and click on the 3 dots as shown below.
(Yes, someone actually has a public calendar for a restroom but it’s not what you think.)
Click on the root permissions link that pops up and click on the add, edit or remove icon to make permission changes.
Use the permission level drop down to set pre determined user permission levels or create custom permission levels using the check boxes.
The permissions are self explanatory (shown below) or click here for the expatiated version.
ReadItems The user can read items within the specified public folder.
CreateItems The user can create items within the specified public folder and send e-mail messages to the public folder if it's mail-enabled.
EditOwnedItems The user can edit the items that the user owns in the specified public folder.
DeleteOwnedItems The user can delete items that the user owns in the specified public folder.
EditAllItems The user can edit all items in the specified public folder.
DeleteAllItems The user can delete all items in the specified public folder.
CreateSubfolders The user can create subfolders in the specified public folder.
FolderOwner The user is the owner of the specified public folder. The user can view and move the public folder, create subfolders, and set permissions for the folder. The user can't read, edit, delete, or create items.
FolderContact The user is the contact for the specified public folder.
FolderVisible The user can view the specified public folder, but can't read or edit items within the specified public folder.
In Exchange you can block emails that have specific keywords. Since many spammers use different email addresses and sending servers, it can sometimes be difficult to block. When you have a recurring email that you want to block, look for a common denominator. Usually the spammer will have a link back to their Web site or show their name or company name in order for you to identify them. With this information, you can block them using a keyword filter.
Open Exchange ECP and navigate to mail flow –> rules. Select + to create a new rule.
Give your rule a friendly name (1). Apply the rule if the subject or body includes specific keywords (2) and finally, add the keyword.
Select message reject (1) and add a NDR message explaining why the message was rejected. Voila!
Exchange will reject any messages that have the specified keywords in the subject or content of the email.
In the future, you can block more keywords by editing the rule, clicking on the linked keyword list..
and then adding more keywords.
When attempting to access Sonicwall Web UI remotely, you receive one of these errors depending on the browser you are using:
Error descriptions include:
Many browsers no longer support the deprecated RC4 encryption cypher. This can be easily fixed by logging in to the Sonicwall’s diagnostic UI and unchecking the RC4 only option. To do this, log in to the Sonicwall’s INTERNAL HTTP URL and after you log in, change the URL to a trailing /diag.html. For example: http://192.168.1.1/diag.html. This will display the diagnostic UI. Note: You can also access the diagnostic UI from the Sonicwall’s outside address if you have HTTP access enabled on the WAN, but this is not recommended.
Uncheck the selection: Enable RC4-Only Cipher Suite
A restart will be required after which you will once again be able to log in using HTTPS.
If you have a gateway spam filter, it can get pretty confusing for end users having to discern whether a lost email is in the gateway’s junk store or in the Outlook junk folder. For this reason it’s usually a good idea to disable the Outlook spam filter option. You can easily accomplish this using group policy so that you don’t have to go one by one.
Open group policy editor and create a new policy. Enable the policy and add the users for whom you want to disable Outlook anti-spam.
Navigate to user configuration –> preferences –> windows settings –> registry and create a new registry item.
Select the following:
Action: Update Hive: HKEY_CURRENT_USER Key Path: Software\Policies\Microsoft\office\nn.n\outlook Value Name: DisableAntiSpam Value Type: REG_DWORD Value Data: 1 Base: Decimal
Replace the nn.n with your own version of Outlook:
12.0 for Outlook 2007 14.0 for Outlook 2010 15.0 for Outlook 2013 16.0 for Outlook 2016
One of the nice feature of virtualization, being able to take a snapshot (checkpoints are also knows as snapshots), can later come back to get you. For this reason, it’s not a good idea to take checkpoints in a production environment. If you do take a checkpoint for some reason, be sure to erase it as soon as possible.
I have run across many servers with multiple snapshots spread over years. Deleting these old checkpoints can be time consuming, stressful and occasionally downright ugly if you run out of disk space. Before deleting snapshots, make sure you have enough free disk space.
You can see the snapshot’s size by right clicking on it, selecting settings and then clicking on the inspect button or by visiting the folder where your aVHDX files are stored.
To be safe, you should have enough disk space free to accommodate the combined size of the main VHD file plus all the snapshots that you are going to merge. When you delete a snapshot, it merges the file into another snapshot. This happens until all snapshots have been deleted and merged. At this point, the last snapshot will merge to the main VHD file. The following is a brief analysis based on my own experience in a lab environment so results may vary depending of the differencing capacity between your files. But you should use the worse case scenario capacity requirement shown below.
Example: You have a 100 GB VHD and two 25GB snapshots, here’s what will happen when you delete a snapshot:
A 25 GB avhdx file will merge into another 25 GB avhdx file creating a 50GB avhdx file. The merge file will grow to 50GB before the old 25GB file is deleted so you will need an extra 25GB space to complete this process.
When you delete the second snapshot, the 50 GB avhdx will merge with the remaining 25GB avhdx creating a 75GB avhdx. The merge file will grow to 75 GB before the old 50 GB file is deleted, so you will need an extra 50GB space to delete the second checkpoint.
When you delete the last snapshot, the 50 final GB avhdx will merge with the original 100GB VHD file creating a 150 GB avhdx. The merge file will grow to 150 GB before the old 50 GB file is deleted, so you will need an extra 50GB space to delete the second checkpoint.
This is the reason you need enough free space to accommodate the size of the original VHD plus all the avhdx (snapshot) files combined.
What if you don’t have enough free space to merge the checkpoints?
If you do not have enough disk space available, there are three options:
Before proceeding, make sure that you have a backup. If you can move it to a volume mounted to the SATA or SCSI interface, it would be faster and more reliable than using an external USB drive. If you are using Hyper-V 2012R2, you can perform the following steps while your VM is on, otehrwise you must shut down the VM first.
Right click on the VM and select move. Choose to move the VM’s storage.
Select the option you want then the storage location.
Depending on the size of the VM and its snapshots, it may take a while.
In our lab, it took about 1 minute per GB to move to a USB 3.0 external drive.
Once the move has completed, proceed to delete the checkpoint.
When the merge completes, repeat the above steps to move the VM back to its original location.
If you have recently installed Comcast Business Class and your network has run afoul after installing and configuring their Cisco or Netgear routers, you may have IPV4 & IPV6 DNS and DHCP problems.
After a new installation of a Comcast issued Netgear router ( I specifically asked NOT to have the Cisco installed because I have had similar IPV6 issues with Cisco), lo and behold, same problem.
After disabling IPV6 and setting the DNS server in IPV4 to use our local DNS, neither one of the two settings are taking effect. So why give the user access to router then?
As you can see by the images, the router settings are useless. The Comcast routers continues to offer IPV6 DCHP and use their own DNS servers in IPV4 and IPV6. Don’t bother to call them, resistance is futile. The best course of action is to bridge the router and use your own router/firewall.
Before proceeding, make sure that you have installed and configured the NetExtender SSL VPN client. In order to log in to the SSL VPN, you must have the NetExtender client installed first!
Go to the SSL VPN login URL using Internet Explorer (note: this will not work with Chrome or Firefox) and enter your VPN credentials. Note that both the user name and password is case sensitive.
Once you have logged in successfully, click on the NetExtender image.
NetExtender will start and connect.
You are now connected!
This is a quick step by step guide and does not cover all the details, but rather the main settings necessary to get the Sonicwall SPAM filter up and running. The details that have been left out are mainly self-explanatory so you can figure them out while browsing through the Sonicwall anti-spam menu settings.
After you have registered you Sonicwall device and have purchased the necessary licenses, navigate to anti-spam –> settings and enable the anti spam service from your Exchange server. You can install the SPAM proxy on any server but in this example we will use the Exchange server for the installation.
Set the email threat categories according to your preferences and click accept.
Open Internet Explorer and go to IE Settings –> Compatibility Mode Add the Sonicwall IP address to the compatibility lists and to to the trusted sites list. Once this is done, click on the Sonicwall Junk Store installer.
You will be prompted to install an active X component. Wait for the download progress bar to finish and then the junk store installer will begin.
When the installer windows appears, click next to start the spam proxy installer process. When prompted, enter the relay domain names, separated by spaces.
Scroll down and expand the advanced settings and make sure that the settings are correct. The Exchange server’s internal IP address should appear in the server private field.
Navigate to Junk Summary and select the SPAM notification frequency.
From active directory, create an account that has AD LDAP read access. Set the password to not expire.
Navigate to Sonicwall LDAP configuration and click add server.
Add the IP address of a domain controller and the credentials you created in the prior step and click on the test LDAP button.
Lastly, open Exchange ECP or EMC (depending on your version), select mail flow –> receive connector and edit the default front end connector. In the security section of the front end connector, enable anonymous users.
This may seem odd to you but don’t worry, the Sonicwall device will proxy SMNP traffic and not allow open relay. If you want to test, go to www.mxtoolbox.com and enter the servers IP address or FQDN to test for open relay.
Now say goodbye to SPAM!