Aside from security, maximizing uptime is likely the top priority of just about any data center, regardless of size, industry or any other factors. Most businesses today run on data and that data is being facilitated by a data center. Businesses, and their employees and customers, depend on data being available at all times so that business processes are not interrupted. Every second a data center experiences downtime, their clients experience downtime as well. Data center managers and personnel are on a constant mission to prevent downtime and they must be vigilant because downtime can occur for a variety of reasons but one has been and remains the #1 threat – UPS battery failure.
UPS (Uninterruptible Power Supply) is the redundant power supply that is supposed to back up a data center in the event of an energy problem such as power failure, or a catastrophic emergency. Having an uninterruptible power supply is necessary in any size data center because no batteries last forever and, unfortunately, even the most observant and effective data center managers cannot prevent some power failures. The UPS also contains a battery that will kick in should the primary power source fail so that a data center (and its clients) can experience continuous operation. Unfortunately, the very thing that is supposed to provide backup power – the UPS – can sometimes fail as well. Emerson Network Power conducted a 2016 study to determine the cost of and root causes of unplanned data center outages, “The average total cost per minute of an unplanned outage increased from $5,617 in 2010 to $7,908 in 2013 to $8,851 in this report… The average cost of a data center outage rose from $505,502 in 2010 to $690,204 in 2013 to $740,357 in the latest study. This represents a 38 percent increase in the cost of downtime since the first study in 2010…UPS system failure, including UPS and batteries, is the No. 1 cause of unplanned data center outages, accounting for one-quarter of all such events.”
In order to properly for a strategy for UPS failure prevention, it is important to look at why UPS failure occurs in the first place. At the heart of the UPS system is its battery which powers its operation. UPS batteries cannot simply be installed and then left alone until an emergency occurs. Even if a brand-new battery is installed and the UPS system is never needed, the battery has a built-in lifespan and it will, over time, die. So even if you think you are safe with your UPS system and your unused battery, if you are not keeping an eye on it, you may be in trouble when a power outage occurs.
Beyond basic life-expectancy in ideal conditions, UPS battery effectiveness may be reduced or batteries may fail for other reasons. Ambient temperatures around the UPS battery, if too warm, may damage the UPS battery. Another reason a battery may fail is what is called “over-cycling” – when a battery is discharged and recharged so many times that it reduces capacity of the battery over time. Further, UPS batteries may fail due to incorrect float voltage. Every battery brand is manufactured differently and has a specific charge voltage range that is acceptable. If a battery is constantly charged outside the recommended charge voltage range – whether undercharging or overcharging – it will reduce the battery’s capacity and may lead to battery failure during a power emergency.
Fortunately, many of these UPS failures can be traced back to human errors that are preventable. This means that data centers looking to prevent UPS failures and maximize uptime can do so by implementing and vigilantly following a UPS failure prevention strategy. First, it is important to develop a maintenance schedule, complete with checklists for consistency, and actually stick to it. Don’t let routine battery maintenance fall off of your priority list, while it may not seem urgent, it will feel very urgent if the power fails.
One of the first and most important things that a data center should implement in their strategy is proper monitoring of batteries. Every battery will have an estimated battery life determined by the manufacturer, some even boast as long of a life cycle as 10 years! But, as any data center manager knows, UPS batteries do not last as long as their estimated life cycle because of a variety of factors. Just how long they will actually last will vary which is why monitoring is incredibly important. Batteries must be monitored at the cell level on a routine schedule, either quarterly or semi-annually and it is important to also check each string of batteries. By doing this on a routine schedule, you can determine if a battery is near its end of life cycle or has already reached its end of life cycle and make any necessary repairs or replacements. If it appears a battery is nearing the end of its life cycle it may be best to simply replace it so as not to risk a potential failure. In addition to physically checking and monitoring UPS batteries, there are battery monitoring systems that can be used. While physical checks are still critical, battery monitoring systems can provide helpful additional support that may prevent a UPS failure. Schneider Electric describes how battery monitoring systems can be a useful tool, “A second option is to have a battery monitoring system connected to each battery cell, to provide daily automated performance measurements. Although there are many battery monitoring systems available on the market today, the number of battery parameters they monitor can vary significantly from one system to another.
– A good battery monitoring system will monitor the battery parameters that IEEE 1491 recommends be measured. The 17 criteria it outlines include:
– String and cell float voltages, string and cell charge voltages, string and cell discharge voltages, AC ripple voltage
– String charge current, string discharge current, AC ripple current
– Ambient and cell temperatures
– Cell internal resistance
With such a system, users can set thresholds so they get alerted when a battery is about to fail. While this is clearly a step up from the scheduled maintenance in that the alerts are more timely, they are still reactive – you only get an alert after a problem crops up.” Further, as your monitor your batteries it is important to collect and analyze the data so that you can make informed decisions about how to best maximize battery life.
Next, it is important to properly store your battery when not in use to maximize its lifespan which will help it function properly in the event of use. A UPS battery must be charged every few months while in storage or its lifespan will be diminished. If you cannot periodically charge your UPS battery while in storage, most experts recommend storing your battery in cooler temperatures – 50°F (10°C) or less – which will help slow down the degradation of your battery.
To keep your UPS battery functioning in optimal conditions, ambient temperature should not exceed 77 degrees Fahrenheit and should stay, generally, as close to that as possible. It is important to not just prevent temperatures from exceeding that but prevent temperatures from frequently fluctuating because it will greatly tax UPS batteries and reduce their life expectancy. It is important that your UPS is stored in an area of your data center where temperatures are carefully monitored and maintained to help promote proper function of your UPS in the event of an emergency. Ideally, your UPS would be maintained in an enclosure with temperature and humidity control.
While routine maintenance will require attention and dedication, it is not without merit. In fact, Data Center Knowledge notes that there are statistics that back up the argument that routine maintenance really does prevent UPS failure, “In one study of more than 5,000 three-phase UPS units and more than 24,000 strings of batteries, the impact of regular preventive maintenance on UPS reliability was clear. This study revealed that the Mean Time Between Failure (MTBF) for units that received two preventive maintenance (PM) service visits a year is 23 times better than a UPS with no PM visits. According to the study, reliability continued to steadily increase with additional visits completed by skilled service providers with very low error rates.” Data centers must implement their own unique UPS maintenance strategy, tailored specifically to individual needs, and remain vigilant in their follow through. Implementing UPS maintenance best practices, including maintaining proper temperatures, maintaining proper float voltage, avoiding over-cycling, properly storing batteries, utilizing UPS battery monitoring systems, and performing routine visual inspections, will help significantly decrease the risk of UPS failure.