Power requirements for data centers and other mission-critical facilities continue to grow. While specific requirements of a facility's power distribution depend on the nature of its critical activities — and its anticipated future growth — most rely on large-scale uninterruptible power supply (UPS) systems. These systems, in turn, depend on effective grounding.
The Nature of Power
Before addressing the issue of effective grounding of UPS systems, it is worthwhile, first, to consider the levels of power reliability that characterize electrical systems for one type of mission-critical facility: the data center.
The Uptime Institute provides a tier system of classifications and certification for reliability of mechanical and electrical systems in data centers.
There are four tiers: I, II, III and IV. Most data centers have relatively similar components, typically designed to meet Tier-IV requirements — but constructed for Tier-III capacity.
Tier IV design, according to the Uptime Institute, is 2N electrical distribution , which means that power is distributed to critical loads via two different — and redundant — paths. Loss of one feeder anywhere in the distribution will not disrupt power to the critical load. To meet the Tier-IV design, equipment must be rated for dual-cord, dual-input configuration. Power is available on both cords, but only one is utilized. Upon loss of power to one cord, the load transfers to the second cord seamlessly.
Most data centers, however, are not constructed to Tier-IV specifications, which can be extremely cost-prohibitive. Instead, data centers and mission-critical facilities are designed and constructed to Tier-III specifications. Tier III employs N+1 redundancy in the service, UPS modules, mechanical systems and concurrent maintenance systems. A simplified Tier-III single line is illustrated below.
Single or dual medium-voltage power is brought to the facility and transformed to low voltage for distribution. Generators are installed to provide 100% backup so that prolonged loss of utility power will have no impact on data-center operation. Transfer between the utility power and generators can be at low or medium voltage.
Loads are segregated into three categories of power: HVAC, critical and house. Each category may have multiple double-ended substations, depending on the load requirements. The HVAC substations provide power to all mechanical equipment associated with cooling the facility. The house-power double-ended substation provides power to all non-critical spaces, such as administration, support spaces and lighting. The critical-power double-ended substation that is protected by a UPS provides conditioned power to the critical components of the data centers — the servers, direct-access storage drives and disk storage.
Most power supplied to a data center is conditioned power with stored energy as reserve. Consequently, the UPS requirements are very large in magnitude, ranging into the megawatts. This creates a distribution nightmare, considering that most readily-available single-module UPS are at most rated at 800 kilovolt-amperes (kVA). To develop high capacity output of UPS power, single-module UPS systems are installed in parallel. As many as seven modules may be installed in parallel to increase the capacity of the UPS systems.
Static-Switch Bypass
A large UPS system is typically provided with a static-switch bypass . If the UPS modules fail, the critical load will transfer to this bypass. Some designers also provide a wraparound maintenance bypass to the static switch, so as to isolate the UPS modules and static-switch bypass. However, the maintenance bypass provides unconditioned power to the critical load—with no stored energy reserve as backup.
In the event of a power loss—even for just milliseconds—the batteries associated with UPS modules provide the power to maintain continuity. If there is loss of power to the UPS module, the batteries will continue to provide power until their capacity is depleted and a low-voltage condition occurs. At that time the static bypass will transfer to a secondary source, if available and within voltage tolerances.
The Importance of Grounding
Tier-III installations are typically designed with N+1 modules — the total number of modules necessary to meet the load requirements, plus one additional module for redundancy — and are provided with a static-switch bypass to transfer power in the event of failure of the UPS modules. It is important that, if there is a problem with the UPS modules, the critical load transfers from UPS modules to static-switch bypass. In order for the transfer to occur, a good solid ground must be established.
Typically a ground wire is run, along with phase conductors, from the service substation to static-switch bypass. If the termination is not installed or maintained properly, an impedance may develop between the two reference grounds and cause increased voltage in the circuit phase conductors, pushing the tolerance range within which the bypass will transfer and causing the static switch to fail to transfer, even though a good secondary source is available.
It is vital that the ground is properly connected at the static-bypass cabinet, and zero potential is maintained at the neutral to ground bond at the static bypass and at the double-ended substation. Otherwise, the UPS system could fail to transfer to static bypass.
Most manufacturers recommend that a neutral and ground conductor be run from output isolation transformers from the UPS modules to static-switch bypass, where they will be connected to their respective bus. The neutral and ground should be bonded — in accordance with National Electrical Code (NEC) requirements — because the output of the transformers are separately derived systems. A ground is also run from the neutral and ground bond of double-ended substations to the neutral ground bond at the static switch. Because there is no transformer at the static switch to help establish a separately derived system, an electrically common point is established between the double-ended substation and static-switch bypass. This configuration, however, has the potential for creating problems with load continuity.
A major reason why data centers go off-line is human error. Where there is human intervention, there are potential problems with ground faults. Ground faults can be very difficult to predict and control and can cause havoc in large multi-module UPS systems. Smaller UPS systems — less than 225 kVA — have output isolation transformers with internal static-switch bypass. The output of the static-switch bypass and UPS is routed through a common output isolation transformer. This in turn protects the UPS system from ground faults and transients that may develop at the critical load.
On the other hand, UPS modules in larger systems typically have an output isolation transformer, but the static-switch bypass does not. If a ground fault occurs downstream from the UPS system but upstream from the power distribution unit, the fault will travel back to the source: the double-ended substation. To reach the source and help clear the fault, the load will transfer to the static-switch bypass. This can cause the main circuit breakers at the double-ended substation to trip on ground fault and take the critical load off-line. Because the breakers at the output of the static bypass and at the double-ended substations are approximately the same size, and are significantly larger than the minimum 1,000-amp setting allowed by NEC, it is possible that a facility's entire system will lose power.
Grounded Solutions
There are various solutions currently employed to help mitigate potential problems of ground faults and impedance between the two separately-derived grounds — at the static-switch bypass and the double-ended substation. A transformer can be installed at the input of the static switch so that the neutral-to-ground bond established at the static switch will come from a separate source.
This approach, however, can be costly, and the required transformer can be extremely large. Also, the transformer will contribute to inrush and additional impedance. But it may be of benefit in limiting maximum fault current available downstream from the UPS system.
Another popular solution is to implement high-resistance grounding. HRG is not commonly used on a low-voltage system. The intent is to introduce a resistor to limit the current that flows at the neutral and ground bond, where the ground-fault current transformer monitors fault current.
This method is difficult to implement, because it requires calculation of system capacitance and requires fine-tuning of the resistor in the field. It is also dangerous, requiring highly trained personnel to monitor the ground-fault alarm and then trace through the distribution system and isolate the source of the fault. Human error is already a major source of data center power loss. It does not seem a good idea to introduce yet more human intervention to trace and isolate fault current. In addition, the setting for the ground-fault sensors needs to be revisited any time a significant load is added that may change the system capacitance.
It may seem that the solution is as simple as providing a zone-interlocking relay-protection scheme from the double-ended substation down to UPS static switch, and a distribution switchboard downstream of the UPS system. But ground-fault coordination is very difficult to design and install. Also, the static switch will transfer from the UPS module at a much faster speed than any fast-acting relay.
It is essential to maintain operation of mission-critical facilities and data centers with a reliable distribution scheme. The design engineer should coordinate with the client to establish design parameters based on economics and level of required reliability for mission-critical facilities and ensure that the final product is a facility that meets all of the client's long-term operational requirement.
About the Blogger
Strategic Media Asia (SMA) is one of the approved CPD course providers of the Chartered Institution of Building Services Engineers (CIBSE) UK. The team exits to provide an interactive environment and opportunities for members of ICT industry and facilities' engineers to exchange professional views and experience.
SMA connects IT, Facilities and Design. For the Data Center Consideration Series, please visit
(1) Site Selection,
(2) Space Planning,
(3) Cooling,
(4) Redundancy,
(5) Fire Suppression,
(6) Meet Me Rooms, and
(7) UPS Selection
