Earthing & Grounding for UPS Systems

Power requirements for data centers and other mission-critical facilities continue to grow. While specific requirements of a facility's power distribution depend on the nature of its critical activities — and its anticipated future growth — most rely on large-scale uninterruptible power supply (UPS) systems. These systems, in turn, depend on effective grounding.





The Nature of Power

Before addressing the issue of effective grounding of UPS systems, it is worthwhile, first, to consider the levels of power reliability that characterize electrical systems for one type of mission-critical facility: the data center.


The Uptime Institute provides a tier system of classifications and certification for reliability of mechanical and electrical systems in data centers. There are four tiers: I, II, III and IV. Most data centers have relatively similar components, typically designed to meet Tier-IV requirements — but constructed for Tier-III capacity.

Tier IV design, according to the Uptime Institute, is 2N electrical distribution , which means that power is distributed to critical loads via two different — and redundant — paths. Loss of one feeder anywhere in the distribution will not disrupt power to the critical load. To meet the Tier-IV design, equipment must be rated for dual-cord, dual-input configuration. Power is available on both cords, but only one is utilized. Upon loss of power to one cord, the load transfers to the second cord seamlessly.





Most data centers, however, are not constructed to Tier-IV specifications, which can be extremely cost-prohibitive. Instead, data centers and mission-critical facilities are designed and constructed to Tier-III specifications. Tier III employs N+1 redundancy in the service, UPS modules, mechanical systems and concurrent maintenance systems. A simplified Tier-III single line is illustrated below.





Single or dual medium-voltage power is brought to the facility and transformed to low voltage for distribution. Generators are installed to provide 100% backup so that prolonged loss of utility power will have no impact on data-center operation. Transfer between the utility power and generators can be at low or medium voltage.

Loads are segregated into three categories of power: HVAC, critical and house. Each category may have multiple double-ended substations, depending on the load requirements. The HVAC substations provide power to all mechanical equipment associated with cooling the facility. The house-power double-ended substation provides power to all non-critical spaces, such as administration, support spaces and lighting. The critical-power double-ended substation that is protected by a UPS provides conditioned power to the critical components of the data centers — the servers, direct-access storage drives and disk storage.

Most power supplied to a data center is conditioned power with stored energy as reserve. Consequently, the UPS requirements are very large in magnitude, ranging into the megawatts. This creates a distribution nightmare, considering that most readily-available single-module UPS are at most rated at 800 kilovolt-amperes (kVA). To develop high capacity output of UPS power, single-module UPS systems are installed in parallel. As many as seven modules may be installed in parallel to increase the capacity of the UPS systems.





Static-Switch Bypass

A large UPS system is typically provided with a static-switch bypass . If the UPS modules fail, the critical load will transfer to this bypass. Some designers also provide a wraparound maintenance bypass to the static switch, so as to isolate the UPS modules and static-switch bypass. However, the maintenance bypass provides unconditioned power to the critical load—with no stored energy reserve as backup.





In the event of a power loss—even for just milliseconds—the batteries associated with UPS modules provide the power to maintain continuity. If there is loss of power to the UPS module, the batteries will continue to provide power until their capacity is depleted and a low-voltage condition occurs. At that time the static bypass will transfer to a secondary source, if available and within voltage tolerances.





The Importance of Grounding

Tier-III installations are typically designed with N+1 modules — the total number of modules necessary to meet the load requirements, plus one additional module for redundancy — and are provided with a static-switch bypass to transfer power in the event of failure of the UPS modules. It is important that, if there is a problem with the UPS modules, the critical load transfers from UPS modules to static-switch bypass. In order for the transfer to occur, a good solid ground must be established.

Typically a ground wire is run, along with phase conductors, from the service substation to static-switch bypass. If the termination is not installed or maintained properly, an impedance may develop between the two reference grounds and cause increased voltage in the circuit phase conductors, pushing the tolerance range within which the bypass will transfer and causing the static switch to fail to transfer, even though a good secondary source is available.

It is vital that the ground is properly connected at the static-bypass cabinet, and zero potential is maintained at the neutral to ground bond at the static bypass and at the double-ended substation. Otherwise, the UPS system could fail to transfer to static bypass.

Most manufacturers recommend that a neutral and ground conductor be run from output isolation transformers from the UPS modules to static-switch bypass, where they will be connected to their respective bus. The neutral and ground should be bonded — in accordance with National Electrical Code (NEC) requirements — because the output of the transformers are separately derived systems. A ground is also run from the neutral and ground bond of double-ended substations to the neutral ground bond at the static switch. Because there is no transformer at the static switch to help establish a separately derived system, an electrically common point is established between the double-ended substation and static-switch bypass. This configuration, however, has the potential for creating problems with load continuity.





A major reason why data centers go off-line is human error. Where there is human intervention, there are potential problems with ground faults. Ground faults can be very difficult to predict and control and can cause havoc in large multi-module UPS systems. Smaller UPS systems — less than 225 kVA — have output isolation transformers with internal static-switch bypass. The output of the static-switch bypass and UPS is routed through a common output isolation transformer. This in turn protects the UPS system from ground faults and transients that may develop at the critical load.

On the other hand, UPS modules in larger systems typically have an output isolation transformer, but the static-switch bypass does not. If a ground fault occurs downstream from the UPS system but upstream from the power distribution unit, the fault will travel back to the source: the double-ended substation. To reach the source and help clear the fault, the load will transfer to the static-switch bypass. This can cause the main circuit breakers at the double-ended substation to trip on ground fault and take the critical load off-line. Because the breakers at the output of the static bypass and at the double-ended substations are approximately the same size, and are significantly larger than the minimum 1,000-amp setting allowed by NEC, it is possible that a facility's entire system will lose power.


Grounded Solutions

There are various solutions currently employed to help mitigate potential problems of ground faults and impedance between the two separately-derived grounds — at the static-switch bypass and the double-ended substation. A transformer can be installed at the input of the static switch so that the neutral-to-ground bond established at the static switch will come from a separate source.

This approach, however, can be costly, and the required transformer can be extremely large. Also, the transformer will contribute to inrush and additional impedance. But it may be of benefit in limiting maximum fault current available downstream from the UPS system.

Another popular solution is to implement high-resistance grounding. HRG is not commonly used on a low-voltage system. The intent is to introduce a resistor to limit the current that flows at the neutral and ground bond, where the ground-fault current transformer monitors fault current.





This method is difficult to implement, because it requires calculation of system capacitance and requires fine-tuning of the resistor in the field. It is also dangerous, requiring highly trained personnel to monitor the ground-fault alarm and then trace through the distribution system and isolate the source of the fault. Human error is already a major source of data center power loss. It does not seem a good idea to introduce yet more human intervention to trace and isolate fault current. In addition, the setting for the ground-fault sensors needs to be revisited any time a significant load is added that may change the system capacitance.

It may seem that the solution is as simple as providing a zone-interlocking relay-protection scheme from the double-ended substation down to UPS static switch, and a distribution switchboard downstream of the UPS system. But ground-fault coordination is very difficult to design and install. Also, the static switch will transfer from the UPS module at a much faster speed than any fast-acting relay.

It is essential to maintain operation of mission-critical facilities and data centers with a reliable distribution scheme. The design engineer should coordinate with the client to establish design parameters based on economics and level of required reliability for mission-critical facilities and ensure that the final product is a facility that meets all of the client's long-term operational requirement.



About the Blogger

Strategic Media Asia (SMA) is one of the approved CPD course providers of the Chartered Institution of Building Services Engineers (CIBSE) UK. The team exits to provide an interactive environment and opportunities for members of ICT industry and facilities' engineers to exchange professional views and experience.

SMA connects IT, Facilities and Design. For the Data Center Consideration Series, please visit 

(1) Site Selection,
(2) Space Planning,
(3) Cooling,
(4) Redundancy,
(5) Fire Suppression,
(6) Meet Me Rooms, and
(7) UPS Selection

Commissioning: Electrical System for Mission Critical Supply (3)

Refer to Previous post: Commissioning: Electrical System for Mission Critical Supply (2)


Generator Paralleling Switchgear

This is a crucial component to a critical facility in situations where the generator supported load exceeds the capacity of one generator.

Generator paralleling switchgear systems should be tested at the rated power factor of the generator paralleling switchgear system—typically 0.8. This is important to show that each generator properly shares the kW and kVAR loads. Just because paralleled generators evenly share kW while serving a resistive load does not always mean that they will evenly share kVAR when serving a reactive load.

A major challenge with testing generator paralleling switchgear systems is that they are often rated for very heavy loads due to the number of generators that can be connected to them. In some cases, it may not be practical and may also be very expensive to load generator paralleling switchgear systems to rated capacity.

It is recommended that enough load be provided so that it exceeds the capacity of one generator. Ideally, the load banks provided will be sized to the expected operational capacity of the generator paralleling switchgear, but not necessarily to its full design capacity.

Generator paralleling switchgear systems rely heavily on programming within the programmable logic controller (PLC) for operation. Knowledge of how this program operates is often limited to a handful of experts. Changes to PLC programming must be documented in a PLC programming change log. The log should include the date of the change, the reason for the change, a description of the change, and the new version number of the program that includes the change. Older versions of the program should be saved in the event that updates create additional problems and reverting back to an earlier version of the program is required.


Main Electrical Switchgear

It is also an important component to a critical facility because it distributes power to all of the downstream electrical distribution equipment.

Circuit breaker settings must be inputted, coordinated, tested, and verified throughout all main electrical distribution equipment. If there is a fault in the system, it is imperative that selective coordination is implemented so that the fault is isolated as far downstream as possible.

Main circuit breakers must be properly set up to ensure that they will stay closed during fault conditions and wait for downstream equipment to clear the fault. This will be ensured by implementing proper National Electrical Testing Association-recommended circuit breaker testing including instantaneous pickup, short time pickup, short time delay, long time pickup, long time delay, ground fault pickup, ground fault time delay, contact resistance tests, and insulation resistance tests.

While main electrical switchgear is an integral part of the electrical distribution system, the system’s current carrying capacity may increase the arc flash hazard. To avoid injury, main electrical switchgear should be disconnected before it is opened or worked on.

Because the owner will often not own a means of disconnect ahead of this equipment, it usually requires involvement from the utility provider, which can be problematic and difficult to schedule.


Static Transfer Switch (STS)

An STS is an important and useful component for a critical facility because it provides the ability to seamlessly transfer load during both failure and maintenance situations.

STSs behave similarly to ATSs, but because they are designed to transfer within a few msec, there are several settings that must be coordinated. STSs are commonly fed from UPS systems. These UPS systems are present to prevent interruptions to the downstream STSs. During a planned maintenance event or during a utility power failure, the UPSs are designed to perform transfers to bypass or battery within a certain time frame.

Because the STSs are set up to transfer on a loss of the primary source for a certain duration, the time frame must be longer than the allowable interruption seen from the UPS. If not coordinated properly, a routine transfer to bypass at the UPS level can cause the downstream STSs to transfer to their secondary source.

On several occasions, phantom voltage and current readings have been observed at the STS screens with no connected load. Rebooting the system typically corrects this problem. While the manufacturers generally indicate that there are no operational risks, this anomaly is puzzling.


Electrical Power Monitoring System (EPMS)

The EPMS allows all of the electrical systems within the critical facility to be monitored from a single location, giving the operator visibility to ensure that all systems are not generating any alarms and are operating properly and efficiently.

When confirming that the EPMS is monitoring systems correctly, multiple states must be checked for each point. Points must be modified in the field and checked to ensure that the same values or statuses observed in the field are properly reported back to the EPMS.

One difficulty encountered in this area has to do with discrepancies with points. Design engineers typically specify points to be monitored by the EPMS, but they often approve equipment submittals that are unable to provide these points. To avoid this problem, it is best to meet with the design engineer and the equipment manufacturers prior to the acceptance of the submittals to ensure that the points that are important to the design engineer can be provided by the equipment.


Conclusion

The equipment in the electrical distribution system of mission critical facilities must operate dependably. After commissioning challenges have been resolved and best practices have been employed, these systems will meet the original design intent and owner’s requirements, ensuring the owner that the facility embodies reliability, redundancy, and resiliency.

About The Blogger

Strategic Media Asia (SMA, www.stmedia-asia.com) is a leading technical training and event organizer for corporations specialized in data center design & build, E&M facilities, telecom, ICT, finance and colocation. Currently, SMA delivers a series of data center trainings and qualification programs in Hong Kong, Taiwan and Macau.

All these events / training seminars are designed to support the leadership needs of senior executives (Chief Information Officers, IT Directors / Managers, Facilities Managers, company decision makers, etc.) and to provide useful and applicable knowledge.

For detail, please visit SMA's Technical Training Seminars & Data Center Qualification Programs.

Commissioning: Electrical System for Mission Critical Supply (2)

Refer to Previous post: Commissioning: Electrical System for Mission Critical Supply (1)


Automatic Transfer Switch (ATS)

The ATS is an important component of the critical facility because it is used commonly in critical facility designs to transfer power from a primary source to a secondary source after the loss of the primary source.

Open transition ATSs are designed to allow for an interruption to the load using a break-before-make transfer. Because of this, loading the ATS during open transition transfers during functional performance testing is not required. Load is also not required when testing an ATS’s ability to perform closed transition transfers. During closed transition transfers, the ATS will parallel the primary and secondary sources prior to transferring.

It is important to ensure that the ATS can properly conduct closed transition transfers and will handle the transition in the same manner, regardless of whether it is carrying load or not. A power quality meter must be connected to the output of the ATS to confirm that the transfer is completed within the specified time for closed transition applications. It should be noted that load is required for all ATSs when conducting infrared scanning.

It is recommended that all components of the ATS are infrared scanned under full load on all primary, secondary, and bypassed power paths after final installation is complete. Load is also required for closed transition applications when the secondary source of the ATS is a generator. This testing is usually conducted as an integrated system test to prove that the generator and ATS work properly together under full load. The integrated system testing is conducted after functional performance testing for the ATS, generator, and other integral systems is completed.

In most cases, for an ATS to be functionally tested, both sources must be available because the ATS will usually inhibit any transfer if there is only one source. This problem can arise in situations where ATSs are added to existing live facilities. Because of their integral role in the power distribution system, they often can’t be tied into the electrical system without bringing down the loads that they will serve. In an effort to minimize disruption to the live facility, the ATS testing will likely occur prior to connecting it to the live facility.

However, the ATS can be connected to the secondary source if the secondary source is a generator. When the primary source serving the load is restored, there is usually limited time for testing the ATS as it will immediately be required to provide power to critical loads.


UPS

The UPS is a critical component to supporting critical loads, as it is the primary system responsible for maintaining continuity of load during a loss of utility. Courtesy: ESDThe UPS is probably the most important piece of equipment in the critical facility because of its ability to maintain power to critical loads, regardless of the operation of all of the other supporting systems.

Monitoring the inputs to the rectifier of the UPS, the static bypass within the UPS, and the UPS output bus is considered best practice during functional performance testing. After each transient, step load, or battery discharge test, the waveforms recorded by the power quality meters set up on the system should be reviewed to confirm that no events were triggered and that the output waveforms stayed within tolerance and recovered within the specified time frame.

UPS systems are often placed into service quickly after functional performance testing, so it is best to check the power quality meter results—including waveform captures—during on-site testing rather than waiting for a report from the meter technician. This way, any problem discovered during UPS testing can be quickly rectified as the manufacturer often has to consult the factory on problematic internal UPS operation.

Full load endurance tests should be conducted on UPS systems after the system has been installed on-site, even if full load testing was conducted in the factory. Many components need to be disconnected for shipping and are then reassembled on-site. Electrical equipment can also be affected by problems that develop during shipping and may not be detected without performing the endurance test on-site. Generally, an 8-hr duration for a full load test is considered adequate to confirm that the system will be capable of functioning at full rated load without problems.

In some cases, it can be difficult to monitor the logic used by the UPS to handle various operations because the actions are carried out by microprocessors installed on circuit boards. This emphasizes the importance of properly setting up power quality monitoring equipment prior to testing the UPS.

If a problem is detected during testing, the manufacturer will have a much easier time solving it if it is provided with significant data generated both by the UPS’s internal monitoring system and the external power monitoring equipment used during testing. When a failure occurs, it can be very difficult to understand what is happening inside the equipment. Captured test data almost always improves the issue resolution process.


Next Post: Commissioning: Electrical System for Mission Critical Supply (3)

Commissioning: Electrical System for Mission Critical Supply (1)

Critical Facilities Engineers should overcome the inherent challenges with functional performance testing for their electrical system.


The overall goal of commissioning must be to ensure that a facility meets the design intent and the owner’s requirements. For critical facilities, this goal is generally achieved by proving to the owner that the reliability, redundancy, and resiliency that he or she paid for is indeed present and operational in the finished facility.

Because there are so many failure scenarios and variables, it is rarely possible or cost efficient to reasonably test each one, but the commissioning authority has an obligation to provide a level of testing that will allow the owner to feel confident that each system is working and capable of maintaining a proper planned operational state during common external events.

As expected, the owner will want to use the commissioning process to be certain that the installation, performance, and operation of new equipment is acceptable before it supports critical load, and he or she will strive to do this as cheaply and as quickly as possible.

In the forthcoming posts, we will explores the best practices for testing several electrical systems, as well as some of the challenges encountered. Implementing these best practices and lessons learned on future projects will improve the quality of the product provided to the owner:


(1) Generator
(2) Automatic transfer switch (ATS)
(3) UPS
(4) Generator paralleling switchgear
(5) Main electrical switchgear
(6) Static transfer switch (STS)
(7) Electrical power monitoring system (EPMS)


Generator

Including the generators in the commissioning scope for a critical facility is imperative because they are the only source of long-term standby power when the utility becomes unavailable.

When testing a generator, it is best practice to ensure that the load for step loading and endurance testing has a power factor rating that matches the nameplate power factor on the generator, as the generator will be tuned and calibrated to operate best at its rated conditions. The manufacturer also will not likely be able to provide documentation on how the generator is expected to perform if the load used for testing deviates from the name plate conditions. The tuning and calibration is especially important when attempting a 0% to 100% step load, and often the system will not respond properly within acceptable tolerance if the power factor of the load does not match the nameplate rating.

Due to new some regulations in several countries, generators are now limited regarding the amount of pollution that they can emit under all running conditions, including when responding to step loads. This has been a challenge for generator manufacturers who in the past simply allowed the system to call for more fuel, which resulted in billows of black smoke entering the environment. In an effort to minimize pollution, manufacturers have had to finely tune the generators, resulting in the increased importance of testing the generators at rated power factor. In addition, because the generators are typically exercised under load for routine maintenance and testing, the owner often buys a permanent resisted load bank (unity power factor) sized for the rated capacity of the generator. It is important to explain to the owner that the permanent load bank that will be used for future load testing may not be appropriate to use during commissioning if it is rated at unity power factor.


Next Post: Commissioning: Electrical System for Mission Critical Supply (2)