Electrical Distribution System in a Data Center

There are many different loads in the data center, such as IT devices, air conditioners (CRAC Units), pumps, lighting, etc. The critical supply from the utility / transformer / generator to the load is enabled by various types of power equipment. We are going to illustrate these equipment that are critical and without which it would not be possible to operate the data center.

Figure 1 is a block diagram of an electrical distribution system showing the name and the typical location of the electrical distribution equipment in a data center and the power flow path (Uptime Institute Tier Level II, N+1 Design) . This diagram is only an example of an electrical architecture and attempts to include all the possible major types of equipment used and their typical location in a data center. In the real world, a typical data center electrical design has much more complexity and diversity than that in this diagram.

Uptime Institute Tier II, N+1 Design

Figure 1 

Uptime Institute Tier III Design

Uptime Institute Tier IV, 2N+1 Design

Tracing the flow of power along its path (starting from the utility to the IT load) in the figure 1 above. It shows 8 essential facilities are critical for the power supply. This facilities distribute power to the downstream loads and also protect the power distribution system in the data center:

• Medium-voltage switchgear
• MV/LV transformer
• Low-voltage switchgear / switchboard / automatic transfer switch (ATS)
• UPS system with input/output switchboard and UPS distribution switchboard
• Power distribution Units (PDUs) and remote power panels (RPPs)
• Busway
• Panelboard
• Rack PDUs (rPDUs) / outlet strips

All facilities above, except for rack PDUs (rPDUs), are considered to be assemblies containing circuit breakers, switches, various types of relays, buses and connections, and control and auxiliary devices. Each device is optimized for long life and ease of maintenance.

The IEC 61947 & IEC 62271, which specifies the HV and LV switchgear terminology, considers switchboard to be the same as switchgear. For some regions like, North American, switchgear and switchboard are specified differently by ANSI and UL standards.

About Us


SMA combines with professional Chartered Engineers (CEng) from the Institute of Engineering Technology (IET), the Chartered Institute of Building Services Engineers (CIBSE) and the Hong Kong Institution of Engineers (HKIE). Our engineers have more than 20 years experience in data center design & build, building services engineering and energy conservation in the private and public sectors.

The team prepares the engineers and IT personnel to face any challenges in data centers and critical facilities of any size, in any location. For other design considerations / topics in data center and critical infrastructure, please visit 


(1) Site Selection,
(2) Space Planning,
(3) Cooling,
(4) Redundancy,
(5) Fire Suppression,
(6) Meet Me Rooms,
(7) UPS Selection,
(8) Raised Floor,
(9) Code & Standards,
(10) Transformers and Harmonic Distortion,
(11) Multi-mode UPS Systems,
(12) Electrical Rooms,
(13) Generator Systems,
(14) Generator Fuel Systems,
(15) Battery Systems,
(16) Earthing / Grounding and Bonding, etc.

Data Center Design Consideration: Electrical Rooms (1)

Switchboards, switchgear, transformers, generators, and UPSs require space for installation, maintenance, heat dissipation and future expansion (if possible). The wiring, busways, and raceways that distribute the electrical power must be accounted for now or in the future.

Electrical engineers should coordinate with mechanical engineers, architects, structural engineers, and others involved in the design of electrical rooms. Documentation and monitoring of electrical system’s equipment and how it connects to the rest of the facility must be accurately maintained.

We are going to explain the applicable code requirements and evaluate the design criteria for appropriate electrical-room size to accommodate present and future needs. Furthermore, we analyze the requirements for coordinating with structural, architectural, fire protection, and HVAC requirements.

   

Electrical rooms and mechanical, electrical, and plumbing (MEP) spaces are often an afterthought when it comes to building design and planning, either relegated to locations that are left over or deemed undesirable for other planning purposes. This shortsightedness can have unfortunate consequences on the cost, operations, and flexibility of the systems for the future.

NFPA 70: National Electrical Code (NEC) dictates the minimum amount of space needed around the equipment for access, operations, safety reasons, and conduit installation. Together, with the actual equipment sizes, this defines the overall minimum dimensional requirements of the room.





There are three types of general interior electrical spaces that factor into new building design: (1) main equipment rooms, (2) distribution pathways, and (3) local/branch equipment rooms. Code-required working space and dedicated space needs must be met. This article will outline important considerations for these spaces in the early stages of building design as they relate to building type, intended occupancy, size, and future expectations of both the building and the electrical systems.


Working Spaces


Let’s see the different between working and dedicated space as stated by the NEC (see picture below). The working space helps safeguard a clear working zone around all equipment and ensures protection for any workers or occupants within the room. This includes defining minimum width, depth, and height requirements for the working space, which varies due to voltage and the specific equipment. The higher the voltage of the equipment, the greater the depth of the working space. The width should be equal to the width of the equipment and no less than 30 in., while allowing for opening any doors or hinged panels to a full 90 deg. The height should be 6 ft 6 in. from the floor, or the height of the equipment if greater than 6 ft 6 in.

The style and construction type of the electrical equipment dictates whether only front access is required, or if rear and/or side access also is required. For each point of access to a piece of equipment, the minimum working clearances must be provided.

The dedicated space (depicted in red) and the working clearances (depicted in blue) are shown in a new emergency distribution room.


Dedicated Spaces


Dedicated space is a zone above the electrical equipment. It’s reserved to provide future access to the electrical equipment, protection of the electrical equipment from foreign systems, and for installing conduit/other raceways supporting incoming and outgoing circuits. The requirement for dedicated space applies primarily to switchgear, switchboards, panelboards, and motor control centers. The space should be equal in width and depth to the equipment size and extend from the floor to a height of 6 ft above the equipment (or to a structural ceiling, whichever is lower). No equipment or systems foreign to the electrical installation are allowed in this zone by the NEC.

The medium-voltage switchgear sections and unit substation transformers in a large data center installation require additional space and clearances.


The area above the dedicated space may contain foreign systems, provided proper protection prevents damage from drips, leaks, or breaks in these systems. However, it’s good practice to avoid having these systems installed in electrical rooms altogether.

While installations of equipment greater than 1,000 V generally follow the same principles, some of the specifics vary, requiring additional clearance around the equipment due to the increased hazard that these voltages impose (see picture above). Access to this equipment is preferably limited to only those deemed qualified to be there. For this reason, electrical equipment should be installed in rooms or spaces that are dedicated for that purpose and have controlled access.


Main Equipment Rooms


The main electrical room, or service entrance space, should coordinate with the local electrical utility. For example, main equipment rooms have requirements that dictate access to the space from the exterior for servicing, maintenance, and service feeder installation. The type of equipment installed will also further determine the room requirements. The service entrance room is typically located on an exterior wall for both code and practical reasons; it makes installation easier and minimizes the length of the service entrance conductors. Because the service conductors are usually the largest in the facility, this can have a substantial impact on cost.

Using arc-resistant switchgear will also impact space needs. This equipment will be taller and may have a larger footprint. Engineers will also need to account for the potential exhaust gases and arc flash energy by providing a pathway to expel them and relieve the pressure buildup from inside the switchgear.

If an exterior transformer is used to provide the service to a building, feeders from the transformer enter the building and transition to the main service entrance disconnect, typically a switchgear, switchboard, or panelboard. These feeders are often routed underground into the building through the exterior foundation wall via a coordinated opening. Additional coordination with the structural engineer is needed to avoid footings.

The elevation of the service entrance conduits many times do not naturally align with the equipment to which it is routed. Additional space in the form of increased height or footprint commonly is required to allow for the successful transition and termination of these conduits and conductors. Service installations that require medium-voltage equipment and/or transformers installed indoors will require additional elements including more space, higher fire ratings of the rooms (per NEC Article 450), and increased ventilation.

The location of any exterior equipment also needs to be coordinated with other architectural and landscaping elements. Minimum separation distances are often dictated by local codes/ordinances or utility requirements for proximity to screen walls, fencing, vegetation, paths of egress, or building fenestration.

Generator installations offer additional challenges when it comes to defining space needs. Noise, odor, and vibration factor into the location of this equipment within a building. The equipment should be located to minimize disturbances to building occupants and adjacent properties. Many jurisdictions have specific requirements on noise emissions, which will impact equipment placement and other components needed to meet requirements. Increasing the distance of this equipment from sensitive areas is one way of dealing with the concerns, but this comes with added feeder costs and may prove to be more costly than other options.

Sound attenuation and equipment required to meet specific emissions requirements, such as diesel oxidation catalyst, particulate filters, urea tanks, and selective catalytic reduction units, have significant cost implications and require a large amount of space to install.

Tier 4 versus Tier 2 compliance is usually dictated by an owner’s desire to use a generator for utility peak shaving or other non-critical proposes. It is crucial to have a clear understanding of current and future implications in both of these areas from the outset of a project and to discuss them thoroughly with the building owner.

The weight of a generator and the vibration experienced during its operation will have an impact on the building’s structural design. Generators require a lot of ventilation for cooling and combustion needs; getting air into and out of the room is critical and will impact placement.

With regard to fuel storage, most installations require a volume of fuel that dictates an external fuel tank with interconnecting fuel lines. NFPA (National Fire Protection Association) limits the overall capacity of diesel fuel inside buildings to 660 gal. The relationship of the exterior tank and the generator is also important to minimize pumping requirements and allow for gravity-drain return-fuel piping. This requires the fuel tank to be lower in elevation than the generator.

Direct access to the outside is preferable for maintenance and testing. All of this requires close coordination with the architectural, structural, and mechanical disciplines.

NFPA 110: Standard for Emergency and Standby Power Systems requires the emergency power supply for Level 1 installations to be installed in a separate room, separated from the rest of the building by 2-hour fire-rated construction. While NFPA 110 does allow the emergency power supply system equipment (EPSS; equipment consists of all components from the emergency power supply, or EPS, to the load terminals of the transfer switches) to be installed in the same room as the EPS, it is good practice to keep these separated to help enhance system resiliency. EPS rooms are also prone to additional dust, moisture, temperature fluctuations, and excessive noise during operation that limits the ability to have a conversation and may have a negative impact on other equipment if co-located.

For mission-critical facilities (e.g., financial institutions, data centers, and airports) and other highly sensitive installations, the use of a dry-type, pre-action, or another type of fire protection system that does not rely on a normally wet piping installation is highly recommended. In cold climates, this has an added advantage of preventing pipes from freezing, rupturing, and potentially flooding the EPS room.


Continue Reading: Part (2) - Data Center Design Consideration: Electrical Rooms


About us

Strategic Media Asia (SMA) is one of the approved CPD course providers of the Chartered Institution of Building Services Engineers (CIBSE) UK. The team exists to provide an interactive environment and opportunities for members of ICT industry and facilities' engineers to exchange professional views and experience.

SMA connects IT, Facilities and Design. For the Data Center Design Consideration, please visit 

(1) Site Selection,
(2) Space Planning,
(3) Cooling,
(4) Redundancy,
(5) Fire Suppression,
(6) Meet Me Rooms,
(7) UPS Selection,
(8) Raised Floor,
(9) Code & Standards,
(10) Transformers and Harmonic Distortion,
(11) Multi-mode UPS Systems, and 

(12) Electrical Rooms

All topics focus on key components and provide technical advice and recommendations for designing a data center and critical facilities.

Data Center Design Case Study

When the reliability demand is high and the data center has to work around pre-existing building constraints, there are significant design challenges.

With approximately 120,000 sq ft of white space divided into eight Tier III, 2 MW power data halls, the 20 MW data center meets a number of design challenges before construction even began.

The data center was originally the site of an old manufacturing facility. Coupled with the owner’s building requirements, this created a few challenges to the data center’s electrical distribution system design. Here’s a look at how Environmental Systems Design, working together with the other members of the building team, was able to solve them.


Challenge No. 1: The transformer


In an effort to maintain consistency across others data centers, the corporation likes to specify company-standard mechanical, electrical, plumbing (MEP), and fire protection equipment for all of its mission critical facilities. However, much of this equipment is designed for outdoor use, and because this particular facility has very limited outdoor space, significant adjustments had to be made for the transformer and switchgear equipment to work indoors.

For example, the heavy transformer couldn’t be placed on the raised floor designed for the data hall and electrical room space and wouldn’t be able to be removed from the building at the end of its useful life, as the 2500 kVA transformers each weigh approximately 16,000 lbs.

Instead, the facility was designed with the intent of moving in the transformer prior to the raised floor construction. Coordinating with the architect, saw cuts were designed into the pre-cast panels on the side of the building to make it easier for the contractor to remove/replace the exterior wall to get the transformer out for future installations or replacement.

Additionally, because the liquid-filled transformers specified are typically used outdoors, special design considerations were needed for the electrical room to contain the fluid in the event of a leak. A 4-in. metal dam was constructed along the perimeter of the room to contain the liquid, while perforated raised-floor tiles were installed around the transformer to facilitate the flow of liquid to under the floor.


Challenge No. 2: Uniform design


The owner wanted a repeatable, scalable design for each of the eight data halls to both create uniformity across the facility and provide the ability to build out data halls as needed over time. However, due to existing building conditions, including the different quantity and compact nature of the structural columns on the building’s first and second floors, an offsite modular construction design was eliminated.

Instead, the solution was to bring each component of the mechanical and electrical systems into the facility individually and build out the data halls one at a time as identically as possible.


Challenge No. 3: Medium-voltage equipment


Because the switchgear and transformers were located inside the building, medium-voltage (MV) feeders were routed throughout. The MV system itself was daisy-chained so it had to be installed and commissioned in its entirety when the first of the eight data halls were installed.

The active MV system presents a potential risk to workers during construction. Therefore, the MV feeders and conduit were routed through the ceiling of the first floor for safety, stubbing back up through the second floor slab and directly into the equipment. The feeders for the first floor equipment were routed similarly under the first floor slab. Because the eight data halls will be built out over time, this technique permits construction on the floor without subjecting workers to the active feeders.


Challenge No. 4: Switchgear and metering equipment


With very limited space for outdoor equipment, there wasn’t enough room for the utility’s 34.5 kV ground-level switchgear and metering. In addition, this equipment has not been fully vetted for the application and was seen as a risk by the owner. The only option was for the utility’s switchgear and metering to be pole mounted. This meant that the site’s 20 poles had to be spaced 20 ft from each other, taking up 400 ft of linear space on a site with little outdoor area.

The Reliability


Utility Service:

2N, or two separate utility sources/feeders, one general source and the second as backup. The switchgear is daisy-chained together, creating a connection from one switchgear to another down the line so that the power can feed into one switchgear, back out and into the other, rather than having multiple connections to the site.


Backup generators:

N+1 redundant swing generator. While each of the eight data halls has its own backup generator, another redundant swing generator for every two to three halls was designed as well, providing additional backup (a total of four swing generators in all). This provides an extra level of redundancy without the cost of providing an additional backup for each generator.


UPS system and distribution (2N):

Two sides to the electrical system were designed, where each side is a mirror of the other. The benefit of the 2N system topology allows for maintenance or a fault to occur on one of the sides, while still maintaining a completely active data center.



About Strategic Media Asia Ltd


Strategic Media Asia Ltd (SMA) is one of the Approved CPD Course Providers of the Chartered Institution of Building Services Engineers (CIBSE). SMA exists to provide an interactive environment and opportunities for members of IDC industry and engineers to exchange professional views and experience on data center design, critical infrastructure system, electrical and mechanical facilities, etc.

SMA connects IT, Facilities and Design. For details, please visit our website at www.stmedia-asia.com/trainings.html.

Data Center Design Consideration: Space Planning

A data hall houses computers and communications equipment that delivers, store and process information. This is the heart of the data center that operates all mission-critical applications. You need to address 2 key considerations:


(1) Restricting Access

Access to the data hall shall be limited and restricted to only authorized people and to minimize any accidental damage to the computer systems.

(2) Maximizing Floor Space

Floor space in the data hall is extremely valuable due to the high investment placed into the supporting infrastructure. It includes precision air-conditioning, standby power supply, equipment-safe fire protection system as well as security and monitoring systems.



"Non-essential" Items

Sometime "non-essential" doesn't mean useless. To maximize the use and enhancing the security of the area, "non-essential" items should be located outside the data hall. Examples of these items are:


(A) Operating Console / Network Operation Center (NOC)

It houses computer stations connect to the equipment or critical facilities in a data center. NOC may be situated in a separated room / area next to the data hall. Staff can monitor and manage the various systems here and enter the data hall only when they install cables or equipment. It also ensures that the operations and maintenance of the console do not affects the racks/cabinets inside the data hall.

(B) Gas Suppression Systems

Gas Suppression Systems such as FM200 are frequently used in a data center for fire fighting purposes. Gas is used over water as they do not damage the data hall's equipment when discharging. Unlike water, gas should be stored in gas cylinders. Placing these cylinders outside the data hall would allow your technicians to serve them easier.

(C) UPS Batteries & Generators

Uninterruptible Power Supply (UPS), high voltage switchgear and Genset are critical to supply backup power and are usually isolated in different rooms / area:

• Batteries will discharge gas during charging.
• Batteries contain acid that may leak and damage equipment
• Prevention of fire
• For safety reason, only qualified electrical workers are allow to enter the room and serve the high voltage facilities

(D) Air Handling Units (AHU) / Computer Room Air Conditioners (CRACs) Units

It's commmon for a dedicated AHU / CRAC Units to be used in a data room which ensure air from other parts of the data center is not recirculated into the critical area. In addition to being bulky, the AHU or CRAC Units generates heat and maintenance issues. AHU / CRAC Units, therefore, are unsuitable to be placed inside a data hall. Should they be installed in a data hall, they should be placed along the hot aisles.

Maintenance Area

It is important equipment installed in a data center can be repaired or replaced easily without affecting normal operations. Unless we absolutely sure that rear access to the equipment is not required, no equipment should be placed along a wall such as switchgear.

A minimum space greater than the depth of the equipment is required in front or at the back of the equipment which allows

- New parts' installation
- Maintenance or repair
- Airflow (such as hot aisle or cold aisle design)

For racks, a minimum of 1m clearance is observed between racks. A 1.5m radius is usually kept at the end of cabinets / racks which facilitates the use of trolley to move heavy equipment into the aisles.



Raised Floors

Typical raised floor inside a data hall / NOC are 150mm to 300mm (or higher depends on the floor height or design request, you may refer to the international standard / advice in TIA-942 or Uptime Institute). A ramp is usually created at the entrance to bridge the different in heights. It also allows trolley to move heavy equipment. The gradient of a ramp should not be steeper than 1:12.

Per normal practices, power cables (high voltage) are usually put under the raised floor, whereas data cables are usually put inside a cable tray overhead.

Cable Routing

We suggest power cables are separated from data cables. It ensures that the power cables do not pose any electromagnetic interference (EMI) to the data cables. Per previous advice, power cables are laid under the raised floors, whereas data cables are laid overhead.

In addition, data and power cables should be entered a data hall (or other faciliities' rooms) at opposite ends of the area whenever possible to avoid too many cables on one side and EMI problem.

Layout of a data center should be designed to minimize the run of data or power cables. For example, power cables enters the data hall and are routed through

- Riser (Cable Entry)
- Electrical Distribution Box / PDU (Power Distribution Units)
- UPS (High Voltage Switches, etc.)
- Equipment Racks

with minimized distance if they are located near each other relatively.

Expansion

Data Center are used to serve enterprises, government or other organizations. As business or services developed, the demand for faster applications and data storage grow as well. You are advised to allow space for expansion purposes. This expansion space could be located near the entrance of a data center / data hall which allows easier renovation and the installation of new equipment , cabinets and racks.

Cabling, cable trays and empty racks, where feasible, can be pre-installed in the expansion areas. This would minimize the amount of work to be done and the disruption of data center operations / equipment during expansion.



About SMA

Strategic Media Asia (SMA), a critical infrastructure training and event organizer based in Hong Kong, provides an interactive environment and opportunities for members of IDC industry and engineers to exchange professional views and experience on critical infrastructure and E&M facilities.

SMA is one of the CPD Course Providers of the Chartered Institution of Building Services Engineers (CIBSE).

For details of other data center courses and seminars, please visit our website at http://www.stmedia-asia.com/trainings.html.

Data Center Power System Design

Engineers should take a closer look at the different power strategies being used to distribute power, and how they impact the data center.

Alternating Current (AC) versus Direct Current (DC) is a battle that has been going on for more than a century and continues today in the data center industry. Although AC power is the standard, based on its potential for eliminating conversion losses and improving efficiency, many believe that DC power is the future of data center distribution. Still others believe that the same level of efficiency can be achieved with AC by using more efficient equipment with higher voltage distribution.

Electrical systems usually waste energy in the form of losses due to inefficiencies in the electrical equipment and distribution system. On average, the electrical distribution system losses account for 12% of the total energy consumed by the data center. For a data center with 2000 kW of IT load (2700 kW total load), that equates to an annual cost of USD$280,000


Power System Design Tips


So please review these six key items when planning a data center power distribution system:

• Install or replace existing power and IT equipment with energy-efficient equipment
• Review the proposed IT equipment to determine if the systems can operate on 240 V AC or 380 V DC
• Review all the advantages and challenges of the different power systems
• Determine how much of the existing infrastructure would need to be replaced to change power systems
• Design flexibility into the power system that will allow the data center to adapt in the future
• Design a power system that is modular to eliminate partial loading

Similar to the mechanical systems, modifications can be made to the electrical system to make it more efficient and save energy. The key to a good mission critical facility design is not to degrade the reliability of the facility in the process.

Typical Electrical Distribution Systems


Typical legacy data center electrical distribution system is made up of five major components:

Power is supplied to the data center at medium voltage from a utility/generator power source. The power is stepped down from medium voltage to distribution voltage by a substation transformer. The power then goes through an Uninterrupted Power Supply (UPS) system that conditions the power and provides ride-through capability during an outage until the generator starts. The power is then stepped down to substation voltage by a Power Distribution Unit (PDU). The PDU supplies power to the IT power supply where it is rectified and stepped down DC power, which is the internal operating voltage of the IT equipment.


Utility / Generator --MV AC--> MV/LV Transformers --480V AC--> Switchgear --480V AC-->
UPS --480V AC--> PDU --208/120V AC--> IT Power Supply --12V DC--> Servers


Four components in the legacy electrical distribution system with the highest losses are:

• Substation transformer: Transformer no-load and core losses
• UPS: Rectifier and inverter losses
• PDU transformer: Transformer no-load and core losses
• IT power supply: Rectifier and transformer losses

One method for increasing efficiency is to replace those pieces of equipment with more efficient equipment. Today with ultra-high-efficient transformers that efficiency is above 99.5%. Conventional double conversion UPS systems range from 84% efficient at 25% load to 94% at 100% load. Using flywheel or passive standby UPS topology can increase that range to 94% efficient at 25% load and 99% at 100% load.

Another method for increasing efficiency is to eliminate partial loading of the data center. Eliminating partial loading reduces losses by allowing the equipment to operate at its peak operating efficiency. This can be performed by designing a power system that is modular, grows with the load, or by designing a power system that uses flexible tiers, and matches the reliability and redundancy to the different programs within the data center.

A third method is to eliminate the inefficient electrical equipment altogether. Increasing efficiency by eliminating the equipment that has the most losses is the reason why different power strategies are being investigated for data center distribution.


About the Blogger

Strategic Media Asia (SMA) is one of the approved CPD course providers of the Chartered Institution of Building Services Engineers (CIBSE).

SMA, a critical infrastructure training and event organizer based in Hong Kong, provides an interactive environment and opportunities for members of IDC industry and engineers to exchange professional views and experience on critical infrastructure and E&M facilities.

For more details of other data center courses and industry events, please visit our website at http://www.stmedia-asia.com/trainings.html.