plantemoran.

com

TIA-942: Data Center Standards

Sri Chalasani

Merit 2010 Conference

May 25, 2010
Objectives
 What are concerns in the data
center?
 Data center standards & best
Practices
Data Center Definition
 Computer facility designed for continuous use by
a n c y
several users, and well equipped with hardware,
u n d
R e d software, peripherals, power conditioning and
backup, communication equipment,
e r
security systems, etc. – businessdictionary.com
w
P o i n g includes redundant or backup
i o n
…..It generally
t supplies, redundant data
d i
power
con communicationsiconnections,
n g &
environmental
i to rcontrols (e.g., air
o n
conditioning,
M fire o
r l s
suppression) and security
n t
Co
devices. – wikipedia.org

 Notice the common terminology

 Levels of implementation set them apart
Why should we care?
 DCs house mission-critical data &
equipment. In addition to protecting
this…
 Challenges… increased demand for:
 Applications / systems availability / SLA
 Complex & heterogeneous systems
 Service levels for uptime and responsiveness
 Amount of data (live and retention)
 Regulatory compliance and security
 Changing business demands
 Green practices & energy costs
Data Center Standards

 Without standards… enormous variation

in data center designs
 Three commonly known tier systems
 Uptime Institute (1995)

 Syska Hennessy Group

 ANSI/TIA-942 or TIA-942 (2005, 2008, 2010)
Data Center Standards
 Uptime and TIA-942
 Neither addresses the challenges
 Both provide a framework to alleviate
challenges
 TIA-942
 Requirements / guidelines for the design
& installation of a data center
 Multidisciplinary Design Considerations
 Intended Audience
TIA-942 Multidisciplinary Design
 Design Considerations
1.  Design Process 12. Access Providers
2.  Space Planning 13. Telecom Spaces
3.  Redundancy 14. Cabinets & Racks
  Architectural Design
4.  Sitefloor,
(space, Selection 15. Cabling Pathways
light, security etc.)
  Structured
5.  Architectural Wiring 16. Cabling Systems

  Electrical
6.  Structural 17. Cabling Field Testing

  Cooling
7.  Electrical 18. Telecom Administration

  Operations
8.  Mechanical/Cooling 19. Information Technology
9.  Fire Protection 20. Commissioning
10. Security 21. Maintenance
11. Building Automation
TIA-942 – Discussion Topics
 For today’s discussion, focus on…
1.  Data Center Spaces.
2.  Data Center Cabling
3.  Electrical
4.  Cooling
5.  Tier System
 Spaces
 TIA-942 – 5-key functional areas:
 (1) Entrance Room (ER)
 (2) Main Distribution Area (MDA)
 (3) Horizontal Distribution Area (HDA)
 (4) Zone Distribution Area (ZDA), opt.
 (5) Equipment Distribution Area (EDA)
 Ideally separate rooms but not practical
for normal organizations; Can be
consolidated with defined areas
Source: Corning – Distribution in the data center
 Spaces
(5) (3) (2) (1)

(4)
ZDA

Source: ADC’s Data Center Optical Distribution Frame: The Data Center’s Main Cross-Connect
 Spaces

Source: Corning – Distribution in the data center

Spaces
 Typical Data Center Requirements:
Location Ceiling Height
  Avoid locations that restrict   Min. 8.5’ from finished floor to
expansion any obstruction (sprinklers,
lighting fixtures, or cameras)
  Redudnat Access
  Cooling architecture may
  Delivery of large equipment
dictate higher ceilings
  Located away from EMI sources
  Min. 18” clearance from water
  No exterior windows (inc. heat & sprinkler heads
security risk)
Flooring / Walls
  Provide authorized access &
monitored   Anti-static properties

Size – no magic formula   Sealed / painted to minimize

dust
  Sized to meet the known
requirements of specific   Light color to enhance lighting
equipment   Min dist floor loading 150 lbf/
  Include projected future as well Sq-ft, Reco. 250 lbf/Sq-ft
as present requirements
Spaces
Doors Operational parameters
  3’ wide x 7’ high, no /   Dedicated HVAC system preferred
removable center obstructions (68 – 77 F); measured every
10-30 ft at 1.5ft height
Lighting   HVAC – min. 100 sqft/ton
  Min. 500 lux in the horizontal   Max. temp rate of change: 5 F/hr
plane and 200 lux in the
  40% to 55% relative humidity
vertical plane
(reduces ESD)
  Lighting on separate circuits/
  Electrical - Signal reference grid
panels
(SRG)
  Emergency lighting & signs
  Sprinkler systems must be pre-
action system
Other Equipment
  UPS, power dist. or conditioner Security
  <= 100kVa inside room   Camera monitoring (int./ext.)
  > 100kVa in separate room   100-yr flood plain
Spaces – Best Practices
 Locate ER outside of the DC for security
purpose; if inside DC, consolidate ER & MDA
 MDA centrally located
 Both MDA & HDA require separate racks for
fiber, UTP and coax cable
 ZDA is optional, but provides additional flexibility
(pre terminated cables)
 EDA – contains equipment only
 Each space requires same power/cooling req.

ER: Entrance Room, MDA: Main Distribution Area, EDA: Equipment

Distribution Area, ZDA: Zone Distribution Area, DC: Data Center
Spaces – Raised vs. Solid Floor
 Raised floor a very common notion, but...
  Older equipment vs. newer   Only place to go is creep into
equipment air flow (bottom-up cold aisle….warmer air at
vs. front to back) cabinet tops.
  Hot aisle – Cold aisle air flow   Typically see passive
dynamics components or open spaces
  Cold air – want to fall, but we near top of cabinets
are pushing – requires   Both use anti-static tiles or
pressure through perf. tiles flooring
  Opening / leaks in flooring has   Data & electrical cabling
impact on pressure restrictions
  Equip. densities increase ->   New build – more expensive
higher head load -> higher   Have to look at your
pressure of cold air through environment to see if raised
restrictive space floor makes sense….do use this
  What happens to hot air? – as the rule of thumb!
flows up, reduces temperature
and begins to fall down again
Cabling Systems
 Structured vs. Unstructured Cabling
  Horizontal cabling   Horizontal cross-connect
(HC) in the
  Backbone cabling
telecommunications
  Cross-connect in the room, horizontal
entrance room or main distribution area or main
distribution area distribution area
  Main cross-connect (MC)   Zone outlet or
in the main distribution consolidation point in
area the zone distribution
area; and
  Outlet in the equipment
distribution area
 Cabling Systems

Source: Corning Cable Systems – Just the Technical Facts

Cabling Systems

  Reduced Data Center Topology

 Consolidated ER/MDA/HAD
 Applicable to most enterprises

Source: Orthronics – Standards-Based Data Center Structured Cabling System Design

Cabling Systems – Transmission Media
 100-ohm twisted-pair copper cable
 Category 5e or 6, 6A
 10GbE: Cat 6 – 37-55mts, Cat 6A – 100mts
 Multimode fiber optic cable
 62.5/125 µm or 50/125 µm
 50/125 µm 850 nm laser optimized mmf
 Singlemode optical fiber cable
 75-ohm coaxial cable
 Type 734 & 735 cable
 Type T1.404 coaxial connector
Cabling Systems – Overhead / Under floor
 Under Floor Cabling
 Less expensive if raised floor than overhead
 Cabling in cable trays to minimize airflow blocks;
consider multilevel trays for fiber/copper
 Provide adequate capacity for growth
 Separate fiber cords from copper cabling from
power
 Typically placed in the hot aisle
 Electrical – color coded PDU with locking
receptacle. Receptacles labeled with PDU/panel
ID & breaker #
Cabling Systems – Overhead / Under floor
 Overhead
 Can be used in raised floor environments also
 Multi level cable tray system (3 Layer)
  Bottom layer – copper
  Middle layer – fiber
  Top layer – power
 Suspended from ceiling; min.12” clearance
above each ladder
 Separation from fluorescent lights (5”) & power
 Avoid blocking cooling ducts (overhead cooling)
Racks / Cabinets
 Placement of racks / cabinets
 Hot aisle / Cold aisle - arranged in an alternating
pattern (with fronts facing each other)
 Cold aisles are front & Hot aisles are rear of
racks/cabinets –
 If there is a raised floor, PDU cables are run in
cold aisle. Data cable trays for telecom cabling
are typically placed in hot aisle.
 Common bonding network (CBN)
  Racks / cabinets individually, cable trays, HVAC,
PDU, panel boards, raised floor structure, columns
 Front clearance – min. 3ft, 4ft recommended
 Raised Flooring vs. Traditional Flooring
Racks / Cabinets
 Placement of racks / cabinets
 Front rails recessed for wire management
 Switch-Panel-Switch arrangement
 Front edge of cabinet on edge of tile
 Perforated tiles at front of cabinets
 Provide blank panels in empty spaces
Electrical Considerations
 Unfortunately no magic bullet!
 Manual process for load configuration
 APC ‘s “Calculating Total Power Requirements for
Data Centers” By Richard Sawyer – framework
for calculating req.
 Color coded PDU with locking receptacle.
Receptacles labeled with PDU/panel ID &
breaker #
 Best Practices
  Multiple power grid connects   Intelligent PDU
  Sub-breakers per relay rack   Generator capacity to include
or lineup for cooling
  Dual A-B cording   UPS capacity to include
cooling and lights
  Accommodate growth
Cooling Considerations
 # 1 Mitigating Factor – heat removal
 No specific guidelines; basic physics
 Cooling reqd. = Heat Generated = Electrical load

 Design Implications
  Layout of racks in alternating   Quantity and location of vents
rows   Sizing of ductwork
  Location of CRAC units   Proper internal configuration of
racks
Cooling Considerations
 Process
  Determine critical heat load   Determine cooling delivery
methodology(s)
  Establish critical loads - watts-
per-RLU   Room, Row, Rack
  Determine the CFM   Blank panels/short circuits
requirements per RLU   Cold air containment
  If possible, divide the room   Special Considerations – high
into cooling zones by RLU BTU
  Determine appropriate air   Establish a floor plan
conditioner type(s)
  Deploy a comprehensive
  Equip. airflow (f->b / s->s) monitoring system

 Cooling is not enough – airflow required

Cooling Considerations - Airflow
Supply &
Return
Based

source: apc.com
Fire Detection and Suppression
 Significant risk of electrical fires
 A comprehensive fire detection &
suppression system is mission-critical
Detection Suppression
  Both heat and smoke detection   Follow NFPA 75 standard firewalls
  Interconnected with the fire   Sprinkler systems — both flooded
suppression system, local and pre-action
alarms, monitoring system, etc   Chemical systems or Clean Agent
  Installed in accordance with (FM 200, Inergen, Ecaro-25(FE 25),
NFPA 72E Novec 1230)
  Installed below raised floors   Manual systems (Manual pull
and other areas stations, Portable fire extinguishers
  Airflow patterns determines
location of detection units
Tier System – Uptime & TIA-942
 4-Tier System based on
 Resilience / Capacity of its MEP systems
 16-pages of criteria
 Primary Categories
  Power and cooling delivery   Support space to raised floor
paths ratio
  Redundancy in components   Raised floor height
  Initial & ultimate watts/sqft   Floor loading pounds/sqft
  Utility voltage
Optimal Criticality – Choosing a tier
C Business characteristics Effect on system design

1
• Typically small businesses • Numerous single points of failure in all aspects of
• Limited online presence design
• Low dependence on IT • No generator if UPS has 8 minutes of backup time
• Perceive downtime as a tolerable • Generally unable to sustain more than a 10 minute
Inconvenience power outage

2
• Some online revenue generation • Some redundancy in power and cooling systems
• Multiple servers • Generator backup
• Phone system vital to business • Able to sustain 24 hour power outage
• Dependent on email • Minimal thought to site selection
• Some tolerance to scheduled downtime • Vapor barrier
• Formal data room separate from other areas

3
• World-wide presence • Two utility paths (active and passive)
• Majority of revenue from online business • Redundant power and cooling systems
• VoIP phone system • Redundant service providers
• High dependence on IT • Able to sustain 72-hour power outage
• High cost of downtime • Careful site selection planning
• Highly recognized brand • One-hour fire rating
• Allows for concurrent maintenance

4
• Multi-million dollar business • Two independent utility paths
• Maj. of rev from electronic transactions • 2N power and cooling systems
• Business model entirely dependent on IT • Able to sustain 96 hour power outage
source: apc.com
• Extremely high cost of downtime • Stringent site selection criteria
• Minimum two-hour fire rating; High phy. security
 Balance cost of downtime and TCO • 24/7 onsite maintenance staff
 Tier System
Attribute / Statistic Tier I Tier II Tier III Tier IV
Power and Cooling 1 Active 1 Active 1 Active 1 2 Active
Delivery Paths Passive
Redundant Components N N+1 N+1 2(N + 1)
Support Space to Raised 20% 30% 80 – 90% 100%
Floor Ratio
Initial Watts / sqft 20 – 30 40 – 50 40 – 60 50 – 80
Ultimate Watts / sqft 20 – 30 40 – 50 100 – 150 150+
Raised Floor Height 12” 18” 30 – 36” 30 – 36”
Floor Loading Pounds / 85 100 150 150+
sqft
Utility Voltage 208, 480 208, 480 12 – 15 kV 12 – 15 kV
Months to Implement 3 3–6 15 – 20 15 – 20
Year First Deployed 1965 1970 1985 1995
Construction $ / sqft $450 $600 $900 $1,100+
Annual IT Downtime Due 28.8 hrs 22.0 hrs 1.6 hrs 0.4 hrs
to Site
Site Availability 99.67% 99.75% 99.98% 100.00%

Source: The Uptime Institute

Next / Action Steps
 Perform a risk assessment to determine
hourly cost of downtime
 Determine current data center tier for
each subsystem in the data center –
you might not need tier-iv across the
board
 Get IT and Facilities to work together on
issues
 Work with finance to justify upgrades
 Understand TIA-942 (requirement &
process)
Outsourced Data Center
 Fits business model - consider
outsourcing
 Affordable co-location/hosted DC and
99.995% uptime are NOT mutually
exclusive
 Understand levels of redundancy and
the uptime SLA in order to get the best
combination of uptime and affordability
 Balance between budget and availability
Outsourced Data Center
 What to look for….
  Hardened data center   Availability of multiple
buildings carriers
  Data center power &   Physical security
cooling redundancy   SAS 70 data center
  Telecom entrance compliance
redundancy

 Claims of Uptime Tiers – III or IV; most

are not certified
Review
  TIA-942
  Key design parameters
  Tier System
  Next Steps
Resources
  Useful links
  Excellent white papers from www.apc.com
  TIA - http://www.tiaonline.org/
  Green data center efficiency savings calculator
http://cooling.thegreengrid.org/namerica/WEB_APP/calc_index.html

  The green grid (thegreengrid.org)

  Department of Energy – DC Profiling Tool
http://www1.eere.energy.gov/industry/datacenters/software.html

  …. and obviously Google or Bing it.

Questions
plantemoran.com

Contact Information

Sri Chalasani
[email protected]
248.223.3707